Seven Alerts at 2 AM: Building an AI On-Call Agent with a Graph Database

It's 2 AM. Your phone buzzes, then buzzes again. Seven alerts in four minutes.

Frontend says Orders failing. The API gateway is throwing 5xx errors. Checkout is erroring, payment is slow, cart is slow. Somewhere near the bottom there's a quiet warning on the cache, and a High CPU alert on the recommendation service.

Most people start with the loudest one. That's usually wrong. The loud alerts are symptoms. The real cause is often a small change on a shared service that everything else depends on. Finding it means connecting a lot of dots: which service calls which, who deployed what and when, who owns it, and what fixed this last time.

That's what I built for Graph Hacks (WeMakeDevs × FalkorDB): RootCause Agent, an AI on-call agent that finds the real cause of an outage by walking a graph.

🔗 Live app: https://rootcauseagent.streamlit.app 💻 Code: https://github.com/beyondions/rootcause-agent 🎥 Demo: https://www.youtube.com/watch?v=YDL6ZGoFmOY

Four days ago I had never written a line of Cypher. I didn't know how an AI agent calls tools. I'd heard "graph database" but couldn't have told you why you'd pick one over Postgres.

I chose Track 1, Agents That Act on Connected Data, because on-call incidents felt like the clearest case where the answer lives in the connections, not in any single document.

Think about what an on-call engineer actually needs to know:

Every one of those is a path through connected data. So I modelled it as a graph in FalkorDB: 25 services of an online shop, plus teams, engineers, deploys, alerts, past incidents and runbooks, joined by relationships like DEPENDS_ON, OWNS, AUTHORED, DEPLOYED_TO, FIRED_ON and RESOLVED. All the data is synthetic.

The rule turned out to be simple once I saw it as a graph:

An alerting service is a symptom if something it depends on is also alerting. What's left are the candidates, ranked by how many alerting services depend on them, and then by whether they had a recent deploy.

In my scenario, frontend, API gateway, checkout, payment and cart are all symptoms, because each of them sits upstream of another alerting service. Two candidates are left:

So the answer is cache, and the deploy behind it is D-1: "Config change: maxmemory lowered". The recommendation CPU alert is a red herring.

The core of it is one line of Cypher, a variable-length traversal:

MATCH (:Service { name: $service }) - [:DEPENDS_ON * 1.. ] -> ( dep ) RETURN DISTINCT dep.name Enter fullscreen mode Exit fullscreen mode *1.. means "follow DEPENDS_ON one or more hops". In SQL that would be recursive joins. In Cypher it's two characters.

From there, the agent also finds the evidence path (frontend → API gateway → cart → cache), the blast radius (5 critical services), who to wake up (Priya, who fixed the same problem before, and Arjun, who made the change), the fix from the past incident's runbook, and then it acts: it writes a new incident back into the graph.

The first versions of the agent let the LLM write the whole answer. It was fluent and confident, and sometimes wrong.

The moment that changed my approach: the agent suggested a fix called "Restore Cache Memory Settings". That runbook doesn't exist. The real one is "Roll back cache config". My tests didn't catch it, because they only checked the root cause.

At 2 AM, a made-up runbook name is worse than no answer. So I set a rule:

Facts come from the graph. Structure is enforced in code. The LLM only explains.

The LLM still decides which graph tools to call and writes the explanation. But the lines that matter (the first line, the suggested fix, yes/no answers like "is recommendation the cause?") are rebuilt in code from the tool results before anyone sees them. [Adjust if needed: the contact line got the same treatment after the model once invented a team name.]

It's easy to say a graph helps. I wanted to measure it.

So I built a vector-RAG baseline over the same data: every node and relationship turned into a sentence, TF-IDF retrieval with stopwords and stemming, the current alerts added to the query, top 8 documents, same LLM. I tried hard to make it fair. My first version was accidentally unfair, and when I fixed it, RAG started finding cache too. That changed my pitch from "RAG gets it wrong" to "here's what RAG can't do".

Then I scored both against an answer key read from the graph, not typed in by hand:

RAG could guess that the cache was involved. But it blamed the API gateway as the primary driver, it never found deploy D-1 or who made it, it couldn't list the full blast radius, it couldn't say who to wake up and why, and it can't take action. Asked "what changed?", it answered an increase in cache latency, which is an alert, not a change.

Same data, same model. The only difference is the graph.

My first root-cause query returned all zeros. I tried to do everything in one big Cypher query. I split it into simple traversals and combined the results in Python, and it worked straight away.

shortestPath inside MATCH threw an error. In FalkorDB it has to go in RETURN or WITH. That took me a while to figure out.

My blast radius was hard-coded in the tests. I noticed when I asked myself why it never changed. Now every expected value comes from the graph.

The agent treated every question as the current outage. I typed "auth is not working, who is responsible?" and it blamed the cache, woke up the wrong people, and opened an incident. Auth wasn't even alerting. That led to service-check mode: if you ask about a service with no alerts, the agent checks its dependencies in the graph, flags a possible monitoring gap, names the owners, and opens nothing. It uses zero LLM tokens.

I ran out of tokens. Groq's free tier gives about 200k tokens per model per day, and a full investigation costs about 11k. Two test runs used up a whole day. I added a model fallback chain, an answer cache, and token logging, so the demo keeps working.

Deployment had its own surprises. My.env still pointed at localhost while I was trying to reach FalkorDB Cloud, and I once pasted the port into the host field, so it tried to connect to...cloud:59676:59676.

🔗 Live app: ( https://rootcauseagent.streamlit.app ) 💻 Code: https://github.com/beyondions/rootcause-agent 🎥 Demo: https://www.youtube.com/watch?v=YDL6ZGoFmOY

Try "Orders are failing. What's the root cause and who do I wake up?", then "Is postgres healthy?"

Right now it's one scripted outage on synthetic data with a fixed clock. The real next step is feeding it a real service map (for example from OpenTelemetry) and real deploy history, then seeing whether the same rule holds up.

Four days ago I didn't know Cypher. Now I understand why some questions are graph questions, and why an agent that answers at 2 AM should take its facts from data, not from the model's imagination.

Thanks to WeMakeDevs and FalkorDB for the push.