An agent that refuses to guess: building RepoGraph AI on FalkorDB

When a new bug report lands in a large open-source project, the first question is rarely "How do I fix this?"

I contribute to Oppia, an open-source learning platform with thousands of files. For Graph Hacks (WeMakeDevs × FalkorDB), I built RepoGraph AI: give it an Oppia issue, and it ranks the files worth investigating, suggests tests to run, and shows the graph evidence behind its recommendations.

The important part is what happens when the evidence runs out: RepoGraph AI is designed to say so rather than inventing a confident answer.

Keyword search over source code is a strong baseline. But some useful clues about a bug aren't obvious from the words in its description. They may be hidden in the project's history: a related issue fixed a similar problem, a previous commit changed the relevant file, or a code relationship points to another place worth investigating.

RepoGraph AI uses FalkorDB to connect issues, commits, files, functions, tests, and other code relationships. Instead of returning only a ranked list, it also exposes the path behind a recommendation, so a developer can inspect why a file appeared.

This doesn't mean a graph is always better than keyword search. In my evaluation, both methods found a relevant file in the top five for 14 of 21 held-out issues. On a second set of 29 issues selected by a precommitted rule, keyword search hit 18, while graph-enhanced search hit 20.

Those results are promising, but modest. The graph alone performed worse than keyword search, and some issues still have insufficient evidence. The goal is not to claim that graphs solve issue localization; it's to test whether connected historical and structural evidence can make the investigation more useful and explainable.

The graph has about 25,700 nodes and 39,000 edges, built from three sources:

Every edge stores a provenance property: git_commit, github_parent, todo_comment, python_ast, registry_mapping, route_table. These sources are not equally reliable. A TODO(#issue) written by a developer is strong evidence; two files that often change in the same commits are only a correlation. So the app labels each piece of evidence as direct, related, structural or weak. An edge records a link; it is not proof that two files are related in meaning.

One of the core queries finds files changed by the fix of an issue's parent:

MATCH (:Issue { number: $n }) - [:CHILD_OF ] -> ( p: Issue ) <- [:FIXES ] - ( c: Commit ) - [:TOUCHES ] -> ( f: File ) WHERE c.date < $cutoff RETURN p.number, c.sha, c.date, f.path Enter fullscreen mode Exit fullscreen mode The WHERE c.date < $cutoff line matters. For an issue that is already fixed, the graph only sees history from before its fix. Otherwise the fix commit itself would point straight at the answer, and every result would be inflated.

Issue #25307 asks for email retries to stop on 4xx errors. It names no file.

#25307 → CHILD_OF → #18276 ← FIXES ← commit 2405253 → TOUCHES → core/controllers/tasks.py

A TODO(#25307) comment in the same file gives a second, independent path. Using only history from before the fix, core/controllers/tasks.py is ranked first, and the real fix did change it. The app also suggests the test to run first: core.controllers.tasks_test.

Not every path leads somewhere useful. For #18371, keyword search ranked the fixed file 3rd. Adding graph evidence pushed it down to 22nd: the graph found strong-looking links into a different part of the code, and they outweighed the keyword signal. For #24470, every method missed.

These failures shaped one design decision: a gated mode that adds graph evidence only when the graph has direct support (the issue names a file, a TODO comment, or a related issue's fix). On the 29-issue set, the gated mode turned one miss into a hit and never turned a hit into a miss.

On top of the graph sits a Gemini agent with three tools: related issues, file context, and keyword search. Three mechanisms keep it honest:

The triage agent writes each plan back into FalkorDB as (:Investigation)-[:SUGGESTS]->(:File). The investigation code never reads these nodes, so the agent's output can never become evidence for itself. Re-running the evaluation after the agent wrote its plans gave identical results.

For one open issue, #27595, the agent pointed to a specific function in email_services.py. I checked the code myself: the email regex really does run on the unstripped address. The pointer was right.

hit@5 means at least one file changed by the real fix is in the top five. Keyword search runs on the code as it was just before each fix, and the selection rule for the 29-issue set was committed to git before the script was first run. The biggest single win was #24156: keyword search ranked the fixed file 28th; with the graph it was 5th, thanks to a sibling issue's fix.

The evaluation dataset, with frozen issue texts, fix commits and file labels, is in the repository so anyone can check these numbers.

The lesson I'm taking away: the most useful thing an investigation tool can say is not just where to look, but why, and when it honestly doesn't know.

Built with FalkorDB for Graph Hacks. I used Claude as a coding and writing assistant; Gemini powers the agent inside the product.