How we use an LLM to triage SAST findings, and what we never send it

Static analysis tools are pattern matchers. That's their strength (fast, deterministic, explainable) and their weakness: a rule that matches string-built SQL can't tell whether the string came from a user or a constant. So you get findings that are technically correct and practically irrelevant, and teams learn to ignore the whole report.

In OpenRouting we keep the scanner as the source of truth and add a review step: Claude, via Amazon Bedrock, looks at each finding with the surrounding code and gives a verdict. This post walks through how that step is built. The design choices matter more than the model.

The model never goes looking for vulnerabilities on its own. Semgrep finds them, every finding is normalised into one schema (rule id, scanner, severity, CWE, file, line range, snippet, message), and only then does the model review it. That gives two properties we care about:

The output is structured, not prose: true_positive or likely_false_positive, a confidence between 0 and 1, a two-to-four-sentence explanation, and for real issues a minimal corrected version of the flagged lines.

One instruction in the prompt does a lot of work:

When unsure, prefer true_positive; a missed vulnerability costs more than a review.

Triage that hides real issues is worse than no triage. The model is allowed to clear a finding only when the code shows why it's safe (a constant, a bound parameter, test fixture code), and it has to say what that reason is.

A generic "you are a security expert" prompt gives generic answers. So each finding is routed to a specialist reviewer whose instructions describe what to look for in that category, what makes it a false positive, and how to fix it.

The reviewers are plain Markdown files with front matter, one folder each:

--- name: SQL injection key: sqli cwes: [ 89, 90, 91, 943 ] keywords: [ sqli, sql-injection, sqlalchemy, raw-query ] --- What to look for... What makes it safe... How to fix it... Enter fullscreen mode Exit fullscreen mode Routing is a simple, predictable order:

The nice property: adding a reviewer is adding a Markdown file. No code change, and the dashboard shows which reviewer judged each finding.

Scans repeat. The same finding shows up in the next scan, and the one after that. Re-analysing it each time would burn money and could flip verdicts for no reason.

code = re. sub ( r " \s+ ", " ", snippet ). strip () key = " \x1f ". join ([ scanner, rule_id, path, code ]) fingerprint = hashlib. sha256 ( key. encode ()). hexdigest () Enter fullscreen mode Exit fullscreen mode It hashes the code, not the line number, so a finding keeps its identity when unrelated edits shift it up or down the file. Findings are stored once per repository by fingerprint, and the review step only picks up open findings that don't have an analysis yet. A finding you saw last week costs nothing the next time it appears, and its triage history stays attached.

If a model call fails, the finding is left without an analysis instead of being guessed at, so it's still visible and the next scan retries it.

This is the part I'd want to know if I were evaluating a tool like this:

On one example run against a deliberately vulnerable Flask app ( we45/Vulnerable-Flask-App ), Semgrep produced 29 results, which became 28 findings, 8 of them high, in 18.4 seconds. Among them was a CWE-250 finding on the Dockerfile: the container runs as root. One run on one intentionally broken repo isn't a benchmark, so I won't claim a false-positive rate from it. It does show the shape of the output: each finding with a verdict, an explanation tied to the actual code, and a fix.

If you want to try it, OpenRouting gives you 100 free credits, enough for five full scans. The review agents page lists every specialist and what it treats as a false positive. Questions about the design are very welcome in the comments.