An AI summary is not a fix. Stop letting it close your security findings.

Static analysis and LLMs are a good pairing. I'm not here to argue otherwise. But a lot of teams are wiring them together in a way that quietly turns the scanner's evidence into the AI's opinion, and then treating the opinion as the record.

In the demo, it looks great. The scanner dumps 400 alerts on a pull request. An assistant reads them, groups them into twelve clusters, writes a plain-English paragraph for each, and marks the ones it thinks are noise. The reviewer sees twelve tidy cards instead of a wall of rule IDs. Everyone relaxes.

In the pipeline, six months later, nobody can tell you why a SQL injection finding in orders/export.py was closed. The ticket says "Resolved: input is validated upstream." There's no link to the upstream validation. The original rule ID, file, line, and data-flow trace are gone, because the summary replaced them.

The scanner and the assistant have different jobs. The scanner says: this rule was violated at this location, and here is the path that got there. The assistant can say: here's what that means, here are three other alerts that are probably the same root cause, and here's where I'd look first. Problems start when the second job silently absorbs the first.

Illustrative example. A taint-tracking rule flags this:

def export ( request ): fmt = request. GET. get ( " format ", " csv " ) query = f " SELECT * FROM orders WHERE tenant = ' { request. user. tenant } ' ORDER BY { fmt } " return run ( query ) Enter fullscreen mode Exit fullscreen mode The assistant reads it and writes: "The tenant value comes from the authenticated user object and is not user-controlled. Likely false positive." That sentence is correct about tenant. It says nothing about fmt, which is the actual sink the rule fired on. The summary was plausible, focused on the wrong variable, and got accepted because it read well.

An explanation is a hypothesis about a finding. It is not evidence that the finding is resolved.

Clustering is genuinely useful. Forty "hardcoded credential" alerts that are all test fixtures should collapse into one decision.

But clusters get dismissed as a unit. If 39 of those 40 are fixtures and one is a real key in deploy/config.py, a "dismiss cluster" button throws it out with the rest. The grouping logic needs to show you what makes each member similar, and the dismissal needs to be recorded per finding, not per cluster.

If your triage flow stores "AI says false positive" and drops the rule ID, location, trace, and the code snapshot it was judged against, you can't audit it later. You can't rerun the check against the fixed code. You can't notice that the same pattern came back in a refactor.

Keep the original finding as the source of truth. Attach the AI explanation next to it, labeled as an explanation. Close the finding only when a rescan of the changed code no longer reports it, or when a human records a reason that points at something concrete.

"The assistant processed 12,000 alerts this quarter" is a volume metric. It tells you nothing about whether the right ones were dismissed.

The number that matters is false dismissals: findings the AI (or a human following the AI) closed that were real. You only learn that rate by sampling closed findings and re-reviewing them, or by tracking findings that were dismissed and later reappeared as incidents or re-detections. If you aren't measuring it, you don't know whether your triage got better or just got faster.

None of this is exotic. It's the same discipline you'd apply to a junior reviewer: helpful, fast, often right, and not the one who gets to sign off alone.

AI really does make security signals easier to understand. The evidence still has to survive the summary.

How are you handling AI-assisted triage on your team? Do you track false dismissals at all, or is "alerts processed" the only number anyone looks at?