auditsift: security-audit every PR with an AI agent, without drowning in false positives

Repo: 👉 github.com/bayuh85/auditsift (MIT, Python 3.10+, 64 offline tests)

Ask a coding agent to "review this PR for security issues" and you usually get back a long, confident list. Some of it is real. A lot of it is plausible-sounding noise: an "SQL injection" in a query that is already parameterized, a "hardcoded secret" that is a test fixture, severities that change every time you ask. There is no file and line you can trust, and nothing your CI can parse.

Then there is the bill. Running a flagship model over every pull request "just in case" is the kind of habit teams drop after the first invoice.

What I wanted was boring and reliable: cheap offline checks first, a model only for the handful of candidates worth explaining, and a second opinion that has to quote the actual code before a finding counts.

auditsift runs an audit in phases. The first two need no API key and never touch the network.

The verify step is the part I care about most. It doesn't see the triage write-up, so it can't just agree with it. A "confirmed" verdict only counts when the quoted evidence is found in the code. Rejected findings are dropped from SARIF and counted in the summary, so you can see how much noise was removed.

The repo ships a deliberately broken sample app in examples/flask-notes. Here is the offline scan:

python -m venv.venv && source.venv/bin/activate pip install -e ".[dev]" auditsift scan examples/flask-notes Enter fullscreen mode Exit fullscreen mode auditsift examples/flask-notes 13 findings kept, 0 rejected high AS-SQLI examples/flask-notes/app.py:28 SQL query is built with string interpolation. impact: An attacker who controls the interpolated value can change the query. fix: Use a parameterized query and pass values as bound parameters. high AS-DESER examples/flask-notes/app.py:41 Untrusted data can reach yaml.load. fix: Use yaml.safe_load, or a format such as JSON that does not construct objects. high AS-SSRF examples/flask-notes/app.py:53 Outbound request target includes external input (target). high AS-TLS examples/flask-notes/app.py:53 TLS certificate verification is disabled. high AS-SHELL examples/flask-notes/app.py:60 Shell command is built from external input.... cost estimate map local 0 in / 0 out (0 calls) scan local 0 in / 0 out (0 calls) Enter fullscreen mode Exit fullscreen mode Secrets are redacted before they are printed, written to a report or sent to a model.

For a pull request you only care about the lines that changed, so --diff filters findings to the git diff against a ref:

auditsift audit. --diff origin/main --fail-on high --out results Enter fullscreen mode Exit fullscreen mode Exit code 1 means a kept finding is at or above the gate, so CI blocks the merge. results/ holds auditsift.json, auditsift.sarif and auditsift.md.

name: auditsift sarif on: pull_request: permissions: contents: read security-events: write jobs: audit: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: fetch-depth: 0 - uses: actions/setup-python@v5 with: python-version: " 3.12" - run: pip install. - run: > auditsift audit. --diff origin/${{ github.base_ref }} --offline --fail-on high --out results - uses: github/codeql-action/upload-sarif@v3 if: always() with: sarif_file: results/auditsift.sarif Enter fullscreen mode Exit fullscreen mode --offline keeps the required check free. When you want explanations and a second opinion on the diff, add your key as a repo secret and drop --offline. Findings show up in the Security tab like any other code-scanning tool.

auditsift keeps the model work small on purpose. The scan narrows a whole repo down to a few snippets, and only those go out. --max-llm caps how many candidates are sent at all. Every run prints a token ledger per phase, so you can see exactly what triage and verify cost.

The default endpoint is AtmoRouter, a cheap AI API gateway and OpenRouter alternative. That matters here because the setup wants two different models through one key: a decent triage model and a cheaper verify model.

export ATMOROUTER_API_KEY = ar-your-key-here export AUDITSIFT_TRIAGE_MODEL = openai/gpt-4o-mini export AUDITSIFT_VERIFY_MODEL = openai/gpt-4.1-nano auditsift audit examples/flask-notes --fail-on high --out results Enter fullscreen mode Exit fullscreen mode AtmoRouter gives you 109 models behind one OpenAI- and Anthropic-compatible key, pay per token, at an average of 83.7% below official prices. Running two passes on every PR stops being a budget conversation. If you'd rather use another provider, set AUDITSIFT_BASE_URL and AUDITSIFT_API_KEY to any OpenAI-compatible endpoint, or AUDITSIFT_API_STYLE=anthropic for /v1/messages.

auditsift serve is an MCP server over stdio with auditsift_map, auditsift_scan and auditsift_audit. Map and scan stay offline, and auditsift_audit only calls models when the agent passes use_llm: true, so an agent doesn't spend tokens by accident.

{ "mcpServers": { "auditsift": { "command": "auditsift", "args": [ "serve" ] } } } Enter fullscreen mode Exit fullscreen mode claude mcp add auditsift -- auditsift serve Enter fullscreen mode Exit fullscreen mode A good agent loop: scan the diff, read the candidates, and only ask for the paid audit on the ones that look real.

It's MIT licensed and the rule set is small and readable, so adding a rule for your own framework is a short PR. If it catches something real in your repo, a star on GitHub helps other people find it.