How to run AI security code review on GitHub pull requests without alert fatigue
To run AI security code review on GitHub pull requests without drowning in false alarms, run deterministic rules first for hard stops (secrets, removed CI safety guards, and protected paths), then use an evidence-gated security checklist that only raises a pull request's risk lane when the model qu
Posted on Oct 24 • Originally published at mergejudge.si
To run AI security code review on GitHub pull requests without drowning in false alarms, run deterministic rules first for hard stops (secrets, removed CI safety guards, and protected paths), then use an evidence-gated security checklist that only raises a pull request's risk lane when the model quotes an exact line from the diff.
MergeJudge is an AI pull request reviewer with an open-source engine for GitHub. Every pull request is sorted into an auto, review or block lane: deterministic rules set the floor first, and the AI step can only raise a lane. auto means low risk, not a guarantee; branch protection and required checks still apply, and MergeJudge doesn't merge pull requests itself.
Before any model is called, MergeJudge runs deterministic checks on the pull request diff using the.mergejudge.yml file from the base branch:
Because.mergejudge.yml is read from the base branch tip, a pull request cannot edit.mergejudge.yml to remove a protected path or turn off a check for itself.
For teams that want a closer security audit on every pull request—especially when reviewing AI-agent pull requests or backend, webhook, and auth changes—MergeJudge includes an optional Security review mode. It is off by default.
Turn it on in.mergejudge.yml on your default branch:
#.mergejudge.yml on the default branch review: mode: context # diff | context | deep security: true # default false; only a real boolean true turns it on Enter fullscreen mode Exit fullscreen mode On the hosted GitHub App, you can also flip the Security review switch in the Set up wizard or Edit settings in the panel; it opens a setup pull request on mergejudge/setup-config for you to merge. On the GitHub Action and runner, add review.security: true to.mergejudge.yml directly.
When review.security: true is on, MergeJudge scores five high-severity attack-class criteria alongside the normal rubric (and overrides rubrics.enabled: false if rubrics were turned off):
A common failure mode of LLM security reviews is flagging theoretical hardening wishes or guessing about middleware that lives outside the diff. MergeJudge constrains sec.audit.* in two ways:
The five sec.audit.* criteria add about 500 tokens to every review prompt, counted against budget.max_tokens_per_pr.
Set review.security: true in.mergejudge.yml on your repository's default branch, or toggle the Security review switch in the MergeJudge panel under Set up or Edit settings and merge the resulting pull request. It is off by default, and only a real boolean true turns it on.
When review.security: true is set, MergeJudge scores five high-severity attack-class criteria alongside its normal rubric: sec.audit.injection (untrusted input reaching SQL, shell, eval, HTML, path, redirect or deserialization sinks), sec.audit.authz (missing authentication, IDOR or dropped role/tenant checks), sec.audit.secrets-crypto (leaked credentials, weak hashes or ciphers, == on secrets, disabled TLS verification), sec.audit.callbacks (unverified webhooks or SSRF), and sec.audit.ai-tools (untrusted text or model output gaining shell, file, network or secret access when the diff touches agents, tools or prompts).
Every high-severity rubric failure must include an exact quote that MergeJudge verifies against the pull request diff. A finding with a verified diff quote is marked confirmed ( ❌ ) and raises the lane to review. If the model suspects a trust-boundary break that depends on code outside the diff and cannot quote a changed line, MergeJudge lists it with a question-mark icon ( ❔ ) as needs validation: it leaves the lane unchanged so a human can check it, while still keeping opt-in lanes.code_auto off for that pull request.
No. MergeJudge always reads.mergejudge.yml and policy files such as AGENTS.md and REVIEW_POLICY.md from the pull request's base branch tip, never from the pull request head. Any pull request that edits.mergejudge.yml is placed in the review lane.
No. The five sec.audit.* criteria add about 500 tokens to every review prompt and count against budget.max_tokens_per_pr. The free AI tier on public repositories keeps its usual token, diff-size and monthly review caps. For larger pull requests or deeper cross-file checks, use the open-source runner on your machine or your own provider API key.