The Problem With Using AI Agents to Evaluate Other AI Agents

If you build agents, you have probably wired one model up to grade another. It is fast, it scales, and it is how most agent evals work now. I recently submitted a public comment to an FDA docket on how generative AI medical devices should be evaluated, and the part I spent the most time on was the part developers tend to skip: who is allowed to be the judge.

This post is the engineering version of that comment. The healthcare setting raises the stakes, but the failure modes apply to any agent you grade with another agent. FDA states that its discussion paper (docket FDA-2026-N-7874) is for discussion only and is not guidance, so none of this is a requirement. It is my argument, with sources where they exist. You can read the full comment on regulations.gov.

Disclosure: I'm building Orinyx, which is in this space. Treat that as an interest.

Self-preference is documented. Panickssery et al. found an LLM evaluator scoring its own generations higher than human annotators did, and that the effect strengthened with the model's ability to recognize its own output. Wataoka et al. reported significant self-preference in GPT-4 and tied it to familiarity: judges rated lower-perplexity text higher than humans did, whether or not they wrote it.

What this does not show is that same-family judging fails on your task. Neither paper tested a deployed clinical device. My position is a design rule: do not let the judge be the same model family, or a fine-tuned variant, as the system under test. A GPT-based agent graded by a GPT-based judge is not an independent check just because two different teams ran it. LLM judges are still a good tool: the MT-Bench paper reports strong judges agreeing with human preferences more than 80% of the time. The rule is about which seat they sit in, not whether they work.

Instead of one model judging another, split the job so that anything a machine can verify is verified without a model. This is the pattern I'm developing and testing. It runs four narrow evaluators against the same output:

Keep the deterministic layer deterministic on purpose. It has to be auditable and it has to fail predictably. Only the judgment-dependent slice goes to a model. This narrows what you have to trust; it does not remove the question.

Evaluating input against output misses failures that happen mid-process: an agent reaches a correct-looking conclusion through a reasoning chain that should have triggered escalation earlier, or through a tool it was not authorized to use. Score the trajectory.

The research agrees directionally. A 2025 paper on reasoning trajectories of tool-augmented agents found that agents with similar answer accuracy differed significantly once trajectories were scored for efficiency, hallucination, and adaptivity. A survey of LLM-agent evaluation maps how the field is moving from scoring answers to scoring behavior.

A minimal shape for a trace you can evaluate (illustrative, not a spec):

{ "session_id": "s-0042", "step": 3, "tool": "chart_write", "authorized_at_session_start": false, "scope_before": [ "chart_read" ], "scope_after": [ "chart_read", "chart_write" ], "tool_output_source": "same_vendor_as_agent", "logged_by": "external_telemetry" } Enter fullscreen mode Exit fullscreen mode Two fields carry most of the weight. logged_by matters because I argued, and I stand by it, that agents under-report their own failures, so tool-call results should be logged independently of the agent's self-reported status. scope_before and scope_after matter because authority can expand mid-session, and an evaluation should test whether the agent recognized its available scope changed, not only whether it refused out-of-scope actions.

An agent that treats a tool's output as ground truth has the same self-affirmation problem as a system grading itself. The call looks like an external check, but if the tool and the agent share a developer, a training lineage, or an incentive structure, it is not one. Tag tool outputs by whether the source is structurally independent of the agent, and weight them accordingly.

If a supervisory agent benchmarks a worker agent, something has to benchmark the supervisor. The chain has to terminate in a structurally independent, non-agentic check. Otherwise you have a stack of agents each vouching for the one below, and accountability regresses one level at a time. I also suggested capping turns and per-session actions so a supervisor's scope is bounded and auditable.

My FDA comment describes a longer-term architecture: continuous per-output verification of live output against current authoritative sources, integrated through CDS Hooks.

The Orinyx MCP launches October 29, 2026. It runs on synthetic data only, uses simplified bounty agents, has no dashboard, no CDS Hooks or EHR integration, and does not monitor live clinical output. If you want a launch notice, the waitlist is at orinyx.io/developers.

The full write-up, with sources and FAQ, is at orinyx.io/blog/who-verifies-the-verifier.

Which of these four evaluators does your current agent eval already have, and which one is a model grading itself?