Pass-Raten über Wiederholungen: KI-Agenten richtig evaluieren

Posted on Oct 27 • Originally published at en.aobaworks.com

Track three AI agent evaluation metrics for Claude Code runs that nobody watches. The first is **pass@k**: can the agent solve the task at all within k tries? The second is **pass^k**, or pass rate across repeat runs: does it succeed every time?

The third is **graded-score drift**: is the quality score from your grader slowly falling on input that hasn't changed? Run each task 10–20 times in a clean environment and log a pass/fail result plus a score for every run. Then compute these numbers before you let a cron job run the agent alone.

Anthropic defines the second metric as one that "measures the probability that all k trials succeed" ( [ Anthropic: Demystifying evals for AI agents ]( https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents ) ). That's the question an overnight automation actually asks. The [ pillar guide on the eval loop ]( https://en.aobaworks.com/ai-agent-evaluation-framework-claude-code-eval-loop-steps/ ) names these metrics.

This guide shows how to choose and compute them. ## Which option is right for you? If you're starting out, use a bash loop around `claude -p` plus a short Python script. Move to a framework only once you have more than a handful of tasks. | Your situation | What to pick first | First page to open | |---|---|---| | Beginner, one or two recurring tasks | Bash loop + `claude -p --output-format json` + pass rate over 10 reruns | [ Claude Code headless docs ]( https://code.claude.com/docs/en/headless ) | | Daily work use, cron jobs already running | pass^k on a fixed set of 20–50 tasks, gated before every prompt/model change | [ Anthropic: Demystifying evals for AI agents ]( https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents ) | | Many tasks, want a results table and assertions | Promptfoo with `--repeat` and the `anthropic:claude-agent-sdk` provider | [ Promptfoo: Evaluate Coding Agents ]( https://www.promptfoo.dev/docs/guides/evaluate-coding-agents/ ) | | Tight budget | Code-based graders only (tests, file checks); skip LLM judges at first | [ Promptfoo command line docs ]( https://www.promptfoo.dev/docs/usage/command-line/ ) | | Output is prose or reviews, not code | Add an LLM-judge score and track its drift | Section on graded-score drift below | Here's why each pick fits.

A bash loop teaches you what the numbers mean before a tool hides them. pass^k is the metric that matches "runs without me". Promptfoo saves you from writing your own repeat-and-cache plumbing. Code graders cost nothing per run.

And prose can't be checked by `npm test`. ## What Are Evaluation Metrics in AI Agent Testing? In Claude Code testing, an evaluation metric is a rule that turns each run's result into a pass/fail or a score. That lets you compare runs instead of eyeballing transcripts.

Anthropic's evals post uses four words you'll need throughout this guide: - A **task** is " [ a single test with defined inputs and success criteria ]( https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents ) ". - A **trial** is one attempt at a task. You run several, " [ because model outputs vary between runs ]( https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents ) ". - A **grader** is the logic that scores the trial. - The **outcome** is the final state of the environment when the trial ends. A metric sits on top of these.

It's what you get when you add up grader results across trials and tasks. For example, say your nightly job asks Claude Code to "bump the patch version, update CHANGELOG.md, and make the test suite pass". The task is that prompt plus a frozen copy of the repo.

A grader might check three things: `npm test` exits 0, `package.json` has the expected version, and `CHANGELOG.md` has a new top entry. One trial gives one pass/fail. Ten trials give a pass rate.

That pass rate is the metric. **Pitfalls:** - **Treating the exit code as the metric.** Claude Code exits with code 0 on success ( [ headless docs ]( https://code.claude.com/docs/en/headless ) ). That means the run ended normally, not that the work is right. A run can exit 0 after editing the wrong file.

Always grade the outcome (tests, files, diff) separately from the exit status. - **Scoring the transcript text.** A confident summary like "All tests pass!" isn't evidence. Run the tests yourself in the grader. - **No reference solution.** Anthropic recommends tasks that are unambiguous, with a known solution that proves they can be solved. If you can't write the passing answer yourself, the metric can't tell a model failure from a broken task. ## What Is Evaluation Metrics in Machine Learning (and How Agent Evals Differ)?

Classic machine learning metrics score one fixed prediction against a fixed label. Agent evals for tools like Claude Code score multi-step runs that can succeed in different ways or fail quietly partway through. scikit-learn describes [ accuracy ]( https://scikit-learn.org/stable/modules/model_evaluation.html ) as "the fraction (default) or the count (normalize=False) of correct predictions". Precision, recall and F1 refine that for imbalanced classes.

These metrics assume something that agent runs break: the same input always produces the same output, and there's one correct label to compare against. | Property | Classic ML metrics (accuracy, F1) | Coding-agent evals (Claude Code) | |---|---|---| | Unit scored | One prediction | A whole trial: tool calls, edits, final repo state | | Determinism | A fixed model gives the same prediction | Same prompt, different paths and results per run | | Ground truth | A label | A checkable end state (tests pass, file exists, no stray changes) | | Valid answers | Usually one | Many diffs can be correct | | Repeats needed | Once per example | Multiple trials per task to estimate a rate | | Silent failure mode | Wrong label | Plausible summary over a broken or partial change | The practical difference is the repeats. A classifier's accuracy on a test set is fixed once you freeze the model. A Claude Code task's pass rate is a probability, and you can only estimate it by sampling.

That's why the Codex paper that made pass@k popular reported two numbers. Its model solved 28.8% of HumanEval problems with 1 sample per problem ( [ Chen et al. ]( https://arxiv.org/abs/2107.03374 ) ), and " [ 70.2% of our problems with 100 samples per problem ]( https://arxiv.org/abs/2107.03374 ) ". Same model, same problems.

The only difference was how many tries it got. **Pitfall:** Don't borrow precision/recall unless your agent really is classifying something. A triage agent that labels issues fits that frame. An agent that writes code needs pass rates. ## What Is Evaluation in AI (Agent Context)?

For Claude Code, evaluation means rerunning a task many times and checking the final state it leaves. A plausible-looking output isn't enough; the outcome has to match what you intended. Anthropic's rule for this is to " [ grade what the agent produced, not the path it took ]( https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents ) ".

If Claude fixes a bug by editing `utils.ts` when you expected `auth.ts`, and the tests pass with no unrelated changes, that counts as a pass. Grading the exact steps penalizes valid alternatives and makes your metric noisier. The same post warns that " [ unnecessary shared state between runs (leftover files, cached data, resource exhaustion) can cause correlated failures ]( https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents ) ".

That's why a single trial needs a clean environment. Steps for one evaluation trial: 1. **Reset the environment.** Make a fresh checkout or worktree of a pinned commit. Don't reuse the last trial's folder.

2. **Run the agent headless.** Use `claude -p "<task prompt>" --output-format json` so the result and metadata come back as JSON ( [ headless docs ]( https://code.claude.com/docs/en/headless ) ). 3. **Grade the outcome.** Run tests, check files, and run `git diff --stat` to spot changes outside the expected paths. 4. **Record one row.** Save the task id, trial number, pass/fail, score if any, cost and duration.

5. **Read some transcripts.** Anthropic notes that 0% pass@100 usually points to a broken task spec, not a hopeless agent ( [ Anthropic ]( https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents ) ). You only find that out by reading what happened. If you already run Claude Code from cron, this is the same command your job uses, pointed at a frozen copy of the repo.

See [ Claude Code Headless Mode: Flags, Auth and Cost ]( https://en.aobaworks.com/claude-code-headless-mode-flags-auth-and-cost-2026/ ) for the flags and auth. ## What Are the Core AI Agent Performance Metrics? The core metrics are pass@k, pass^k, success rate, cost, latency and turns per task. Compute all of them on a fixed task set.

Here are the definitions and the numbers to compute: | Metric | What it answers | How to compute from n trials with c passes | Where the data comes from | |---|---|---|---| | pass@1 (success rate) | How often does one run succeed? | c / n | Your grader | | pass@k | Can it solve this at all within k tries? | 1 − C(n−c, k) / C(n, k) | Your grader | | pass^k | Will k runs in a row all succeed? | C(c, k) / C(n, k) | Your grader | | Cost per task | What does one trial cost? | Mean of `total_cost_usd` | [ `--output-format json` ]( https://code.claude.com/docs/en/headless ) | | Latency per task | How long does a trial take? | Mean or p90 of `duration_ms` | [ SDK result message ]( https://code.claude.com/docs/en/agent-sdk/typescript ) | | Turns per task | How much back-and-forth did it need? | Mean of `num_turns` | [ SDK result message ]( https://code.claude.com/docs/en/agent-sdk/typescript ) | | Cost per successful task | What do you pay per good result? | Total cost / c | Both | The pass@k formula is the unbiased estimator from the Codex paper: pass@k:= E [ 1 − C(n−c, k)/C(n, k)]. The paper calls the shortcut 1 − (1 − p̂)^k "[biased ]( https://arxiv.org/html/2107.03374 ) ", and its authors drew n=200 samples per problem for k ≤ 100 ( [ Chen et al., section 2.1 ]( https://arxiv.org/html/2107.03374 ) ). pass^k comes from τ-bench, which introduced it as " [ a new metric to evaluate the reliability of agent behavior over multiple trials ]( https://arxiv.org/abs/2406.12045 ) ". The C(c,k)/C(n,k) form above uses the same combinatorial idea: the share of k-sized subsets of your trials in which every trial passed.

The two metrics move in opposite directions as k grows. Anthropic's example uses a 75% per-trial success rate, where the chance that all 3 of 3 trials pass is about 42% (0.75³) ( [ Anthropic ]( https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents ) ). As k rises, pass@k heads toward 100% and pass^k falls. τ-bench found leading function-calling agents were " [ quite inconsistent (pass^8 <25% in retail) ]( https://arxiv.org/abs/2406.12045 ) ".

To see how this works out, here's an example calculation (illustrative numbers, not a measurement). You run a task 10 times and 7 pass. - pass@1 = 0.70 - pass@3 = 1 − C(3,3)/C(10,3) = 1 − 1/120 ≈ 0.99 - pass^3 = C(7,3)/C(10,3) = 35/120 ≈ 0.29 "It basically always works" (pass@3) and "three nights in a row, it fails at least once about 70% of the time" (pass^3) describe the same 10 runs. For unattended work, believe pass^k. **Which k should you use?** Match k to how you actually run the job.

If a human picks the best of three drafts, pass@3 is fine. If cron runs it daily and nobody checks, use pass^k with k equal to how many runs you'd let happen before you look. For example, k=7 if you review weekly. ## How Do AI Agent Performance Metrics Work in Amazon Connect?

Amazon Connect's agent metrics measure contact-center agents and AI assistants, not code correctness. Only the idea of "a rate over many interactions" carries over to Claude Code. AWS defines metrics like these in the [ Amazon Connect metric definitions ]( https://docs.aws.amazon.com/connect/latest/adminguide/metrics-definitions.html ): | Amazon Connect metric (API id) | What AWS says it measures | Nearest Claude Code equivalent | Does it transfer? | |---|---|---|---| | Average handle time ( `AVG_HANDLE_TIME` ) | Agent interaction, hold and after-contact work time | Mean `duration_ms` per trial | Partly: latency only | | AI handoff rate ( `AI_HANDOFF_RATE` ) | Share of AI-handled contacts escalated to humans or extra support | Share of runs that need your manual fix | Yes, as an idea | | AI agent invocation success rate ( `AI_AGENT_INVOCATION_SUCCESS_RATE` ) | Share of invocations that " [ executed successfully without technical failures ]( https://docs.aws.amazon.com/connect/latest/adminguide/metrics-definitions.html ) " | Share of runs with exit code 0 | Only as a health check | | Agent answer rate ( `AGENT_ANSWER_RATE` ) | (Contacts accepted / total routing attempts) × 100 | None | No | If you came here from a search about Connect, the useful lesson is the gap between the last two rows.

"Invocation success" in Connect, and exit code 0 in Claude Code, both tell you that nothing crashed. Neither tells you the answer was right. **Pitfall:** Reporting invocation success as your quality metric. A coding agent can have 100% clean exits and a 60% pass rate.

Track both. The pass rate is the one that matters, and clean exits are only a health signal. **Fix:** Pair any health metric with an outcome grader. In Connect terms, that's like pairing invocation success with handoff rate.

In Claude Code, pair exit status with your test-based grader. ## What Are the Common AI Agent Evaluation Metrics for Coding Agents? Three metrics matter most: pass rate across N reruns, run-to-run variance, and graded-score drift. This holds especially for solo builders who run Claude Code repeatedly.

Most top-ranking pages on this query list categories: task completion, tool-call accuracy, latency, business KPIs. They rarely show how to separate a flaky task from a reliable one. That's the part that decides whether a cron job is safe. ### Pass rate across N reruns Run each task n times (10 is a sensible minimum, 20 if the task is cheap).

Record c passes per task and compute pass@1, pass@k and pass^k as shown above. Anthropic suggests starting with " [ 20-50 simple tasks ]( https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents ) " taken from real failures, so you don't need a big benchmark. ### Variance across reruns (flakiness) Flakiness is simple to detect. A task is flaky when 0 < c < n: it passed sometimes and failed sometimes on the same input.

The harder question is how sure you can be about its pass rate. A pass rate p from n trials has a standard error of about √(p(1−p)/n). Here's an example: at p=0.7 and n=10, that's √(0.021) ≈ 0.145, so roughly ±14 percentage points.

That's arithmetic, not a measurement. So with 10 reruns, you can't reliably tell a 70% task from an 85% task. Two rules follow: 1. **Count flaky tasks, not just the average.** "18 of 20 tasks are 10/10, two are 4/10" is far more useful than "88% overall".

The two flaky tasks are where your overnight failures will come from. 2. **Rerun before you conclude.** If a prompt change moves a task from 7/10 to 9/10, run another 10 before you believe it. Promptfoo's coding-agent guide puts the next step bluntly: " [ If a prompt fails 50% of the time, the prompt is ambiguous. ]( https://www.promptfoo.dev/docs/guides/evaluate-coding-agents/ ) " Its advice is to fix the instructions rather than add retries.

Retries raise pass@k but leave pass^k where it was. ### Graded-score drift Drift is a slow fall in a 1–5 or 0–100 quality score on input that hasn't changed. It matters for output that tests can't fully check, like PR review comments, changelogs or docs. You track it by keeping a baseline and comparing each new batch against it.

Steps: 1. Freeze 10–20 inputs, and have an LLM judge score each output with a fixed rubric (for example, 1–5 for "covers every changed function"). 2. **Measure judge noise first.** Grade the *same* saved transcripts twice.

Anthropic notes model-based graders are " [ non-deterministic ]( https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents ) ", so part of any movement is the judge, not the agent. 3. Set the baseline as the mean score of your first full batch.

4. On each later batch, flag drift when the mean drops by more than about twice the standard error of the batch mean, or by an absolute threshold you choose (for example, 0.3 on a 5-point scale). These numbers are illustrative.

5. When drift fires, diff the transcripts from a high-scoring baseline run and a low-scoring new run. Look at what changed, such as a new model, a CLAUDE.md edit or an MCP server. **Pitfall:** Changing the judge prompt or judge model mid-series.

Every judge change resets the baseline. Keep a `judge_version` column and never compare scores across versions. ## How Do You Set Up Metrics Visualization for an AI Agent Evaluation Framework? A minimal Claude Code framework appends one JSON line per trial to a file and then plots pass rate and score per task over time.

Here's the logging loop. It's an example script, so adapt the paths and the grader to your project. It uses only flags from the [ headless docs ]( https://code.claude.com/docs/en/headless ): `-p`, `--output-format json`, `--allowedTools`, and `--bare`.

Bare mode skips local hooks, plugins, MCP servers and CLAUDE.md, so every trial starts the same way. It needs `ANTHROPIC_API_KEY`, because bare mode "doesn't use your subscription login" ( [ headless docs ]( https://code.claude.com/docs/en/headless ) ). Enter fullscreen mode Exit fullscreen mode bash

set -u TASK="$1"; PROMPT=$(cat "$2"); RUNS="${3:-10}" for i in $(seq 1 "$RUNS"); do DIR=$(mktemp -d) git worktree add --detach "$DIR" eval-base >/dev/null 2>&1 OUT=$(cd "$DIR" && claude --bare -p "$PROMPT" \ --allowedTools "Read,Edit,Bash" --output-format json) EXIT=$? (cd "$DIR" && npm test >/dev/null 2>&1) && PASS=1 || PASS=0 echo "$OUT" | jq -c --arg t "$TASK" --argjson i "$i" \ --argjson p "$PASS" --argjson e "$EXIT" \ '{task:$t, trial:$i, pass:$p, exit:$e, cost:.total_cost_usd, ts:(now|todate)}' >> results.jsonl git worktree remove --force "$DIR" done

Then summarize and plot the results (also an example): Enter fullscreen mode Exit fullscreen mode python import json, math, collections import matplotlib.pyplot as plt rows = [json.loads(l) for l in open("results.jsonl")] by = collections.defaultdict(list) for r in rows: by[r["task"]].append(r["pass"]) def pass_at(n,c,k): return 1 - math.comb(n-c,k)/math.comb(n,k) def pass_hat(n,c,k): return math.comb(c,k)/math.comb(n,k) for t, v in by.items(): n, c = len(v), sum(v) print(t, f"p1={c/n:.2f}", f"pass@3={pass_at(n,c,3):.2f}", f"pass^3={pass_hat(n,c,3):.2f}", "FLAKY" if 0<c<n else "") tasks = list(by) plt.bar(tasks, [sum(by[t])/len(by[t]) for t in tasks]) plt.ylabel("pass rate"); plt.ylim(0,1); plt.savefig("pass_rate.png")