What exactly are AI evals?

Posted on Nov 1 • Edited on Nov 1 • AI-assisted

You ask an AI to write a configuration. It returns something that looks reasonable, explains what it did, and says the result is ready.

Reading the answer might catch an obvious mistake. But if you change the prompt tomorrow, how will you know whether the new version is better? What if it fixes one example and breaks another?

That is where evaluations, usually shortened to evals, become useful.

I want to start this series with the basics. The case studies that follow will make more sense if we agree on what we are measuring, how we judge it, and what a score can leave out.

An eval tests an AI system against a defined expectation and records what happened.

That expectation needs to be more useful than “give me a good answer.” For example:

The eval gives the system a task, collects its answer or actions, and checks them against the success criteria. A score is a result of that process, not the whole process.

For an agent that can take actions, its final message is only part of the evidence. Saying “I moved the meeting” does not establish that the calendar changed. We need to inspect the resulting state too. Anthropic makes this distinction between a recorded interaction and its actual outcome in its agent evaluation guide.

This also explains why an eval can contain ordinary software tests. A unit test may be one of its checks. The larger evaluation adds the chosen tasks, run conditions, repeated attempts where appropriate, and the records we need to interpret the results.

Running an eval does not, by itself, train or improve the model. We use its findings to decide what to change, then test again. The change might be in a prompt, a retrieved document, a tool, application code, or the model itself.

Here is the running example for this article:

Create a Kubernetes Deployment with three replicas.

Kubernetes runs applications, and a replica is a copy of a workload. You do not need to know Kubernetes to follow the example: we asked for three copies.

Suppose the answer contains this excerpt:

{ "spec": { "replicas": 2 }} Enter fullscreen mode Exit fullscreen mode It is readable JSON. But it says two, not three.

The excerpt answers the first two questions. It cannot establish the third. It is also not a complete Kubernetes manifest, so accepting this JSON does not establish full schema validity.

Figure 1. An illustrative example. Readable output can still miss a requirement, and neither observation proves runtime behavior.

An eval should say which of these questions it checks. Otherwise, “passed” is easy to overinterpret.

The terminology becomes easier when we attach each word to the same example.

Frameworks use these words somewhat differently. In Inspect AI, a task brings together a dataset, a solver that produces answers, and a scorer that grades them. Its task documentation shows how those pieces fit together.

A benchmark is usually a defined collection of tasks and a way to report performance. A public benchmark can be useful for comparisons. A project-specific eval asks whether a system meets the requirements of that project. Doing well on one does not automatically answer the other question.

There is no single, fixed number that covers all the ways people use the word. Different labels answer different questions.

I find it easier to sort them along three separate axes.

A failure in an application does not immediately tell us which component caused it. The retrieved document might be wrong even when the model follows it correctly.

Code-based grading uses explicit checks: parse the output, compare a value, run tests, or inspect an environment. It works well when success can be stated precisely. It can also be wrong if the checks are incomplete or unnecessarily strict. Inspect's custom scorer examples show this kind of scoring logic.

Model-based grading asks another model to assess the response against instructions or a rubric. A rubric is a written set of judging criteria. This can help with open-ended answers, but the grader model can make mistakes too. Its judgments need comparison with trusted examples and human review. See Inspect's model-grading guide.

Human grading uses people with the relevant understanding to review the work. It is valuable for ambiguous cases and for checking automated graders. People also need clear criteria and a way to resolve disagreements.

These methods can be combined. We can use code to check structure, a model to flag a possible factual error, and a person to review disputed cases.

We do not always need one reference answer that every response must resemble. A code change can be judged by tests, and an explanation can be judged against a rubric. The expectation still needs to be defined even when several different answers could satisfy it.

Offline does not mean “no network.” An offline eval may call a hosted model API. It describes how the experiment relates to users and deployment.

We can also name evals by their purpose: checking a new capability, catching regressions, probing safety requirements, or measuring reliability.

So our future case study can be an offline, code-graded evaluation of generated configurations, used to investigate scoring reliability. Those labels fit together. They are not competing categories.

Start with a decision you need to make. “Is this model good?” is too broad. “Did this prompt change improve the required configuration settings without increasing failures?” is something we can investigate.

Repeat trials when the behavior varies. Keep each attempt rather than quietly replacing failures with successful retries. Repeating the same cases helps measure variation on those cases; it does not make the case collection more representative.

Correctness is not the only useful measurement. Record latency, token usage, cost, and resource use when they matter to the decision. Report these alongside the task results so readers can see what a successful attempt required.

An eval also needs records: the input, response, relevant actions, configuration, check results, and errors. Otherwise, the headline number may survive while the explanation disappears.

Hamel Husain's practical evaluation workflow connects this work to a useful development cycle: evaluate, inspect the problems, change the system, and evaluate again.

This is the question that connects the introduction to our case study.

Suppose our grader only checks that replicas is present. The answer containing two replicas could pass even though the task asked for three.

We could get a clean report from a broken scoring rule.

Before using that grader to compare models, give it examples whose expected outcomes we understand:

Figure 2. Test the measuring instrument too. These constructed examples check the grader, not a model's ability.

The third case is easy to overlook. An extra label that does not break a requested selector should not automatically make a configuration wrong. A grader needs to catch mistakes while respecting the task's allowed variations.

These controls still cover only the situations we chose. A passing set of controls is useful evidence, not a guarantee that the grader is complete.

Here is a made-up run, not a result from our benchmark:

If we report passing answers among completed responses, the rate is 6/8 = 75%.

If we report successful attempts among everything started, it is 6/10 = 60%.

Figure 3. Same run, different denominators. Both calculations need their definitions beside them.

Neither rate should be presented without explaining what counts. There is also a difference between a model producing an invalid answer and a grader crashing while trying to inspect it. Inspect's scoring policy explains why those outcomes need separate treatment.

A partial score is another choice. An answer may satisfy several requirements and still fail the full task. “Most checks passed” and “the task passed” are different statements.

For comparisons, look beyond one percentage: case coverage, variation across runs, failure categories, and operational measurements can all matter. More trials and uncertainty estimates help describe the evidence we have; they do not repair a badly chosen dataset. Inspect documents several scoring metrics and uncertainty measures.

In IaC Eval Bench, we are studying model-generated Kubernetes configuration through offline structure and task checks. We retain the responses and individual scoring outcomes.

The upcoming case study examines the evaluator itself. A review found cases where it accepted a contradictory selector and rejected an acceptable extra label. It shows why the original positive and negative examples could produce expected totals while other scoring errors remained.

That work does not demonstrate runtime correctness. It does not establish a ranking of models. It gives us specific examples of how an evaluation can mislead us, and how to test the checks more carefully.

The three questions I want readers to carry into it are:

I plan to follow this introduction with practical case studies: a reproducible example, the failure, the investigation, and the limits of the fix. Each should give readers something they can inspect or try themselves.

I will add links here as the case studies are published.

For the framework terminology and design choices behind this introduction, the linked Inspect, Anthropic, and Hamel Husain articles are useful places to continue reading.