Monitoring AI Quality: Evaluation Pipelines for Non-Deterministic Products

Shipping an AI feature is the easy part. Knowing that it still works next month is the hard one.

In a traditional application, the same input always gives you the same output. That's why tests work so well: you write expect(sum(2, 2)).toBe(4) once, and it protects you forever. AI features break that contract. Send the same input to an LLM twice and you get two different answers. Both may be good. One may be a bit worse. And one day, after a prompt tweak or a silent model update, all of them may be wrong, with no error in the logs.

In this article we'll walk through how to monitor the quality of an AI feature: a pipeline that generates real outputs, asks a second model to grade them, and tells the team in Slack when quality drops. Then we'll go one step further and run it on a schedule with cron jobs, so you also catch the problems that don't come from your own deploys.

How to read this article: if you're on the product or leadership side, the first sections explain the problem and what to measure, with no code required. If you're a developer, the implementation and the cron examples are in "Building the Pipeline" and "Beyond CI".

To keep things concrete, we'll use one example throughout the article: a meeting assistant. It reads the transcript of a recorded call and writes a structured summary: key decisions, action items with owners, and highlights tied to moments in the recording. Each team picks a summary template (sales call, standup, job interview, customer support and so on), so the same call should produce a different summary depending on that choice. That's a lot of ways to be almost right.

With AI, quality is not a value, it's a distribution. Your feature doesn't produce "the right answer"; it produces a range of answers, and most of them are good. The job is to keep that range where you want it.

Think of it like a restaurant chain. The same dish is cooked by different cooks, in different kitchens, with ingredients from different suppliers. Nobody tastes every plate, but someone tastes a sample against the recipe on a regular schedule. That's what an evaluation pipeline does for an AI feature.

The tricky part is that quality can change even when nobody touches the code. There are three common sources of change:

None of these throws an error. The output just gets a little worse, and your users notice before you do. For an AI feature, that's the real cost: not a crash, but trust. A feature that is great 90% of the time and strange the other 10% teaches users to double-check everything, and then it stops saving them time.

An evaluation pipeline gives you three things:

You stop checking for the answer and start checking for properties of a good answer. Instead of expect(output).toBe, you ask: is it well-formed? Does it cover the whole call? Is it faithful to the transcript? Each question needs a different kind of check, and they get more expensive as they get smarter.

All of them run on top of the same foundation:

A golden dataset is a fixed set of realistic inputs that represent how your customers use the feature. For our meeting assistant, that could be 20 fixtures: 15 call transcripts and 5 written meeting notes, covering every summary template. A second set of 5 timestamped recordings is used for the coverage check. Use synthetic or anonymized data, never raw customer content.

Because the inputs never change, the scores are comparable over time. If last week's average was higher than today's, something on our side or the provider's side changed.

Some qualities can be measured with plain code, no AI needed. A good one for our example is coverage. If the summary only cites moments from the first ten minutes of a one-hour call, the reader gets half the picture. So we split the recording into bins of about four minutes and measure how evenly the summary's timestamps spread across them. A score of 0 means everything is bunched in one place; 1 means perfectly even.

For the things code can't measure, like "was this decision really made in the call?", you use a second model as a judge. It receives the source, the generated output and a rubric, and grades the output.

The most important part is the rubric, and it should be yours. Start by writing down the metrics that define a good output for your product. For our meeting assistant, they could be:

A support bot would have different ones, like tone, policy compliance or whether the issue was solved. Then ask the judge to return a score from 1 to 100 for each metric, with a one-sentence reason for each score. Run it with temperature 0 to keep it as stable as possible.

Why a separate score per metric instead of one grade? Because it lets you decide what "good enough" means for each metric on its own, as we'll see in the thresholds.

The judge is also an LLM, so treat its scores as measurements with some noise, like a thermometer that can be off by a degree. Before trusting the numbers, compare a few judge scores with a human expert's opinion, and adjust the rubric until they mostly agree.

A score is only useful when it's compared against something. With a 1 to 100 score per metric, you set a threshold for each metric, based on how much that metric matters to your product:

These numbers are only an example. Choosing them is a product decision, not an engineering one: a healthcare product may want accuracy at 98, while a brainstorming tool may be fine with 70. Together, the thresholds are the quality level you expect from the feature, written down.

There are two ways to apply them, and you'll probably want both:

Code metrics, like coverage, work the same way: decide how even the spread must be for your product, and fail the run below that.

A natural next step for consistency is to run the same input several times and measure the spread of the scores. A feature with an average of 80 and a small spread is a better product than one with the same average that swings between 60 and 95.

The pipeline itself doesn't need to be complex. It's a script that runs the golden set against a real environment, calling the same endpoints the product uses, so it tests the whole stack at once: prompts, model, parsing and response format. Run it as a CI job, like a GitHub Action, every time something that can change quality changes: a prompt, a model version or the code around them.

![Evaluation pipeline: a prompt or model change, or a scheduled run, runs the golden set; a rubric check and a coverage check run in parallel; the results go to the job summary and to a Slack message that is green only when both checks pass]

The two checks run side by side, and a failure in either one turns the Slack message red.

The core of the pipeline is a Python script, run_evals.py. For each fixture it makes two calls to OpenAI: one to generate the summary with the same prompt and model the product uses, and one to ask the judge to grade it. Here is a simplified version:

import asyncio import sys from openai import AsyncOpenAI client = AsyncOpenAI () # reads OPENAI_API_KEY from the environment MODEL = " gpt-6-luna " # the same model the product uses THRESHOLDS = { " accuracy ": 90, " owners ": 85, " completeness ": 80, " clarity ": 70 } async def generate_summary ( source ): # Step 1: generate the summary with the product's own prompt response = await client. chat. completions. create ( model = MODEL, messages = [ { " role ": " system ", " content ": SUMMARY_PROMPT }, { " role ": " user ", " content ": source }, ], ) return response. choices [ 0 ]. message. content async def judge_summary ( source, summary ): # Step 2: ask the judge for a 1-100 score per metric response = await client. chat. completions. create ( model = MODEL, reasoning_effort = " none ", # temperature only works with reasoning off temperature = 0, response_format = { " type ": " json_object " }, messages = [ { " role ": " system ", " content ": JUDGE_PROMPT }, { " role ": " user ", " content ": f " Source: \n { source } \n\n Summary: \n { summary } " }, ], ) # JudgeResult validates the answer, we'll see it in the next section return JudgeResult. model_validate_json ( response. choices [ 0 ]. message. content ) async def process_fixture ( fixture ): source = fixture [ " source " ] # a transcript or written notes summary = await generate_summary ( source ) return await judge_summary ( source, summary ) async def run_all (): fixtures = load_fixtures () results = await asyncio. gather ( * ( process_fixture ( f ) for f in fixtures )) failed = [] for metric, threshold in THRESHOLDS. items (): average = sum ( result. scores [ metric ] for result in results ) / len ( results ) status = " PASS " if average >= threshold else " FAIL " print ( f " { metric }: { average:. 1 f } (threshold { threshold } ) { status } " ) if average < threshold: failed. append ( metric ) sys. exit ( 1 if failed else 0 ) asyncio. run ( run_all ()) Enter fullscreen mode Exit fullscreen mode If you read our article on async tasks in arrays, asyncio.gather will look familiar: it's Python's Promise.all. With the SDK's async client, all fixtures run in parallel.

LLM calls are slow, so running them one by one would make the pipeline far slower.

SUMMARY_PROMPT and MODEL should be exactly what the product uses, and JUDGE_PROMPT is the rubric from the next section. To keep the example short, step 1 calls OpenAI directly. If your feature runs behind your own API, call that API instead, so each run also tests what's around the model, like how the prompt is built and how the answer is parsed.

A complete version should also keep failed requests apart from low scores. A request that errors is reported by name, and any failure fails the run, because "the AI service returned a 500" and "the AI got worse" are different problems.

The judge's prompt lists your metrics and asks for a strict JSON answer. A trimmed version for our example:

You are evaluating a meeting summary against the call transcript. Score each metric from 1 to 100 and explain each score in one sentence. - accuracy: every decision and action item really happened in the call - completeness: no decision or action item was left out - owners: every action item has a person responsible for it - clarity: someone who missed the meeting understands the summary Answer only with JSON: {"scores": {"accuracy": <1-100>,...}, "reasons": {"accuracy": "<one sentence>",...}} Enter fullscreen mode Exit fullscreen mode Validate the answer before using it. The judge is an LLM too, so sometimes it returns a score out of range, or something that isn't JSON at all. With Pydantic, that becomes an error instead of a made-up number:

from typing import Annotated from pydantic import BaseModel, Field Score = Annotated [ int, Field ( ge = 1, le = 100 )] class JudgeResult ( BaseModel ): scores: dict [ str, Score ] # one 1-100 score per metric reasons: dict [ str, str ] # one sentence per metric Enter fullscreen mode Exit fullscreen mode A score per metric tells you what kind of problem you have. Low accuracy usually means the AI is inventing things that didn't happen in the call. Low completeness means items were skipped. Low clarity means the summary is vague. Each one points to a different fix, and the reasons tell you where to look.

Version the judge's prompt too, just like the prompts it evaluates; a prompt management tool like Langfuse makes that easy. And if you want one headline number for a dashboard, the average of your metrics works well.

A check that nobody sees is a check that doesn't exist. Every run publishes its results in three places:

Evaluation pipelines are not immune to the problem they solve. They call slow, non-deterministic services, so they time out, the judge occasionally returns something unparseable, and a deploy may not be live yet when the run starts. When that happens often enough, teams tend to switch the automatic run to manual, and a manual check is one that people forget to run.

The lesson: separate "the pipeline broke" from "quality dropped". They need different alerts and often different owners. If every red message might be noise, people stop reading them, and then the pipeline protects nobody.

A pipeline that runs after a deploy answers one question: did we break something? But remember the three sources of change. Two of them, the model provider and your customers' inputs, never come with a deploy. A model can change on a Tuesday afternoon while your team is in a meeting. To catch that, the evaluation has to run on a clock, not on a commit.

There are two kinds of scheduled runs, and they answer different questions:

The golden set is your control group: the inputs never change, so any change in the score comes from the system. Production sampling is your reality check: the inputs change every day, and that's the point.

The easiest step is to reuse what you already have. Take the same validation script and add a schedule trigger:

on: schedule: - cron: " 0 11 * * 1-5" # weekdays at 11:00 UTC workflow_dispatch: Enter fullscreen mode Exit fullscreen mode A few things to keep in mind with scheduled GitHub workflows:

The golden set tells you if the system changed. It doesn't tell you how the AI is doing with the inputs customers send today. For that, you sample real outputs and grade them with the same judge.

Here's what that job can look like in Node, using node-cron for the schedule and p-map to limit how many judge calls run at once:

import cron from ' node-cron '; import pMap from ' p-map '; const SAMPLE_SIZE = 20; const MAX_DROP = 5; // points below the 7-day average const SIX_HOURS = 6 * 60 * 60 * 1000; const average = ( values ) => values. reduce (( sum, value ) => sum + value, 0 ) / values. length; cron. schedule ( ' 0 */6 * * * ', async () => { // 1. Sample real outputs generated since the last run const samples = await AiOutputs. aggregate ([ { $match: { createdAt: { $gte: new Date ( Date. now () - SIX_HOURS ) } } }, { $sample: { size: SAMPLE_SIZE } }, ]). toArray (); if (! samples. length ) return; // 2. Grade them with the same judge and metrics used in CI, 4 at a time const results = await pMap ( samples, ( sample ) => judge ( sample ), { concurrency: 4, }); // 3.

Store the run, so we can see trends and not only pass/fail const metrics = Object. keys ( results [ 0 ]. scores ); const averages = Object. fromEntries ( metrics. map (( metric ) => [ metric, average ( results. map (( result ) => result. scores [ metric ])), ]) ); await EvalRuns. insertOne ({ createdAt: new Date (), averages, promptVersion: samples [ 0 ]. promptVersion, model: samples [ 0 ]. model, scores: results. map (({ outputId, scores }) => ({ outputId, scores })), }); // 4. Alert on any metric that dropped below its recent baseline const baseline = await getRollingAverages ({ days: 7 }); const dropped = metrics. filter ( ( metric ) => averages [ metric ] < baseline [ metric ] - MAX_DROP ); if ( dropped. length ) { await notifySlack ( dropped. map ( ( metric ) => ` ${ metric }: ${ averages [ metric ]. toFixed ( 1 )} vs 7-day average ${ baseline [ metric ]. toFixed ( 1 )} ` ). join ( ' \n ' ) ); } }); Enter fullscreen mode Exit fullscreen mode AiOutputs, EvalRuns, judge, getRollingAverages and notifySlack stand for your own collections and helpers. The shape is what matters: sample, grade, store, compare.

A fixed threshold catches big drops. But if accuracy goes from 97 to 91, it still passes a threshold of 90, and something clearly got worse. Comparing each run against a rolling baseline, like the 7-day average in the example above, catches the slow slides that a fixed number misses.

And remember the lesson from the previous section: give "the job failed to run" its own alert. A monitor that fails silently looks exactly like a monitor that says everything is fine.

One number is enough to start, but a few more turn a red light into a diagnosis. These are the metrics we find useful, and who usually reads each one:

Put the numbers where people already look. For leadership, that can be a single daily line in Slack: the average, the trend against the baseline, passed or failed. For engineers, it's the job summary with each metric's score and a link to the generated outputs. The goal is that nobody has to remember to go check a dashboard.

AI features don't fail like regular code. They don't crash; they drift. A prompt change, a model update or a new kind of input can make the output a little worse, and no log will tell you. The only way to know is to keep measuring.

The good news is that the tools for this are the ones you already use: a test script, a CI workflow, a cron job and a Slack channel. What changes is what you assert. Not the exact answer, but the properties of a good one: well-formed, complete, faithful to the source, and good enough by a rubric that your team agreed on.

If you want to start this week, here's the path we'd follow again:

Your AI feature will never give the same answer twice. With an evaluation pipeline watching it, it can still give the same quality.