AI Agent Regression Testing: Production Traces to CI

Posted on Oct 28 • Originally published at fp8.co

TL;DR: Trace-based regression testing turns a failing production agent trace into a CI test case. The trace's recorded tool and model outputs become fixtures, so the test replays offline and deterministically on every pull request. Tracely is an MIT-licensed open-source project that packages this loop: OTLP ingest, graders, failure clustering, promotion of a trace to a case, and a GitHub Action gate. Langfuse, LangSmith and Promptfoo cover parts of the same loop. None of them removes the main engineering cost, which is routing every tool and model call through a seam you can record and replay.

Agent failures rarely show up in the test inputs a team writes on day one. They come from real conversations: a user phrases a refund request the planner did not expect, a tool returns an empty list instead of an error, or a model upgrade changes how often the agent calls a search tool. By the time someone sees the failure, it lives in a trace (a nested record of every prompt, model response, tool call and result in one run).

Teams usually handle this in one of two ways, and both leak:

Trace-based regression testing closes that gap. The trace is the test: its exact inputs, the exact tool outputs the agent saw, and the model responses in recorded order. A grader decides pass or fail. The suite runs on every pull request and blocks merges that bring a known failure back.

This is the same discipline as "write a failing test before you fix the bug" in ordinary software. The difference is that agents are non-deterministic and depend heavily on external services, so the test needs recorded fixtures before it can be repeated.

The loop has five stages. Every implementation, whether bought or built, needs all five.

The Promote stage is where most homegrown setups stop. Copying a trace's input into a dataset is easy. Copying the world the agent saw (tool results, retrieval hits, intermediate model outputs) and replaying it in order is the actual work.

Tracely is an open-source project by Julien Wuthrich that describes itself as "trace-native CI/CD for AI agents." It is MIT-licensed, has about 1.2k stars at the time of writing, and is still early (a few dozen commits). Its stack is a Python backend (FastAPI, Celery, uv) and a Next.js 15 UI. It stores traces in ClickHouse, the case registry in Postgres with pgvector, the queue in Redis and blobs in MinIO/S3. A local demo starts with docker compose --profile demo up -d --build --wait.

Any OpenTelemetry exporter can post to POST {endpoint}/v1/traces with a Bearer ingest key. Tracely reads both gen_ai.* semantic-convention attributes and OpenInference attributes, so existing instrumentation keeps working. The Python SDK ( tracelyai-sdk ) adds auto-instrumentation for OpenAI, Anthropic, Google, Mistral and LangChain:

import tracely_sdk as tracely tracely. init ( endpoint = " http://localhost:8000 ", api_key = " tracely_dev_key ", service_name = " support-agent ", env = " prod ", # prod | staging | ci | dev — the gating axis instrument = " auto ", ) with tracely. trace ( agent = " support-agent ", conversation = " conv-1 ", user = " u_42 " ): client. chat. completions. create ( model = " gpt-4o ", messages = [{ " role ": " user ", " content ": " Where is order ORD-4471? " }], ) Enter fullscreen mode Exit fullscreen mode The env field matters more than it looks. It separates production traces (the source of failures) from CI traces (the ones the gate grades), so the same pipeline can serve both.

Evaluators come in two kinds. Structural checks need no model. LLM judges run at conversation, run or span level and can output a score, boolean, number, text or JSON-schema result. Failing runs are then clustered "structurally and semantically" with a structural signature plus embedding similarity. In the project's words, "31 broken runs become one issue with a count."

This is the most important idea in the project. Promoting a failing trace bundles "recorded input, tool and LLM outputs" as fixtures. The case is trusted only if it "must fail on the old code and pass on the fix." A regression test that passes on the buggy code is a test that cannot fail, which is the most common way evaluation suites lie to their owners. The contract makes that check part of the workflow instead of leaving it to discipline.

Multi-turn behaviour uses scenarios instead of single cases: either a scripted conversation, or an adversarial goal that a red-team model improvises against.

Replay runs offline against the recorded fixtures, with no API keys and no model spend. tracely gate exits non-zero, posts a commit status and upserts a PR comment. The bundled action has two modes. Tracely can call your agent through an entrypoint ( tracely replay planner --entrypoint my_pkg.agent:run, or --cmd "node run.js" for non-Python agents), or it can grade the CI traces your own test job emitted:

- uses: jwuthri-cloud/Tracely-ai/.github/actions/tracely-gate@master with: mode: gate agent: planner api: https://tracely.your-co.dev key: ${{ secrets.TRACELY_KEY }} Enter fullscreen mode Exit fullscreen mode Pin that @master reference to a commit SHA before you rely on it. A gate that can change underneath you is not a gate.

The README states it plainly: hermetic replay "requires your agent to route tool/model calls through the SDK's call_tool / call_llm seam." Auto-instrumentation is enough to record a trace. It is not enough to replay one, because replay means intercepting the call and returning the recorded value instead of making it. If your agent calls requests.get from inside a tool, or a framework makes model calls you cannot reach, those calls go live during replay.

Other caveats from the project itself: judges and failure analysis degrade without an LLM key, the default AUTH_MODE=dev with tracely_dev_key is open access and only suitable for local use, and the Celery worker does not hot-reload.

None of the established tools is called "trace-native CI," but each covers some of the stages. The question is which stages you get for free and which you build.

Langfuse and LangSmith datasets store inputs and expected outputs, not the world. Adding a trace to a Langfuse dataset or a LangSmith dataset captures what went in and, optionally, what should come out. When you run an experiment, your application runs again, live. That is the right design for comparing a new prompt or model against a fixed set of inputs: you want the new model's real behaviour. It is the wrong design for a regression test of agent logic, because the tools return today's data instead of the data that triggered the bug. For the trade-offs between these two platforms, see our LangSmith vs Langfuse vs Phoenix comparison.

Promptfoo is the strongest CI gate and the weakest trace source. Its GitHub Action and assertion language are mature, and it runs red-team suites no trace tool matches. But it never sees production. Every case is something a human typed, so the "trace to test" step is copy-paste. Our Langfuse vs Promptfoo comparison covers why many teams pair the two.

Tracely's real differentiator is the fixture bundle plus the fail-to-pass contract. The tracing and judging overlap heavily with Langfuse. Indeed, the project describes parts of its design as mirroring Langfuse's. What is new is treating promotion as a test-engineering operation with a validity check, not as a "save to dataset" button.

If you already run Langfuse or LangSmith and do not want another stack, you can build the missing stages yourself. The core is a single seam that every tool and model call goes through. This is a generic sketch of the pattern, not any vendor's API:

import json, hashlib from contextvars import ContextVar _mode = ContextVar ( " mode ", default = " live " ) # live | record | replay _tape = ContextVar ( " tape ", default = None ) # list of recorded calls def _key ( kind, name, args ): blob = json. dumps ([ kind, name, args ], sort_keys = True, default = str ) return hashlib. sha256 ( blob. encode ()). hexdigest ()[: 16 ] def seam ( kind, name, fn, ** args ): """ Every tool call and LLM call goes through here. """ mode, tape = _mode. get (), _tape. get () if mode == " replay ": rec = tape. pop ( 0 ) # strict order, not lookup if rec [ " key " ]!= _key ( kind, name, args ): raise AssertionError ( f " divergence at { kind }: { name } " ) return rec [ " output " ] out = fn ( ** args ) if mode == " record ": tape. append ({ " key ": _key ( kind, name, args ), " kind ": kind, " name ": name, " output ": out }) return out Enter fullscreen mode Exit fullscreen mode Three design choices in that sketch decide whether the tests are useful.

Then write the fail-to-pass check as an actual test step: run the case against the parent commit and require a failure, run it against the fix and require a pass. Without that step you will end up with cases that were promoted from a trace but never reproduced the bug, and they pass forever.

It is not a replacement for every kind of evaluation.

The approach pays off when you have steady production traffic, recurring failure modes, and orchestration code that changes often. That is the profile of most customer-facing tool-calling agents.

A sequence that keeps the investment proportional:

AI agent regression testing checks that a change to an agent's code, prompts or model does not bring back a failure that was already fixed. The strongest form builds each test from a real failing trace and replays it on every pull request.

Hermetic replay runs an agent test with recorded tool and model responses instead of live calls. The test needs no network access, no API keys and no token spend, and it gives the same result on every run, which makes it usable as a CI gate.

A fail-to-pass contract is a rule that a regression case must fail on the code that had the bug and pass on the code that fixes it. It proves the test actually reproduces the failure, instead of being a test that cannot fail.

Not yet for most teams. Tracely overlaps with their tracing and judging but is an early project with a small community. Its distinctive parts are fixture-bundled promotion, failure clustering and an offline CI gate, which you can also build on top of an existing platform.

Promptfoo can gate CI on agent outputs and run red-team suites, but it does not ingest production traces. You copy the trace's input into a test case by hand and write your own mocks for tool responses.

No, but it keeps your options open. Tracely, Langfuse, LangSmith and Phoenix all accept OpenTelemetry traces, so instrumenting once with gen_ai.* attributes lets you change backends without re-instrumenting the agent.

Originally published at fp8.co. Subscribe for weekly AI engineering analysis at fp8.co/newsletters.