How to Turn Production Sessions into Regression Tests for Your AI Agent

Most agent test suites are written at a desk. Someone imagines twenty realistic questions, writes the expected answers, and runs them before each release. They pass, the agent ships, and then production produces a failure nobody imagined: an odd phrasing, a tool that times out at the wrong moment, a customer who changes their mind halfway through.

Your best source of test cases is already sitting in your logs. Every real failure, escalation, and dispute is a case your agent has actually met. This guide shows how to turn those sessions into regression tests with ZizkaDB, so each incident you fix stays fixed.

A note on scope: the changelog documents these SDK and API capabilities: log_batch, iter_events, export_events and import_events, verify_chain, search with a mode option, baseline, and idempotency_key on writes. It does not give every parameter, so the code below is illustrative. Check the SDK READMEs for exact signatures, and use zizkadb-sdk 0.5.0 or later.

Why production sessions make better tests

They are real. Synthetic cases reflect what you expect users to do. Production sessions show what they did.

They carry the full story. A session of linked events records the path: which tools ran, what they returned, and where the first wrong step happened.

They match your failure history. Every incident becomes a permanent check, so a bug cannot quietly return after a prompt change.

They keep your suite honest. As users and tools change, new sessions refresh your picture of normal.

Step 1: Define what regression means for your agent

Decide what a test should check before collecting anything. A final-answer check catches only one kind of regression:

Outcome: did the agent reach the right result? Path: did it take the required steps, such as verifying an amount before a refund? Tools: did it call the right tool with sensible arguments, and avoid forbidden ones? Cost and speed: did it stay within a token and time budget? Safety: did it avoid saying or doing something it must never do?

Write two or three checks for each important workflow. Path checks are the most valuable and the least common, because they catch an agent that reaches a correct-looking answer by skipping a safeguard.

Step 2: Make sure the record is complete

A regression test is only as good as the session it came from. Before relying on your record, check these:

Events are linked with parent IDs, so the path can be read as a chain. Tools are named, with results recorded. The SDK’s log_tool and log_tool_result do this, and the LangChain, LiveKit, and MCP integrations already name tools. A prompt or configuration version is logged, so you know which version produced the session. Token usage is recorded, for cost budgets. The Python SDK’s log() accepts token_usage, while the TypeScript SDK does not, so check the README for the shape. Retrieved context is recorded or referenced, since a wrong answer is often a wrong retrieval.

Cases missing these can still be kept as outcome-only tests, but you lose the path checks.

A well-chosen set of fifty to two hundred cases beats thousands of random ones. Look in these places:

Failures: sessions that ended in an error, a timeout, or an abandoned conversation. Escalations and disputes: anything a human had to fix or a customer complained about. Expensive sessions: the top few percent by cost, which often reveal loops. Drift periods: the baseline view accepts time windows of 24h, 7d, and 30d, so you can pull sessions from the period when behavior shifted. Described patterns: use search to pull sessions matching a behavior you can describe in words. Normal sessions: a sample of ordinary successful cases, so the suite also protects what works. python

async for event in db.iter_events("support-bot"): # cursor paging, stable at any depth... # filter by type, session, or time

hits = await db.search("agent kept asking for documents already uploaded", agent="support-bot", mode="auto")

One accuracy point on search: with embeddings off, which is the self-hosted default, search is keyword based. Any word can match and prefixes count, so refund finds refunded. Meaning-based matching needs embeddings, either a hosted provider or a local OpenAI-compatible endpoint such as Ollama. Describe the behavior using words that would actually appear in the logs.

Step 4: Replay each case, find the first divergence, and trace its impact

Open the session in replay and read it from the first message to the last. Write one sentence on what should have happened. Mark the first step where the session diverged. Many failures begin two or three steps before they become visible. Run zizkadb why on that step to find the cause: bad input, bad context, a failing tool, or a reasoning error. Use the impact route, GET /v1/events/{event_id}/impact, to see everything that descended from the bad step. It is the mirror of why, and it shows which later events your test should also check. Label the case with workflow, cause category, and whether the original outcome was good or bad. Step 5: Verify the source, then export the cases

Before you turn sessions into fixtures, confirm the record is intact. ZizkaDB links every event into a per-agent hash chain, and a verify call checks the whole chain and every checksum:

report = await db.verify_chain("support-bot") print(report) # intact flag, summary, problems, and chain_head

Save the chain_head outside ZizkaDB. Later you can pass it back as an anchor to show that older history has not been rewritten. Note that checksums show a record has not changed since it was written, not that the logged content was correct.

Then export. The export route streams an agent’s events as NDJSON, oldest first, and you filter down to your chosen sessions in your own script:

events = await db.export_events("support-bot") cases = [e for e in events if e["session_id"] in chosen_sessions]

Because export works per agent, filter by session yourself. For CI jobs and evaluation tools, create a read-only API key with scope: "read_only". It can query, search, export, and verify, but every write or delete route refuses it, and it belongs to one agent. You can also set expires_in_days so the key stops working after a set time.

Production sessions contain real people’s information, and test suites live in repositories and CI systems with wider access.

Check redaction. ZizkaDB replaces values under secret-named keys, such as password and api_key, with [REDACTED] before storage, and optional patterns can redact emails, phones, cards, IBANs, and IPv4 addresses inside strings. This applies at write time, so older events written before you configured it are not cleaned retroactively.

Inspect your exports. Replace personal details with synthetic ones that keep the same structure, such as a made-up name and order number. Never copy secrets into case files.

Link to the source session by ID only, and keep access to the original controlled. Tag cases with a subject identifier where a real person’s data remains, so an erasure request can find the fixtures as well as the production events. Remove fixtures when the source is erased.

Forget deletes events from ZizkaDB, but copies in your repository stay until you delete them. Tombstones in the chain mark where data was removed without keeping content. Set a retention rule for fixtures, since a copy is also stored data.

Step 7: Freeze the case

A regression test needs the same inputs every time. For each case capture:

The input: the user messages in order. The context: retrieved documents or memory given to the agent, or fixed stand-ins. The tool results: what each tool returned, so the test stubs them and avoids live systems. The expected result: the outcome, plus the path and tool checks from Step 1. Metadata: source session, date, configuration version, cause category, and who labeled it. python

{ "case_id": "refund-amount-001", "source_session": "sess_8842", "captured_with": "prompt v14", "inputs": ["I was charged twice, please refund the second charge"], "tool_stubs": {"lookup_order": {"order_id": "ORD-8842", "charges": [49.0, 49.0]}}, "expect": { "outcome": "refund of 49.00 offered", "must_call": ["lookup_order", "verify_charge"], "must_not_call": ["close_account"], "max_input_tokens": 12000 }, "cause_category": "skipped verification" }

Stubbing matters. Without it, tests depend on live data that changes, and failures become hard to interpret.

Step 8: Run the suite on a disposable instance

This is where self-hosted setups differ from the earlier advice. A self-hosted deployment runs one agent with one API key, and multiple agents are a Cloud feature. That means you should not write test runs into your production agent, because they would distort your behavior baselines and cost figures.

A throwaway instance in CI. The single container image starts with one command, so each run can start fresh, load the cases, run the candidate agent, assert, and discard. The container image is ghcr.io/zizka-ai-sl/zizkadb, and ENV=development gives you the local one-click setup. Separate agents on Cloud, where multiple agents are available, with test traffic under its own agent name.

Load the cases with import_events. The documented behavior keeps original timestamps, remaps parent links, gives every event fresh checksums and chain links, and returns duplicates instead of copies if you re-run an import. Run the candidate agent against each case with tool results stubbed.

Write the runs in batches with log_batch, up to 500 events per request, and give each write an idempotency key built from run, case, and step, so a retried CI job does not double count. Assert by reading events back and checking required tools, order, arguments, budgets, and outputs. Compare against the last passing release, and list the cases whose results changed.

Compare overall behavior. With enough runs, the event and transition distribution of your test sessions shows shifts that individual cases did not cover. Block the release, or require a named approver, if a must-pass case fails. python

await db.log(agent="support-bot", event="tool_call", data={"tool": "verify_charge", "case": "refund-amount-001"}, parent_id=prev_event_id, idempotency_key=f"{run_id}:refund-amount-001:step3") Step 9: Handle non-determinism honestly

Language models do not give identical output twice. A test that passes once may fail on the next run for no real reason.

Run each case several times and track a pass rate, not a single result. Set thresholds by risk. A safety check might require every run to pass, while a style check might tolerate a few failures. Lower randomness in tests where your provider supports it. Investigate flaky cases. One that fails one run in five is telling you your prompt or tools are fragile. Pin the model version in tests, and treat a provider model update as a release that needs a test run. Step 10: Grow, prune, and review

Add every incident. After each production failure and fix, add the session as a case. Tag cases by workflow and cause, so you can see which weaknesses recur. Retire stale cases when a product rule changes. Use point-in-time state. The at() function shows which prompt, tools, and model were live when a case was captured, so you can tell whether an old case fails because of a regression or a deliberate change. Mark replaced facts. The supersedes option on writes lets a newer event replace an older one in memory and search without deleting history. It helps if your fixtures rely on memory. Review monthly: number of cases, pass rate, flaky cases, and how many production incidents were already covered. What the business case looks like

These figures are modeled assumptions, not measured results. Replace them with your own.

Assume a team ships 8 changes a month and 1 in 8 causes a regression that reaches customers. Each costs about 20 engineering hours to find, fix, and communicate, at 100 euro per hour, plus 1,500 euro in support and goodwill. That is about 3,500 euro per regression, so 3,500 euro a month.

If a suite built from real incidents catches two thirds of them before release, the saving is about 2,300 euro a month, or 28,000 euro a year.

Add the cost to build it. Reviewing and labeling 100 sessions at about 20 minutes each is roughly 33 hours, or about 3,300 euro, plus a few days of engineering for the harness. Payback is a matter of months, and each later incident adds a case for about an hour of work.

A second benefit is hard to price: confidence. Teams that can show a release passed a suite of real past failures ship prompt and model changes more often and with less fear.

Collecting thousands of cases. A small, varied, well-labeled suite is easier to maintain and trust.

Testing only the final answer. Path and tool checks catch regressions a correct-looking answer hides.

Running tests against your production agent. On self-hosted, there is only one agent, so use a disposable instance or a separate Cloud agent.

Using live tools. Results change, and failures become hard to read.

Leaving personal data in fixtures. Redact, replace, and tag before saving, and delete copies when the source is erased.

Skipping source verification. Check the chain before trusting a session as a fixture.

Re-running CI without idempotency keys. Retries create duplicate events.

Ignoring flaky cases. They point to fragile prompts or tools.

Never retiring cases. Stale tests block valid changes.

Pick your last five production incidents. For each one, open the session in replay, find the first divergent step, check what descended from it, and write a case file with inputs, stubbed tool results, and the expected path. Verify the chain, export those sessions, load them into a throwaway instance, and run them against your current build and then your next change. That is a working regression suite, and every future incident makes it stronger.

If you tell me what your agent does and share a few failed sessions with personal data removed, I can help you draft the case format and the first assertions.

Want to test ZizkaDB on your agent? Try our open source version here: https://github.com/ZIZKA-AI-SL/ZizkaDB

Interested in a design partnership? Fill in the form on our site or reach me directly at founder@zizka.ai

Sources: ZizkaDB changelog · ZizkaDB repository README · ZizkaDB Python SDK wiki

Expand hacksicilian hacksicilian hacksicilian Follow I run a small fleet of AI agents and measure what they actually cost — by effect, not estimate. I write about agent cost, governance, and reading sessions token by token. Joined Oct 22, 2026 • Oct 29 Dropdown menu Copy link Hide Step 1's "stay within a token and time budget" is the check I'd handle most carefully, because it's the one most likely to go flaky for reasons that have nothing to do with the agent's logic, which is your Step 9 problem one layer down.

A raw max_input_tokens assert on a replayed session drifts: the same conversation costs different amounts depending on cache state (cold vs warm prefix), where the tool list changed between turns, and how much prior context got re-sent after an invalidation. None of that is behavior you're trying to pin.

The budget check gets stable if you assert on the part the agent controls: tokens it added this turn (new input + output), kept separate from tokens that were only re-sent (the cached/repeated prefix). A real regression shows up as more added tokens for the same task; a cache or infra wobble shows up in the re-sent column and shouldn't fail the test. If your event record already logs token_usage, splitting it into added vs re-sent (cache-read vs cache-write where the provider exposes them) turns one noisy number into two that mean different things, and only one of them is a regression signal. Same spirit as pinning the model version in tests: pin what the agent did, and let the billing-side variance live in its own column.