Case Study: A Record/Replay Cassette That Makes an AI Agent Test Deterministic

The first failure in this build had nothing to do with model quality: the agent's test suite went red because a network call timed out, then green again ten minutes later with no code change. Record every model response once into a plain JSONL file, replay it forever in CI, and fail loudly whenever the prompt or request shape drifts. That single rule turned a flaky agent job into a deterministic one, and the replay side costs nothing to run.

Below is the full case study of a small classify-and-route agent: the constraints, the cassette module, the drift guard that saved the most debugging time, and the parts I would not recommend.

The project was small on purpose. One job reads a failing test log, asks a model to pick the likely owner module, and posts a suggestion to a review thread. Three functions, one prompt file, and roughly forty lines of glue code — the kind of thing you write in an afternoon and then maintain for a year.

The problem showed up in week two. Every assertion compared model output against expected strings, so the suite depended on sampling, provider latency, and whatever the current model snapshot preferred that morning. A red build told you nothing, because you could not separate a regression from a slow Tuesday.

The usual fixes are worse than the disease. Mocking the client returns whatever you typed, so the test asserts your own fiction. Skipping the test in CI means nobody notices when the prompt file quietly loses a section.

Providers attach ids, timestamps, and usage counters to responses, and tool-call blocks carry randomly generated identifiers. If you hash the raw payload, identical requests produce different keys, and your cassette grows one duplicate entry per run.

import hashlib, json VOLATILE = { " id ", " created ", " system_fingerprint ", " usage " } def normalize ( request: dict ) -> dict: body = { k: v for k, v in request. items () if k not in VOLATILE } body [ " messages " ] = [ { " role ": m. get ( " role " ), " content ": m. get ( " content " )} for m in body. get ( " messages ", []) ] return body def request_key ( request: dict ) -> str: blob = json. dumps ( normalize ( request ), sort_keys = True, separators = ( ", ", ": " ), ensure_ascii = False ) return hashlib. sha256 ( blob. encode ( " utf-8 " )). hexdigest () Enter fullscreen mode Exit fullscreen mode Put a template_sha field into the request as well. Editing prompts/classify.md then changes the key automatically, which is exactly the behaviour you want.

One line per call keeps diffs readable and lets you append without rewriting the file. Record mode is idempotent: if the key already exists, return the stored response instead of paying for a second call.

# cassette.py import json, time from dataclasses import dataclass, field from pathlib import Path from typing import Callable class CassetteMiss ( RuntimeError ): def __init__ ( self, key: str, mode: str ): super (). __init__ ( f " no recorded response for key { key [: 12 ] } (mode= { mode } ) " ) self. key = key @dataclass class Cassette: path: Path mode: str = " replay " # "record" | "replay" call: Callable [[ dict ], dict ] | None = None entries: dict = field ( default_factory = dict ) def __post_init__ ( self ): self. path = Path ( self. path ) if self. path. exists (): for line in self. path. read_text ( encoding = " utf-8 " ). splitlines (): if line. strip (): entry = json. loads ( line ) self. entries [ entry [ " key " ]] = entry def send ( self, request: dict ) -> dict: key = request_key ( request ) if self. mode == " record ": if key in self. entries: return self. entries [ key ][ " response " ] # idempotent re-record if self. call is None: raise RuntimeError ( " record mode requires a call function " ) response = self. call ( request ) entry = { " key ": key, " recorded_at ": int ( time. time ()), " request ": normalize ( request ), " response ": response, } self. entries [ key ] = entry with self. path. open ( " a ", encoding = " utf-8 " ) as fh: fh. write ( json. dumps ( entry, ensure_ascii = False ) + " \n " ) return response try: return self. entries [ key ][ " response " ] except KeyError: raise CassetteMiss ( key, self. mode ) from None Enter fullscreen mode Exit fullscreen mode 3. Turn a miss into a readable diff A missing key is the most common failure, and a bare hash tells you nothing useful. Because you stored the normalized request next to the response, you can show the closest entry and print only the fields that changed.

def explain_miss ( cassette, request ): current = normalize ( request ) best, best_score = None, - 1 for entry in cassette. entries. values (): score = sum ( 1 for k, v in current. items () if entry [ " request " ]. get ( k ) == v ) if score > best_score: best, best_score = entry, score if best is None: return " cassette is empty; record a new entry " diffs = [ f " { k }: recorded= { best [ ' request ' ]. get ( k )!r} current= { current. get ( k )!r} " for k in sorted ( set ( best [ " request " ]) | set ( current )) if best [ " request " ]. get ( k )!= current. get ( k ) ] return " \n ". join ( diffs ) Enter fullscreen mode Exit fullscreen mode Nine times out of ten the diff names template_sha, and you immediately know a prompt edit caused the miss rather than a provider outage.

# tests/conftest.py import os, pytest from pathlib import Path from cassette import Cassette @pytest.fixture def cassette ( request ): mode = os. environ. get ( " CASSETTE_MODE ", " replay " ) path = Path ( __file__ ). parent / " cassettes " / f " { request. node. name }.jsonl " return Cassette ( path, mode = mode ) Enter fullscreen mode Exit fullscreen mode Run plain pytest and the suite replays. Run CASSETTE_MODE=record pytest -k classify only when you intentionally change behaviour, then review the cassette diff exactly like any other fixture file.

Replay suites are dangerously easy to keep green forever. Add one test that compares the recorded template hash with the current prompt file, and make the failure message actionable.

# tests/test_freshness.py import hashlib, json from pathlib import Path def test_prompt_matches_cassette (): template = Path ( " prompts/classify.md " ). read_bytes () first = Path ( " tests/cassettes/test_classify_issue.jsonl " ). read_text ( encoding = " utf-8 " ). splitlines ()[ 0 ] recorded = json. loads ( first )[ " request " ][ " template_sha " ] assert recorded == hashlib. sha256 ( template ). hexdigest (), ( " prompt changed; re-record the cassette on purpose " ) Enter fullscreen mode Exit fullscreen mode Where free model access and a free server fitted in The record pass is the only step that needs a live model, and it is also the step that runs least often. That is precisely where MonkeyCode's free model access and free server option were useful in this build: the recording run happened on the hosted free server rather than behind a paid key on someone's laptop, and the resulting JSONL was committed for everyone else.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two cautions matter more than the convenience. Treat any free tier as a moving target and check the project's current terms for which models are available and what limits apply, because availability and quotas change. Also remember that a free model is a reasonable recording source and a poor benchmarking source, so do not publish quality comparisons built on it.

If you want to try the record pass before wiring a paid key, start with the project's free model access and free server option, commit the cassette, then delete your provider credentials and confirm the replay suite still passes.