Part 2: We Fixed the Root Causes of Our AI Agent Incident. Routing Models by Step Cut Latency 86%

Note: this is a composite scenario built from common production failure patterns in agent systems. Ticket IDs, dollar amounts, latencies, and probabilities are illustrative. I drafted it with AI assistance and edited it myself.

In part 1, our billing agent settled a $48,000 credit it should have routed to a human. We showed why re-running the agent five times proved nothing, then used cut-point replay and a deterministic gate to prove the fix.

The gate stopped the bleeding. It did not explain why the agent reached the wrong conclusion in the first place. This post is about the upstream causes, and about the second half of the TICKET-8841 trace: why a six-step ticket took 42.6 seconds and cost $0.184.

The gate sits at Boundary 5. The agent got there because of what happened at Boundaries 2 and 3. When we held the post-mortem, someone immediately asked:

"Why are we still maintaining a RAG pipeline? Frontier models have 1M-token context windows now. Let's dump the entire 350-page Finance & Contract Handbook into the prompt."

Cost. A 350-page handbook is roughly 175,000 tokens, not 1M. Using an illustrative $3.00 per 1M input tokens:

175,000 tokens × $3.00 / 1M = $0.525 per call × 45,000 tickets/month ≈ $23,600/month (uncached, one call per ticket) with ~90% prompt-cache discount ≈ $2,400/month Enter fullscreen mode Exit fullscreen mode Our retrieval pipeline cost about $600/month. With caching, the cost gap is real but not dramatic, so cost alone isn't the strongest argument. (Plug in your own rates and call counts. Every extra step that carries the handbook multiplies these numbers.)

Recall. Long context is good now, but not perfect. Anthropic's launch numbers for Opus 4.6 on the 8-needle, 1M-token variant of MRCR v2 were around 76%. That's a synthetic benchmark at a much longer length than our handbook, so it says nothing precise about our documents. What it does say is that "the model will find the clause" is a claim you have to measure, and a miss on a financial contract is expensive.

Traceability. This is the argument that mattered most to us. With retrieval, the exact chunks that entered the prompt are a small, loggable, replayable artifact (exactly what a cut-point envelope wants). With a 175K-token dump, you can't tell which clause the model leaned on, and every edit to the handbook changes your entire prompt.

So we kept RAG. Our problem was that we were running 2023-era "RAG v1" in 2026.

Our ingestion script split PDFs every 512 tokens. At 2:14 AM, the vector store held this:

chunk_041 (ended at token 512): "...For Enterprise Annual plans, seat reductions must be submitted in writing at least 30 calendar days prior to the renewal date. Mid-term or late seat reductions are non-refundable and apply only to the following renewal term."

chunk_042 (started at token 513): "Table 4.2 — Seat Downgrade Pro-Rata Credit Formula: credit_usd = (prior_seats - new_seats) * annual_seat_price * (days_remaining / 365)..."

The customer's message mentioned "downgraded 40 seats" and "billed", which matched the formula table well. chunk_042 scored 0.86 cosine similarity; chunk_041, the condition that made the formula inapplicable, scored 0.71 and missed the top-3 cutoff. The planner saw a formula with no precondition.

def test_frozen_bad_rewrite_still_retrieves_notice_clause (): session = CutPointSession ( TRACE, mode = " replay " ) bad_query = session. envelopes [( " query_rewrite ", 1 )]. output # "...credit calculation table" (no "annual") chunks = retrieve ( bad_query ) # new hybrid + rerank + parent-section pipeline assert any ( " 30 calendar days " in c. text for c in chunks ) Enter fullscreen mode Exit fullscreen mode The upstream condition, a sampled rewrite that drops "annual", is still there. The pipeline is now robust to it.

At Boundary 3, the agent wrote SELECT active_seats FROM subscriptions against the warehouse through MCP. The query was valid. The problem was semantics: in our warehouse, active_seats is a live operational telemetry gauge, while contracted_billable_seats is the governed financial metric. The model picked a plausible column, and the nightly dbt job was mid-refresh, so it read 80 instead of 120.

Text-to-SQL was never the hard part; text-to-the- right -SQL is a semantics problem.

So we removed the run_sql() tool. The MCP server now exposes one governed tool that returns defined metrics in a single atomic read:

{ "account_id": "acct_99201", "contracted_billable_seats": 120, "telemetry_active_seats": 80, "notice_window_days_remaining": -13 } Enter fullscreen mode Exit fullscreen mode -13 is the whole story: the downgrade was requested 17 days before renewal, against a 30-day requirement. The agent no longer chooses between active_seats and contracted_billable_seats. The contract chooses, and the data comes from a consistent snapshot rather than a table mid-refresh.

With the full parent section and the governed metrics, the planner can't reach "automated overcharge error." And because the gate from part 1 sits at the exit, even a wrong planner decision can't move money.

Once the replay tests were green, we looked at why TICKET-8841 took 42.6 seconds and cost $0.184. The answer was embarrassing: we'd routed every boundary through a frontier reasoning model.

Asking a frontier model to rewrite a search query or format a JSON payload is paying for deliberation on a task that needs none. The planner decides whether the run succeeds, and the worker tier is close to interchangeable once the spec is unambiguous.

So we kept a frontier model on Boundary 4, the only step that needs multi-clause contract judgment, and routed the rest by task. I'm using model tiers instead of product names, because the rankings change monthly. Benchmark your own candidates against your own replay suite.

End to end, latency fell 86% (42.6s to 5.86s) and cost per ticket fell 82.7%. At 45,000 tickets a month, that's about $8,280 versus $1,430 in inference, before the hosting cost of any self-hosted models.

Across both posts, the same shape keeps showing up. Probabilistic models are extraordinary at turning messy human intent into structured plans. They're bad at being their own database catalog, access-control layer, and regression suite. What worked was wrapping them in deterministic engineering:

If you're debugging an agent right now, try this first: pick your last real incident and ask whether your fix has ever been run against the exact upstream state that caused it. If the answer is "I re-ran it a few times," start there.