Measuring error in support AI, and what accuracy hides

Posted on Oct 9 • Originally published at field-notes-6cd.pages.dev

The way I test a support or ticketing system is against a task book, not a demo. The number people ask for afterwards is accuracy, and I have stopped handing that over on its own: on every realistic test set I have built, the average was the least informative thing in the results file.

What follows is the method I use now, plus a run on public data so the numbers can be checked.

A support model fails in at least five ways, and they do not cost the same.

Fabrication. The reply asserts something absent from the policy or the record: a fee that does not exist, a wrong refund window, an invented delivery date. This is what people call hallucination. The taxonomy I lean on, OlaBench, splits it into four sub-types — factual hallucination, misuse of retrieved results, relevance hallucination, and logical inconsistency ( OlaBench, arXiv:2510.22143v3, §3.1; v1 25 Oct 2025, v3 25 May 2026, retrieved 9 Oct 2026). A sentence that reads well and is false costs money, and the cost usually arrives months later.

Missed detection. The ticket contains a legal threat, a regulator's name, a safety issue or a fraud signal, and the system files it as routine. Nothing false was said. The failure is that no human ever reads it. On a scoring sheet this often counts as a correct classification, because the model did pick a queue that exists.

Misrouting. The reply is reasonable and the queue is wrong, so a differently-skilled team applies its own policy. The measurable cost is time to resolution and repeat contact.

Out-of-scope answers. The system answers a question it was never built for — tax, legal advice, another provider's product. The content may be accurate. The exposure is regulatory.

Refusal. The system declines, loops, or hands off. There is no answer, so there is no error to score. This one gets tracked as a transfer or containment rate and drops out of the error count entirely.

One accuracy figure blends these five at whatever ratio the sample happens to contain, and that ratio is set by whoever picked the sample.

I needed a routing task with a ground-truth label and text a model would really see. Public complaint data has both.

The US CFPB publishes its Consumer Complaint Database under CC0 ( CFPB Consumer Complaint Database, retrieved 9 Oct 2026); the search API's own metadata reports 18,274,023 records, license CC0, last updated 2026-10-08 ( API metadata, retrieved 9 Oct 2026).

The JSON search endpoint returns 15 fields and no narrative ( field reference, retrieved 9 Oct 2026). That is new: the September 2026 release removed complaint narratives from the database ( CFPB release notes, Release 24, September 2026). So I took the text from a Hugging Face mirror instead, BEE-spoke-data/consumer-finance-complaints ( dataset card, CC0, mirror last modified 29 Dec 2025); its has-text config holds 1,689,573 rows ( split sizes, retrieved 9 Oct 2026).

Task: read the narrative only, predict the product queue. Ground truth: the product label the CFPB publishes. I drew 3,894 complaints from it. The oldest is dated 2015-03-20, the newest 2024-01-04.

The raw product field holds 17 distinct strings in this sample, and they are not 17 products. The date ranges are what give it away.

Read the dates down the table. Bank account or service last appears here on 2017-01-31, and Checking or savings account first appears on 2017-04-24. Three more strings end in 2017 and their successors start the same year. The three credit-reporting strings sit end to end: 2015 to 2017, 2017 to 2023, 2023 into 2024. Same category, renamed.

I collapsed the 17 strings into 9 categories before scoring anything. The mapping is in the script, applied in one place rather than inferred per row. Without that step an accuracy compares a 2016 row and a 2023 row whose label sets do not line up, and the model gets credit or blame for the taxonomy.

The live database disagrees with the mirror about which names are current. Aggregated over complaints received in 2026, the CFPB API returns 11 product strings, with Credit card and Prepaid card listed separately and no Credit card or prepaid card at all ( product aggregation, complaints received 2026-01-01 onward, retrieved 9 Oct 2026). I did not reconcile the two taxonomies.

Imbalance. After collapsing, the sample looks like this:

A predictor that answers "credit reporting" for every ticket scores 0.572 here. That is a constant, and it sits close enough to a real model's score to be mistaken for one.

Sample selection. Same data, same model, two splits:

The chronological run scores 7.6 points higher while getting worse on most categories. The test set became more concentrated in the biggest class, and the dominant-class predictor rides that concentration up. Anyone reporting the 0.832 as an improvement after retraining would be reading their own sampling as progress.

Coverage. Sort the test set by the model's confidence and score only the top slice:

The 0.904 at half coverage is a real number computed on held-out data, and published alone it misdescribes the system by 15 points. Note where it stops moving: from 50% coverage down to 10%, accuracy drifts from 0.904 to 0.897, so the model's confidence and its correctness come apart on the tail. That tail is where the misroutes are.

Underneath the average, the per-category picture on the random split:

Three categories are never once predicted correctly. Accuracy is 0.756. Macro F1 is 0.428. The distance between those two is the honest headline for this model.

The useful published idea here is OlaBench's risk axis, Critical Business Risk Rate. It counts only responses that inappropriately assert one of four things, each a high-stakes failure that may trigger compliance exposure, user disputes or reputational damage: admitting platform liability (618 test cases), misidentifying the ICS role (228), overcommitting (138), and disparaging individuals or merchants (16) ( OlaBench, arXiv:2510.22143v3, §3.1 and Table 1; 25 May 2026, retrieved 9 Oct 2026). It is reported as its own number, beside the hallucination rate.

The shape of that is what to copy: choose failures whose consequences differ in kind, count them separately, publish both, and never average them.

For the routing test I defined three tiers, and I chose them:

The two splits swap places depending on which tier you read: the one with the prettier accuracy hides a critical error rate almost twice as high.

Published work also shows how far the same named metric can move when the population changes. The OlaBench paper reports a critical business risk rate of 8.7% for its OlaMind-Stage-2 model on the benchmark, and a critical business risk rate below 0.05% in daily manual annotation of live dialogues after launch ( OlaBench, arXiv:2510.22143v3, §5.2 and Table 3; 25 May 2026, retrieved 9 Oct 2026). Both are the same metric, measured on different populations with different annotators. Quoting one without the other is a choice.

I keep the longer write-ups and the raw measurements at Field Notes — no ads, no tracking, one-click unsubscribe.