How to Build Reliable LLM Classifiers With Confidence Thresholds

Signup Write New Story How to Build Reliable LLM Classifiers With Confidence Thresholds by JasonL by JasonL | @ jasonl888 Jason Lau, founder of SophiArch. 20+ years in software engineering before moving into teaching applied ML, LLM apps, and data science.

Jason Lau, founder of SophiArch. 20+ years in software engineering before moving into teaching applied ML, LLM apps, and data science.

TL;DR: Routing, triage, moderation and approval prompts pick an answer from a list you wrote in advance. That makes them classifiers, but they usually skip what makes a classifier safe to run: label definitions the model actually follows, a labelled test set, a probability that means something, and a threshold with an escalation path. Schema-constrained output guarantees a valid label, not a right one.

In a September 2026 study, typed decision models were given the same definitions with the label names swapped between them. With yes/no names, one model changed 76.9% of its decisions, against 6.5% with 0/1 names, and its ranking went from 93.8% AUC to a below-chance 23.2%. The type-error rate was 0% the whole time.

The same test runs on your own prompt in an afternoon and needs no labels. Then check whether the model's confidence ranks its own errors, set the threshold from what a mistake costs, and route everything below it to a person.

The ticket router had been in production for five months when the billing team asked why they kept getting password resets.

It was a single LLM call. The prompt listed six queues (billing, account access, technical, sales, urgent and general), each with a one-line description, and asked for JSON with one field: queue. The response was validated against a schema, so the queue name was always one of the six. The integration had never thrown an error. Latency was fine. The dashboard showed six queues filling up at plausible rates.

When someone finally labelled 500 recent tickets by hand, 40 of them were in the wrong queue. That's 8%, and at 20,000 tickets a month it's about 1,600 tickets a month landing with a team that has to read them, work out they aren't theirs and pass them on. Nothing had flagged it, because nothing in the system could. Every output was a valid queue name.

This example is invented, but the design is common. The fix isn't a better prompt. It's recognising what the call actually is.

A call with a fixed set of possible answers, known before any input arrives, is a classification problem. Ticket routing, content moderation, "does this transaction need review", "is this lead qualified", "which of these five intents is the user expressing". These are classifiers, and classification has a mature discipline that most LLM pipelines skip:

The router had a version of the first item and nothing else. Its prompt returned a label and no score, so there was nothing to set a threshold on and nowhere to send the unsure cases.

Structured output made this harder to see, not easier. Constraining the response to a JSON schema removes the one error you would definitely have noticed, a malformed answer, and leaves every other kind in place. In September, TypeSafe AI took the idea one step further with Jev, a model that doesn't generate text at all. It takes a question, an input and a declared set of options, and returns a probability distribution over those options, so its answer can only ever be one of them. That's a real improvement over parsing free text. It is also a guarantee about the format of the answer, and a study published the same month shows how little that guarantee covers.

Type-Safe Is Not Error-Free (Sun, Xu, Shi and Yang, Fudan University and USTC, September 2026) tested Jev and two open-weight models built the same way: Laya, built on ModernBERT, and Open-Jev, built on DeBERTa-v3. In all three, each option is a short name plus a definition of what the option means. For example, the name "yes" might carry the definition "A human should inspect this run", and "no" the definition "No human attention is warranted."

The experiment was simple. The authors kept the question, the input, the names and the definition texts exactly the same, and only swapped which definition each name was attached to. The new pairing was written out in the input. A model that reads the definitions picks the same definition in both versions, so it returns the other name. A model that reads the names keeps returning the same name, which now means the opposite. The test set was 1,200 binary decisions across four workflows: invoice reconciliation, agent-trace triage, security-alert classification and customer-service escalation.

Selected rows from Table 1 of the paper.

With yes/no names, three decisions in four flipped. The AUC of 23.2% is the more important number. It's below the 50% you'd get by guessing, which means the model's scores still separated the two classes, just in the wrong direction. As the authors point out, AUC doesn't depend on the decision threshold, so "this error cannot be corrected by adjusting the threshold."

Jev itself was less extreme but showed the same pattern. With yes/no names it flipped 32.5% of decisions, 30.4 points more than with 0/1, and its AUC fell from 81.5% to 58.1%. Because Jev isn't deterministic, the authors also ran the original version twice to see how much it changes on its own. At most 1.33% of decisions changed between those runs, so the yes/no flip rate was 24 times that floor.

Across every condition in the paper, the type-error rate was 0%. Every answer was a valid option. The authors' conclusion is the one to remember: "Type safety guarantees where a decision can land, but not what the options mean to the model."

Two results point at the fix. Replacing the names with random five-character strings put all three models back at the neutral-name flip rate, and their accuracy before the swap was within 3.3 points of what they scored with neutral names. The meaning was carried by the definitions, and the model followed them. Polarity mattered more than familiarity, too: "absent/present", "negative/positive" and "rejected/accepted", pairs Laya had never produced during training, still raised its flip rate by 41 points over neutral names.

Nobody deliberately attaches "no" to the definition of yes, so it's fair to ask what this has to do with a production prompt. The swap is a measurement tool, not a production scenario. It's the cleanest way to find out whether a model reads your definitions or your label names. On ordinary inputs the two agree, so accuracy can't tell you which one the model is following.

It matters wherever your definition is narrower than the word. The router's "urgent" queue was defined as "a production system is down or customers cannot log in." A ticket that says "I urgently need a copy of last month's invoice" is a billing ticket by that definition and an urgent one by the dictionary. If the model weights the name over the definition, that's where it goes. The same applies to "spam" defined to exclude legitimate cold sales email, "fraud" defined as a specific set of chargeback patterns, or "qualified" defined by your sales team's criteria rather than the word's general sense.

None of this is new for language models in general. Wei et al. (2023) showed that small language models "ignore flipped labels presented in-context and thus rely primarily on semantic priors from pretraining," that overriding those priors is "an emergent ability of model scale," and that instruction tuning "strengthens both the use of semantic priors and the capacity to learn input-label mappings, but more of the former." Liusie, Manakul and Gales (2023) found that in zero-shot prompt classifiers "the precise choice of the prompt template and label words can largely influence performance, with semantically equivalent settings often showing notable performance difference." What the 2026 paper adds is a clean way to measure the effect, run on the newest kind of model built to avoid it.

The useful part is that the test needs no labels. It only needs your prompt, a sample of real inputs and two extra calls per input. Here's a minimal version for a prompt that lists options as name: definition. classify stands for your existing call, constrained to return one of the allowed names:

def build_prompt(text options): """options: list of (name, definition) pairs, shown in this order.""" listed = "\n"join(f"- {name}: {definition}" for name definition in options) return ( "Pick the option whose definition fits the ticket. " "Reply with the option name only.\n\n" f"Options:\n{listed}\n\nTicket:\n{text}" ) def picked_definition(text options classify): name = classify(build_prompt(text options), allowed=[n for n _ in options]) return dict(options)[name] # compare what was chosen by meaning, not by name def flip_rate(texts names definitions classify shift=): """Share of inputs where moving each name to another definition changes the decision.""" original = list(zip(names definitions)) moved = list(zip(names[shift:] + names[:shift], definitions)) # shift=0 measures run-to-run noise flips = sum( picked_definition(t original classify)!= picked_definition(t moved classify) for t in texts ) return flips / len(texts) Run it three ways on a few hundred real inputs:

If the first number is close to the other two, the model is reading your definitions. If it's far above them, your definitions are partly decoration, and the label names are making some of the decisions. In that case the fix the paper suggests is cheap: use neutral IDs as option names, put all the meaning in the definitions, and map the ID back to your queue name in code. Then re-run your accuracy check, because the change isn't automatically free. On questions with three to sixteen options, even switching Laya to neutral letters changed about half its decisions, though that model was only 56.4% accurate on those questions to begin with. Measure your own prompt rather than assume.

A caveat on scope: the paper tested typed decision models, not chat models answering through a JSON schema. Whether your model behaves like Laya, like Jev or like neither is exactly what these three numbers tell you.

The second thing the router lacked was a score. There are two common ways to get one from an LLM, and only one of them is worth much.

Asking the model for its confidence. Adding a "confidence" field to the schema is easy and mostly uninformative. In a large evaluation of this approach, Xiong et al. found that LLMs "tend to be highly overconfident when verbalizing their confidence," and that the values "predominantly fall within the 80% to 100% range and are typically in multiples of 5, similar to how humans talk about confidence." In the router example, the model reported 0.9 or higher on 488 of the 500 labelled tickets. A threshold has nothing to separate.

Reading the probability of the label itself. If your provider exposes token log-probabilities for the chosen output (not every API or model does), use the probability the model assigned to the label it picked. Single-token, distinct option names make this straightforward. It's a real score, but don't assume it's calibrated. The GPT-4 technical report showed that "the pre-trained model is highly calibrated," but "after the post-training process, the calibration is reduced," and its calibration plot is captioned "the post-training hurts calibration significantly." Typed decision models like Jev return a distribution by design, which is one of their real advantages. The same checks still apply.

What you need to know is whether the score ranks the model's mistakes. Group your labelled set by score and count how often each group is right. Here is what that table might look like for the router (illustrative numbers):

The errors aren't spread evenly. A third of the tickets (the 170 below 0.95) contain 33 of the 40 mistakes. That's what makes a threshold possible.

The table also shows the scores aren't perfectly calibrated: the tickets below 0.60 are right two times in three. If you want the number itself to be trustworthy, fit a calibration correction on a held-out slice of your labels: temperature scaling, which Guo et al. (2017) found "surprisingly effective" for neural networks, or Platt scaling or isotonic regression on the raw scores. Reliability diagrams and those correction methods are the subject of the calibration lesson in SophiArch's Applied Machine Learning course, which works through them on gradient-boosted and random-forest models, whose raw probabilities need the same kind of correction.

A threshold isn't a quality setting. It's a trade between two costs, and you should write both down. Suppose a misrouted ticket costs about 20 minutes of someone's time (the wrong team reads it, works out it isn't theirs, passes it on, and the customer waits longer), and a person sorting a ticket by hand takes about 1 minute and gets it right. Per 500 tickets:

At these costs, sending everything below 0.95 to a person cuts the cost of routing by about 60% (from 800 minutes per 500 tickets to 310). If sorting a ticket by hand took three minutes instead of one, the 0.80 threshold would win: 320 + 240 = 560 minutes against 140 + 510 = 650. The right threshold comes from your costs, not from the model.

The tickets sent to a person do something else too. Every one of them comes back with a correct label, from exactly the cases the model was least sure about. After a few months that's a better training set than you could have designed.

Once a decision has labels, a score and a threshold, you can compare different ways of making it on equal terms. A reasonable order to try them in:

The exit test is the same at every rung: accuracy, the score table and the swap flip rate, all on your own labelled traffic. Not the vendor's benchmark, and not the model's own description of how confident it is.

The router team answered all five in two weeks. The billing queue stopped getting password resets, and the "urgent" queue got a lot quieter.

Why Adding More AI Agents Doesn't Always Make Your System Smarter