Beyond LLM JSON mode: decisions your code can branch on

Your support pipeline asks a model which queue a ticket belongs to. It comes back with three sentences: "This appears to be a billing issue, as the customer mentions being charged twice. It could also relate to their account. Billing seems like the best fit." Now your code has to find the queue in that paragraph. So you reach for LLM JSON mode, and the paragraph turns into {"queue": "billing"}.

That is better. It is still not a decision your code can trust, and this post explains why, and what a typed decision looks like instead.

When software gets its decisions back as text, three things go wrong.

You can't see how sure it is. "Billing seems like the best fit" reads the same whether the model was certain or guessing. A ticket that is clearly about a double charge and one that could be billing or account access look identical by the time they reach your if.

The words come first. When a model writes its explanation before it settles the answer, the answer tends to follow the text it already wrote, not the other way round.

Parsing is a guess. Your regex works until the model says "Billing" with a capital B, or "the billing team", or adds a second suggestion at the end.

Structured outputs and JSON mode fix the last problem. The format is constrained, so the response parses and the field is where you expect it.

They don't fix the first two. The model is still writing text underneath, token by token, with no sense of how sure it is. The queue inside your JSON is a token in a string. {"queue": "billing"} carries no number that tells you whether to act on it or send it to a person.

That missing number is the part that matters in production. Routing a ticket wrongly is cheap to undo. Issuing a refund wrongly is not. Without a probability, both get the same treatment.

A typed decision is an answer your code can branch on directly: one of the options you defined, with a probability for every option. Here is the first request from the Wity quickstart, in Python:

import os, requests r = requests. post ( " https://api.wity.alphanimble.com/v1/systemone ", headers = { " Authorization ": f " Bearer { os. environ [ ' WITY_API_KEY ' ] } " }, json = { " state ": " Hi, I was charged twice for my subscription this month. Please refund the duplicate. ", " questions ": { " intent ": { " type ": " choice ", " instructions ": " What does the customer want?

", " criteria ": { " billing ": " A charge, refund, invoice or payment problem ", " technical_support ": " Something in the product is broken ", " account_access ": " Cannot log in or is locked out ", " cancellation ": " Wants to cancel or downgrade ", }, } }, " reasoning ": " auto ", }, timeout = 60, ) r. raise_for_status () intent = r. json ()[ " answers " ][ " intent " ] print ( intent [ " choice " ], intent [ " probabilities " ][ intent [ " choice " ]]) # billing 0.99 Enter fullscreen mode Exit fullscreen mode You send the state (the thing to decide about) and the question, with the options spelled out. Nothing asks the model to "answer in JSON", because the answer was never text. This is what comes back:

{ "model": "wity-1", "answers": { "intent": { "type": "choice", "choice": "billing", "probabilities": { "billing": 0.9912, "technical_support": 0.0031, "account_access": 0.0019, "cancellation": 0.0038 }, "confidence": 0.93, "reasoning": { "mode": "auto", "thought": false, "reason": null, "forecast": false, "thought_tokens": null } } }, "usage": { "input_tokens": 212, "output_tokens": 0 }, "metadata": { "reasoning": "auto", "elapsed_ms": 94.1 } } Enter fullscreen mode Exit fullscreen mode Reading the answer choice is the option to act on. It can only be one of the keys you put in criteria, so there is nothing to normalise. probabilities covers every option you offered and sums to 1. You can read any of them, not only the winner. confidence says how peaked the distribution is.

The docs are clear that it describes the shape of the answer, not the chance of being right: use probabilities for anything that acts, and confidence for sorting a review queue. reasoning.thought is false here. The question was clear, so Wity answered in one fast pass. On a close call it thinks first, and the answer comes back in the same shape.

The full distribution is where the value is. A single label tells you what was picked. The probabilities tell you how the evidence split, and whether a second option deserves a look.

Once the answer is a number, the rule lives in your code, not in a prompt. This is the three-zone pattern from the docs on probabilities, for a question named route with a refunds_queue option:

ans = r. json ()[ " answers " ][ " route " ] p = ans [ " probabilities " ][ ans [ " choice " ]] if ans [ " choice " ] == " refunds_queue " and p >= 0.95: issue_refund () # cheap to undo: act on your own elif p >= 0.80: queue_for_one_click ( ans ) # a person confirms with one click else: send_to_human ( ans ) # genuinely unsure: full review Enter fullscreen mode Exit fullscreen mode Refunds get a stricter bar because they cost money. Other routes need 0.80 because a wrongly routed ticket is cheap to move. You set the lines from what a mistake costs, and you can move them without touching a prompt.

No model is perfect, and these numbers are not a promise about any single answer. That's the reason to have them: act on the clear calls and hand the close ones to a person.

Some actions need words: a city to type into a form, a one-line reply, a field value. For those, Wity's generate primitive writes short text in a shape you define, from the same state, after the decision is made. Output is capped at 100 tokens. If the answer is one of a set you could list, use choice instead, because a decision gives you probabilities and generated text doesn't.

The rule of thumb: decide first, in a type, and write text only for the part of the action that needs it.

The quickstart gets you to your first typed decision in about a minute, and every account starts with $5 of free credit to use within 48 hours of signing up.