Stop Calling an LLM for Every Ticket: Compile the Decision Once, Then Run It Locally

A support queue gets a few thousand new tickets a day. Most need one of three destinations: billing, engineering, or account support. The usual AI implementation sends each ticket to a hosted model, asks for a category, parses the answer, then repeats the process for the next ticket.

That is a reasonable first prototype. It can become an expensive and difficult-to-audit production path when the decision space is small, the same instructions repeat, and every output must fit a strict schema.

A different design is showing up in the open-source project STROT: use a larger language model once to compile the task, then use a smaller decision model locally for each record. The runtime selects from known labels instead of generating free-form text. If its confidence gate cannot make a safe decision, it abstains for a human or another system to handle.

This guide walks through that pattern with STROT's Python API. It is based on the project's public README and developer guide, not a claim of personal testing. STROT is new, and its performance figures are project-reported measurements. Treat the example as a starting point for an evaluation, not as a production endorsement.

This architecture is for repeated, bounded decisions. The system receives a record and must choose among known actions, answer a boolean question, or assign a score inside a defined range.

It is a poor fit when the task needs fresh world knowledge, open-ended writing, complex multi-step reasoning, or an answer that is not represented by your labels. A small classifier cannot replace a generative model just because it is cheaper. The design works by making the decision space explicit and bounded.

STROT describes its architecture as a compile phase followed by a runtime phase. The compile phase profiles a sample, asks an LLM to synthesize a configuration, validates the result, tests known answers, and optionally fits calibration. The saved artifact then handles later records without another LLM call on a cache hit. The repository describes this lifecycle in its developer guide.

The project separates its base package from optional runtime integrations. Its README lists these extras:

pip install 'strot-ai[local,anthropic]' Enter fullscreen mode Exit fullscreen mode The local extra supplies dependencies for the default local decision model. The anthropic extra enables the documented Anthropic synthesizer. If your environment uses a different supported synthesizer, install the matching integration and configure its credential using that provider's instructions. Do not store API keys in source code, fixtures, or a committed.env file.

For a small experiment, start with a representative fixture rather than a production export. Remove secrets and personal information first. The compile stage can inspect sample fields and values, so the data you provide is part of the prompt and the resulting artifact's design.

The registry is the boundary of what the runtime may choose. Keep the keys stable and descriptions distinct. Overlapping descriptions make a classification problem harder, not easier.

ACTIONS = { " billing ": " Invoices, duplicate charges, refunds, and payment failures ", " engineering ": " Crashes, broken features, error messages, and technical defects ", " account_support ": " Login trouble, profile changes, and account access ", } Enter fullscreen mode Exit fullscreen mode These are examples, not universal categories. If a ticket can reasonably belong to two actions, clarify your routing policy before asking a model to learn it. Decide whether the system should choose the first owning team, the team that can resolve it, or a queue for further triage.

A golden record is an input paired with the answer your application expects. It acts as a promotion gate: an artifact that fails one of these known cases should not become the active artifact.

GOLDEN = [ { " record ": { " subject ": " Charged twice for my monthly plan ", " body ": " I see two payments on my bank statement. ", }, " expect ": " billing ", }, { " record ": { " subject ": " The export button crashes ", " body ": " Clicking export returns an error every time. ", }, " expect ": " engineering ", }, { " record ": { " subject ": " I cannot sign in ", " body ": " The password reset link says my account is unknown.

", }, " expect ": " account_support ", }, ] Enter fullscreen mode Exit fullscreen mode The examples should cover one clear case per action at minimum. Add edge cases that have caused real confusion, such as a refund request caused by a software defect. Do not build goldens out of only easy examples.

The developer guide says goldens are checked before promotion and recommends three to five as a practical starting range, while emphasizing that compile results can vary and the gate is there to catch failures.

Golden records are not a substitute for a held-out evaluation set. They are a compact regression check, not evidence that the task works on all future traffic.

Here is the documented run() shape, with a small list of representative records:

import strot sample = [ { " subject ": " Refund for a duplicate payment ", " body ": " My card was charged twice this morning. ", }, { " subject ": " Search returns an error ", " body ": " The results page fails after I submit a query. ", }, ] result = strot. run ( sample, query = " Route each support ticket to the team that can resolve it.

", action_registry = ACTIONS, llm = " anthropic/claude-opus-4-8 ", golden = GOLDEN, state = ".strot ", ) for outcome in result. outcomes: print ( outcome [ " action " ], outcome [ " confidence " ]) for fallback in result. fallbacks: print ( " Needs review: ", fallback [ " reason " ]) Enter fullscreen mode Exit fullscreen mode The provider identifier above is the example documented by the project, and may change as provider models and integrations evolve. The code shows the expected shape, not a verified compatibility test for today's account or API.

On a cache miss, STROT's guide describes profiling the sample schema, synthesizing labels and routing logic, validating the candidate, checking goldens, and saving an artifact. Later calls with the same cache key load the saved artifact instead of involving the synthesizer again. Changing configuration, registry, schema fields or types, goldens, package version, or other keyed settings can trigger a new compile. Review and monitor that boundary instead of assuming every deployment is a cache hit.

A classifier should not be forced to guess every time. STROT reports outcomes and fallbacks separately. That separation is operationally useful: the application can route confident decisions automatically while putting uncertain records into a human queue.

Do not equate a confidence number with a guarantee. Calibration requires representative labeled examples, and its behavior depends on the assumptions being met. The STROT guide distinguishes a marginal error bound across all records from the error rate among records that were actually dispatched. Those are not interchangeable. It also warns that distribution shift and imperfect teacher labels weaken the interpretation.

For a consequential workflow, start by sending every record to a human while logging the model's proposed outcome. Compare those proposals against reviewed labels. Only then consider automating a narrow, high-confidence slice. Keep the fallback path real and staffed; an abstention that silently disappears into a backlog is not a safety mechanism.

Before comparing the result with a large model, compare it with the simplest reasonable alternative:

Report more than overall accuracy. Look at per-class precision and recall, false routing cost, fallback rate, latency on your hardware, and performance on ambiguous or newly worded tickets. A 90% aggregate score can hide a failure on a rare but high-impact category.

STROT's repository reports 92.5% compiled routing accuracy versus 74.8% for a naive approach on a 398-record task, and 86.7% versus 72.7% on an AG News task. These are project-reported measurements, not independent results or a forecast for your data. Read the benchmark code and reproduce it before relying on the numbers. The project's own guide also explains that calibration samples need to match the runtime distribution and that small samples limit the granularity of statistical claims.

A useful mental model is to treat the compiled artifact like generated code. Put it through review and deployment controls:

There is an important security caveat. STROT's developer guide says its security lint is validation, not a sandbox, and that candidate code is executed during the golden gate. Do not compile untrusted intent on a sensitive machine without external isolation, locked-down permissions, and resource limits. The project recommends CI compilation and shipping a frozen artifact as a deployment pattern.

Use compile-once decision workflows when the task is repetitive, bounded, and measurable; labels are stable; local runtime is valuable; and an uncertain answer can safely be escalated. It may reduce runtime API calls, but there are still costs for compilation, model downloads, hardware, operations, evaluation, and human fallback.

Keep a generative model in the loop when the output must be novel, the answer requires current knowledge, or the input may introduce categories that your registry cannot express. You can also use a hybrid: let the local decision layer handle routine cases and send abstentions or unusual records to a more capable model or a person. That fallback is often more sensible than asking a small model to solve every case.

The useful question is not whether a small local model beats a large hosted model in general. It is whether your particular workflow contains many repeated decisions whose answer space you can define, test, and safely constrain.

The project is interesting because it reframes where the LLM belongs. Instead of paying for a fresh generation on every record, it uses the model to help construct a bounded decision process, then keeps that process inspectable and local at runtime. That is promising for narrow jobs, but the tests, monitoring, and human fallback still belong to the team shipping it.

I write about practical AI infrastructure and developer tools. Subscribe if you want more build-focused guides like this.

Have you found a repeated classification or routing task that could be compiled into a bounded decision process? Where would you draw the line between automatic handling and human review?