Small LLM vs Frontier Model: Which One Does Your AI Feature Actually Need?

Posted on Oct 19 • Originally published at pykero.com

Most AI features inside a SaaS product do not need a frontier model. If the task is bounded (classify a ticket, extract fields from a document, draft an email from facts you already have), a small model with a strict prompt and an output validator is usually good enough and costs a fraction per call. Reserve the frontier model for open-ended reasoning, long multi-step tool use, and anywhere a wrong answer costs real money.

Founders tend to ask us which model is best. That is the wrong question, because the best model on a leaderboard is rarely the best model for a feature that will run fifty thousand times a month inside your product. The right question is: what is the cheapest model that passes my eval set at the accuracy I need, with the guardrails I can afford to build?

Take the ticket-classification example from the top of this post and make it concrete. At fifty thousand tickets a month, the leaderboard winner and a distilled "flash" model will both tag the obvious tickets the same way. The difference shows up on the ambiguous ones, and a validator that checks the tag against your allowed list and retries on a miss catches a good share of what the small model gets wrong. The only way to know whether the remainder is acceptable is to run the fifty to a few hundred real tickets in your eval set through both models and compare, with the per-call price next to each result.

Framed that way, the decision stops being a bet on vendor marketing and becomes an engineering measurement. The small-model category has moved fast; the recent wave of "flash" and distilled releases from several labs means a model that would have been state of the art two years ago now costs very little per token and runs on a single GPU. Whether that is enough for you depends on four things.

A bounded task has a known input shape, a known output shape, and a small set of acceptable answers. "Pull the invoice number, due date and total from this PDF" is bounded. "Read this customer's last six emails and decide what to do" is not.

Small models are strong on bounded tasks because the prompt can carry most of the intelligence: the schema, the rules, three worked examples. They degrade on unbounded tasks because they need to improvise, and improvisation is exactly what gets worse as parameter count drops.

If a deterministic check can catch most bad outputs, you can run a smaller model and let the validator do the heavy lifting. JSON schema validation, regex on dates and amounts, a lookup against your own database, a length cap: these cost nothing per call and turn a model's occasional mistake into a retry instead of a bug a customer sees. We wrote about this pattern in deterministic guardrails for AI feature reliability.

If the only way to check the output is for a human to read it, the model has to be right the first time more often, and that pushes you up the price list.

A mis-tagged support ticket costs an agent ten seconds. A wrong dosage summary in a healthcare tool, or a wrong refund decision in an e-commerce agent, costs a lot more. Price the failure, not the call. Where the failure is cheap, optimise for cost per call. Where the failure is expensive, the frontier model's per-token premium is trivial next to the risk, and you should also add a human review step regardless of model.

Small models typically have shorter usable context and get lost sooner in long documents. If your feature needs to hold a 200-page contract in view, or carry state across a twenty-step workflow, you are either chunking aggressively (more engineering) or paying for a model that can hold it. Multi-step agentic work with tools is the clearest case for frontier models, and it is worth reading agents versus workflows before you assume you need the agent at all.

Our own outbound tool is a concrete example. It crawls each prospect's website with a self-hosted Firecrawl instance, hands the page text to a local model, and writes one tailored email per company. The task is bounded: the input is a few pages of text, the output is a short email following our template and tone, and a validator rejects anything that is too long, mentions a product we do not offer, or fails to cite a fact from the site.

What surprised us was that a single call doing "extract the facts and draft the email" beat a multi-step chain (extract, then plan, then write) on both cost and quality, even with the smaller model. Fewer hops meant fewer places for the model to drift from the source text. We covered the general case in single call versus agent chains. The point for a buyer: a well-constrained small model plus a validator was the right tool, and a frontier model would have added cost without changing the reply rate.

Be honest about the cases where cheap is expensive. The outreach engine above gets away with a small model for one reason: there is a single hop, page text in and email out, with a validator standing behind it. Remove any of those conditions and the picture changes.

That last one matters. Starting on a frontier model and moving down is a measured decision backed by an eval set. Starting on a small model and discovering in production that it is not good enough is a rewrite of your prompts and guardrails under pressure, with no eval set to tell you when you are done.

Founders expect the small model to make the project cheaper to build. It mostly does not. The build cost is in the prompt design, the validator, the eval set, the retry and fallback logic, and the integration into your product. That work is the same whichever model sits behind it. If anything, a small model needs a thicker validation layer, which is slightly more engineering, not less.

What changes is the running cost. Per-call prices between a frontier model and a small hosted model differ by a large multiple, and over a year of real traffic that difference can exceed the original build budget. Ask for a projection at three volume levels before you sign; our guide to LLM cost optimisation lists the levers that matter beyond model choice, such as caching and prompt length.

Self-hosting the small model is a separate decision. Tools like vLLM and Ollama make it technically easy, but it only pays off at steady volume or when data residency forces your hand. At typical early-stage volumes, a cheap hosted API model wins once you count the engineering time.

Whatever you decide, do not let it be baked into the code. Insist on three things in the brief:

Done this way, the model question becomes a dial you turn as traffic and evidence come in, instead of a decision you make once with no data.

If you are scoping an AI feature and want a straight answer on which model tier it needs and what it will cost to run, let's talk.

Originally published on the Pykero blog.