What If a Small AI Model Could Speed Up a Big One Without Changing Its Answers?

Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

Language models generate text one token at a time. Each new token depends on the tokens generated before it, so the model usually needs another forward pass before it can produce the next token.

For a small model, this may be fast enough. For a large model, however, generating text can feel slow. The reason is not always what you might expect.

You might assume the GPU spends all its time performing heavy computations. In many inference workloads, that isn't the case.

To generate a token, a large language model must access its weights, which can occupy many gigabytes of memory. For models limited by memory bandwidth, moving these weights from memory to the GPU's compute units can take longer than performing the calculations themselves.

As a result, the GPU's compute units may sit underutilized while waiting for data.

Think of a chef who spends 10 minutes fetching ingredients from a distant pantry and only 10 seconds cooking. Faster cooking wouldn't solve the main problem. Fewer trips to the pantry would.

This is similar to memory-bound inference: the bottleneck is moving data, not necessarily performing calculations.

During autoregressive generation, the model typically processes one new token at a time. But when processing multiple known tokens together, the model can reuse the weights it has loaded instead of repeatedly loading them for each token.

Processing five tokens together can therefore cost much less than running five separate single-token steps, especially when inference is memory-bound.

This does not mean five tokens always cost the same as one. The exact cost depends on the model, hardware, sequence length, and workload. But processing several tokens in parallel can make better use of the available computing resources.

Now imagine you already had a good guess for the next five tokens. The large model could check those guesses together rather than generating every token sequentially.

But where would those guesses come from?

Speculative decoding uses a second, smaller model called a draft model to propose upcoming tokens.

The larger model, called the target model, then verifies those proposals.

The editor can review an existing draft more efficiently than writing every word from scratch. If most of the draft is correct, the editor can approve several words at once and correct the first mistake.

That is the basic idea behind speculative decoding.

Suppose the draft model proposes four tokens:

In this example, the target model produces three accepted tokens plus its correction in one verification round. That gives four new tokens instead of the single token it would ordinarily produce in a standard autoregressive step.

If all four draft tokens are accepted, the target can also contribute a bonus token, potentially producing five new tokens in that round.

The important point is that the target model verifies several proposed tokens together rather than generating each one separately.

Sometimes the very first guess is wrong. When that happens, the big model rejects it and picks the word itself.

So you still get one correct word, just like you would without this trick. The only cost is a little wasted time: the small model made a guess, and the big model checked it, but the guess was thrown away.

If the small model is wrong more often, each check produces fewer words, so you save less time.

But here’s the important part: if it’s set up correctly, the final text is exactly what the big model would have written on its own.

In other words, this trick makes writing faster without changing the quality or the style of the answer. Only the speed changes.

When a model writes, it gives a score to every possible next word. There are two common ways to pick one:

This section is about the first way, where the big model always picks its top choice.

In that case, checking the small model’s guess is simple:

For example, say the sentence so far is “The capital of France is”. The small model guesses “ Paris”, and the big model’s top choice is also “ Paris”. Match, so we keep it. If the small model had guessed “ Lyon”, we would replace it with “ Paris”.

Result: the final text is exactly the same as what the big model would have written alone, just faster. (This holds as long as you use the same big model and the same settings.)

Sometimes the big model adds a bit of randomness when it picks a word. If we kept every guess from the small model just because it sounds reasonable, the final text would start to follow the small model’s habits instead of the big model’s.

So we don’t keep guesses automatically. Each guess gets a chance of being kept, based on how much the big model likes it. If a guess is rejected, the big model picks a replacement.

This way, the final text still matches what the big model would have written on its own.

In short: the small model suggests, but the big model decides.

The speedup depends heavily on how often the target model accepts the draft model's proposals.

Under this simplified model, the expected number of tokens produced per target-model verification pass is:

For k=4, the expected number of tokens per verification pass is approximately:

These numbers show how many words we get each time the big model checks the small model’s guesses. They do not show how much faster the whole thing runs.

The real speed also depends on other things: how fast the small model is, how long the checking takes, and the extra work of running two models together.

So getting more words per check doesn’t mean you get the same amount of speedup overall. To know the real gain, you have to test it on your own model, computer, and task.

Speculative decoding works best when the draft model can predict the target model's next tokens accurately.

It may offer less benefit for open-ended creative writing, where many continuations are plausible and the draft model may disagree with the target more often.

The task alone doesn't determine the outcome. The acceptance rate between the particular draft and target models is what matters.

Several inference frameworks support speculative decoding. In Hugging Face Transformers, one approach is called assisted generation, which allows a smaller assistant model to help a target model generate tokens.

output = target_model. generate ( ** inputs, assistant_model = draft_model, max_new_tokens = 200, ) Enter fullscreen mode Exit fullscreen mode Here, target_model is the model whose output you want, and draft_model proposes candidate tokens for it to verify.

This is an illustrative example. The models and inputs must be loaded and configured appropriately, and the selected model combination must be supported by the Transformers version you use.

Speculative decoding addresses a key bottleneck in language model inference: generating tokens sequentially can repeatedly incur the cost of accessing a large model's weights.

A smaller model proposes several tokens, and the larger model verifies them together. When the proposals are accepted frequently enough, the target can produce multiple tokens per verification pass while preserving its original output distribution.

The central trade-off is simple: the more accurately the draft model predicts the target, the more opportunity there is to accelerate generation.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

LiveReview is built for engineering teams that ship often, but also care about keeping production stable for customers. It approves some 40-50% of low-risk changes automatically, and it gives your team the tools to handle the rest:

Auto-Approve Upto 50% of Your Pull Requests, Safely. Risk scoring decides which MRs/PRs are safe enough to ship without a human, so your team only spends review time on risky and critical changes.

LiveReview auto-approving a low-risk change while …