Choosing an LLM Inference Optimization Technique: A Decision-Tree Approach

Share Post Share In this article, you will learn how to match an LLM inference optimization technique to the actual bottleneck in your workload by working through a six-question decision tree.

LLM inference gets slow and expensive as traffic grows. Requests queue behind long generations, GPU memory fills with cached attention state, and the hardware spends much of its time waiting on memory reads.

Inference optimization is the set of techniques that help here. Some techniques target memory, some target compute, and some target how requests are scheduled against each other. Each technique assumes something specific about your workload, so the right choice depends on which assumptions hold for yours. This article covers:

We start with what each technique assumes.

LLM inference has many optimization techniques, but each one addresses a different type of bottleneck. Before using one, it helps to understand the workload it is designed for.

Choose the technique based on the bottleneck in your workload to avoid unnecessary complexity.

The tree below has six branching questions. Each one narrows the field based on a specific property of your model or your traffic. Every question ends the same way: a direct answer if the condition holds, and a pointer to the next question if it doesn’t.

Before moving on to the serving stack, check whether the model itself is the problem. A model that barely fits in GPU memory leaves almost no room for KV cache, which caps your batch size no matter how good your scheduler is.

If the model is too large, or inference is compute-bound even at batch size one, start with model-level compression:

These techniques are mostly independent of the serving stack, and you may want to consider them first because they determine how much hardware capacity remains.

If the model already fits with headroom to spare, move to Question 2.

Chat applications resend the full conversation history on every turn. Agent frameworks reuse the same system prompt across thousands of calls. Retrieval-augmented generation (RAG) pipelines often prepend the same instructions to different retrieved contexts.

If a meaningful share of your traffic has an identical prefix, whether that’s a system prompt or a growing chat history, prefix caching is the highest-leverage technique available. It skips recomputing the KV cache for tokens the model has already processed, and the gains compound as conversations get longer.

⚠️ Prefix caching only works when the shared prefix matches exactly, character for character. Put stable, reusable content first and variable content at the end of the prompt.

If your prompts are mostly unique with little shared structure, move to Question 3.

A support chatbot that returns a one-sentence answer and a code assistant that returns four hundred lines rarely finish at the same time, even if they started together. Under naive batching, the whole batch waits on the slowest request.

If your traffic mixes short and long generations and arrives in overlapping bursts, continuous batching fixes the idle time this creates. Finished sequences leave the batch immediately, allowing new requests to start without waiting for a batch boundary. Most serving engines enable this by default.

If your traffic is already well-batched and GPU idle time is not a problem, move to Question 4.

Some engines reserve a fixed, contiguous block of memory per request sized for the maximum possible sequence length, even when most requests are much shorter. That reserved space sits unused for the life of the request and blocks other requests from using it.

If KV cache is wasting memory or limiting batch size, PagedAttention can help. It splits the cache into fixed-size pages, allowing memory to be allocated in smaller chunks instead of large contiguous blocks. Most modern serving engines already support it, so check whether it is enabled before adding new configuration.

If memory is not the constraint, move to Question 5.

If your goal is faster token streaming for each user rather than higher overall throughput, consider speculative decoding.

A smaller draft model predicts several tokens ahead, and the main model verifies them in a single pass. When the draft model predicts correctly, multiple sequential decode steps can be completed together, reducing inter-token latency.

The trade-off is the extra model and verification overhead. If you don’t have a suitable draft model or its predictions are accepted too rarely, the added complexity may not pay off. In that case, move to Question 6.

This is mainly a production-scale concern because it changes the serving architecture.

If long prefills are delaying decode requests and hurting both time-to-first-token and inter-token latency, prefill-decode disaggregation can separate the two phases onto dedicated hardware. Prefill nodes process prompts, then pass the KV cache to decode nodes optimized for token generation.

This adds KV-cache transfer and routing overhead, so it makes sense only when the traffic justifies that complexity. For smaller workloads or non-interactive requests, offline batch inference is often simpler. If none of these conditions apply, start with continuous batching and prefix caching.

Working through the tree lands you on one of the techniques in the following table:

A technique that you think will work may behave quite differently in practice. The signals below tell you when that is happening and what to check first.

If a technique consistently underperforms, check its underlying assumption before adding more complexity.

Choosing an inference optimization technique starts with identifying your bottleneck. Model size, prompt overlap, traffic concurrency, memory fragmentation, and the balance between throughput and per-user latency each point toward a different answer, and most systems will need a combination of a few of these rather than relying on just one.

If you’re interested in learning more, you may find The Roadmap to Mastering LLM Inference Optimization useful.