When Should an LLM Router Stop Being Sticky? llm-d 1.0 Cache-Affinity Routing, Break-Even Math, and a Simulation

The announcement lists a lot: prefix-cache and load-aware routing, KV cache offloading, prefill/decode disaggregation, wide expert parallelism, batch and async inference, multi-tenant quality of service, router high availability with hot standby, and session-aware routing for agents. The founding contributors named are Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, with more than 500 contributors from over 25 organizations. The GitHub release notes add a few operational details: a DisaggregatedSet that manages prefill and decode replicas as one logical unit, a router configuration API that graduates to v1, and deprecation of the llm-d-rocm container image builds in favor of upstream vLLM.

Most of that is plumbing you will only appreciate during an incident. The routing default is different, because it decides how often your GPUs recompute a prompt prefix they have already seen. If you serve long shared system prompts (coding agents, RAG with a fixed preamble, multi-turn chat), prefix reuse is the largest single lever on cost per token, and the router is the component that either preserves it or throws it away.

IBM's announcement also quotes figures such as 11.5 times lower inference cost than an all-proprietary configuration at roughly a 98% prefix-cache hit rate. Those are the project backers' numbers with workload details I could not check, so I treat them as a statement of how much headroom a high hit rate can give, not as something to plan capacity around.

Round-robin ignores caches. A fleet of 10 endpoints, each able to hold only a few long prefixes, ends up with every endpoint seeing every prefix, evicting constantly, and recomputing most of what it receives. Affinity routing fixes that by sending a request to an endpoint that already has its prefix.

The failure mode is the other direction. In llm-d's own words, affinity concentrates traffic on cache-warm endpoints, and without an explicit saturation release the warm endpoint keeps absorbing load past the point where a cold endpoint would serve the request faster. Real traffic is skewed. A few popular prefixes carry a large share of requests, so the endpoint that owns the most popular prefix becomes a queue.

The previous llm-d default tried to handle this with a four-signal weighted blend: a prefix-cache scorer at weight 3, plus queue, KV-cache-utilization and no-hit-LRU scorers at weight 2 each. The authors describe the behavior as hard to predict and harder to tune. That is a fair description of any weighted sum where one signal is in tokens, one is a count and one is a ratio: the weights encode an exchange rate between units nobody measured.

The replacement splits the decision in two. A filter keeps only endpoints whose cached prefix match is at or above an affinityThreshold of 0.8. Then one load scorer, matched to the workload's bottleneck, picks among them: token-load-scorer (uncached prefill tokens in flight) for prefill-bound traffic, active-request-scorer (active request count) for decode-bound traffic. When the best warm endpoint's load passes the saturation limit tau, the filter stops restricting and the request goes wherever load is lowest.

The limit comes from two things you can measure:

tau = R_peak x T_max R_peak = peak prefill throughput per endpoint (tokens/s) T_max = how much TTFT degradation you will tolerate Enter fullscreen mode Exit fullscreen mode For the benchmark setup (Qwen3-32B, tensor parallel 2 across two H100s per server, vLLM with max-num-batched-tokens=8192 ), the post reports T(B = 8192) = 0.40 s. I checked the rest of the chain: 8,192 / 0.40 = 20,480 tokens/s; 20,480 x 14 s = 286,720; and 286,720 / 8,192 = 35.0 chunks. The numbers are internally consistent.

Here is the same calculation as code, with the break-even and the KV sizing I use below. The KV geometry comes from the model's config.json on Hugging Face (64 layers, 8 KV heads, head dimension 128, bf16):

def saturation_tau ( batch_tokens, batch_seconds, ttft_tolerance_s ): r_peak = batch_tokens / batch_seconds # prefill tokens/s per endpoint return r_peak, int ( r_peak * ttft_tolerance_s ) def kv_bytes_per_token ( layers, kv_heads, head_dim, dtype_bytes = 2 ): return 2 * layers * kv_heads * head_dim * dtype_bytes # K and V r, tau = saturation_tau ( 8192, 0.40, 14 ) print ( r, tau, tau / 8192 ) # 20480.0 286720 35.0 per_tok = kv_bytes_per_token ( 64, 8, 128 ) print ( per_tok, 55000 * per_tok / 2 ** 30 ) # 262144 bytes, 13.43 GiB per 55k prefix print ( int ( 64 * 2 ** 30 // ( 55000 * per_tok ))) # 4 prefixes fit in a 64 GiB budget Enter fullscreen mode Exit fullscreen mode That last line matters. At 256 KiB per token, one 55,000-token prefix is 13.4 GiB of KV. A server with a 64 GiB KV budget (my assumption, not a figure from the post) holds four of them.

A coding-agent fleet with dozens of distinct long prefixes cannot cache them all on every endpoint, which is exactly why routing decides the hit rate.

The post compares matched configurations against Kubernetes Service round-robin on 10 Qwen3-32B servers (8 for the third workload). All figures below are from llm-d's own write-up, not an independent test.

The pattern is the useful part. Where prefill dominates and prefixes are long, affinity roughly doubles to triples capacity. Where decode dominates, there is little prefill to save, so throughput is at parity and the only gain is a later knee. The same routing rule is not a universal win, and the post's choice to pair the load signal with the bottleneck is the honest way to say so.

Take one request whose prefix of P tokens is cached on endpoint A and absent from endpoint B. It carries U new tokens. On A it costs U tokens of prefill after waiting q_A tokens in queue. On B it costs P + U tokens after waiting q_B. Sticking is faster for that request while

q_A + U <= q_B + P + U => q_A - q_B <= P Enter fullscreen mode Exit fullscreen mode The queue gap you should accept is exactly the number of tokens the cache saves. For P = 55,000 at 20,480 tokens/s that is 2.69 s. A cached request costs 73 ms of prefill; a cold one costs 2.76 s, a ratio of 37.7.

So llm-d's 14 s tolerance is not the per-request break-even. It is 5.2 times larger. That looks wrong until you count what a spill costs beyond the one request. A cold fill burns 2.76 s of the receiving endpoint's capacity, and it inserts a 13.4 GiB prefix into a cache that holds four, evicting something another request was about to hit. The single-request formula ignores both. The next section tests whether that externality is large enough to matter.

I built a small discrete-event simulation. It is illustrative, loosely modeled on the code-generation workload, and it is not a reproduction of llm-d's benchmark. The assumptions:

Three policies: round-robin; always sticky (the warm endpoint with the least queue, spill only when no endpoint is warm); and sticky until saturated (spill when the best warm endpoint's queue passes tau).

warm = [ i for i in range ( N ) if g in cache [ i ]] least = min ( range ( N ), key = lambda i: pend [ i ]) # pend = queued tokens if warm: bw = min ( warm, key = lambda i: pend [ i ]) e = bw if pend [ bw ] <= tau else least # stick until saturated else: e = least Enter fullscreen mode Exit fullscreen mode

Three things in that table can be checked by hand, which is how I convinced myself the simulation is not producing noise.

Round-robin's hit rate is 35%, so its average request costs 0.35 x 1,500 + 0.65 x 56,500 = 37,250 tokens. Ten endpoints deliver 204,800 tokens/s, which caps the fleet at 5.5 requests/s. The simulation falls over between 4 and 6. Its huge numbers at 10 req/s and above are the backlog after a 10 minute window, not a steady-state latency.

Always-sticky ties each prefix to one endpoint. The hottest prefix takes 27.2% of traffic at 73 ms per request, so its endpoint saturates at 1 / (0.272 x 0.0732) = 50 req/s. The simulation is healthy at 40 and in trouble at 60. That is hot-spotting, and a saturation release fixes it: at 60 req/s p90 drops from 121.8 s to 3.21 s.

At low rates p90 sits near 2.76 s for both sticky policies. That is one cold fill, and it comes from first-time prefixes and LRU evictions. No router removes it.

The last row is the part I did not expect to write. At 80 req/s the saturating policy does worse than always-sticky (1,000 s against 292 s) because the fleet is simply short of capacity, and spilling turns scarce capacity into cold fills. A saturation release helps a skewed fleet that has total headroom. It does not create headroom.

Now the sweep that answers the break-even question. Same simulation at 60 req/s, varying tau only:

Below roughly 110,000 tokens the system falls into the same state as round-robin. A spill creates a cold fill, the cold fill evicts a warm prefix, the next request for that prefix misses, spills again, and the hit rate sinks to 35% and stays there. Above the cliff everything is fine, and then latency climbs slowly as stickiness turns back into hot-spotting. The shape is a cliff on the low side and a slope on the high side, which argues for erring high.

Be careful with the exact position of the cliff. It comes from my parameters (4 slots, 40 groups, 60 req/s) and a router that sees perfect state. A real fleet has chunked prefill, imprecise cache indexing and a different mix. What transfers is the mechanism and the asymmetry, not the number 110,000.

The 2.9x and 2.0x figures come from the llm-d authors on their own 10-server cluster and workloads. They are a strong reason to test, not a forecast for your traffic. The decode-bound result at parity says the same thing from the other side.

My simulation is a one-queue-per-endpoint toy. It omits decode, continuous batching, cache-indexing error and multi-tenant priorities, all of which the production router handles. It supports one claim only: a stickiness limit set at the single-request break-even can be catastrophic, because the cost of a spill lands on other requests.

What is prefix-cache-aware routing? Sending a request to an inference endpoint that already holds the KV cache for the start of its prompt, so the endpoint can skip recomputing those tokens and only prefill the new ones.

What does "sticky until saturated" mean in llm-d? Requests stay on the cache-warm endpoint until its queued token load passes a calibrated limit (tau = R_peak x T_max). After that, routing falls back to load alone.

How is llm-d's 286,720-token limit derived? In the published setup, 8,192 tokens take 0.40 s, so one endpoint prefills 20,480 tokens/s. With a 14 s TTFT degradation tolerance, 20,480 x 14 = 286,720 tokens, which equals 35 batches of 8,192.

Is round-robin ever the right choice? Yes, when all your prefixes fit on every endpoint, or when requests share little prefix. Then there is no cache to protect, and the decode-bound benchmark at parity shows the gain can vanish.

Why not set the limit at the break-even queue gap? Because a spill costs more than one request: a cold fill uses 2.76 s of the receiving endpoint's capacity and can evict a warm prefix. In my simulation, limits at or below about twice the break-even collapsed the hit rate to 35%.

Does this apply outside Kubernetes? The logic does. It needs only per-endpoint queue depth in tokens and some record of which endpoint holds which prefix, which you can approximate by hashing prompt prefixes.

Sources: llm-d, "Sticky Until Saturated: Token-Aware Routing in llm-d"; llm-d, "llm-d v1.0: State-of-the-art inference, ready for Day 2"; IBM Newsroom, 2026-10-20; Qwen3-32B config.json.