How to Stop a Long LLM Answer From Freezing When the Server Runs Out of Memory

Posted on Nov 1 • Originally published at theaicommit.com

Engineer's view. This is memory overcommit. Back every page up front and half your RAM sits idle. Overcommit, and an out-of-memory killer stops a process at peak. An LLM server faces the same choice, because each request's memory grows with its answer. You pay in idle GPU memory or in frozen streams.

TL;DR. An LLM server must decide how much memory to hold for an answer before the model has written it. Hold too much and you waste capacity. Hold too little and a running answer gets paused to make room.

Picture a restaurant where nobody knows how long each family will stay. You could save a giant table for every family, just in case. Most families leave early, so half your seats sit empty while people wait outside.

Or you could seat everyone at small tables and hope. When the room gets full, you ask the newest family to stand up and wait, even in the middle of dinner. They keep their food. They just can't eat for a while.

An AI server runs that restaurant, and every answer is a family.

You have shipped this bug already, in a system with no model in it.

You ran a service on Kubernetes. You set its memory request to the worst case you had ever seen. Nothing crashed. But the cluster reported itself full while the real usage sat near 40%, and new pods stayed pending.

So you lowered the request to the average. The cluster filled up nicely. Then a busy afternoon came, several pods grew at once, and the kernel killed one in the middle of its work.

You were guessing a number before you knew the answer. An LLM server guesses that number for every request.

The memory here is the KV cache: the model's saved work for every token so far. Without it, the model would reread the whole conversation for each new token. It grows by one token per token generated. The model only decides the length by stopping, so at the moment a request starts, nobody knows its final size. The caller sends max_tokens, but that is a ceiling, not a forecast.

Under load, that guess shows up in two ways. Your GPU memory looks full while throughput sits far below what you paid for. Or a streaming answer stops mid-sentence for many seconds, then carries on as if nothing happened. Neither one logs an error.

The fix: find out which rule your server uses, measure how long your answers really are, then set max_tokens and your concurrency limit from those lengths. New to the KV cache? Start at How to Work Out How Much GPU Memory One Request Really Needs.

Admission is the moment a server lets a waiting request start. Three popular servers give three different answers to one question: what must be free first?

TGI counts the prompt plus the full max_new_tokens against its budget. It cannot run out mid-answer.

SGLang counts max_new_tokens times a ratio, clipped at 4,096 tokens. The ratio starts at 0.7 and falls to about 0.1 while things go well. When memory runs out, it resets from what requests really used.

vLLM counts only what exists now. Its --watermark setting can keep some memory free, but the default is zero.

The server evicts a running request. It frees that request's cache and sends it to the front of the waiting line. vLLM picks the newest request. SGLang picks the one with the least output.

The answer isn't lost. Tokens already generated are kept. The request must rebuild its cache by rereading the prompt and its own output, and only then does the stream move again. In vLLM, nothing new starts during a step that evicted.

I expected it to be. It wasn't. My first simulation was built to show a server thrashing, and throughput never collapsed. Rereading is much faster than generating.

Eviction redid 80% of the prefill work and still delivered about five times the throughput. The price landed elsewhere: 26% of finished requests were evicted, and their median freeze was 58 seconds. Throughput hid that completely.

When this matters. You run your own model server, or you are choosing one. It matters most when answers vary widely in length, as they do with reasoning models and agents. If you only call a hosted API, this is the provider's problem. You can still detect it from your side, and your max_tokens may still matter.

How it affects you. The same GPU can give very different results depending on which rule it runs. On a reserving server, one team's careless limit costs everyone. In my chat simulation, moving max_tokens from 8,192 to 4,096 raised throughput 29%, from 351 to 454 tokens per second. Answers were short, so about 0.5% would have been cut. On an evicting server, average throughput looks healthy while a few users stare at a stopped stream. Your dashboard's mean hides them.

What to do about it. Start with no setup. Log the largest gap between chunks on your streaming calls. A gap of many seconds in the middle of an answer is your first sign.

Next, log how many tokens your answers really use. Compare that with the max_tokens you send. Most teams send a default nobody chose.

Then cap concurrency from those lengths. On an evicting server, that cap removed more freezing than any other setting I tried. Finally, check the eviction counter and alert on it.

Four jobs, and one person may do all of them.

Job 1, whoever owns the client. Measure the freeze from the outside. This works against any server. In your streaming call:

gap, last = 0.0, None for chunk in client. chat. completions. create ( model = MODEL, messages = msgs, stream = True ): now = time. monotonic () gap, last = max ( gap, now - last ) if last else 0.0, now metrics. observe ( " stream_max_gap_seconds ", gap ) Enter fullscreen mode Exit fullscreen mode Start timing after the first chunk, as this does, so that queue time isn't counted as a freeze.

Job 2, whoever sets max_tokens. Choose it from logged lengths, not from the model's limit. Take a value a little above your longest normal answer, then count what it would cut:

limit = sorted ( lengths )[ int ( 0.999 * len ( lengths ))] cut = sum ( n > limit for n in lengths ) / len ( lengths ) Enter fullscreen mode Exit fullscreen mode A wrong limit shows up as finish_reason: "length". Alert on that rate.

Job 3, whoever runs the server. Work out two limits from the same logs. safe_concurrency() in code_example.py does this:

never = pool / ( mean_prompt + p95_answer ) # almost no evictions, low throughput average = pool / ( mean_prompt + mean_answer ) # the memory is full on an average mix Enter fullscreen mode Exit fullscreen mode pool is your KV cache size in tokens. Those limits came out at 4 and 15 for the long-answer case. The upper limit is the useful one here. It gave up 17% of throughput, cut evicted requests from 26% to 7%, and cut the median freeze from 58 to 17 seconds.

Job 4, the setting for your server. On vLLM, lower admission without shrinking the batch buffers:

vllm serve " $MODEL " --max-num-active-seqs 15 Enter fullscreen mode Exit fullscreen mode --max-num-seqs also works, and the docs recommend it. Be careful with --watermark: at 0.2 it cut redone work from 80% to 53%, but the median freeze rose to 247 seconds, because an evicted request then waits for extra room.

On SGLang, watch for the pool-full warning. Its tuning guide says to raise --schedule-conservativeness to about 1.3 if the warning fires often. Lower it to about 0.3 if token usage stays low while requests queue. On TGI, set --max-total-tokens as low as your longest real request. TGI is in maintenance mode, so weigh that for a new deployment.

rate(vllm:num_preemptions_total[5m]) Enter fullscreen mode Exit fullscreen mode vllm:kv_cache_usage_perc should stay high, or your cap is too low. On any server, p99 of stream_max_gap_seconds should fall while total tokens per second holds. If usage drops and waiting requests climb, you set the cap too low. Raise it until evictions just reappear.

When you never fill the pool. If vllm:kv_cache_usage_perc stays well under 1 at peak, no rule is making a difference. Fix something else.

When batch work doesn't care. A nightly job nobody watches can freeze for a minute at no cost. Leave concurrency high, and take the throughput.

When you are simply out of capacity. A cap moves waiting from the stream to the queue. It doesn't make more GPU. In my runs, the queue still grew, and your time to first token will show that.

When a limit would cut real work. A low max_tokens on a reasoning model can end the answer before it starts. Check finish_reason first.

Treat my numbers as a shape, not a measurement. They come from a simulation that follows each server's source code, with invented lengths and an assumed prefill speed. Single runs also varied widely: the same setup froze for 26 to 170 seconds across nine seeds. It ignores prefix caching, which can make a reread much cheaper. I did not run these servers on a GPU. If prefill is slow, or contexts are long, eviction costs more than it did here.

Three questions before you change anything:

Originally published at theaicommit.com, a daily AI engineering lab for software engineers. The original has an interactive version you can play with, and the full runnable code is on GitHub.