LLM API Slows to 30s, Your Server Holds 600 Requests: Little's Law and a Load Shedder

Posted on Nov 1 • Edited on Nov 1 • AI-assisted

Your LLM provider is up. Every call just takes 30 seconds instead of one. Nothing errors. Your health check is green. And your Node process is slowly filling up with requests that are all waiting on the same slow upstream.

The loud failures make the status pages. DeepSeek logged an incident at 8:11 Beijing time on October 15, and chat and API users got "the server was temporarily unavailable" until a fix moved to monitoring at 12:49, per TechNode. Five days later OpenAI posted authentication errors for some users near Illinois, API included, for an hour, fixed by rolling back a configuration change. The slow afternoon usually never gets an entry.

You can predict exactly how full your process gets with one line of arithmetic, and you can stop it with a counter and a fast 503, checked against a simulation you can run.

Little's Law, proved by John Little in 1961, says that in a stable system the average number of items inside (L) equals the arrival rate (λ) times the average time each one spends inside (W). It does not care about the arrival distribution or the order things are served in. That is why it is so useful for back-of-the-envelope capacity work.

Apply it to an endpoint that proxies an LLM call:

Your traffic did not change. Your code did not change. Your concurrency went up 30x because W did. Each of those 600 requests holds a socket to the client, a socket to the provider, the prompt, the buffered response, and a promise chain in memory. Users rarely wait 30 seconds either. Some hit refresh, which raises λ while W stays high.

Netflix's concurrency-limits library builds on the same identity: limit = average RPS × average latency. Flip it around and you get the two levers you have. You can cap W (timeouts) or you can cap L directly (a concurrency limit). λ belongs to your users.

When W goes from 1 s to 30 s, the pipe holds 30 times as much, with no extra traffic.

Real load tests against a slow LLM are expensive and hard to reproduce. A deterministic, tick-based simulation is neither. It steps time forward 100 ms at a time, sends 20 requests per second, and makes the upstream take 30 seconds per call between second 30 and second 90.

It also needs a way to die. Here the server has a 400 MB budget and each in-flight request holds 1 MB. Both are assumptions, picked to make the failure visible; measure your own. When the budget is exceeded, the process crashes, loses everything it was holding, and takes 10 seconds to come back. Save the file as sim.ts and run it with tsx; there are no dependencies.

// sim.ts: run with `npx tsx sim.ts` const TICK = 100; // ms per simulation step const END = 180 _000; // simulate three minutes const RPS = 20; // requests arriving per second const PATIENCE = 15 _000; // a user gives up after 15 s const TARGET = 2 _000; // "healthy" upstream latency const HEAP_MB = 400; // memory budget (assumption) const MB_PER_REQ = 1; // held per in-flight request const RESTART = 10 _000; // crash-to-ready time // Fast, then 30 s per call for one minute, then fast. const upstream = ( t: number ) => t >= 30 _000 && t < 90 _000? 30 _000: 1 _000; type Policy = { name: string; timeout: number; // Infinity = wait forever limit: () => number; // Infinity = no cap onSlow?: () => void; // a call passed TARGET onFast?: ( inFlight: number ) => void; }; type Req = { start: number; end: number; slow: boolean }; Enter fullscreen mode Exit fullscreen mode Each tick of run() does three things: settle what is in flight, admit new arrivals, and check the memory budget. Append it to the same sim.ts.

function run ( p: Policy ) { let inFlight: Req [] = []; let downUntil = 0; let due = 0; // fractional arrivals carried between ticks const s = { ok: 0, late: 0, timedOut: 0, shed: 0, lost: 0, crashes: 0, peak: 0 }; for ( let t = 0; t < END; t += TICK ) { // finish, time out, or flag what is in flight const still: Req [] = []; for ( const r of inFlight ) { if ( r. end <= t ) { const took = r. end - r. start; if ( took <= PATIENCE ) s. ok ++; else s. late ++; if ( took <= TARGET ) p. onFast?.( inFlight. length ); } else if ( t - r. start >= p. timeout ) { s. timedOut ++; } else { if (! r. slow && t - r. start > TARGET ) { r. slow = true; p. onSlow?.(); } still. push ( r ); } } inFlight = still; // new arrivals this tick (exact for any RPS) due += ( RPS * TICK ) / 1000; for (; due >= 1; due -- ) { if ( t < downUntil ) { s. lost ++; continue; } if ( inFlight. length >= p. limit ()) { s. shed ++; continue; } inFlight. push ({ start: t, end: t + upstream ( t ), slow: false }); } s. peak = Math. max ( s. peak, inFlight. length ); // out of memory: the process dies with its requests if ( inFlight. length * MB_PER_REQ > HEAP_MB ) { s. crashes ++; s. lost += inFlight. length; inFlight = []; downUntil = t + RESTART; } } return s; } Enter fullscreen mode Exit fullscreen mode The counters mean: ok answered within the user's patience, late answered after the user left (you paid for those tokens and nobody read them), timedOut your server gave up on, shed rejected instantly, lost dropped by a crash or refused while restarting.

Start with what a naive proxy does: no limit, no timeout. Then add a timeout, then a fixed limit.

const unbounded: Policy = { name: " unbounded ", timeout: Infinity, limit: () => Infinity, }; const timeoutOnly: Policy = { name: " timeout only ", timeout: PATIENCE, limit: () => Infinity, }; const fixed50: Policy = { name: " limit 50 + timeout ", timeout: PATIENCE, limit: () => 50, }; Enter fullscreen mode Exit fullscreen mode You can predict the unbounded run before you execute it. Little's Law says the in-flight count heads toward 600, but the budget is 400, so the process dies about 20 seconds into the slowdown (20 per second × 20 seconds = 400). It restarts empty, refills at the same rate, and runs into the same wall. Every crash throws away hundreds of requests that had each been waiting for many seconds.

The timeout-only run caps W at 15 seconds, so L tops out near 20 × 15 = 300, under the budget. No crash. That is a real improvement, and it shows that a timeout is the cheapest fix there is. The catch is in timedOut: during the slow minute, every single user waits 15 seconds to get an error. And the cap only holds while λ holds. Set RPS = 40, the refresh-button scenario, and 40 × 15 = 600 puts you back over the budget.

The fixed limit caps L at 50 no matter what λ does. Most users in the slow minute land in shed: they get a 503 right away instead of a 15-second spinner. The AWS Builders' Library puts the reason plainly: "Early rejection is important because it's the cheapest place to drop excess traffic, but it comes at a cost to visibility." That second half is why you count and graph every shed request. The weakness is the number 50. You picked it for one latency, and the upstream decides what latency you get.

A concurrency limit turns a slow upstream into fast rejections instead of a growing queue.

Here is the same policy in an HTTP handler. The rejection carries Retry-After, which on a 503 tells the client how long the service is expected to be unavailable, as a number of seconds or a date.

import { createServer } from " node:http "; const LIMIT = 50; const TIMEOUT_MS = 15 _000; let inFlight = 0; createServer ( async ( req, res ) => { if ( inFlight >= LIMIT ) { res. writeHead ( 503, { " Retry-After ": " 5 " }); res. end ( " busy, retry shortly \n " ); return; } inFlight ++; try { const up = await fetch ( process. env. UPSTREAM_URL!, { signal: AbortSignal. timeout ( TIMEOUT_MS ), }); res. writeHead ( up. status ); res. end ( await up. text ()); } catch { res. writeHead ( 504 ); res. end ( " upstream failed or timed out \n " ); } finally { inFlight --; } }). listen ( 3000 ); Enter fullscreen mode Exit fullscreen mode The check runs before any work: no body parsing, no auth lookup, no upstream socket. The finally is the part people get wrong; if the counter does not come down on every path, the limit slowly leaks until the server sheds everything.

On the client side, honour the header and add jitter, or 50 rejected clients all retry in the same second. Google's SRE book describes the client half too: a per-client retry budget where a request is retried only while retries stay below 10% of traffic.

Additive increase, multiplicative decrease is the feedback rule from TCP congestion control: grow slowly while things are fine, cut sharply on a sign of congestion. Netflix's AIMDLimit does this for concurrency: multiply the limit by 0.9 on a drop or a call slower than its timeout, add 1 on success, but only grow when at least half the current limit is in use.

function aimd ( min = 5, max = 200 ): Policy { let limit = 20; return { name: " aimd + timeout ", timeout: PATIENCE, limit: () => Math. floor ( limit ), onSlow: () => { limit = Math. max ( min, limit * 0.9 ); }, onFast: ( inFlight ) => { if ( inFlight * 2 >= limit ) { limit = Math. min ( max, limit + 1 ); } }, }; } const policies = [ unbounded, timeoutOnly, fixed50, aimd ()]; console. table ( policies. map (( p ) => ({ policy: p. name,... run ( p ) })), ); Enter fullscreen mode Exit fullscreen mode The main change from Netflix's version: the sim flags a call as slow the moment it passes TARGET, not when it finally returns. With a 30-second upstream, a completion-based signal arrives 30 seconds after the trouble started. Two seconds in is when you want to know.

Run all four and compare the shed and timedOut columns for the last two rows. During the slow minute the AIMD limit drops toward its floor of 5, so far fewer users sit through a 15-second wait for an error, and the in-flight count stays tiny. When the upstream recovers, fast completions push the limit back up by one each, and the "half in use" guard stops it from inflating while traffic is light.

TARGET is the knob that carries your judgement. At 2 seconds, an upstream that settles at 10 seconds would also be treated as broken, even though a patient user could still be served. Set it from your own latency data, not from this post.

Do the arithmetic for your worst endpoint first: peak RPS times the worst latency your provider has shown you. If that number of in-flight requests would hurt, you need a timeout today and a concurrency limit that answers with a fast 503 and Retry-After. AIMD is the next step once you are tired of guessing the right number. Then change upstream, RPS and HEAP_MB in the sim to match your own service and see which row you are living in.

Little's Law, timeouts, load shedding and backpressure are the parts of system design that only feel abstract until a provider has a slow afternoon. The System Design Pocket Guide: Fundamentals covers those building blocks, with the capacity arithmetic that tells you where your service breaks before your users find it.