Your LLM gateway's token budget is a suggestion

If several teams share one LLM bill, someone eventually asks for a hard cap: team A gets two million tokens an hour, and not one more. Most LLM gateways offer that. I wanted to know how hard the cap really is, so I built a gateway, put a budget on it, and hammered it with concurrent streams.

With 128 streams in flight per tenant, spread across three gateway replicas, the usual implementation let a tenant spend 3.3× its budget. The approach this post describes held the same tenant to 0.03% over: 5 tokens on a 20,000-token budget.

The code, the benchmark and the raw results are open source: github.com/saikirankalidindi/hardcap.

A classic rate limiter counts requests. That's fine when every request costs about the same. An LLM request can cost 10 tokens or 100,000, and you don't know which until the response is done: output length is decided by the model, one token at a time, while the stream is open.

So gateways count tokens instead. The common implementation is:

Step 3 is the problem. Between admission and settlement, a request is invisible to the budget. Run one request at a time and you overshoot by at most one response. Run 128 at once and all 128 are admitted while the tenant still looks under budget, and all 128 are charged when they finish. The overshoot grows linearly with concurrency, which is exactly what the blue line shows.

Replicas make it worse, not better. Even with shared state in Redis, nothing reserves the tokens that in-flight requests are about to use.

The fix comes from payments, where it is old news: a card terminal authorizes an amount before the sale and captures the real amount after. Applied to tokens:

admit used + reserved + estimate ≤ limit? → reserved += estimate stream every 64 tokens: grow the reservation (or stop if the budget is gone) settle reserved -= reservation; used += actual Enter fullscreen mode Exit fullscreen mode Each of these is a single Lua script in Redis. Redis runs one script at a time, so for a given tenant the checks and updates are serialized, no matter how many gateway replicas send them. That gives one invariant, checked inside every admit and every extension:

Settlement swaps a reservation for the real count, so the only way past the limit is a token that was generated without a reservation. The whole design comes down to deciding what happens to those tokens.

What should you reserve? There are two obvious answers, and both are wrong.

Reserve max_tokens. It's safe, because the model can't exceed it. But most completions use a fraction of their cap. With max_tokens: 1024 and a median response of about 150 tokens, every in-flight request holds about 6× what it will use. Near the end of a budget, the remaining tokens are all held and none are used, so new requests get rejected even though the budget isn't spent. In the benchmark, about 4% of every budget was never usable, and at 128 streams each tenant's clients saw roughly 2,900 "come back later" rejections.

Reserve a small fixed number. It's efficient, until a long response blows straight past it.

Hardcap reserves the rolling p95 of real output lengths for that route (the last 1,000 completed responses) and treats the 5% tail as an extension problem. As a stream gets within 32 tokens of its reservation, the gateway asks Redis for 64 more. If the budget can't cover them, there are two policies:

Truncated responses are kept out of the p95 sample. Otherwise the estimate would drift down, cause more truncation, drift down further, and spiral.

That 0.03% isn't a rounding error, and I didn't want to hide it. It is the tokens a provider generates between the gateway closing the connection and the provider noticing: a chunk or two sitting in a TCP buffer. The benchmark only catches them because ground truth is what the mock upstream says it generated, not what the gateway thinks it forwarded. A real provider bills the former.

Reservations created a problem I didn't expect. A tenant with 900 tokens left, all held by in-flight requests, is in a different position from a tenant whose 900 tokens are spent. The first will have room again in a second; the second has to wait for the window to reset. Return the same error for both and clients either give up too early or retry a spent budget forever.

So the admission script checks one more thing when it rejects: would this fit if nothing were in flight? If yes, it answers budget_busy with Retry-After: 1; if no, budget_exceeded with the time until the window resets.

Reservations are only safe if they can't leak. Each one is stored with an expiry. A live stream refreshes it, and the next admission for that tenant reclaims any that expired. If a replica crashes, its tenants get their held budget back after the TTL (two minutes by default). The benchmark checks this too: across every run, 0 reserved tokens were left behind once traffic stopped.

What it can't recover is the usage of the streams that died with it. The provider billed those tokens and the gateway never learned the count. That's in the limitations section of the README, because it's a real gap.

Two Redis round trips per request, plus a small extension call every 64 tokens. Through the full path (auth, rate limit, reservation, settlement, Prometheus metrics), measured open-loop on one laptop:

For calls that take hundreds of milliseconds to seconds, that's noise. It isn't free at the top end: five Python workers saturate somewhere between 2,500 and 5,000 req/s on my machine, and the knee moved between runs. Profiling showed about 65% of the handler's CPU going to the Redis client's pure-Python protocol code. Switching to its C parser ( hiredis ) moved one worker's limit from about 1,300 to over 1,600 req/s. If raw throughput were the goal, the hot path would be Go. It wasn't: the goal was a budget that holds.

A budget gateway also has to route around a failing provider. With a primary → secondary route at 200 streaming req/s:

The benchmarks use a mock upstream, because measuring overshoot exactly needs ground truth. Before release I also ran the same paths against a real API (Groq, gpt-oss-20b ). Ledger charges matched the provider's reported usage to the token, on unary and streaming calls. The truncation check failed: a stream on a 60-token output budget ran to 400 billed tokens.

The model was a reasoning model. Reasoning models stream their thinking in a separate delta.reasoning field before any delta.content appears, and Hardcap counted only content. For the whole reasoning phase, the gateway saw zero tokens. It never asked to extend the reservation and never cut the stream, so the budget was right back to "count it afterwards". The mock upstream couldn't catch this: it only ever sent content.

The fix counts every delta the provider bills for: reasoning, reasoning_content, Anthropic thinking blocks and tool-call arguments. Rerun: truncated after 49 tokens. The lesson generalizes. Any part of the output the gateway doesn't count is output with no budget.

Publish the uncomfortable numbers. Where it saturates, what a crash loses, which estimate the benchmark doesn't test (input tokens, counted with a heuristic that the mock upstream shares). A benchmark that only shows wins doesn't tell anyone much.

Test against a real provider, even when the mock is the better benchmark. The mock gave exact ground truth, and it also shared my assumptions about what a stream looks like.

Everything here reproduces with one command per benchmark. The raw JSON, the machine specs and the parameters are committed next to the charts: github.com/saikirankalidindi/hardcap.