Prompt Caching: The 90% Discount You're Missing

Posted on Oct 18 • Originally published at apicostlab.com

There is a 90% discount sitting in your API bill, and most developers are either not using it or using it in a way that doesn't work. Prompt caching — paying a tenth of the input price for tokens the provider has already seen — is the highest-leverage cost optimization available on modern LLM APIs. Here's how it actually works and how to get the savings.

The idea is simple. When you send a request, the provider processes your input tokens through the model — that's the expensive computation you're paying for. But if your next request starts with the exact same tokens — the same system prompt, the same repo context, the same few-shot examples — the provider doesn't need to recompute all of that. It kept the intermediate results cached. So it charges you a fraction of the price for those repeated tokens.

The key phrase is stable prefix. Caching works from the start of your prompt forward: the provider matches your prompt's beginning against what it has cached, and everything that matches gets the discounted rate. The moment the tokens diverge — your new user message, the latest tool result — matching stops and the rest is billed at full price.

This has one big architectural implication: put stable content first, variable content last. System instructions, repo context, few-shot examples, reference documents — all of that goes at the top, byte-identical every time. The thing that changes per request goes at the end. Get this ordering right and the cache does the rest.

On GPT-5.6 Sol, the numbers are published and stark:

That's the 90% discount. Now run it through the reference workload — 1,100 tasks/month, each with 12K fresh input, 48K cached input, and 6K output tokens:

Notice: the 48K cached tokens — four times the volume of fresh tokens — cost less than half as much. Without caching, those 48K tokens at full price would cost $0.1920 per task instead of $0.0192. Across 1,100 tasks a month, caching saves roughly $190/month on this workload alone. That's most of the input bill, gone.

One cost people miss: cache writes. The first time a prompt prefix is cached, the provider charges a premium — on GPT-5.6 Sol, cache writes cost 1.25x the input rate. So the first call with a new prefix costs slightly more than an uncached call. Every reuse after that is nearly free. For anything you call more than a handful of times, the math is overwhelmingly in your favor — but don't model the first call as discounted.

Caching fails more often from prompt structure than from provider issues. The rules:

1. Fixed order, every time. The stable prefix must be byte-identical across calls — not just semantically the same, but the exact same tokens in the exact same order. Build your prompts programmatically from fixed templates, not by string-concatenating in whatever order the code happens to run.

2. Nothing dynamic in the prefix. Timestamps, request IDs, user names, "today's date" — if any of it sits inside the cached region, every request has a different prefix and nothing ever hits the cache. Move all dynamic content after the stable block, or accept that the cache starts after it.

3. Longest stable content first. Order the stable blocks by how rarely they change: system prompt first (never changes), then reference documents (rarely change), then conversation summaries (change slowly), then the live request (changes every time). This maximizes the cached portion even when later blocks rotate.

4. Don't reshuffle tool definitions. In agentic loops, the tool schemas are often a large, stable block — perfect cache material. But if your framework reorders them or regenerates them per call, the cache breaks. Pin them.

5. Verify, don't assume. After restructuring, check the cached-token ratio in your usage dashboard. You're aiming for 80%+ of input tokens served from cache on repetitive workloads. If you're at 20%, something in the prefix is still varying — diff two consecutive prompts and find it.

What's actually achievable? It depends on the workload shape:

The pattern: the more your workload looks like "same setup, different question," the higher the hit rate. The more it looks like "different everything," the lower.

Biggest winners: agentic coding loops (huge stable context, repeated thousands of times), RAG systems with fixed document sets, chatbots with long system prompts, eval harnesses running the same prompts across many inputs. These are exactly the workloads where input volume is highest — which is why caching matters so much.

Modest benefit: mixed workloads with partial repetition. You'll get the system prompt cached but not much else. Still worth doing — it's nearly free to set up.

Skip it: truly one-off inference, bulk jobs over unique documents, anything where the input is never repeated. The cache-write premium means you'd pay more for zero benefit.

When the hit rate stays low after restructuring, the culprit is almost always something small and dynamic hiding in the prefix. The debugging method is unglamorous but effective: capture two consecutive prompts, diff them, and look at exactly which tokens differ. Common offenders:

Fix them one at a time and watch the dashboard after each change. The hit rate should climb in steps; when it plateaus, you've found everything that was varying.

One more subtlety: minimum cacheable lengths. Providers typically require the stable prefix to exceed a minimum token count before caching engages at all — short prompts may never qualify regardless of structure. If your prompts are small (a few hundred tokens), caching may simply not apply, and that's fine: there's little to save on small inputs anyway. Caching earns its keep on large, repeated context.

Prompt caching is the closest thing to free money in LLM APIs: a ~90% discount on your largest token category, available for the cost of ordering your prompts correctly. The developers paying full price for repeated context aren't being overcharged — they're just sending the same tokens in a slightly different order every time. Fix the order, check the dashboard, and watch the input line of your invoice collapse.