HotKV cut Claude token use by 70% in our Banking77 benchmark

Posted on Oct 19 • Originally published at hotkv.com

A semantic cache promises to cut LLM spend by answering repeated and similar questions from memory instead of calling the model. We measured that end to end: the same 3,000 customer support requests, sent to each of four Claude models twice, once straight to the model and once with HotKV in front, with Anthropic's own prompt caching left on in both runs. Here is exactly what we ran and what it found, including where it falls short.

In short. HotKV cut model calls, input plus output tokens and API cost by about 70% on each of the four models. Most of that comes from questions that repeat exactly. The semantic layer adds roughly 12 points on top. At the default 0.81 similarity threshold, cached answers were judged to fit the exact question 5.8 to 10.7 percentage points less often than fresh answers. On the one model we swept (Sonnet 5.5), a threshold of 0.88 or higher showed no such gap.

Updated 12 October 2026: this article now reports our larger run on four current Claude models. It replaces an earlier 1,500-request run on GPT and Claude Sonnet 5 that measured about 60% fewer tokens, which we no longer present.

We replayed customer support questions from the test split of Banking77 (PolyAI, CC BY 4.0), a public dataset of short banking questions labelled with 77 intents. It was introduced in Casanueva et al., 2020. The 3,000 requests were drawn with a fixed seed. Intent popularity follows a Zipf distribution with exponent 1.0, a synthetic skew in which a few intents come up far more often than the rest, and the question within an intent is picked uniformly, with repeats. All 77 intents appear, and 1,264 of the 3,000 requests are distinct questions (counted by replaying the seeded request generator). The bank and its support policy are fictional. We wrote a policy of about 2,200 tokens, sent as the system prompt on every request, that asks for two to four short sentences.

We ran claude-fable-5-1, claude-opus-5-5, claude-sonnet-5-5 and claude-haiku-5-5. Fable 5.1 and Opus 5.5 ran at their lowest effort setting, because their thinking cannot be switched off. Sonnet 5.5 ran with up-front thinking off, its lowest setting, and Haiku 5.5 with thinking disabled. Maximum output was 2,000 tokens for Fable and Opus and 400 for Sonnet and Haiku, and every answer finished normally rather than being cut off. Eight requests ran at a time in both modes, calls were not streamed, and each condition ran once.

Every token count comes from the usage block of the API response: uncached input, cache reads and writes, and output (thinking included). Embedding tokens are separate. Costs use Anthropic's list prices of 11 October 2026, include the embedding calls for semantic lookups, and exclude HotKV hosting and operations. They are not a quote for current pricing.

For each question, HotKV tries two lookups before the model is called:

Only a miss on both reaches the model, and its answer is written back with SET and SCACHE.SET. The similarity threshold for a semantic hit was fixed at 0.81 before the run. We chose it on a separate Banking77 train split as the lowest value, in steps of 0.01, whose semantic hits pointed at the same intent at least 98% of the time (98.2% there). In this run, on the untouched test split, 96.7% to 96.8% of semantic hits matched the intent of the question asked: 30 or 31 wrong-intent hits per model out of 926 to 936.

Across the four models, calls fell from 12,000 to 3,588, tokens by 70.1% and cost by 69.6%. About 70% of requests were answered from the cache on every model: roughly 39% by exact match and 31% by semantic match.

Calls, tokens and dollars fall together because HotKV removes whole calls. Read the token figure with care: 90% to 93% of the counted tokens are cache reads of the same 2,200-token policy prompt, which Anthropic bills at a small fraction of the normal input price. Output tokens are only 6% to 9% of the tokens but 73% to 92% of the baseline cost.

The median response time across the whole run fell from 2.7 seconds to 127 ms on Fable 5.1, from 2.9 seconds to 126 ms on Opus 5.5, from 1.8 seconds to 128 ms on Sonnet 5.5 and from 1.4 seconds to 125 ms on Haiku 5.5, which is 11 to 23 times faster. A repeated question came back in a median of about 0.05 ms, measured by a Python client over loopback on the same machine as HotKV, so it is a local figure rather than a network one. A paraphrased question, served from the semantic cache, took about 130 ms, almost all of it the embedding API call.

The whole-run median sits inside that semantic-hit band because exact hits (about 39% of requests) are too few to carry the median on their own, while exact and semantic hits together (70%) pass the halfway mark. The slow end moves less: 30% of requests still call the model. The 99th percentile was about 6.4 seconds with HotKV against 6.6 seconds without it on Fable 5.1, and about the same on Haiku 5.5. It was lower on Opus 5.5 (4.7 seconds against about 6) and on Sonnet 5.5 (2.7 seconds against 3.3). Calls were not streamed, so these are times to the complete answer, not to the first token.

The cache starts empty, so early requests still reach the model. The share of requests served from the cache was about 52% in the first thousand requests and about 81% in the last thousand.

Most of the saving is plain repetition. An exact-match cache alone would have needed at most 1,264 model calls out of 3,000, one for each distinct question, which is about 58% fewer. The semantic layer took that to about 70%, so it contributes roughly 12 points on this workload. It is also the part that carries the answer-quality risk, as the next section shows. This is derived from the request sequence and the logs, not a separate measured run.

A semantic hit means the same intent, not the same correct answer. To measure correctness we had a different Claude model judge every semantic hit. The judge saw the customer's question and two answers in random order: the cached answer HotKV served, and the fresh answer the same model gave to that exact question in the baseline run. It said whether each answer correctly answered that exact question. Opus 5.5 judged the answers of Fable 5.1, Sonnet 5.5 and Haiku 5.5, and Fable 5.1 judged Opus 5.5, so no model graded its own output.

At 0.81, a cached answer fit the exact question 5.8 to 10.7 percentage points less often than a fresh answer from the same model. Semantic hits are about 31% of requests, so across all responses, and counting exact hits as neutral, that is roughly 2 to 3 points (1.8 to 3.3). The judges differ in strictness, so compare cached with fresh only within one model.

We reran Sonnet 5.5 at four more thresholds, each from an empty cache, with the same judge. The baseline cost was $5.51.

From 0.88 upward cached answers did not show a deficit against fresh ones, while still cutting cost by 64% at 0.88 and 61% at 0.90. The evidence is thin. At 0.90 the 95% interval for the difference runs from about 4 points worse to 7 points better, the 0.95 row has only 50 hits, and each row judges a different subset of hits. This is one model, one judge and one run per threshold, so it is not proof of equivalent quality or a universal safe threshold.

The real risk is questions that look almost identical but need different answers, such as depositing versus withdrawing. We had Opus 5.5 write 119 pairs, each a real training-split question plus a version with one decisive detail changed, and counted how many changed questions would clear the threshold against a cache holding only the original. These pairs are built to be hard, were not reviewed by a person, and show a worst case rather than how often it happens in real traffic.

Raising the threshold reduces the risk but does not remove it. Even at 0.90, 25% of the deliberately close pairs would match, and fewer true paraphrases match too.

Extrapolating the measured cost per request to 1 million requests a day over a 30-day month, and assuming the same traffic pattern, gives these monthly figures:

This is arithmetic on the measured cost per request, not another measured run. It excludes hosting, and the capacity needed for that volume was not measured. At the 3,000 requests we ran, the saving is only $0.19 for Haiku 5.5 and $21.12 for Fable 5.1. The four models ran with short answers, and Fable 5.1 and Opus 5.5 at their lowest effort, so longer or reasoning-heavy answers would make each avoided call worth more.

This is not a general "HotKV saves you 70%" claim. It is what we measured on this workload, and the workload does the heavy lifting.

The per-model figures are also on the AI page.

The exact-match cache in this test uses plain GET and SET, which every HotKV plan includes. The semantic cache ( SCACHE.* ) comes with the Performance plan and above ( pricing ). Start on HotKV Cloud and point your existing traffic at it, or talk to sales if you want to work through the numbers for your own workload first.