Claude's Context Window in Practice: Caching, Chunking, and Cost

A bigger context window raises a design question that didn't exist a couple of years ago: if the model can read 100 pages at once, do you still need retrieval?

Sometimes you don't. Often you do. This post walks through three ways to give Claude your documents (stuff, cache, retrieve), with minimal Python for each and a rough way to compare cost. The code is a sketch written from the Anthropic documentation, not a tested library, so check parameter names and pricing against the current docs before you copy anything.

Before choosing an approach, find out how big your material really is. The SDK can count tokens for a request without running it:

import anthropic client = anthropic. Anthropic () count = client. messages. count_tokens ( model = " claude-sonnet-5-5 ", system = " You answer questions about our policy handbook. ", messages = [{ " role ": " user ", " content ": handbook_text + " \n\n Question:... " }], ) print ( count. input_tokens ) Enter fullscreen mode Exit fullscreen mode If the number is a small fraction of the model's context window, long context is on the table. If it's close to the limit, or your corpus is many times larger, retrieval is the only real option.

For a stable document set that many requests share, such as a handbook, a product spec, or a contract template, you can send the whole thing each time and mark it for caching. Repeated requests then reuse the processed prefix instead of paying full price for it again.

response = client. messages. create ( model = " claude-sonnet-5-5 ", max_tokens = 500, system = [ { " type ": " text ", " text ": " You answer questions about our policy handbook. " }, { " type ": " text ", " text ": handbook_text, " cache_control ": { " type ": " ephemeral " }, }, ], messages = [{ " role ": " user ", " content ": question }], ) usage = response. usage print ( usage. cache_creation_input_tokens, usage. cache_read_input_tokens ) Enter fullscreen mode Exit fullscreen mode Three habits make caching work:

When the corpus is large, changes often, or has to respect per-user permissions, retrieval is the better fit. You split documents into chunks, find the few that matter for each question, and send only those.

A simple chunker that keeps paragraphs together and adds a small overlap:

def chunk ( text, max_chars = 3000, overlap = 300 ): paragraphs = text. split ( " \n\n " ) chunks, current = [], "" for p in paragraphs: if len ( current ) + len ( p ) > max_chars and current: chunks. append ( current ) current = current [ - overlap:] current += p + " \n\n " if current. strip (): chunks. append ( current ) return chunks Enter fullscreen mode Exit fullscreen mode Then retrieval and prompt assembly. embed here stands for whatever embedding model you use, since that's a separate choice:

import numpy as np def top_chunks ( question, chunks, chunk_vectors, k = 5 ): q = np. array ( embed ( question )) scores = [ float ( np. dot ( q, v ) / ( np. linalg. norm ( q ) * np. linalg. norm ( v ))) for v in chunk_vectors ] best = sorted ( range ( len ( chunks )), key = lambda i: scores [ i ], reverse = True )[: k ] return [ chunks [ i ] for i in best ] def answer ( question, chunks, chunk_vectors ): context = " \n\n --- \n\n ". join ( top_chunks ( question, chunks, chunk_vectors )) return client. messages. create ( model = " claude-sonnet-5-5 ", max_tokens = 500, system = " Answer only from the provided excerpts. If they don ' t contain the answer, say so. ", messages = [{ " role ": " user ", " content ": f " Excerpts: \n { context } \n\n Question: { question } " }], ) Enter fullscreen mode Exit fullscreen mode Retrieval has its own failure modes: the right chunk may not rank in the top five, and answers that span several sections can get cut.

Budget time to evaluate it, not just build it.

Prices change, so I won't print any. This helper takes your own numbers. The write and read multipliers describe how cached input is billed relative to normal input, so take them from the current pricing page:

def monthly_input_cost ( doc_tokens, calls_per_month, price_per_mtok, cache_write_mult = None, cache_read_mult = None, ): full = doc_tokens * calls_per_month / 1_000_000 * price_per_mtok if cache_write_mult is None: return full # no caching # simplified: one cache write, the rest reads (ignores expiry gaps) write = doc_tokens / 1_000_000 * price_per_mtok * cache_write_mult reads = doc_tokens * ( calls_per_month - 1 ) / 1_000_000 * price_per_mtok * cache_read_mult return write + reads Enter fullscreen mode Exit fullscreen mode Run it for your own document size and question volume, then compare against the retrieval path, where each call sends only a few thousand tokens. Two cautions. Caching entries expire, so a low-traffic app will rewrite the cache more often than this simplified function assumes.

And retrieval has costs beyond tokens: embeddings, a vector store, and the engineering time to keep the index fresh.

Whichever you pick, test with your own questions. Models can miss details buried deep inside very long inputs, and a small test set of real questions with known answers will tell you more than any benchmark.

Whatever the approach, record these per call: input tokens, cached tokens read and written, output tokens, latency, and which chunks or documents were included. When cost or quality moves, those fields tell you why.

I've gone deeper on window sizes and cost patterns in Claude context window limits and cost-smart patterns.

How big is the biggest document set you've tried to put in a single prompt, and did it hold up? I'd like to hear what worked.

I'm Darius Tran, an AI Implementation Specialist at AI Hive, working on getting AI agents into production for enterprise teams.