Decide which prompt prefix to cache vs re-run to reduce LLM latency

Posted on Nov 1 • Originally published at chriseugenerodriguez.com

This post gives a practical decision flow for splitting an LLM prompt into a reusable prefix and a volatile suffix so you can use provider-side prompt caching safely and effectively. The included helper finds the longest contiguous leading stable prefix, produces a conservative token estimate for that prefix, and compares it to a configurable minimum cacheable token threshold. Use the server-inserted example to plug this helper into tests and CI.

Use this post as a checklist: identify ownership (what changes each request), measure prefix size before committing to cache, pick keys and TTLs to separate scope, prefer explicit-only when later stable blocks exist, and verify with real instrumentation. The server will append the canonical URL and insert the verified implementation example.

export function simpleTokenEstimate ( text ) { if (! text ) return 0; // Very small, deterministic estimator: split on whitespace and punctuation groups. // This is intentionally conservative and inexpensive; real apps should use a tokenizer // matching the actual model. return String ( text ). trim (). split ( / \s +| (?=[.,;:!?() \[\] {}"' ]) | (?< = [.,;:!?() \[\] {}"' ]) / ). filter ( Boolean ). length; } export function analyzePromptParts ( parts, { model = ' gpt-5.6 ', minCacheableTokens = 64 } = {}) { // parts: array of { role: 'system'|'developer'|'user'|'assistant', text: string, stable?: boolean } // Determine the longest leading contiguous stable prefix (from start) that can be cached. if (! Array. isArray ( parts )) throw new TypeError ( ' parts must be an array ' ); let prefixParts = []; let prefixTokenCount = 0; for ( const p of parts ) { if (! p || typeof p. text!== ' string ' ) throw new TypeError ( ' each part must have a text string ' ); const isStable = Boolean ( p. stable ); if (! isStable ) break; // only contiguous leading stable parts are cacheable as one prefix prefixParts. push ( p ); prefixTokenCount += simpleTokenEstimate ( p. text ); } const prefixText = prefixParts. map ( p => p. text ). join ( ' \n ' ); const isCacheable = prefixTokenCount >= minCacheableTokens; // Model-specific guidance: on gpt-5.6+, explicit-only is recommended when suffixes are volatile. const major = String ( model ). toLowerCase (); const prefersExplicit = major. includes ( ' gpt-5.6 ' ) || major. includes ( ' gpt-6 ' ) || major. includes ( ' gpt-5. ' ); // Suggest cache mode: if there are trailing stable parts later (non-leading), explicit is safer. const hasLaterStable = parts. slice ( prefixParts. length ). some ( p => p. stable ); const suggestedMode = (! isCacheable || hasLaterStable )? ' explicit ': ( prefersExplicit? ' implicit ': ' explicit ' ); const contiguousLeadingStableCount = prefixParts. length; return { prefixText, prefixTokenCount, isCacheable, minCacheableTokens, suggestedMode, // top-level convenience for tests and callers contiguousLeadingStableCount, reason: { contiguousLeadingStableCount, hasLaterStable, model, } }; } export function chooseCacheKey ( baseKey, { scope = ' global ', id = null } = {}) { // Simple helper to build a cache-accounting key. Do not include secrets here. if ( typeof baseKey!== ' string ' ) throw new TypeError ( ' baseKey must be a string ' ); const safeScope = String ( scope ). replace ( / [^ a-z0-9_:- ] /gi, ' _ ' ); if ( id == null ) return ` ${ safeScope }: ${ baseKey } `; return ` ${ safeScope }: ${ id }: ${ baseKey } `; } export default { simpleTokenEstimate, analyzePromptParts, chooseCacheKey }; Enter fullscreen mode Exit fullscreen mode Read the full article: https://chriseugenerodriguez.com/blog/prompt-prefix-caching-decision