Putting a semantic cache in front of an AI agent, from signup to dashboard

An AI agent asks the model the same things every time it runs the same task: the same plan step, the same "did that work?" check, the same summary of the same docs. We recorded the whole path of putting the Sulci semantic cache in front of one, from signup to dashboard, in six minutes:

This post covers the parts worth reading slowly: the three code changes, the agent settings, and the moment the cache chose not to answer.

The demo is a planner–reflector loop: plan a step, act, then ask the model "did that step succeed?" Its task is "deploy the web service to AWS", and every plan step and every check is a call to Claude. Run the same task tomorrow and it makes the same calls again.

Each prompt carries its own state: the task, the steps already done, and the last tool output.

You are an operations agent. Task: deploy the web service to AWS. Completed steps: configure AWS credentials.

Reply with the single next step to perform. Enter fullscreen mode Exit fullscreen mode Three code changes import anthropic import sulci from sulci import Cache client = anthropic. Anthropic () # reads ANTHROPIC_API_KEY MODEL = " claude-haiku-5-5 " # any Claude model works cache = Cache ( backend = " sulci " ) # 1. reads SULCI_API_KEY from the environment def ask_claude ( prompt ): msg = client. messages. create ( model = MODEL, max_tokens = 512, messages = [{ " role ": " user ", " content ": prompt }]) return sulci. with_usage ( msg. content [ 0 ]. text, msg. usage ) # 3. token counts for the dashboard def plan_step ( prompt ): result = cache. cached_call ( prompt, ask_claude, # 2. route the call through the cache standalone = True, threshold = 0.95 ) return result [ " response " ] Enter fullscreen mode Exit fullscreen mode Create a cache. backend="sulci" is the managed tier; the free tier needs no card.

Self-hosted backends (SQLite and others) work the same way; Tokens Saved on the dashboard is a managed-tier feature. Route the call through cached_call. On a hit you get the stored answer.

On a miss Claude answers and the cache keeps the answer. Return the answer with with_usage. It attaches the call's token counts, so the dashboard can show the tokens each hit saved.

Agent-safe settings standalone=True: the prompt carries everything needed to answer it, so its answer is stored even when other calls came before it. threshold=0.95: strict. Agent prompts often differ in one detail (step 2 versus step 3, exit code 0 versus 1), and a near-miss must go to the model, not get another step's answer. No session_id.

With a session, the lookup blends the query with the conversation's recent turns, which is what lets a vague follow-up like "how do I fix it?" search with its conversation's context. Agent prompts already carry their own state, so they're looked up on their own: earlier steps can't pull a prompt toward another step's answer. The moment that matters: a cache that knows when not to answer In run 3, one step's tool output is a failure:

You are an operations agent. Task: deploy the web service to AWS. Step just performed: configure AWS credentials.

Tool output: exit code 1; AccessDenied: not authorized to perform this action; request id req-b41e5c. Did the step succeed? Answer yes or no with a reason.

Enter fullscreen mode Exit fullscreen mode A naive cache would hand back run 1's "Yes, it succeeded." Here the failed prompt scored below the threshold against the cached success, so it missed, and Claude answered "No, the step did not succeed…". In our offline replay of the same prompts with the same embedder, it scores 0.927 against the cached success: under the 0.95 bar. At the hosted default of 0.65 it would have been served, which is why agent prompts get a strict threshold.

Catching repeats is the easy part; knowing when not to answer is the hard part.

That is one run of a demo app. What you save depends on how repetitive your traffic is.