Cut AI agent token costs with decision models for tool selection

Posted on Oct 13 • Originally published at builder.aws.com

Give an AI agent a real toolset (25, 40, more) and every call pays for it. The agent reads every tool description to use one, so input tokens grow with each tool you add. And with many similar tools in view ( search, search_hotels, search_flights...), the model picks the wrong one more often.

Two costs on every turn: tokens you did not need to spend, and picks you did not want. Giving the agent fewer tools is not the fix, because different turns need different tools. You want the full toolset available but only the relevant few in front of the model each turn.

In an earlier post I solved this with semantic tool selection over a vector store: embed every tool description into an index (FAISS), embed the query, and keep the nearest few. It works, and that walkthrough covers it in depth. But it has a cost of its own: you maintain an embedding index for your tools, every query pays an embedding call, and the index has to stay in sync as tools change.

A new class of model changes this. A decision model reads the request and the list of tools and returns a calibrated choice directly, with no embeddings and no index. It generates no text, so there is nothing to parse, and the choice comes with a confidence score.

Three have arrived recently: Strands Decider, an open-source 2B model that runs locally; Jev (TypeSafe), a hosted one; and the OpenAI Decisions API ( gpt-6-luna, public beta since Oct 2026), also hosted. All three answer the same bounded question, which tool fits this request?, and all three skip the vector store entirely. You get the token and accuracy win of tool selection without building or maintaining an index.

Strands lets you plug any of these in the same way, through a hook that runs before the model and swaps the agent's tools for the turn. The selector inside the hook can be the vector store or a decision model, and the same idea extends to model routing, choosing which model serves the request.

class ToolFilterHook ( HookProvider ): def register_hooks ( self, registry ): registry. add_callback ( BeforeModelCallEvent, self. filter ) async def filter ( self, event ): tools = await self. select ( query ) # vector, decider, Jev, or OpenAI Decisions swap ( event. agent, tools ) # the model now sees only these agent = Agent ( tools = ALL_TOOLS, hooks = [ ToolFilterHook ( select )]) Enter fullscreen mode Exit fullscreen mode The agent is built once with every tool; the hook trims the registry per request. This is where Strands earns its place: it swaps tools on the live agent at runtime, so the conversation survives. Frameworks that force you to rebuild the agent to change its tools lose that history. Here you keep the memory, and you switch strategy by swapping select in one line.

The swap itself is a few lines against the agent's tool registry. Clear it and register the chosen tools; the running agent and its message history stay intact:

def swap ( agent, tools ): reg = agent. tool_registry reg. registry. clear () # drop the previous turn's tools reg. dynamic_tools. clear () for t in tools: # register only the ones picked this turn reg. register_tool ( t ) # agent.messages is untouched — the conversation is preserved Enter fullscreen mode Exit fullscreen mode References: Strands hooks, hook events, and model routing.

Keeping the live conversation is memory within a session. For memory that lasts across sessions, Strands has a dedicated memory library: pluggable memory stores on a unified storage interface. You attach a store, a local file store for prototyping or Amazon Bedrock Knowledge Bases with semantic search for production (or your own backend), and the agent gets recall, automatic injection into the prompt, and extraction of new memories. So the hook keeps this turn's context, and the memory store carries durable knowledge between runs.

The hook decides which tools. Strands model routing decides which model. Give a ModelRouter a few candidates and a strategy; the strategy picks one per request. A decision model fits this role well: picking the model is a choice among a few unordered options — exactly what these models are built for — and it votes with a calibrated pick instead of a chat call.

All three decision models plug into the same RoutingStrategy interface: Jev (hosted), Strands Decider (local 2B), and the OpenAI Decisions API ( gpt-6-luna, a choice question over the candidate models). The example below uses Jev — the one wired in the demo ( model_routing_with_jev.py ); swapping in Strands Decider or OpenAI Decisions is a one-class change, so you pick on hosted-vs-local and which provider you are already on. Routing to a cheaper model for the easy requests is where this pays off — OpenAI reports a run that cut cost 54% by classifying each job before picking the model.

router = ModelRouter ( models = [ small, large ], strategy = JevRoutingStrategy ()) agent = Agent ( model = router ) Enter fullscreen mode Exit fullscreen mode Same shape as the tool hook: a cheap decision in front, the expensive model only when it is needed.

Same 12 queries, same ~25-tool pool, one demo run (AWS Bedrock Claude Haiku as the agent):

Both selectors reach the same accuracy and cut chat tokens by about two-thirds. But each adds a call before the model: vector embeds the query (and pays embedding tokens), the decision model runs one choice, so here they are slower end to end, not faster. At 25 tools, sending everything is the quickest.

That ordering flips as the toolset grows. The model's prefill (reading the prompt before it answers) scales worse than linearly with input length, so a bigger pile of tool descriptions makes every call slower, and an agent makes many calls. Past some tool count the baseline's growing prefill overtakes the fixed cost of one selection step, and selecting first becomes the faster path, not just the cheaper one.

So the real takeaway is not a fixed latency winner; it is that you spend a small, constant selection cost to keep the prompt, the token bill, and the prefill small, and that pays off more as you add tools.

Follow the request through the pipeline and you can see exactly where each approach spends time:

So the extra latency lives in the selection step, before the model runs. The model call itself is cheaper afterwards (a short tool list), but not cheap enough at this scale to beat skipping selection entirely.

They are not exclusive. The hook makes them interchangeable in one line.

Measuring only the pick — which tool each selector returns — over 20 labelled queries against a ~25-tool pool (warm, single run on 2026-10-07):

Small set, one run — treat it as an early signal, not a verdict. Both decision models matched the right tool on every query here; vector missed one. (This is the selection step in isolation; the end-to end table above measures a different 12-query run including the agent's answer.)

All of this runs in the open-source demo (Demo 02), why-agents-fail-sample-for-amazon-agentcore. It has chat.py (vector), chat_decider.py (local decision model), model_routing_with_jev.py (model routing), and a notebook that measures accuracy and tokens for each — including the hosted OpenAI Decisions API as a third selector.

Does trimming tools hurt accuracy? No. In the demo it raised accuracy, because fewer similar tools means less confusion. The risk is dropping the right tool; tune how many you keep.

Why a hook instead of rebuilding the agent? Rebuilding loses the conversation. The hook changes the tools on the live agent, so memory survives and you switch strategy in one line.

Does a decision model save tokens? It sends fewer tool descriptions to the agent and generates no text itself. It is a separate, cheap call, not a chat turn.

Is this tied to Strands? The idea (filter before the model) is general. Strands makes it first-class: hooks, the tool-registry swap, and model routing are built in.

What do I need to run the OpenAI Decisions selector? An OPENAI_API_KEY and openai>=3.26.0 (or just POST to /v1/decisions ). It is in public beta on gpt-6-luna, returns a typed choice with confidence, and bills input tokens only — no output or cache charges.

🇻🇪 Dev.to Linkedin GitHub Twitter Instagram Youtube