ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model

Models Datasets Spaces Buckets new Docs Enterprise Pricing Website Tasks HuggingChat Collections Languages Organizations Community Blog Posts Daily Papers Hardware Learn Discord Forum GitHub Solutions Team & Enterprise Hugging Face PRO Enterprise Support Inference Providers Inference Endpoints Storage Buckets Log In Sign Up Back to Articles a]:hidden"> ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model Enterprise Article Published October 20, 2026 Upvote 2 Ashwath Vaithinathan Aravindan ashwath-vaithina Follow ibm-research Vatche Isahagian Vatche Follow ibm-research Vinod Muthusamy vinodmut Follow ibm-research Jayaram Radhakrishnan jayaramkr Follow ibm-research Evelyn Duesterwald evduester Follow ibm-research Gaodan Fang gaodan-fang Follow ibm-research Punleuk Oum illeatmyhat Follow ibm-research G Thomas gsthomasx Follow ibm-research Michał Ulewicz Michal Follow ibm-research Merve Unuvar mrvnvr Follow ibm-research Ayhan Sebin ayhansebin Follow ibm-research:last-child]:mb-0"> Learned memory takes a basic ReAct agent with GPT-4.1 to the top τ²-bench tests policy-bound customer service Guidelines improve capability and reliability The best amount of guidance depends on the domain Better performance costs more tokens, not more turns Learned guidelines also improve agents using frontier models ALTK-Evolve tops the leaderboard comparison Agents can improve without weight updates Links: ALTK-Evolve learns and applies a playbook using GPT-4.1, outperforming the retail entries on τ²-bench, a customer service agent benchmark—without updating model weights.

Learned memory takes a basic ReAct agent with GPT-4.1 to the top Using GPT-4.1, an older, non-reasoning model, a basic customer-service agent reaches 67.5% Pass^4 on τ²-bench retail — meaning it solves the task on all four independent attempts for 67.5% of tasks. It scores above every retail entry in our October 12, 2026 leaderboard snapshot. It runs on the benchmark's stock ReAct scaffold, with learned guidelines appended to its system prompt.

ALTK-Evolve builds that playbook using GPT-4.1. It analyzes the agent's training-task attempts and uses the model to distill successes and failures into short behavioral rules: how to authenticate a customer, confirm a transaction, or handle a return. Those guidelines are then added to the agent's prompt for subsequent conversations. The same model powers both guideline extraction and the agent; no weight updates or human annotation are needed.

Our previous work demonstrated this approach on coding agents. Here, we test it on customer service, where success requires following domain policies, understanding a customer's request, and executing the right tool actions.

τ²-bench tests policy-bound customer service τ²-bench (from Sierra) measures how well an agent handles customer-service conversations under real constraints. Each task pairs the agent with an LLM-simulated customer who has a goal in mind, a set of domain tools (look up an order, issue a refund, rebook a flight), and a written policy document the agent must follow. The agent has to draw out the right details, stay within policy, and take the correct — often irreversible — actions; a task counts as solved only when the final database state matches the expected outcome, a strict binary reward. We use its two customer-service domains, retail and airline.

We run τ²-bench's own implementation of a basic ReAct agent with a two-phase THINK → ACT loop, using gpt-4.1 as the agent model and gpt-5.5 as the customer-simulator model; 4 trials per task. The only change we make is a block of learned guidelines appended to the system prompt — everything else (policy, tools, scaffold) is stock τ²-bench. Concretely, the learned guidelines are tacked on after the domain policy as a plain markdown list:

You are a customer service agent that helps users according to the policy below. Always follow the policy strictly. Use the provided tools when needed. <policy> # Retail agent policy As a retail agent, you can help users: - cancel or modify pending orders - return or exchange delivered orders - modify their default user address … </policy> ## Guidelines learned from past task attempts (follow these carefully): - Authenticate the user before taking any order-related action.

If email is unavailable, verify identity via full name and zip code using the find_user_id_by_name_zip API. - Always confirm all intended changes (modify, cancel, return, exchange) with a clear, structured summary, and require explicit user approval before executing any database-updating action. - When processing a return or exchange, communicate the refund with an itemized breakdown and clarify refund timing — don't wait for the user to ask. - Take one action per step: never combine a tool call with a user-facing message in the same turn, and wait for the tool's result before replying. - Before looking a user up, parse and clean their name — e.g. split a combined handle like `daiki_johnson_9523` into first/last name — and after a couple of failed lookups, state the exact format you need. … We extract guidelines from the agent's trajectories on τ²-bench train tasks using gpt-4.1, the same model that powers the agent, then apply them unchanged on the held-out test set.

Guidelines improve capability and reliability In a prior post, we talked about how different models / tasks require different dosage of guidance. Inline with the idea, we also evaluate different amounts of guidelines provided to the model.

Each guideline carries a support count — the number of distinct training tasks it was extracted from, so higher support means a rule that generalized across more tasks rather than a one-off. We compare three configurations: baseline (no guidelines), sup3 (keep only guidelines with support ≥ 3 — a small, high-confidence core), and lossless (keep every guideline retained after consolidation, i.e. support ≥ 1).

Table 1. Guideline configurations and retained guideline counts by domain.

The metrics we track. Every task is attempted 4 times, and we read those trials two ways:

Figure 1. Mean Accuracy and Pass^4 on held-out retail and airline tasks for agents using gpt-4.1 across four trials per task.

Agent model: gpt-4.1 (user simulator: gpt-5.5 ).

Table 2. Held-out agent performance using gpt-4.1 across four trials per task. Parentheses show percentage-point changes from baseline.

Mean Accuracy climbs in both domains — retail +4.4 pts (78.1 → 82.5) and airline +7.5 pts (57.5 → 65.0). Reliability moves further: the strict Pass^4 rises +10 pts in retail (57.5 → 67.5) and +10 pts in airline (35.0 → 45.0). And the gains come from better-informed calls, not longer conversations — we look at conversation length directly in Better performance costs more tokens, not more turns below.

The best amount of guidance depends on the domain The two domains disagree about how much guidance pays off. In retail, more is better on every metric — the small sup3 core barely beats baseline on accuracy, and you need the full lossless set for the real jump. In airline the picture flips at the reliability end: lossless leads on accuracy (65% vs 60%), but the compact sup3 core (14 guidelines) is the more reliable one, taking Pass^4 to 45% against lossless's 40%. Echoing the earlier finding that delivery is a dial, not a constant, support count is a knob worth tuning per domain, not a value to hard-code.

Better performance costs more tokens, not more turns Guidelines aren't free, but the cost lands where you'd expect — in prompt input tokens, not in a longer, more meandering conversation:

Figure 2. Total LLM cost per conversation by domain and guideline configuration on the full base split.

Table 3. Total LLM cost per conversation (agent and user simulator) on the full base split across four trials. Lower is better.

Does the agent get there by talking longer? To answer that cleanly we compare like for like: the task runs solved under both baseline and the guideline variant — identical scenario and seed, a true "same task, done right" set — and count assistant turns on each side.

Table 4. Assistant turns per conversation for matched runs solved under both settings on the test split, pooling four trials.

The agent isn't talking longer — on the shared set, guidelines make conversations slightly shorter: lossless trims ~0.3–0.4 turns in both domains (retail 10.59 → 10.16, airline 9.15 → 8.82), and no variant lengthens them. That matters beyond cost: each turn is a sequential LLM round-trip, so fewer turns means lower end-to-end latency — a faster time-to-completion for the customer, not just a cheaper one. The only real price is on the input side — the larger system prompt riding on every LLM call: retail's 292-guideline prompt pushes cost from ~$0.084 to ~$0.139 per conversation, while airline's smaller playbook is cheaper. And in production even that is largely reclaimable with prompt caching, since the guideline block is identical across every call.

So the trade is a favorable one: for a slightly higher input cost (much of it cached away in practice) the agent lands higher accuracy, higher reliability, and lower time-to-completion at once — not the usual quality-for-latency tradeoff, but all three moving the right way together.

Learned guidelines also improve agents using frontier models So far, we have used an older, non-reasoning model — gpt-4.1. A fair objection is that learned guidelines might have less to add to an agent using a frontier model: maybe they only patch the gaps of a weaker model. So we tested it directly — we reran airline using gpt-5.5 for both the agent and guideline extraction, on the same τ²-bench ReAct scaffold, with guidelines re-mined from the agent's own runs. One detail changes with the model: none of the guidelines extracted in this setup reached support ≥ 3 (they top out at support 2), so the high-support core here is sup2 rather than sup3.

Table 5. Held-out airline performance for the gpt-5.5 agent across four trials per task.

The learned guidelines improve this setup too. On the same held-out test tasks, appending them to the prompt lifts accuracy +10 pts (72.5 → 82.5) and the strict Pass^4 by +15 pts (55.0 → 70.0) — a larger reliability gain than we observed with gpt-4.1 on these very tasks (+10 pts), and from a much higher floor. Learning guidelines around the model can improve an agent even when it starts from a stronger base. And where the baseline's reliability swung between reruns, the lossless configuration reproduced exactly — the gain it buys is dependable, not a lucky draw.

ALTK-Evolve tops the leaderboard comparison Submissions are scored on the full base split (train ∪ test), as τ²-bench requires, and reported at Pass^4. Because that covers the full task set rather than the held-out test split, these figures differ from the test-task numbers reported above. This is where the practical upshot shows up — prompt-level memory carries agents using each base model into, or past, current frontier company:

Figure 3. Pass^4 comparison between ALTK-Evolve runs and the τ²-bench leaderboard snapshot from October 12, 2026.

Same message in both domains: learning guidelines around the model works across different base models — bringing an agent using an older, non-reasoning model into frontier company on retail, and taking an agent using a frontier model to the top on airline.

Agents can improve without weight updates Fine-tuning a model on a benchmark is expensive, brittle, and doesn't transfer to the next domain. Here, ALTK-Evolve improves a stock agent on a stock model in both domains — and matches or beats a fine-tuned system on retail — using only prompt-level guidelines extracted from the agent's own runs, automatically, without human annotation, and without any test-set leakage.

Links: Prior posts Thinking of ACE? We Can Do It with Fewer Tokens How much memory does your agent actually need? ALTK-Evolve library — https://github.com/AgentToolkit/altk-evolve τ²-bench — https://github.com/sierra-research/tau2-bench Retail leaderboard submission — τ²-bench PR #542 Airline leaderboard submission - τ²-bench PR #551 More from this author

122 September 15, 2026 IBM releases SOTA Granite Time Series PatchTST-FM-r2 model with commercial-friendly license 60 September 9, 2026 Community Edit Preview Upload images, audio, and videos by dragging in the text input, pasting, or clicking here. Tap or paste here to upload images Comment · Sign up or log in to comment