IBM's ALTK-Evolve tops τ -bench retail with GPT-4.1 memory
IBM Research has published results for ALTK-Evolve in a Hugging Face blog post , dated October 20, 2026. The method extracts short behavioural rules from an agent's earlier attempts and appends them to the system prompt. It needs no weight updates and no human annotation. The authors report that a
Posted on Oct 21 • Originally published at inite.ai
IBM Research has published results for ALTK-Evolve in a Hugging Face blog post, dated October 20, 2026. The method extracts short behavioural rules from an agent's earlier attempts and appends them to the system prompt. It needs no weight updates and no human annotation. The authors report that a plain ReAct agent on GPT-4.1 reached 67.5% Pass^4 on the retail track of τ²-bench, above every retail entry in their October 12, 2026 leaderboard snapshot.
τ²-bench from Sierra pairs an agent with a simulated customer, a set of tools and a written policy. A task counts as solved only if the final database state matches the expected result. Pass^4 means the agent solved a task on all four attempts, so it measures reliability rather than average success. In this setup GPT-4.1 powered both the agent and the rule extraction, and GPT-5.5 played the customer.
The learned rules are plain text. Examples include verifying identity before any order action, getting explicit approval before changing the database, and giving an itemised refund breakdown without being asked.
On held-out retail tasks (40), Pass^4 rose from 57.5% to 67.5% with the full set of 292 rules. On airline tasks (20), the smaller 14-rule core lifted Pass^4 from 35.0% to 45.0%, while the full 101-rule set reached 40.0%. The authors conclude that the right amount of guidance depends on the domain.
On the full base split, the retail result is ahead of RAFT-30B-A3B at 61.4%, a model fine-tuned on synthetic retail data. For airline, GPT-4.1 with the compact rules reached 44%, eighth on the board. Swapping in GPT-5.5 gave 82.0%, against 66–72% for the frontier entries listed.
The rules add input tokens to every call. Retail cost per conversation went from $0.084 to $0.139 with the full set, a 65% increase. The 31-rule core cost $0.095. Airline went from $0.081 to $0.097 with the core and $0.107 with the full set. On tasks solved both with and without the rules, conversations were slightly shorter, not longer. The authors add that the identical rule block should be largely recoverable through prompt caching in production, though they do not measure this.
The figures are self-reported by IBM. The held-out sets are small, at 40 retail and 20 airline tasks, and the customer is simulated. The post links the ALTK-Evolve library and the two leaderboard pull requests, which readers can inspect.