How a Bank Serving 100 Million Users Tests Agentic AI in Production

Signup Write New Story How a Bank Serving 100 Million Users Tests Agentic AI in Production by sarahevans by sarahevans | @ sarahevans CEO at Evans Global Communications Strategist & Technology Builder 🔮 Partner Zen Media | Building the human layer between brands and AI

Communications Strategist & Technology Builder 🔮 Partner Zen Media | Building the human layer between brands and AI

TL;DR: A KDD 2026 paper follows five customer support AI agents through testing and deployment. The team checks automated graders against expert judgments, starts new versions at 1% to 5% exposure, and tracks whether satisfaction gains come with more self-service or more work for human agents.

An automated grader can give an AI agent a passing score and still be wrong. In a June 2026 paper accepted to ACM’s KDD 2026 conference, engineers at a digital bank report that one grader initially performed worse than simply choosing the most common answer. Before relying on it to evaluate customer-facing agents, they had to improve the evaluator itself.

The paper describes how a bank serving more than 100 million users tests agentic AI in production across card delivery, debt management, credit limits, card management and product explanation. That customer base provides context for the deployment; it is not the test sample size.

For companies buying agentic AI, the case offers a view into how a team decides an agent is ready to handle more work. The evidence comes from customer support, rather than paid media or back-office workflows. Its useful contribution is the detail behind each readiness decision, from the tests a new version must pass to the point where a person takes over.

All five agents improved transactional Net Promoter Score (tNPS), collected after a support interaction, in online A/B tests. The comparison varied by agent, including earlier versions and a control group. Scores nevertheless remained below the bank’s expert human-agent benchmarks, which were measured separately within the support funnel.

Source: Gupta et al., KDD 2026, Table 3. Human scores were measured separately from the AI scores.

The self-service rate tracks how many requests the AI resolved without handing them to a person. Card delivery posted the largest gain, 29 percentage points, alongside a 37-point rise in satisfaction. The product explainer was the one exception, with a slight dip the authors attribute to temporary errors in calls to the language model.

The team separates an agent’s instructions and business procedures into modules that can be revised individually. Domain experts change those modules, then language models acting as automated graders evaluate candidate versions before live testing.

To establish the reference answers, three operations analysts labeled each evaluation sample and explained their decisions. Majority vote determined the final label. Those explanations then helped GEPA, a prompt-optimization method, refine the graders’ instructions.

On customer input verification, the initial grader scored 68.88% accuracy. Always choosing the most common label would have scored 77.78%. Optimization raised accuracy to 88.89%, measuring agreement with the expert labels rather than the customer-facing agent’s overall accuracy.

The authors also tested whether different grader models agreed after optimization. Those tests covered models from one vendor, leaving broader model coverage for future work.

Offline checks let the team test targeted fixes before exposing customers to them. Live metrics then reveal problems those checks missed, including changes in escalation patterns. New versions begin at 1% to 5% exposure while the team tracks self-service, tNPS, escalations and errors.

The bank’s agents transfer conversations to people when a request falls outside their routines, necessary data cannot be retrieved or confidence is low. According to the authors, the agents operate within existing policies and human oversight, without unrestricted or irreversible decisions about credit or debt management.

Some handoffs can be removed by giving an agent the tools to finish a defined task. In the card-delivery deployment, adding a card-reissue capability removed the need for a human transfer from the most common resolution path. Other handoffs remain part of the operating design.

Average handle time can rise even when automation is working. If AI resolves simpler requests, the human team is left with harder cases that take longer.

TELUS Digital, a customer experience transformation partner, makes that point in an Oct. 6 release on adjusting which interactions go to automation and which reach people. The release weighs handle time against the quality of the interactions that reach agents.

“AI should take the parts of a customer interaction that slow an agent down, so the agent can spend their judgment where it counts,” said Erin Walker, Global Vice President of CX AI, Business & Delivery at TELUS Digital.

The company’s framework for choosing a contact center AI partner also covers adoption and change management. Any routing change lands on the team receiving the transferred cases, and that team needs the customer’s context and the capacity to take them on.

A higher satisfaction score can come from sending more difficult cases to people and leaving simpler requests with the AI. Evaluating agentic AI in production therefore means examining why cases escalated. A missing tool and a decision reserved for human judgment call for different responses. The satisfaction score alone cannot show whether the agent improved or the workload moved.

If AI Isn’t Your Competitive Advantage, Stop Building It