When AI Calls Underperform Humans: A 12-Scenario Gate for Collections Voice Agents

Posted on Nov 4 • Originally published at vocalis.blog

If a voice agent gets a borrower to say “I’ll pay Friday,” the call has produced a promise, not a repayment. The right release question is whether the system can support a permitted, accurate, respectful resolution and whether that outcome holds up against a fair human-led comparison. A demo conversion rate, number of calls completed, or promise-to-pay rate cannot answer that on its own.

The Vocalis guide to collections controls and human handoffs covers the adjacent operating design: eligibility, suppression, policy-bound dialogue, and escalation. This piece focuses on a different decision: how to evaluate whether an AI-first collections workflow improves a real outcome without making a promise-to-pay metric the target. It offers a reusable 12-scenario acceptance suite and a field-evaluation contract. The suite is a proposed synthetic test, not a benchmark or customer result.

Do not make a collections voice agent autonomous because it sounds fluent or gets more people to say yes. Start with one narrow, counsel-approved portfolio and compare an AI-assisted path with the existing process. Measure completed, policy-permitted outcomes over a defined follow-up window; keep promises kept, disputes, wrong-party events, contact restrictions, complaints, and human handoff failures visible beside any recovery or efficiency metric. If a safety-critical event is mishandled, pause the workflow even if a financial metric looks favorable.

This is not a claim that human collection is always better, or that voice AI cannot be useful. It is a decision rule for a high-consequence workflow: the system should earn scope through evidence. In practice, the safest initial job is often a bounded reminder, a neutral route to account information, or a request for a human callback. The model should not decide whether a debt is valid, invent settlement terms, push a distressed person, or infer that a spoken promise is binding.

Search results for queries such as “how can voice AI improve collections?” often foreground reach, automation, and payment conversion. Product pages and conference demos show what a particular system can do in a selected interaction. They are useful for generating test questions, but they are not controlled evidence that another team will see the same results. The evidence gap is not another list of hypothetical benefits; it is a comparison protocol that makes the denominator, follow-up period, guardrails, and exclusions explicit.

One unusually relevant empirical paper is Choi, Huang, Yang, and Zhang’s NBER Working Paper 33669, issued in April 2025. The study uses data from a Chinese consumer-finance company covering 22 million delinquent-loan cases that entered collections from April 2021 through December 2022. The firm’s allocation rule sent nearly all newly delinquent debts below a 300-yuan principal threshold to AI callers, while larger debts were assigned to human callers; the researchers use a regression-discontinuity design around that threshold.

For some larger accounts, the company also randomly assigned half to receive AI calls for the first five days and then human calls, while the other half received human calls from the beginning. The authors report that AI callers were less effective than human callers in that setting and that promises made to AI were broken more often. The NBER paper is the primary research record; Yale’s summary of the study describes the sample and both designs.

There is a material reporting difference in the accessible summaries of the long-run repayment result. The NBER abstract says borrowers first contacted by AI repaid 1% less of the initial late payment one year later, while Yale’s summary says AI callers collected 5% less even a year later. Those sentences may be referring to different outcome definitions or denominators, but the accessible summaries do not establish a like-for-like reconciliation.

Do not average them or transfer either percentage to a new portfolio. For this article, the defensible takeaway is narrower: in one large company’s pre-generative-AI collection workflow, promises and later repayment did not move together, and human follow-up did not erase every difference. The denominator and table in the full working paper must govern any precise reproduction of its estimates.

The external-validity limit matters. The data came from one Chinese online lender and its specific accounts, scripts, technology, call rules, and borrower population. The assignment rule and randomized intervention offer stronger causal evidence than a vendor testimonial, but they do not test a 2026 speech model, a US third-party collector, a UK first-party reminder, or another institution’s portfolio. This is a reason to design a local evaluation, not a reason to ignore the study.

The CFPB’s 2025 Consumer Response annual report records consumer complaints about contact frequency, calls outside permitted hours, and collectors continuing to contact people after stop-contact requests. Those are complaint themes, not a representative estimate of how often every collector or AI system behaves this way. The CFPB explicitly says its published complaint database is not a statistical sample of consumer experiences. Use complaints as a source of failure scenarios, not as a market-wide defect rate. The annual report and the database’s methodology note make that limitation clear.

A current vendor demonstration can be informative in a different way. Veritus’s October 1, 2026 Finovate page describes a six-minute live servicing-call demo, including identity checks, account changes, a specialist-agent route, a secure keypad for card details, and a payment authorization read back to the customer. The same page claims “1 in 5” collections calls handled by its agents result in a payment, but it does not give the call count, portfolio mix, control group, measurement window, or definition of “result in a payment” on that page.

Treat the demo as a product illustration and the conversion statement as a vendor-reported claim until those details are available. Do not compare it directly with the NBER study. The demo and its transcript are useful precisely because they prompt questions a polished interaction cannot answer: what counts as authorization, how a payment is verified, and what happens when the next tool call fails?

Write down the unit of analysis, population, treatment, comparator, and outcome before enabling any live calls. A “call” is a poor unit if the same person receives a human call, an AI call, and an SMS for the same account. A “promise” is a weak outcome if one group records verbal intent more generously than another. A collection amount is incomplete if it is collected through a more frequent or more forceful contact pattern that creates more complaints or disputes.

An evaluation contract should contain the following fields:

The primary metric should be tied to the decision. If the business question is whether AI-first outreach increases timely resolution, a reasonable candidate is the percentage of eligible accounts assigned to each path with a verified payment or documented non-payment resolution by the chosen follow-up date. Define the denominator as assigned eligible accounts, not only answered calls: answer rates can differ by caller identity, time, number reputation, and reach strategy. Keep “contact reached,” “identity verified,” “promise made,” “promise kept,” “payment scheduled,” and “payment settled” as different events.

Use an intent-to-treat analysis when the operational assignment is randomized: keep accounts in their assigned group even if a call fails, a human takes over, or the agent never reaches the person. Otherwise the evaluation can accidentally exclude the very failures caused by the intervention. If randomization is not legally or operationally acceptable, document why and use the strongest feasible matched or phased comparison, with the residual confounders stated in plain language. Do not describe an observational comparison as causal.

Track a small number of outcomes that remain meaningful after the call: verified payment received, a documented dispute routed correctly, a customer-directed callback completed, or the issue closed through the agreed service path. A successful payment should be confirmed by the payment or ledger system, not inferred from an affirmative phrase in the transcript. A payment scheduled is a separate event from a payment settled. If a payment is reversed, returned, or not completed, the report should say so instead of keeping the earlier “success.”

The NBER evidence makes this distinction more than a reporting nicety. In the studied company, AI callers were reported to obtain fewer reliable promises, and later repayment did not map one-for-one to what callers said during a conversation. Your local outcomes can differ, but a verbal “yes” should never be the only outcome that determines whether the system appears to work.

Define hard-stop events separately from routine quality measures. Examples include sensitive debt details disclosed to a wrong party; a new outbound attempt after a valid restriction has reached the source of truth; a model inventing an amount or legal consequence; a dispute treated as an admission; or a payment taken without the required confirmation step. A weighted dashboard can hide a small number of serious events under a large number of successful calls. A gate should instead specify that certain events trigger immediate review or pause regardless of the aggregate recovery result.

Other measures can be monitored as rates or counts with clearly stated denominators: wrong-party outcomes per attempted contact, disputes correctly routed per detected dispute, stop requests propagated before the next queue dispatch, completed transfers per transfer request, and complaint events per eligible account. Do not use a single call-quality score as proof of compliance. The OECD describes privacy, safety, security, and human autonomy as AI risks and recommends risk assessment and governance; that is useful policy context, but it is not a debt-collection rule. The OECD’s current AI policy overview says the AI Principles were revised in 2024 and presents them as recommendations for policymakers and AI actors.

Once guardrails are stable, measure time to resolution, human minutes per resolved account, transfer wait, repeated calls, contact attempts per account, and cost per verified resolution. When comparing these values, preserve the campaign calendar, portfolio eligibility, language mix, and follow-up period. A shorter conversation is not an improvement if it defers the same unresolved issue to a human team. A reduction in human minutes can also be a transfer of work to complaint teams or customer-support queues; measure the receiving teams rather than assuming the work disappeared.

The vendor’s “1 in 5” payment statement illustrates the need for a denominator contract. Ask whether “calls” means attempts, answered conversations, verified borrowers, or all calls handled; whether a payment was merely scheduled or actually received; how many calls and accounts were included; whether multiple calls to one account count separately; which product and delinquency segments were included; what time window was used; and how the result compared with a control. Without those fields, the claim can guide a demo question but cannot justify a rollout decision.

The suite below is a reusable synthetic test asset derived from official rule concepts, complaint themes, the empirical result that spoken promises and later repayment can diverge, and failure cases visible in vendor demonstrations. It was designed on November 4, 2026. It contains twelve proposed scenarios; it is not a test run, not based on customer recordings, and not a measured failure rate. The scenarios should be adapted by counsel to the actual jurisdiction, debt type, collector role, contract, and contact policy. Exclude live consumer data from the test harness; use fictional names, accounts, balances, and numbers.

Score each scenario on separate gates: correct spoken response, correct policy decision, correct persistent state, correct downstream action, and sufficient audit evidence. Do not collapse those into a single “passed call” label. A transcript can be acceptable while the queue still sends a prohibited callback; a policy tool can return the right result while the caller hears a misleading explanation. Both layers need to pass.

Before a pilot, run the suite with the model disconnected from production systems. Then run integration tests against a staging copy of the account rules and queues. Inject delayed, duplicated, unavailable, and stale responses for the ledger and preference services. Review the call transcript only alongside the backend event log. For any failed case, save the exact synthetic input, model and prompt version, rules version, integration response, event IDs, expected state, actual state, reviewer decision, and fix. Re-run the same case after a prompt change, policy update, vendor release, CRM migration, or dialler change.

The suite does not claim to cover every law or every way a conversation can go wrong. It is deliberately small enough to run before a deployment and strict enough to expose common mismatches between words, state changes, and side effects. Expand it using actual incidents, regulator guidance, customer feedback, internal QA, and the portfolio’s approved procedures. Do not turn a synthetic test into a public accuracy or compliance percentage.

Create a signed configuration record for the pilot: portfolio, jurisdiction, collector role, permitted channels, call windows, approved disclosure and dialogue versions, call-frequency policy, human escalation routes, payment instrument, and account exclusions. Pin the agent build, model, system prompt, tools, source data fields, and policy table. A model update in the middle of a comparison is a new treatment, not an invisible maintenance detail.

The CFPB’s Regulation F applies to debt collectors as defined by the law and addresses communications, harassment, false or misleading representations, and unfair practices. The CFPB’s call-frequency FAQs explain a presumption involving more than seven calls in seven days or another call within seven days of a telephone conversation, subject to exceptions and context; they also warn that conduct may violate the broader rule even when it does not violate the frequency presumption. The rule’s scope and exceptions are fact-dependent. Read the CFPB’s Regulation F materials and the official call-frequency FAQs before translating any U.S. rule into a production counter. Do not treat “seven calls” as an entitlement or copy it into other jurisdictions.

For covered U.S. third-party collectors, the Fair Debt Collection Practices Act and Regulation F are not replaced by a model prompt. The FTC summarizes the federal baseline against deceptive, unfair, and abusive practices; the CFPB’s official rule text explains the detailed requirements and interpretations. The FDCPA definition and first-party/third-party boundary matter. Federal guidance is not a substitute for state rules, the TCPA analysis for artificial or prerecorded calls, creditor-specific obligations, or legal review of the actual campaign. The FTC’s business collection resources are a starting point for a legal review, not a deployment approval.

The strongest practical comparator is often the existing human process with the same eligibility rules and contact schedule, not an artificially weak process with fewer attempts or less follow-up. If the question is whether AI-first improves outcomes, compare assignment groups with a consistent measurement window. Consider a hybrid group only if it answers a real operational question, such as whether a human callback after a neutral AI intake improves resolution without increasing burden.

If random assignment is used, get legal, compliance, and operational approval first. Randomize at a unit that limits contamination: person, account, or another defined unit depending on the actual system and permitted contact policy. If a person has multiple accounts, randomizing each account independently may cause them to receive conflicting paths. Record assignment before contact, retain every assigned unit in the analysis, and prevent the operator from moving difficult accounts out of the AI group after seeing early outcomes. If the team instead uses matched comparisons, report the fields used to match and the differences that remain.

Pre-register the primary endpoint, secondary endpoints, guardrail definitions, exclusions, and follow-up window. Set the minimum meaningful difference and the required sample size using a statistician or qualified analyst with the actual baseline and outcome variance. There is no universal sample size for every portfolio. Do not choose a sample because it is convenient and then call a non-significant result proof that the systems are equivalent.

The outcome data model should distinguish a contact attempt, a live answer, an identity result, an interaction path, a promise, a payment schedule, a settled payment, a dispute, a stop request, a handoff, a complaint, and a follow-up. Store timestamps, portfolio, approved language, treatment assignment, rule version, and minimum event identifiers needed for reconciliation. Avoid adding sensitive free text simply because it is available; define access, retention, redaction, and deletion schedules before collection.

The CFPB annual report gives one useful source of consumer-reported question themes: people describe contact occurring too often or at inconvenient times, and contact continuing after they requested it stop. Because complaints are filtered and non-representative, use these themes to seed scenarios and monitoring prompts, not to estimate the probability of a failure. The CFPB database methodology explicitly warns against interpreting complaint counts as a statistical sample of all consumers.

At a weekly review, answer a fixed set of questions. Did stop requests reach every queue before another contact? How many promise events became verified payments during the agreed follow-up window? Did disputes trigger the approved route? How many people requested a person, how many reached one, and how many abandoned the transfer? Did wrong-party events or recognition failures cluster by language, accent, phone quality, or account-data source? Did human teams receive a new pile of unresolved cases? A report should show counts and denominators side by side, with small cells protected and uncertainty visible.

Separate outcome metrics from process metrics. “Payment received per assigned eligible account” is an outcome; “number of calls made” is an activity. “Stop request persisted across the system within the approved service level” is a control test; “the model said it would stop” is only dialogue. “Customer was transferred to a human” is not the same as “the human answered and resolved the request.” Naming the event precisely makes it harder for a dashboard to turn an intermediate step into a headline result.

Before launch, establish an explicit gate with three classes:

The thresholds must be chosen locally, before results are visible, and signed by people accountable for compliance, servicing, and measurement. This article does not invent a universal permitted complaint rate, a universal non-inferiority margin, or a universal required sample. A regulator or legal adviser may require a stricter outcome than an operational pilot design. KPMG’s public AI governance material describes the general value of a system inventory, policy controls, and outcome observability; because it is a vendor alliance page, treat it as a commercial framing rather than independent evidence that any given platform’s controls work. KPMG’s AI governance overview can help teams ask what their own evidence bundle should contain.

An evaluation can fail before statistics enter a dashboard if the system cannot tell what state the account is in. A basic event record should distinguish requested action from confirmed action:

{ "evaluationId": "synthetic-run-2026-11-04-0042", "assignment": "ai-first", "accountRef": "TEST-ACCT-17", "policyVersion": "collections-policy-7", "agentVersion": "voice-agent-build-31", "eventType": "payment_requested", "eventTime": "2026-11-04T10:15:00Z", "sourceEventId": "evt-018a", "ledgerResult": "unknown", "nextAction": "pause_and_reconcile" } Enter fullscreen mode Exit fullscreen mode This example is a proposed test fixture, not a sample production schema. It contains no personal identifiers or actual debt data. The critical idea is the separation between “requested,” “authorized,” “scheduled,” and “settled.” If a payment endpoint times out after receiving a request, the conversational agent must not repeat the request until an authoritative ledger or processor lookup settles the state.

Otherwise a retry can turn a network uncertainty into a duplicate payment or conflicting customer message.

Use an idempotency key for a side-effecting request and a correlation identifier for the wider conversation. The backend—not the language model—should enforce uniqueness, account state, amount limits, and policy conditions. Keep the agent’s tool result small: permitted next action, safe explanation code, and required handoff destination. Do not ask the model to reconstruct the balance from a transcript or infer whether an earlier payment actually cleared.

The event chain for a call should be auditable: assignment, eligibility check, connection, identity result, rule decision, spoken path, tool request, authoritative response, restriction or dispute update, human handoff if any, and final outcome. Use server-side timestamps and immutable identifiers. The transcript is valuable context, but it should not be the only record of a system action. If a policy update is accepted but not propagated to the dialler, the later event history should show that gap instead of pretending the call stopped.

Promise substitution. A verbal promise is counted as a payment. Fix the event model and report kept promises separately from settled transactions.

Denominator drift. The AI metric uses answered calls while the human metric uses assigned accounts. Use the same assignment-level denominator for the primary comparison, then report reach and answer as secondary outcomes.

Selective handoff. Difficult AI calls are transferred to humans and then the full successful result is attributed to AI. Keep original assignment and handoff as separate fields; analyze both assignment and actual exposure.

Policy updates arrive late. A dispute or stop request is in the CRM but a dialler has a stale queue item. Test the dispatch-time eligibility check and capture acknowledgements from every queue.

The wrong business rule is automated. A model or prompt faithfully repeats a bad contact policy. Treat legal interpretation and the approved policy table as externally governed inputs; version them and require review when changed.

A payment is scheduled but fails. The conversation says “you are all set,” but the payment processor later rejects or reverses the instruction. Keep the conversation promise distinct from processor confirmation, and send a corrected follow-up through an approved channel.

The model fills a silence with pressure. A long pause or emotion cue triggers persuasion. The safe behavior should be a neutral check-in, offered human support, or an end to the call—not an invented deadline, threat, or repeated demand.

A transfer exists on paper only. The tool reports “transferred,” but the queue is closed or the caller waits indefinitely. Test transfer completion, wait time, abandonment, and fallback wording; a transfer attempt is not service access.

Vendor statistics lack a usable comparison. A payment ratio is shown without the account count, time window, eligibility rule, or control group. Request a measurement sheet and validate a local result rather than importing the headline.

A result is averaged across unlike segments. A strong routine segment hides a failure for a language group or hardship scenario. Stratify the approved report, protect small groups from exposure, and do not expand a segment that has not passed its own tests.

A model version changes during the test. Outcome differences combine product changes and model changes. Freeze the treatment or mark the change as a new evaluation period and rerun the relevant acceptance cases.

A complaint is treated as a statistical rate. Complaint data can highlight a failure mode, but the CFPB says its published database is not a representative sample. Use complaint narratives to improve case coverage and use internal, well-defined denominators for rate estimates.

Collection conversations can contain financial, health, family, employment, or legal information. The purpose of evaluation is not to create a permanent audio archive. Decide whether raw audio is necessary; if not, use a structured event record or narrowly retained transcript segment. Restrict access by role, log review activity, redact payment credentials, and define deletion before recordings arrive. Apply jurisdiction-specific recording and notice requirements rather than assuming that a vendor’s default setting is acceptable.

A model should not create a new inferred “vulnerability score” and feed it back into collection pressure. If the system recognizes a hardship cue, the control should direct it toward a permitted support path and a trained person. The purpose of detecting a cue is to reduce the risk of an inappropriate automated response, not to exploit an individual’s circumstances. Document why each field is collected and how it can change an allowed action.

The OECD’s AI principles are policy recommendations and governance guidance, not a substitute for the laws applicable to debt collection. The CFPB and FTC sources describe U.S. debt-collection concepts, but the FDCPA’s coverage and state protections matter; the European Union, United Kingdom, Canada, and other jurisdictions use different frameworks. A cross-border voice campaign needs a jurisdiction map before a common script or metric is applied everywhere.

Voice AI can participate in a collections workflow, but it should not be judged by whether a call sounds empathetic or ends with a verbal yes. The NBER study is a caution against that shortcut: in its specific Chinese consumer-finance setting, AI callers’ promises did not translate into the same later repayment pattern as human callers, and a human switch did not make every difference disappear. The exact long-run percentage needs careful denominator reconciliation across summaries; it is not a forecast for another portfolio.

Use an evidence contract that compares verified outcomes on the same eligible population, runs safety and service guardrails alongside operational metrics, and treats a spoken promise as an intermediate event. The synthetic 12-case suite gives product, compliance, and operations teams a concrete starting artifact. If a system cannot show the approved state change, safe fallback, and audit evidence for those cases, it has not earned the next level of autonomy. A good evaluation can still support a narrow launch; its purpose is to make the boundary and the proof visible.

Facts are the descriptions of the cited rule text, complaint-data limitations, and research design/results. Inference is that evaluation should distinguish a promise from a realized outcome and test state changes at backend boundaries. Recommendation is the proposed outcome contract, guardrails, 12-case suite, and staged rollout. None is a ranking, compliance, or collection-performance guarantee.