I read every AI claim on 4,335 earnings calls. A keyword filter would have got most of them wrong

Posted on Nov 2 • Originally published at equibles.com

Apollo's chief economist argued in late October that AI has delivered "zero financial payoff" to S&P 500 companies outside tech, based on flat margins and revenue per worker. I wanted the firm-level version of that question: when executives outside tech talk about AI on their earnings calls, how often do they put a number on what it delivered?

The sample was every earnings call held since January 2024 by S&P 500 companies outside the tech sector, 4,335 calls from speaker-by-speaker transcripts. The headline:

Counting mentions is easy. Counting "put a number on AI" is not, and the gap between what a filter finds and what a person reading the sentence finds was the most useful lesson of the project.

The first pass looked for sentences by company speakers (not analysts) that mention AI and contain a number and a benefit word in the same or the next sentence:

import re AI = re. compile ( r " (?<![A-Za-z])A\.?I\.?(?![A-Za-z]) " # "AI" or "A.I.", not "AIG" or "said" r " |(?i:\bartificial intelligence\b|\bagentic\b|\bgen?AI\b|\bLLMs?\b|\bmachine learning\b) " ) NUM = re. compile ( r " \$\s?\d[\d,.]*|\b\d[\d,.]*\s?(%|(percent|bps|million|billion|hours?|days?|times|x)\b) " r " |\b(half|halved|doubled|two-thirds)\b ", re. I ) BENEFIT = re. compile ( r " \b(productiv\w*|efficienc\w*|sav\w*|reduc\w*|cut|faster|automat\w*|headcount)\b ", re. I ) def candidates ( sentences ): for i, s in enumerate ( sentences ): if AI. search ( s ): window = s + " " + ( sentences [ i + 1 ] if i + 1 < len ( sentences ) else "" ) if NUM. search ( window ) and BENEFIT. search ( window ): yield window Enter fullscreen mode Exit fullscreen mode The AI part is case-sensitive and allows "A.I.", the way some transcripts write it; the letter lookarounds keep it from matching inside names such as AIG.

It produced 534 candidate passages on 343 calls. Reading them, 122 calls had a real figure for what AI delivered. The rest were things a regex can't tell apart:

A second pass dropped the benefit-word requirement and widened it to revenue words (conversion, approval rates, growth, uplift), and listed every AI sentence with a number on calls the first pass hadn't flagged, or had flagged for a different sentence. It found 36 more calls.

The misses were mostly revenue and customer outcomes, phrased in ways a cost-savings vocabulary doesn't catch: an AI shopping assistant whose users convert at "triple" the rate (no digit at all), approval rates up 50 basis points from AI decisioning, an AI product line adding 300 basis points of organic growth. A few were cost claims split across sentences further apart than the window.

The lesson I'd keep: for a claim-level count, treat the regex as a way to rank what to read, not as the classifier. I read every candidate and every second-pass sentence; the labels took longer than all the code.

A day after publishing I found two more holes, both in the filters rather than the reading, and both in the first pass and the second.

The first was the speaker filter. Transcripts give each turn a role, and I kept CEO, CFO, chair, COO and the like. Executives whose titles don't fit the role list, such as a chief commercial officer or a division president, are stored as "Other", and I had grouped "Other" with the operator. C.H. Robinson's results were mostly presented by those executives:

"In the AI agent's first few months, it has been determining the freight class and code for about 2,000 orders a day, and it has reduced the processing time from 10 minutes or more per shipment to 10 seconds or less."

The second was the window. A figure often comes first and the AI credit second. A sentence with both in it works, but "total non-standard surcharges are generating a significant year-over-year benefit of over $180 million annualized. This is the result of a new AI image capture process" puts the number one sentence before the AI. Fixing both added 385 passages to the first pass and 459 to the second, all read by hand. They added 65 calls with a figure, 223 in all, and raised last quarter's share from 6.5% to 9.0%. Turns with no role at all I only counted when reading them showed a company speaker, because some of them are analysts.

Many figures credit AI along with other things: "efficiencies from organizational realignment, process redesign through AI and automation, procurement and vendor rationalization... more than $1.5 billion". I labeled those separately from figures credited to AI alone. Last quarter 27 of the 35 calls with a number credited AI alone; most of the shared-credit ones are company-wide cost programs. Without that split, the count mixes an AI agent that resolves a stated share of support issues with a reorganization that mentions AI once.

Before computing shares by quarter I hashed every transcript and grouped by company. Across the whole transcript archive, 182 transcripts were stored on more than one call event of the same company, 67 of them in different quarters; one company's investor webinar sat on five consecutive quarters' calls. In the S&P 500 sample there were 18 such pairs. Keeping one call per company per quarter handled the same-quarter ones, and I dropped the three pairs that crossed quarters:

-- PostgreSQL: identical transcripts on more than one call of the same company with h as ( select call_id, company_id, md5 ( string_agg ( text, ' ' order by turn_index )) as hash from transcript_turns group by call_id, company_id ) select company_id, hash, count ( * ) as calls from h group by company_id, hash having count ( * ) > 1; Enter fullscreen mode Exit fullscreen mode What the clean count shows Executives outside tech talked about AI on 61.2% of last quarter's calls and put a number on its payoff on 9.0%, about one in seven of the calls that mentioned it. Eleven companies account for 68 of the 223 calls with a number since 2024. The most common figure is a savings target for a company-wide program; the concrete results are narrow, in customer service, claims and document processing, and code.

Analysts asked about AI savings, productivity or returns on 6.7% of calls last quarter, against about 1% in 2024. The full write-up, with the charts, the sector table and every quote linked to its call: AI Talk Has Nearly Doubled Outside Tech

If you've labeled claims in transcripts or other long documents, did you find a way to get the filter's precision up, or did you also end up reading everything?

Data: S&P 500 earnings call transcripts, processed by Equibles.