AI Hiring in a Live Company: The Agent Filled the Funnel. The AI Interview's Labels Did Not Line Up With Who Is Still Working

Posted on Nov 3 • Originally published at arsentev.ai

Part 2 of "AI managers in a live company". In five weeks an AI agent read 1,973 job applications and booked interviews without a recruiter. On day 36, 2 of 7 "hire" and 6 of 12 "maybe" candidates were still working, too few to tell the labels apart.

AI hiring means software agents take over parts of a recruiter's job. They read applications, decide who gets an interview and book it. Sometimes they run the first interview too. The AI interview is the newest piece. The candidate opens a link and talks to a voice AI instead of a person. Then a model writes a verdict into the company's hiring system.

This is the second part of a series on AI agents doing management work in one live business, a US-facing telemedicine company. Part 1 followed an AI supervisor running the company's phone team. This part follows the same company as it hired people for that team. It covers the path from the first application to the first working day, and beyond that as far as today's data goes.

The short version. The agent took the slow, repetitive part of hiring and handled it at a scale no one on the team could match. The AI interview was meant to replace the human first round. Instead it ran alongside it. Its verdict showed no sign of separating the people still working from those who stopped, though the numbers are small. And a large part of the losses came not from choices but from no-shows and decisions left waiting.

The company's name is under a non-disclosure agreement. Days are counted from the day the first candidate entered the CRM (day 1). The snapshot is day 36. All numbers come from the company's CRM, the AI interview records and the hiring agent's own files. The method is in the companion technical report.

The role is a phone job. The person calls partner physicians, helps them get started and keeps them active. It calls for people who have done outbound phone sales before and can work a full day from home.

One job posting brought in about 2,400 applications by day 32. A second, similar posting added more. Reading the 1,973 worth a close look would take one recruiter 65 to 100 hours, time this small team did not have. The person who would normally run the first interviews is the team's human supervisor, who also runs the floor every day.

So the company gave the front of the funnel to an agent.

The hiring agent is a Claude agent. It reads incoming applications on the job site, including the answers to the posting's screening questions and the work history. It picks candidates by rules that leadership writes. It enters them into the CRM, offers a time and books an interview slot. The invitation goes out in the same message as the proposed time, with "reply if this does not suit you". So the data has no separate "candidate replied" step. A candidate moves from picked to booked at once.

Over 36 days the agent read 1,973 applications in detail. The CRM received 142 candidates, and 138 human interviews were booked for 130 of them.

The rules changed often. Leadership changed the selection rules eight times in the first 15 days. The changes included phone sales only, no second chance after a missed interview, and a stricter checklist built from the supervisor's past decisions. The agent applied each change from the next batch on.

Here is the human-interview part of the funnel, with all bookings counted.

Almost one booking in three ended in a no-show. In the second calendar week it was close to four in ten.

The second leak was waiting. On day 15, 37 candidates had passed the supervisor's interview, and the CRM showed no hiring decision for any of them, although a few had already been given CRM accounts. The oldest had been waiting 13 days. Nobody had decided "no". Nobody had decided "yes" either. The AI supervisor from part 1 found this pile and kept raising it in its daily summary, but it could not hire anyone. A hire needed a CRM account, a start date and someone to meet the newcomer, and all three were people's jobs.

On day 15, leadership decided to remove the human supervisor from the first round. From then on, candidates would get a link to an AI interview instead of a booking.

The CRM stores a number alongside the verdict: 90 for HIRE, 50 for MAYBE, 10 for NO. In part 1, the AI supervisor's weekly report said three candidates had "scored 90/100" in the AI interview. That was a fixed label for HIRE, not a measured score. The part 1 report flagged it as misleading.

From day 16 to day 36, 41 candidates were invited. 22 finished the interview, or 54%. All 22 finished between day 16 and day 26. In the last ten days of the period, nobody finished one, even though invitations kept going out; the records do not show why. Human interviews, meanwhile, came back. Four were booked in the week of the rule that was meant to end them, and 23 more in days 21 to 36.

The final verdicts were HIRE 7, MAYBE 12, NO 3. One early NO was re-scored as MAYBE after the scoring rule was corrected.

For 16 candidates, there is both an AI verdict and a human interview result. The AI agreed with the supervisor on all five HIRE verdicts and disagreed on both NO verdicts: in both cases the supervisor had passed the candidate weeks earlier. MAYBE results split six passes to four fails across nine candidates (one was interviewed twice). The order varies: for some candidates the human interview came first.

Trial days began on day 21. A trial is one working day in the CRM with actual calls. The team lead judges it and records the result the same day. The table shows where everyone who received a CRM account stood on day 36.

(One more account went to a candidate who was invited to the AI interview but never took it. That person made calls and left.)

Thirty candidates received a CRM account. Eleven of them never made a single call. Nineteen started, and nine of those stopped after one to four days of calls: for five the company ended it after a trial day, one left on their own, and for three the records do not say who ended it. Ten are working.

The three candidates behind the AI supervisor's "90/100" line were all scheduled for trial days on day 21. One is still working. One never made a call. One made calls for one day.

The numbers are small. They do not prove that MAYBE is a better sign than HIRE. They do show no sign that the label sorted people the way its name suggests. Of the ten people now on the team, six came through as MAYBE and two as HIRE (Fisher exact test for 2 of 7 against 6 of 12: p ≈ 0.63).

Speed changed more clearly, though not only because of the AI interview. Compare everyone who entered the CRM with the same seven-day window. Of 112 candidates who entered before day 16, one made a first call within seven days of entry. Of 28 who entered on days 16 to 29, six did. Trial days, which started on day 21, are part of that change too. A CRM account turned into a first call in one to four days for all 19 who started. The wait always came before the account, never after it.

The hiring agent decides who gets an interview. It does not decide who gets hired. (This is a different agent from the part 1 supervisor, with its own rules: it may record decisions in the CRM.)

The CRM holds 48 hire or reject stage changes; some candidates have more than one. People entered 36 and the agent entered 12. All four hires the agent entered cite the team lead's decision in the team chat; one also adds the agent's check of the trial day against CRM activity. The trial verdict belongs to the team lead.

There is one exception, and it matters. All three candidates with an AI NO were rejected on that verdict. The agent entered two of those rejections, and a person entered the third, citing "score 10". Two of the three had already passed the human supervisor's interview. There is no record that a person read the transcript before any of the three rejections. That is the point where an AI verdict alone ended an application. It is also where rules on automated hiring decisions, such as Illinois's reporting duty for AI-only screening, would focus if they applied.

Here is how this AI interview measures up against them.

What it did. The page tells the candidate that answers are saved as text for the team. It says plainly that an AI plays the physician in the role play. The scorer was told to judge the content of the answers only. Candidates could record an answer again. Most HIRE and MAYBE candidates (14 of 19) also had a human interview.

There is also the matter of what the scorer was asked to weigh. One NO was tied to working-hours availability. A full working day is a real requirement of the job. But a rule like that can fall harder on some groups, and impact ratios by sex and race/ethnicity, of the kind a Local Law 144 audit requires, is where that pattern could show up.

A page that met the yardstick would say this before the first question: you will talk to an AI. The video is an AI avatar of a company executive. A model will score your answers, and a person makes the decision. You can ask for a human interview instead. And no NO would end an application unless a person had read the transcript.

The agent replaced the reading. At two to three minutes per application, 1,973 applications come to 65 to 100 hours of a recruiter's time (the author's estimate). It also replaced the booking and the reminder messages.

It did not replace the human first round. The AI interview was meant to, and in the week of the rule few human interviews were booked. Then human interviews came back, and most people who passed the AI interview were also interviewed by a person. It did not replace the team lead's trial-day decision, the operations manager who creates accounts, or the onboarding call.

Nobody hired through this funnel has reached 30 days yet. On day 36 the longest-working person had 22 calendar days since their first call. Version 1.1 of the technical report will add 30-day retention for everyone who started, split by AI verdict, once the first group passes that mark.

This is one company, two similar job postings and 36 days. There is no control group. The AI interview was not randomized, and the people who took it entered the funnel at a different time from those who did not. The groups in the outcome table are small, so the differences between them are not statistically meaningful. Candidates were matched to CRM accounts by name. 30-day retention cannot be observed yet. Business results are under the NDA. The full method, formulas and limitations are in the companion technical report.

The calculations behind every figure: technical report, doi.org/10.5281/zenodo.23927437.

The full version: arsentev.ai/press/ai-hiring. The technical report page: arsentev.ai/research/ai-hiring. Part 1 of the series: An AI Supervisor Ran a Call Center Team for Three Weeks.

Evgenii Arsentev, MD, PhD, Founder & CEO of Dad4Kids. ORCID 0000-0002-9120-7298

AI assistance: drafted and edited with an AI tool; the figures come from the company's records, and the calculations are in the technical report.