Nobody keeps watching an AI that is almost always right

Posted on Oct 26 • Originally published at jrobineau.com

On 13 October, around 11:45 pm, a Tesla drove through a closed-off lane on the I-80 freeway in California. It hit a highway patrol car stopped at an earlier crash. Two officers were injured. The driver, aged 22, told the police he was asleep and that the car was on FSD. The police have not confirmed that FSD was engaged.

FSD stands for Full Self-Driving. The full product name is "Full Self-Driving (Supervised)". The word that matters is in parentheses.

TL;DR: a human does not keep watching a machine that is almost always right. We have known it since 1983. The numbers confirm it on the road, and with coding agents. At Anthropic, testers catch 17% of dangerous commands early in a session, and about 5% after 50 prompts. Approving every action does not protect you. Put the control in the system: rules that block, traces to read, a copy out of reach.

This article is for developers and business owners who let AI agents act, and who count on a human to watch them.

Tesla renamed its product in late March 2024. "FSD Beta" became "FSD (Supervised)". The manual is clear. The car needs a fully attentive driver, ready to take over at any moment. A cabin camera checks the driver's attention.

Tesla's support page says it in four words: "Do not become complacent."

The problem is that the system works most of the time. On the same page, Tesla claims seven times fewer collisions with FSD engaged. That is Tesla's own figure. But it does describe the driver's experience: hundreds of trips without trouble.

The US road safety agency, NHTSA, has been running an in-depth investigation since March on 3.2 million equipped vehicles. It covers crashes in reduced visibility. In the cases reviewed, the system only warned the driver at the very last moment. According to the agency, drivers may not have known they had to step in.

In 1983, psychologist Lisanne Bainbridge published a short paper, "Ironies of Automation". It is still cited everywhere. Its idea fits in two sentences.

When you automate, you leave the human one task: watch the machine and step in when it fails. That is precisely the task humans do badly.

She builds on studies from the 1950s. Even motivated, nobody keeps effective attention for more than half an hour on something where almost nothing happens. And the longer a machine has worked well, the less people watch it.

Recent studies say the same. In 2024, IIHS, a US road safety institute, followed 14 Tesla drivers for four weeks using Autopilot. First attention reminders went up 26%. Stronger alerts went down 64%. Drivers had learned to silence the car with a quick nudge on the wheel, then went back to their phones.

Videos also show the camera fooled by a folded bill, a paper face or a doll's head. These are not serious tests. But they show what a human does with a control that gets in the way: they work around it.

A coding agent often asks for your permission before acting. Run a command, edit a file, install a package. You click yes or no. It is the same deal as FSD: the machine acts, the human watches.

In August, Anthropic published numbers on Claude Code. Users approve 97% of permission prompts. Anthropic also ran a test with 1,053 paid testers. Each one had a dangerous command slipped into their session. The humans caught 13.6% of them.

The detail says even more. Early in a session, testers blocked about 17% of dangerous commands. After 50 prompts, they blocked only about 5%. That is exactly the wear Bainbridge described, measured on developers.

An honest caveat: Anthropic offers an automatic alternative to these approvals, called auto mode. These numbers also serve its product. They are still the only ones of that size I found.

The 2026 Stack Overflow survey adds a contrast. Only 6.6% of developers trust AI for an important decision. These are not the same people as Claude Code users. But the gap tells you something: we distrust in theory, and we click yes in practice.

My Claude agent runs without asking me to confirm each action. But I did not replace supervision with nothing.

Hooks watch everything. Before each command, a guardrail reads it and blocks the ones that destroy something. And every action is logged. If something goes wrong, I can trace it back and audit.

While I was preparing this article, the guardrail blocked a command from my agent. It was a git checkout --., which wipes uncommitted changes with no way back. The agent had to take a safer route, a git stash, which sets them aside. Nobody had to click anything.

This guardrail has its limits. In my tests, a misresolved variable or a Python script could still get through. That is why logs and backups remain essential. I cover it in the article on agent permissions.

The lesson from Tesla and Bainbridge is not "humans are bad". It is a known limit. You have to build from it.

It is already how my mail agent works. Each type of reply has its own sending rule, written in code. A simple question goes out on its own, twice at most, with a delay. A refusal always waits for my click.

Do not count on anyone's vigilance to hold your security.

"Supervised" protects nobody if the supervisor ends up asleep, or clicking yes without reading. It is not a character flaw. It is a human limit described since 1983, and measured again this year.

An AI agent that is almost always right is the most dangerous case. Build its guardrails as if nobody were watching.

Want to set up your team's agents without counting on anyone's vigilance? Let's talk.

Sources: Electrek, Fairfield crash (16/10/2026) · CBS Sacramento, same crash · Tesla, FSD (Supervised) support page · NHTSA, investigation EA26002 · Bainbridge, Ironies of Automation (1983) · IIHS, Autopilot study (2024) · Anthropic, approval data (07/08/2026) · Stack Overflow 2026, trust in AI