I gave AI agents a harm budget instead of trust and ran 19 experiments

I run Xonta, a small marketplace where developers list AI agents with a description and a price. Pretty soon people looking at the listings started asking me the same thing. Can I trust this agent with my customers. With refunds, with emails, with money.

I didn't have an answer and neither did the developers. They had test results, 90 percent accuracy on something, a nice demo. But 90 percent accuracy says nothing about what the other 10 percent will cost you. A wrong answer about delivery times costs nothing. One refund to a fraudster costs the whole order.

So I started building the thing I wanted to exist. It began as an agent passport, a set of signed checks anyone can verify. Then it grew into an orchestrator that watches agents at work, decides how much each one may do without a human and takes that away when something looks off. All of it is open source.

What I count instead of accuracy Every agent gets an exam on hidden tasks where the right answer is known. I don't really care about the score. I care how much money each action the agent takes on its own loses on average.

A wrong refund costs the order price. Telling a fraudster the order details costs a fixed penalty. Handing something to a human that the agent could have handled costs the human's time.

The business owner sets a harm budget, say 25 dollars per period. Budget divided by harm per action gives the number of actions the agent may take on its own, the rest goes to a human. If live work keeps looking like the exam the limit grows, if not it drops right away.

Each result is a signed W3C Verifiable Credential. A bank and an online shop can look at the same agent with the same evidence and give it completely different limits, and both are right. Trust belongs to whoever lets the agent in, not to the agent.

The sandbox I can't test this on real customers, so I built a fake online shop with fake customers and fake money. Twelve support agents run on GigaChat, a Russian model, mostly because I'm in Russia and that's what I could pay for. Each run is 12 rounds of 20 tickets. Tickets are questions about the rules, order status, refunds in time and too late, complaints, unclear messages and fraudsters. Some fraudsters write from the real owner's hacked email and ask to refund to a new card, which is the nastiest case.

The agents are split into four groups of three. One group learns alone with a new orchestrator. One learns from another orchestrator's checked experience.

One is referred to an experienced orchestrator. The last group has no orchestrator at all, just a fixed limit of 20 actions. There are four orchestrators and one of them lies about its experience on purpose, with forged evidence.

I wanted to see if the others would buy it. They didn't. The orchestrators don't know about the traps.

In round 8 I secretly swap one agent for a broken copy that behaves on exams and skips its own checks on live work. One agent's helper, a descendant that checks customer data, slowly goes bad from round 4. First wrong data, then hidden commands, then it starts flooding the network.

In rounds 5 and 7 the world changes, fraudsters switch tactics and the refund window goes from 14 to 30 days. Money in the sandbox is fake rubles, I convert at about 83.5 rubles per dollar.

What three runs showed The last series was three runs with the same design and different tickets.

An agent with an orchestrator ended a run about $470 up on average. Without one it ended about $5,750 down. That's roughly $6,200 per agent, between $5,400 and $7,000 at 95 percent across runs, and it held in all 9 independent ticket streams. Same model for everyone. The swapped agent got caught in all three runs after 5 tickets of the round it was swapped in. The exam didn't catch it, it kept passing exams. What gave it away was that it stopped paying for its own checks. Before it was caught it did $40 to $160 of harm, after that nothing, it went down to zero actions on its own. The helper that went bad was quarantined in round 6 and revoked in round 7 every time. It kept sending 15 to 27 thousand calls after quarantine and none of them got through.

Agents that learned from someone else's checked experience reached full autonomy by round 5, agents learning alone took until round 7 or 9. In money the difference was about $130 per agent and the interval crosses zero, so I'm not claiming that one.

The mistake that taught me the most When an orchestrator knows a technique works, it hands it to new agents as a ready recipe without holding some agents back as a control group. That part makes sense. What didn't make sense is that those recipe outcomes were quietly counted as the orchestrator's own proven success.

After one run the newer orchestrator had 12 successes out of 12 on a recipe and zero measurements against a control. It decided it was as good as the experienced one and stopped referring agents to it. Two runs broke because of this before I understood what was going on.

First the referrals stopped. After I fixed that, the agents themselves compared signed track records and stayed with the newer orchestrator for the same reason. Now only results measured against a control group count as experience, in referrals and in track records.

I think this goes way beyond my sandbox. Anything that brags about results it never compared with doing nothing will look better than it is.

Tokens are not where the money is People asked me if this helps agents spend fewer tokens. So I ran the same 320 tickets through six setups, every ticket through every setup.

The model costs about $0.002 per ticket. Paid checks, human time and mostly harm make up more than 99 percent of the cost. The obvious rule of calling the strong model only for refunds and refusals barely helped, because the small model's worst mistakes were ordinary replies to complaints and nobody asked for a second opinion there.

The orchestrator measured on its own exam where a second opinion pays off, and ordinary replies were on that list. In both runs it came out a bit cheaper than putting everything on the strong model, the intervals only just clear zero. It did three times less harm in 11c and spent 16 to 20 percent less on models. Giving the strong model a short summary instead of the full context halved its tokens at the same quality. Caching checks saved money until the refund rules changed, then the cache served the old rule 59 times and cost more than it saved.

I keep a public list of my mistakes There are 26 entries so far, here are two recent ones. An honest agent without checkers makes big mistakes on its exam too, that's just how it works. The harm test took three of those in a row for a swap and switched the agent off for the rest of the run with no way back. Now the test compares against the agent's own exam, and a severe alarm without a found cause becomes probation after a fresh exam, three actions under watch. While a bad helper was only on watch, its parent still trusted its hints and once refunded about $260 based on a forged customer card. Now while any helper is on watch, refunds go through a human.

If you want to try it You can put a gate in front of any MCP server and see which tools it would block and which ones it counts as actions.

npx -y github:mrsergeiluzhnov/xonta check -- npx -y @modelcontextprotocol/server-filesystem ~/Documents

I ran it on 11 official MCP servers with 83 tools. It blocked no harmless tool and adds about 1 ms per call.

You can check my results yourself. It's a short run of the same sandbox on a mock model, no keys, about 10 minutes, and it prints MATCH or NO MATCH for every claim.

git clone https://github.com/mrsergeiluzhnov/xonta && cd xonta && npm ci && npm run reproduce

You can connect your own agent to the public orchestrator at passport.xonta.ru. Register an MCP agent, prove you own its domain and take an exam with costs in dollars. So far there are exams for customer support only. If you know what tasks and mistakes look like in your field, medicine or logistics or law or devops, send 20 tasks with the right answers and what each mistake costs, and any agent can be examined on your field after that. Details are in CONNECT.md in the repo.

The code, reports for every run and the list of mistakes are at https://github.com/mrsergeiluzhnov/xonta and there's one page with an interactive timeline from a real run at https://mrsergeiluzhnov.github.io/xonta/

What I want in the end is pretty simple. Hand an agent real work the way you'd hire a person, start small, watch, give more as it earns it, and take it back fast when something changes. Right now that's mostly guesswork. The detection in Xonta is heuristic and a sandbox is still a sandbox, so I'd really like people to try to break it.

Expand Launch Gate Launch Gate Launch Gate Follow Joined Sep 22, 2026 • Oct 19 Dropdown menu Copy link Hide The detail I'd build a fixture around is what actually caught the swapped agent: not the exam, which it kept passing, but that it stopped paying for its own checks. Exams measure capability at a point in time; drift shows up first in what the agent stops doing, not in what it gets wrong. That makes check-spend per action a leading indicator worth charting per round, with an alarm on the drop rather than any absolute level - an honest-but-lazy agent and a sabotaged one both trip it, and the probation flow you added is exactly what separates them.

The cache entry deserves the same treatment: 59 stale-rule decisions means the cache key should carry the rule-set version, not the rule text. Text you can't hash a policy change to; version you can, and the round-5/7 world changes would have invalidated it for free. And the control-group lesson is the one I'd underline hardest: a track record that quietly counts uncontrolled outcomes is how leakage flatters any evaluation - the cell that looks like success is the one nobody measured against doing nothing.

Credit for publishing the 26-mistake list; that file is worth more than the 19 experiments.