Stop Trusting Your AI Agents to Tell You They Are "Done"

I spent three days last week watching two AI agents argue in a circle.

Neither one was hallucinating. Neither one was broken. Both were being perfectly obedient to their prompts.

Agent A (the coder) wrote a function. Agent B (the reviewer) said "change the variable name." Agent A changed it and passed it back. Agent B said "actually, change it back, the first name was more descriptive." Agent A changed it back.

They did this for three days. The API bill was $42. The code was exactly what it was on hour one.

This is the dirty secret of the current AI agent hype cycle. We are building incredibly complex, multi-agent swarms, giving them tools to interact with the real world, and then trusting them to tell us when the job is done.

They cannot be trusted. Not because they are malicious, but because they are text predictors.

The "Black Box With a Credit Card" Problem

The most dangerous part of an AI agent isn't when it crashes. It's when it confidently says "Done!" and you have absolutely no way to replay the exact tool calls and state changes that led to that claim.

When you ask a LangChain or AutoGen agent to complete a task, it usually responds with something like: "I have successfully updated the database and emailed the client."

But did it? Or did it just predict the most likely sequence of words that usually follows a task like that in its training data?

If you cannot independently verify that the database write actually happened, and that the email payload actually left your server, you don't have an autonomous system. You have a black box with a credit card.

The standard advice right now is to slap a step limit on your agent.

agent = create_agent( model="gpt-4o", tools=[search_tool, db_write_tool], max_iterations=10 ) If you do this, you are going to have a bad time.

max_iterations is the equivalent of a doctor telling you, "If the surgery takes more than 10 minutes, just stop and walk away."

An agent can loop 5 times and burn $20 doing absolutely nothing.

An agent can legitimately need 20 steps to solve a complex problem, but gets killed at step 10.

Step count measures effort. It does not measure progress.

Why do agents loop? It's almost never because the model is "too dumb." It's because the state is desyncing.

When an agent calls a tool and the tool fails (e.g., a 500 error, a malformed JSON response), what happens? If you are relying on the prompt to handle it, the model will usually say, "My apologies, let me try again," and call the exact same tool with the exact same arguments.

The model thinks it is making progress because the conversation history is growing. The tokens are changing. But the actual state of the world hasn't changed at all.

You end up with a "polite infinite loop." The agent keeps saying "sorry" and retrying a broken tool until it hits the max_iterations cap or drains your token budget.

* The Real Fix: Move the Guardrails Outside the Model * You cannot fix a state problem with a better prompt. You have to move the enforcement out of the LLM and into a deterministic layer that wraps around it.

If you are building agents for production, you need three things:

1. A Pre-Execution Validation Layer Never let the model execute a tool blindly. Before the tool call runs, validate the arguments against a strict schema. If the model hallucinates a parameter, reject the call and return a structured error. Don't let a malformed payload hit your API.

2. State-Change Diffing (The Loop Killer) If an agent calls search_flights("NYC", "LA") and gets a bad response, and then 3 steps later calls search_flights("NYC", "LA") again, your harness needs to catch that. If the same action is repeated without the underlying state changing, kill the execution.

3. Receiver-Side Idempotency If your agent is making network requests or writing to a database, the receiver must be idempotent. If the agent passes a unique idempotency key, the database knows to ignore the duplicate write, even if the agent retries 5 times because it didn't understand the first response.

We need to stop obsessing over "smarter models." A 2-million token context window just gives an agent more room to get lost.

The next wave of AI engineering isn't about better prompts. It's about better plumbing. It's about building harnesses that assume the model will hallucinate, the API will timeout, and the agent will get stuck.

If you can't replay the exact tool calls and state changes that led to an agent saying "done," you don't have a system. You have a liability.

Build the hard stop. Build it outside the model.