Your AI Agent Said “Success” The Database Says Otherwise

Why testing AI agents requires verifying what actually happened, not just what the agent reported.

Imagine asking an AI agent to update a customer's subscription.

The agent calls a tool. The tool returns a success response. The agent tells the user:

“Your subscription has been updated successfully.”

The logs look fine. The tool call completed. The final answer is confident.

But the database still contains the old subscription.

Perhaps the SQL UPDATE matched zero rows. Perhaps the API accepted the request but rejected the operation later. Perhaps a browser agent clicked a button, but the form submission failed because the session had expired.

From the agent's perspective, the task is complete.

From the application's perspective, nothing changed.

This is one of the important testing problems we need to solve as AI agents move from answering questions to taking actions.

Traditional application code usually has explicit success and failure conditions. An HTTP response code, a database result, or an exception can tell us something about what happened.

Agentic systems add another layer of uncertainty.

An agent decides which tool to call, supplies its arguments, interprets the response, and decides whether to continue or stop. Each step can appear successful while the overall task remains incomplete.

result = await db. execute ( """ UPDATE subscriptions SET plan = ' premium ' WHERE customer_id =:customer_id """, { " customer_id ": customer_id }, ) return { " success ": True } Enter fullscreen mode Exit fullscreen mode The update might affect one row, several rows, or zero rows. Returning success: True without checking the result makes all three outcomes look identical to the caller.

The problem is not necessarily the LLM. The tool's success contract is too weak.

If an agent is allowed to treat a tool response as proof of completion, the application can report a successful action that never happened.

assert " updated successfully " in response Enter fullscreen mode Exit fullscreen mode The test passes when the agent produces the expected sentence.

It says nothing about whether the database changed.

Even checking whether the agent called the correct tool is insufficient. A tool can be called with the correct arguments and still fail to produce the intended result.

For action-taking agents, there are at least three different things to verify:

These are related, but they are not interchangeable.

A correct tool call does not guarantee a successful operation. A successful operation does not automatically guarantee the intended end state either.

A stronger approach is to check the result against the system of record.

For a database update, inspect the affected-row count and verify the relevant fields. For an external API, retrieve the resource and confirm its persisted state. For a browser agent, check the resulting page or the underlying record rather than relying on the click event.

result = await update_subscription ( customer_id, " premium " ) if result. affected_rows!= 1: raise AssertionError ( " Expected exactly one subscription to change " ) subscription = await get_subscription ( customer_id ) assert subscription. plan == " premium " Enter fullscreen mode Exit fullscreen mode This is illustrative code; the actual result fields depend on the database driver and application's data-access layer.

The key principle is independent verification. The check should obtain evidence from a source that can establish whether the requested action really took effect.

For eventually consistent systems, verification may need to wait for a final state rather than treating an intermediate pending status as completion.

And when the evidence is inconclusive, the system should say so. “Unverified” is more honest—and often safer—than “Done.”

There is a further problem: a test can contain a success check that looks meaningful but does not detect failure.

A useful negative test deliberately simulates an action that reports success without changing anything.

Then ask: does the verification still pass?

If it does, the check is not proving the behavior we care about.

This is the same idea behind testing failure paths in ordinary software. We should not only demonstrate that a system works when everything goes right. We should also demonstrate that our tests catch cases where an operation silently does nothing.

These problems are part of a broader shift in AI testing.

An evaluation should not stop at whether the answer sounds correct. It should also examine tool selection, tool arguments, action order, side effects, and the resulting application state.

This matters particularly for agents that handle refunds, update records, manage bookings, send communications, or trigger other consequential operations.

A useful regression test should establish what the agent was expected to do, record what it actually did, and verify whether the intended outcome occurred.

That gives us evidence we can compare across prompt changes, model upgrades, tool changes, and application releases.

This is one of the problems I have been exploring while building Assay Evals, an open-source project for behavioral regression testing of AI applications and agents.

Assay focuses on evaluating more than the final response. It can check agent trajectories, expected tool calls, end states, and behavioral changes between runs.

The project's roadmap also explores stronger independent verification: reading the result back from an API, database, or another system of record, and treating an inconclusive outcome separately from verified success.

You can explore the implementation and roadmap in the Assay repository.

As AI agents become more capable of changing the world around them, our definition of success must become stricter.

A successful model response is not necessarily a successful tool call. A successful tool call is not necessarily a successful business operation.

The right question is not “Did the agent say it worked?” It is “What evidence proves that it worked?”

That distinction is essential to making agentic systems reliable.

Because in production, confidence is not evidence, a completed tool call is not proof of an outcome, and a green evaluation is only as useful as the behavior it actually verifies.