How to Vet an AI Agency Demo Before You Sign

Posted on Oct 9 • Originally published at pykero.com

An AI agency demo proves exactly one thing: the system works on the inputs the agency chose to show you. It does not prove it works on your data, your edge cases, or your users at 2 a.m. Before you sign, you need to replace their inputs with yours, look at the failures instead of the successes, and get the test set written into the contract.

This matters more for AI projects than for ordinary software. A CRUD app either saves the record or it does not. An LLM-powered agent can look perfect on fifty curated examples and then confidently produce garbage on the fifty-first. A founder who cannot tell a rigged demo from a robust system is buying on faith.

Nobody has to be dishonest for a demo to mislead you. Three things happen naturally:

None of this is malicious. But the burden is on you to see past it.

The single most useful thing you can do is show up with your own data. Not a description of your data, the actual records.

Pull 20 to 50 real examples from the workflow the agent will handle. Deliberately include the ugly ones:

Then ask the agency to run these live, in front of you, and show you every output. Not an accuracy percentage. The raw outputs, side by side with the inputs.

If they ask for a week to "prepare the environment" first, that is useful information too. A system that needs a week of prep to handle 30 of your records is a system that will need a week of prep for every new customer segment you add.

Flip the usual demo script. Instead of "show me it working", ask:

A demo hides architecture, and architecture is what you are paying for over the next two years. Ask them to walk through the request path for one input, start to finish:

Be especially alert to complexity theater. Agencies sometimes present a five-agent pipeline with planners, critics and orchestrators because it looks sophisticated on a slide. In our own outreach tooling, where we scrape each prospect's site and have a local model draft one tailored email, the version that extracted facts and wrote the draft in a single call beat the multi-step chain on both cost and output quality. Every extra hop was another place for the pipeline to lose context or hallucinate a bridge between steps. We wrote up the trade-offs in single call vs agent chains.

Simple is not automatically better, but a team that can explain why each component exists is a team that understands the system. A team that cannot is maintaining something they inherited from a tutorial.

The gap between a demo and a deployed system is where most AI projects die. The good news is that the 30 records you brought and the request walkthrough you just sat through give you everything you need to make these questions concrete instead of hypothetical:

A team that has taken agents from pilot to production will answer these without hesitation, because they have been burned by each one. Our pilot to production guide lists the specific failure modes we see most often.

Here is the move that separates a hopeful engagement from a safe one. Take the inputs you brought to the demo, agree on what a correct output looks like for each, and write that into the statement of work as the acceptance test.

This does three things at once. It forces both sides to define "working" in concrete terms before money changes hands. It gives the agency a target they can actually hit, which good agencies appreciate. And it gives you a regression suite you will keep running for the life of the product.

Agree on a threshold, not perfection. Something like "at least 90 percent of the acceptance set correct, zero outputs in the unsafe category, and every incorrect output routed to review" is realistic for most document and sales workflows. A hundred percent is a sign nobody has thought about the failure path.

Walk in with this and you will learn more in an hour than in three polished pitches:

An agency that welcomes these questions is one you can work with. One that treats them as hostile is telling you how the project will go.

If you are evaluating AI partners right now and want a second opinion on a proposal, or want to see how we would run your 30 ugliest records, let's talk.

Originally published on the Pykero blog.