AI Proof of Concept to Production: What Demos Skip

Posted on Oct 21 • Originally published at dablewdevs.com • AI-assisted

A good AI demo takes an afternoon. A model reads a document, produces a summary, and everyone in the room is impressed. Then the project stalls for months, because the demo worked on five hand-picked examples and nobody knows what it does on the five-thousandth real one.

The gap between proof of concept and production is not mainly about choosing a better model. It is ordinary engineering that demos skip: knowing whether the output is right, making it fit your software, controlling what it can do, knowing what it costs, and rolling it out so a bad day does not reach every customer. This is the checklist we work through when taking AI features to production.

The single biggest difference between teams that ship AI features and teams that stall is an evaluation set: a collection of real inputs, with what a correct output looks like, that you can score automatically. Without one, every prompt change is a guess and every model upgrade is a gamble.

For open-ended output, a second model scoring the first against a rubric ("LLM-as-judge") is often the only scalable option. It is also easy to get wrong: an uncalibrated judge happily passes bad answers or fails good ones, and you end up optimising toward its quirks.

The method we use, including in our own testing: have people label a sample of outputs against the same rubric, run the judge on that sample, and compare. Track false passes and false fails separately — a judge that passes everything looks great on average. Iterate on the rubric and judge prompt until agreement with human labels is high, and only then trust the judge on the full set. We have also found that a model's self-reported confidence is a poor stand-in for this; measure against labels instead.

Five specific yes/no questions written from how a human actually grades the output are far easier to calibrate than a single "quality out of 10".

If code consumes the model's output, "usually valid JSON" is a production incident waiting to happen. Major model APIs support schema-constrained (structured) output: you supply a schema and the response is generated to match it. Validate it anyway before it touches your database.

In an AI assessment and reporting app we built for a leadership consultancy, templates are generated as typed JSON from a free-text topic, and executive summaries follow a fixed section contract. We also gave the summariser labelled findings instead of raw scores, which removed a whole class of confident, numerically wrong statements. Giving the model less — only what it needs — often improves accuracy and lowers cost at the same time.

Prototypes typically run with an admin API key. Production AI that takes actions must act as a specific user with that user's permissions, so it can never do more than the person on whose behalf it runs.

These are the defaults behind the remote MCP server we built so a consultancy's staff could run operations from Claude: each tool proxies an existing authenticated endpoint and inherits the signed-in user's permissions.

Track tokens and cost per feature, per customer and per model from day one. It turns pricing decisions — included, metered, add-on — into arithmetic, and it catches runaway loops early. It also lets you safely move easy steps to a smaller model: if the evaluation score holds, the saving is real. Our AI agent cost breakdown walks through the token arithmetic at current prices.

When a customer reports a strange answer, support needs to see exactly what the model received, which tools it called, what came back and how long each step took. LLM-observability tools such as Langfuse, or traces exported through OpenTelemetry, give you that per-request history. Store prompts and outputs only where your data agreements allow, and redact what you do not need.

Sometimes the evaluation set shows the idea does not work well enough on real data, and no amount of prompt tuning closes the gap. That is a successful outcome of a proof of concept, not a failure — it cost weeks instead of a launch. The honest options are then to narrow the scope, fix the underlying data, keep a person in the loop permanently, or drop the feature.

If you have a prototype that needs to become a dependable feature, that is the core of our AI integration services. For agents that take actions across systems, see AI agent development.

Why do so many AI proofs of concept never reach production?

Usually because nobody can show the feature is reliable on real data. Without an evaluation set, teams cannot tell whether changes help, cannot set an accuracy bar, and cannot justify the risk of launching. Missing permissions, cost tracking and failure handling add to the delay.

Large enough to cover the real variety of inputs, including the awkward cases. Many features start with 50 to 200 examples and grow as production failures are added. A small honest set beats a large set of easy, invented examples.

Only after checking it against human labels. Compare the judge with people on a sample, track false passes and false fails separately, and refine the rubric until they agree. Then re-check periodically.

Do we need to rebuild our prototype for production?

Not usually. Most of the work is adding what the prototype skipped — evaluation, structured output, permissions, failure handling, tracing and a rollout plan — around the core idea, inside your existing codebase.

Originally published on the DablewDevs blog. DablewDevs builds production AI systems and software — AI agents, RAG, MCP servers and SaaS products. See what we build and what it costs.