Why My AI Agents Aren't Allowed to Make Things Up

Posted on Oct 27 • Originally published at Medium

The first time an AI assistant handed me a confident, beautifully formatted answer that turned out to be invented, it wasn't a disaster. It was a phone number. It looked right, it had the right area code, and it belonged to nobody. I caught it because I happened to know the real one. That's the part that stuck with me: I caught it by luck, not by design. At INASNAP, we decided early that luck isn't a control. So the agents we build follow one rule above almost every other: if you didn't get it from a real source, you don't say it.

I've been building sites and apps for about twenty years, and running a business for nine. In that time I've learned that the most expensive bugs aren't the ones that crash. They're the ones that look correct. A crashed page gets fixed in an hour. A wrong number in a proposal, an invented customer quote in a case study, or a made-up statistic in a press release can sit there for months, quietly costing you trust. Large language models are very good at producing things that look correct. That's exactly why they need rules about what they're allowed to claim.

People talk about "hallucination" like it's a defect that the next model release will fix. Models do get better. But the underlying behavior doesn't go away, because it comes from what a language model is: a system that predicts plausible text. Plausible and true overlap most of the time. When they don't, the model has no built-in alarm that says "you're guessing now."

So I stopped waiting for the model to fix it and started treating it like any other engineering problem. You don't trust user input in a web form; you validate it. You don't trust a network call to succeed; you handle the failure. In the same way, you don't trust a model's factual claims; you require them to come from somewhere you can check.

That's a harness problem, not a model problem. I wrote about the broader idea in The AI Agent Harness Matters More Than the Model. Zero fabrication is one of the most practical pieces of that harness.

Here's how I phrase it for every agent we run:

Never invent numbers, quotes, citations, sources, names, or other facts you didn't get from a real tool, file, or source. When you don't have the data, say so, and point to the real way to get it.

That sounds obvious. In practice, it changes a lot of behavior. A few examples from our own work:

The pattern I keep coming back to is what I call, for myself, "source or silence." Every factual claim an agent makes should trace back to one of three places:

If a claim can't point to one of those, it doesn't go in the output. The agent can still reason, suggest, and draft. It just has to be clear about which parts are facts and which parts are its own suggestions.

The engineering side of this is less glamorous than it sounds. Mostly it's:

That last one ties into what I call glass-house transparency. If an agent sends you a number, you should be able to see where it got it. If you can't, treat it as a guess.

Here's a subtle one. Sometimes you genuinely need placeholder data. A demo dashboard, a mockup of a report, a training video showing what a feature looks like. Fake data is fine there. Unlabeled fake data is not.

Our rule: placeholder data is always clearly marked as example data. In a mockup, it says so on the screen. In a video, the caption says it's a demo. In a spreadsheet template, the column header or a note says "sample values."

Why be so strict about it? Because example data escapes. A screenshot from a demo ends up in a slide deck. A sample figure from a template gets copied into a real proposal. Six months later nobody remembers it was made up, and now it's a "fact" your company has been repeating. A label costs nothing and stops that drift.

When zero fabrication is working, the agent's answers get a little less smooth and a lot more useful. Instead of:

"Your top competitor has around 40 employees and raised a Series A last year."

"I couldn't find a reliable headcount or funding history for them. Their site lists a team page with 12 people, and I didn't find any press about funding. If you want, I can check a business registry or you can share what you know."

The second answer is less impressive at a glance. It's also the one you can actually put in front of a client, a partner, or an investor without a knot in your stomach.

The same goes for writing. When I have an agent help draft an article like this one, it isn't allowed to invent a "recent survey" to open with, or pull a quote from someone who never said it. If I want a stat, I go find it, or we write the point without one. Honestly, most of the time the writing is better for it. Plain experience beats a borrowed number.

I hear this a lot, and I understand it. If the selling point of AI is that it always has an answer, then teaching it to say "I don't know" feels like a step backward.

I'd put it differently. An assistant that always has an answer isn't helpful. It's agreeable. Helpful is an assistant whose answers you can act on. Every time an agent fabricates something and you catch it, you trust it a little less, and you start double-checking everything, which erases the time it was supposed to save. Every time it says "I don't have that, here's how we can get it," you trust it a little more.

For small businesses especially, that trust is the whole game. A big company might have a legal team, a comms team, and three layers of review between an AI draft and the public. A small business might have one owner and a laptop. If the AI makes something up, there may be nobody standing between that mistake and a customer. That's exactly the kind of business I want these tools to help, which is why I care so much about the rules.

You don't need a custom agent platform to use these ideas. If you're using AI tools in your business today, a few habits go a long way:

At INASNAP Labs, zero fabrication isn't a feature we bolt on at the end. It sits next to read-only-first access, permission-denied defaults, and human approvals as part of how every agent we build is supposed to behave. The goal isn't an AI that sounds smart. It's an AI you can actually hand real work to, because when it doesn't know, it tells you.

If that's the kind of AI you want working in your business, or the kind of work you want to be doing, I'd like to hear from you. You can learn more at inasnap.org, and if you're interested in joining us, take a look at inasnap.org/apply.

Michael Petler, Founder & Principal Engineer, INASNAP LLC (INASNAP Labs), Pennsylvania.