"I asked ChatGPT" is the new "works on my machine"

When you work with agents, you learn fast that the answer depends on the context you give them. You forget it the moment you report their answer to someone else.

We've all met someone who counters our arguments with "I asked ChatGPT" or "Claude says." The discussion ends there, because arguing with that sentence means arguing with a machine everyone uses and nobody can question on the spot. I sometimes teach software development at a university, and I hear it from my students often enough to want to ban it from class.

That didn't stop me from saying it myself. I fight that sentence, and it still slipped out, in the middle of a discussion, with the answer as my argument. At the time, it seemed obvious to me that the others knew what context I had given the AI. They had no idea. All they had was the answer.

Developers have an expression for this kind of situation. Whoever says "works on my machine" isn't lying: the code really runs on their computer. Nobody else can check, because what makes it run, a version or a setting, only exists on that computer. An AI answer quoted in a discussion is in the same spot. It comes out of a conversation only the person quoting it has seen, and nobody in the room can check it.

I work with agents that write code, and I've already written about what that changed in the way we work. The skill you learn with them got a name in 2025, context engineering, which Anthropic describes as "the natural progression of prompt engineering." Phrasing the question well is no longer enough. You have to choose what the model has in front of it when it answers, the files as well as the conversation history, and with a different context, the same question gets a different answer.

That context is the answer's environment, and it stays in a private conversation. When you report the answer, nobody has seen the question, or what you pasted before asking it.

The scene also plays out in writing, on tech teams. In September 2025, someone wrote to the advice column Ask a Manager about a coworker who answered their questions by pasting ChatGPT's reply into the team's chat thread. The writer was asking for context on a problem, to understand its cause. What came back was a quick fix, produced by a machine nobody had given that context to.

I wasn't acting in bad faith the day the sentence slipped out, any more than the developer who swears the code works. What happens to both of us has had a name since 1989, when three economists described it: the curse of knowledge. Once you know something, it's hard to imagine someone else not knowing it.

It works both ways. On the machine side, anyone who works with an agent knows it: an instruction feels clear because we have everything else in our head, and the agent misses the point. The same lapse happens again when you report the answer, this time to humans.

In 1990, a Stanford graduate student, Elizabeth Newton, asked participants to tap out the rhythm of a well-known song with a finger, and asked others to name the song. The tappers always overestimated their chances of being understood, because they could hear the melody in their head. Whoever says "I asked ChatGPT" is in the same position. They hear their question, and the room only hears the answer.

A development environment takes nobody's side. An assistant tends to side with whoever is asking. Anthropic researchers showed it in 2023 on five commercial assistants, and OpenAI rolled back a ChatGPT update in April 2025 because it made the model too sycophantic.

In March 2026, a Stanford team measured the effect on the people asking, in a study published in Science. Across eleven models, the AIs affirmed users' actions 49% more often than humans did. In experiments with more than 2,400 participants, built around conflicts between people, a single conversation with a sycophantic AI was enough to make them more convinced they were right, and less willing to repair the conflict. They still preferred those models, and trusted them more.

The study isn't about work meetings. It still describes what we often bring back from a conversation with an assistant: a confirmation, which "I asked ChatGPT" dresses up with an authority nobody would dare claim for their own opinion.

Asking the same machine again checks nothing. In 2023, in New York, a lawyer filed a brief that cited case law ChatGPT had made up. He had asked ChatGPT whether one of those cases existed, and ChatGPT had said yes. The judge sanctioned the lawyers $5,000.

Yes, as long as you show where the answer comes from. Against "works on my machine," developers demand steps to reproduce, and a bug report without them goes back to its author.

An AI answer reported in a discussion should come with the same thing: the question, and what you had given the machine before asking it. The assistants already allow it. ChatGPT, for example, can share a conversation through a link, history and files included, and since September 2025 the APA has explained how to cite a specific conversation through that link in academic work.

The link never carries everything: for an agent working in a code repository, the context is the whole repository. It carries the question, at least. A model's answer never replays exactly the same way anyway. Neither does an intermittent bug, and we still ask for the steps.

These tools help me search, and find the objection I hadn't seen. Quoting them doesn't bother me, as long as the room can see what I asked.

Next time someone answers me with "I asked ChatGPT," at work or in class, I'll ask a question instead of wanting to ban it: what did you ask it? And next time the sentence slips out of me, I'll try to answer that before anyone has to ask.

Mathieu Poli, Head of Frontend Engineering at GoodBarber. I write about frontend engineering, product design and AI.

X: hellomathieup · LinkedIn: hellomathieup