Why ChatGPT Makes Things Up — And the Fix Nobody Told You

Posted on Nov 2 • Originally published at paragonaimag.substack.com

Originally published on Substack: https://paragonaimag.substack.com/p/why-chatgpt-makes-things-up-and-the

Full video: https://youtu.be/tI_02Sl3gHM

Podcast: [podcast URL once known; edit in after step 16]

Why ChatGPT Makes Things Up — And the Fix Nobody Told You, from Paragon AI MAG. ChatGPT invented six court cases that never existed — and real lawyers filed them in a real court — and got fined $5,000. This is not a story about AI lying. It is a story about a tool that guesses beautifully when it has nothing to check. Today: why models invent facts, the Mata v. Avianca case and its $5,000 lesson, how RAG actually works, a 3-step walkthrough you can use today, and the 5-rule check-before-you-trust list. Subscribe free to Paragon AI MAG for one source-checked AI story every day.

By Paragon AI MAG — one video and one podcast episode every day, every fact checked against sources several times before release.

ChatGPT invented six court cases that never existed — and real lawyers filed them in a real court — and got fined $5,000.

That is not a hypothetical. It is the story of Mata v. Avianca, Inc., and it is the clearest warning we have about how AI works — and how to use it safely. This is not a story about AI lying to you on purpose. It is a story about a tool that guesses beautifully when it has nothing to check.

Large language models like ChatGPT are not databases. They are next-word prediction machines. They are trained to predict the most plausible next word, not to look up a fact. They have no fact book inside. And their confidence is fluency, not truth: if an answer sounds smooth and confident, it feels true — even when it is completely invented.

The US National Institute of Standards and Technology prefers the term "confabulation" over "hallucination" for this reason — the AI does not literally see things that are not there. It strings together highly probable words that happen to be totally false. In its Generative AI Profile (NIST AI 600-1, July 2024), NIST describes hallucination as the production of confidently stated but erroneous or false content.

Not all hallucinations look the same. There are two main types.

Intrinsic hallucinations: the answer directly contradicts the source material. You upload a contract and ask for the renewal date, and the AI gives you a date that is not in the document at all.

Extrinsic hallucinations: the answer sounds right but cannot be verified from the source. This is the more dangerous type. It includes invented citations, wrong dates, invented software functions, and made-up quotations. It sounds authoritative, so you are less likely to check it.

In Mata v. Avianca, the lawyers fell into the second trap. They asked ChatGPT for legal research, received six cases that sounded perfectly real, and filed them — without verifying a single one.

In February 2022, Roberto Mata filed a personal injury lawsuit against the airline Avianca. In March 2023, his lawyers, opposing a motion to dismiss, filed a legal brief citing six federal decisions. None of them actually existed.

When the court could not find them and requested copies, the lawyer did something remarkable: instead of checking a real law library, he went back to ChatGPT and asked whether cases like Varghese were real. Optimizing for fluency, the AI cheerfully assured him that yes, they definitely were. The lawyer then submitted the AI's completely fabricated full-text opinions to the federal court.

On 22 June 2023, Judge P. Kevin Castel of the US District Court for the Southern District of New York issued the sanctions opinion (reported at 678 F. Supp. 3d 443): a $5,000 penalty, jointly and severally, on the two lawyers and their firm for submitting those six fictitious cases. The crucial detail: the judge's sanction was for the unchecked output, not for the mere use of AI itself. As the opinion put it, there is nothing inherently improper about using a reliable AI tool for assistance — but existing rules impose a gatekeeping role to ensure the accuracy of filings.

And Mata was not a one-off. A public hallucination-cases database tracked more than 1,300 court decisions globally dealing with this exact issue by April 2026, and more than 1,500 US cases by October 2026, as reported. The problem has also reached the public sector: Washington Post voter-guide tests, as reported, found that six chatbots named the wrong congressional district up to 48% of the time. When real decisions are on the line, the stakes could not be higher.

Here is the fix nobody told you: make AI look things up first.

Think of it like an exam. A closed-book exam forces the student to rely on memory — and memory fails. An open-book exam lets the student check the source before answering. AI works the same way.

This is called retrieval-augmented generation, or RAG. It was named in a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, working alongside University College London and New York University.

You already use RAG in everyday tools: the ChatGPT file-upload feature, chatting with your documents in NotebookLM, Microsoft 365 Copilot summarizing a thread, and Gemini's search grounding all work this way. They do not just guess from memory — they retrieve first. That is why RAG has become the industry-wide standard for safely chatting with your own data.

You can use this method yourself, right now:

Step 1: Upload YOUR document. Use file-upload mode, not general chat. Use a clearly labelled invented handbook example for a test — and never upload real private documents into a test you do not control.

Step 2: Ask only from that document. Say: "Answer only from the document I uploaded. Do not use outside knowledge."

Step 3: Demand the quote and the page, and check it yourself. Say: "Quote the exact passage and give me the page number." Then open the document and verify the quote yourself. If the AI cannot quote it, do not trust it.

This turns AI from a guesser into a research assistant with receipts.

RAG heavily reduces made-up answers. It never entirely eliminates them. Failures happen in two distinct ways.

A search failure: the system retrieves the wrong passage — say, an outdated employee handbook from three years ago that nobody bothered to delete — and the AI perfectly quotes the wrong answer. It inherits your messy files and delivers a beautifully, faithfully wrong answer.

A model failure: the AI straight-up ignores the chunks it was handed and blends in outside guesses from its general training data anyway.

That is why AI systems must be explicitly built to just say "I don't know" when the answer is not clearly in the text — and why human responsibility remains paramount. An unchecked guess from an AI is a massive liability, no matter how advanced the system seems.

For your daily workflow, live by this list before you trust any AI output:

AI does not lie to you on purpose. It guesses beautifully when it has nothing to check.

Give it the book — and check the page it points to — and the guessing stops. That one habit is the difference between being fooled by AI and using it well. In an age of automated answers, are you willing to be the human gatekeeper of the truth?

Watch the full Paragon AI MAG video and listen to the podcast episode on this topic. Subscribe to Paragon AI MAG for one source-checked AI story every day. Sources: Mata v. Avianca, Inc. (S.D.N.Y., Judge Castel, 22 June 2023); Lewis et al. (2020) RAG paper; NIST Generative AI Profile (AI 600-1, July 2024); Washington Post voter-guide tests as reported; SmartChunks, ToolBit, Regulayer and JustWebWorld explainers. Note: specific model hallucination rates mentioned in the video's narration are not repeated here, because they were not independently verified from the production sources.