Checking what an LLM writes costs more than the writing

Getting a model to write a page is the easy part. You give it a topic and some sources, and it gives you something that reads well. If that were the whole job, a reference site would take a weekend.

This year I built a Spanish political test, and around it a set of reference pages; one for each thinker the test is anchored on, a glossary, and an essay comparing each pair of parties. Agents drafted most of that text. The drafting was fast. Checking it is what took the work, and this post is about that part. The website is testpolitica.

If you ask a model to read a page and quote it, it reads the page and then writes the quote from memory. The result looks right and is often not literal. Sometimes it comes back translated.

I measured it a few times. Asking a small model for literal quotes, between 24% and 35% of them survived a check against the source. In the glossary, written before I knew this, 141 of 285 quotes did not appear in their own source. One entry had translated every quote into Spanish while its sources were in English, French and Russian, so nine of its ten failed.

That is, a big part of the LLM generated content on the Internet is completely wrong.

What fixed it was taking the quoting away from the model:

With that, 36 of 39 research quotes came out literal, and the thinker biographies ended with 95% of their quotes verifying against the source.

The literal check is a script: download the source, normalise both texts and search. In the first runs most of the failures were the script's fault.

Before trusting a failure, run the verifier on quotes you know are good.

A literal quote proves the text is in the source. It does not prove that it supports the sentence next to it.

The claim "the PP backed Sánchez's investiture in 2023" is false, and it still got a literal quote with a perfect score; that is, a sentence about Sánchez being appointed. So there is a second check, a model acting as judge that reads each claim with its quote and says whether it holds. A partial match only survives if the judge rewrites the claim into something the quote does support.

Later I ran a review with agents over the 26 riskiest pages. It caught things no script would... a page attributing a court case and a conviction to a living person with no source behind it, a quote about Putin whose source was about Kennedy, and an argument credited to Hayek that is Mises'. Reviewers also over-report, so every finding went to a second agent whose job was to refute it, and 45% of them fell.

Some fixes had nothing to do with facts. "Fabricated" became "started" in a sentence about a named journalist, and "the best ammunition against" became "the standard reference".

No check flags that kind of thing. Someone made of flesh has to read it.

You need to Budget for checking, as the writing is the cheap part now. Do not let the model produce what you cannot check and give it files instead of URLs, and let code extract the quotes.

I also recommende to test the verifier before you trust it and also keeping a threshold.

In many cases a shorter hand-written version should be chosen instead of an entry with fewer than four verified quotes.

Search engines are getting better at telling thin generated content apart, so this matters for ranking too.