Lint your technical drafts for AI writing tells before you publish

Posted on Oct 28 • Originally published at chainofthought.show

A draft about your retrieval layer can read like a launch announcement and still say nothing you could test. Checking the prose is cheap. Checking the engineering claim is a different job. I treat AI-writing lint the way I treat a style checker on code: fast feedback on patterns, and zero proof that the logic is right.

A large language model is trained to produce likely text, not verified text, so a polished explanation can be wrong and still read like documentation. Dan Klein, co-founder and CTO of Scaled Cognition, described the problem in his episode: older language technology often showed its mistakes, because a translation that read badly also tended to be unfaithful. A technology that always sounds right but often is not makes it hard to know what to trust.

Like I said, one of my biggest fears is that, as a field, we are building systems which produce outputs indistinguishable from the truth.

The Chain of Thought writing page makes the same point with a made-up paragraph about a retrieval layer. It marks four patterns from the open-source avoid-ai-writing catalog, and then notes that fixing them makes the paragraph read better while telling you nothing about whether the retrieval layer reduces hallucinations.

When you edit, highlight every instance. If you cannot attach something verifiable on the next pass, cut the sentence or rewrite it into a claim a reviewer could prove wrong.

Two passes on a technical post, in this order:

On a long post I would rather let a script find the pattern families first. This sketch is mine, and deliberately small; a maintained detector is the better tool for real use.

# Sketch: flag a few AI-writing tells in a markdown draft import re from pathlib import Path text = Path ( " draft.md " ). read_text ( encoding = " utf-8 " ) rules = [ ( r " \b(experts agree|studies show|researchers say)\b ", " vague attribution " ), ( r " \bnot (just|only|merely)\b[^.]*\bit '?s\b ", " contrast frame " ), ( r " \bthe future (looks|is)\b ", " generic conclusion " ), ( r " \b(watershed|sea change|in the evolution of)\b ", " significance inflation " ), ] for pattern, label in rules: for m in re. finditer ( pattern, text, re. I ): line = text. count ( " \n ", 0, m. start ()) + 1 print ( f " { label } ~ line { line }: { m. group ()[: 60 ] } " ) Enter fullscreen mode Exit fullscreen mode Treat the hits like linter warnings: review each one, because some will be false positives, and fix the real ones.

A detector only sees shape. Removing an unnamed expert does not add evidence. Replacing a claim of historic weight with a concrete before and after does, but only if the before and after are real.

That is why the writing page sits next to three episodes on the gap between plausible and true: Dan Klein on plausibility engines (episode 54), Sudhir Hasbe on hallucinations as a data architecture problem (episode 55), and Alex Ratner on the evaluation gap, where everyone says their system works and almost nobody can say what "works" means in a number (episode 57). Lint gets you to honest prose. Those conversations are about what honest prose is still allowed to claim.

If you want to see the patterns in a real draft, the essays include a piece where an AI agent's first draft runs with formatting fixes only, so you can read it for tells yourself. More about the show and how it is made is on the about page.

More on this topic, with the related episodes, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.