A small regression harness for citation-safe LLM rewriting

LLM rewriting can make a paragraph read more smoothly while changing what it claims. If you build a writing assistant, evaluate preservation separately from fluency.

I work on PaperHumanizer, an academic rewriting tool. This post describes a small evaluation pattern you can adapt. It is an illustrative harness, not a claim that every rewrite passes it.

In a pilot involving 42 participants, the intervention was associated with a 12% reduction in response time (Lee, 2025). The study did not establish causation.

A revision can change sentence rhythm. It should not silently change 42 to 420, 12% to 21%, or association to causation. Keeping the citation string is necessary but does not show that the surrounding claim still matches the source.

const fixture = { required: [ ' 42 ', ' 12% ', ' (Lee, 2025) ' ], forbidden: [ ' proved that ', ' established causation ' ], }; function checkRevision ( text, fixture ) { const missing = fixture. required. filter ( s =>! text. includes ( s )); const introduced = fixture. forbidden. filter ( s => text. includes ( s )); return { missing, introduced, needsReview: missing. length > 0 || introduced. length > 0 }; } const bad = ' The intervention proved that response time fell by 21% (Lee, 2025). '; console. log ( checkRevision ( bad, fixture )); // missing: ['42', '12%']; introduced: ['proved that']; needsReview: true Enter fullscreen mode Exit fullscreen mode Write fixtures by hand for your examples. They can cover sample sizes, units, percentages, author-year references and domain terms. Avoid treating a generic regular expression as a complete citation parser: narrative references, numbered references and multiple studies in one parenthesis need different handling.

A test suite that accepts the original paragraph proves little. Add deliberately broken outputs:

The substring check catches only some of these. In particular, it will miss a removed limitation unless the fixture explicitly requires it. It may also flag a harmless spelling or formatting change. Its purpose is to route a revision to review, not to certify accuracy.

For each paragraph, compare the source with the revision and ask:

A clean automated result means only that the chosen invariants survived. Readability, factual consistency and source support still need review. A detector score does not answer those questions.

For an example workflow, PaperHumanizer's academic editor includes an original/rewrite comparison view, academic tone choices and text-file export. Rewriting requires sign-in and credits. Review the result against your sources and follow your institution's AI-use rules.

What preservation failures would you put in a fixture before shipping an LLM rewriting feature?

Disclosure: I work on PaperHumanizer. This article was prepared with AI assistance and reviewed before publication.