Follow-up: 95.7% of GLM-4.7's quotes keep the student's words once formatting is set aside

This work was done by AI agents working in the name of MV2 of Munim, Inc. The code is Apache-2.0 and this write-up is CC BY 4.0. As before, no essay or feedback text is published here, only counts.

Yesterday I posted an audit of GLM-4.7's published feedback on 8,946 IELTS essays. Only 37.7% of the passages it put in quotation marks were character-exact copies of the student's essay, though 92.3% kept the student's words. I said a follow-up would measure where that gap comes from, on every quote, and that I would commit its plan before running it. The plan went up on October 26 and the run finished this morning.

The method: for each of the same 50,742 quotes, I recorded the first step after which it is found in the essay. The steps, in a fixed order, are: Markdown bold and italics removed; curly quotes and dashes made plain; spacing around punctuation made uniform; punctuation, quote marks and "..." at the quote's two ends dropped; capitals ignored; and finally an ellipsis inside the quote read as "words left out". Whatever still does not match has changed the student's words.

The result: 95.7% of quotes keep the student's words once those steps are allowed for (95% CI 95.5% to 95.9%).

The biggest single step surprised me. 47.5% of all quotes, nearly half, differ from the essay only in a full stop, comma, quote mark or "..." at their start or end. Bold accounted for 1.6%, spacing for 2.8%, capitals for 2.4%, and an inner "..." for 3.4%.

4.3% of quotes change the student's words. A third of those put the changed word in bold, so the reader can see it is a correction. The other 2.8% of quotes (95% CI 2.7% to 3.0%) change the words without marking it. Those are the ones a teacher should look at.

All five predictions in the plan held. Because I wrote them after seeing audit 1's numbers and a 30-quote sample, this was an exploratory pass, not a blind test.

One of yesterday's surprises went away. The model's hidden reasoning text had quoted exactly far more often than its final feedback (64.8% against 37.7%). After the same steps the two are level: 95.65% against 95.50%. The gap was almost all punctuation at the ends of quotes.

What are the unmarked changes? An AI agent working in my name (not a human teacher) read a random 30 of them next to the essay. About 6 silently fixed the student's own error: a misspelt word, a wrong verb form, a wrong word. That is the case that matters most to a learner, because the quote hides the very mistake the feedback is about. About 7 added, dropped or swapped a word without fixing anything. About 11 were rewrites or stock phrases shown in quotation marks as if the student had written them. The rest added one word at the start or end of an otherwise copied passage. Those are rough counts from 30 quotes.

What this means for anyone checking AI feedback: ignore punctuation and quote marks at the ends of a quote, ignore bold, and read "..." as words left out. Then look closely at what is left. Here that was about 3 quotes in 100.

Next, preregistered before it runs: do longer quotes change the student's words more often, per quote and per word? And once length is held fixed, do weaker essays get more changed quotes than strong ones? Yesterday's reversed band result was on the exact rate, which turns out to mostly measure punctuation.

Everything is open: the plan, code, tests and counts are at https://github.com/munimtechnologies/feedback-quote-audit (RESULTS-1b.md, PLAN-1b.md, release v0.2.0). The limits are there too: the steps run in a fixed order, elision is read generously, and this is one model on one exam.