The Pipeline That Keeps an AI Tutor From Inventing Math Problems

Posted on Oct 21 • Originally published at automata.sale

A week ago I wrote about Space Academy, the AI homework trainer I built for my second-grader on top of his actual school textbooks. Here's the update, and it's really about one thing: what it takes to keep generated content honest when the consumer of that content is a nine-year-old who will happily learn whatever you get wrong. The numbers first: the app now covers 41 pages of two textbooks (math, Moro grade 2, pp. 4-35, and Russian, Kanakina, pp. 37-58), 199 tasks, 214 check questions. Every one of them traces back to a scan of the printed book.

The rule is boring and absolute: no task enters the database without verification against the printed textbook. A vision model transcribes each task with a narrow prompt (one task at a time, not the whole page), then a second pass compares the transcription against the scan. On mismatch, the scan wins.

The interesting decision is what happens where automatic verification can't be honest. Clock readings on a small dial, ruler measurements: the model can read the scan, but a wrong reading would poison a whole exercise chain. So those cards deliberately ship without a reference answer.

The child enters their own result, the card is marked for an adult to check, and the expected format sits in the hint. It feels like a regression until you frame it right: an unverified answer in a solver is worse than no answer.

The learn method (probe, plan, teach) is enforced by a build gate: a page without a lesson plan (a small human-readable DAG from base facts to the lesson goal) and without a motivation line for every task fails the build. No prompt injection can route around it because the check runs in CI, not in the model. Two smaller things that came out of live use: