We quantized our AI judge. Here's exactly what broke.

Our production judge — a small 1.7B model with a LoRA adapter that grades other AI outputs as pass / fail / insufficient_evidence (88.5% accuracy, ECE 0.072) — is cheap to run. The obvious next step was quantization: serve it in int8 or int4 and cut the serving cost further.

Before flipping the switch, we did something unfashionable: we froze the acceptance criteria first — "a quantized judge may ship only if it agrees with the bf16 anchor on ≥99% of 291 held-out cases " — and published them before looking at any number.

Results (all measured, artifacts sha-anchored):

Drift grows monotonically with quantization strength. The two int8 flips and seventeen int4 flips are each a case where the judge changed its verdict — not its confidence, its answer. In a self-improving training loop, a judge flip is not a leaderboard wobble; it is wrong training data, injected silently.

Complete per-case judgment matrix, gating criteria, runner, and the pre-registered protocol: nautilus-compass on GitHub and Hugging Face. The arXiv version ships this week.

Need an independent judge for your own benchmark? Our judging lane is free: preregistered criteria, three-state verdicts, every artifact recomputable. Intake is open at nautilus.social/intake.html · live example: nautilus.social/leaderboard.html

Disclosure: this post was written by the author with AI assistance; every number in it is measured and sha-anchored in the linked artifacts.

Chunxiao — nautilus-compass, Nautilus Platform