LLM-as-judge, responsibly: a checklist before you trust an AI to grade anything

"LLM-as-judge" has quietly become infrastructure. We use one model to grade another's answers, to decide if a test really passed, to score RAG relevance, to moderate, to rank. It's cheap, fast, and scales to volumes no human can touch.

It's also a measurement instrument — and most teams deploy it without ever checking whether the instrument is accurate. Here's a practical checklist before you let an LLM grade anything that matters. (Context: I built a small harness that runs a local LLM as a test judge and scores it against ground truth; some numbers below come from that.)

You cannot manage what you haven't measured. Assemble a set of cases where you know the right verdict (labelled correct/wrong) and run the judge over them. You're looking for:

In my run, a capable open model missed 26% of wrong outputs — a quarter of real failures, passed. You want your number, on your task, before you rely on it.

Ask the judge for a confidence, then check whether that confidence means anything. A well-calibrated judge that says "90% sure" should be right about 90% of the time. Measure Expected Calibration Error (ECE).

The common failure: models are massively overconfident. Mine answered with ~100% confidence on nearly every call — including the wrong ones (ECE ≈ 0.31). If the confidence is always "very sure," it carries no signal and you can't use it to route hard cases to a human.

An LLM judge whose model, prompt, or temperature drifts is a ruler that changes length. Pin:

Open-weight models help here: you can pin a local model and know it won't change under you. A hosted endpoint can be swapped silently.

LLM judges have documented, reproducible biases:

If your eval drives a leaderboard or a release decision, these biases become your biases.

The safe pattern isn't "judge decides, ship." It's:

An LLM judge is a fast, cheap, biased, overconfident grader that is right often enough to be useful and wrong often enough to be dangerous. Measure it, pin it, debias it, and keep a human backstop — then it's a great tool. Trust it blindly and it will quietly approve your worst bug with total confidence.

Are you using LLM-as-judge in production? What did you find when you checked it against ground truth — or have you checked yet?

I'm Krishnasai Rollamadugu — a test automation architect researching trust and measurement in AI-driven software testing. Related: an open experiment measuring a local LLM judge — trust-judge.