How LLM Evaluation Actually Works: Inside the LFORLA Team Recruitment Benchmark

TL;DR: LLM evaluation only produces comparable numbers when every model faces the same prompts, the same fixed judge, and the same per-axis rubrics — and when the raw answers are public so anyone can re-read them. This post walks through how the LFORLA Team Recruitment benchmark does exactly that, using real scores from its leaderboard.

Ask five people to rank language models and you will get five different answers. Ask them why and you will usually get a vibe: it felt smarter, it wrote nicer prose, it answered faster. That is fine for a demo and useless for a decision. A benchmark earns trust only when it can explain, step by step, how a number was produced. That is the whole point of a mechanism explainer.

The LFORLA Team Recruitment benchmark, called Team Recruitment (Oracle) with the slug recruit-equipe, is a good case study because its task is concrete: a recruitment agent must query an oracle of résumés and then build a team. The output is scored on composition criteria, with binary gates for budget and seat constraints. No prose quality, no charisma — just whether the team is valid and how well it is composed.

The mechanism has four moving parts, and each one exists to remove a source of noise.

1. Same prompts. Every model receives an identical task description and an identical oracle interface. The résumés do not change between runs. If model A sees a different candidate pool than model B, the comparison is meaningless. Fixed prompts are the baseline of any credible LLM evaluation.

2. Same fixed judge. Scoring is not delegated to a different model per submission. A single fixed judge applies the rubric consistently. This matters because judge drift is one of the quietest ways a leaderboard becomes fiction: if the judge changes, the scores change, and history becomes incomparable.

3. Per-axis rubrics. The benchmark does not collapse everything into one fuzzy score. It scores composition criteria on separate axes, and it applies binary gates for budget and seat constraints. A team that exceeds the budget or fills the wrong number of seats fails the gate regardless of how elegant the composition looks. Binary gates are harsh, but they are honest: they encode hard requirements as hard requirements.

4. A public verbatim trail. Every answer is preserved so anyone can re-read what a model actually answered. This is the part most benchmarks skip. Without a verbatim trail, you are trusting the judge. With it, you can audit the judge. You can disagree with a score and point to the exact response that produced it.

Here is the current leaderboard for Team Recruitment (Oracle):

And for transparency: we submitted our own model, GLM 5.2, to this same benchmark. It scored 78.0 overall. We are publishing that number next to the leaders because a benchmark you only cite when you win is not a benchmark.

The gap between first and third is about 7.8 points. On a task with binary gates, that gap usually does not come from one catastrophic failure. It comes from how often a model clears the gates and how well it composes the team once it does. A model that occasionally overshoots the budget or miscounts seats loses points in chunks, not slivers.

The interesting result is Qwen2.5 7B Instruct at 88.46 from a local provider, sitting between two hosted free models. That is a useful reminder that LLM evaluation measures task behavior, not brand size. A smaller model that follows the oracle protocol cleanly and respects the constraints can outperform a larger one that is sloppier about gates.

GLM 5.2 at 78.0 is a real result, not a rounding error. It tells us where our model stands on this specific task under this specific judge. It does not tell us GLM 5.2 is worse at everything — it tells us it is worse at this recruitment composition task as measured here.

If you are picking a model for an agent that must satisfy hard constraints — budgets, seat counts, schema validity — a benchmark with binary gates is more informative than a general chat score. Look for the task shape that resembles yours. A model that tops a prose leaderboard may still fail your budget gate.

Also look at the spread, not just the winner. A 90.87 versus an 88.46 is close enough that cost, latency, and provider reliability should decide. A 78.0 versus 90.87 is a different conversation.

This benchmark measures one task: recruitment composition against an oracle with budget and seat gates. It does not measure reasoning breadth, multilingual ability, or long-context recall. The leaderboard is small — three models plus our own submission — so rank stability is limited. Scores are judge-dependent even with a fixed judge; a different rubric would produce different numbers. And a single overall score, like GLM 5.2 at 78.0, compresses per-axis detail that practitioners should inspect before drawing conclusions.

LLM evaluation is not a number. It is a mechanism: same prompts, same fixed judge, per-axis rubrics, and a public verbatim trail. The Team Recruitment benchmark shows what that looks like in practice, including the uncomfortable parts — like publishing our own 78.0 next to a 90.87 leader.

Explore the full leaderboard and the verbatim answers at https://lforla.org.