I set filmmaker traps for AI "directors." The models fell for one. My rubric fell for three.

I QC frames for photorealism and anatomy before anything becomes video. I regenerate bad frames; I do not argue with them. I also direct: corporate events, mythology films, horror shorts, stock shoots.

If AI models are going to help on a real set, "usually right" is not good enough. The bar is three things: never invent a defect, show your constraints, and write prompts as if a human crew will read them. The benchmarks I have read do not measure those jobs. So I built DirectorBench: adversarial tasks for AI models doing a video director's work.

This is a lean pilot: 24 adversarial items, two models (Anthropic Opus 5.5, Grok 4.6), both at Medium effort via Notion AI. Scoring is deterministic: regex checks plus word-overlap checks, no human judgment. Runs happened in Notion AI; the Kaggle benchmark tasks were stuck in the build queue.

Opus 5.5 scored higher on all three. Do not over-read that. On six-item tasks a single item is a large swing, and as you are about to see, two of the three tasks were partly measuring my keyword scorer. The useful question is not which model scored higher. It is what each lost point actually means.

24 items, not the planned 70. Two models, not six. Medium effort, not max. Notion AI, not a clean API harness. One run per item, no variance data.

Two of the three findings below are as much about my rubric as about the models. I would rather say that here than have someone discover it in the comments.

The QC task judges written descriptions of frames, not pixels. That is a real limitation; the full run needs actual images. Within that limit, the design is: 6 decoy descriptions that are correct by design and must be passed, plus 6 with genuine defects.

Opus passed all 6 decoys with the exact right codes. Zero false alarms, zero misses.

Grok caught all 6 real defects, then rejected a decoy: "the discus spins upright, balanced on the tip of his raised index finger." It marked it FAIL [PHYS].

That discus is the Sudarshana Chakra. The scene is iconography, and the instructions explicitly exempt iconography from physics review. I cannot see inside the model, so I will not claim to know why. What I can observe is that it applied a physics objection where the written instructions said not to.

One false alarm in six decoys is a single data point, not a personality. But it is exactly the failure the decoys exist to catch, and the one that matters most in my workflow. A rejection sends a usable frame back into regeneration. A judge that invents defects burns trust: once it flags something that is not wrong, I have to recheck its judgments myself.

This is the only finding in the pilot that is about a director's job rather than about my scorer.

ShotList is where both models scored lowest, and most of the damage came from my own checks. Each task score is the mean of six items; each item is the mean of six sub-checks. Here is where the points actually went.

Naming the exclusion gets punished. Both models lost about 14 points of ShotList score, roughly two-thirds of everything they lost, on a single sub-check: f_clean. Five of six items per model tripped it:

Grok slipped past it once: "blank cream pages, no writing, no print." That dodged the trigger word; it is not evidence of a better shot list.

The models did the right directing thing: state the constraint so the crew can see it. The checker reads the mention as the violation. That is a rubric edge, not a model failure. If your eval rewards silence over clarity, you will select for models that hide their constraints.

Natural phrasing gets punished. In PromptCraft, Grok wrote "moonlight" and "moonlit" instead of "moon," and "candlelight" instead of "candle." The noun scorer wanted a whole-word match on "moon"; "moonlight" does not count. Opus wrote the bare nouns and went 6 for 6.

That does not prove Opus wrote better prompts. It proves this slice of the score measured lexical form, not prompt quality. When a benchmark tells you a model writes bad prompts, check whether it is grading craft or grep patterns.

Paraphrase scores zero. The b04 ShotList item carried a brand promise: the call is heard but never printed. Both models landed the intent in my reading. Opus wrote "coach calls out in Hindi, stopwatch face turned away." Grok wrote "Split is heard, not seen." The word-overlap sub-check scored both 0.0; it cannot see paraphrase. Those zeros sit inside the ShotList averages in the table above.

The full run is 70 items and six models, with two fixes this pilot earned: compliance scored by meaning instead of keyword presence, and QC judged on actual frames instead of descriptions. The test of DirectorBench is whether it can tell a model that breaks a rule from one that explains it, and a model that invents a defect from one that knows when the rules do not apply.

A benchmark should distinguish a bad answer from a good answer expressed differently. This pilot showed me where mine does not.

Items, answer keys, scorers: https://github.com/rsaxena3011/directorbench