Testing AI workflow automation on a tender: a cite check with a planted answer key
This post walks through how to test AI workflow automation on a document check before you trust it, so that by the end you can rebuild the test on your own documents: invent a document with numbered claims that cite sources, plant errors you know about, send one fixed request through one fixed tool
This post walks through how to test AI workflow automation on a document check before you trust it, so that by the end you can rebuild the test on your own documents: invent a document with numbered claims that cite sources, plant errors you know about, send one fixed request through one fixed tool, score each answer against the key by location and by evidence, and recompute every number with a second script that shares no code with the first. Everything below comes from the real files, and every code block is trimmed from them. The trigger was Anthropic's post [Introducing artifacts for legal work]( https://claude.com/resources/articles/introducing-artifacts-for-legal-work ) from 28 October.
One of the six artifacts in its legal gallery is a cite check that, in Anthropic's words, compares a legal document with its sources and flags quotes that don't match, wrong page numbers and sources that don't support the claim. It is built for lawyers. A sales team's version of that document is the tender answer, so I had Claude Code build the same kind of check as a skill, plus a test harness around it, and a GitHub workflow made the API calls.
I did not test Anthropic's artifact. ## Why test AI workflow automation against a planted answer key? Because without a key you are grading the model on how confident it sounds.
A document check returns a list of flags, and a list of flags looks like a result whether or not anything was checked. If you plant the errors yourself, you know exactly which claims should be flagged, which should not, and what a correct reason looks like. That turns "it seemed to find stuff" into caught, missed and false alarm. ## The pack: one tender, five annexes, ten errors pack.py\ writes pack.json\.
An invented software company, Lindholt Systems, answers a tender for a field-service planning system from an invented Danish utility. There are five sources: the customer's requirement list (K1 to K10), a product sheet, an SLA annex, a security annex and a reference case. Section 4 of the tender answer makes 26 numbered claims, 4.1 to 4.26, each citing one passage.
Ten claims carry a planted error, split across the three kinds Anthropic names: | Kind | Count | Example | |---|---|---| | figure or quote does not match | 4 | 99.9% uptime where the SLA says 99.5% | | wrong reference | 2 | the map "meets K5", but K5 is uptime; the map is K4 | | source does not support the claim | 4 | "ISO 27001 certified" where the annex says certification is in progress | The other 16 claims are correct and must not be flagged. Two of them sit beside a trap on purpose: 4.19 carries the 90 days that 4.18 gets wrong, so a check that flags every number gets caught out. Each claim is a small record.
Trimmed from pack.json\, the one that matters most later: \ json { "n": 17, "en": "Users sign in with single sign-on through Microsoft Entra ID, and administrators must use two-step verification.", "cite\_en": "Security annex 3.2", "trap": { "kind": "henvisning", "kind\_en": "Wrong reference", "why\_en": "True, but it is in 4.1. Section 3.2 is the UK subprocessor." } } \ \ pack.py\ also refuses to build a broken pack. Every cite has to resolve to a passage that exists, and the counts are pinned, so a trap cannot quietly disappear while someone edits a sentence: \ python def check(pack): """The pack must hang together before a single call is spent on it.""" refs = {(s\["id"\], p\["ref"\]) for s in pack\["sources"\] for p in s\["passages"\]} for c in pack\["claims"\]: assert (c\["cite"\]\["source"\], c\["cite"\]\["ref"\]) in refs, f"claim {c\['n'\]} cites a passage that does not exist" assert pack\["counts"\] == {"claims": 26, "traps": 10, "clean": 16, "by\_kind": {"tal": 4, "henvisning": 2, "stoette": 4}}, pack\["counts"\] \ \ ## One request, one tool: api_job.py The request is identical in every run, in Danish, from an invented sales lead: here is section 4 and the annexes it cites, it goes out Friday at 12, can you read it through and say what needs fixing before we send?
Nothing in it mentions citations. The model gets one tool, submit\_review\, and the tool does not name citation errors either: each issue is typed as content, reference, language or other. Trimmed from api\_job.py\, the parts of the schema that the scorer reads: \ python TOOL = { "name": "submit\_review", "input\_schema": { "type": "object", "properties": { "issues": {"type": "array", "items": { "type": "object", "properties": { "punkt": {"type": "string"}, # the claim number, e.g. "4.8" "skal\_rettes": {"type": "boolean"}, # must be fixed before sending "type": {"type": "string", "enum": \["indhold", "henvisning", "sprog", "andet"\]}, "reason\_en": {"type": "string"}, "kildens\_ordlyd": {"type": "string"}, # the source's own words "correction\_en": {"type": "string"}, }, "required": \["punkt", "skal\_rettes", "type", "reason\_en"\]}}, }, "required": \["issues"\], }, } \ \ There are six conditions, six runs each, one call per run: Claude Sonnet 5.5 and Claude Haiku 5.5, each given the tender alone, the tender with its annexes, or the tender with its annexes and the cite-check skill in the system prompt.
A fake client ( CITE\_FAKE=1\ ) runs the whole job offline first, so the scorer is exercised before a single paid call is made. ## The first run failed on the plumbing The first version of the job is kept in the repo, and it measured the harness, not the model. Two faults, both the builder's: - It forced the tool call with tool\_choice\. Claude Sonnet 5.5 does not take a forced tool call, so all 12 Sonnet runs came back as a 400 error and no Sonnet review exists from that run. - The tool had no way to say "checked, no error".
Haiku listed every claim, most with a note that nothing was wrong, and the scorer read each note as a flag: 16 of 16 correct claims "flagged" in most runs. The fix changed only the plumbing. The model now calls the tool itself, with at most two short nudges if it does not, and every issue carries the required skal\_rettes\ flag.
An issue with that flag set to false counts as a note and never as a flag. The request, the documents, the skill and the conditions stayed as they were. None of the first run's numbers are quoted anywhere. ## Scoring: location decides, evidence decides the reason score.py\ was written before the paid run.
An issue counts as caught when it lands on a trap claim with a type other than language. It counts as caught for the right reason only if its text contains one of the trap's evidence strings: the true figure, the right section, or a word that shows the source was read. Trimmed and simplified (the counters for notes and language remarks are left out): \ python EVIDENCE = { 6: \["k4"\], 8: \["99,5", "99.5"\], 13: \["2027", "not certified", "working towards", "in progress", "not yet"\], 17: \["4.1"\], 18: \["35"\], } def score\_run(rec, traps, clean): issues = (rec.get("review") or {}).get("issues") or \[\] per = defaultdict(list) for it in issues: ns = claim\_nos(it.get("punkt")) if it.get("skal\_rettes") is False: # checked and commented, not a flag continue if it.get("type") == "sprog" or not ns: continue for n in ns: per\[n\].append(it) caught = sorted(n for n in traps if per.get(n)) right = sorted(n for n in caught if any(any(e in text\_of(i) for e in EVIDENCE\[n\]) for i in per\[n\])) false = sorted(n for n in clean if per.get(n)) return {"caught": caught, "right\_reason": right, "false\_alarms": false} \ \ Two rules in claim\_nos\ were settled while reading real answers and are written into its docstring: an issue that names two claims ("4.8 og 4.9") counts for both, and an issue that names a range ("4.1-4.26") is a remark about the section and counts for none. ## The condition added after the results The pre-registered conditions were the tender with annexes, with and without the skill. When those came back, the skill had made almost no difference, which was not the expected story.
So one condition was added after seeing the results, and it is labelled that way everywhere it is quoted: the same request with the tender alone, the way a sales team most often asks. It ran six times per model through the same tool and the same base system prompt. That ordering matters for how you read the numbers.
The tender-alone result is real, but it was not planned before the run. ## What the numbers came back as Caught, out of 60 (ten traps times six runs): | Condition | Sonnet 5.5 | Haiku 5.5 | |---|---|---| | tender alone (added after) | 0 | 13, only 3 for the right reason | | tender and annexes | 58 | 54 | | annexes and the skill | 59 | 54 | With the annexes, every run caught at least nine of ten, and there was one false alarm in 24 runs: Sonnet asked for a source for the figure of 60 technicians in claim 4.1. With the tender alone, every Sonnet run wrote that it could not check without the sources, and it did flag one correct claim, 4.25, in four of its six runs. Every miss with the annexes was the same claim, 4.17: a true claim citing the wrong section.
Haiku caught it 0 times in 12, Sonnet 9 times in 12. The skill tells the model to treat a claim found in the wrong section as a wrong reference even when it is true, and Haiku still read past it with the skill installed. ## The cite-check SKILL.md The skill was written and frozen before the run.
It is generic: it names kinds of error and never these claims. Trimmed from kit/cite-check/SKILL.md\: \ markdown 2. \*\*Look up the source, for every claim.\*\* Find exactly the section it cites. Quote the source's own words.
Never take a claim as right because it sounds right, or because it agrees with another claim in the document. - If the section does not exist, or is about something else, it is a \*\*wrong reference\*\*, even when the claim itself is true. Search the other sources and say where it actually is. 5. \*\*Give each claim one of four verdicts:\*\* - \*\*matches\*\*: the source says the same thing, to the same extent - \*\*figure or quote does not match\*\*: the source says something else - \*\*wrong reference\*\*: the claim is somewhere else, or the requirement number is wrong - \*\*source does not support the claim\*\*: the source says less than the claim promises \ \ With the annexes attached it moved Sonnet from 58 to 59 and Haiku not at all.
On one invented tender with fairly clear errors, the annexes did the work. That says little about skills in general, and the write-up does not stretch it further. ## verify.py recomputes everything verify.py\ re-derives every quoted number from the raw run log and compares it with facts.json\. It shares no code with score.py\: its own parser for the claim number, its own copy of the evidence strings typed in again rather than imported, its own arithmetic.
It exits 1 on any difference. It also has a selftest, because a checker that has never failed proves nothing: \ text python3 verify.py re-derives from the run log and compares with facts.json python3 verify.py --selftest plants four errors in a copy of facts.json and must catch all four \ \ ## What it cost Prices from the Claude Platform pricing page on 29 October: Sonnet 5.5 at $2 per million input tokens and $10 per million output, Haiku 5.5 at $0.10 and $0.50 for prompts up to 100,000 tokens. Every run here was far under that.
A check with the annexes cost about 7.8 US cents with Sonnet and 0.38 cents with Haiku, and took about 40 and 27 seconds. The 36 runs that are quoted cost $1.30 together. The failed first run cost a few cents more, all of it Haiku, and is reported separately in the build notes. ## What the test doesn't show It is one invented pack, in Danish, and the same setup that planted the errors also wrote the skill, so there is no held-out tender.
Six runs per condition show spread, not a rate to quote with decimals. The check can only compare the tender with what it is given: if an annex is itself wrong, the claim that repeats it passes. And no person was involved, so it says nothing about what a bid team does with the answer. ## Rebuild it on your own tenders Take a tender you have already sent, write down every claim that cites something, and plant a handful of errors you know about, including at least one true claim with a wrong reference.
Send the same request every time, give the model a tool with a must-fix flag so a note cannot pass as a flag, score by location and by evidence, and recompute with a second script before you quote anything. You can [get the free test pack with the cite-check skill and the invented tender]( https://brinvik.com/en/resources/proposal-cite-check?utm_source=devto&utm_medium=article&utm_campaign=2026-10-29-claude-kildetjek&utm_content=devto-pack): the skill in English and Danish, all five annexes, the answer key and the scoring script. To see the runs without installing anything, the [interactive demo of every claim looked up in its source]( https://brinvik.com/demos/proposal-cite-check/index.html?utm_source=devto&utm_medium=article&utm_campaign=2026-10-29-claude-kildetjek&utm_content=devto-demo ) lets you pick any of the 36 runs and click a claim to read Claude's own reason.
There is also [a short film of the check]( https://brinvik.com/claude-news/2026-10-29-claude-kildetjek/brinvik-kildetjek-film-en-1080x1080.mp4?utm_source=devto&utm_medium=article&utm_campaign=2026-10-29-claude-kildetjek&utm_content=devto-film): The full write-up is [the English article on checking a tender against its annexes with Claude]( https://brinvik.com/en/journal/claude-ai-workflow-automation-tender-cite-check?utm_source=devto&utm_medium=article&utm_campaign=2026-10-29-claude-kildetjek&utm_content=devto-en), and there's [a Danish version of the article on tenders and annexes]( https://brinvik.com/da/journal/claude-ai-automatisering-tilbud-kildetjek?utm_source=devto&utm_medium=article&utm_campaign=2026-10-29-claude-kildetjek&utm_content=devto-da). I'm Kim Olsen, a one-person AI consultant in Copenhagen. If your sales team wants Claude checking tenders, I set it up where the tenders are written, with the annexes in reach, and keep a person on the references at the end.
That's part of my [work on internal AI tools for sales teams]( https://brinvik.com/en/services/internal-ai-tools?utm_source=devto&utm_medium=article&utm_campaign=2026-10-29-claude-kildetjek&utm_content=devto-service).