How a GPT-5.4 harness scored above GPT-6 Astra and Claude Code on ExtractBench

On LlamaIndex ExtractBench (370 documents, 4,869 pages), my open-source harness KYNEX scores 92.97 F1. It is ranked 12th of 53 systems on the leaderboard, and the result was independently re-run and verified by the LlamaIndex team.

It runs on GPT-5.4. On the same leaderboard:

The top two are commercial APIs with their own infrastructure. A cheaper model inside a well-built harness still finished above several frontier models.

This post covers what the harness does differently and the specific failures each part fixes. My takeaway: for structured extraction, the model's access to the document mattered more than the model itself.

One benchmark document asks for every row of a large unclaimed-property table: 26,725 rows.

Neither approach is a prompt problem. Both are access problems.

KYNEX has no chunking, no vector store and no retrieval step. The document is parsed once into a page-level "world model" with a lexical index. The model then gets typed tools and decides what to read, the way an analyst flips through pages:

Every tool result carries provenance (page, location, snippet), so each answer can be traced back to evidence.

documents ──> parse once ──> WorldModel (pages, tables, lexical index) │ schema ──> per-field contracts ───┤ ▼ Agent (LangGraph, 3 phases) <── Supervisor │ calls typed tools (tracks progress, │ (text/table/search/ resumes stalls) │ OCR/vision/Azure DI) ▼ Admission gate (deterministic, outside the model) - numbers must appear literally in the evidence - text needs enough token overlap with the evidence ▼ Repair pipeline (5 rules, each with a kill switch) ▼ JSON output Enter fullscreen mode Exit fullscreen mode Two design choices matter most here:

After the agent finishes, five deterministic post-processing rules run. Each one targets a failure I actually saw, each has its own environment-variable kill switch, and each was ablated on its own:

The biggest gain is page-wise enumeration. Instead of asking for the whole array at once, the harness probes two pages to learn the row shape. It then enumerates the table page by page and merges the results. The output never has to fit in one generation.

Frontier models are smarter, but in a single pass they all hit the same limits:

None of these get fixed by more model intelligence. They get fixed by what the model can see, how its work is split up, and what checks its output. A harness turns those into engineering problems you can measure, ablate and switch off.

The same harness runs at ABB on engineering RFQs. It produces RFQ summaries and CPQ configuration sheets from 200+ page technical documents with tens of thousands of table rows.

If you've worked on extraction from long, table-heavy documents: did RAG or an agentic, tool-based approach work better for you, and where did it break? I'd like to hear about the failure cases.

Expand Nikit Kashyap Nikit Kashyap Nikit Kashyap Follow Joined Nov 4, 2026 • Nov 4 Dropdown menu Copy link Hide Repo (Apache-2.0): github.com/Nikit117/kynex-harness. LlamaIndex verification: github.com/run-llama/ExtractBench/...