How a GPT-5.4 harness scored above GPT-6 Astra and Claude Code on ExtractBench

On LlamaIndex ExtractBench (370 documents, 4,869 pages), my open-source harness KYNEX scores 92.97 F1. It is ranked 12th of 53 systems on the leaderboard, and the result was independently re-run and verified by the LlamaIndex team.

It runs on GPT-5.4. On the same leaderboard:

The top two are commercial APIs with their own infrastructure. A cheaper model inside a well-built harness still finished above several frontier models.

This post covers what the harness does differently and the specific failures each part fixes. My takeaway: for structured extraction, the model's access to the document mattered more than the model itself.

The failure that started it: a 26,725-row table

One benchmark document asks for every row of a large unclaimed-property table: 26,725 rows.

• One-shot LLM call: the model cannot emit an array that long. It returned null, and the score was 0.00.
• RAG: chunking separates table headers from their rows, and embeddings flatten table structure. Top-k retrieval also caps how many rows you can get back, so you can't retrieve 26,725 rows with k=10.

Neither approach is a prompt problem. Both are access problems.

The idea: give the model tools instead of retrieved chunks

KYNEX has no chunking, no vector store and no retrieval step. The document is parsed once into a page-level "world model" with a lexical index. The model then gets typed tools and decides what to read, the way an analyst flips through pages:

1. Page text extraction: read specific pages.
2. Table parsing: get rows with their headers preserved.
3. Full-text search: find where a field is mentioned.
4. OCR: for scanned pages with no text layer.
5. Vision: look at a rendered page when the layout matters.
6. Azure Document Intelligence: for complex layouts and tables.

Every tool result carries provenance (page, location, snippet), so each answer can be traced back to evidence.

Architecture

documents ──> parse once ──> WorldModel (pages, tables, lexical index)
                                  │
schema ──> per-field contracts ───┤
                                  ▼
                 Agent (LangGraph, 3 phases)  <── Supervisor
                   │ calls typed tools          (tracks progress,
                   │ (text/table/search/        resumes stalls)
                   │  OCR/vision/Azure DI)
                   ▼
           Admission gate (deterministic, outside the model)
             - numbers must appear literally in the evidence
             - text needs enough token overlap with the evidence
                   ▼
           Repair pipeline (5 rules, each with a kill switch)
                   ▼
                JSON output

Two design choices matter most here:

• The supervisor sits outside the agent loop. Long documents make agents stall or loop. The supervisor watches progress per field and resumes the agent where it stopped.
• Verification is not the model's job. A candidate answer is admitted only if a deterministic check finds it in the evidence. Numbers have to match literally. The model can propose, but it can't approve its own answers.

Five repair rules, measured one at a time

After the agent finishes, five deterministic post-processing rules run. Each one targets a failure I actually saw, each has its own environment-variable kill switch, and each was ablated on its own:

The biggest gain is page-wise enumeration. Instead of asking for the whole array at once, the harness probes two pages to learn the row shape. It then enumerates the table page by page and merges the results. The output never has to fit in one generation.

Why architecture beat model size here

Frontier models are smarter, but in a single pass they all hit the same limits:

• Output length: they can't write 26k rows in one response.
• Hallucination on missing values: redacted or empty cells get "filled in".
• Layout: scanned pages and multi-column tables.

None of these get fixed by more model intelligence. They get fixed by what the model can see, how its work is split up, and what checks its output. A harness turns those into engineering problems you can measure, ablate and switch off.

Limitations (please read before quoting the number)

• Possible leakage: the repair rules were developed while looking at failures on the same benchmark documents.
• Long documents are weak: short docs score 94.29, medium 92.48, long only 78.77. Long documents are also slow.
• Run-to-run variance is about 0.02 F1.
• Grounding metrics are 0: provenance is tracked internally but not yet exported as citation boxes. This is open as issue #5.
• The leaderboard compares systems, not raw models. Other entries may use different prompting and setups.

Used in production

The same harness runs at ABB on engineering RFQs. It produces RFQ summaries and CPQ configuration sheets from 200+ page technical documents with tens of thousands of table rows.

Try it / contribute

• Repo (Apache-2.0): https://github.com/Nikit117/kynex-harness
• LlamaIndex verification: https://github.com/run-llama/ExtractBench/pull/84
• Good first issues are open: local models (Ollama/vLLM), a 2-minute example, Docling tables, citation export, and long-document speed.

If you've worked on extraction from long, table-heavy documents: did RAG or an agentic, tool-based approach work better for you, and where did it break? I'd like to hear about the failure cases.