Source-linked AI summary
ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction
Nick Ferguson, Josh Pennington, Narek Beghian, Aravind Mohan, Douwe Kiela, Sheshansh Agrawal, Thien Hang Nguyen
TL;DR
ExtractBench addresses missing end-to-end benchmarks and principled evaluation methods for PDF-to-JSON extraction with enterprise-scale schemas. It introduces a schema-driven benchmark and evaluation framework, finding that frontier-model reliability degrades sharply with schema breadth and reaches 0% valid output on a 369-field financial reporting schema.
Problem
Existing benchmarks do not jointly evaluate PDF extraction, enterprise-scale JSON schemas, and fine-grained key-level correctness, while existing methodologies treat nested fields and arrays uniformly.
Method
ExtractBench pairs 35 PDF documents with JSON Schemas and human-annotated gold JSON, using schema-driven per-field metrics, semantic array alignment, and explicit omission-versus-hallucination handling.
Results
Frontier-model performance degrades sharply as schema breadth increases, culminating in 0% valid output on a 369-field financial reporting schema across all tested models.
Takeaways & Limitations
Realistic enterprise-scale schemas expose reliability failures that benchmarks limited to small schemas can hide, making complexity coverage central to structured-extraction evaluation.
Takeaways & Limitations
Constrained decoding may fail on very large schemas because grammar complexity and rigid structural enforcement can exceed provider limits or reduce usable content-reasoning capacity.
Abstract
from arXiv · showhide
Unstructured documents like PDFs contain valuable structured information, but downstream systems require this data in reliable, standardized formats. LLMs are increasingly deployed to automate this extraction, making accuracy and reliability paramount. However, progress is bottlenecked by two gaps. First, no end-to-end benchmark evaluates PDF-to-JSON extraction under enterprise-scale schema breadth. Second, no principled methodology captures the semantics of nested extraction, where fields demand different notions of correctness (exact match for identifiers, tolerance for quantities, semantic equivalence for names), arrays require alignment, and omission must be distinguished from hallucination. We address both gaps with ExtractBench, an open-source benchmark and evaluation framework for PDF-to-JSON structured extraction. The benchmark pairs 35 PDF documents with JSON Schemas and human-annotated gold labels across economically valuable domains, yielding 12,867 evaluatable fields spanning schema complexities from tens to hundreds of fields. The evaluation framework treats the schema as an executable specification: each field declares its scoring metric. Baseline evaluations reveal that frontier models (GPT-5/5.2, Gemini-3 Flash/Pro, Claude 4.5 Opus/Sonnet) remain unreliable on realistic schemas. Performance degrades sharply with schema breadth, culminating in 0% valid output on a 369-field financial reporting schema across all tested models. We release ExtractBench at https://github.com/ContextualAI/extract-bench.
1 Introduction
ExtractBench addresses missing end-to-end evaluation for PDF-to-JSON extraction with enterprise-scale schemas and field-specific correctness. Its benchmark and schema-driven framework show that frontier models remain unreliable as structured output demands grow.
- Research gaps: Existing benchmarks do not jointly evaluate PDF extraction, enterprise-scale JSON schemas, and fine-grained key-level correctness.They separately emphasize span extraction, schema conformance, or modest-schema nested extraction.
- Methodological contribution: The evaluation framework assigns field-specific metrics, including exact matching for identifiers, numeric tolerance for quantities, and semantic equivalence for free text.It also supports semantic array matching and distinguishes missing fields from explicit nulls.
- Baseline findings: 0% valid output occurred on a 369-field financial reporting schema across all tested frontier models.The result exposes failures in generating long, deeply structured JSON at enterprise scale.
- Baseline findings: 90% valid output yielded only a 12.5% pass rate in one domain, showing that syntactic validity does not ensure correct extraction.Structural correctness can remain poor even when the output parses successfully.
- Contributions: ExtractBench contributes a multi-domain dataset, schema-driven evaluation, frontier-model baselines, and complexity analysis identifying schema breadth as a dominant reliability predictor.The dataset and evaluation framework are released openly.
2 The ExtractBench Dataset
ExtractBench is a quality-over-quantity diagnostic benchmark designed to isolate how schema breadth, output volume, document length, nesting depth, and arrays affect structured extraction. Its curated domains show that schema breadth and output volume, rather than document length or nesting depth, dominate failure.
- Dataset design: 35 documents across five domains, 2,076 pages, and 12,867 evaluatable fields form a diagnostic benchmark focused on complexity coverage.The dataset prioritizes systematic failure diagnosis over scale, with expert-created gold annotations.
- Complexity dimensions: 369-field SEC 10-K/Q schemas produce 0% valid output across frontier models, while depth-6 sports schemas reach 90% validity.These contrasting results show that nesting depth alone is a weak failure predictor.
- Complexity dimensions: Research papers produce approximately 25k output tokens from only 16 schema fields, whereas credit agreements require about 0.9k tokens and achieve higher success.Output volume, rather than input length, is identified as the binding constraint in this comparison.
- Key finding: Schema breadth and output volume—not document length or nesting depth—are the dominant predictors of extraction failure.The benchmark deliberately curates domains to expose production-relevant failure modes.
- Complexity dimensions: The benchmark varies schema breadth, depth, and array complexity as orthogonal dimensions that interact across domains.This design enables diagnosis of which complexity factors drive failure.
3 A Principled Evaluation Methodology
ExtractBench treats the JSON Schema as an executable specification for evaluating complex document extractions with field-specific semantics, recursive traversal, and explicit error handling. The methodology supports heterogeneous nested fields, arrays, missing values, and extensible metrics.
- 3.1 The Evaluation Challenge: Realistic enterprise schemas combine heterogeneous fields, nested structures, and arrays whose correctness cannot be captured by one global metric.Exact matching, numeric tolerance, semantic equivalence, and order-insensitive array evaluation may be required within one schema.
- 3.1 The Evaluation Challenge: Schema-driven evaluation distinguishes structural errors, omissions, hallucinations, and inaccurate extracted values.The framework separates missing existing information from fabricated information absent from the source.
- 3.2 Schema-Driven Evaluation: Each schema node carries an evaluation configuration, and AST-based dual traversal recursively scores paired gold and predicted JSON instances.Local node results are aggregated into global scores over the schema tree.
- 3.2 Schema-Driven Evaluation: The metric library supports exact, case-insensitive, fuzzy, semantic, tolerance-based numeric, and semantic array comparisons selected through field-level configuration.Domain experts can encode additional equivalences and add metrics through a plugin registry without changing core evaluation logic.
- 3.2 Schema-Driven Evaluation: Semantic array alignment matches predicted and gold items before recursively evaluating their contents, while identifying missed and spurious items.The matcher handles partial matches, duplicates, empty arrays, and large arrays processed in batches.
- 3.3 Extensible Infrastructure: The framework is released as extensible open-source infrastructure designed for reusable benchmarking beyond a one-off scoring script.Its plugin interface lets new metrics register without modifying core traversal logic.
4 Experiments
Across ExtractBench, frontier models were unreliable on realistic PDF-to-JSON tasks: performance varied by model and domain, degraded with output complexity, and remained weak even when JSON was valid. Structured output APIs did not improve the benchmark results.
- Main Results: 51% valid JSON and a 4.6% aggregate pass rate were achieved across 210 extraction attempts, while Gemini 3 Flash reached the best pass rate at 6.9%.Excluding SEC 10-K/Q filings, the pass rate rose to 28.0% (844/3,018 fields).
- Main Results: 0% valid output occurred for all seven SEC 10-K/Q documents, whose 369-field schema accounted for 84% of all field evaluations.The schema requires approximately 24,400 structured-JSON tokens per document.
- Main Results: When JSON was valid, aggregate field-level accuracy was 72.9%, but this valid-only result reflects a biased sample dominated by easier extractions.Valid-only accuracy ranged from 65% to 80%, while the aggregate end-to-end pass rate was 4.6%.
- Complexity Analysis: Schema breadth and output volume strongly constrained performance: 13-field credit agreements reached a 56.3% pass rate, whereas 369-field 10-K/Q filings reached 0% validity.GPT-5 had 100% validity on credit agreements but only 0–14% elsewhere, where outputs required 3k–25k tokens.
- Complexity Analysis: Array complexity produced failures beyond schema size: research papers had 39% validity despite 16 keys, while sports results had 90% validity but only a 12.5% pass rate.Research papers averaged 309.5 array items and 25,366 gold tokens; sports results were penalized for structural discrepancies in nested athlete records.
- Structured Output Mode: Structured output APIs reduced overall validity from 51% to 37% and the best-model pass rate from 6.9% to 5.5%.Provider schema restrictions and constrained decoding introduced additional failure modes on complex schemas.
5 Conclusion
ExtractBench combines a broad PDF-to-JSON benchmark with schema-driven evaluation and shows that frontier models remain unreliable on realistic enterprise schemas. The benchmark is released openly and supports further research on scaling complex extraction.
- 5 Conclusion: ExtractBench spans PDF-to-JSON tasks from 13 to 369 fields and evaluates per-field semantics, omission versus hallucination, and nested-array alignment.The dataset pairs PDFs, JSON Schemas, and gold JSON labels; the framework assigns field-specific evaluation behavior.
- 5 Conclusion: 0% valid output occurs on 369-field financial reports, with performance degrading sharply as schema breadth increases.The conclusion identifies output volume, rather than input length or nesting depth, as the dominant failure predictor.
- 5 Conclusion: The released benchmark supports research on constrained decoding, schema decomposition, retrieval, and agentic context management for long complex documents.These directions are proposed alongside the open-source release of the dataset and evaluation framework.
A Ethical Considerations
The benchmark uses public-domain or synthetic documents and contains no proprietary or restricted-access documents. Its ethical scope is limited by English-only, US-centric financial data, five domains, a small dataset, and three annotators without reported agreement statistics.
- A Ethical Considerations: ExtractBench uses public-domain or synthetically generated documents, with no real personally identifiable information in synthetic resumes.Sources include SEC filings, open-access research papers, public sports results, and synthetic resumes.
- A Ethical Considerations: No proprietary or restricted-access documents are included.
- A Ethical Considerations: All documents are in English, and financial documents are US-centric because they reflect SEC filing requirements.
- A Ethical Considerations: The dataset covers only five domains and 35 documents, prioritizing diagnostic quality over statistical power across fine-grained subgroups.
- A Ethical Considerations: Gold annotations were created by three annotators, with review for conformance and accuracy but no reported inter-annotator agreement statistics.
B Additional Tables and Figures
Supplementary tables provide performance breakdowns by document domain, while a separate table reports per-document token statistics. The listed tables cover SEC filings, credit agreements, resumes, and research papers.
- B Additional Tables and Figures: Tables 9–13 provide per-domain model-performance breakdowns that complement aggregate results in Table 5.
- B Additional Tables and Figures: Table 9 covers SEC 10-K/Q filings across 7 documents.
- B Additional Tables and Figures: Table 10 covers credit agreements across 10 documents.
- B Additional Tables and Figures: Table 11 covers professional resumes across 7 documents.
- B Additional Tables and Figures: Table 12 covers research papers across 6 documents.
C Evaluation Configs
ExtractBench uses field-level evaluation configurations so different value types receive appropriate comparison rules. Its configurations include exact, tolerant, fuzzy, semantic, and array-alignment metrics.
- C Evaluation Configs: Each schema field specifies an evaluation_config that determines how extracted values are compared with gold annotations.
- C Evaluation Configs: string_exact performs exact matching for timestamps and codes.
- C Evaluation Configs: string_case_insensitive performs case-folded matching for units and enums.
- C Evaluation Configs: string_fuzzy uses Levenshtein similarity for entity names.
- C Evaluation Configs: string_semantic uses LLM-based semantic equivalence for free-text fields.
- C Evaluation Configs: integer_exact and number_exact require exact matches for years, counts, and monetary amounts, while number_tolerance allows relative numeric deviation for reported financial values.
- C Evaluation Configs: boolean_exact requires exact boolean matching, while array_llm performs LLM-based semantic alignment for variable-length lists.
D Extraction Prompt
The extraction prompt instructs the model to use a JSON template to extract required information and return only schema-conforming JSON. The section also defines token-statistics terminology for evaluating document processing.
- The baseline prompt directs models to extract all required information from the named document using the JSON template as a guideline.
- The prompt requires output to be valid JSON conforming to the schema, without explanatory text before or after it.
- Table 14 defines input tokens as Vision tokens representing the raw image and defines compression ratio as Input ÷ Output.