PaperlayerView exact revision on arXiv

Source-linked AI summary

ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

Boyang Zhang, Adrian Lyjak, Eli Stewart, Zhaoqi Li, Simon Suo

arXiv:2607.29677v2cs.AI

TL;DR

Enterprise extraction needs outputs that are accurate, complete, traceable, and affordable, but existing evaluations do not measure these requirements together at scale. ExtractBench evaluates schema-guided extraction across diverse enterprise documents and finds that LlamaExtract Agentic Plus delivers the strongest overall performance at lower cost than coding agents.

  • Problem

    Enterprise document extraction requires accurate, complete, traceable, and cost-effective outputs, but existing evaluations lack broad evidence measuring these requirements together at scale.

  • Method

    ExtractBench benchmarks schema-guided extraction across 370 documents and 4,869 pages, using challenge tags, multiple grounding metrics, cost measurement, and scalable ground-truth curation.

  • Results

    LlamaExtract Agentic Plus achieves the strongest overall performance at lower cost than coding agents, while direct VLMs often truncate long record lists.

  • Takeaways & Limitations

    Reliable enterprise extraction requires jointly evaluating output completeness, source traceability, robustness across document challenges, and sustainable cost at scale.

  • Takeaways & Limitations

    Grounding remains limited: word-level grounding F1 reaches only 46.4% even for specialized systems that return source boxes.

Abstract

from arXiv · show

Enterprise workflows increasingly rely on agents for \emph{schema-guided extraction}: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a benchmark for schema-guided extraction and, to our knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together. The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with clear tags differentiating their challenge scenarios. The scalable schema and ground-truth curation pipeline combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. We report order-insensitive value F1 for value accuracy, plus two grounding metrics for source traceability: word- and page-level F1. Commercial VLMs perform well on short documents but often truncate record lists on long ones, while coding agents retain higher accuracy at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost. Dataset and evaluation code are available on \href{https://huggingface.co/datasets/llamaindex/ExtractBench}{HuggingFace} and \href{https://github.com/run-llama/ExtractBench}{GitHub}.

1 Introduction

ExtractBench addresses the need for schema-guided enterprise document extraction by evaluating accuracy, completeness, grounding, and cost across diverse real-world documents and challenge types. It also compares 14 extraction systems, finding strong long-document and cost performance from LlamaExtract Agentic Plus.

  • Motivation: Schema-guided extraction lets enterprises define one schema per document type and require source evidence as grounding metadata.The schema is shared across documents of the same type, despite vendor or layout variation.
  • Challenges: Enterprise extraction fails through missing long-list rows, wrong sparse-fact occurrences, overfilled forms, confused values, and perception errors from scans, handwriting, or complex structure.These failures motivate challenge tags covering task and document perception conditions.
  • Benchmark: ExtractBench evaluates 370 documents and 4,869 pages across 8 business domains and 67 document types, tagged by task challenge, perception challenge, table structure, domain, and length.The benchmark jointly evaluates long-record completeness, real scans and handwriting, word- and page-level grounding, and measured cost.
  • Method: The benchmark’s curation pipeline combines frontier-model ensembles for real documents, programmatic generation for synthetic long lists, and human verification for scanned forms.This pipeline produces schema–ground-truth pairs without labeling every field by hand.
  • Evaluation: 14 frontier methods are compared across commercial VLMs, open-source pipelines, coding agents, and specialized APIs using accuracy, grounding, cost, and quality–cost tradeoffs.Commercial VLMs and coding agents do not return word-level boxes, requiring specialized extraction APIs for workflows that need source evidence.

2 ExtractBench

ExtractBench defines schema-guided extraction as returning schema-valid structured data with evidence, requiring exhaustive, accurate values and nulls for absent fields. It evaluates these systems with challenge-tagged enterprise documents and a scalable ground-truth pipeline tailored to real, synthetic, and scanned sources.

  • Schema-guided extraction: A schema lists fields, types, and natural-language descriptions, guides documents of the same type despite structural variation, and requires unanswered fields to be null.Schemas are expressed as JSON Schema and may request more or less information than a document contains.
  • Schema-guided extraction: The output is a schema-valid JSON object containing source pages and bounding boxes for each value, with correct exhaustive values, repeated records, nulls, and grounding.The formal mapping is from a document and schema to structured data and evidence.
  • Challenge-tagged corpus: ExtractBench tags documents across five independent axes: task challenge, perception challenge, table structure, length, and business domain.The corpus spans 8 business domains and 67 document types, while length ranges from short to long documents exceeding 50 pages.
  • Challenge-tagged corpus: The task-challenge axis includes long-list completeness, needle-in-a-haystack extraction, and dense documents, capturing truncation, false records, missed targets, and over-extraction.Long-list completeness concerns repeated structures spanning many pages; needle-in-a-haystack tasks have a median of 1.6 fields per page; dense schemas may exceed 150 leaf fields.
  • Schema and ground truth: Ground truth combines frontier-model ensembles for real documents, programmatic generation for synthetic long lists, and human labelers for scanned forms.Each field has a deterministic expected value, while real-document agreement—including agreement on null—produces candidate ground truth and disagreements are reviewed by cause.

3 Experiments

ExtractBench evaluates 14 systems across VLMs, coding agents, and specialized APIs using shared document–schema pairs, measuring value accuracy, grounding, and cost. Results reveal a quality–cost frontier, severe long-document and large-table truncation for many VLMs, and substantial remaining grounding gaps.

  • Evaluation setup: 14 systems are evaluated across direct-generation VLMs, iterative coding agents, and specialized document-extraction APIs.All systems receive identical document–schema pairs without benchmark-specific tuning; self-hosted VLMs are excluded from cost comparisons.
  • Overall quality–cost tradeoff: 95.6% value F1 at 8.1 ¢/page makes LlamaExtract Agentic Plus the quality–cost leader, outperforming coding agents at no more than half their cost.Coding agents reach 87.1% and 93.6% F1 at 16.2 and 27.8 ¢/page, while LlamaExtract Cost-Effective and Agentic reach 86.8% at 1.0 ¢/page and 89.5% at 3.1 ¢/page.
  • Document length: 96.6/93.3/94.4% across short, medium, and long documents makes LlamaExtract Agentic Plus strongest at every length, while commercial VLMs fall below 40% on long documents.The long-document failure is concentrated in recall: systems drop entire records because they cannot iterate through long documents and stop early.
  • Table structure: 95.9% on enormous tables beyond 1,000 rows puts LlamaExtract Agentic Plus first, ahead of Reducto Deep Extract at 95.3% and Claude Code Opus 4.8 at 87.8%.Every VLM scores below 10% on this slice, indicating that most systems return only a small fraction of records.
  • Grounding: 84.9% page-level grounding F1 versus 46.4% word-level grounding F1 for LlamaExtract Agentic Plus demonstrates that exact-word evidence is substantially harder than page localization.VLMs and coding agents score zero at both grounding levels because they do not return evidence by default; the best overall word-level grounding F1 is only 46.4%.

4 Related Work

Document extraction benchmarks span fixed-ontology and schema-guided settings, with increasing attention to complex schemas, long documents, and large outputs. ExtractBench combines broad coverage with production-oriented evaluation of accuracy, grounding, cost, and failure modes.

  • Benchmark settings: Document extraction benchmarks cover fixed-ontology tasks with predefined fields and schema-guided tasks where users specify fields and output structure at inference time.Fixed-ontology benchmarks increasingly test diverse layouts, languages, domains, tables, line items, long documents, and unfamiliar templates.
  • Schema-guided benchmarks: Schema-guided benchmarks increasingly test complex schemas, longer documents, and larger outputs without requiring system retraining.Prior work includes schemas with up to 369 fields but limited shared-scenario coverage.
  • ExtractBench: ExtractBench provides the broadest combined coverage of these dimensions, spanning real document families and targeted synthetic stress tests across 8 business domains.Its dataset is designed around production requirements at scale and uses challenge tags to cover task and perception difficulties and diagnose system-family failures.
  • ExtractBench: ExtractBench jointly measures value accuracy, source grounding, and per-page cost to assess whether outputs are correct, traceable, and economical to produce.The benchmark combines these measurements with challenge-based stratification for diagnosing failure modes.
  • System families: Schema-guided extraction systems comprise vision-language models, coding agents, and specialized extractors, which trade off completeness, visual robustness, grounding, and cost.Vision-language models generate structured outputs directly, coding agents iteratively inspect documents with tools, and specialized platforms provide managed extraction workflows and metadata.

5 Conclusion · Appendix Contents

ExtractBench unifies accuracy, completeness, grounding, robustness, and cost to reveal both system performance and failure modes. The conclusion highlights truncation, cost tradeoffs, challenge-dependent degradation, and persistent grounding gaps, while the appendix documents taxonomy, corpus, annotation, and capability comparisons.

  • 5 Conclusion: ExtractBench evaluates correct and complete outputs, source traceability, robustness across document challenges, and cost at scale in one benchmark.The combined evaluation shows not only which systems perform well, but also where and why they fail.
  • 5 Conclusion: Direct VLM extraction is inexpensive but often truncates long record lists, whereas coding agents are more robust but substantially more expensive.This contrast is especially relevant for long-record workloads.
  • 5 Conclusion: LlamaExtract Agentic Plus achieves the strongest overall performance at a lower cost than coding agents.Specialized APIs span the quality–cost frontier.
  • 5 Conclusion: Systems degrade differently on long documents, dense schemas, scans, handwriting, and enormous tables.Challenge-tagged results expose variation in robustness across these document conditions.
  • 5 Conclusion: 46.4% is the word-level grounding F1 achieved even by specialized systems that return boxes.Evaluated VLMs and coding agents do not return source evidence by default.
  • 5 Conclusion: Reliable extraction requires complete outputs, traceable evidence, and sustainable cost at scale.The conclusion identifies grounding as the clearest area for improvement.
  • Appendix Contents: Appendix A.1 provides a taxonomy and coverage reference, while A.2 describes corpus composition.These sections organize the benchmark’s scope and dataset structure.
  • Appendix Contents: Appendix A.3 details annotation methodology, and A.4 presents the full capability comparison.Together, these sections document how the benchmark was annotated and how systems were compared.

A Benchmark Details

The benchmark details include procedures for cell matching, missing-value semantics, array alignment, and aggregation with grounding.

  • A Benchmark Details: The benchmark details cover cell matching and normalization, missing-value semantics, array alignment, and aggregation and grounding.These topics are presented as subsections B.1 through B.4.

B Metric Details

This section covers system configuration and cost accounting and pricing.

  • The section includes subsections on system configuration and cost accounting and pricing.

C Evaluation Protocol … A.1 Taxonomy and Coverage Reference

ExtractBench organizes evaluation around a shared taxonomy of task and grounding challenges, with coverage reported across documents and pages. The taxonomy supports challenge-level result slices and explicit evidence requirements for auditable extraction.

  • D Detailed Results: The document’s detailed-results materials include analyses of document length, task challenges, pipeline failure modes, and quality–cost scaling.The listed qualitative examples cover long-list completeness, needle-in-a-haystack cases, and dense documents.
  • A.1 Taxonomy and Coverage Reference: Task challenges include long-list completeness, needle-in-a-haystack retrieval, and dense-document extraction.The evaluation materials separately identify T1, T2, and T3 as task-challenge categories.
  • A.1 Taxonomy and Coverage Reference: The taxonomy table defines each tag, its stressor, representative document types, and tagged document and page coverage.These tags are reused in the capability comparison and every result slice, while Figure 2 summarizes challenge-level coverage.
  • A.1 Taxonomy and Coverage Reference: Coverage is reported as shares of the 370-document, 4,869-page benchmark, with overlapping tags across task, perception, structure, size, and business-domain slices.Table 4 provides representative documents and tagged document and page coverage; Figure 5 visualizes page-share coverage.
  • A.1 Taxonomy and Coverage Reference: T1 tests large-scale record completeness using lists with 3,063 holdings, 8,624 creditor records, and 26,725 unclaimed-property rows.The notes warn that a well-formed array can still omit a substantial fraction of records.
  • A.1 Taxonomy and Coverage Reference: Grounding tags require extracted values to point back to source evidence, including value boxes, checkbox boxes with booleans, multi-record fields, and nested arrays.The tags are G1 for value boxes, G4 for checkbox handling, G2 for expanding fields, and G3 for deeply nested objects and arrays.

A.1.1 Document and Ground-Truth Size by Task Challenge … A.2.3 Long Documents (>50 pages)

ExtractBench spans diverse document lengths and challenge types, from sparse needle-in-a-haystack documents to dense long-list records requiring tens of thousands of returned fields. Its corpus combines public and synthetic documents across multiple business domains, with short, medium, and long-document slices designed to expose scale, density, truncation, and evidence-grounding challenges.

  • A.1.1 Document and Ground-Truth Size by Task Challenge: 1.6 fields per page is the median for T2 needle-in-a-haystack documents, versus 62.6 for long-list documents and 50.0 for dense documents.T2 returns a median of 35 fields after reading 25 pages, while T1 and T3 re-encode much of the document.
  • A.1.1 Document and Ground-Truth Size by Task Challenge: 86,242 ground-truth fields occur in the largest example: a 114-page bankruptcy creditor matrix.Across the corpus, 48 documents exceed 1,000 fields and 12 exceed 10,000, creating long-list completeness demands.
  • A.1.1 Document and Ground-Truth Size by Task Challenge: T1 documents have a 17-page median while T3 documents have a 1-page median, so challenge and document length are tagged independently.T1 and T3 overlap in field density but differ in scale, with records arriving either on one dense page or across tens of pages.
  • A.2 Corpus Composition: Every document receives exactly one length class, while scan-degraded re-captures are paired one-to-one with clean originals under the same schema and expected values.This corpus organization places forms, valuations, and re-captures inside length-based slices rather than treating them as separate categories.
  • A.2.1 Short Documents (≤10 pages): 252 short documents cover 615 pages, including 68 multi-domain business documents, 155 regulatory and tax forms, 6 automotive valuations, and 23 re-captures.The short-document slice includes finance, government, customs, healthcare, mortgage, product-specification, and automotive materials.
  • A.2.2 Medium Documents (11–50 pages): 98 medium documents cover 2,438 pages, including 64 multi-domain reports, 14 Form 1040 bundles, 8 automotive valuations, and 12 re-captures.Examples require KPI canonical-occurrence selection, reconciliation, hierarchical-header handling, nested detail, multi-line addresses, and cross-page table interpretation.
  • A.2.3 Long Documents (>50 pages): 20 long documents cover 1,816 pages, including 17 multi-domain filings and registers and 3 scan-degraded re-captures.Examples include a 3,063-row SEC 13F table, a 62-page CLIN schedule, a 95-page SF1449 solicitation, and a 56-page DD1155 continuation.
  • A.2.3 Long Documents (>50 pages): 26,725 rows make the unclaimed-property list the benchmark’s extreme truncation stress, alongside an 8,624-block creditor matrix and a 163-page clinical listing.These synthetic re-renders provide exact page- and word-level box evidence by construction.

A.3 Annotation Methodology in Detail … A.4 Full Capability Comparison

The appendix details ground-truth construction for real documents, synthetic long lists, scanned forms, and degraded re-captures, then presents a capability-comparison matrix organized by benchmark task specification. The procedures combine schema refinement, multi-system extraction, adjudication, human review, and controlled degradation while preserving source evidence and evaluation comparability.

  • A.3 Annotation Methodology in Detail: The appendix describes complete procedures for the three ground-truth methods introduced in Section 2.3.These methods cover real documents, synthetic long lists, and scanned forms.
  • A.3 Annotation Methodology in Detail: Evidence lists store expected values, alternate defensible readings, source pages, and reviewed word-level boxes, while OR-acceptance permits any listed reading and recall counts every expected record.A field may have multiple entries when a value is cited in several places or has multiple defensible readings.
  • A.3.1 Real Documents: Real-document workflows collect and verify representative PDFs, draft schemas with aliases and location guidance, and run multiple extraction systems to drive schema discovery and review.The same machinery supports schema discovery for new families and audits of existing ground truth by investigating cells where many pipelines fail.
  • A.3.2 Synthetic Long Lists: Synthetic long-list ground truth begins with a real layout pattern, builds complete structured content, renders it into a realistic PDF, and paginates measured blocks by actual space.Patterns include repeated rows, sectioned registers, continuation pages, totals, and nested records.
  • A.3.3 Scanned Forms: Scanned-form annotation freezes an agent-only schema before labeling, aggregates up to five systems’ votes, adjudicates contested leaves against rendered pages, and subjects every field to human review.Annotators may accept, edit, null, or redraw proposed values and boxes; schema revisions after aggregation require a version bump and targeted re-review.
  • A.3.3 Scanned Forms: Value consensus normalizes votes by field comparator, aligns array rows by declared identity key, and assigns each leaf to unanimous, majority, split, or all-null tiers.Omitting a path is treated as a null vote rather than an abstention, and split proposals are queued for adjudication.
  • A.3.4 Scan-Degraded Re-Captures: 38 degraded re-captures reuse clean documents’ values, schemas, and tags while varying only controlled capture conditions, enabling clean-versus-degraded comparisons and regeneration from recorded recipes and seeds.Boxes remain normalized to the page, with only rotation and perspective transformations moving their corners.
  • A.4 Full Capability Comparison: The full capability comparison scores every taxonomy tag separately and groups benchmark columns by schema-guided versus fixed-ontology task specification.Filled circles indicate covered and scored capabilities, hollow circles indicate partial or incidental coverage, and blanks indicate absence.

B Metric Details … D Detailed Results

The appendix defines deterministic schema-guided extraction metrics, evaluation configurations, and cost accounting, then extends headline results with analyses of quality, failures, operational outcomes, and quality–cost scaling.

  • B Metric Details: Scoring is deterministic, with identical predictions and ground truth always producing the same unified value F1.No model is involved in scoring.
  • B.1 Cell Matching and Normalization: Dates are canonicalized to ISO format, strings use whitespace normalization followed by exact case-sensitive matching, and only long-list free-text fields use edit-distance ratios.Numbers require plain equality, scalar lists remain opaque and order-sensitive, and no other fields use fuzzy matching.
  • B.2 Missing-Value Semantics: Every scalar field contributes to both precision and recall denominators, while missing keys score like explicit nulls and faithful nulls for blank fields receive credit.Hallucinated values on blank fields therefore reduce both precision and recall.
  • B.3 Array Alignment: Records are aligned by globally optimal one-to-one Hungarian assignment, maximizing total field agreement without identity keys or order dependence.Exactly matching rows are pre-paired with a score-preserving hash join.
  • B.4 Aggregation and Grounding: Per-document scores are micro precision, recall, and F1, while slice and overall scores are unweighted means across documents; missing outputs score zero.Grounding requires a value-correct cell and a citation box overlapping field evidence at IoU ≥0.5 on the correct page.
  • C Evaluation Protocol: The evaluation protocol documents system execution, prompts, configurations, provider integrations, and benchmark-specific options.VLM APIs receive the document and shared prompts, while specialized extraction systems receive the same document and schema without a benchmark-specific prompt.
  • C.1 System Configuration: Coding agents run in isolated directories with local computation permitted, a 1,200-second per-document timeout, and instructions to return schema-conforming JSON using nulls and complete lists.Self-hosted VLMs use provider-specific guided-JSON or prompt-based configurations, and specialized APIs use citations and extraction settings.
  • C.2 Cost Accounting and Pricing: Commercial costs use public standard-tier or pay-as-you-go rates as of July 1, 2026, measured token, credit, or page usage, and exclude discounts; self-hosted open-weight pipelines are omitted.Managed extraction APIs separately charge parsing and field extraction, with provider-specific adjustments for schema size, multi-pass extraction, and page or credit rates.

D.1 Document-Length Analysis

Document length changes the quality–cost frontier: systems may become cheaper per page as documents grow while losing extraction quality through missing records. Agentic Plus remains the high-quality endpoint, while precision–recall gaps reveal severe long-document truncation among commercial VLMs.

  • Quality and Cost: Agentic Plus remains the high-quality endpoint on short and medium documents while costing less than coding agents and the most expensive specialized APIs.
  • Quality and Cost: On long documents, token-metered VLMs and coding agents become substantially cheaper per page, but one-shot VLMs degrade sharply and Codex loses ground.Per-document overhead is spread across more pages, but quality does not scale uniformly.
  • Quality and Cost: Robust long-document extraction requires both favorable cost scaling and stable quality, because lower per-page cost can coincide with extracting a smaller fraction of requested records.
  • Precision and Recall: Large positive L3 precision–recall gaps for commercial VLMs show that returned values are often correct while many requested records are missing.Missing records lower recall, whereas duplicated or hallucinated records lower precision.

D.2 Task-Challenge Analysis … T3: the location and operations lines of a Railroad Commission Form P-18

ExtractBench’s challenge analysis shows that long-list completeness, retrieval behavior, schema breadth, and extraction operations drive major performance differences, while qualitative examples illustrate both strong extraction and grounding failures across document types. Operational limits and cost scaling further separate systems, with some pipelines completing all documents and others failing through capacity, schema, or timeout constraints.

  • D.2 Task-Challenge Analysis: Long-list extraction most clearly separates systems because preserving record boundaries and cell associations becomes difficult in packed or multi-row layouts.LlamaExtract Agentic Plus and Reducto Deep Extract remain strong across regular and difficult long-table layouts, while several systems degrade.
  • D.2 Task-Challenge Analysis: T2 sub-tags change system ordering: Gemini leads paraphrased matching, while LlamaExtract Agentic Plus and Reducto lead cross-modal deduplication and reconciliation.Sparse narrative retrieval is broadly tractable for the strongest systems, but the family-level score conceals differences among retrieval behaviors.
  • D.2 Task-Challenge Analysis: Dense-document performance is strong for receipts and invoices but spreads more for classify-then-extract, filer-reviewer separation, and large schemas.Visible density is only part of the challenge; schema breadth and the requested extraction operation also matter.
  • D.2 Task-Challenge Analysis: Seven systems score below 50 F1 on T3.e, while Codex GPT-5.5, Qwen3.6 35B-A3B, and all LlamaExtract systems complete every T3.e document and score above 80.Gemini returns no output for the 152-field W-2 schema; Lift, Gemma4, Reducto, and Extend return no output for all 18 Form 1040 bundles.
  • D.3 Failure Modes Across Pipelines: Codex GPT-5.5 and LlamaExtract Agentic Plus successfully extract all 370 documents, whereas capacity or schema limits and timeouts account for many other failures.Missing outputs receive zero and lower F1; capacity or schema limits dominate failures for Gemini, Claude Code, Qwen, Reducto, and Extend, while timeouts dominate Datalab, NuExtract3, and Gemma4.
  • D.4 Quality–Cost Scaling Within Model Families: GPT is the only family with monotonic quality gains: GPT-5.4 Nano, GPT-5.4 Mini, and GPT-5.5 score 74.9, 85.2, and 88.7 F1 at 0.21, 0.72, and 6.86 ¢ per page.Gemini remains essentially flat, moving from 79.6 F1 at 0.17 ¢ per page for Flash Lite to 79.8 at 1.00 ¢ for Flash.
  • E.1 Long-list completeness (T1): LlamaExtract Agentic Plus returns all 8 values and grounds 7 in the degraded bankruptcy service-list example, while the comparison systems make errors or provide no citations.The selected crop comes from a 17-page service list with 250 parties and 1,457 values; Lift 9B, Codex GPT-5.5, and Extend Max Context each exhibit distinct extraction errors, and none cites these values.
  • E. Qualitative Examples; E.2 Needle-in-the-haystack (T2); E.3 Dense documents (T3); T2: three claims inside a Medi-Cal remittance advice; T3: the location and operations lines of a Railroad Commission Form P-18: Across qualitative examples, LlamaExtract Agentic Plus returns all 12 selected Medi-Cal values and all 12 Form P-18 values, grounding 11 in the latter, while other systems omit, merge, shift, or misclassify fields.The Medi-Cal example has no box ground truth, whereas the Form P-18 crop contains 12 grounded reference fields; the examples use value correctness and citation status as shown in the saved predictions.
Loading 2607.29677v2…