PaperlayerView exact revision on arXiv

Source-linked AI summary

ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

Boyang Zhang, Adrian Lyjak, Eli Stewart, Zhaoqi Li, Simon Suo

arXiv:2607.29677v2cs.AI

TL;DR

Enterprise document extraction is manual and repetitive, while production workflows need traceable outputs and attention to cost. ExtractBench evaluates schema-guided extraction across accuracy, completeness, grounding, and cost, finding specialized APIs span the quality–cost frontier with strong overall performance at lower cost than coding agents.

  • Problem

    Enterprise document extraction is manual and repetitive, and production use makes source traceability and per-page cost important alongside extraction quality.

    Source: §1 · pp. 1–2

  • Method

    ExtractBench is a challenge-tagged benchmark that evaluates schema-guided extraction for correctness, completeness, traceability, robustness, and cost, with real-document ground truth curated through cross-system agreement and human review.

    Source: §2.3 · p. 5, §5 · p. 11

  • Results

    95.6% versus 93.6% accuracy for LlamaExtract Agentic Plus and Codex GPT-5.5, respectively, accompanies costs of 8.1 ¢/page versus 27.8 ¢/page; specialized APIs span the quality–cost frontier.

    Source: §1 · p. 3, §5 · p. 11

  • Takeaways & Limitations

    Evaluating core extraction requirements together shows which systems perform well and where they fail across document challenges.

    Source: §5 · p. 11

    Word-level grounding remains limited: even specialized systems with boxes reach only 46.4% F1, while evaluated VLMs and coding agents do not return source evidence by default.

    Source: §5 · p. 11

Abstract

from arXiv · show

Enterprise workflows increasingly rely on agents for \emph{schema-guided extraction}: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a benchmark for schema-guided extraction and, to our knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together. The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with clear tags differentiating their challenge scenarios. The scalable schema and ground-truth curation pipeline combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. We report order-insensitive value F1 for value accuracy, plus two grounding metrics for source traceability: word- and page-level F1. Commercial VLMs perform well on short documents but often truncate record lists on long ones, while coding agents retain higher accuracy at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost. Dataset and evaluation code are available on \href{https://huggingface.co/datasets/llamaindex/ExtractBench}{HuggingFace} and \href{https://github.com/run-llama/ExtractBench}{GitHub}.

1 Introduction

ExtractBench targets schema-guided enterprise extraction, where agents must follow user-defined schemas and provide source evidence. It benchmarks broad document coverage, traceability, and cost, and shows that performance can vary sharply with document length.

Motivation

  • Extraction failures include missing rows in long lists, selecting the wrong occurrence of sparse facts, overfilling forms, confusing similar values, and misreading document structure.Visual grounding helps humans find and correct failures, while per-page cost can determine whether high-volume extraction is viable.Source: §1 · p. 2

Related work

  • Existing fixed-KIE benchmarks lack user-specified schemas, while the closest schema-guided benchmarks each cover only a slice of the requirements and omit cost, grounding, and scanned or handwritten documents.Those slices include enterprise-scale schema-conformant JSON, completeness on synthetic rendered arrays, and long statistical reports and filings.Source: §1 · p. 2

Benchmark

  • ExtractBench covers 370 documents and 4,869 pages across 8 business domains and 67 document types, and jointly evaluates long-record completeness, real scans and handwriting, word- and page-level grounding, and measured cost.Documents are tagged by task challenge, perception challenge, table structure, domain, and length.Source: §1 · pp. 2–3, Table 1 · p. 2

Benchmark design

  • Its scalable curation pipeline adjudicates independent-system proposals for real documents, sets synthetic-list values before rendering, and uses human verification for scanned forms.Accuracy uses order-insensitive value F1, while grounding scores whether correct values point to their sources for fields with human-verified boxes.Source: §1 · pp. 2–3

Results

  • Accuracy on short documents fell from 87.9% to 27.9% on long documents for an unnamed system, while LlamaExtract Agentic Plus was more consistent at 96.6% and 94.4%, respectively.LlamaExtract Agentic Plus outperformed Codex GPT-5.5, 95.6% versus 93.6%, at 8.1 ¢/page versus 27.8 ¢/page.Source: §1 · p. 3

2 ExtractBench

ExtractBench specifies schema-guided extraction as structured values plus source evidence, and builds a challenge-tagged corpus with source-specific ground truth and separate value and grounding metrics.

2.1 Task Definition

  • A user-authored JSON Schema guides extraction across documents; systems must return schema-valid JSON with exhaustive values, nulls for absent fields, and page-and-box evidence.Schemas support scalar fields, nested objects, record arrays, nullable fields, and value constraints.Source: §2.1 · p. 3

2.2 Taxonomy and Coverage

  • Documents are tagged independently by task challenge, perception, table structure, length, and business domain so low scores can be traced to their causes.Task tags cover long-list completeness, needle-in-a-haystack retrieval, and dense documents; table tags capture layouts that can misassemble otherwise correctly read values.Source: §2.2 · pp. 3–4
  • Coverage spans 8 business domains and 67 document types, while separate tags capture rotated, scanned, or handwritten pages and short, medium, or long documents.Length has its own axis because systems may stop before the end of a long schedule even when they read each page correctly.Source: §2.2 · p. 4

2.3 Schema and Ground-Truth Construction

  • Each document type has one deterministic schema, and ground truth is built through independent-system agreement for real documents, exact-by-construction synthetic lists, and human-verified scanned forms.For real documents, disagreement prompts schema refinement when multiple readings are defensible and reviewer adjudication when only one is; scanned-form annotation yielded 169 verified documents, with 84% of verified fields carrying a human-placed box.Source: §2.3 · pp. 5–6

2.4 Metrics

  • Unified value F1 compares normalized scalar and aligned record cells, crediting correct nulls; repeated records are matched as an unordered set.Dates are normalized to ISO format and strings by collapsing whitespace; other values require exact equality, and omitted keys score as explicit nulls.Source: §2.4 · p. 6
  • Word-level grounding requires a correct value and a predicted box overlapping its verified box at IoU 0.5; page-level F1 requires only the correct source page.Grounding scores apply only where box ground truth is verified.Source: §2.4 · p. 6, §2.3 · p. 6

3 Experiments

LlamaExtract Agentic Plus achieves the strongest reported quality–cost result, while dimension-level results expose long-document truncation and grounding remains substantially weaker at word level than at page level.

3.1 Setup

  • The benchmark evaluates 14 extraction systems across VLMs, coding agents, and specialized APIs using shared document–schema pairs without benchmark-specific tuning.It reports unweighted document-level value F1, word- and page-level grounding F1, and cost; self-hosted VLMs are excluded from cost comparisons.Source: §3.1 · pp. 6–7

3.2 Quality–Cost Frontier

  • 95.6% value F1 at 8.1 ¢/page lets LlamaExtract Agentic Plus outperform both coding agents while costing no more than half as much.The LlamaExtract frontier also includes Cost-Effective at 86.8% F1 and 1.0 ¢/page, and Agentic at 89.5% F1 and 3.1 ¢/page.Source: Figure 4 · p. 7, §3.2 · p. 7

3.3 Results Across Dimensions

  • On long documents, commercial VLMs fall below 40% F1 while Agentic Plus stays above 90% across all lengths and it and Reducto lead the T1 and T2 challenges.Claude Code Opus 4.8 scores 88.1% and Reducto 92.0% on long documents; long-document failures primarily drop records, consistent with context limits and systems stopping early without an iteration strategy.Source: §3.3 · pp. 7–8
  • On dense T3 documents, Agentic Plus (95.5%) and Codex GPT-5.5 (95.4%) lead, while perception challenges reveal system-specific blind spots.Codex drops to 81.0% on rotated or image-only capture, Reducto to 81.1% on scanned pages, and Qwen3.6 35B-A3B exceeds 92% on scanned pages and handwriting; Agentic Plus is strongest across all three perception challenges.Source: §3.3 · pp. 8–9

3.4 The Grounding Gap

  • VLMs and coding agents score zero at both grounding levels by default, while Agentic Plus achieves 84.9% page-level grounding F1 but only 46.4% word-level F1.Grounding requires both the extracted value and its citation to be correct; page evidence narrows the search but leaves reviewers to locate the value among similar candidates.Source: Table 3 · p. 9, §3.4 · p. 9

3.3 Results Across Dimensions

  • On tables exceeding 1,000 rows, every VLM scores below 10%, while Agentic Plus leads at 95.9%; domain difficulty largely reflects the mix of task challenges.Reducto scores 95.3% and Claude Code Opus 4.8 scores 87.8% on enormous tables; cross-page tables pose a milder structural challenge, and pivoted layouts are least discriminative.Source: §3.3 · p. 9

3.4 The Grounding Gap

  • Extend Max Context falls from 61.7% page-level grounding F1 on short documents to 0.0% on long documents, while Reducto declines from 72.6% to 67.3%.The best overall word-level grounding F1 is only 46.4%, so reliably connecting extracted values to exact supporting evidence remains an open problem.Source: §3.4 · pp. 9–10

4 Related Work

Related-work benchmarks differ in whether schemas are fixed or user-supplied, and prior schema-guided evaluations cover these challenges only in fragmented ways. ExtractBench combines real and synthetic enterprise documents with accuracy, grounding, cost, and challenge-tag evaluation; extraction system families also make distinct workflow tradeoffs.

4.1 Benchmarks for Document Extraction

  • Fixed-ontology benchmarks test robustness as documents grow more diverse or complex, but their target fields remain fixed rather than user-supplied [21] [17] [20] [39] [40] [41].They span forms, receipts, multilingual layouts, enterprise domains, line items, tables, long documents, unfamiliar templates, and spatial localization.Source: §4.1 · p. 10
  • Prior schema-guided benchmarks vary in schema reuse, document scale, and synthetic-versus-real coverage, leaving fragmented evidence for comparing systems and diagnosing failures [12] [4] [22].ContextualAI’s ExtractBench [12] uses five shared schemas; LongExtractBench-50 and VAREX [4] use a different schema per document, while LongArray-Extract and LongExtractBench-50 emphasize long-document completeness but have test sets of only dozens of documents.Source: §4.1 · p. 10
  • ExtractBench combines real document families and targeted synthetic stress tests across 8 business domains, measuring value accuracy, source grounding, and per-page cost with challenge tags.The tags stratify task and perception difficulties to help diagnose where system families fail.Source: §4.1 · p. 10

4.2 Methods for Document Extraction

  • General-purpose vision-language models turn pages and schema instructions into one-pass generation, but can miss values in long repeated structures and do not return source evidence.They accept new schemas without task-specific retraining, either through prompted text or schema-compliant structured output.Source: §4.2 · p. 10
  • Coding agents inspect documents, write and validate parsing code, and can revisit long documents, but their flexibility brings cost and reliability risks.They require a filesystem-enabled agent runtime, depend on rendering and available tools, and are not designed around extraction-specific grounding metadata.Source: §4.2 · p. 10
  • Specialized document extractors provide managed upload, preprocessing, parsing, and schema-guided APIs, sometimes with visual grounding, but vary in completeness, robustness, grounding quality, and cost.Some expose source pages and word-level boxes, making outputs easier to audit than raw model responses.Source: §4.2 · p. 11

5 Conclusion · Appendix Contents

ExtractBench evaluates schema-guided extraction across output quality, source traceability, challenge robustness, and cost, showing that specialized systems balance performance and cost better than coding agents while grounding remains a major gap.

5 Conclusion

  • ExtractBench brings correct and complete outputs, source traceability, challenge robustness, and cost at scale into one evaluation.Its challenge tags expose the requirements of real schema-guided extraction and support comparisons of system performance and failure modes.Source: §5 · p. 11
  • LlamaExtract Agentic Plus achieves the strongest overall performance at lower cost than coding agents, while direct VLM extraction is inexpensive but often truncates long record lists.Coding agents are more robust on these workloads but substantially more expensive; specialized APIs span the quality–cost frontier.Source: §5 · p. 11
  • Challenge-tagged results show that systems degrade differently on long documents, dense schemas, scans, handwriting, and enormous tables.These challenge-specific results help identify where and why systems fail across extraction workloads.Source: §5 · p. 11
  • Word-level grounding F1 remains 46.4% even for specialized systems that return boxes, while evaluated VLMs and coding agents do not provide source evidence by default.The conclusion sets complete outputs, traceable evidence, and sustainable cost at scale as the bar for reliable extraction.Source: §5 · p. 11

A Benchmark Details · A.1 Taxonomy and Coverage Reference

The taxonomy defines challenge tags, representative document types, and tagged document/page coverage, which are reused in capability comparisons and result slices. It distinguishes task and grounding demands across a benchmark of 370 documents and 4,869 pages.

A.1 Taxonomy and Coverage Reference

  • The taxonomy table defines each tag’s challenge, representative document types, and document/page coverage; the tags also organize capability comparisons and result slices.Figure 2 presents challenge-level coverage as a distribution, while Figure 5 shows page shares and counts; tags may overlap.Source: §A.1 · p. 14, Figure 5 · p. 14, Table 4 · p. 15
  • T1 tests large-scale record completeness, including real tables or lists with 3,063 holdings, 8,624 address-block records, and 26,725 rows.A well-formed array can still omit a third of its rows, so partial credit can conceal F1 errors.Source: §A.1 · p. 16
  • T2 captures canonical-value selection when KPIs recur under paraphrase and target fields are buried in procurement narrative.Source: §A.1 · p. 16
  • T3 highlights field-localization errors despite dedicated tools [41], while T3.d’s W-14 schema separately extracts original filer values and later regulator annotations.T3.d covers 13 administratively reviewed W-14 filings.Source: §A.1 · p. 16
  • T3.e marks schemas with strictly more than 150 leaf fields; its 35 documents and 421 pages overlap other T3 tags.The cited schemas have 152 leaf fields for four W-2 documents, 175 for 13 reviewed W-14 documents, and 1,343–1,368 for 18 Form 1040 bundles.Source: §A.1 · p. 16
  • Grounding tags specify evidence beyond values: boxes at extracted values (G1), checkbox boxes plus boolean reads (G4), many-record fields (G2), and deeply nested objects or arrays (G3).These tags constrain output; every ground-truth value is paired with a location, and grounding is scored in Section 3.4.Source: §A.1 · p. 16, Table 5 · p. 16

A.1.1 Document and Ground-Truth Size by Task Challenge … A.2.3 Long Documents (>50 pages)

The appendix maps extraction burden across challenge types and document lengths, contrasting sparse needle-in-a-haystack tasks with dense, record-heavy documents and describing distinct structural and annotation demands across corpus slices.

A.1.1 Document and Ground-Truth Size by Task Challenge

  • Long-list and dense tasks return median rates of 62.6 and 50.0 fields per page, respectively, with up to 86,242 fields in one 114-page bankruptcy matrix.For long lists, systems can maintain high precision while recall falls because they fail to return every record; T1 and T3 also differ in scale, with median document lengths of 17 and 1 pages, respectively.Source: §A.1.1 · p. 16, Figure 6 · p. 17
  • Needle-in-a-haystack tasks yield a median of 1.6 fields per page: systems read a median of 25 pages to return 35 fields.An extreme example is an SF-1449 solicitation whose schema requests 6 fields across 31 pages; the other challenge types return over an order of magnitude more fields per page.Source: §A.1.1 · p. 16, Figure 6 · p. 17

A.2 Corpus Composition

  • Corpus size and breadth are summarized by document and page counts alongside domain and schema counts, but schema counts are not directly comparable across datasets.The 38 scan-degraded re-captures are omitted from Figure 6 because each retains its clean original’s schema, values, and page count; each re-capture is assigned to its original’s length class.Source: §A.2 · pp. 16–17, Table 6 · p. 17, Figure 6 · p. 17

A.2.1 Short Documents (≤10 pages)

  • The short-document slice contains 252 documents and 615 pages, spanning business records, regulatory and tax forms, automotive valuations, and scan-degraded re-captures.Its examples include financial, government, healthcare, mortgage, product, and automotive documents, with challenges such as disambiguating identifiers and interpreting pivoted or matrix layouts.Source: §A.2.1 · p. 18
  • Short regulatory and tax forms span 594 pages and 17 frozen schemas, with human review of every form field and additional schema-first adjudication and rendered-page auditing for tax forms.Across these documents, 16,534 evidence rules were human-verified, and 13,867 (∼84%) include a human-placed value box; public releases mask taxpayer identifiers, and the ground truth expects the masked strings.Source: §A.2.1 · p. 18

A.2.2 Medium Documents (11–50 pages)

  • The medium slice contains 98 documents across 2,438 pages, including financial KPI reports, nested schedules, audit lists, and large Form 1040 bundles with absent sub-forms that must return null.Its tasks include canonical KPI selection and reconciliation, multi-level rollups, classification, and extracting hierarchically structured or multi-page records.Source: §A.2.2 · p. 18

A.2.3 Long Documents (>50 pages)

  • The long-document slice contains 20 documents across 1,816 pages, including a 3,063-row SEC 13F table, an 8,624-block bankruptcy matrix, and a 26,725-row unclaimed-property list.These documents include sparse narrative fields, line items spanning page breaks, handwritten signatures, and exact page- and word-level box evidence for synthetic re-renders.Source: §A.2.3 · pp. 18–19

A.3 Annotation Methodology in Detail … A.4 Full Capability Comparison

The appendix details distinct ground-truth pipelines for real documents, synthetic long lists, scanned forms, and degraded re-captures, then presents a tag-by-tag capability matrix.

A.3 Annotation Methodology in Detail

  • Ground truth stores each field as an evidence list of expected and alternate defensible readings, with source pages and reviewed word boxes; scoring accepts any listed reading while counting every expected record for recall.Multiple entries capture either evidence from several locations or genuinely ambiguous readings, which an annotator ratifies.Source: §A.3 · p. 19

A.3.1 Real Documents

  • Real-document annotation aligns repeated rows by declared identity keys, combines system agreement with coding-agent review, and sends blocked or uncertain disagreements to an annotator for final classification.Schema ambiguities trigger schema revision and another system run; model failures are verified or corrected, while unresolved cells remain blocked.Source: §A.3.1 · p. 19

A.3.2 Synthetic Long Lists

  • Synthetic long lists are rendered from complete structured content and paginated by measured block size, then ground-truthed from the final PDF and audited for semantic defects.This avoids artificial page breaks and incorrect source-page labels from fixed rows-per-page pagination.Source: §A.3.2 · p. 19

A.3.3 Scanned Forms

  • Form review has annotators accept, edit, null, or redraw proposed values and boxes; each box comes from one citation-emitting pipeline or is drawn by an annotator if no valid citation exists.Adjudicators inspect rendered pages; a checkbox is false when unmarked on an included page, but null when its page is absent from the filing.Source: §A.3.3 · p. 20
  • Scanned-form consensus uses a frozen schema, normalized votes, and identity-key row alignment to classify leaves as unanimous, majority, split, or all-null, with contested and null-majority fields queued for adjudication.An ensemble spans independent model families; adjudication load ranges from none on small forms to a handful of fields per document on the largest forms.Source: §A.3.3 · p. 20

A.3.4 Scan-Degraded Re-Captures

  • The scan-degraded slice contains 38 documents paired with clean versions under unchanged expected values, so clean-versus-degraded scores isolate capture degradation.Recorded, reproducible scan recipes introduce rotation, perspective shifts, and photometric effects; normalized boxes are invariant to photometric effects and uniform rescaling, with rotation and perspective applied to box corners.Source: §A.3.4 · p. 20

A.4 Full Capability Comparison

  • Table 7 compares task specifications and capability coverage for every taxonomy tag separately, distinguishing schema-guided extraction from fixed-ontology benchmarks and marking full, partial, or absent coverage.The appendix says each mark is checked against the benchmark’s primary source; the main-text Table 1 summarizes domain breadth and measured cost.Source: §A.4 · p. 20, Table 7 · p. 21

B Metric Details … B.4 Aggregation and Grounding

Appendix B defines deterministic rules for cell matching, record alignment, aggregation, and grounding. Its scoring choices determine how missing values, record order, and source evidence affect evaluation.

B Metric Details

  • The appendix makes scoring fully deterministic: identical predictions and ground truth always produce the same value F1, without a model in the scoring loop.It specifies the matching, normalization, and aggregation rules underlying the unified value F1.Source: B Metric Details · p. 22

B.1 Cell Matching and Normalization

  • Common date formats are canonicalized to ISO YYYY-MM-DD, except guarded strings such as all-digit values, strings outside 4–50 characters, or parses outside 1900–2100.Strings containing runs of ten or more digits also pass through unchanged.Source: §B.1 · p. 22
  • For long-list comparability, scoring uses Extend’s public reference scorer verbatim, including edit-distance matching for two free-text fields; no other fields use fuzzy matching.This choice enables comparison with Extend’s LongArray benchmark.Source: §B.1 · p. 22
  • Strings are trimmed and whitespace-collapsed before exact, case-sensitive comparison; numbers have no tolerance, and scalar lists remain opaque and order-sensitive.No punctuation stripping, Unicode folding, or currency/thousands handling applies, and a number does not match its string rendering.Source: §B.1 · p. 22

B.2 Missing-Value Semantics

  • Every scalar schema field contributes to precision and recall denominators, and an absent key scores like explicit null; hallucinations on blank fields cost both, while correct nulls earn credit.The scoring table states that scalar-cell outcomes are symmetric between precision and recall; repeated records alone separate their contributions.Source: §B.2 · p. 22, Table 9 · p. 23

B.3 Array Alignment

  • Records are paired by a global optimal one-to-one assignment that maximizes field agreement, making value scores independent of record order and identity keys.Exact matching rows are pre-paired by a score-preserving hash join; duplicate predictions pair at most once, and nested record lists use independent assignments.Source: §B.3 · p. 23

B.4 Aggregation and Grounding

  • Per-document precision, recall, and F1 are micro-averaged over cells, while slice and overall scores are unweighted means of per-document values.Documents with errors, refusals, or no JSON score zero rather than being dropped; mean F1 is not necessarily the harmonic mean of mean precision and recall.Source: §B.4 · p. 23
  • Grounding requires an accepted value and a citation overlapping verified evidence on the correct page; for defined documents, value F1 ≥ page F1 ≥ word-level grounding F1.Word-level overlap uses IoU ≥0.5; page-level grounding tests page membership, and documents without verified boxes receive no grounding score.Source: §B.4 · p. 23

C Evaluation Protocol · C.1 System Configuration · C.2 Cost Accounting and Pricing

The appendix specifies system prompts and configurations for the evaluated extraction systems, then describes how their costs are reconstructed from usage and provider pricing rules. It distinguishes token-metered systems from managed document APIs and documents adjustments used for provider-specific charges.

C Evaluation Protocol

  • The protocol documents the prompts, configurations, and cost rules underlying the evaluation, with system execution specified in C.1 and pricing in C.2.Source: C Evaluation Protocol · p. 24

C.1 System Configuration

  • GPT-5.4 Nano and Gemini 3.5 Flash receive the same benchmark prompts, with provider-specific schema settings and mechanical schema transformations.GPT uses the Responses API with strict=false; Gemini uses response_json_schema with temperature zero and low thinking. The transformations do not change requested fields or descriptions.Source: §C.1 · p. 24
  • The VLM system and user prompts require JSON output, null for absent fields, and enumeration of every visible row in list fields.Source: §C.1 · p. 24
  • Self-hosted VLMs use differing schema interfaces, while specialized extraction APIs receive the document and schema without a benchmark-specific prompt.Lift and NuExtract3 use model-native interfaces; Gemma4's JSON mode enforces valid JSON but not the complete schema. Specialized systems use their respective parse, extract, citation, and output options.Source: §C.1 · pp. 24–25
  • Claude Code Opus 4.8 and Codex GPT-5.5 run in isolated directories with the staged document and schema, may use local computation, and have a 1,200-second per-document timeout.They must rely only on provided materials; Codex uses low reasoning effort, and the shared task prompt directs agents to enumerate visible rows and validate the output JSON.Source: §C.1 · p. 24

C.2 Cost Accounting and Pricing

  • Costs are reconstructed from recorded or estimated token, credit, or page usage at public standard-tier rates as of July 1, 2026, excluding discounts.Self-hosted open-weight pipelines are omitted because they have no vendor API price; measured costs can differ from headline per-page rates because pricing depends on observed usage.Source: §C.2 · p. 25
  • Table 10 gives token-metered list prices; OpenAI adds input and output surcharges above 272K input tokens, and Anthropic’s Opus 4.7+ tokenizer emits about 30% more tokens for the same text.Codex GPT-5.5 and Claude Code Opus 4.8 use the corresponding GPT-5.5 and Opus 4.8 rates.Source: §C.2 · p. 26, Table 10 · p. 26
  • Table 11 reports parse and extract charges for managed APIs, with provider-specific rules adjusting credit accounting for schema size, document size, and extraction strategy.Reducto charges max(30, 4p + 0.1f) extract credits plus 1–2 parse credits per page; Extend doubles the extract component for large_array_max_context, and Datalab treats one cent as one credit.Source: Table 11 · p. 26, §C.2 · p. 26

D Detailed Results

The detailed results extend the headline findings by examining performance across document lengths, task failures, operational outcomes, and commercial model families.

D Detailed Results

  • The analyses cover quality, cost, precision, and recall by document length; fine-grained task failures; completed extractions versus operational failures; and quality–cost scaling within commercial model families.These are examined in Sections D.1–D.4, respectively.Source: D Detailed Results · p. 27

D.1 Document-Length Analysis

Short- and medium-document systems follow a quality–cost frontier, while long documents lower per-page costs for some systems but expose uneven quality scaling. Precision–recall gaps show that commercial VLMs often return correct values while missing requested records.

D.1 Document-Length Analysis

  • On short and medium documents, the quality–cost frontier runs from GPT-5.4 Nano through LlamaExtract Cost-Effective, Agentic, and Agentic Plus.Agentic Plus remains the high-quality endpoint while costing less than coding agents and the most expensive specialized APIs.Source: §D.1 · p. 27, Figure 7 · p. 27
  • On L3, token-metered VLMs and coding agents cost less per page than on L1, but one-shot VLMs and Codex lose quality; Agentic Plus remains high-quality.Claude preserves more of its short-document quality and becomes a competitive intermediate point; Reducto remains accurate but at substantially higher per-page cost.Source: §D.1 · p. 27, Figure 7 · p. 27
  • Document growth can lower per-page cost while reducing the fraction of requested records extracted, so robust scaling requires both favorable cost scaling and stable quality.The analysis distinguishes these as separate dimensions of scaling.Source: §D.1 · p. 27
  • Commercial VLMs show large positive L3 precision–recall gaps, indicating returned values are often correct but many requested records are missing.Precision and recall expose failures hidden by F1: missing records reduce recall, while duplicated or hallucinated records reduce precision.Source: §D.1 · p. 28, Table 13 · p. 28

D.2 Task-Challenge Analysis … T3: the location and operations lines of a Railroad Commission Form P-18

The analysis finds that extraction performance varies by challenge subtype, operational limits, and model family; qualitative examples show how these differences appear in document-level outputs and grounding. Long-list completeness and large-schema capacity are recurring weaknesses, while quality–cost tradeoffs differ sharply across model families.

D.2 Task-Challenge Analysis

  • Long-list extraction separates systems: LlamaExtract Agentic Plus and Reducto Deep Extract remain strong across regular and packed layouts, where less regular rows challenge other systems.The analysis attributes the difficulty to preserving record boundaries and associating cells across continuations, repeated regions, and compact layouts. T2 sub-tags also change system ordering: Gemini leads on T2.b, while LlamaExtract Agentic Plus and Reducto are strongest on T2.c and T2.d; receipt and invoice extraction is strong across most systems, but classify-then-extract, filer-reviewer separation, and large schemas spread scores more widely.Source: §D.2 · p. 29
  • Seven systems score below 50 F1 on T3.e; Gemini scores 0, while Codex GPT-5.5, Qwen3.6 35B-A3B, and all LlamaExtract systems complete every document and score above 80.Gemini returns no output even for the 152-field W-2 schema; Claude Code completes W-2 but fails on W-14 and Form 1040, and Lift, Gemma4, Reducto, and Extend return no output for all 18 Form 1040 bundles.Source: §D.2 · pp. 29–30

D.3 Failure Modes Across Pipelines

  • Codex GPT-5.5 and LlamaExtract Agentic Plus extracted all 370 documents; capacity or schema limits caused most failures for several systems, while timeouts dominated for Datalab, NuExtract3, and Gemma4.Missing outputs receive zero and lower F1.Source: §D.3 · p. 30, Table 15 · p. 30

D.4 Quality–Cost Scaling Within Model Families

  • GPT alone improves monotonically with model tier, while the nine-model quality–cost frontier is Gemini 3.1 Flash Lite, GPT-5.4 Mini, and GPT-5.5.GPT-5.4 Nano, Mini, and GPT-5.5 score 74.9, 85.2, and 88.7 F1 at 0.21, 0.72, and 6.86 ¢ per page; Gemini Flash Lite, Flash, and Pro score 79.6, 79.8, and 78.2 F1 at 0.17, 1.00, and 3.83 ¢ per page. Claude Haiku, Sonnet, and Opus score 29.9, 31.5, and 30.1 F1 at 0.80, 2.40, and 4.89 ¢ per page.Source: §D.4 · p. 30, Figure 8 · p. 31

E Qualitative Examples

  • The qualitative examples pair page crops with field-level predictions, distinguishing correct grounded values, correct ungrounded values, incorrect values, and values not returned.For examples with ground-truth boxes, dashed boxes mark the ground-truth location when a system provides no usable citation; grounding requires a correct value and a citation at IoU 0.5.Source: E Qualitative Examples · p. 32

T1: the first two records of a 250-party service list

T2: three claims inside a Medi-Cal remittance advice

T3: the location and operations lines of a Railroad Commission Form P-18

Related papers

Loading 2607.29677v2…