Source-linked AI summary
VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents
Udi Barzelay, Ophir Azulai, Inbar Shapira, Idan Friedman, Foad Abo Dahood, Madison Lee, Abraham Daniels
TL;DR
Structured document extraction lacks systematic evaluation for small models and variable schemas. VAREX builds such a benchmark through Reverse Annotation, using synthetic values in PDF templates, deterministic value-level ground truth, and four controlled modalities. Across 20 models, it identifies small-model compliance failures and shows that layout-preserving text and benchmark design reveal meaningful performance differences.
Problem
Existing benchmarks provide limited evidence about small-model failures and schema generalization in structured document extraction.
Method
VAREX generates documents from fillable PDF templates, maps placeholders to semantic schemas, injects synthetic values, validates the results, and exports four input modalities.
Results
VAREX evaluates 20 models on 1,777 documents with 1,771 unique schemas and identifies schema echo, under-extraction, and modality-dependent accuracy differences.
Takeaways & Limitations
Layout-preserving text provides the largest modality-related accuracy gain, while VAREX most effectively discriminates models in the 60–95% accuracy range.
Takeaways & Limitations
The benchmark uses single-page English-language U.S. government forms with typed values from digital PDFs and lacks handwriting, scan artifacts, and multilingual content.
Abstract
from arXiv · showhide
We introduce VAREX (VARied-schema EXtraction), a benchmark for evaluating multimodal foundation models on structured data extraction from government forms. VAREX employs a Reverse Annotation pipeline that programmatically fills PDF templates with synthetic values, producing deterministic ground truth validated through three-phase quality assurance. The benchmark comprises 1,777 documents with 1,771 unique schemas across three structural categories, each provided in four input modalities: plain text, layout-preserving text (whitespace-aligned to approximate column positions), document image, or both text and image combined. Unlike existing benchmarks that evaluate from a single input representation, VAREX provides four controlled modalities per document, enabling systematic ablation of how input format affects extraction accuracy -- a capability absent from prior benchmarks. We evaluate 20 models from frontier proprietary models to small open models, with particular attention to models <=4B parameters suitable for cost-sensitive and latency-constrained deployment. Results reveal that (1) below 4B parameters, structured output compliance -- not extraction capability -- is a dominant bottleneck; in particular, schema echo (models producing schema-conforming structure instead of extracted values) depresses scores by 45-65 pp (percentage points) in affected models; (2) extraction-specific fine-tuning at 2B yields +81 pp gains, demonstrating that the instruction-following deficit is addressable without scale; (3) layout-preserving text provides the largest accuracy gain (+3-18 pp), exceeding pixel-level visual cues; and (4) the benchmark most effectively discriminates models in the 60-95% accuracy band. Dataset and evaluation code are publicly available.
1. Introduction
VAREX addresses the lack of systematic evaluation for small models and schema variability in structured document extraction. Its Reverse Annotation pipeline creates deterministic value-level ground truth and supports controlled multimodal evaluation.
- No existing benchmark systematically evaluates models below 4B parameters on structured document extraction.
- Existing benchmarks use fixed templates or schemas, limiting evaluation of schema generalization and creating annotation-scale challenges.
- VAREX fills PDF templates with deterministic placeholders, uses an LLM to map them to semantic fields, and injects realistic synthetic values into widgets.
- VAREX contains 1,777 documents and 1,771 unique schemas across three structural categories, evaluated through 21,084 fields and three-phase quality assurance.
- The benchmark evaluates 20 models across four modalities and reports schema echo, under-extraction, and an instruction-following threshold between 2–4B.
2. Related Work
Prior document benchmarks generally rely on fixed schemas or single input representations. VAREX combines variable schemas, multimodal inputs, and structured JSON output to evaluate generalization more directly.
- Early benchmarks use fixed schemas or a small number of schemas, limiting evaluation of adaptability to new document types.
- VRDU and DocILE increase template diversity but retain fixed extraction schemas across documents.
- Concurrent benchmarks cover schema-grounded output, PDF-to-JSON extraction, constrained decoding, or broad document annotations.
- Table 1 characterizes VAREX as combining unique per-document schemas, multimodal inputs, and structured JSON output.
- VAREX differs from SO-Bench and ExtractBench by combining per-document variable schemas, four controlled modalities, and deterministic ground truth.
3. The VAREX Benchmark
VAREX constructs a large, schema-diverse benchmark by generating documents from fillable government forms, validating their mappings and values, and exporting each document in controlled modalities. Its protocol combines broad structural coverage with explicit quality controls.
- Reverse Annotation Pipeline: The pipeline processes fillable PDF templates from U.S. government sources into synthetic documents generated from structured data.
- Reverse Annotation Pipeline: Deterministic placeholders uniquely identify widgets, allowing downstream semantic fields to be traced back to their source locations.
- Reverse Annotation Pipeline: A 24B instruction-tuned model generates semantic JSON Schemas from rendered images and spatial text, including nested and repeated structures.
- Reverse Annotation Pipeline: LLM schema discovery can misattribute fields, miss arrays, or hallucinate fields, making it the main source of ground-truth errors.
- Reverse Annotation Pipeline: Synthetic values are generated with Faker and LLM assistance, then written to specific PDF widget IDs with verification for write failures and truncation.
- Multi-Modal Export: Each document is exported as plain text, layout-preserving spatial text, image, or spatial text plus image.
- Dataset Composition: The final benchmark contains 1,777 documents, 1,771 unique schemas, and 21,084 evaluation fields, with a median of 11 fields per document.
- Quality Assurance: Three-phase quality assurance combines automated screening, review of 428 flagged documents, and sampled review of unflagged documents.
4. Results and Analysis
VAREX shows that model scale, output compliance, input representation, and training strongly shape structured extraction accuracy. Small-model failures often reflect formatting and instruction-following bottlenecks, while layout-preserving text and extraction-specific fine-tuning substantially improve results.
- Main Results: The benchmark spans an 88 pp accuracy range, from 9.7% to 98.0%, and most strongly discriminates models in the 60–95% band.This range is reported for Image modality results.
- Document Structure: Above 90% overall, models typically perform similarly across Flat, Nested, and Table documents, although lower-scale models show larger Flat–Table gaps.The gap reaches 7 pp for mid-range models and 8–20 pp for non-echo models below 80%.
- Semantic Categories: Format-sensitive fields produce the widest semantic-category gaps: monetary values range from 97% to 82%, while email addresses range from 99% to 82%.Zip codes and state abbreviations show narrower cross-scale gaps of less than 10 pp.
- Modality Analysis: Layout-preserving text delivers the largest modality gain, improving over raw text by +3 to +8 pp for high-performing models and up to +18 pp at smaller scales.Adding vision after spatial text yields only −1.0 to +2.1 pp, while combining spatial text and vision adds +0.5 to +2.2 pp.
- Output Compliance: Below 4B parameters, schema echo and other structured-output failures dominate apparent extraction errors, especially when schemas contain $defs references.Non-compliance rises from 2–3% on flat schemas to 68–69% with $defs; image inputs further exacerbate failures for Qwen3-VL 2B.
- Scaling and Fine-Tuning: Extraction-specific fine-tuning enables a 2B model to bypass the instruction-following threshold, while scaling from 1B to 2B also produces large gains.NuExtract 2.0 2B reaches 90.8% EM with zero schema echo, and its fine-tuning gain is approximately +81 pp over its 9.7% base.
5. Discussion
VAREX’s scope is bounded by synthetic, single-page U.S. government forms and LLM-discovered schemas, while residual annotation noise limits precision at the highest accuracies.
- VAREX covers single-page, English-language U.S. government forms with typed values rendered from digital PDFs, excluding handwriting, scan artifacts, and multilingual content.
- Synthetic Faker-generated values do not match real-world distributions, although the paper states this is immaterial to extraction accuracy.
- The benchmark most effectively discriminates models in the 60–95% accuracy range; above 95%, residual ground-truth noise limits comparison precision.
- Order-invariant array matching removes row-order ambiguity but does not penalize within-row field misattribution.
6. Conclusion
The paper presents VAREX as a multimodal structured-extraction benchmark with deterministic value-level ground truth, varied schemas, and controlled input modalities. Across 20 models, it identifies structured-output compliance as the dominant source of sub-4B failures and targets the 60–95% accuracy range for effective discrimination.
- VAREX contains 1,777 documents, 1,771 unique schemas, and 21,084 evaluation fields across four controlled input modalities.
- The Reverse Annotation pipeline creates deterministic value-level ground truth and validates it through three-phase quality assurance.
- 20 models from 800M to frontier scale reveal schema echo, under-extraction, and attention decay as concrete sub-4B failure modes.
- An 88 pp performance range makes VAREX most discriminative in the 60–95% accuracy band, where small-model failures are dominated by structured output compliance deficits rather than vision limitations.
Supplementary Material
The supplementary material provides prompts, complete document examples, schema-and-ground-truth illustrations, schema-echo outputs, resolution-robustness examples, and modality comparisons.
- The supplement includes the evaluation prompt used for all models and complete Nested and Table document examples with schemas and ground truth.
- Nested examples use $defs/$ref structures, while Table examples demonstrate arrays of objects and visible-text value truncation.
- The schema-echo examples show Gemini 2.5 Pro extracting values correctly, Qwen3-VL 2B producing correct values with invalid structural metadata, and InternVL3.5 1B reproducing the schema without values.
- Resolution examples report 38–40 pp losses for open 8–17B models and losses under 3.5 pp for Gemini models when rendering falls from 200 DPI to 50 DPI.
- Spatial Text adds layout to plain text and yields the largest accuracy gain, while remaining reconstructible from OCR word-level bounding boxes for cost-constrained deployments.