PaperlayerView exact revision on arXiv

Source-linked AI summary

ExStrucTiny: A Benchmark for Schema-Variable Structured Information Extraction from Document Images

Mathieu Sibue, Andres Muñoz Garza, Samuel Mensah, Pranav Shetty, Zhiqiang Ma, Xiaomo Liu, Manuela Veloso

arXiv:2602.12203v1cs.CL

TL;DR

Existing VRDU benchmarks do not adequately test holistic structured extraction across heterogeneous documents, novel entities, and flexible schemas. EXSTRUCTINY addresses this gap with a benchmark built from manual and human-validated synthetic samples, then analyzes open and closed VLMs. The experiments identify persistent challenges in schema adaptation, query underspecification, answer localization, and complex or less structured contexts.

  • Problem

    Existing VRDU datasets inadequately assess holistic, fine-grained structured extraction across heterogeneous documents with novel entities and user-specified schemas, especially in zero- and few-shot settings.

  • Method

    EXSTRUCTINY combines manually annotated and LLM-generated human-validated samples from varied document types to benchmark structured IE with diverse query formats.

  • Results

    The analysis finds persistent challenges with schema adaptation, query underspecification, and answer localization in open and closed VLMs.

  • Takeaways & Limitations

    EXSTRUCTINY provides a benchmark designed to reflect structured extraction needs through JSON outputs, multi-entity and cross-page queries, and unanswerable cases.

  • Takeaways & Limitations

    EXSTRUCTINY is currently English-only, and its schema mapping relies on a slower text-only LLM that may not produce perfect mappings.

Abstract

from arXiv · show

Enterprise documents, such as forms and reports, embed critical information for downstream applications like data archiving, automated workflows, and analytics. Although generalist Vision Language Models (VLMs) perform well on established document understanding benchmarks, their ability to conduct holistic, fine-grained structured extraction across diverse document types and flexible schemas is not well studied. Existing Key Entity Extraction (KEE), Relation Extraction (RE), and Visual Question Answering (VQA) datasets are limited by narrow entity ontologies, simple queries, or homogeneous document types, often overlooking the need for adaptable and structured extraction. To address these gaps, we introduce ExStrucTiny, a new benchmark dataset for structured Information Extraction (IE) from document images, unifying aspects of KEE, RE, and VQA. Built through a novel pipeline combining manual and synthetic human-validated samples, ExStrucTiny covers more varied document types and extraction scenarios. We analyze open and closed VLMs on this benchmark, highlighting challenges such as schema adaptation, query under-specification, and answer localization. We hope our work provides a bedrock for improving generalist models for structured IE in documents.

1 Introduction

Enterprise documents contain information needed for workflows and analytics, but existing VRDU benchmarks inadequately test holistic structured extraction across heterogeneous documents and flexible schemas. EXSTRUCTINY addresses this gap with a unified benchmark and analysis of VLM challenges.

  • Enterprise forms, receipts, and reports encode information needed for storage, automation, compliance, and analytics, while manual review is labor-intensive and error-prone.
  • Text-only IE distinguishes closed, open, and on-demand extraction by how specifically the input query defines the requested structure.
  • Existing VRDU datasets do not adequately evaluate fine-grained structured extraction across heterogeneous documents, novel entities, and user-specified schemas, especially in zero- and few-shot settings.
  • EXSTRUCTINY contributes a document-image benchmark with 304 closed and on-demand query-answer pairs spanning 110 multi-page documents, unifying KEE, RE, and VQA aspects.
  • The benchmark supports analysis of open and closed VLMs on schema adaptation, query underspecification, and answer localization.

2 Related Work

Earlier VRDU benchmarks typically emphasize fixed ontologies, simple extractive questions, or isolated layout elements. EXSTRUCTINY broadens evaluation with flexible, structured queries across varied documents and challenging answer conditions.

  • KEE: KEE datasets target document-specific entity ontologies, while specialist sequence-tagging systems are generally trained for data-rich, domain-specific settings.
  • Benchmark comparison: Table 1 compares EXSTRUCTINY with KEE and VQA test splits using statistics covering document, query, answer, entity, unanswerability, lexical-overlap, and ontology characteristics.
  • VQA: VQA datasets broaden entity coverage but often use questions with document-language overlap, request few related entities, and rarely include unanswerable questions.
  • EXSTRUCTINY: EXSTRUCTINY includes closed and on-demand queries that are multi-entity, low-lexical-overlap, and potentially unanswerable across four document types without a fixed entity ontology.
  • Synthetic data: LLM-based data generation is emerging in VRDU, and EXSTRUCTINY uses this strategy to scale and enrich its benchmark.

3 Methodology

EXSTRUCTINY formalizes structured IE with variable user-specified schemas and builds its benchmark through manually annotated and synthetically generated, human-validated document QA pairs. It also introduces semantic evaluation to handle structurally different but informationally equivalent model outputs.

  • 3.1 Task Formulation: EXSTRUCTINY requires models to organize multiple extracted values according to variable user-specified schemas rather than fixed entity ontologies.
  • 3.1 Task Formulation: Each query-answer pair contains document images, an entity-extraction prompt, and a structured output with extracted text, page indices, and bounding boxes.
  • 3.2 Small-Scale Manual Data Annotation: The benchmark supports closed plain-text, closed schema, and on-demand plain-text queries, with all answers required in JSON format.
  • 3.2 Small-Scale Manual Data Annotation: The dataset draws documents from forms, financial reports, slide decks, and webpage screenshots, while annotations target multiple related entities, missing values, low lexical overlap, multi-page values, and challenging layouts.
  • 3.2 Small-Scale Manual Data Annotation: 102 manually annotated QA pairs were reviewed by a separate annotator, producing high-quality examples spanning varied layouts and extraction challenges.
  • 3.3 Large-Scale Synthetic Data Generation: Synthetic generation uses Gemini-2.5-Flash-Thinking with few-shot examples, iterative prompting, programmatic cleanup, and augmentation for reformulated or unanswerable requests.
  • 3.4 Dataset Quality Validation: Human validation retained 202 synthetic QAs after an average of 25.5 edits per QA, with value changes and additions comprising 47.8% and 23% of modifications.
  • 3.5 Evaluation Metrics for Structured IE: The final benchmark contains 304 QAs over 110 documents, and its evaluation uses semantic schema mapping before measuring extraction, localization, and structured-output performance.

4 Experiments

Experiments show that closed VLMs generally outperform open models, while extraction becomes harder with larger schemas, underspecified or reformulated queries, and difficult document contexts. Models also struggle with spatial grounding and valid structured outputs, although larger open models perform better on several measures.

  • The highest-performing closed model outscored the best open model by more than 18 points across document classes.
  • Performance increased with model size, including Qwen2.5-VL from 38.5 points at 3B parameters to 61.4 points at 72B.Gemma-3 likewise increased from 25 points at 3B to 52.9 points at 27B.
  • Closed-with-schema and on-demand queries underperformed plain-text closed queries, with on-demand queries usually the most difficult.Schema queries request approximately three times as many entities, while on-demand queries require deeper interpretation of underspecified requests.
  • Closed models maintained stable extraction quality beyond 50 expected values, whereas open-model performance declined as extraction length increased.The authors cannot attribute this advantage to a specific difference in training data, model size, or training recipe.
  • Manual QAs scored 13.6% lower than synthetic QAs on average, while the best models achieved 81.2% ANLS on manual data and 78.8% on synthetic data.Neither split was saturated, and the authors found no evidence of bias favoring Gemini-2.5-Flash in synthetic data.
  • ANLS fell by nearly 27% on reformulated queries relative to basic queries, and visual information improved overall performance by approximately 10% over OCR-only input.Models also struggled with unanswerable requests, charts, and free-text contexts.
  • Localization remained difficult: maximum bounding-box IoU was 14.4% and best page accuracy was 84.3%, with correct text not reliably implying correct grounding.These results motivate caution where precise evidence localization is required.
  • All models produced Python-executable structures, but sufficient valid extraction leaves generally required at least 12B parameters; Kimi-VL-A3B-16B reached only 5.1%.Closed models achieved the highest entity recall, while Gemma-3-27B and Qwen2.5-VL-72B were among the strongest open models for structural alignment.

5 Conclusion

EXSTRUCTINY benchmarks structured information extraction from visually rich documents using varied query types, document contexts, and challenging output requirements. Experiments reveal strong but incomplete model capabilities, with persistent difficulties in complex queries and answer localization.

  • EXSTRUCTINY evaluates structured IE across closed, schema-based, and on-demand queries spanning varied document types and contexts.The benchmark expects JSON outputs and includes multi-entity, cross-page, and unanswerable queries.
  • Closed-source VLMs lead in recall and robustness, while larger models generally produce more valid outputs.
  • All evaluated models struggle with intricate queries and answer localization, especially in charts and free text.
  • EXSTRUCTINY is intended to support research on more adaptable, robust, and semantically grounded extraction models for real-world applications.

Limitations

The benchmark has limitations in evaluation metrics, language coverage, and schema mapping. These constraints affect how accurately and efficiently its results can be interpreted.

  • ANLS is suboptimal for numbers and dates because character-based similarity may not accurately reflect extraction quality.The paper identifies these values as 26% of all annotated values.
  • EXSTRUCTINY is currently English-only and does not evaluate information extraction on documents in other languages.
  • Schema mapping relies on a text-only LLM, which is slower than programmatic solutions and may not produce perfect mappings.

Disclaimer

The document is presented for informational purposes by JP Morgan and is not a product of its Research Department. JP Morgan disclaims warranties and liability for the information and states that it is not investment advice or a recommendation.

  • The document was prepared for informational purposes by JP Morgan and its affiliates, not by its Research Department.
  • JP Morgan makes no representation or warranty about the information's completeness, accuracy, or reliability and disclaims liability.
  • The document is not intended as investment research, investment advice, or a recommendation, offer, or solicitation.

A Data Generation Instructions

The data-generation instructions define three extractive query types and prescribe structured, ordered answers with explicit handling for schemas, missing information, and document layout. They also require semantic variation and distinguish absent information from empty values.

  • Query types: The instructions define closed plain-text, closed schema, and on-demand query types for extractive information requests.Closed queries name entities directly or through a schema, while on-demand queries guide extraction through a parent entity.
  • Query types: Entity names in closed queries should not always overlap document wording, encouraging semantic understanding rather than straightforward template matching.
  • Query types: On-demand queries should remain vague about child entities so the answering model proposes a document-informed schema.
  • Answer format: Answers should use nested dictionaries or lists, preserve requested entity order, and represent extraction leaves with verbatim text, page, bounding-box, and context fields.
  • Answer format: The instructions preserve reading order, avoid numeric suffixes for repeated line-item keys, encode checkbox states as Yes or No, and keep separate on-demand extractions separate when they are separate document items.
  • Answer format: Missing requested information uses null extraction fields or an empty answer, whereas document fields marked N/A are extracted as empty text values.

B Dataset Comparison Metrics

The comparison harmonizes KEE and VQA datasets into QA-level statistics under multiple recasting schemes. ExStrucTiny differs through broader query and answer structures, greater entity diversity, and lower lexical overlap.

  • Comparison setup: KEE and VQA are harmonized into a common QA-level view before metrics are computed in Table 1.This common representation addresses their different annotation formats.
  • KEE recasting: KEE test sets are recast as either one document-level query requesting all k entities or k entity-level queries, one per entity.The corresponding QA counts are n and nk, respectively.
  • VQA adaptation: VQA analysis retains extractive questions whose annotated answers appear verbatim in document text and estimates requested entities by parsing each query.Examples distinguish single-entity and multi-entity questions.
  • Statistics: The analysis measures document, query, and answer lengths alongside entity counts, extracted-value counts, missing-entity rates, lexical overlap, and normalized ontology size.Missing-entity rate counts queries where documents lack at least one requested entity.
  • Interpretation: ExStrucTiny offers longer multi-entity queries, more expected answer values, higher entity diversity, realistic missing-entity rates, and lower lexical overlap than the compared datasets.KEE entity-level recastings instead have short single-entity queries, few answers, high overlap, and very low normalized ontology size; VQA has near-1 normalized ontology size but mostly single-entity, single-value queries.

C Dataset Annotation Details

ExStrucTiny combines manually annotated and large-scale synthetic QA data with validation and targeted sampling procedures. Synthetic validation achieved 96.2% inter-validator agreement in the reviewed sample.

  • Manual annotation: Manual annotation used an intermediate validator review after four annotators completed approximately half their QAs, followed by revisions and comprehensive final review.The process incorporated feedback before annotators completed the remaining questions.
  • Sampling: Target-guided importance sampling normalizes metadata features, samples target vectors, and assigns nearest examples without replacement to form a subsample.The method uses distances in normalized feature space, including regularized Mahalanobis distance as an example.
  • Synthetic generation: Nearly 3,000 synthetic QAs were generated across 350 test documents, with Gemini-2.5-Flash Thinking proposing nine QAs per document.Programmatic fixes repaired JSON formatting and reconciled bounding boxes against OCR-processed source documents.
  • Validation: 96.2% inter-validator agreement was estimated from a stratified sample of 60 synthetic QAs previously validated by other validators.Because of resource constraints, the full synthetic set was not revalidated from scratch.

D Evaluation Metric for Structured IE

The structured IE metric maps flattened ground-truth and predicted answer leaves before comparing them. A few-shot text-only LLM performs holistic key-value matching while allowing unmatched fields and limited one-to-many mappings.

  • Matching rules: The mapper primarily matches key-value pairs by their values, using key meaning and numeric path components to resolve parent-object correspondences.The query is also supplied as a string to the mapping task.
  • Mapping flexibility: Most matches are one-to-one, but one ground-truth field may map to multiple prediction fields when their values concatenate into the ground-truth value.The converse allows multiple ground-truth fields to map to one prediction field when a shared prediction value contains both.
  • Global matching: The mapper evaluates each local match in the context of the overall mapping rather than greedily optimizing only a few ground-truth keys.This reflects the instruction to match as many ground-truth key-value pairs as possible while respecting reuse constraints.
  • Unmatched fields: Ground-truth fields with no equivalent prediction field remain unmatched and receive None in the output mapping.The required output maps all ground-truth keys to prediction-key tuples or None.

E Text Extraction Results: Breakdown by Answer Context

Table 8 breaks model performance down by answer context, covering five document-answer formats. The listed formats include free text, key-value fields, templated items, charts, and checkboxes.

  • Answer contexts: Table 8 reports model performance separately for free-text answer contexts.
  • Answer contexts: The breakdown includes key-value fields, templated items with table rows, charts, and checkboxes alongside free text.
  • Comparison scope: The table’s comparison unit is answer format, enabling performance comparisons across these five context types.

F VLM Benchmarking Prompt

The benchmark prompt requires extractive answers from document images in structured formats, supporting plain-text and schema-based queries across three query types. It also specifies ordering, nesting, localization, and handling of missing or empty values.

  • Input query: Queries are strings that can be plain text or empty structured schemas to populate with extracted values.The prompt defines three extractive query types: closed IE with plain text, closed IE with schema, and on-demand IE.
  • Query types: Closed IE queries explicitly identify the entities to extract, either in natural-language text or through schema key names.Plain-text closed IE names entities in a sentence, while schema-based closed IE provides entity names in a structured schema.
  • Query types: On-demand IE queries name a parent entity rather than every child field, requiring the model to propose a document-informed schema for related entities.The example requests dependents and returns nested name and age fields.
  • Output structure: Extractions are ordered by queried-entity order and then document reading order, with key naming determined by the input schema, query wording, document tokens, or suggested semantic names.Numerical suffixes are unnecessary for repeated templated items when items already follow document reading order.
  • Output structure: Structured answers use nested dictionaries or lists for hierarchical and templated entities, with leaf extraction dictionaries containing text, page, and bounding-box fields.Line items group fields belonging to the same underlying parent entity, while leaf nodes preserve extracted document text and localization metadata.
  • Special cases: Missing requested values use null text, page, and bounding-box fields, whereas explicitly empty fields such as N/A use empty text with their document location; checkboxes map to Yes or No.If all requested keys are absent, the answer is an empty list.
Loading 2602.12203v1…