PaperlayerView exact revision on arXiv

Source-linked AI summary

From Pixels to Pairs: A Comprehensive Benchmark of LLM-Based Key-Value Extraction in Noisy Document Settings

Zahra Anvari, Vassilis Athitsos

arXiv:2609.17538v1cs.CLcs.CV

TL;DR

Reliable KVP extraction by text-only LLMs under realistic OCR noise remains insufficiently understood. This paper benchmarks open-source instruction-tuned models across datasets, OCR engines, and input regimes using a unified protocol. The results show strong semantic extraction from clean text, but OCR corruption narrows model differences and makes input fidelity the limiting factor.

  • Problem

    Existing evaluations provide limited evidence about text-only LLM extraction under realistic OCR-generated inputs, despite OCR corruption affecting downstream document processing.

  • Method

    The paper benchmarks Gemma, Mistral, Qwen2.5, LLaMA 3, and DeepSeek on FUNSD, CORD, and SROIE using Gold-text, three OCR engines, and a unified evaluation protocol.

  • Results

    LLMs perform strongly on Gold-text, but OCR inputs reduce Exact Match and Value F1, with model differences narrowing as corruption increases.

  • Takeaways & Limitations

    Real-world text-only extraction depends on both semantic reasoning and preservation of textual fidelity under OCR corruption.

  • Takeaways & Limitations

    Flattened text loses spatial relationships, hierarchical grouping, and visual context needed by many document-understanding tasks.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used for structured information extraction from documents, yet their behavior under realistic OCR noise remains poorly understood. We present a systematic benchmark of open-source instruction-tuned LLMs for key-value pair (KVP) extraction under both clean-text and noisy OCR conditions. We evaluate representative decoder-only models (Gemma, Mistral, Qwen2.5, LLaMA 3, and DeepSeek) on the FUNSD, CORD, and SROIE benchmarks using both Gold-text annotations and OCR outputs from PaddleOCR, EasyOCR, and Tesseract. A unified evaluation protocol isolates the effects of input quality, model design, and prompting under consistent conditions. The results show that modern LLMs act as strong semantic extractors when high-quality text is available, in some cases approaching supervised layout-aware systems. Under OCR noise, however, performance degrades substantially and performance gaps between models narrow as input corruption increases. Across all datasets, extraction performance is governed by two factors: semantic reasoning over text and preservation of textual fidelity under OCR noise. While larger models improve results on clean text, these gains diminish under noisy inputs, where OCR quality becomes the dominant factor. We also identify recurring failure modes, including key-value misalignment, hallucination, and numeric corruption. Our findings highlight the gap between clean-text evaluation and real-world deployment, emphasizing the need to jointly improve OCR quality, structural reasoning, and LLM-based semantic modeling.

1 Introduction

This paper addresses the limited understanding of text-only LLM extraction under realistic OCR noise by introducing a controlled, reproducible benchmark. It shows that model strength helps on clean text, while input quality increasingly governs performance as OCR degrades.

  • The benchmark evaluates instruction-tuned open-source LLMs across FUNSD, SROIE, and CORD using Gold-text and OCR outputs from three engines.Models include Gemma, Mistral, Qwen2.5, DeepSeek, and LLaMA 3.
  • Strong models improve extraction on clean text, but their gains become less pronounced as input quality degrades.This comparison is evaluated across multiple datasets and input regimes.
  • The unified protocol combines consistent prompting, deterministic decoding, and non-semantic canonicalization to isolate model capability from input quality.Evaluation uses Key Recall, Exact Match, and Value-level F1 to capture complementary aspects of extraction.
  • Results show strong clean-text extraction, lower performance under OCR, narrowing model differences, and dataset-dependent benefits from few-shot prompting.Few-shot gains are reported to depend on dataset regularity and input quality.
  • The benchmark enables controlled comparison across models, prompting strategies, and document collections under realistic input conditions.It is also intended as a reproducible framework for future research on LLM-based document understanding.

2 Related Work

Prior document-extraction systems use rules, layout-aware models, or multimodal pipelines, but their robustness and comparability under OCR noise remain limited. This work positions a unified text-only LLM benchmark as a controlled way to study those conditions across diverse documents and OCR engines.

  • Traditional rule-based systems are brittle under layout changes, OCR artifacts, and domain shifts because fixed patterns limit semantic generalization.They also require extensive manual adaptation across document types.
  • Layout-aware models achieve strong benchmark performance but depend on accurate OCR boxes, annotated corpora, and more complex vision–language pipelines.Their reliance on explicit layout signals can limit robustness when structure is degraded.
  • OCR corruption directly causes extraction failures through character substitutions, token fragmentation, line reordering, and numeric errors.Such corruption is a primary failure source in document-processing pipelines.
  • The benchmark varies model families, document types, OCR quality, prompting, and canonicalization under one reproducible protocol.It evaluates forms and receipts with Tesseract, EasyOCR, and PaddleOCR to isolate input-quality effects.
  • Flattening OCR text removes spatial relationships in FUNSD, producing frequent key–value misalignment and reduced Exact Match.This especially affects fields whose associations depend on positional cues rather than explicit lexical markers.
  • SROIE reduces structural ambiguity through a small schema but remains sensitive to numeric and address normalization errors.Semantically correct fields outside the annotated schema are ignored during evaluation.

3 Methodology and Experimental Setup

The benchmark evaluates text-only KVP extraction with open-source instruction-tuned LLMs across clean and OCR-derived inputs, using fixed prompting, decoding, and evaluation procedures. It covers diverse document structures and OCR conditions to support controlled, reproducible comparison.

  • 3.1 Task Formulation: KVP extraction maps OCR-generated document text into structured key–value pairs, where keys denote semantic fields and values are their corresponding contents.OCR text may contain character errors, token fragmentation, and lost structural ordering.
  • 3.2 Models Evaluated: The benchmark evaluates six instruction-tuned decoder-only LLMs across diverse document types and OCR conditions using a unified experimental pipeline.Models span the Gemma, Mistral, Qwen2.5, DeepSeek, and LLaMA 3 families; datasets include FUNSD, CORD, and SROIE.
  • 3.3 Prompting Strategy: A shared prompt defines the task, enforces JSON output, discourages hallucinated fields, and requires models to infer keys from input text.Prompt sensitivity is tested with zero to three disjoint in-context examples from each dataset’s training split.
  • 3.4 Output Canonicalization: Malformed model outputs are handled through deterministic parsing and recovery that reads only already-generated key and value fields without re-inspecting the source document.Unrecoverable outputs are treated as empty, keeping evaluation focused on model generation rather than heuristic extraction.
  • 3.5 Datasets and Splits: The study compares Gold-text inputs with OCR-text generated by PaddleOCR, EasyOCR, and Tesseract to isolate semantic extraction from OCR robustness.Gold-text is derived from ground-truth annotations, while OCR-text captures varying levels of realistic noise.
  • 3.6 Inference Settings: All models use identical greedy decoding, generation limits, chat-template handling, and fixed pipeline components to ensure reproducible comparisons.No model-specific prompt tuning, decoding adjustment, or reranking is applied.

4 Evaluation Methodology

The evaluation protocol measures text-only KVP extraction under clean and OCR-degraded inputs using complementary metrics and fixed normalization rules. It emphasizes field detection, exact pair correctness, and partial value recovery while preserving document-level comparability.

  • 4.1 Evaluation Overview: The protocol evaluates JSON-predicted key–value pairs against annotations using Key Recall, Exact Match, and token-level Value F1.These metrics respectively capture field detection, strict pair correctness, and approximate value recovery under textual variation.
  • 4.2 Matching and Normalization: Predicted keys absent from the annotation schema are ignored, avoiding penalties for plausible additional fields and supporting comparison across datasets with incomplete annotations.This design is especially relevant to SROIE, whose annotations cover only a subset of document information.
  • 4.2 Matching and Normalization: Keys are matched by exact equality after lowercasing, whitespace trimming, and repeated-space collapsing, while values are compared at the whitespace-token level.This normalization reduces sensitivity to superficial formatting differences before matching and scoring.
  • 4.4 Metrics: Exact Match counts a pair only when both normalized key and value match exactly, providing a strict extraction-accuracy measure.Value F1 instead grants partial credit when predicted values preserve most ground-truth content despite formatting or OCR noise.
  • 4.5 Macro Averaging: Dataset-level scores use macro averaging so each document contributes equally rather than allowing long or densely annotated documents to dominate.The pipeline fixes normalization, matching, and scoring procedures and stores per-document predictions for reproducibility and error analysis.
  • 4.3 Input Regimes: Gold-text and noisy OCR tracks isolate the effect of OCR errors by comparing clean annotation-derived text with outputs from PaddleOCR, EasyOCR, and Tesseract.The noisy track introduces token fragmentation, numeric corruption, and loss of structure.

5 Results

Across FUNSD, SROIE, and CORD, Qwen2.5–7B is generally the strongest model, while OCR corruption reduces extraction quality and narrows model differences. Few-shot prompting often improves results, but its gains depend on dataset structure and input quality.

  • Qwen2.5–7B is consistently among the strongest models, achieving the best reported 0-shot Gold-text Value F1 across all three datasets and strong OCR performance.It also provides the best 0-shot PaddleOCR result on FUNSD, SROIE, and CORD, while remaining strongest overall in several dataset-specific analyses.
  • Model differences narrow as OCR noise increases, most sharply on CORD under Tesseract, where all models fall below 0.30 Value F1.This pattern contrasts with clearer model separation under Gold-text, especially on SROIE and CORD.
  • Key Recall generally exceeds EM because models can identify candidate fields yet assign values to the wrong fields, especially in forms and dense receipt regions.FUNSD shows persistent key–value alignment errors, while CORD contains repeated numeric content that increases assignment difficulty.
  • OCR severity varies by dataset and engine: PaddleOCR is consistently cleanest, Tesseract most corrupt, and CORD is especially difficult.On CORD, Tesseract approaches a WER of 1.0, while SROIE corruption includes broken numeric fields and fragmented addresses.
  • OCR lowers EM and Value F1 more than broad field detection, with the largest declines under EasyOCR and Tesseract.On CORD, Qwen2.5–7B drops from 0.8267 Value F1 under PaddleOCR to 0.2679 under Tesseract.
  • Few-shot prompting improves performance across many settings, with gains depending on document type and input quality and often saturating after a few examples.SROIE shows especially strong and stable gains across 1–3 shots, whereas CORD gains are largest from 0-shot to 1-shot and then level off.

6 Discussion

The discussion identifies a two-stage bottleneck: LLMs reason effectively over intact text, but OCR corruption shifts errors toward alignment, value fidelity, and structural recovery. Scaling and prompting help mainly when input structure remains reliable, while text-only pipelines remain limited by lost layout information.

  • 6 Discussion: OCR noise shifts extraction from semantic reasoning toward fragile key–value alignment and value corruption, making input fidelity the dominant bottleneck.Key Recall can remain high while Exact Match and Value F1 fall because keys are identified without correctly aligning their values.
  • 6.1 Separation Between Semantic Capacity and Input Fidelity: Gold-text inputs yield strong extraction, but OCR corruption disrupts lexical boundaries, ordering, and numeric content across document types.FUNSD is especially affected by lost spatial relationships, SROIE by numeric corruption, and CORD by dense-region ambiguity.
  • 6.2 Model Scaling: Larger models improve Gold-text performance, yet architectural differences compress as OCR noise increases because scaling cannot reliably restore distorted upstream information.Scaling primarily strengthens semantic reasoning and only partially compensates for minor inconsistencies.
  • 6.3 Prompting Effects: Few-shot prompting improves performance most on regular, short documents such as SROIE, but gains diminish when OCR disrupts key spans or structural relationships.The examples help exploit recurring patterns rather than recover missing structure.
  • 6.5 Structural Limitations of Text-Only Pipelines: Flattened text loses spatial, hierarchical, and visual cues, producing failures such as misalignment, hallucination, address fragmentation, and format instability.These failures expose the gap between text-only pipelines and layout-aware or multimodal approaches.
  • 6.7 Evaluation and Dataset Effects: Text-only LLMs and layout-aware models are complementary: robust deployment requires upstream quality, structural recovery, validation, and semantic interpretation rather than model choice alone.Metric interpretation also depends on dataset characteristics and input quality, because EM is formatting-sensitive while Value F1 tolerates some variation.

7 Conclusion and Future Work

The paper concludes that instruction-tuned LLMs are strong semantic extractors on preserved text but remain constrained by OCR fidelity in realistic settings. It therefore points toward reproducible evaluation and hybrid systems that combine semantic modeling with layout, OCR, and structural mechanisms.

  • 7 Conclusion: LLMs recover many annotated fields from Gold-text inputs, but OCR corruption consistently lowers Exact Match and Value F1 and shifts errors toward alignment and value fidelity.The largest degradation occurs in structurally complex and numerically dense documents such as CORD.
  • 7 Conclusion: Scaling and few-shot prompting improve results conditionally, with diminishing benefits as OCR noise increases or structural patterns become inconsistent.Neither scaling nor prompting alone overcomes upstream degradation.
  • 7 Conclusion: Text-only LLM extraction is powerful but incomplete because models lack explicit spatial structure and remain sensitive to OCR-induced errors.Reliable real-world pipelines will likely integrate complementary components instead of relying on prompt-based extraction alone.
  • 7 Conclusion: The standardized framework uses unified prompting, deterministic decoding, non-semantic canonicalization, and consistent cross-dataset benchmarking to support reproducible comparisons.The protocol isolates model capability from input quality and provides a foundation for future robust-extraction research.
  • Future Work: Future work should combine layout-aware or vision–language components with LLM semantic decoders and address OCR degradation through correction, numeric repair, and structure recovery.The proposed directions target the missing structural and input-fidelity mechanisms identified by the benchmark.
  • Future Work: More realistic evaluation requires broader annotation schemas, diverse document types, and controlled degradation levels that better represent deployment conditions.Systematic analysis of performance across input quality could better align model improvements with real-world gains.

A Hyperparameters and Implementation Details

The appendix provides implementation details intended to make the experiments fully reproducible.

  • A Hyperparameters and Implementation Details: The appendix documents implementation details necessary for reproducing the experiments.

A.1 Model Configurations

The model configuration uses public instruction-tuned checkpoints without fine-tuning or parameter updates.

  • A.1 Model Configurations: The experiments use publicly available HuggingFace checkpoints for Mistral, Qwen2.5, DeepSeek, and LLaMA 3.
  • A.1 Model Configurations: All evaluated models remain in their instruction-tuned variants without fine-tuning or parameter updates.

A.2 Decoding Parameters

All models use identical deterministic decoding settings, enabling consistent outputs across runs.

  • The evaluation fixes decoding behavior rather than allowing model-specific sampling choices.
  • Greedy decoding with a 256-token maximum generation length standardizes inference across models.These settings eliminate stochastic variation and support consistent outputs across runs.
  • Deterministic decoding improves comparability by removing run-to-run stochastic variation.

A.3 Prompt Template

All models and datasets use one shared prompt structure for key–value extraction, JSON formatting, hallucination control, and optional few-shot guidance.

  • The unified prompt defines key–value extraction, requires JSON output, and adds an anti-hallucination constraint.
  • Optional few-shot examples provide one to three demonstrations within the common prompt template.
  • Applying the same prompt structure across models and datasets enables fair comparison.The full template and examples are available in the accompanying code repository.

A.4 OCR Engines

The OCR evaluation uses direct OCR outputs and a deterministic JSON-first parsing pipeline, with shared metrics and reproducible inference conditions.

  • OCR outputs are used directly without correction, filtering, or post-processing.PaddleOCR is identified as the primary evaluation system among the three OCR systems evaluated.
  • The parser first attempts JSON parsing and otherwise recovers only explicitly generated key and value fields.It performs no rule-based source extraction, heuristic completion, semantic correction, or post-hoc field inference.
  • Key Recall, Exact Match, and Value F1 use a shared normalization and matching framework across datasets.
  • Experiments use a unified transformers-based inference pipeline with independently processed documents stored in JSONL format.The setup supports reproducibility and downstream analysis.
  • Single-GPU evaluation processes 7–8B models sequentially, while larger models may use CPU offloading that increases latency without affecting output quality.
Loading 2609.17538v1…