Source-linked AI summary
Identify, Locate, Link: End-to-End Key-Value Extraction from Document Images
A. Said Gurbuz, Ahmed Nassar, Christoph Auer, Maksym Lysak, Lucas Morin, Matteo Omenetti, Tim Strohmeyer, Panagiotis Vagenas, Nikolaos Livathinos, Michele Dolfi, Peter Staar
TL;DR
Structured extraction from documents traditionally depends on OCR cascades, while key-value extraction also requires spatial localization and relation linking. The paper fine-tunes a compact SmolDocling model for single-pass OCR-free extraction, extends DocTags and training data, and evaluates spatial correctness. Across three datasets, it outperforms larger zero-shot VLM baselines while using substantially fewer parameters and less inference time.
Problem
Key-value extraction must jointly understand text, layout, and relationships, but existing OCR-based cascades introduce error propagation, language dependence, engineering complexity, and fragmented outputs.
Method
The paper fine-tunes a 256M SmolDocling VLM for single-pass OCR-free extraction, extending DocTags with key-value tags and combining synthetic form filling with graph-based crops.
Results
Across FUNSD, XFUND, and a private dataset, the model outperforms larger zero-shot VLM baselines under layout-aware evaluation while being 27× smaller than Qwen2.5-VL (7B) and over 5× faster at inference.
Takeaways & Limitations
The approach supports spatially grounded key-value evidence for deployment directly on document images, including disambiguation and visual retrieval workflows.
Takeaways & Limitations
The model does not match OCR/text-layout encoder models with pre-extracted text and boxes, and available spatial key-value datasets limit training diversity and evaluation coverage.
Abstract
from arXiv · showhide
Document processing pipelines traditionally cascade optical character recognition (OCR) engines with downstream models for structured information extraction, leading to multi-stage error propagation. We fine-tune SmolDocling, a compact 256M-parameter vision-language model (VLM), to perform end-to-end key-value extraction directly from document images, jointly solving identification, localization, and association in a single pass without OCR preprocessing. We extend DocTags with specialized key, value, region, and link tags, enabling many-to-many relationships in a unified output sequence. To address data limitations, we design an augmentation pipeline combining synthetic form filling and graph-based crops that preserve complete key-value subgraphs. We further introduce a layout-aware evaluation framework extending text matching with spatial bounding box verification. On FUNSD, XFUND, and a large-scale private dataset, our model outperforms larger zero-shot VLM baselines under layout-aware evaluation, while being 27 times smaller than Qwen2.5-VL (7B) and over 5 times faster at inference. The model weights will be released publicly after publication.
1 Introduction
Key-value extraction must jointly identify keys and values, localize them, and link related regions, but conventional OCR cascades introduce practical limitations. This work addresses the task end to end with an extended structured representation, targeted data augmentation, and layout-aware evaluation.
- Task definition: Key-value extraction jointly requires semantic identification, bounding-box localization, and relation association.Unlike OCR, the task must connect textual content, spatial layout, and semantic relationships.
- Task definition: Many-to-many links form directed graph structures in which keys can map to multiple values and values can serve as keys to sub-options.These relationships are illustrated as linked regions rather than isolated key-value pairs.
- Limitations of prior pipelines: OCR-based cascades introduce error propagation, language dependence, engineering complexity, and fragmented outputs requiring post-processing.They first extract text and layout, then apply models for classification and relation extraction.
- Proposed approach: The proposed 256M vision-language model performs identification, localization, and association in one forward pass without external OCR or layout analysis.The approach extends DocTags with specialized region, key, value, and directed-link tags supporting unique identifiers and complex relationships.
- Results: Across FUNSD, XFUND, and a private dataset, the model outperforms larger zero-shot VLM baselines while being 27× smaller than Qwen2.5-VL (7B) and over 5× faster.Evaluation uses layout-aware metrics that combine text matching with spatial bounding-box verification at IoU ≥0.7.
2 Related Work
Prior work spans OCR-based multimodal and structure-aware models, decoder-only architectures, and OCR-free end-to-end systems. This paper extends SmolDocling’s compact document-conversion framework with explicit key-value relationship tags, data augmentation, and spatially verified evaluation.
- OCR-based multimodal transformers: OCR-based multimodal transformers combine OCR-extracted text with visual and layout features, with LayoutLM variants progressively unifying these modalities.LayoutXLM extends the paradigm to multilingual documents through pre-training across seven languages.
- Structure-aware models: Structure-aware models add explicit spatial inductive biases through token adjacency graphs, graph convolution, or geometry-focused pre-training.These methods target geometric relationships that standard attention mechanisms learn only implicitly.
- Decoder-only models: Decoder-only models inject layout awareness through spatial attention or projected bounding-box tokens but still require pre-extracted text.They produce natural-language or token-tag outputs while leveraging large generative architectures.
- End-to-end approaches: OCR-free systems such as Donut provide structured outputs but do not explicitly model spatial grounding and complex key-value link graphs.This limits their representation of one-to-many and many-to-many relationships.
- End-to-end approaches: SmolDocling provides normalized bounding-box tokens and a compact 256M model, but its original DocTags vocabulary lacks key-value relationship tags.It was designed for general document conversion, including tables, text blocks, and captions.
- End-to-end approaches: This work extends SmolDocling with four specialized key-value tags, synthetic form filling, graph-based crops, and layout-aware evaluation.The evaluation verifies spatial correctness alongside text matching.
3 Method
The method formulates key-value extraction as joint identification, localization, and association, producing structured DocTags directly from document images. It combines a compact vision-language architecture, key-value markup extensions, targeted augmentation, and layout-aware evaluation.
- Problem Definition: Key-value extraction jointly identifies keys and values, predicts their bounding boxes, and establishes directed many-to-many relationships.The relationships form a directed graph over keys and values.
- DocTags Representation: DocTags represents document elements with normalized bounding-box tokens and is extended with region, key, value, and directed-link tags.Unique identifiers allow links to connect elements in a unified output sequence.
- DocTags Representation: The link-in-key strategy places multiple link tokens inside a key, while inference parses well-formed key and value spans into graph cells.The method also evaluates link-in-value and spatial-ordering alternatives in ablations.
- Model Architecture: The 256M-parameter SmolDocling model combines a 93M-parameter SigLIP vision encoder with a 135M-parameter SmolLM2 decoder to generate DocTags sequences.Visual patch embeddings are projected into the decoder token space before autoregressive generation.
- Training: Training uses cross-entropy over DocTags sequences and also tests λ = 5 weighting for structural key-value tokens.The weighted variant is intended to reduce domination by frequent content tokens and is evaluated through ablations.
- Data and Augmentation: Training data combines annotated FUNSD, XFUND, and DocLayNetV2 with synthetic form filling, graph-based crops, and image-preserving transformations.Graph-based crops preserve complete key-value subgraphs, while synthetic filling supplies values for empty form fields.
- Evaluation Framework: Layout-aware evaluation extends text and label matching with spatial overlap verification for entities and relations.The framework uses globally optimal Hungarian matching for compatible entity assignments.
4 Experiments
The experiments evaluate end-to-end SmolDocling against VLM and contextual baselines using layout-aware metrics, then test data and model configurations through ablations and qualitative analysis.
- Evaluation protocol: Layout-aware evaluation combines text, label, and spatial agreement, requiring IoU ≥0.7 for SmolDocling and Qwen.Models without localization use text-only evaluation, while OCR/text-layout models are not directly comparable because spatial positions are inputs.
- VLM comparison: Our 256M-parameter model outperforms all zero-shot VLMs on relation extraction and is competitive on entity recognition across FUNSD and XFUND.The comparison avoids external OCR and layout systems for the model under evaluation.
- Comparison with Qwen2.5-VL: 0.27s/sample versus 1.40s/sample makes the model over 5× faster than Qwen2.5-VL on A100 GPUs.Qwen2.5-VL is the only baseline with box prediction and is therefore compared under the same layout-aware metric.
- Metric sensitivity: Relaxing the localization gate raises FUNSD scores from 0.514/0.275 to 0.679/0.376 and XFUND scores from 0.412/0.206 to 0.516/0.289 for SER/RE.The stricter metric is retained because it better reflects spatial grounding.
- Data pipeline ablation: Crop augmentation improves XFUND RE by ∼41%, from 0.122 to 0.172, while synthetic form filling reaches 0.206 XFUND RE and 0.543 FUNSD SER.Image transformations help noisy FUNSD scans but slightly reduce scores on clean DocLayNetV2 documents.
5 Conclusion
The paper concludes that a compact VLM can jointly extract, localize, and link key-value information directly from document images. Its strongest practical scope is OCR-free, spatially grounded deployment, while dataset and serialization limitations remain.
- Conclusion: A 256M-parameter model performs end-to-end key-value extraction by jointly solving identification, localization, and association in one forward pass.The approach is designed for direct document-image processing without external OCR or layout components.
- Conclusion: Crop augmentation and synthetic form filling provide large gains, while spatial ordering and link placement trade off entity recognition against relation extraction.No single configuration dominates across all metrics.
- Practical scope: The approach is most suitable for image-direct deployment that avoids external OCR/layout components and returns spatially grounded evidence.The stated applications include disambiguation and visual retrieval workflows.
- Limitations: Limited spatial key-value datasets restrict training diversity and evaluation coverage, while coordinate serialization is suboptimal for complex multi-column layouts.The model also does not yet match OCR/text-layout encoders that receive pre-extracted text and boxes.
- Future work: Scaling to 1B–7B parameters and adding reading-order prediction are proposed to address capability and complex-layout limitations.The compact model is also identified as a candidate for on-device deployment where latency and privacy constrain cloud inference.