Source-linked AI summary
A Neural Model for Text Localization, Transcription and Named Entity Recognition in Full Pages
Manuel Carbonell, Alicia Fornés, Mauricio Villegas, Josep Lladós
TL;DR
Document-image information extraction combines localization, transcription, and named-entity recognition, yet remains challenging and is traditionally handled with separate methods. This paper unifies the three tasks in an end-to-end full-page model. The authors conclude joint modeling suits strongly interdependent tasks, but not cases dominated by one difficult, unrelated task.
Problem
Document-image information extraction remains challenging, and its localization, transcription, and entity-labeling tasks are traditionally handled separately despite their interdependence.
Source: §1 · p. 1Method
The paper explores a unified end-to-end architecture for combining localization, transcription, and named-entity recognition on full pages.
Source: §1 · p. 1Results
Joint models are suitable when tasks are strongly interdependent, but not for unstructured documents where one independent task is the main difficulty.
Source: §6 · p. 7Takeaways & Limitations
Task interdependence can guide whether to use a joint model or a separate approach for a difficult, unrelated task.
Source: §6 · p. 7A unified model can be limited when one task is much harder and unrelated to the others.
Source: §6 · p. 7
Abstract
from arXiv · showhide
In the last years, the consolidation of deep neural network architectures for information extraction in document images has brought big improvements in the performance of each of the tasks involved in this process, consisting of text localization, transcription, and named entity recognition. However, this process is traditionally performed with separate methods for each task. In this work we propose an end-to-end model that combines a one stage object detection network with branches for the recognition of text and named entities respectively in a way that shared features can be learned simultaneously from the training error of each of the tasks. By doing so the model jointly performs handwritten text detection, transcription, and named entity recognition at page level with a single feed forward step. We exhaustively evaluate our approach on different datasets, discussing its advantages and limitations compared to sequential approaches. The results show that the model is capable of benefiting from shared features for simultaneously solving interdependent tasks.
1. Introduction
Document-image information extraction requires localization, transcription, and semantic labeling, but these tasks are interdependent and remain challenging to automate. The paper proposes a unified full-page approach and evaluates where joint modeling helps.
1. Introduction
- Information extraction from document images remains challenging despite neural-network improvements, and human intervention is still required,[2].The process includes localizing, transcribing, and annotating text, and varies across domains.Source: §1 · p. 1
- Strong task interdependence motivates unifying localization, transcription, and named-entity recognition, whose interactions can affect one another’s performance.For handwritten text, recognition depends on segmentation accuracy, while recognized words can in turn boost segmentation.Source: §1 · p. 1
- The paper explores an end-to-end full-page architecture combining all three information-extraction tasks across different document scenarios.The evaluation considers geometric context, layout regularity, and the strength of named entities.Source: §1 · pp. 1–2
- The authors report this as the first method for end-to-end information extraction in full handwritten pages, with a baseline for the IEHHR competition dataset.They describe the model as versatile across information-extraction scenarios and beneficial when tasks are strongly interdependent.Source: §1 · p. 2
2. Related Work
Related work treats localization, transcription, and named-entity recognition with separate methods as well as pairwise joint models. The review motivates integrating all three while noting how segmentation quality and contextual information shape prior approaches.
2. Related Work
- One-stage detectors such as SSD [8] and YOLO [9] prioritize speed, while focal loss addresses class imbalance and improves their accuracy-speed trade-off [10].The review contrasts these with two-stage methods, including R-CNN and Faster-RCNN.Source: §2 · p. 2
- HTR methods jointly segment and recognize at line level to address Sayre’s paradox, but inaccurate line or paragraph segmentation can still harm recognition [11],.A prior full-page model predicts text boxes and transcription together using RoI pooling, whereas another approach reports stronger transcription with separately trained networks.Source: §2 · p. 2
- For named-entity recognition, LSTM and CRF methods model sequential word patterns [14],,, while image-based CNN approaches can classify words without transcription.CNN-LSTM and joint recognition approaches use context and, [3],, but the reviewed methods lack bi-dimensional context analysis [3],.Source: §2 · p. 2
- Prior work suggests integrating localization, transcription, and entity recognition, as existing joint approaches combine only pairs of tasks or address controlled layouts [3],, [21].The review distinguishes sequential entity labeling from scanned-document settings, where context around a word can matter.Source: §2 · p. 2
3. Methodology
The method uses shared page features and task-specific branches to localize text, transcribe boxes, and assign semantic tags. Their predictions and losses are integrated in a single architecture.
3. Methodology
- Shared convolutional page features feed separate branches for classification, localization, text recognition, and named entity recognition.The model combines their losses through backpropagation across the whole network.Source: §3 · p. 2, Figure 1 · p. 3
3.1. Shared features
- ResNet18 with an FPN supplies five feature maps at 1/8 through 1/128 of input resolution; a lighter two-block variant converged slower and had higher IAM validation error.The authors chose an intermediate-depth model for the combined tasks.Source: §3.1 · pp. 2–3
3.2. Classification branch
- The classification branch predicts one-hot object-class probabilities at each location and uses cross-entropy; neighborhood context may miss dependencies between distant image regions.The authors chose this classification approach over IoU regression for its performance across varied datasets.Source: §3.2 · p. 3
3.3. Regression branch
- The regression branch predicts offsets from predefined anchors to produce positive-box coordinates, using mean squared error.The anchors combine the listed ratios and scales with a base size of 32.Source: §3.3 · p. 4
3.4. Feature pooling
- After thresholding anchor scores and suppressing overlaps, the model applies RoI pooling to shared page features for each selected box.All five FPN levels serve detection, while recognition and named entity branches use the least-downsampled features for higher resolution.Source: §3.4 · p. 4
3.5. Text recognition branch
- The recognition branch produces character probabilities across pooled-feature columns and uses CTC to score the ground-truth text sequence.CTC sums the probabilities of possible paths; its loss is backpropagated together with classification and regression losses.Source: §3.5 · p. 4
3.6. Semantic annotation branch
- For simple layouts, pooled text-box features are sorted in reading order, padded, and processed as a sequence to predict semantic tags.The branch uses two convolutions, a fully connected network, and cross-entropy loss.Source: §3.6 · p. 4
3.7. Receptive field calculation
- The shared feature map has a receptive field of 1559 pixels, so predictions on 1250×1760 inputs mostly depend on page-wide content, with greater weight near the field center.The authors note that the context may be limiting when related entities are far apart.Source: §3.7 · p. 5
4. Datasets
The experiments use three datasets spanning handwritten marriage records, refugee registration forms, and synthetic pages, with distinct data and annotation challenges.
4. Datasets
- Only a few publicly annotated datasets are available, so the study evaluates its approach on three datasets.Table 4 reports page, word, out-of-vocabulary word, and partition counts.Source: §4 · p. 5, Table 4 · p. 5
4.1. IEHHR
- IEHHR contains 125 handwritten pages with 1,221 marriage records covering family names, occupations, locations, and civil states.The records also include family names and marriage fees at the sides of paragraphs.Source: §4.1 · p. 5
4.3. Synthetic GMB
- The synthetic GMB pages depict a standard named-entity task with sparse entities relative to other words.The example page is from the SynthGMB dataset.Source: Figure 5 · p. 5
4.2. War Refugees
- WR forms show a strong but non-fixed relation between text location and tag, alongside 82% out-of-vocabulary words and mixed printed and handwritten text.A key challenge is distinguishing relevant from non-relevant text, which can differ mainly by location or nearby printed keys.Source: §4.2 · p. 5, Figure 4 · p. 5
5. Experiments
The experiments compare joint and sequential variants across detection, transcription, and entity recognition, using separate metrics for each. Localization is generally high and relatively insensitive to setup, while entity recognition varies with dataset context.
5. Experiments
- Training uses stochastic gradient descent on three losses, with a batch size of one page because the unified model is memory-expensive.The reported settings also include a learning rate of 10^-4, non-maximum suppression threshold of 0.2, box sampling score threshold of 0.5, and patience of 100 epochs.Source: §5 · p. 6
5.1. Setup
- The five setups compare all-task objectness classification, sequential triple-task labeling, detection with entity recognition, detection with transcription, and a word-only CNN.The CNN uses no shared recognition or localization features and receives no context information.Source: §5.1 · p. 6, Figure 6 · p. 6
5.2. Metrics
- Localization uses average precision, entity recognition uses F1, and transcription uses character error rate (CER).Entity recognition is scored by word location and tag, independently of transcription; CER counts insertions, deletions, and substitutions relative to string length.Source: §5.2 · pp. 6–7
5.3. Results
- Localization performance is generally high and the least challenging task, with no significant differences across setups or datasets.Table 5 presents performance for the method variations on each dataset.Source: §5.3 · p. 7, Table 5 · p. 7
- Entity-recognition outcomes vary by dataset: context helps over the word-only CNN, with layout especially useful for WR and sequential context for sGMB.The sequential triple-task model also performs well on WR; IEHHR shows no significant differences among end-to-end approaches.Source: §5.3 · p. 7
6. Discussion and Conclusion
Joint models suit cases with strong task interdependence, but may underperform when one independent task dominates. Localization can interact beneficially with transcription and named entity recognition when spatial or neighborhood context matters.
6. Discussion and Conclusion
- Localization can interact beneficially with transcription and named entity recognition when spatial information or neighboring text influences the entity value.Source: §6 · p. 7
- A unified model can be limited when one task is much harder and unrelated to the others; page-image input also makes semantically meaningful word embeddings difficult for named entity recognition.Source: §6 · p. 7
- Joint models are suitable when tasks strongly depend on one another, but not when an unstructured document makes one independent task the main difficulty.Source: §6 · p. 7