Source-linked AI summary
Towards End-to-End Unified Scene Text Detection and Layout Analysis
Shangbang Long, Siyang Qin, Dmitry Panteleev, Alessandro Bissacco, Yasuhisa Fujii, Michalis Raptis
TL;DR
Text detection and geometric layout analysis are typically studied separately, although text understanding also requires layout comprehension. This paper introduces HierText and an end-to-end model that detects text and clusters it, outperforming multi-stage baselines and achieving state-of-the-art results on multiple scene-text benchmarks without complex post-processing.
Problem
Text detection and geometric layout analysis are usually treated as separate tasks, despite visual text understanding requiring semantic and geometric layout comprehension.
Source: §1 · p. 1Method
The paper introduces HierText with hierarchical annotations and Unified Detector, which detects text entities and groups them into clusters end to end.
Source: §1 · p. 2, §3.3 · p. 4Results
Unified Detector outperforms competitive multi-stage baselines on unified text detection and layout analysis and achieves state-of-the-art results on multiple public scene-text benchmarks without complex post-processing.
Source: §1 · p. 2Takeaways & Limitations
The unified task, hierarchical dataset, and model are presented as supporting text extraction and understanding in images and downstream tasks.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Scene text detection and document layout analysis have long been treated as two separate tasks in different image domains. In this paper, we bring them together and introduce the task of unified scene text detection and layout analysis. The first hierarchical scene text dataset is introduced to enable this novel research task. We also propose a novel method that is able to simultaneously detect scene text and form text clusters in a unified way. Comprehensive experiments show that our unified model achieves better performance than multiple well-designed baseline methods. Additionally, this model achieves state-of-the-art results on multiple scene text detection datasets without the need of complex post-processing. Dataset and code: https://github.com/google-research-datasets/hiertext and https://github.com/tensorflow/models/tree/master/official/projects/unified_detector.
1. Introduction
The paper argues for treating text detection and geometric layout analysis jointly, because text understanding requires both entity localization and layout comprehension. It introduces HierText and Unified Detector, which jointly detect text and cluster text entities, outperforming competitive baselines and reaching state-of-the-art results on multiple scene-text benchmarks.
1. Introduction
- Most prior work treats text-entity detection and layout analysis separately, although text understanding requires semantic and geometric layout comprehension.The paper motivates joint treatment as a route to simpler, more efficient, and potentially more accurate models.Source: §1 · p. 1
- HierText is introduced as the first dataset with hierarchical text annotations in natural scenes and documents, covering word, line, and paragraph levels.Images average more than 100 words, twice the density of the then-highest-density scene-text dataset.Source: §1 · p. 2
- Unified Detector combines instance segmentation for arbitrarily shaped text with multi-head self-attention to cluster text entities in an end-to-end pipeline.It avoids complex label generation during training and sophisticated post-processing during inference.Source: §1 · p. 2
- The model achieves state-of-the-art scene-text detection on ICDAR 2017 MLT, Total-Text, CTW1500, and MSRA-TD500 without fine-tuning on each target dataset.It is directly trained using a combination of datasets and simultaneously recovers text-layout information.Source: §1 · p. 2
- Unified Detector outperforms competitive multi-stage baselines on unified text detection and layout analysis.The model also achieves state-of-the-art results on multiple challenging public text-detection benchmarks without complex post-processing.Source: §1 · p. 2
2. Related Works
Prior scene-text datasets generally annotate individual words or lines, while document datasets often omit atomic text entities or focus on specific document domains. Earlier scene-text detectors and layout-analysis methods also leave the two tasks separate; this work brings layout analysis into the scene-text domain with an end-to-end unified model.
2.1. Scene text and documents datasets
- Scene-text datasets focus on individual words or lines, while document datasets often omit word labels or cover specific domains; HierText supports joint detection and layout analysis in general natural images.Prior text-reading-order data also focuses on images with well-defined reading order.Source: §2.1 · p. 2
2.2. Scene text detection
- Scene-text detection methods address irregular text representations and contour-recovery post-processing, while prior detectors do not also perform layout analysis.The paper positions its end-to-end unified network as simplifying that pipeline.Source: §2.2 · p. 2
2.3. Layout analysis
- Some document layout methods detect text blocks but omit word- or line-level detections, while graph-based methods analyze word-level layout; both mainly target document datasets.The paper introduces layout analysis into the scene-text domain through an end-to-end unified model.Source: §2.3 · pp. 2–3
3. Hierarchical Text Dataset (HierText)
HierText provides hierarchical word, line, and paragraph annotations in a dataset with high word density and broad spatial text distribution. The section defines unified detection and layout analysis tasks and their evaluation using Panoptic Quality.
3.1. Data collection
- HierText images come from Open Images v6, filtered for word count, recognition confidence, and dominant language, yielding 11,639 images.The long side is constrained to 1600 pixels; word polygons are annotated first, then grouped into lines and paragraphs to form a tree-structured hierarchy.Source: §3.1 · p. 3
3.2. Dataset characteristics
- 103.8 words per image on average makes HierText approximately three times denser than the second-densest dataset, TextOCR.HierText includes 5.3K images with more than 100 words, compared with 3.4K for Intel OCR, and is the only dataset listed with hierarchical annotations.Source: §3.2 · p. 3
- HierText captures an image domain not covered by existing datasets and enables research on unified text detection and layout analysis.Its images have more uniformly distributed text and a larger share of high-density examples than other datasets described.Source: §3.2 · p. 3
3.3. Task and evaluation protocol
- Text detection is evaluated at word or line level, while layout analysis treats each paragraph as an instance formed from underlying word masks.Candidate methods should detect words or lines and group those entities into paragraphs.Source: §3.3 · p. 4
- Panoptic Quality is the common evaluation metric because it combines an F1 score with average IoU and applies uniformly to word, line, and paragraph segmentation.The passage notes that text entities are sensitive to missing or superfluous pixels, while cited text-detection scoring methods do not generalize to complex line and paragraph geometries.Source: §3.3 · p. 4
4. Methodology
Unified Detector jointly predicts text masks and pairwise affinities for grouping text instances. Its architecture uses object queries and separate detection, layout, and textness branches, trained with matched targets and joint losses.
4.1. Unified detector
- Unified Detector produces a fixed set of softly exclusive text masks and an affinity matrix for clustering detections, with thresholds and union-find used at inference.The affinity between two masks represents the probability that their text belongs to the same semantic or paragraph group.Source: §4.1 · p. 4
4.2. Model architecture
- The MaX-DeepLab backbone uses bidirectional communication between pixels and learned object queries, which feed separate mask, layout-affinity, and textness branches.The backbone combines hourglass-style CNNs with a dual-path transformer; the mask branch predicts masks from mask queries and normalized pixel features.Source: §4.2 · pp. 4–5
4.3. Training targets
- Training matches unordered predictions to ground truth by maximizing total similarity, using the Hungarian algorithm during training.The similarity combines classification agreement with Dice mask similarity.Source: §4.3 · p. 5
- The layout loss supervises predicted affinities using ground-truth text-cluster IDs, and the model jointly optimizes detection, layout, segmentation, and instance-discrimination losses.The layout loss balances positive and negative pairs; the full loss is a weighted sum of the four task losses.Source: §4.3 · pp. 5–6
5. Experiments
Experiments compare the unified detector with separate detection and layout-analysis baselines, test its design choices, and evaluate scene-text detection on public benchmarks. The unified approach improves HierText layout analysis and achieves strong detection results across multiple datasets.
5.1. Baselines
- Baselines include GCP API, GCN post-processing, and two-stage Mask R-CNN or MaX-DeepLab cluster models paired with the detector’s text branch.The object-detection baselines assign detected text entities to the cluster mask with the largest intersection.Source: §5.1 · p. 6
5.2. Experimental settings
- Main experiments train on HierText with a MaX-DeepLab-S backbone, 384 object queries, and line-level text detection.The setup uses 1024 × 1024 inputs and filters text masks and clustering affinities at inference.Source: §5.2 · p. 7
5.3. Main Results
- The unified approach improves layout-analysis PQ by 1.08% over standalone clustering methods and scores 62.23 versus 56.17 for the GCP API on text detection.The standalone methods use the unified detector’s text detections and require almost double the computational resources.Source: Table 2 · p. 6, §5.3 · p. 7
- Across varied image domains, the detector handles curved text and non-uniform fonts and colors while producing text detections and layout-analysis outputs.Figure 4 pairs detection results on the top or left with layout-analysis results on the bottom or right.Source: Figure 4 · p. 6, §5.3 · p. 7
5.4. Ablation studies
- Line-based models consistently outperform word-based models across object-query counts, while α-balancing improves both text detection and layout analysis.The authors suggest fewer line-level objects may make clustering-head optimization easier.Source: §5.4 · p. 7, Table 3 · p. 7, Table 4 · p. 7
- Additional transformer layers improve performance; separating features is necessary, and a multi-head self-attention head outperforms a fully connected layer.The comparisons also include a detector without a layout-analysis branch.Source: §5.4 · p. 8, Table 6 · p. 8
5.5. Scene text detection on public datasets
- Word detection achieves 77.24 on MLT 17 and 87.94 and 87.90 on Total-Text with and without HierText, respectively.The reported results are state of the art; the table caption says models are not fine-tuned on target datasets.Source: §5.5 · p. 8, Table 5 · p. 8
- Adding HierText improves line-detection results from 84.88 to 85.97 on CTW1500 and from 86.69 to 87.70 on MSRA-TD500.Across public benchmarks, the detectors show higher recall and lower precision than state-of-the-art methods.Source: §5.5 · p. 8
6. Conclusion
The paper concludes that its hierarchical dataset and end-to-end unified model advance scene text detection and layout analysis while simplifying the pipeline and supporting downstream tasks.
6. Conclusion
- The end-to-end unified model outperforms previous methods while greatly simplifying the detection and layout-analysis pipeline.Source: §6 · p. 8
- The paper collects a dataset with hierarchical text annotations to facilitate research on unified detection and layout analysis.Source: §6 · p. 8
- The authors state that the task, dataset, and model enable better support for downstream tasks.Source: §6 · p. 8