Source-linked AI summary
A Two-Stage Method for Text Line Detection in Historical Documents
Tobias Grüning, Gundram Leifert, Tobias Strauß, Johannes Michael, Roger Labahn
TL;DR
Historical-document text line detection must handle complex, curved, and arbitrarily oriented lines while limiting annotation demands. The paper combines ARU-Net pixel labeling with bottom-up superpixel clustering, achieving strong results with fewer than 50 annotated images and outperforming state-of-the-art methods.
Problem
Historical-document text line extraction remains difficult because degradations, complex layouts, arbitrary orientations, and curved lines challenge existing methods.
Method
The method uses ARU-Net to label baseline, separator, and other pixels, then applies bottom-up superpixel clustering to build baselines.
Results
The method outperforms state-of-the-art approaches across three datasets, including an F-value increase from 0.859 to 0.922 on the cBAD complex track.
Takeaways & Limitations
The workflow handles complex layouts and straight, curved, and oriented text lines, while its open-source framework supports practical adaptation.
Takeaways & Limitations
Reported training-sample numbers are not directly transferable to collections containing entirely different scenarios, which may require more data.
Abstract
from arXiv · showhide
This work presents a two-stage text line detection method for historical documents. Each detected text line is represented by its baseline. In a first stage, a deep neural network called ARU-Net labels pixels to belong to one of the three classes: baseline, separator or other. The separator class marks beginning and end of each text line. The ARU-Net is trainable from scratch with manageably few manually annotated example images (less than 50). This is achieved by utilizing data augmentation strategies. The network predictions are used as input for the second stage which performs a bottom-up clustering to build baselines. The developed method is capable of handling complex layouts as well as curved and arbitrarily oriented text lines. It substantially outperforms current state-of-the-art approaches. For example, for the complex track of the cBAD: ICDAR2017 Competition on Baseline Detection the F-value is increased from 0.859 to 0.922. The framework to train and run the ARU-Net is open source.
1 Introduction
Historical-document text line extraction remains difficult because documents contain degradations, complex layouts, arbitrary orientations, and curved lines. The paper proposes a two-stage baseline-detection workflow using ARU-Net pixel labeling and image-processing clustering, with limited annotation and broad applicability.
- Text line extraction is a mandatory first step before transforming document pixels into text for ATR and KWS.
- Historical documents challenge text line detection through degradations, capture conditions, complex layouts, arbitrary orientations, and curved lines.
- The method represents each text line by its baseline because text line representations otherwise vary across downstream systems.
- ARU-Net labels pixels as baseline, separator, or other, while a second stage clusters image-processing elements to construct baselines.
- Less than 50 full-page training samples support high-quality results through data augmentation, while curved and arbitrarily oriented lines approach straight-line performance.
- The workflow combines deep learning with image processing, introduces learned separators for complex layouts, and is freely usable through open-source components and Transkribus.
2 Related Work
Prior text line extraction methods include dynamic programming, interest-point or superpixel clustering, state-based clustering, and neural pixel-labeling or regression approaches. The paper positions its method as combining deep learning with image processing.
- Dynamic-programming approaches calculate cost-optimal paths to separate text lines, differing mainly in preprocessing and cost-function design.
- Interest-point clustering forms word clusters, separates them into sub-word segments, and groups those segments into text lines.
- State-based clustering encodes local text orientation and interline distances, then minimizes operation costs to obtain final segmentation.
- The proposed work combines deep learning strategies with state-of-the-art image-processing techniques in a two-stage method.
- Existing neural approaches include recurrent regression for line boxes and starts, alongside fully convolutional pixel-labeling methods.
3 Methodology
The proposed baseline detector uses two stages: ARU-Net performs task-specific pixel labeling, and superpixels from its output are clustered to form baselines.
- Stage I applies ARU-Net to perform task-specific pixel labeling rather than detecting every foreground element.
- Stage II extracts superpixels from the Stage I output and clusters them to build baselines.
3.1 Problem Statement
The paper formulates baseline detection as learning a detector from annotated training images that generalizes to unseen test images. Baselines are represented as polygonal chains and evaluated through similarity and F-value measures.
- A baseline is the virtual typographical line where most characters rest and descenders extend below.
- Each baseline is represented by a polygonal chain, and a baseline detector maps an image to a set of such chains.
- Baseline ground truth consists of polygonal chains representing an image’s baselines, potentially annotated by a human operator.
- Ground-truth variety means no single unique annotation exists, so creator bias must be considered in evaluation and detector design.
- The detector is learned from Ttrain and must generalize to unknown Ttest using only the training set during design.
- The framework can adapt to differently biased annotations by retraining the first stage without expert fine-tuning.
3.2 Stage I: ARU-Net
Stage I uses ARU-Net as a supervised neural pixel labeler for baseline detection, extending residual U-Net processing with multi-scale spatial attention. It classifies pixels into baseline, separator, and other classes, producing outputs for the subsequent baseline-construction stage.
- Pixel labeling: The network is trained as a supervised neural pixel labeler using cross-entropy loss, backpropagation, and gradient-descent parameter updates.The loss compares predicted and ground-truth pixel distributions, and training repeatedly updates the model parameters using the loss gradient.
- Architecture: ARU-Net extends an RU-Net, which replaces U-Net convolution blocks with residual blocks that support identity and error propagation.Residual blocks provide shortcut connections, and RU-Net is defined as a U-Net with residual blocks.
- Architecture: A multi-scale attention mechanism processes downscaled inputs with shared RU-Net and A-Net weights, then upsamples and combines their feature maps.Pixel-wise softmax-normalized attention maps weight RU-Net feature maps at different scales before summation and classification.
- Architecture: The attention mechanism allows ARU-Net to focus on different image positions at different scales, supporting pages containing varied font sizes.The authors describe the RU-Net as specializing in a font size while the A-Net distinguishes areas with different font sizes.
- Pixel labeling: ARU-Net assigns each image pixel a confidence distribution over the classes baseline, separator, and other.Pixel ground truth uses one-hot class assignments, while the neural pixel labeler predicts class confidences for every pixel.
- Baseline detection: The network output is designed for baseline detection, where explicit separator predictions help distinguish closely spaced text lines such as those in adjacent columns.The separator class marks the beginning and end of each text line and supplements information implicitly encoded by baseline pixels.
3.3 Stage II: Baseline Estimation
Stage II estimates baselines from ARU-Net outputs by extracting superpixels, assigning each a local text orientation and interline distance, and clustering them into baselines. It combines local periodicity and orientation cues with neighborhood-based smoothing and layout constraints.
- Pipeline: The second stage estimates baselines through superpixel calculation, state estimation, and superpixel clustering.It operates on the output of the ARU-Net.
- Superpixel calculation: Superpixels are selected from skeletonized, thresholded baseline-confidence maps, so missing superpixels can cause a baseline to be missed.Pixels are ordered by baseline confidence before the superpixel set is constructed.
- State estimation: Each superpixel state is the pair of its local text orientation and interline distance.Interline distance is measured orthogonally to the local text direction.
- State estimation: Local text orientation is estimated from neighboring superpixels with the strongest baseline connectivity, while interline distance is inferred from regional text-line periodicity.The orientation uses the slope through selected neighbors; periodicity is analyzed with projection profiles and a DFT.
- State estimation: Interline-distance labeling combines data costs from local periodicity with smoothing costs that discourage large differences between adjacent superpixels.The smoothing cost becomes large when adjacent superpixels receive interline distances with different sizes.
- Superpixel clustering: Available polygonal text-region information is incorporated by removing edges connecting superpixels that do not belong to the same region.This separates superpixels from different non-overlapping regions during clustering.
4 Experiments
The experiments assess training-data requirements, augmentation, geometric robustness, architecture choices, and comparisons with state-of-the-art systems. Results show strong performance with limited data, curved and oriented text, and challenging benchmark datasets.
- Training sample number and augmentation: All three augmentation strategies significantly improve performance over the base strategy, with the largest differences occurring for small training sets.The best mean F-value is 0.975 with all 350 training samples and S+A+E augmentation.
- Training sample number and augmentation: The reported training-data results are not directly transferable to collections combining substantially different page scenarios.The paper gives mixed census tables and postal cards as examples and expects that more than 30 samples may be needed in such settings.
- Curved and oriented text lines: Curved text lines achieve results as good as the base scenario, while arbitrarily oriented lines perform slightly worse but remain excellent.The authors report applicability without remarkable workflow adaptation; additional elastic transformations benefit the oriented scenario.
- Architecture and workflow comparison: The ARU-Net performs significantly better than the compared architectures with less computational effort, and the second stage improves overall performance.The LARU-Net does not prove superiority and is dismissed in this comparison.
- Comparison with the state of the art: The framework is compared against methods from three recent historical-document text-line competitions, including ICDAR 2015, ICDAR2017 layout analysis, and cBAD baseline detection.The cBAD comparison uses the competition datasets and metrics described in the respective competition papers.
- Comparison with the state of the art: For ANDAR-TL, the system outputs the leftmost point of each detected baseline as the competition’s origin-point representation.The ARU-Net was trained on cBAD data, so its results are difficult to compare directly with systems trained on the original ANDAR-TL training data.
- Comparison with the state of the art: The ARU-Net performs considerably better than all submissions in both cBAD tracks, with the largest increase in the complex track.The comparison is made against the average ARU-Net result and results reported in the cBAD competition paper.
5 Conclusion
The method detects historical-document text lines through baseline representations and supports complex layouts with limited training data. It was reported to outperform prior state-of-the-art methods, while still producing some errors.
- Text lines are represented by their baselines, and ARU-Net predicts baseline positions plus line beginnings and endings.These predictions support documents with tables, marginalia, and multi-column layouts.
- The system can be trained from scratch with manageably few training samples for a complex but homogeneous collection.
- The proposed method substantially outperforms previous state-of-the-art approaches, although missed baselines and segmentation errors remain.The cited evaluation table presents P-, R-, and F-values for the cBAD simple and complex tracks, but no values are included here.
- Ground-truth production is described as inexpensive and feasible without expert knowledge.The authors report that a page with annotated baselines can be prepared in a few minutes.
Algorithm S.1: Pixel Ground Truth Generation
The appendix describes constructing pixel ground truth from baseline annotations by drawing baseline and endpoint-related polygonal chains, then outlines clustering operations used to form optimized partitions. Supplementary figures show results across training-sample counts, augmentation strategies, and degraded or layout-free test images.
- Pixel Ground Truth Generation: Pixel ground truth generation takes an image and corresponding baseline ground truth as input and produces pixel ground truth.
- Pixel Ground Truth Generation: Polygonal chains centered at the first and last baseline points are oriented θ + 90° and drawn into S, while P is drawn in B.
- SP Clustering: SP Clustering processes a set of SPs and sorted edges to produce an optimized partition P∗ through edge addition, new-cluster creation, cluster extension, and cluster merging.
- SP Clustering: Clustering decisions use curvature and distance conditions involving γ, δ, and cluster scale, including tests for new clusters, extensions, and merges.
- Supplementary Results: Supplementary results vary training samples from 5 to 350 and augmentation strategies, and also show arbitrarily curved or rotated images and cBAD images without layout information.