Source-linked AI summary
WordSup: Exploiting Word Annotations for Character based Text Detection
Han Hu, Chengquan Zhang, Yuxuan Luo, Yuzhuo Wang, Junyu Han, Errui Ding
TL;DR
Character detectors offer a common basis for text detection across languages and scenarios, but character-level annotations are costly and real datasets are mostly labeled at the word level. WordSup addresses this gap with weak supervision that iteratively learns character models from word annotations, enabling strong scene-text benchmark performance and flexible downstream text-structure analysis.
Problem
Character-level location labels are expensive to obtain, while most large-scale real text datasets provide word-level annotations instead.
Method
WordSup uses word annotations to iteratively refine character center masks and train a character detector, which feeds a text structure analysis pipeline.
Results
The character-based pipeline achieves state-of-the-art performance on several challenging scene-text benchmarks and applies to deformed text detection and structural math expression recognition.
Takeaways & Limitations
Word-level annotations from large real datasets can strengthen character models and support a flexible text detection pipeline across varied scenarios.
Takeaways & Limitations
The approach assumes that at least a bounding box for each word is available for supervision.
Abstract
from arXiv · showhide
Imagery texts are usually organized as a hierarchy of several visual elements, i.e. characters, words, text lines and text blocks. Among these elements, character is the most basic one for various languages such as Western, Chinese, Japanese, mathematical expression and etc. It is natural and convenient to construct a common text detection engine based on character detectors. However, training character detectors requires a vast of location annotated characters, which are expensive to obtain. Actually, the existing real text datasets are mostly annotated in word or line level. To remedy this dilemma, we propose a weakly supervised framework that can utilize word annotations, either in tight quadrangles or the more loose bounding boxes, for character detector training. When applied in scene text detection, we are thus able to train a robust character detector by exploiting word annotations in the rich large-scale real scene text datasets, e.g. ICDAR15 and COCO-text. The character detector acts as a key role in the pipeline of our text detection engine. It achieves the state-of-the-art performance on several challenging scene text detection benchmarks. We also demonstrate the flexibility of our pipeline by various scenarios, including deformed text detection and math expression recognition.
1. Introduction
WordSup builds a common character-based text detection engine by learning character detectors from weak word-level supervision, addressing the scarcity and cost of character annotations. Its character-centered pipeline supports diverse languages, text structures, and scenarios, achieving strong benchmark performance and extending to deformed text and mathematical expressions.
- 1. Introduction: Character is a shared basic element across languages and visual text scenarios, motivating a common detection framework based on character detection.The paper contrasts differences in language organization and visual distortion with the common character-level foundation.
- 1. Introduction: Character-location annotation is expensive and inconvenient, while most large-scale real text datasets provide only word-level labels.Characters are small, prone to gluing and blur, whereas datasets such as ICDAR15 and COCO-Text are coarsely annotated at word level.
- 1. Introduction: WordSup iteratively refines character center masks and the character model, using word annotations as weak supervision.The framework alternates between generating character masks from the current model and updating the character network from those masks.
- 1. Introduction: The character model achieves state-of-the-art performance on ICDAR13, ICDAR15, and COCO-Text, while the pipeline also applies to deformed text lines and structural math expression recognition.The paper presents the resulting pipeline as effective across challenging scene-text benchmarks and multiple scenarios.
- 1.1. Related Works: The approach exploits word annotations from large-scale real datasets, strengthening character models beyond reliance on synthetic character data.This addresses the limited coverage of challenging real scenes in synthetic datasets and also accommodates loose word annotations.
- 1.1. Related Works: The pipeline detects characters first, then performs application-dependent text structure analysis for lines, words, text blocks, and mathematical expressions.Its structure analysis handles horizontal, oriented, and deformed lines, while math expressions use non-sequential character or symbol connections.
2. Our Approach
The approach builds a shared character-based text detection pipeline and trains its character detector from word annotations through alternating mask generation and network updating. It combines high-resolution detection with word-supervised refinement, then analyzes character structure for lines and other text scenarios.
- Pipeline: The pipeline first detects characters with a shared module, then applies application-dependent text structure analysis for sequential lines and mathematical expressions.Detected characters support horizontal, oriented, and deformed text-line extraction, as well as recognition and structural recovery for math expressions.
- Basic Character Detector: The basic character detector uses 1/4-resolution feature maps and merges deeper, coarser cues to preserve small-character detail while retaining representation power.The network uses an FPN-style element-sum fusion, while the figure identifies VGG16 as its inherited network model.
- Basic Character Detector: Two-level hard-negative mining addresses varied backgrounds by selecting high-scoring negatives and repeatedly sampling patches containing current false positives.Online mining limits the negative-to-positive ratio to 3:1; hard patch mining begins after 10k iterations.
- Learning From Word Annotations: The method supports word annotations as bounding boxes and can use tighter quadrangles or word categories to strengthen supervision.Most cited real-text datasets provide word-level annotations, including quadrangles in ICDAR15 and bounding boxes in COCO-Text.
- Learning From Word Annotations: Word-level supervision alternates between generating character masks from word annotations and updating the character network with those masks.The procedure is used during training only; inference remains unchanged.
- Learning From Word Annotations: Candidate characters are selected by maximizing a score that balances word-box coverage with alignment of character centers along a straight line.The coverage term uses the selected-character and annotation bounding boxes, while the alignment term uses covariance eigenvalues; the default weight is 0.5.
- Learning From Word Annotations: More reliable generated masks receive greater influence through a loss combining confidence and localization terms, and both masks and the network improve during training.The mask reliability score is computed from the mask-generation objective.
- Text Structure Analysis: For each character group, text-line models increase in complexity from horizontal or vertical lines to arbitrary orientations and piecewise linear polygons.The three models are 0-order, 1-order, and piecewise linear.
3. Experiments
Experiments evaluate the detector on synthetic data and scene-text benchmarks, showing that word supervision can train effective character models and support challenging text scenarios.
- Datasets and evaluation: The experiments use VGG SynthText-part for ablations and ICDAR13, ICDAR15, and COCO-Text for benchmark evaluation.VGG SynthText-part provides synthetic character annotations, while ICDAR15 uses word-level quadrangles and COCO-Text is a large-scale real-image dataset.
- Synthetic-data ablations: The basic network preserves finer feature-map resolution than faster RCNN and SSD, and two-level hard negative mining provides a moderate gain.Its character-response maps use 1/4 resolution rather than 1/16 while merging cues from deeper stages.
- Synthetic-data ablations: Word supervision performs better than a 2k-character-trained model, with insignificant degradation relative to the full 50k-character-trained model.This demonstrates that weak word annotations can effectively train the character model.
- Scene-text benchmarks: On COCO-Text, the best model achieves 30.9% recall, 45.2% precision, and 36.8% F-measure, exceeding Yao’s method by 3.5% in F-measure.Adding real text images also improves the VGG-synth-coco character model over the synthetic-only model.
- Applications: The pipeline successfully handles blurry, perspective-distorted, handwritten or artistic-font, and extremely illuminated text, as well as deformed lines and structural math expressions.Detected characters support math-symbol recognition even when expressions are non-sequential.
4. Conclusion
The conclusion presents weakly supervised word annotation learning as a way to strengthen character-based text detection while preserving flexibility across languages and scenarios.
- 4. Conclusion: Character-based text detection methods are flexible across various scenarios.
- 4. Conclusion: The weakly supervised framework enables real word-level annotated images to train stronger character models.
- 4. Conclusion: Experiments demonstrate the effectiveness of the weakly supervised approach and the flexibility of the character-induced text detection pipeline.
5. Appendix
The appendix details character grouping, text-line modeling, rectification, and optional word partitioning for the text detection pipeline. It adapts graph-based grouping to multi-oriented text and converts grouped lines into rectified strips for recognition.
- Character Grouping: Character pairs replace individual characters as graph nodes, allowing pairwise costs to encode angle cues for multi-oriented text lines.Spatially close, high-scoring character pairs are used so higher-order relations among characters can be represented within a pairwise graph.
- Character Grouping: Unary costs combine average text/non-text scores with the Euclidean distance between the two characters in a pair.The pairwise graph cost uses the two characters’ scores and their center distance, while pairwise edge costs encode angle distance.
- Text Line Models: The best text-line model minimizes a height-based fitting term plus a complexity penalty, with penalties of 1.0, 1.2, and 1.4 for 0-order, 1-order, and piecewise linear models.Smaller estimated height indicates better model fitting; the model represents lines using center lines and a height value.
- Text Line Rectification: A closed polygon built from estimated center lines and heights is mapped to boundary control points on a 32 × W strip and rectified using thin plate splines.The polygon contains 2n vertices or control points, and the rectified strip has fixed height H = 32.
- Word Partition: An optional CNN-RNN word partitioning module separates words after line rectification for recognition and benchmark formats that require word-level evaluation.Detected separation positions are mapped back to produce word polygons for benchmarks including ICDAR13, ICDAR15, and COCO-Text.