Source-linked AI summary
TextScanner: Reading Characters in Order for Robust Scene Text Recognition
Zhaoyi Wan, Minghang He, Haoran Chen, Xiang Bai, Cong Yao
TL;DR
Scene text recognition methods can fail through attention drift or unreliable character separation after segmentation. TextScanner addresses both issues with separate class and geometry branches, order-aware character extraction, and mutual supervision, and reports strong benchmark performance, including gains on irregular and Chinese text.
Problem
Existing attention-based methods can suffer attention drift, while segmentation-based methods can mis-separate characters through improper binarization.
Method
TextScanner uses separate branches for character class and geometry, fuses their maps to form ordered characters, and adds mutual supervision for sequence-level training.
Results
TextScanner achieves higher or highly competitive benchmark accuracy, with reported gains of 3.3% on ICDAR 2015 and 4% on CUTE80, and demonstrates Chinese-recognition superiority.
Takeaways & Limitations
TextScanner provides stronger adaptability to irregular, longer, and more complex text, including Chinese scripts.
Abstract
from arXiv · showhide
Driven by deep learning and the large volume of data, scene text recognition has evolved rapidly in recent years. Formerly, RNN-attention based methods have dominated this field, but suffer from the problem of \textit{attention drift} in certain situations. Lately, semantic segmentation based algorithms have proven effective at recognizing text of different forms (horizontal, oriented and curved). However, these methods may produce spurious characters or miss genuine characters, as they rely heavily on a thresholding procedure operated on segmentation maps. To tackle these challenges, we propose in this paper an alternative approach, called TextScanner, for scene text recognition. TextScanner bears three characteristics: (1) Basically, it belongs to the semantic segmentation family, as it generates pixel-wise, multi-channel segmentation maps for character class, position and order; (2) Meanwhile, akin to RNN-attention based methods, it also adopts RNN for context modeling; (3) Moreover, it performs paralleled prediction for character position and class, and ensures that characters are transcripted in correct order. The experiments on standard benchmark datasets demonstrate that TextScanner outperforms the state-of-the-art methods. Moreover, TextScanner shows its superiority in recognizing more difficult text such Chinese transcripts and aligning with target characters.
1 Introduction
TextScanner addresses complementary weaknesses in attention-based and segmentation-based scene text recognition by separating character geometry from classification and explicitly encoding reading order. It also introduces mutual supervision so sequence-level annotations can support training with real data, while experiments report strong benchmark and difficult-text performance.
- Motivation: RNN-attention methods can suffer attention drift, while segmentation methods can mis-separate characters through improper thresholding.These failures can respectively corrupt character order and class, or create spurious and missing characters.
- Our Work: The geometry branch assigns characters to distinct channels, allowing contiguous or same-class characters to be extracted while preserving order.This design is intended to avoid attention drift and improve character separation.
- Our Work: TextScanner’s mutual-supervision mechanism enables fine-tuning with real or synthetic data that has only sequence-level annotations.The class and geometry branches supervise one another when character-level annotations are unavailable.
- Results: 3.3% on ICDAR 2015 and 4% on CUTE80: TextScanner’s reported recognition-accuracy gains over prior art.The paper also reports higher or highly competitive accuracy on regular datasets and significantly enhanced performance on irregular datasets.
- Our Work: TextScanner uses separate class and geometry branches to predict character class, position, and order.Its segmentation maps are fused through element-wise multiplication to form characters in sequence order.
- Results: TextScanner shows stronger adaptability to longer and more complex text, including Chinese scripts.The paper reports this capability as part of its benchmark and Chinese-recognition evaluation.
2 Related Work
Related work centers on attention-based sequence recognition and segmentation-based recognition of irregular text. TextScanner retains segmentation flexibility while separating localization and ordering into geometry processing, with mutual supervision enabling the branches to use sequence-level annotations.
- Attention-based methods: Attention-based methods encode image features and use attention to align and decode characters as a sequence.They have achieved state-of-the-art performance but remain vulnerable to attention drift.
- Segmentation-based methods: Segmentation-based methods use fully convolutional networks, connected components, and component-level classification to recognize characters.They are generally more flexible for oriented and curved text than attention decoders.
- Segmentation-based methods: Segmentation post-processing may fail on closely arranged characters because connected-component separation depends on the segmentation map.This can limit recognition accuracy for irregular text.
- TextScanner: The geometry branch combines pyramid feature processing with an RNN module for context modeling.This preserves contextual modeling while supporting explicit character geometry.
- TextScanner: TextScanner dispatches character localization to different channels so characters are naturally separated and ordered.Its mutual-supervision mechanism lets the two branches use sequence-level annotations to supervise and enhance one another.
Overview
TextScanner decodes text by combining character-class segmentation with geometry maps that encode character positions and order. Its differentiable decoding aggregates these maps into ordered character predictions and can use sequence- or character-level annotations.
- Architecture: TextScanner's decoder has separate class and geometry branches for character classification, localization, and order prediction.The class branch produces multi-class segmentation maps, while the geometry branch produces localization and order maps.
- Geometry branch: The geometry branch uses a localization map and RNN-encoded context to generate one order map for each sequence position.Order maps are formed by element-wise multiplication of the localization map Q and the corresponding order-segmentation channel S_k.
- Word Formation: Word formation multiplies class and order maps so each sequence position receives class scores from spatial locations associated with that character.The resulting scores are used to identify the class of the kth character.
- Decoding: Decoding stops when the maximum character probability falls below Tscore or the predefined maximum sequence length N is reached.The procedure is differentiable and therefore supports training with sequence-level and character-level annotations.
Pre-Training with Character-Level Annotations
Character-level pre-training constructs segmentation, order, and localization targets from annotated character geometry. The training objective combines losses for these tasks, with mutual supervision enabled after pre-training.
- Data: TextScanner can be pre-trained on synthetic data using character-level annotations.
- Character segmentation labels: Polygon character regions are shrunk to P′ to reduce overlap between adjacent characters, and pixels outside P′ are ignored for segmentation optimization.Polygons are used because rectangular boxes are inaccurate for curved or dense text.
- Order-map labels: Order targets are generated from Gaussian maps centered on character bounding-box centers, thresholded at ζorder = 0.5, and normalized to [0, 1].The resulting normalized maps provide the ground truth for each character's order map.
- Localization labels: A localization target Q is formed by combining the normalized Gaussian heatmaps for all characters.
- Loss function: The overall objective is a weighted sum of localization, order-segmentation, character-segmentation, and mutual-supervision losses.λl and λo are set to 10; λm is 0 during pre-training and 1 otherwise.
Mutual-Supervision Mechanism
TextScanner uses mutual supervision between class and geometry branches to reduce reliance on character-level annotations. Sequence labels guide cross-branch refinement, enabling fine-tuning with sequence-level annotations.
- Mutual-Supervision Mechanism: Mutual supervision lets the class and geometry branches supervise each other using sequence-level labels when character-level annotations are unavailable.The branches generate character segmentation and order maps, then exchange supervision during training.
- Mutual-Supervision Mechanism: The process scans the sequence from the first character to the last, selecting pixels matching each character’s class and order.For character k, its order is k and its class is T(k).
- Mutual-Supervision Mechanism: Selected regions are refined with Q and mapped into the opposite prediction map to provide cross-branch supervision.The mechanism applies this refinement to regions in both ˆG and ˆH.
- Mutual-Supervision Mechanism: Repeated characters can select multiple regions, so those samples are removed when they cannot provide unambiguous supervision.The example selects two regions for ‘N’ in “LONDON”.
- Mutual-Supervision Mechanism: The loss weights branch confidences so inaccurate predictions have reduced influence during mutual supervision.The implementation sets λ to 0.2 and γ to 2.
Other Details
The implementation uses a ResNet-50 backbone within a CA-FCN-derived model and fixes score and size thresholds empirically.
- Other Details: TextScanner removes character attentions from CA-FCN, replaces its VGG blocks with ResNet-50, and uses ζscore = 0.3 and N = 32.These settings are used in the reported implementation.
4 Experiments
Experiments evaluate TextScanner on standard English benchmarks, irregular text, localization, and Chinese recognition. Results show consistent gains over comparable methods, especially for curved, oriented, and Chinese text.
- Datasets: The evaluation covers regular datasets IIIT, SVT, and IC13, plus irregular datasets IC15, SVTP, and CT.The experiments compare recognition accuracies across these standard benchmarks.
- Recognition Performance Evaluation: TextScanner variants consistently outperform previous methods under matched training data, with particularly strong gains on curved text.TextScanner+90k improves 3.3% on IC15, 4.1% on SVTP, and 4.0% on CT.
- Recognition Performance Evaluation: The geometry branch separates adjacent characters even when their segmentation masks connect, while tracking character arrangement in long or oriented text.Intermediate results highlight separation of adjacent ‘O’ characters.
- Recognition Performance Evaluation: TextScanner+real gains substantially from real training data, supporting the use of mutual supervision with sequence-level annotations.Fine-tuning with synthetic SynthText and 90k data also consistently improves performance.
- Recognition Performance Evaluation: TextScanner has a higher probability of accurate localization than the attention decoder when normalized center distance is D < 0.1.The comparison uses character-center distance on IC13.
5 Ablation Study
The ablation study isolates the geometry branch and order-map word formation. Both improve recognition, with order-aware decoding providing especially large gains on irregular text.
- 5 Ablation Study: Order maps improve recognition by ensuring characters are scanned in the correct order, yielding gains of 7.4% on IC15 and 10.2% on SVTP.The ablation compares geometry and order components separately.
- 5 Ablation Study: The geometry branch improves performance even with regular post-processing, indicating benefits for optimizing the class branch.The reported ablation attributes additional gains to geometry-based decoding.
6 Conclusion
TextScanner is presented as a segmentation-based dual-branch framework for scene text recognition that addresses challenging scenarios and incorporates mutual supervision for real and synthetic data.
- TextScanner is an effective segmentation-based dual-branch framework for scene text recognition.
- The framework is designed to overcome problems in previous methods and perform well under various challenging scenarios.
- Its mutual-supervision mechanism enables the use of both real and synthetic data.