Source-linked AI summary
Convolutional Character Networks
Linjie Xing, Zhi Tian, Weilin Huang, Matthew R. Scott
TL;DR
Existing joint text detection and recognition systems commonly rely on two-stage, RoI-based architectures and RNN recognition branches. CharNet uses a one-stage convolutional architecture with parallel character and text-detection branches, plus iterative character detection for training without extra real-world character boxes. It consistently outperforms recent approaches across benchmarks, including 65.33%→71.08% on ICDAR 2015 with a generic lexicon and 54.0%→69.23% on Total-Text.
Problem
Existing joint systems use two-stage RoI operations and RNN-based recognition, creating limitations for jointly optimizing text detection and recognition.
Method
CharNet is a one-stage convolutional model with parallel branches for direct character detection and recognition and text-instance detection, supported by iterative character detection.
Results
CharNet consistently outperforms recent two-stage approaches, including 65.33%→71.08% on ICDAR 2015 with a generic lexicon and 54.0%→69.23% on Total-Text.
Takeaways & Limitations
The character-based one-stage design yields a compact model that jointly handles text detection and recognition and works reliably on curved text.
Takeaways & Limitations
Training the character branch requires character-level bounding boxes and labels, which are more expensive to obtain than word-level annotations.
Abstract
from arXiv · showhide
Recent progress has been made on developing a unified framework for joint text detection and recognition in natural images, but existing joint models were mostly built on two-stage framework by involving ROI pooling, which can degrade the performance on recognition task. In this work, we propose convolutional character networks, referred as CharNet, which is an one-stage model that can process two tasks simultaneously in one pass. CharNet directly outputs bounding boxes of words and characters, with corresponding character labels. We utilize character as basic element, allowing us to overcome the main difficulty of existing approaches that attempted to optimize text detection jointly with a RNN-based recognition branch. In addition, we develop an iterative character detection approach able to transform the ability of character detection learned from synthetic data to real-world images. These technical improvements result in a simple, compact, yet powerful one-stage model that works reliably on multi-orientation and curved text. We evaluate CharNet on three standard benchmarks, where it consistently outperforms the state-of-the-art approaches [25, 24] by a large margin, e.g., with improvements of 65.33%->71.08% (with generic lexicon) on ICDAR 2015, and 54.0%->69.23% on Total-Text, on end-to-end text recognition. Code is available at: https://github.com/MalongTech/research-charnet.
1. Introduction
CharNet addresses limitations of sequential and two-stage joint text reading by using characters as the basic unit in a one-stage model. It directly detects words and characters, transfers character detection from synthetic to real images, and reports large benchmark gains.
- Sequential detection and recognition can cause sub-optimization, system complexity, and dependence of recognition performance on detection results.
- Existing joint models commonly use RNN-based recognition and RoI cropping or pooling, complicating joint optimization and potentially including background features.
- CharNet is a one-stage model that uses characters as the basic unit for direct character detection and recognition alongside text-instance detection.
- Iterative character detection transfers character-detection capability from synthetic data to real-world images without additional real-world character bounding boxes.
- 65.33%→71.08% on ICDAR 2015 with a generic lexicon and 54.0%→69.23% on Total-Text show large end-to-end recognition gains over recent two-stage approaches.
2. Related Work
Related work largely treats text detection and recognition as sequential or combines them with two-stage, RoI-based frameworks and recurrent recognition. CharNet instead provides a one-stage character-based approach that directly predicts character boxes and labels without requiring full real-world character annotations.
- Traditional text-reading systems process detection and recognition as separate sequential tasks, while recent work seeks unified end-to-end recognition.
- Text detection research commonly extends general object detectors with text-specific mechanisms such as sequential proposals or linked segments.
- Text recognition commonly uses CNN features followed by RNN-based sequence decoding, often with CTC or related sequence methods.
- Unified end-to-end methods often use RoI pooling or alignment and recurrent recognition branches, forming two-stage frameworks.
- CharNet is the first one-stage end-to-end recognition model described here to directly output character-level boxes and labels.
- Its iterative character detection enables training without additional real-world character-level bounding boxes, unlike approaches requiring full character annotations.
3. Convolutional Character Networks
CharNet is a one-stage convolutional architecture that jointly detects text instances and characters, directly producing bounding boxes and character labels in one pass. It uses iterative character detection to adapt synthetic-data character localization to real-world images without additional character-level annotations.
- Overview: CharNet runs character detection and recognition alongside text-instance detection in parallel branches, avoiding RoI pooling and RNN-based word recognition.The model directly outputs instance-level and character-level bounding boxes with corresponding labels during inference.
- Iterative Character Detection: Iterative character detection adapts character localization learned from synthetic data to real-world images using only text-instance-level annotations.This addresses the high cost and limited availability of real-world character-level bounding boxes.
- Character Branch: The character branch densely predicts text segmentation, character boxes, and character labels from shared backbone feature maps.Character boxes use five parameters, while recognition predicts among 68 character classes.
- Text Detection and Grouping: CharNet groups detected characters into text instances by assigning each character to an instance box when their boxes overlap.The final output contains both text-instance and character bounding boxes with character labels.
- Specialized Text Shapes: The architecture supports multi-orientation and curved text through oriented text detection and a direction field for separating adjacent curved instances.The direction field encodes directions away from text boundaries and is predicted in parallel with the other branches.
4. Experiments, Results and Comparisons
CharNet is evaluated on three benchmarks using iterative character detection to adapt synthetic-data character knowledge to real-world images. It improves text detection and end-to-end recognition, including curved and multi-oriented text.
- Evaluation Setup: CharNet is evaluated on ICDAR 2015, Total-Text, and ICDAR MLT 2017.The benchmarks include multi-oriented, small-scale, curved, and multilingual text.
- Iterative Character Detection: Training on real-world data with iterative character detection substantially improves performance over direct synthetic-data transfer.Direct transfer from synthetic data performs poorly because of the domain gap, while iterative training improves text detection and end-to-end recognition.
- Iterative Character Detection: 92.65% correct words are collected from Total-Text after four iterative character-detection steps.The identified-word ratio rises from 64.95% at step 0 to 88.94% at step 1 before reaching 92.65%.
- Text Detection: 4.13%: CharNet’s ICDAR 2015 detection F-measure increases from 85.57% to 89.70% with joint recognition optimization.The gain exceeds FOTS’s reported 2.68% improvement under the corresponding comparison.
- Text Detection: 90.97% F-measure is achieved on ICDAR 2015 text detection, compared with 87.99% for the recent strong baseline.Reported single-scale gains are 80.3%→85.6% on curved Total-Text text and 67.25%→75.77% on ICDAR 2017 MLT.
5. Conclusions
CharNet is a compact one-stage fully convolutional model for end-to-end text recognition that jointly detects and recognizes text. It improves collaborative training, transfers character detection from synthetic to real images, and works reliably on curved text.
- CharNet is the first one-stage fully convolutional model to jointly implement text detection and recognition.It introduces a direct character-recognition branch integrated into a text-detection framework.
- The joint one-stage design enables more effective collaborative training of text detection and recognition.
- Iterative character detection transfers character-detection capability learned from synthetic data to real-world images.
- CharNet is compact with fewer parameters and works reliably on curved text.
- CharNet consistently outperforms existing approaches by a large margin on ICDAR 2015, MTL 2017, and Total-Text.