Source-linked AI summary
Reading Scene Text in Deep Convolutional Sequences
Pan He, Weilin Huang, Yu Qiao, Chen Change Loy, Xiaoou Tang
TL;DR
Scene-text recognition must recover word strings despite difficult character segmentation and ambiguous images, while many existing approaches remain tied to character-level classification. DTRN generates an explicitly ordered deep CNN sequence from a word image and decodes it with an LSTM-based recurrent model. The paper reports state-of-the-art performance, with improvements over several comparison systems and support for unknown words and arbitrary strings.
Problem
Existing scene-text methods rely on character-level classification and segmentation, while recognizing word strings requires context and explicit character order.
Method
DTRN slides a CNN window through a word image to generate an ordered deep sequence, then uses an LSTM-based recurrent model for sequence labeling.
Results
The model achieves state-of-the-art performance and produces clearer recurrent output maps than DeepFeatures on highly ambiguous word images.
Takeaways & Limitations
DTRN can recognize unknown words and arbitrary strings without a predefined dictionary, language model, or pre- and post-processing.
Abstract
from arXiv · showhide
We develop a Deep-Text Recurrent Network (DTRN) that regards scene text reading as a sequence labelling problem. We leverage recent advances of deep convolutional neural networks to generate an ordered high-level sequence from a whole word image, avoiding the difficult character segmentation problem. Then a deep recurrent model, building on long short-term memory (LSTM), is developed to robustly recognize the generated CNN sequences, departing from most existing approaches recognising each character independently. Our model has a number of appealing properties in comparison to existing scene text recognition methods: (i) It can recognise highly ambiguous words by leveraging meaningful context information, allowing it to work reliably without either pre- or post-processing; (ii) the deep CNN feature is robust to various image distortions; (iii) it retains the explicit order information in word image, which is essential to discriminate word strings; (iv) the model does not depend on pre-defined dictionary, and it can process unknown words and arbitrary strings. Codes for the DTRN will be available.
Related Work
Prior scene-text systems largely relied on isolated character classification, segmentation, and post-processing, while whole-word methods used global representations that did not explicitly preserve character order. DTRN instead uses recurrent deep features to exploit word context and ordered spatial information.
- Existing systems commonly separate isolated character classification from subsequent word recognition, limiting their use of whole-word context.
- DTRN differs by exploring recurrence over deep features to recognize whole word images directly without a language model or post-processing.
- Whole-word approaches such as subspace regression and global CNN representations do not provide DTRN’s explicitly ordered sequential feature representation.
- The model is related to CNN-RNN image-captioning systems but preserves strict spatial order because word strings are order-sensitive.
Deep-Text Recurrent Networks
DTRN formulates word-image recognition as sequence labelling: a Maxout CNN produces an ordered feature sequence, and an LSTM-based recurrent model decodes it into text.
- Pipeline: The system encodes a word image into an ordered CNN sequence, then uses an RNN to decode that sequence into a word string.The end-to-end pipeline accepts varying-length images and strings without pre- or post-processing.
- CNN sequence generation: The CNN avoids character-level segmentation by retaining explicit spatial order while extracting strong features from sequential image regions.This design targets robustness to distortions and background clutter while preserving information needed to distinguish word strings.
- CNN sequence generation: A Maxout CNN generates the sequence features, using sliding-window convolution over resized word images and no pooling operation.The architecture contains five convolutional layers followed by grouped Maxout operations; its final layers produce one-pixel outputs for efficient whole-image convolution.
- Recurrent sequence labelling: The recurrent component uses LSTM memory cells to learn long-range dependencies and mitigate the standard RNN’s vanishing-gradient problem.The LSTM uses input, forget, and output gates to regulate memory and retain meaningful interdependencies over long sequences.
- Recurrent sequence labelling: CTC aligns variable-length LSTM outputs with target word strings by removing repeated and non-character labels, enabling end-to-end training without exact character segmentation.The bidirectional LSTM processes the CNN sequence forward and backward, allowing the output to use both past and future information.
Experiments and Results
Experiments on three cropped-word benchmarks show that DTRN improves recognition over character-independent and heavily engineered alternatives while supporting unconstrained strings.
- Evaluation setup: The evaluation uses SVT, ICDAR 2003, and IIIT5K cropped-word recognition benchmarks.SVT provides 647 word images and a 50-word lexicon per image; the passage introduces the benchmark setup.
- Mid-level representation: Mid-level methods show significant accuracy reductions relative to DTRN across all three datasets.The comparison includes Strokelet and Lee et al.’s method.
- Deep neural networks: DTRN outperforms DeepFeatures on SVT, improving accuracy from 86.1% to 93.5%.The authors attribute this to recurrently connecting sequential deep features rather than classifying isolated regions independently.
- Whole image representation: DTRN reaches 94% and 91.5% accuracy on IIIT5K small and large lexicons, respectively.The authors associate these gains with explicit order information in the CNN sequence.
- Training on additional large datasets: DTRN improves over PhotoOCR by 3.1% on SVT while using two orders of magnitude less training data.Unlike PhotoOCR, it does not require a language model or post-processing.
- Unconstrained recognition: DTRN can process unknown words and arbitrary strings, unlike approaches constrained by fixed word sets.The system is presented as supporting unconstrained recognition of any number of characters or word strings.
Conclusion
The paper presents DTRN as a deep sequence-labelling model for scene text recognition. Its ordered deep features and recurrence support robust recognition of ambiguous images, unknown words, and arbitrary strings, with state-of-the-art results.
- Conclusion: DTRN computes explicitly ordered deep features that are robust to image distortions and discriminative for word strings.The conclusion links explicit ordering to word-string discrimination.
- Conclusion: Its recurrence leverages meaningful word context to recognize highly ambiguous images.The conclusion identifies recurrence as the mechanism for using contextual information.
- Conclusion: DTRN processes unknown words and arbitrary strings, providing a more principled approach to scene text recognition.This capability is stated as a consequence of the model’s design.
- Conclusion: Experimental results show that DTRN achieves state-of-the-art performance.The conclusion reports this outcome without specifying a single benchmark value.