Source-linked AI summary
Recursive Recurrent Nets with Attention Modeling for OCR in the Wild
Chen-Yu Lee, Simon Osindero
TL;DR
Lexicon-free scene text recognition remains challenging beyond constrained vocabularies and hand-engineered features. R2AM combines recursive CNN image encoding, recurrent character-level language modeling, and soft attention in an end-to-end image-to-sequence system, achieving state-of-the-art results with absolute gains of 9% on Street View Text and 8.2% on ICDAR 2013. The authors also report that backward and bidirectional RNN extensions did not improve results and therefore did not explore LSTM cells for these short scene-text sequences.
Problem
Lexicon-free scene text recognition must handle words absent from a testing lexicon, whereas earlier methods primarily targeted constrained scenarios with hand-engineered features.
Method
R2AM uses recursive CNNs for image encoding, RNNs for implicit character-level language modeling without manually defined N-grams, and sequential soft attention for feature selection in end-to-end training.
Results
9% on Street View Text and 8.2% on ICDAR 2013 were the absolute accuracy improvements over previous best unconstrained results, with state-of-the-art results across reported benchmarks.
Takeaways & Limitations
The same framework is reported as effective in both constrained and unconstrained scenarios and practically capable of recognizing real-world scene text.
Takeaways & Limitations
Backward and bidirectional RNN extensions delivered no further improvements, and LSTM memory cells were not explored because the scene-image sequences averaged around eight characters.
Abstract
from arXiv · showhide
We present recursive recurrent neural networks with attention modeling (R$^2$AM) for lexicon-free optical character recognition in natural scene images. The primary advantages of the proposed method are: (1) use of recursive convolutional neural networks (CNNs), which allow for parametrically efficient and effective image feature extraction; (2) an implicitly learned character-level language model, embodied in a recurrent neural network which avoids the need to use N-grams; and (3) the use of a soft-attention mechanism, allowing the model to selectively exploit image features in a coordinated way, and allowing for end-to-end training within a standard backpropagation framework. We validate our method with state-of-the-art performance on challenging benchmark datasets: Street View Text, IIIT5k, ICDAR and Synth90k.
1. Introduction
Photo OCR supports applications including navigation, assistive technologies, mobile translation, and image or video search, but prior work largely focused on constrained recognition with hand-engineered features. R2AM addresses lexicon-free scene text recognition using recursive CNNs, recurrent language modeling, and attention, reporting state-of-the-art results with substantial gains on SVT and ICDAR 2013.
- Photo OCR reads scene text in natural images for navigation, assistive technologies, mobile translation, and web image or video search.
- Prior photo OCR research primarily addressed constrained scenarios using hand-engineered image features, fixed lexicons, or known word lengths.Constrained recognition includes region-based methods, pictorial structures, integer programming, CRFs, and related models.
- Deep methods performed well when candidate ground-truth word strings were known but did not generalize to words absent from the testing lexicon.
- R2AM directly learns image-to-word sequences with recursive CNNs, RNN-based character-level language modeling, and sequential soft attention trained end-to-end.Its contributions include weight-sharing for image features, implicit language modeling without manually defined N-grams, and coordinated feature selection during character prediction.
- 9% on Street View Text and 8.2% on ICDAR 2013 were the reported absolute accuracy improvements over previous best unconstrained results.Experiments covered Street View Text, IIIT5k, ICDAR, and Synth90k, with an accompanying ablation study.
2. Methodology
R2AM extends character-sequence OCR with recursive convolutional feature extraction, recurrent decoding, and soft attention. Its architecture addresses contextual dependencies while controlling parameter growth and supports end-to-end image-to-word learning.
- 2.1. Character sequence model review: The proposed system predicts a word’s character sequence from a cropped image using recursive or recurrent CNN features and RNN-based decoding.The Base CNN model is extended with recursive image encoding, character-level language modeling, and attention-based mechanisms.
- 2.2.2 Untying in recursive convolutional layers: Recursive convolutional layers reuse weights across steps, increasing network depth and receptive-field modeling without significantly increasing parameter count.They also provide compact feature responses and can capture higher-order dependencies through lateral interactions within feature maps.
- 2.2.2 Untying in recursive convolutional layers: Untied recursive layers separate the initial feed-forward weights from later recursive weights, allowing different channel counts across layers and more specialized recursive weights.The untied first step also controls computational cost through the output dimensions of the initial transformation.
- 2.2.2 Untying in recursive convolutional layers: Recursive Base CNN variants consistently outperform recurrent versions in the explored tasks, motivating the recursive architecture used in the overall pipeline.Both recursive and recurrent variants improve performance on benchmarks including Synth90k, SVT, and ICDAR13.
- 2.3. RNNs for character-level language modeling: The recurrent decoder models character dependencies by generating one character per time step conditioned on image features, prior hidden state, and prior generated input.The explored designs vary RNN depth, modality factorization, image-captioning-style recurrence, and attention modeling.
- 2.4. Attention modeling: Attention computes positive location weights from image features and the first RNN stack, then sends the weighted context vector to a second RNN stack for output prediction.The mechanism performs soft feature selection at each output step using energy coefficients and weighted image features.
3. Experiments
Experiments evaluate R2AM through component ablations and comparisons on standard lexicon-free scene-text benchmarks. Recursive CNNs, recurrent language modeling, and attention each contribute to the final architecture, which improves substantially over prior unconstrained recognition results.
- Experimental setup: Experiments cover ICDAR 2003, ICDAR 2013, Street View Text, IIIT5k, and Synth90k, with training performed purely on Synth90k training data.Evaluation follows established recognition protocols, including alphanumeric words with at least three characters.
- Recursive and recurrent convolutional layers: More iterations improve both recurrent and recursive CNN accuracy across evaluated datasets, while recursive CNNs outperform recurrent CNNs on all three datasets.The recursive design shares convolutional weights, increasing receptive fields under similar parametric capacity.
- Character-level language modeling: Feeding image features to the RNN at every time step improves performance over first-step-only access, and factored RNNs outperform unfactored variants.The factorized design separates character-statistics modeling from access to encoded image features.
- Character-level language modeling: Attention-based RNNAtten gives the best performance across the five explored RNN variants by rescaling image features before the top-level RNN.The final architecture combines recursive CNN image extraction with RNNAtten character-level modeling and attention.
- Constrained and unconstrained text recognition: R2AM improves absolute accuracy over the previous unconstrained state of the art by 9% on Street View Text and 8.2% on ICDAR 2013.The method also achieves new best unconstrained results on several benchmarks, including SVT, IIIT5k, and ICDAR 2013.
- Character-level language modeling: Qualitative examples show recognition under low contrast, strong transformation, clutter, and missing or occluded characters.The implicitly learned language model can recover characters in examples such as PARK, BURBANK, SAN, and STAR.
4. Conclusion and future directions
The paper presents a lexicon-free photo OCR framework combining recursive CNNs, RNNs, and attention, with state-of-the-art real-world scene-text results. Future work targets improved feature localization, attention control, supervision, and end-to-end text detection.
- The framework combines recursive CNN image encoding, RNN language modeling, and attention-based image-feature usage for lexicon-free photo OCR.
- The authors report effectiveness of each component, generalizability across constrained and unconstrained scenarios, and state-of-the-art recognition of real-world scene text.
- Future work will explore recursive fully convolutional networks to better connect extracted features with their input-image locations and visualize attention coefficients.
- Additional planned extensions include gated units for attention-state updates and deep supervision.
- The authors also plan to extend recognition to text detection, enabling an end-to-end full-image-to-text reading pipeline.