Source-linked AI summary
An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition
Baoguang Shi, Xiang Bai, Cong Yao
TL;DR
The paper addresses image-based sequence recognition, especially scene text, where images contain variable-length sequences rather than single labels. It proposes CRNN, which integrates convolutional feature extraction, recurrent sequence modeling, and transcription in one trainable framework. CRNN achieves superior or highly competitive scene-text results and significantly outperforms competitors on image-based music-score recognition, while remaining compact and applicable beyond scene text.
Problem
Image-based sequence recognition requires predicting variable-length label sequences, while conventional fixed-dimension models and existing pipelines rely on preprocessing or detailed character annotations.
Method
CRNN combines convolutional layers for sequential feature extraction with bidirectional recurrent layers for contextual sequence labeling and produces predictions from image inputs of varying dimensions.
Results
CRNN achieves superior or highly competitive performance on scene-text benchmarks and significantly outperforms competitors on an Optical Music Recognition benchmark.
Takeaways & Limitations
CRNN provides a compact image-based sequence-recognition framework that operates without detailed character annotations and generalizes from scene text to image-based music-score recognition.
Takeaways & Limitations
Further speeding up CRNN for greater practicality in real-world applications remains an open direction.
Abstract
from arXiv · showhide
Image-based sequence recognition has been a long-standing research topic in computer vision. In this paper, we investigate the problem of scene text recognition, which is among the most important and challenging tasks in image-based sequence recognition. A novel neural network architecture, which integrates feature extraction, sequence modeling and transcription into a unified framework, is proposed. Compared with previous systems for scene text recognition, the proposed architecture possesses four distinctive properties: (1) It is end-to-end trainable, in contrast to most of the existing algorithms whose components are separately trained and tuned. (2) It naturally handles sequences in arbitrary lengths, involving no character segmentation or horizontal scale normalization. (3) It is not confined to any predefined lexicon and achieves remarkable performances in both lexicon-free and lexicon-based scene text recognition tasks. (4) It generates an effective yet much smaller model, which is more practical for real-world application scenarios. The experiments on standard benchmarks, including the IIIT-5K, Street View Text and ICDAR datasets, demonstrate the superiority of the proposed algorithm over the prior arts. Moreover, the proposed algorithm performs well in the task of image-based music score recognition, which evidently verifies the generality of it.
1. Introduction
Image-based sequence recognition requires predicting variable-length label sequences from images, but conventional fixed-dimension models and existing pipelines impose preprocessing or annotation constraints. The paper introduces CRNN to address these limitations with an integrated neural architecture for sequence-like objects.
- Problem: Image-based sequence recognition concerns objects such as scene text, handwriting, and musical scores that require predicting label sequences rather than single categories.These objects can vary greatly in length, from two-character words such as “OK” to 15-character words such as “congratulations”.
- Problem: Fixed-dimension DCNN models cannot be directly applied to sequence prediction when input and output lengths vary.Sequence-like objects may have drastically different lengths, creating a mismatch with fixed-dimensional model operations.
- Prior limitations: Character-based scene-text systems detect and crop individual characters before recognition, requiring a strong character detector and detailed character annotations.These pipelines separately train recognition models using labeled character images.
- Prior limitations: RNN-based systems handle sequences without requiring element positions but usually depend on independently designed preprocessing that converts images into feature sequences.This preprocessing separates feature construction from later components, preventing end-to-end training and optimization.
- Proposed approach: CRNN combines DCNN and RNN components to learn directly from sequence labels, avoid hand-crafted features and preprocessing, handle unconstrained sequence lengths, and use fewer parameters than standard DCNNs.The model requires only height normalization and is designed specifically for recognizing sequence-like objects in images.
2. The Proposed Network Architecture
CRNN integrates convolutional feature extraction, recurrent sequence labeling, and transcription into one jointly trained architecture. Its sequence representation preserves spatial order, while bidirectional recurrent layers model context and transcription converts frame predictions into label sequences.
- CRNN combines convolutional layers, recurrent layers, and a transcription layer in a unified network.Convolutional layers extract feature sequences, recurrent layers predict per-frame label distributions, and transcription produces the final label sequence.
- 2.1. Feature Sequence Extraction: Convolutional layers convert each input image into an ordered feature sequence by concatenating feature-map columns from left to right.Each feature vector corresponds to a receptive field in the original image and serves as its local image descriptor.
- 2.2. Sequence Labeling: A deep bidirectional recurrent network predicts a label distribution for each feature frame while using contextual information from both directions.Bidirectional LSTMs combine forward and backward context, and stacked layers form the deep recurrent component.
- 2.2. Sequence Labeling: The recurrent component operates on sequences of arbitrary lengths and propagates error differentials back to the convolutional layers for joint training.Back-propagation through time is connected to convolutional layers through a Map-to-Sequence bridge.
- 2.2. Sequence Labeling: LSTM memory cells and input, output, and forget gates support long-range dependencies in image-based sequences.The recurrent layers use LSTM units to address vanishing gradients and retain relevant past contexts over longer ranges.
- 2.3. Transcription: CTC transcription converts per-frame predictions into label sequences without requiring the positions of individual characters.The conditional probability sums over frame-level paths mapped to a label sequence, and the forward-backward algorithm computes it efficiently.
- 2.3. Transcription: Lexicon-based transcription restricts decoding to nearby lexicon candidates around the lexicon-free prediction.The search uses candidates within a maximum edit distance and approximates lexicon-free decoding by selecting the most probable label at each timestamp before collapsing the path.
3. Experiments
Experiments evaluate CRNN on standard scene-text benchmarks and musical-score recognition, examining accuracy, generality, trainability, and model size. CRNN uses a compact convolutional-recurrent architecture and performs competitively across constrained and unconstrained recognition settings.
- Training and evaluation setup: CRNN is trained once on 8 million synthetic word images and tested on real-world datasets without fine-tuning.The synthetic images are generated by a highly realistic text engine and paired with ground-truth words.
- Scene-text evaluation: Four public datasets—IC03, IC13, IIIT5k, and SVT—are used to evaluate scene-text recognition.The datasets contain cropped or street-view word images with lexicon settings that vary by benchmark.
- Network configuration: The architecture uses rectangular pooling in later convolutional layers to preserve wider feature maps and produce longer feature sequences.A typical 100×32 image containing 10 characters produces a feature sequence of 25 frames.
- Scene-text results: CRNN consistently outperforms most state-of-the-art methods with constrained lexicons and achieves the best performance on SVT in unconstrained recognition.It is lower than some approaches on IC03 and IC13 without a lexicon, while remaining applicable beyond fixed dictionaries.
- Model properties: CRNN is end-to-end trainable, uses learned convolutional features, requires no character ground-truth boxes, supports unconstrained recognition, and has 8.3 million parameters.Weight sharing and the absence of fully connected layers reduce the model to 33MB of RAM.
- Musical-score recognition: On musical-score recognition, CRNN outperforms two commercial systems and demonstrates generality beyond scene text, although the system remains preliminary and misses many functionalities.The score-recognition experiments use convolutional features and recurrent context to handle noise, distortions, and nearby-note information.
4. Conclusion
CRNN integrates CNN and RNN advantages to recognize image-based sequences from varying-dimension inputs with compact models. It performs strongly on scene text and OMR, while further speed improvements remain an open practical direction.
- CRNN accepts input images of varying dimensions, produces predictions of different lengths, and learns directly from coarse labels without character-level annotations.It integrates CNN and RNN advantages within one architecture.
- CRNN achieves superior or highly competitive performance on scene text benchmarks and significantly outperforms competitors on an Optical Music Recognition benchmark.The OMR result supports the framework’s generality beyond scene text.
- Further speeding up CRNN to improve practicality in real-world applications is identified as future work.