Source-linked AI summary
Deep Structured Output Learning for Unconstrained Text Recognition
Max Jaderberg, Karen Simonyan, Andrea Vedaldi, Andrew Zisserman
TL;DR
The paper tackles word recognition in natural images without a fixed lexicon, including unknown-length and non-language strings. It combines position-wise character and N-gram CNN predictors in a CRF trained by structured-output back-propagation, achieving strong unconstrained and state-of-the-art dictionary-constrained recognition. The model also generalises to previously unseen words while using purely synthetic training data.
Problem
The central problem is recognising words in natural images without restricting outputs to a fixed dictionary, including previously unseen and non-language strings.
Method
A CNN-based CRF combines position-dependent character predictions with position-independent N-gram detection and is jointly trained using structured-output loss.
Results
The model outperforms character prediction alone on standard benchmarks, achieves state-of-the-art dictionary-constrained accuracy, and generalises to unseen words.
Takeaways & Limitations
One model supports both language and non-language recognition, including random alphanumeric strings, while retaining strong performance under dictionary constraints.
Takeaways & Limitations
The N-gram representation relies on language statistics, so non-language words may contain fewer modelled N-grams and have more ambiguous encodings.
Abstract
from arXiv · showhide
We develop a representation suitable for the unconstrained recognition of words in natural images: the general case of no fixed lexicon and unknown length. To this end we propose a convolutional neural network (CNN) based architecture which incorporates a Conditional Random Field (CRF) graphical model, taking the whole word image as a single input. The unaries of the CRF are provided by a CNN that predicts characters at each position of the output, while higher order terms are provided by another CNN that detects the presence of N-grams. We show that this entire model (CRF, character predictor, N-gram predictor) can be jointly optimised by back-propagating the structured output loss, essentially requiring the system to perform multi-task learning, and training uses purely synthetically generated data. The resulting model is a more accurate system on standard real-world text recognition benchmarks than character prediction alone, setting a benchmark for systems that have not been trained on a particular lexicon. In addition, our model achieves state-of-the-art accuracy in lexicon-constrained scenarios, without being specifically modelled for constrained recognition. To test the generalisation of our model, we also perform experiments with random alpha-numeric strings to evaluate the method when no visual language model is applicable.
1 INTRODUCTION
The paper addresses unconstrained recognition of text in natural images without a fixed dictionary, using a joint CNN–CRF model that combines character and N-gram evidence. It reports strong unconstrained and dictionary-constrained performance while training solely on synthetic data.
- Problem: Unconstrained text recognition targets natural-image words without restricting outputs to a fixed lexicon or dictionary.The recognition stage operates on word regions after detection produces bounding boxes.
- Motivation: Existing CNN word-recognition methods often rely on known dictionaries, limiting generalisation to unseen or non-language strings.Examples include generic alphanumeric strings such as number plates and phone numbers.
- Method: The proposed architecture combines position-dependent CNN character predictions with position-independent CNN N-gram terms inside a CRF.The output sequence maximises the CRF score, enforcing consistency among individual predictions.
- Training: Structured-output learning jointly trains the character predictor, N-gram predictor, and their combination in the CRF.Training uses back-propagation of the structured output loss and constitutes multi-task learning.
- Results: The system achieves excellent unconstrained recognition and state-of-the-art accuracy with standard dictionary constraints, despite training only on synthetic data.It is also evaluated on synthetic data under different scenarios.
2 RELATED WORK
Prior text-recognition systems commonly use sequential or holistic representations and strong dictionary or language-model constraints. The reviewed character-sequence and N-gram approaches instead provide complementary ways to represent word images, including synthetic-data generalisation to real-world recognition.
- Scope: The paper focuses on recognition from cropped single-word images rather than the preceding text-detection stage.Text spotting systems first generate word detections; this work addresses recognition within those regions.
- Sequential recognition: Traditional methods classify characters sequentially and integrate classifier predictions left to right before making a word prediction.Reported classifiers include random ferns and CNNs.
- Proposal-based recognition: Other methods generate candidate character regions through oversegmentation, classify them, and use beam search or dictionary re-ranking.PhotoOCR combines classifier scores with a static N-gram model and a 100k-word dictionary.
- Holistic recognition: Holistic word-recognition methods pool evidence from the full word image, sometimes retaining explicit character classifiers within an inference graph.The cited approaches include graph-based and Fisher-vector representations.
- Language constraints: The reviewed methods make use of strong static language models through constrained dictionaries or re-ranking mechanisms.
- Synthetic training: CNN position-sensitive character outputs have been applied to street numbers and CAPTCHA sequences using synthetic training data.The paper extends the relevance of synthetic training beyond synthetic problems to real-world recognition.
- Graphical models: Graphical models with back-propagated losses have previously combined neural predictions for text recognition and human pose estimation.Examples include segmentation-based character models and an MRF-like spatial model with one message-passing round.
3 CNN TEXT RECOGNITION MODELS
The component models encode whole word images either as position-wise character sequences or sparse bags of N-grams. The latter exploits compositional structure, while the character model remains dictionary-free but assumes independent character positions.
- Character sequence model: The character sequence model predicts each character position independently from shared CNN features, allowing completely unconstrained recognition.It uses the whole word image and makes no assumptions about the underlying language.
- Character sequence model: Variable-length words are represented using Nmax output positions, set to 23 as the maximum training-word length.Each position has a classifier over 36 alphanumeric classes plus a null character.
- Character sequence model: The character model assumes independence between characters when estimating the most probable word.This independence follows from the unconstrained-recognition formulation.
- Character sequence model: Each position-specific probability is produced by a classifier operating on the shared CNN representation Φ(x).The predicted word is formed from the most probable character at each position.
- Bag-of-N-grams model: The bag-of-N-grams model represents a word as an unordered set of character substrings up to length N.For example, G3(spires) contains character unigrams, bigrams, and trigrams.
- Bag-of-N-grams model: With N = 4, the N-gram encoding has only 7 collisions among 90k dictionary words and averages about 22 active N-grams out of 10k.The resulting binary vector is sparse and nearly unique for natural-language words.
- Bag-of-N-grams model: A CNN N-gram detector predicts the presence of each modelled N-gram somewhere in the input image using independent logistic outputs.Training therefore comprises separate binary classification tasks for the N-gram classes.
- Bag-of-N-grams model: Selecting N-grams from language statistics compresses the representation, but non-language strings may have fewer and more ambiguous encodings.This creates a scope boundary for the language-informed representation.
4 JOINT MODEL
The joint model combines position-dependent character scores with position-independent N-gram scores in a CRF, then trains their combination using structured output learning. Beam search approximates the highest-scoring word path, while back-propagation jointly optimises the CNN components and CRF weights.
- Joint CRF model: The character and N-gram CNN outputs provide the CRF score terms, with optional weights that may be shared across characters, positions, or N-gram orders.The character predictor supplies unary terms and the N-gram predictor supplies edge terms; unmodelled N-grams receive score 0.
- Joint CRF model: The CRF score combines unary character predictions with higher-order N-gram scores applied across word positions.Character scores are position-dependent, whereas the N-gram scoring function is position-independent and applied repeatedly at each position.
- Structured-output training: Training imposes a margin requiring the ground-truth word to outscore the highest-scoring incorrect word, using a soft convex loss.The structured objective averages this loss over training examples and can learn the scoring weights jointly with the CNN functions.
- Structured-output training: Back-propagation through the structured loss jointly optimises the character predictor, N-gram predictor, and their combination in the CRF.Errors are propagated through the CNN outputs responsible for margin violations, enabling end-to-end learning with standard back-propagation and SGD.
- Inference: Beam search approximates the highest-scoring incorrect or predicted path because exhaustive search is too expensive with high-order terms.The CRF graph contains cliques of order N, making exhaustive maximisation impractical when N is moderately large.
- Inference: The model uses a weak language model that gives nominal scores to infrequent N-grams without forbidding their recognition.N-grams outside the target set receive score 0 rather than −∞, so they remain possible predictions.
5 EVALUATION
The evaluation uses standard real-world and synthetic datasets to test recognition accuracy, vocabulary generalisation, random-string handling, and lexicon-constrained performance. JOINT consistently improves on CHAR, generalises to unseen words, and remains competitive with dictionary-based systems.
- Datasets: The models are evaluated on ICDAR 2003, ICDAR 2013, SVT, IIIT5k, Synth90k, and SynthRand, combining real-world and synthetic benchmarks.Synth90k is also used for training, while SynthRand tests recognition of random strings.
- N-gram Encoding: 87.0% maximum F-score on Synth90k and 87.1% on IC03 show that NGRAM accurately models N-gram presence despite class imbalance.The reported F-score is the maximum obtained by sweeping the threshold for classifying an N-gram as present.
- Character Sequence and Joint Models: +4% accuracy improvement on IC03 and SVT shows that JOINT consistently outperforms the CHAR model.Both models are trained on Synth90k and evaluated on standard and synthetic benchmarks.
- Unseen Words: 89.1% JOINT accuracy on a 50/50 vocabulary split is only -2% below the 91.0% accuracy when training and testing vocabularies match.This experiment evaluates recognition of words unseen during training.
- Random Strings: 79.5% JOINT accuracy on random SynthRand strings is close to CHAR’s 80.7%, because N-grams nudge scores rather than impose hard word constraints.The random-string setting lacks the common N-grams found in natural language.
- Comparison to the State of the Art: 93.1% accuracy on IC03 matches the DICT model when JOINT output is constrained with the same 90k dictionary.Unlike DICT, CHAR and JOINT are not trained on a specific dictionary and can operate beyond fixed vocabularies.
6 CONCLUSION
The paper introduces a joint graphical model for word recognition that combines character positions with common N-gram presence and is trained using structured output loss. It supports language and non-language scenarios, generalises to unseen words, and matches state-of-the-art accuracy under lexicon constraints.
- Conclusion: The joint model combines character-position predictions with common N-gram presence in a graphical model trained by back-propagating structured output loss.The formulation is designed to operate identically in language and non-language scenarios.
- Conclusion: The model is more accurate than predicting characters alone and generalises recognition to previously unseen words.The conclusion reports impressive unconstrained recognition performance.
- Conclusion: The model matches state-of-the-art accuracy in lexicon-constrained scenarios without being specifically modelled for constrained recognition.