Source-linked AI summary
End-to-End Text Recognition with Hybrid HMM Maxout Models
Ouais Alsharif, Joelle Pineau
TL;DR
Natural-image text recognition is challenging, and prior work has often addressed only one part of the problem. This paper combines Maxout character recognition with hybrid HMM word recognition in an end-to-end system. The authors report state-of-the-art character and end-to-end results across the benchmarks described in the paper.
Problem
Natural-image text recognition is challenging, and previous work has often focused on a single part of the problem.
Source: §1 · p. 1Method
Results
The system reports state-of-the-art character-recognition results and outperforms previous end-to-end systems on standard challenging benchmarks.
Source: §1 · p. 1, §6.2 · p. 8, §8 · p. 9Takeaways & Limitations
The proposed word recognizer is described as fast, tunable, and scalable with lexicon size, while the integrated system performs well on end-to-end benchmarks.
Source: §1 · p. 1, §8 · p. 9MSER-based character-location extraction does not offer high recall, while the cited highly accurate alternative is too time-consuming for real-time use.
Source: §7 · p. 8
Abstract
from arXiv · showhide
The problem of detecting and recognizing text in natural scenes has proved to be more challenging than its counterpart in documents, with most of the previous work focusing on a single part of the problem. In this work, we propose new solutions to the character and word recognition problems and then show how to combine these solutions in an end-to-end text-recognition system. We do so by leveraging the recently introduced Maxout networks along with hybrid HMM models that have proven useful for voice recognition. Using these elements, we build a tunable and highly accurate recognition system that beats state-of-the-art results on all the sub-problems for both the ICDAR 2003 and SVT benchmark datasets.
1. Introduction
Natural-image text recognition is challenging because lighting, shadows, styles, fonts, and backgrounds make it unlike document recognition. The paper tackles the end-to-end task through character recognition, word recognition, and text detection, proposing solutions across these stages.
1. Introduction
- Lighting, shadows, styles, fonts, and backgrounds shift natural-image text recognition away from document recognition and toward object and handwriting recognition.Source: §1 · p. 1
- The end-to-end task comprises text detection, character recognition, and word recognition, with character recognition nested within word recognition and word recognition within detection.Source: §1 · p. 1
- The design goals are high character accuracy, fast and scalable word recognition, and fast query-time end-to-end performance with high F-measure.Source: §1 · p. 1
- The paper combines new solutions for character and word recognition into an end-to-end system that achieves state-of-the-art F-measure on ICDAR 2003 and SVT.It uses convolutional Maxout networks for character recognition and a hybrid HMM/Maxout architecture for word sequencing.Source: §1 · p. 1
2. Related Work
Prior work addresses character recognition, word recognition, and text detection, but most studies focus on one stage; relatively few evaluate end-to-end systems. The surveyed approaches trade accuracy, speed, and scalability in different ways.
2. Related Work
- Most previous studies focus on one pipeline stage, while relatively few examine end-to-end text recognition systems.Source: §2 · p. 2
- Character recognition is treated as classification and has used CNNs, deformable parts models, or engineered features followed by a classifier.Source: §2 · p. 2
- Word recognition is a sequence-recognition problem commonly approached with CNNs, CRFs, or pictorial structures, often using lexicons despite the need for scalability to general lexicons.Source: §2 · p. 2
- Text detection approaches trade accuracy, speed, training time, and feature-engineering effort, with fast MSER and Stroke Width Transform methods generally less accurate.Source: §2 · p. 2
3. Technical Background
The technical background introduces convolutional networks, dropout, Maxout networks, and hybrid HMMs as components relevant to the proposed recognition system. The hybrid model uses a probabilistic classifier to estimate HMM state posteriors in place of an explicit observation model.
3. Technical Background
- The section supplies technical background on methods used in the proposed system.Source: §3 · p. 2
3.1. Convolutional Neural Networks
- Convolutional neural networks alternate convolution and pooling layers, with a softmax or RBF layer usually at the end.Source: §3.1 · p. 2
3.2. Dropout
- Dropout stochastically omits network units during learning, reducing overfitting and acting like model averaging over many shared-parameter models.Source: §3.2 · p. 2
3.3. Maxout Networks
- Maxout networks use dropout and a max activation that produces a sparse gradient; they have achieved benchmark results without pretraining.Source: §3.3 · p. 2
3.4. Hybrid HMM models
- Hybrid HMMs use a probabilistic classifier to estimate state posteriors and approximate the observation model through Bayes’ rule.The exposition assumes p(o) is equal for all observations and notes embedded-Viterbi and discriminative training variants.Source: §3.4 · p. 3
4. Character Recognition
The character recognizer uses a convolutional Maxout network trained on augmented character images. It reaches state-of-the-art accuracy on the ICDAR 2003 character test set, while tested binarization and preprocessing methods do not improve accuracy.
4. Character Recognition
- The character task classifies images into 62 classes: 26 uppercase letters, 26 lowercase letters, and 10 digits.Source: §4 · p. 3
- The recognizer is a five-layer convolutional Maxout network trained on 32-by-32 grayscale patches with mean subtraction, standard-deviation scaling, stochastic gradient descent, momentum, and dropout.Source: §4 · p. 3
- Binarization and preprocessing methods including Otsu, random walkers, GrabCut, and ZCA did not improve test accuracy and sometimes decreased it.Source: §4 · p. 3
- 85.5% accuracy on the 62-way case-sensitive benchmark and 89.9% on the case-insensitive 36-way benchmark were reported for the ICDAR 2003 character test set.The table compares character-recognition accuracy on that test set, with all methods using the same augmented training dataset.Source: Table 1 · p. 3, §4 · p. 3
5. Word Recognition
The section presents a segmentation-based word recognizer that combines hybrid HMM/Maxout segmentation, corrected segmentation alternatives, and beam-based inference. A lexicon can be integrated during inference or used afterward.
5. Word Recognition
- Word recognition uses a Hybrid HMM/Maxout model to segment images, builds a cascade of possible characters, then uses speed-accuracy trade-off inference to return candidate words.The lexicon may be incorporated during inference or used as a post-processing tool.Source: §5 · pp. 3–4
5.1. Hybrid HMM Maxout Model
- The hybrid HMM segments word images into character/inter-character regions; each inter-character training region extends 10% into both adjacent characters.The model uses four HMM states per region, and one embedded-Viterbi iteration sufficed to learn segmentation.Source: §5.1 · p. 4
5.2. Constructing the Cascade
- The correction net predicts over- and under-segmentation, joins adjacent intervals when combined confidence exceeds both pieces, then splits resulting intervals at midpoints to form a graph-linked cascade.Over-segmentation can split a character into other characters; under-segmentation is associated with difficult fonts, blurry images, or complex background noise.Source: §5.2 · p. 4
5.3. Word Inference
- Word decoding is a path search over node-and-edge costs; Viterbi-style inference is optimal without a language model or with bigrams, while higher n-grams use top-B beam search to avoid O(K^n).When each interval has O(1) predecessors, computing S takes O(K^2V + V log(V)).Source: §5.3 · p. 5
5.4. Word Recognition Results
- 85.1% accuracy with a 1,000-fold lexicon compares favorably with Novikova et al.’s 78% at 90-fold growth; the method also outperforms prior state-of-the-art algorithms on the specified benchmarks.Tests covered ICDAR 2003 lexicon scenarios and an SVT distractor-word setting; evaluation excluded words with non-alphanumeric characters and words of length two or less.Source: Table 2 · p. 6, §5.4 · p. 6
5.4.1. EFFECT OF BEAM WIDTH
- Almost 15 times faster, the word-recognition module retains most accuracy with a small beam width on the ICDAR 2003 small-lexicon task.Beam width trades inference speed against accuracy.Source: Figure 4 · p. 6, §5.4.1 · p. 6
5.4.2. EFFECT OF LANGUAGE MODEL ORDER
- 67.0% was the highest accuracy in Large*, where higher language-model orders overfit and reduce recognition accuracy.Large* uses the large lexicon without adding ground-truth words and is evaluated case-insensitively.Source: Figure 5 · p. 7, §5.4.2 · p. 7
6. End-to-End Text Recognition
The section integrates text localization with the paper’s word-recognition module and evaluates the resulting system on ICDAR 2003 and SVT. The authors report outperforming previous end-to-end systems, including on large lexicons.
6. End-to-End Text Recognition
- Text localization is the main issue the preceding character- and word-recognition sections leave unaddressed.This section integrates the preceding modules into a full end-to-end system.Source: §6 · p. 7
6.1. End-to-End Pipeline
- MSERs quickly propose image regions, which DB-SCAN clusters at multiple distances into candidate line-level boxes.MSER regions are imprecise, but their speed avoids costly multi-scale recognition.Source: §6.1 · p. 7
- The pipeline segments candidate lines into words, adds words by introducing gaps in descending size order, then filters word boxes with a detector and recognition-derived scores.The segmentation model is trained on ICDAR 2003 scene data; non-max suppression removes boxes overlapping by 30% according to visual cost.Source: §6.1 · p. 7
6.2. End-to-End Results
- The evaluation uses 249 images per dataset and tests five ICDAR 2003 lexicon settings alongside the SVT-provided lexicons.A prediction counts as a hit only when box overlap exceeds 50% and the predicted text matches exactly.Source: §6.2 · pp. 7–8
- The system outperforms previous end-to-end systems and produces reasonable results for large lexicons despite using MSERs.Table 3 compares end-to-end F-measures on ICDAR 2003 and SVT.Source: §6.2 · p. 8, Table 3 · p. 8
7. Discussion
The discussion identifies accuracy losses in word recognition and recall limitations in end-to-end detection, then describes possible improvements and their trade-offs.
7. Discussion
- Most word-recognition accuracy loss comes from hybrid-HMM segmentations; context-conditioned neural posteriors are proposed to reduce it.Learned edit distances are another proposed way to improve word-recognition accuracy.Source: §7 · p. 8
- MSERs do not offer high recall for character locations, while the cited higher-accuracy classifier is too time-consuming for real-time operation.The authors propose a feature-learning Viola–Jones-style cascade as a promising detector approach.Source: §7 · p. 8
8. Conclusion
The conclusion presents a system that combines solutions across the text-recognition pipeline. It reports state-of-the-art character recognition and end-to-end benchmark performance, alongside a fast, tunable word recognizer.
8. Conclusion
- The system combines convolutional Maxout character recognition, a fast and tunable word recognizer, and an end-to-end system that outperforms prior work on standard challenging benchmarks.The word recognizer is described as highly accurate and scaling elegantly with lexicon size.Source: §8 · pp. 8–9