Source-linked AI summary
Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks
Ian J. Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, Vinay Shet
TL;DR
Recognizing multi-digit street numbers in unconstrained Street View imagery is difficult because localization, segmentation, and recognition are typically separated. The paper unifies these steps with a deep convolutional neural network and reports strong performance across street-number and distorted-text recognition tasks.
Problem
Automatically recognizing multi-digit numbers in complex Street View imagery remains difficult despite its importance for accurately locating buildings on maps.
Method
The paper uses a unified deep convolutional neural network that jointly localizes, segments, and recognizes digit sequences directly from image pixels.
Results
The approach performs strongly across street-number and distorted-text recognition, achieving 99.8% transcription accuracy on the hardest reCAPTCHA category.
Takeaways & Limitations
At specific confidence thresholds, the system produces street-number transcriptions comparable to human operators and suitable for map inclusion.
Takeaways & Limitations
The method assumes sequences have a reasonably small bounded maximum length and is unlikely to scale well to long or unbounded sequences.
Abstract
from arXiv · showhide
Recognizing arbitrary multi-character text in unconstrained natural photographs is a hard problem. In this paper, we address an equally hard sub-problem in this domain viz. recognizing arbitrary multi-digit numbers from Street View imagery. Traditional approaches to solve this problem typically separate out the localization, segmentation, and recognition steps. In this paper we propose a unified approach that integrates these three steps via the use of a deep convolutional neural network that operates directly on the image pixels. We employ the DistBelief implementation of deep neural networks in order to train large, distributed neural networks on high quality images. We find that the performance of this approach increases with the depth of the convolutional network, with the best performance occurring in the deepest architecture we trained, with eleven hidden layers. We evaluate this approach on the publicly available SVHN dataset and achieve over $96\%$ accuracy in recognizing complete street numbers. We show that on a per-digit recognition task, we improve upon the state-of-the-art, achieving $97.84\%$ accuracy. We also evaluate this approach on an even more challenging dataset generated from Street View imagery containing several tens of millions of street number annotations and achieve over $90\%$ accuracy. To further explore the applicability of the proposed system to broader text recognition tasks, we apply it to synthetic distorted text from reCAPTCHA. reCAPTCHA is one of the most secure reverse turing tests that uses distorted text to distinguish humans from bots. We report a $99.8\%$ accuracy on the hardest category of reCAPTCHA. Our evaluations on both tasks indicate that at specific operating thresholds, the performance of the proposed system is comparable to, and in some cases exceeds, that of human operators.
1 Introduction
The paper presents a unified deep convolutional approach that directly recognizes multi-digit numbers from Street View pixels while integrating localization, segmentation, and recognition. It reports over 96% street-number accuracy on SVHN, 97.84% per-digit accuracy, over 90% on a larger Street View dataset, and 99.8% transcription accuracy on the hardest reCAPTCHA category.
- Motivation: Street View number recognition remains difficult because text varies in fonts, colors, styles, orientations, arrangements, lighting, shadows, occlusions, resolution, motion, and focus blur.The task supports map making by transcribing address numbers from geo-located image patches and associating them with street addresses.
- Method: The proposed model unifies localization, segmentation, and recognition using a deep convolutional neural network operating directly on image pixels.The best configuration had eleven hidden layers and was implemented with DistBelief for large-scale distributed training.
- Results: Over 96% accuracy was achieved for complete street-number recognition on SVHN, with 97.84% accuracy for per-digit recognition and over 90% on a larger, more challenging Street View dataset.The per-digit result improved upon the state of the art; the larger dataset contained several tens of millions of street-number annotations.
- Results: 99.8% transcription accuracy was achieved on the hardest category of reCAPTCHA images.CAPTCHAs synthetically distort text to distinguish humans from automated recognition systems.
- Contributions: The paper identifies a unified sequence model with a conditional probabilistic output layer as a key contribution.Its contributions also include evidence that performance is best with a deep architecture and results on difficult reCAPTCHA images.
2 Related work
Convolutional neural networks use tied parameters and convolutional filtering layers, while larger datasets, computational resources, and algorithmic advances have driven recent deep-network successes. Prior applications largely focused on single-object recognition or used CNNs as components within larger systems, motivating the use of scalable DistBelief infrastructure here.
- Convolutional neural networks: CNNs use neurons with tied parameters and layers that apply affine transformations followed by elementwise nonlinearities, with convolution replacing general matrix multiplication.This convolutional structure distinguishes them from fully general neural-network transformations.
- Advances in deep networks: Recent deep-CNN successes have followed increased computational resources, larger training sets, and advances including piecewise linear units and dropout training.These developments accompanied dramatic improvements in object recognition reported by Krizhevsky et al. (2012).
- Scalable training infrastructure: DistBelief provides scalable deep-neural-network infrastructure with convolutional-network support, which this paper uses as the basis for its experiments.Dean et al. (2012) developed DistBelief for large-scale settings where increasing network size can improve training and testing accuracy.
- Prior applications: CNNs were previously used mainly for recognizing single objects, or as feature-extraction components in larger systems for tasks such as object detection and localization.The cited object-detection system included components beyond the neural network portion trained with backpropagation.
3 Problem description
Street number transcription is sequence recognition over digit strings, evaluated by exact recovery of both sequence length and every digit. Because incorrect map entries are costly, systems may reject low-confidence inputs; the task assumes street numbers have at most five digits and targets high-accuracy coverage.
- Accuracy requires predicting the sequence length and every digit correctly, with no partial credit for individual-digit matches.
- Confidence thresholding trades coverage for reliability by discarding uncertain inputs, with map-making focused on coverage at 98% accuracy or better.Coverage is the proportion of inputs not discarded; 98% roughly corresponds to human accuracy.
- Thresholding supports incremental mapping by automating high-confidence cases and reserving the remainder for future systems or human operators.
- The work assumes bounded sequence length, setting the maximum street-number length to N = 5 and requiring systems to reject longer numbers rather than transcribe them incorrectly.Very few street numbers contain more than five digits.
4 Methods
The method learns a probabilistic model P(S | X) for image-conditioned digit sequences, representing sequence length and digit identities with softmax classifiers over convolutional features. It predicts the most likely sequence by independent character-wise maximization with linear-time decoding.
- Probabilistic sequence model: The model maximizes log P(S | X), with S comprising digit variables and a length variable L, while assuming digit identities are mutually independent.The length variable includes an additional value for sequences longer than the maximum modeled length.
- Model parameterization: For street numbers, L has 7 values and each digit has 10, enabling softmax classifiers over features extracted from X by a convolutional neural network.The CNN features are represented by H, whose value is deterministic given the input image.
- Training: Training maximizes log P(S | X) with stochastic gradient descent using standard softmax backpropagation, except absent digits receive no classifier gradient.The length model and each digit model use the same backpropagation rule as isolated softmax layers.
- Inference: At test time, the system predicts the sequence s = (l, s1, . . . , sl) that maximizes log P(S | X).Each character argmax is computed independently and combined with the length probability.
- Inference and preprocessing: The decoding procedure runs in O(N) time by incrementally summing character log probabilities and adding log P(l | x) for each candidate length.Preprocessing subtracts each image mean without whitening or local contrast normalization.
5 Experiments
The experiments show that a deep convolutional network can recognize multi-digit street numbers directly from images, achieving high sequence accuracy on public and internal datasets and strong performance on distorted CAPTCHA text. Analyses further examine depth, while deployment results demonstrate worldwide-scale street-number extraction.
- 5.1: The experiments evaluate complete multi-digit sequences in original SVHN images rather than cropping and recognizing individual digits separately.SVHN contains about 200,000 street numbers and roughly 600,000 individually boxed digits.
- 5.1–5.3: The system achieved 96.03% sequence accuracy on SVHN, 91% on the larger internal dataset, and 99.8% on the hardest reCAPTCHA puzzles.These results cover public street-number recognition, noisier large-scale street-number recognition, and distorted text recognition.
- 5.1: On SVHN, confidence thresholding provided 95.64% coverage at 98% accuracy, matching human-operator performance for map inclusion.The unthresholded 96.03% sequence accuracy was insufficient for automated map placement, whereas the thresholded operating point was considered acceptable.
- 5.2: On the internal dataset, confidence thresholding reached 83% coverage at 99% accuracy or 89% coverage at 98% accuracy despite broader variation and noisier labels.The dataset spans more than 12 countries, includes non-digit characters, and lacks ground-truth bounding boxes; its imagery contains challenging layouts, blur, shadows, and occlusions.
- 5.3: The reCAPTCHA experiment tested generality beyond street numbers using case-sensitive sequences of up to eight characters from the hardest distorted puzzles.Its model used nine convolutional layers and accepted CAPTCHA-word crops resized to 200x40 pixels.
- 5.4: Performance analysis indicates that fairly deep architectures are needed for sequence transcription because the task jointly involves localization, segmentation, and recognition.The analysis hypothesizes that depth is crucial for an efficient representation of this complicated task.
6 Discussion
The authors argue that a single neural network can jointly localize and segment ordered sequences end to end, with potential applications beyond number recognition. They identify bounded sequence length and training-set quality and size as key limitations or opportunities for improvement.
- The authors believe the model solves OCR for short sequences in many applications, while further gains could come from improving and enlarging the training set for general OCR transcription.
- The method assumes a reasonably small maximum sequence length N and may not scale well to long or unbounded sequences because each digit classifier requires a separate weight matrix.For long sequences, the separate matrices could impose excessive memory costs, although DistBelief can mitigate memory constraints by using more machines.
- A possible extension is to train a model that outputs one N-character word at a time, slide it across the image, and decode the resulting predictions.Early experiments with this approach were promising.
- A single neural network can learn simultaneous localization and segmentation of ordered object sequences, potentially extending to general text transcription and speech recognition.The authors present this end-to-end capability as their most interesting finding.
Appendix A: Example inference
The appendix illustrates inference for transcribing a house number: the network computes sequence probabilities from image features, then selects the highest-probability transcription using numerically stable log probabilities. In the worked example, the model correctly outputs 175.
- Appendix A: Example inference: Inference computes P(S | X) from one image-wide feature vector H extracted by a convolutional network with a fully connected final layer.The computational graph in Figure 8 describes the parameters of the sequence distribution from the input image.
- Appendix A: Example inference: The selected transcription is argmax_s log P(S = s | H), avoiding numerical underflow from multiplying many small probabilities.Log softmax values should be computed directly with a stable formula rather than by taking logarithms after softmax.
- Appendix A: Example inference: Under the independence assumptions, the most likely digit can be selected separately at each position before comparing all possible sequence lengths.The cumulative log-probability calculation can be performed in linear time and generated incrementally without storing the final table.
- Appendix A: Example inference: The model outputs the correct transcription, 175, with maximal log probability −0.42144.The example distribution assigns the correct length 3 and highest probability to digits 175, while retaining uncertainty about alternative digits and length 4.