Source-linked AI summary
Attention-based Extraction of Structured Information from Street View Imagery
Zbigniew Wojna, Alex Gorban, Dar-Shyang Lee, Kevin Murphy, Qian Yu, Yeqing Li, Julian Ibarz
TL;DR
The paper tackles extraction of useful text from cluttered street-view imagery, where conventional transcription must identify relevant subsets rather than all visible text. It introduces an end-to-end CNN–RNN model with spatial attention, achieving 84.2% accuracy on FSNS versus 72.46% for the previous state of the art while also showing strong results on a harder business-name dataset.
Problem
Extracting useful text subsets from natural scenes is challenging because the relevant text may occupy only part of cluttered, variable imagery.
Method
The paper uses an end-to-end CNN–RNN architecture with novel spatial attention to read selected text from images.
Results
84.2% FSNS accuracy surpasses the previous state of the art’s 72.46%, while the model also gives strong results on Street View Business Names.
Takeaways & Limitations
The attention mechanism supports structured text extraction by reading relevant image regions rather than the whole scene.
Abstract
from arXiv · showhide
We present a neural network model - based on CNNs, RNNs and a novel attention mechanism - which achieves 84.2% accuracy on the challenging French Street Name Signs (FSNS) dataset, significantly outperforming the previous state of the art (Smith'16), which achieved 72.46%. Furthermore, our new method is much simpler and more general than the previous approach. To demonstrate the generality of our model, we show that it also performs well on an even more challenging dataset derived from Google Street View, in which the goal is to extract business names from store fronts. Finally, we study the speed/accuracy tradeoff that results from using CNN feature extractors of different depths. Surprisingly, we find that deeper is not always better (in terms of accuracy, as well as speed). Our resulting model is simple, accurate and fast, allowing it to be used at scale on a variety of challenging real-world text extraction problems.
I. INTRODUCTION
The paper addresses extracting useful text subsets from cluttered natural scenes rather than merely transcribing complete images. It proposes an attention-based CNN–RNN model and evaluates it on challenging Street View text datasets.
- Natural-scene text recognition is difficult because images contain distortion, occlusion, blur, clutter, and viewpoint variation.
- The task requires focusing on useful text while ignoring visual clutter, not simply transcribing all text in an image.
- The proposed model combines CNNs, RNNs, and a novel spatial-attention mechanism.
- 84.2% FSNS accuracy exceeds the previous state of the art’s 72.46%.The FSNS dataset contains over 1M labeled images derived from Google Street View.
- The model is presented as simpler, more accurate, and less assumption-dependent than the previous approach, with strong results on Street View Business Names.
- Across CNN feature extractors, the study examines speed–accuracy tradeoffs and finds that increasing depth can eventually reduce accuracy.
- The paper recommends a CNN configuration intended to balance accuracy and efficiency.
II. METHODS
The model extracts visual features with CNNs, combines them through spatial attention, and uses an RNN to generate text sequentially. Its design compares alternative CNN backbones and examines how feature depth affects the representation.
- The architecture processes images with a CNN and passes attentionally weighted features into an RNN.
- Three CNN feature extractors are considered: inception-v2, inception-v3, and inception-resnet-v2.
- For FSNS, four views share one CNN extractor, whose feature maps are concatenated horizontally into a single feature map.
- The CNN feature map f indexes spatial locations i,j and channels c.
- The RNN acts as a character-level language model that receives image-derived inputs.
- Spatial attention forms a weighted combination of image features into the RNN input at time t.
- The RNN computes an output and next hidden state from its current input and previous state.
- The final letter prediction combines recurrent output with the attentional feature vector and uses greedy decoding.
C. Spatial attention
The paper’s spatial attention mechanism combines image content with time-varying information and explicit pixel coordinates, making attention location aware. This design addresses limitations of permutation-invariant attention and prior local-transition approaches for multiline text.
- Prior spatial-attention methods predict attention masks from the current RNN state.
- The baseline attention combines image content with a time-varying offset through an elementwise tanh transformation.
- The baseline is permutation invariant because the mapping from image features to attention is applied independently at each pixel location.
- The proposed location-aware mechanism adds one-hot encodings of spatial coordinates to image features, equivalent to spatially varying bias terms.
- A prior convolutional location mechanism cannot capture large jumps to the left side of the line below in multiline recognition.
D. Handling multiple views
For FSNS, the model processes four sign views independently with shared CNN parameters, then concatenates their feature maps horizontally into one input representation.
- Each FSNS input sign provides four views, each processed independently by the same CNN-based feature extractor.
- The four resulting feature maps are concatenated horizontally to form a single input feature map.For example, four 16 × 16 × 320 maps become a 64 × 16 × 320 map.
- Shared CNN parameters allow the model to extract features from each view using the same representation process.
E. Training
The model is trained with autoregressive maximum likelihood rather than CTC, using ground-truth histories during training and no text bounding boxes. This training approach improves accuracy and training speed according to the paper.
- The model maximizes autoregressive conditional log-likelihood over output labels, with T = 37 for FSNS.Outputs shorter than 37 characters are padded with a null character.
- Ground-truth labels are supplied as history during training, while predicted labels are used at test time.
- The training procedure does not require ground-truth text bounding boxes.The paper states that this makes collecting training data much easier.
- 6% accuracy improvement and 2x faster training are reported for autoregressive dependencies compared with the CTC-based alternative.The paper contrasts this with the previous FSNS state-of-the-art model, which used CTC loss.
- Training uses stochastic gradient descent with augmentation, regularization, asynchronous multi-machine batches, and network-specific batch sizes.
III. DATASETS
This section introduces the datasets used in the paper’s experiments. It identifies the source material and frames the experimental evaluation around dataset description.
- The section describes the datasets used in the experiments.
- Dataset descriptions provide the experimental data context for the paper.
- The section precedes the paper’s empirical evaluation of its model on those datasets.
A. FSNS dataset
The paper evaluates on FSNS, a large French Street View dataset of street-name signs, using full-sequence accuracy. Its models outperform the previous state of the art, with location-aware attention providing an additional gain.
- FSNS contains 965,917 training images, 38,633 validation images, and 42,839 test images, with up to four 150x150-pixel views per sign.
- Street-name transcriptions contain up to 37 characters, and the model predicts 37 steps with 134 possible characters at each location.
- Full-sequence accuracy requires every predicted character to agree with the ground truth.
- All evaluated models significantly outperform the previous state of the art on the FSNS test set.
- Location-aware attention improves accuracy by 0.9% over standard attention.
B. The effect of depth on FSNS
The study evaluates how CNN feature-extractor depth affects FSNS accuracy and inference speed. Accuracy rises initially but later declines, motivating a shallower configuration that balances performance and computational cost.
- The effect of depth on FSNS: Accuracy improves initially but decreases after CNN depth passes an optimum across inception-v2, inception-v3, and inception-resnet-v2.The authors attribute this pattern to overly high-level features and reduced spatial resolution after max pooling.
- The effect of depth on FSNS: Inception-resnet-v2 reaches 0.833 accuracy, followed by inception-v3 at 0.831 and inception-v2 at 0.807.The reported comparison uses the full-sequence accuracy measure.
- The effect of depth on FSNS: Inception-v3 features from the mixed-5d layer are selected because they are nearly as accurate as the best inception-resnet-v2 cut while being almost twice as fast.This configuration is chosen as optimal for the stated computational budget.
- Visualization: The visualization procedure computes pixel saliency from character-logit derivatives and displays upsampled attention maps alongside it.Saliency maps are smoothed by averaging results from 16 Gaussian-noise perturbations.
- Visualization: During prediction of “Fonds,” both attention and saliency are zero in a view where the word is blurred by preprocessing errors.The model still correctly emits the initial letter “F” at the ninth step.
D. Error Analysis on FSNS
The error analysis finds that many apparent FSNS model errors originate in the reference transcriptions rather than the predictions. Accent mistakes are especially common in both model outputs and ground truth.
- Error sources: 48% of 100 randomly sampled wrong predictions are attributed to incorrect ground truth.The paper reports a more detailed breakdown in Table V.
- Error sources: The most common error concerns the incorrect accent over the letter e, which is also the most common mistake in the ground-truth transcription.The analysis distinguishes acute and grave accent variants.
- Error sources: Figure 4 presents test cases where the model prediction differs from the ground truth, including an example caused by a ground-truth error.These examples illustrate that prediction-reference disagreement does not always indicate a model mistake.
E. Results on internal datasets
On an internal Business Names dataset, the same model produces qualitative examples of extracting names from varied storefront layouts and fonts. Its attention can select a clearer occurrence or move across lines to read multi-line text.
- Dataset scope: The internal dataset is not public, so evaluation is qualitative and is intended to demonstrate applicability across different street-sign types.The authors use the same model rather than introducing a task-specific variant.
- Visualization examples: The FSNS visualization shows saliency maps in red and attention masks in green for the correctly predicted string “Rue des Fonds Gadons.”The figure displays the first 12 prediction steps.
- FSNS comparison examples: Figure 4 provides FSNS examples in which the model prediction differs from the ground truth.The supplied figure caption identifies these as disagreement cases without quantifying their outcomes.
- Business Names examples: For “Lambert’s Tire Service,” the model attends to the horizontal, standard-font occurrence rather than the wavy text at the top of the store.The name appears in two locations in the image.
- Business Names examples: For “Autopartes” followed by “Lubricantes Tauro,” attention scans left to right, then jumps down and left to continue on the second line.The movement resembles a carriage return in a mechanical typewriter.
V. CONCLUSIONS AND FUTURE WORK
The paper presents an end-to-end scene-text approach that achieves state-of-the-art results while extracting structured text by attending to informative image regions. Future work includes extending the system to full structured extraction of business information from storefronts.
- The end-to-end approach achieves state-of-the-art results on the challenging FSNS and internal datasets.
- The attention mechanism extracts structured text by reading only the interesting parts of the whole image.
- The model correctly predicts “Autopartes Lubricantes Tauro” while visualizations show saliency maps and attention masks on a Business Names image.
- Future work includes extending the system to full structured extraction of business information from storefronts.