PaperlayerView exact revision on arXiv

Source-linked AI summary

Reading Text in the Wild with Convolutional Neural Networks

Max Jaderberg, Karen Simonyan, Andrea Vedaldi, Andrew Zisserman

arXiv:1412.1842v1cs.CV

TL;DR

Text spotting must handle natural-scene text that varies in appearance and layout beyond document OCR, while also supporting retrieval from large visual collections. The paper addresses this with a high-recall proposal pipeline and whole-word CNN recognition trained on synthetic data. It reports substantial improvements across standard spotting and retrieval datasets and demonstrates instant search over thousands of hours of news footage.

  • Problem

    Natural-scene text varies in fonts, lighting, occlusion, orientation, noise, and layout, making document-oriented OCR unsuitable for this setting.

  • Method

    The system combines fast complementary region proposals, filtering and bounding-box refinement, whole-word deep-CNN recognition, and recognition-informed merging and ranking.

  • Results

    The system significantly improves text spotting and image retrieval over previous methods across standard datasets and enables instant search through thousands of hours of archived news footage.

  • Takeaways & Limitations

    High-recall proposals and accurate recognition support a fast, scalable pipeline for text-based visual search from hundreds to millions of images.

  • Takeaways & Limitations

    The system can miss slanted or vertical words lacking suitable proposals, and sub-word or multiple-word detections can produce false positives.

Abstract

from arXiv · show

In this work we present an end-to-end system for text spotting -- localising and recognising text in natural scene images -- and text based image retrieval. This system is based on a region proposal mechanism for detection and deep convolutional neural networks for recognition. Our pipeline uses a novel combination of complementary proposal generation techniques to ensure high recall, and a fast subsequent filtering stage for improving precision. For the recognition and ranking of proposals, we train very large convolutional neural networks to perform word recognition on the whole proposal region at the same time, departing from the character classifier based systems of the past. These networks are trained solely on data produced by a synthetic text generation engine, requiring no human labelled data. Analysing the stages of our pipeline, we show state-of-the-art performance throughout. We perform rigorous experiments across a number of standard end-to-end text spotting benchmarks and text-based image retrieval datasets, showing a large improvement over all previous methods. Finally, we demonstrate a real-world application of our text spotting system to allow thousands of hours of news footage to be instantly searchable via a text query.

1 Introduction

The paper presents an end-to-end system for detecting and recognising text in natural images, addressing the variability that makes document OCR unsuitable. It combines high-recall proposals, whole-word CNN recognition, and large-scale retrieval, reporting strong performance across standard benchmarks and a news-video application.

  • Motivation: Text spotting is important for decoding semantic content from the labels and textual cues scattered throughout images and videos.The paper connects text spotting with understanding, annotating, and retrieving visual media.
  • Motivation: Natural scene text varies in fonts, styles, lighting, occlusion, orientation, noise, and background clutter, limiting document-oriented OCR.Traditional OCR is tuned to largely black-and-white, line-based printed documents.
  • Contributions: A deep CNN recognises whole word images across a dictionary such as the 90k-word dictionary, trained purely on synthetic data without human labelling.The paper also introduces incremental learning for the large number of classes.
  • Contributions: The system uses complementary proposal methods and filtering to achieve around 98% word recall on ICDAR 2003 and Street View Text.A random forest filters false-positive proposals, while a CNN refines their bounding-box coordinates.
  • Results: The complete system significantly outperforms previous methods across standard text-spotting and retrieval datasets and supports instant search through thousands of hours of archived news footage.The authors report this application as a real-world demonstration of text-based visual retrieval.

2 Overview of the Approach

The approach preserves high recall early by generating many inexpensive word proposals, then improves precision through filtering, regression, recognition, and recognition-informed merging. Whole-word CNN recognition uses synthetic training data and produces state-of-the-art accuracy on real-world word images.

  • Pipeline: The pipeline generates word proposals, filters and adjusts them, recognises text, and finally merges detections for the task-specific output.The full process is illustrated as a sequence of proposal generation, filtering, recognition, and merging stages.
  • Pipeline: The system treats detection and recognition as connected stages because recognition outputs help merge and rank detections.This creates a more holistic text-spotting process than a strictly separated detector-recogniser pipeline.
  • Detection: Fast Edge Box and aggregate-channel proposals avoid exhaustive search while providing candidate word bounding boxes for later processing.A random forest reduces false positives, and a CNN regresses more accurate bounding boxes.
  • Recognition: Whole-word recognition feeds each cropped proposal to a deep CNN that classifies it across a dictionary of potential words.Synthetic rendering produces variable word images that transfer to real-world recognition.
  • Merging and ranking: Recognition information supports multiple rounds of non-maximal suppression and bounding-box regression before final results are produced.These operations update detection results after recognition.

3 Related Work

Earlier text-spotting systems commonly separate detection and recognition, using character regions, sliding windows, character classifiers, or whole-word representations. This paper instead carries many high-recall proposals through recognition and uses recognition results to complete detection, while applying whole-word dictionary classification with a deep CNN.

  • Text Detection Methods: Text detection methods produce word segmentations or bounding boxes in noisy natural scenes using character-region or sliding-window approaches.Character-region methods segment and group characters, whereas sliding-window methods frame detection as object detection.
  • Text Recognition Methods: Earlier CNN-based systems used text, character, or bigram classifiers, often combining their outputs with sequence scoring or language models.These systems commonly relied on explicit character-level processing.
  • Position of This Work: The paper carries many candidates through a high-recall, low-precision pipeline and completes detection only after recognition-based merging and ranking.This differs from approaches that select a single word location proposal before recognition.
  • Text Recognition Methods: Whole-word recognition methods pool evidence across the word image, but several prior approaches still depend on explicit character classifiers or word-image representations.Other methods use embeddings or compare word images with rendered word forms.
  • Position of This Work: The proposed recogniser uses the whole word image as input to a deep CNN and performs multi-way classification over a dictionary of potential words.The authors contrast this dictionary model with prior character-based recognition systems.

4 Proposal Generation

The proposal-generation stage combines fast, complementary detectors to produce high-recall word bounding boxes while keeping computation manageable. Edge Boxes and an aggregate channel features detector together substantially improve coverage over either method alone.

  • Proposal-generation strategy: The pipeline uses a fast, high-recall detection phase followed by increasingly informative models that improve precision by rejecting false positives.Region proposals reduce the search space relative to exhaustive multi-scale, multi-aspect-ratio sliding-window classification.
  • Proposal-generation strategy: Edge Boxes and an aggregate channel features detector generate complementary candidate word bounding boxes.Edge Boxes score enclosed contours, while the aggregate channel features detector uses ACF features with an AdaBoost classifier.
  • Implementation: The aggregate channel features detector accelerates multi-scale processing by estimating intermediate feature-pyramid scales from resampled existing features.This avoids recomputing channel features independently at every scale.
  • Results: Edge Boxes provided the best recall and overlap ratio among the evaluated region proposal algorithms.The authors also found some alternatives too slow, while BING produced poorer overlap and overall recall.
  • Results: 98% recall is achieved by combining Edge Boxes and the aggregate channel features detector, compared with 92% and 70% recall independently.Recall is computed at 0.5 overlap.

5 Filtering & Refinement

The filtering and refinement stage reduces the thousands of false-positive proposals and improves their localization for accurate text recognition. It combines HOG-based random-forest filtering with CNN bounding-box regression.

  • Filtering and refinement: Thousands of high-recall proposals are filtered before the expensive text-recognition stage, while a regressor refines proposals with poor ground-truth overlap.The filtering stage targets computational manageability and the regression stage targets localization quality.
  • Word classification: A random forest classifies HOG descriptors for every proposal and rejects proposals below a threshold.The retained proposals form the filtered set Bf.
  • Localization motivation: A 0.5 overlap can be inadequate for text recognition because a proposal may expose only half of a word despite meeting the detection criterion.This occurs when one bounding-box dimension is substantially misestimated while the other is accurate.
  • Bounding-box regression: CNN bounding-box regression updates each filtered proposal using encoded corner coordinates predicted from an enlarged crop centered on the proposal.The crop is inflated by a factor of two, and training minimizes L2 loss between encoded predicted and ground-truth boxes.
  • Design choice: A CNN classifier was not significantly better than HOG with a random forest, while requiring greater complexity.The authors therefore selected the simpler classification approach after comparing alternatives.
  • Bounding-box regression: Bounding-box regression improves overlap for individual samples and causes duplicate proposals for one word instance to converge toward shared coordinates.This also supports the later voting and merging mechanism.

6 Text Recognition

Text recognition classifies whole cropped word images with a dictionary-based CNN rather than recognizing characters separately. Synthetic rendering supplies the large, varied training set needed for a 90k-word model, while fixed-size resampling handles variable word widths.

  • CNN word recognition: The recognition CNN maps whole word images to a dictionary-based probability distribution over possible words.The model formulates recognition as multi-class classification with one output class per dictionary word.
  • CNN word recognition: A 90k-word dictionary is supported by training the recognition model purely on synthetic data.The synthetic dataset replaces the unavailable large real-world collection required for classification at this scale.
  • Synthetic training data: Available real word-image datasets contain only thousands of samples and limited vocabularies, motivating synthetic whole-word training.Earlier approaches consequently relied on character classifiers, data mining, or proprietary datasets.
  • Synthetic training data: The synthetic engine renders varied scene-text samples through font, border or shadow, coloring, and projective-distortion operations applied to layered images.The process is designed to mimic real-world scene-text appearance while generating labels from a corpus or dictionary.
  • CNN architecture: The five-convolutional-layer, three-fully-connected-layer CNN uses a final layer whose size matches the target dictionary.Its feature-map dimensions are illustrated in the network schematic.
  • CNN architecture: Fixed-size input requires resampling word images to a common width and height even though word widths vary with character count.The authors note that this does not preserve aspect ratio but may provide word-length cues through horizontal feature distortion.
  • Recognition pipeline: For each proposal, the system crops and resamples the image, computes CNN word probabilities, and combines them with a task-dependent language prior.The resulting distribution ranks dictionary words for the proposal.

7 Merging & Ranking

The pipeline merges, ranks, and refines word detections using recognition scores, label-aware suppression, and recurrent bounding-box regression. For retrieval, it aggregates per-image word probabilities and answers multi-word queries efficiently.

  • Text spotting: Each detected box receives a word label and score from its maximum word probability.
  • Text spotting: Greedy NMS clusters duplicate detections sharing a word label, then suppresses overlapping detections with different labels.Scores of suppressed same-label proposals are aggregated as positional voting.
  • Text spotting: Multiple regression and NMS rounds update localization context and group boxes that become similar after refinement.This recurrent process improves overlap for applications requiring more than 0.5 overlap.
  • Image retrieval: For retrieval, the system averages detection-level word distributions to estimate each image's distribution across the word space.The per-image distribution is computed offline for every image.
  • Image retrieval: At query time, images are scored for one or more query words and ranked in descending score order.An inverted index enables processing queries across millions of images in less than a second after offline preprocessing.

8 Experiments

The experiments evaluate the pipeline across standard text spotting and text-based image retrieval benchmarks. They separately report datasets, component-level implementation and results, and end-to-end benchmark performance.

  • The evaluation covers standard benchmarks for both text spotting and text-based image retrieval.
  • The experimental structure introduces datasets, details each pipeline component, and reports text spotting and retrieval results separately.These stages correspond to Sections 8.1 through 8.4.

8.1 Datasets

The evaluation uses diverse recognition, spotting, retrieval, and video datasets, including synthetic data and large-scale BBC news footage. Dataset annotations and lexicons vary across benchmarks.

  • The evaluation combines datasets for text recognition, text spotting, and image retrieval, with lexicons narrowing recognition search spaces where available.
  • Synthetic data: The synthetic dataset contains 9 million 32 × 100 word images drawn from a 90k-word dictionary.It is split into 900k testing, 900k validation, and the remainder for training.
  • Recognition and spotting datasets: SVT contains 249 high-resolution Google StreetView road scenes with substantial noise and many unannotated words.Per-image 50-word lexicons are also provided.
  • Recognition and spotting datasets: IIIT 5k contains 3000 cropped scene-text and digital word images, each paired with 50-word and 1k-word lexicons.
  • Retrieval datasets: IIIT STR and Sports-10k evaluate retrieval using 10k-image collections with query words but no word bounding-box annotations.Sports-10k uses low-resolution, noisy or blurred sports-video frames.
  • Video dataset: BBC News contains approximately 12 million frames from around 5000 hours of programming, processed into 2.3 million keyframes.

8.2 Implementation Details

The implementation combines complementary proposal generators, filtering, box regression, and whole-word CNN recognition. Experiments show high proposal recall, reduced proposal counts, improved localization overlap, and strong recognition from synthetic training data.

  • Pipeline setup: The same models and hyperparameters are used across datasets and experiments to assess framework generalisability.
  • Proposal generation: Combining Edge Boxes and ACF proposals reaches 98% recall at 6k proposals on IC03 and 97% at 11k proposals on SVT.Recall is measured at 0.5 overlap with ground-truth words.
  • Proposal generation: The proposal stage reduces the search space from tens of millions of possible boxes to around 10k proposals per image.
  • Proposal filtering: The random-forest filter retains 96.6% of IC03 and 94.8% of SVT positive proposals while reducing averages to 650 and 900 proposals per image.
  • Bounding-box regression: Bounding-box regression raises average positive overlap from 0.61 to 0.88 on IC03 and from 0.60 to 0.70 on SVT.
  • Text recognition CNN: The recognition CNN classifies whole-word crops across a 90k-word dictionary and is trained on synthetic data.
  • Text recognition results: Recognition accuracy reaches 98.7% on IC03-50, exceeding prior state of the art by 5% on SVT-50 and 3% on IC13.SVT accuracy without a dataset-specific lexicon is 80.7%.
  • Synthetic data effects: Adding random image-layer colouring improves accuracy by 44% on IC03 and 40% on SVT, while natural-image blending adds 1% and 6%.

8.3 Text Spotting

The pipeline achieves strong end-to-end text spotting performance across standard datasets, combining high-recall detection with recognition that supports precise ranking. Its evaluation also exposes annotation and proposal-coverage limitations.

  • Results: 0.85/0.68/0.76 P/R/F on SVT-50 improves over 0.73/0.45/0.56 by +20%.On IC03, F-measure improves by at least +10% across all lexicon scenarios, reaching 0.96/0.85/0.90 P/R/F.
  • Results: The pipeline maintains very high recall while recognition scores provide a strong cue for selecting suitable detections.Precision/recall curves are reported for IC03-50, IC03-Full, and SVT-50.
  • Caveats: SVT precision and F-measure are underestimated because the dataset is only partially annotated, although recall is reported as 71% on SVT-50.Unannotated detected words are recorded as false positives.
  • Results: At a 0.3 overlap threshold, SVT-50 improves by +8%, indicating accurate recognition of loosely cropped detections.Pure word detection reaches 0.85 F-measure on IC03 and 0.81 on IC11.
  • Qualitative results: The system can detect and recognise disjoint, occluded, and blurry words without connected-component algorithms or explicit character recognition.Example results are shown for SVT-50 and IC11, with precision, recall, and F-measure reported above each image.
  • Caveats: A common failure mode is missing slanted or vertical words because the proposal stage does not explicitly model them.Sub-word detections and multiple words grouped together can also produce false positives.

8.4 Image Retrieval

The text spotting pipeline also performs strongly for text-based image retrieval, substantially outperforming prior methods across evaluated datasets. Its scalability enables search over thousands of hours of video, while incomplete annotations constrain some reported metrics.

  • Evaluation: The method computes ranked retrieval lists for each query and evaluates them using average precision and mean average precision.Sports additionally uses precision at n for top-n retrieval.
  • Benchmark results: 90.3% mAP on IC11 exceeds 65.3% from Mishra et al., while Sports P@20 reaches 92.5% versus 43.4%.On SVT, the method achieves 86.3% mAP compared with 56.2% for Mishra et al.
  • Caveats: Reported retrieval precision is hurt by incomplete annotations on the SVT and Sports datasets, so true SVT mAP is higher than 86.3%.The BBC News dataset lacks ground-truth annotations for quantitative retrieval evaluation.
  • Scalability: Processing time is approximately 5–20 seconds per image on one CPU core and one GPU, or 1–2 images per second on a workstation with 16 CPU cores and four GPUs.The pipeline reduces proposals through successive stages, allowing greater computation per remaining proposal while keeping stage times stable.
  • BBC News application: The BBC News application retrieves text occurrences from 5,000 hours of video, with P@100 of 100% for hollywood and boris johnson and 93% for vision.The top two retrieval results are illustrated for the three queries.

9 Conclusions

The paper presents a scalable end-to-end system that substantially improves text spotting and image retrieval across standard datasets. Its synthetic-data training also supports retraining for other languages or scripts without human labelling.

  • Conclusions: The end-to-end pipeline significantly improves text spotting and image retrieval over previous methods on all standard datasets without dataset-specific tuning.The authors attribute this largely to high-recall proposals and a highly accurate text recognition model.
  • Conclusions: The system scales from hundreds of images to millions of images for instant text-based retrieval without perceivable degradation in accuracy.The conclusion characterises the system as fast and scalable.
  • Conclusions: Synthetic-data training enables retraining the recognition model for other languages or scripts without human labelling effort.The paper proposes exploring recognition of unknown words and arbitrary strings in future work.
Loading 1412.1842v1…