Source-linked AI summary
Synthetic Data for Text Localisation in Natural Images
Ankush Gupta, Andrea Vedaldi, Andrew Zisserman
TL;DR
Natural-image text spotting is hindered by slow and inaccurate word-region proposal pipelines. The paper introduces geometry-aware synthetic scene-text generation and a fully convolutional regression detector, achieving state-of-the-art detection and faster test-time processing. The detector also improves end-to-end word recognition when substituted into existing pipelines.
Problem
Text detection is the bottleneck in text spotting: incorrect and missed word proposals reduce end-to-end F-score to 69% despite 98% recognition accuracy on correctly cropped words.
Method
The paper generates synthetic text blended with natural scenes according to scene geometry, then trains an FCRN that densely predicts word bounding boxes across image locations and scales.
Results
The new data and detector achieve state-of-the-art text detection on standard benchmarks, with an improvement of 6% in F-measure across all datasets, and process up to 15 images per second on a GPU.
Takeaways & Limitations
A CNN trained only on sufficiently realistic synthetic images exceeds state-of-the-art performance for detection and end-to-end text spotting on real images.
Takeaways & Limitations
The detector can fail on unseen fonts, text-like symbols or patterns, extremely small text, and words split or merged by character spacing.
Abstract
from arXiv · showhide
In this paper we introduce a new method for text detection in natural images. The method comprises two contributions: First, a fast and scalable engine to generate synthetic images of text in clutter. This engine overlays synthetic text to existing background images in a natural way, accounting for the local 3D scene geometry. Second, we use the synthetic images to train a Fully-Convolutional Regression Network (FCRN) which efficiently performs text detection and bounding-box regression at all locations and multiple scales in an image. We discuss the relation of FCRN to the recently-introduced YOLO detector, as well as other end-to-end object detection systems based on deep learning. The resulting detection network significantly out performs current methods for text detection in natural images, achieving an F-measure of 84.2% on the standard ICDAR 2013 benchmark. Furthermore, it can process 15 images per second on a GPU.
1. Introduction
The paper addresses text detection as the bottleneck in natural-scene text spotting by combining scalable synthetic data generation with a dense, efficient bounding-box regression network. The resulting system improves detection performance while operating substantially faster than traditional detectors.
- Motivation: Text detection limits end-to-end text spotting because missed or incorrect word proposals reduce F-score despite 98% recognition accuracy on correctly cropped words.The cited pipeline reports 69% end-to-end text spotting F-score versus 98% recognition accuracy for correctly cropped words.
- Synthetic data: The synthetic engine inserts text into natural scenes while aligning it with scene geometry and respecting scene boundaries.It automatically generates SynthText in the Wild, including cluttered conditions and word-level annotations.
- Synthetic data: The approach is trained without manual annotation and uses synthetic images designed to provide detector-suitable scene-level regions rather than only isolated word regions.Existing synthetic text datasets are described as unsuitable for detector training because they contain only word-level image regions.
- Detector: The FCRN predicts bounding-box parameters densely at every image location, rather than only classifying locations as text or non-text.Its design draws on fully convolutional prediction and YOLO-style direct bounding-box regression.
- Results: The new dataset and detector achieve state-of-the-art text detection on standard benchmarks while reaching up to 15 images per second on a GPU.The paper also reports improved end-to-end word recognition after replacing existing detectors in state-of-the-art pipelines.
2. Synthetic Text in the Wild
The paper builds a fast, automated engine for generating large-scale synthetic scene-text data that respects local appearance, region boundaries, and estimated 3D geometry. It produces 800,000 annotated images with varied text rendered into natural backgrounds.
- Limited labelled datasets do not adequately cover natural-scene text variation in fonts, colours, sizes, and positions.
- The automated engine segments backgrounds using local colour and texture cues, estimates dense depth, and processes suitable regions for text placement.
- Text is rendered with region-matched colours, optional borders, perspective alignment to estimated surface orientation, and Poisson blending.
- 800,000 scene-text images are generated, each containing multiple word instances rendered in different styles.
- Text placement is constrained within contiguous regions to avoid crossing strong image discontinuities and is checked for collisions between instances.
- The engine uses imperfect CNN-estimated depth rather than RGBD data to support broader scene coverage despite RGBD datasets’ limited size, resolution, and variability.
3. A Fast Text Detection Network
The paper introduces a fast, end-to-end text detector that densely predicts word presence and bounding-box pose across image locations. Its local, translation-invariant predictors combine direct YOLO-style regression with multi-scale processing.
- Existing CNN text-detection pipelines rely on many region proposals and repeated CNN evaluations, making them slow.
- The detector uses a fixed field of predictors that estimates object presence and bounding-box pose around each image location.
- The proposed FCRN directly predicts detections without voting accumulation while retaining local, translation-invariant, shared predictors.
- Each predictor regresses object confidence and up to six pose parameters: position, size, and bounding-box rotation.
- Multi-scale detection uses input scales {1, 1/2, 1/4, 1/8} and merges overlapping detections through non-maximal suppression.
- The FCRN has 30× fewer parameters than YOLO, whose model size also increases when retrained for each image size and scale.
4. Evaluation
The evaluation compares FCRN variants and synthetic-data settings across standard text-localisation benchmarks, showing strong accuracy, recall, and speed. Multi-scale inference, realistic scene-aware synthesis, and high-recall proposals improve localisation and downstream spotting.
- Evaluation setup: The networks are evaluated on ICDAR 2011, ICDAR 2013, and Street View Text using DetEval and PASCAL VOC-style IoU protocols.The evaluation also compares single-scale and multi-scale inference and uses the detector as a proposal generator.
- Text localisation: Multi-scale FCRN improves maximum recall by more than 12% over single-scale FCRN and outperforms Neumann et al.Single-scale precision at maximum F-measure is comparable to Neumann et al., but recall is 12% worse.
- Text localisation: The high-recall method improves the state-of-the-art by 6% in F-measure across all datasets.The low-recall method already achieves better-than-state-of-the-art text-detection performance.
- Text localisation: On IC13, multi-scale FCRN reaches 85.9% maximum recall before filtering, while FCRNall + multi-filt gains 10.3% maximum recall and 11.1% AP over Jaderberg et al.The refined detections are produced by applying multi-filtering post-processing to the multi-scale FCRN outputs.
- Limitations: The detector fails on unseen fonts, text-like symbols or patterns, extremely small text, and words split or merged by character spacing.The image is not scaled up during detection, limiting performance on extremely small instances.
- Synthetic dataset evaluation: On SVT, restricting synthetic text to local colour and texture regions improves maximum recall by 6.8%, AP by 3.85%, and maximum F-measure by 2.1% over random placement.Adding perspective distortion yields only marginal AP and maximum-F-measure gains and no maximum-recall change.
- End-to-end text spotting: FCRN improves end-to-end text spotting by 8% in F-measure on ICDAR datasets and 3% on SVT, where unlabelled text lowers measured precision.The method performs worse on SVT-50 despite the smaller gain on SVT.
- Timings: FCRN processes about 15 images per second at multiple scales on a GPU and makes the region-proposal stage about 45 times faster.End-to-end spotting is 3× to 23× faster than Jaderberg et al., depending on the variant.
5. Conclusion
The paper concludes that realistic synthetic training data can support a new CNN architecture for text proposals. Trained only on synthetic images, the CNN exceeds state-of-the-art performance for detection and end-to-end text spotting on real images.
- Conclusion: The paper develops a new CNN architecture for generating text proposals in images.The conclusion links this architecture to synthetic training because available annotated datasets contain too few samples.
- Conclusion: Synthetic images with sufficient verisimilitude enable training that exceeds state-of-the-art performance on real-image text detection and end-to-end text spotting.The conclusion states that the CNN is trained only on synthetic images.
A. Appendix
The appendix documents the synthetic dataset and compares the method with Jaderberg et al. on ICDAR 2013 and Street View Text.
- Appendix: The appendix presents synthetic text dataset components and sample images, then compares FCRNall multi-filt with Jaderberg et al. on ICDAR 2013 and SVT.The appendix sections cover dataset examples and detection results on both datasets.
A.1. Variation in Fonts, Colors and Sizes
The appendix varies font, colour, and size while keeping text location and background approximately fixed, illustrating the dataset’s visual diversity.
- Variation in Fonts, Colors and Sizes: Rows render the same text in approximately the same location and background using different fonts, colours, and sizes.The example text is “vamos!”.
A.2. Poisson Editing vs. Alpha Blending
The comparison contrasts simple alpha blending with Poisson Editing for compositing text into images. Poisson Editing preserves local illumination gradients and texture details.
- Poisson Editing is compared directly with simple alpha blending.
- The comparison uses alpha blending in the bottom row and Poisson Editing in the top row.
- Poisson Editing preserves local illumination gradients and texture details.
A.3. SynthText in the Wild
The synthetic text dataset contains varied text rendered against diverse backgrounds. Text instances are transformed using local scene geometry and constrained to contiguous colour and text regions.
- Synthetic images vary text fonts, colours, sizes, borders, and shadows.
- Text is rendered against different backgrounds and transformed according to local geometry.
- The synthesis constrains text placement to local contiguous regions of colour and text.
- Ground-truth word bounding boxes are marked in red.
A.4. ICDAR 2013 Detections
Example ICDAR 2013 detections compare FCRNall + multi-flit with Jaderberg et al. Each image reports precision, recall, and F-measure values.
- The examples compare FCRNall + multi-flit detections with those from Jaderberg et al.
- FCRNall + multi-flit detections are shown in the top row.
- Precision, recall, and F-measure values are indicated at the top of each image.
A.5. Street View Text (SVT) Detections
Example SVT detections compare FCRNall + multi-flit with Jaderberg et al. Both methods have precision of 1 in these examples except one case attributed to missing ground-truth annotation.
- The examples compare FCRNall + multi-flit detections with those from Jaderberg et al.
- Both methods have a precision of 1 on these images.
- One exception is attributed to missing ground-truth annotation.
- Precision, recall, and F-measure values are indicated at the top of each image.