Source-linked AI summary
Rosetta: Large scale system for text detection and recognition in images
Fedor Borisyuk, Albert Gordo, Viswanath Sivakumar
TL;DR
Rosetta targets robust OCR for large-scale social-media images, where varied text and massive upload volumes make extraction difficult. It combines Faster-RCNN detection with fully convolutional CTC recognition, evaluates efficiency and accuracy trade-offs, and reports a deployed production system processing Facebook and Instagram uploads.
Problem
OCR for social-media images must handle varied text and image conditions while processing hundreds of millions of images per day in realtime.
Method
Rosetta uses a two-stage OCR pipeline with Faster-RCNN for text detection and a fully-convolutional CTC model for character-based recognition.
Results
Rosetta was deployed in production, and its CTC recognition model improved accuracy by 48.06% over the CHAR baseline while running 26% faster.
Takeaways & Limitations
The paper provides practical approaches and trade-offs for building OCR that balances model accuracy with efficiency at Facebook and Instagram scale.
Takeaways & Limitations
The CHAR baseline cannot correctly recognize words that are too long, while its large classifiers can produce overfitting models.
Abstract
from arXiv · showhide
In this paper we present a deployed, scalable optical character recognition (OCR) system, which we call Rosetta, designed to process images uploaded daily at Facebook scale. Sharing of image content has become one of the primary ways to communicate information among internet users within social networks such as Facebook and Instagram, and the understanding of such media, including its textual information, is of paramount importance to facilitate search and recommendation applications. We present modeling techniques for efficient detection and recognition of text in images and describe Rosetta's system architecture. We perform extensive evaluation of presented technologies, explain useful practical approaches to build an OCR system at scale, and provide insightful intuitions as to why and how certain components work based on the lessons learnt during the development and deployment of the system.
1 INTRODUCTION
Rosetta addresses the challenge of extracting text from rapidly growing volumes of varied social-media images for downstream applications. Its deployed OCR pipeline separates word detection from lexicon-free recognition to support scalable processing.
- Social networks receive hundreds of millions of image uploads daily, creating major challenges for processing visual information at scale.
- Extracting textual information from images facilitates applications such as image search and recommendation.
- OCR must handle varied fonts, languages, lexicons, symbols, non-dictionary words, URLs, email ids, backgrounds, and image quality.
- Rosetta is a deployed Facebook OCR system that detects text regions with Faster-RCNN and recognizes their contents using a fully-convolutional character-based model.
- The system performs detection and recognition as two independent steps, with the recognition model producing lexicon-free word transcriptions.The architecture is illustrated as a two-step process with independently trained models.
2 RELATED WORK
Related work established CNN-based approaches for detecting and recognizing text in natural images, while newer recognition systems removed dictionary restrictions and supported arbitrary-length words. Rosetta builds on fully convolutional sequence prediction trained with CTC loss and synthetic data.
- Earlier CNN recognition systems classified words from a predefined dictionary after candidate word locations were detected.
- Recent character-based recognizers avoid lexicons and can recognize arbitrary-length words unseen during training.
- Fully-convolutional recognition models produce feature sequences, with CTC or related sequence-to-sequence losses used during training.
- Rosetta’s recognition model produces feature sequences, trains with CTC, and recognizes dictionary-free words of arbitrary length.
- Synthetic training data combines fonts, styles, and font sizes to provide varied examples for robust text recognition classifiers.
3 TEXT EXTRACTION MODELS
Rosetta separates OCR into detection and recognition, using efficient convolutional models to locate text and transcribe words of arbitrary length. The CTC recognizer reduces model size and inference time but requires curriculum learning because it is difficult to train directly.
- 3 TEXT EXTRACTION MODELS: Rosetta performs OCR in two independent steps: detecting text-containing regions, then recognizing each detected word with a CNN.This separation supports independent model updates, parallel recognition, and different languages.
- 3.1 Text Detection Model: Faster-RCNN detects candidate text regions using a ShuffleNet convolutional body selected for efficiency.The detector is bootstrapped on synthetic data and fine-tuned on COCO-Text and human-annotated datasets.
- 3.2 Text Recognition Model: The CHAR recognizer uses k independent classification heads but cannot correctly recognize words longer than k and has many parameters that encourage overfitting.Its fixed-size input and maximum word length limit recognition of long words such as URLs.
- 3.2 Text Recognition Model: The fully-convolutional CTC recognizer outputs character probabilities across image positions and aligns them with labels using dynamic programming.CTC permits arbitrary-length words without a dictionary, while greedy decoding removes duplicate characters not separated by blanks.
- 3.2 Text Recognition Model: The CTC model has significantly fewer parameters and runs 26% faster than CHAR, while ResNet-18 gives higher accuracy with only a small computation overhead.The speed difference is attributed mostly to CHAR’s k fully connected classification layers.
- 3.2 Text Recognition Model: CTC training was harder than CHAR training, so the authors used pretraining or curriculum learning; curriculum learning was adopted because it was slightly faster and removed model dependency.Curriculum learning warms up on short words, increases learning rate, and progressively increases word width during subsequent training.
4 SYSTEM ARCHITECTURE
Rosetta uses a pull-based, distributed architecture to process uploaded images in realtime: detection locates words, recognition extracts their characters, and results are stored for downstream applications.
- 4 SYSTEM ARCHITECTURE: Rosetta pulls image-processing jobs from a distributed queue onto inference machines for realtime processing at Facebook scale.The system is deployed in production and designed for images uploaded daily.
- 4 SYSTEM ARCHITECTURE: Each image is downloaded to a local machine, then resized and normalized before text detection runs.
- 4 SYSTEM ARCHITECTURE: The text detection model returns bounding-box coordinates and scores for all words, which are passed to recognition for character extraction from cropped regions.
- 4 SYSTEM ARCHITECTURE: Extracted text and its image locations are stored in TAO, Facebook’s distributed graph database.
- 4 SYSTEM ARCHITECTURE: Downstream applications such as Search access the image’s extracted textual information directly from TAO.
- 4 SYSTEM ARCHITECTURE: Development experiments evaluated trade-offs between system accuracy and inference speed.
5 EXPERIMENTS
Rosetta’s experiments evaluate accuracy and inference time across detection and recognition choices, emphasizing production-scale trade-offs. The results show gains from domain-specific fine-tuning, efficient architectures, and targeted model adjustments.
- Training and test datasets: 57% relative mAP improvement came from fine-tuning synthetic detection training with COCO-Text and then human-rated data.The model was first trained on synthetic data, fine-tuned on COCO-Text, and finally fine-tuned on a human-rated dataset.
- Detection model experiments: 2x faster inference resulted from reducing RPN_POST_NMS_TOP_N from 1000 to 100, with an acceptable mAP decrease.Reducing the setting below 100 produced diminishing returns in inference time.
- Detection model experiments: Images wider than 800px slightly increased mAP@0.5 but significantly increased inference time.The comparison exposed a direct resolution-versus-latency trade-off in detection.
- Detection model experiments: SoftNMS improved mAP@0.5 by an absolute 1.5 points, while F1-score fluctuated by 5.87 points across identical training configurations.The authors replaced F1-score with mAP because Average Precision is less sensitive to threshold variation across runs.
- Recognition model experiments: Character-based recognition avoids the limitations of predefined dictionaries for URLs, emails, special symbols, and multiple languages.This architectural choice supports recognition beyond fixed sets of dictionary words.
- Recognition model experiments: Random jittering of ground-truth bounding boxes produced a 1.54% relative end-to-end improvement, especially with limited application-specific training data.The augmentation models localization noise from the detection stage.
6 DEPLOYMENT
Rosetta is deployed as a cloud OCR service that processes Facebook and Instagram images at scale through detection, cropping, and recognition stages.
- Rosetta provides a cloud API for extracting text from images and processes images uploaded to Facebook and Instagram every day.
- Images are resized to 800px on the larger dimension before detection produces bounding boxes around each word.
- Detected word patches are cropped, resized to 32px height while preserving aspect ratio, and passed to the recognition model.
- Deployment increased traffic weekly from internal users through 1%, 5%, 10%, 25%, 40%, 80%, and finally 100%.Resource utilization was evaluated and machines were added incrementally as traffic increased.
- Detection and recognition were trained with Detectron and PyTorch, then both deployed using Caffe2 after recognition-model conversion through ONNX.
7 CONCLUSION
The paper presents robust and efficient text detection and recognition models alongside scalable architecture for Rosetta. Evaluation demonstrates trade-offs between processing efficiency at scale and model accuracy, while the deployed system processes Facebook and Instagram images daily.
- The paper presents robust, efficient models for text detection and recognition and architectural approaches for a scalable OCR system.
- Evaluation demonstrates trade-offs between efficiency in scale and processing time and model accuracy.
- Rosetta is deployed in production and processes images uploaded to Facebook and Instagram every day.