PaperlayerView exact revision on arXiv

Source-linked AI summary

TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text

Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojciech Galuba, Tal Hassner

arXiv:2105.05486v1cs.CV

TL;DR

Scene-text reasoning is limited by weak OCR supervision and the lack of real-image annotations that support isolated reasoning evaluation. The paper introduces TextOCR and PixelM4C, then reports stronger OCR and downstream TextVQA performance, while noting a scope limitation around direct training on TextOCR.

  • Problem

    Real-image scene-text annotations and end-to-end OCR feedback are limited, hindering OCR evaluation and scene-text reasoning in TextVQA and TextCaps.

  • Method

    The paper builds TextOCR and connects its trained Mask TextSpotter v3 OCR module with M4C in the end-to-end PixelM4C architecture.

  • Results

    Training on TextOCR improves text-recognition models on most benchmarks, while PixelM4C surpasses existing state-of-the-art methods on TextVQA.

  • Takeaways & Limitations

    TextOCR and PixelM4C bridge OCR and downstream scene-text applications while enabling reasoning evaluation with OCR effects separated more fairly.

  • Takeaways & Limitations

    The authors do not claim direct TextOCR training as state-of-the-art because doing so would be non-ideal for isolating reasoning capabilities.

Abstract

from arXiv · show

A crucial component for the scene text based reasoning required for TextVQA and TextCaps datasets involve detecting and recognizing text present in the images using an optical character recognition (OCR) system. The current systems are crippled by the unavailability of ground truth text annotations for these datasets as well as lack of scene text detection and recognition datasets on real images disallowing the progress in the field of OCR and evaluation of scene text based reasoning in isolation from OCR systems. In this work, we propose TextOCR, an arbitrary-shaped scene text detection and recognition with 900k annotated words collected on real images from TextVQA dataset. We show that current state-of-the-art text-recognition (OCR) models fail to perform well on TextOCR and that training on TextOCR helps achieve state-of-the-art performance on multiple other OCR datasets as well. We use a TextOCR trained OCR model to create PixelM4C model which can do scene text based reasoning on an image in an end-to-end fashion, allowing us to revisit several design choices to achieve new state-of-the-art performance on TextVQA dataset.

1. Introduction

Scene-text reasoning depends on OCR, but real-image annotations and end-to-end OCR feedback remain limited. TextOCR and PixelM4C address these gaps with dense annotations and trainable image-to-reasoning integration.

  • Real-world OCR still performs poorly because large annotated scene-text datasets, ground-truth text, and downstream end-to-end feedback are unavailable.
  • TextOCR provides high-quality annotations on TextVQA images and supports task-specific end-to-end fine-tuning of OCR pipelines.
  • 903k words across more than 28k images average 32 words per image, providing dense arbitrary-shaped scene-text supervision.
  • PixelM4C connects TextOCR-trained Mask TextSpotter v3 with M4C to perform scene-text reasoning directly from images.
  • PixelM4C achieves new state-of-the-art TextVQA performance under comparable settings and enables TextOCR impact analysis on TextCaps.

2. Related work

Prior scene-text datasets span recognition, detection, multilingual, synthetic, and downstream reasoning settings, but differ substantially in shape, density, language, and realism. TextVQA and TextCaps specifically require reading and reasoning over scene text in images.

  • Recognition benchmarks range from horizontally aligned word datasets to irregular datasets containing oriented, perspective, curved, or otherwise transformed text.
  • Detection datasets provide full-image word locations using horizontal boxes, quadrilaterals, curved polygons, or line-level labels.
  • Multilingual datasets cover multiple scripts or Chinese and English, but distribute less data per language and often withhold test labels.
  • Synthetic datasets such as Synth90k and SynthText supply large-scale word data and are commonly used for pretraining or joint training with real datasets.
  • TextVQA and TextCaps require reading scene text and reasoning about its relationships with objects to answer questions or produce coherent captions.
  • Existing TextVQA and TextCaps systems typically use pre-extracted OCR tokens alongside image, question, and textual embeddings.

3. TextOCR dataset

TextOCR annotates arbitrary-shaped scene text on dense real images with polygon locations and transcription rules. Its scale, density, diversity, and coverage of rotated and curved words distinguish it from existing datasets.

  • 3.1. Annotation Details: All words receive polygon detection annotations, while only Latin words are transcribed and non-Latin or illegible words receive a “.” label.
  • 3.1. Annotation Details: Annotators use rectangles for near-horizontal words and multipoint polygons for curved words while preserving reading direction and minimizing background.
  • 3.2. Statistics and Visualizations: TextOCR shows greater word density and more uniform word-location distributions than COCO-Text and ICDAR15.
  • 3.2. Statistics and Visualizations: TextOCR contains more annotated words than existing public datasets, with at least 3x more than ICDAR19-LSVT and 10x more than most others.
  • 3.2. Statistics and Visualizations: 32.1 words per image gives TextOCR roughly 3x the density of any existing comparable dataset.
  • 3.2. Statistics and Visualizations: 523k words remain after excluding non-alphanumeric or short words, including 150k rotated and 19.3k curved words.

4. OCR Experiments

The experiments evaluate TextOCR as a challenging recognition benchmark and as training data for improving OCR performance across datasets. Results show gains in recognition and end-to-end recognition, while TextOCR remains difficult even for fine-tuned state-of-the-art methods.

  • TextOCR experiments cover cross-dataset analysis, state-of-the-art performance on public benchmarks, and evaluation of existing methods on TextOCR.The evaluation includes text recognition and end-to-end recognition settings.
  • 4.1. Cross-dataset empirical analysis: 22.45% better TextOCR-trained recognition than COCOText-trained recognition on TextOCR, and 10.56% better on COCOText’s test set.The same TPS-ResNet-BiLSTM-Attn model was trained separately on each dataset for 100k iterations.
  • 4.1. Cross-dataset empirical analysis: 80.07% word accuracy is achieved by training on TextOCR alone, near state-of-the-art performance.Using matched data settings, TextOCR also performs 2.5% better than ICDAR15 and 0.3% better than COCO-Text on average in recognition benchmarks.
  • 4.2. State-of-the-art on public benchmarks: Fine-tuning state-of-the-art recognition models on TextOCR significantly increases accuracy on almost all evaluated datasets, especially irregular datasets.The reported gains are attributed to TextOCR’s rich diversity, with ICDAR2015, SVT Perspective, and CUTE80 benefiting most.
  • 4.1. Cross-dataset empirical analysis: TextOCR fine-tuning improves end-to-end recognition by 3.3% without a lexicon and 3.2% with a weak lexicon on the Total Text test set.The improvement is reported after fine-tuning official Mask TextSpotter V3 weights on TextOCR and Total Text.
  • 4.3. The challenges of TextOCR: TextOCR remains challenging: methods fine-tuned on its 21k-image training set score much lower than on other popular OCR datasets.The results indicate substantial room for improvement on the TextOCR test set; higher input short sides may help because of its high resolution.

5. TextVQA and TextCaps Experiments

The experiments evaluate TextOCR-trained OCR within downstream TextVQA and TextCaps systems, including upper bounds, heuristics, ablations, and end-to-end PixelM4C training. TextOCR improves TextVQA substantially, while TextCaps shows limited gains, highlighting remaining OCR and reasoning challenges.

  • Experimental setup: TextOCR annotations and trained OCR models are evaluated in TextVQA and TextCaps using PixelM4C with OCR extracted from MTS v3.The OCR module is connected to M4C so tokens and features can be extracted on-the-fly during end-to-end training.
  • Upper Bounds and Heuristics: OCR UB is 11% and 22% higher for TextOCR than for MTS v3 and Rosetta, respectively, on TextVQA validation.The comparison uses OCR tokens without a fixed vocabulary and supports the need for stronger OCR mechanisms.
  • Improving the state-of-the-art: 42.12% TextVQA accuracy is achieved after adding STVQA as additional data, establishing a new state-of-the-art over M4C.TextOCR-trained models provide better OCR tokens than models trained on COCO-Text and Total-Text.
  • Improving the state-of-the-art: TextCaps shows no significant metric improvement from TextOCR-trained OCR tokens except in B4, indicating greater complexity for that task.The result contrasts with the stronger TextVQA improvements.
  • Improving the state-of-the-art: 6% TextVQA accuracy and 3% TextCaps metric gains result from using TextOCR directly as the OCR source.These gains indicate remaining room to improve the OCR module beyond reasoning improvements.

6. Conclusion

The paper introduces TextOCR, a large arbitrary-shaped scene-text dataset collected from TextVQA images, and PixelM4C, an end-to-end model connecting text recognition with downstream reasoning. TextOCR-trained recognition improves OCR benchmarks and PixelM4C surpasses prior TextVQA performance, although direct TextOCR training is not claimed as state of the art for community evaluation.

  • Conclusion: TextOCR provides large-scale arbitrary-shaped scene-text annotations, while PixelM4C performs scene-text reasoning directly from images through an incorporated recognition module.PixelM4C connects text recognition and downstream reasoning in an end-to-end model.
  • Conclusion: Training on TextOCR produces better text-recognition models that outperform state of the art on most text-recognition benchmarks.The conclusion attributes this outcome to TextOCR-trained recognition models.
  • Conclusion: PixelM4C surpasses existing state-of-the-art methods on TextVQA using a TextOCR-trained text-recognition module.The model can use different recognition features and potentially provide feedback within the integrated system.
  • Conclusion: Direct training on TextOCR is not claimed as state of the art because it would be non-ideal for community evaluation outside isolated reasoning analysis.The authors frame direct TextOCR training as useful for understanding reasoning capabilities separately from OCR systems.

(Supplementary Material)

TextOCR uses an annotation interface designed for arbitrary-shaped words. Annotators draw polygons around words, with interface cues supporting point ordering and corner placement.

  • Annotation UI: The TextOCR annotation UI allows annotators to label arbitrary-shaped text as polygons.The interface supports drawing any number of points around a word.
  • Annotation UI: Annotators are instructed to use quadrilaterals whenever appropriate, while retaining polygonal labeling for other word shapes.The UI displays the edge between the first and second points differently to validate point placement and ordering.

B. Dataset Instance Location Heatmap

The supplementary heatmap compares instance locations across TextOCR, COCO-Text, ICDAR15, and TotalText. It reports that TextOCR annotations are more uniformly distributed than those in existing datasets.

  • Dataset Instance Location Heatmap: Figure B.1 compares the instance locations of TextOCR, COCO-Text, ICDAR15, and TotalText.The heatmap expands the main-paper comparison of dataset instance locations.
  • Dataset Instance Location Heatmap: TextOCR annotations are reported as more uniformly distributed than those in the existing datasets.The comparison concerns the spatial distribution of annotated instances.

C. OCR Model Implementation Details

The study evaluates two OCR model types: text recognition and end-to-end recognition. Text recognition uses four model variants, while end-to-end recognition uses Mask TextSpotter V3 with specified optimization and augmentation settings.

  • The experiments cover text recognition and end-to-end recognition OCR models.
  • Text recognition compares CRNN, Rosetta, STAR-Net, and TPS-ResNet-BiLSTM-Attn models.The implementation follows Baek et al.'s model configurations.
  • Text-recognition training uses AdaDelta with decay rate 0.95, batch size 192, and gradient clipping at magnitude 5.
  • End-to-end recognition uses the official Mask TextSpotter V3 implementation, trained with SGD, momentum 0.9, weight decay 0.0001, and a 0.001 initial learning rate.The learning rate is divided by 10 every 100K iterations over 300K iterations, with batch size 8, rotation augmentation, and multi-scale training.

D. Experiments on same number of instances

Controlled-size experiments compare TextOCR against ICDAR15 and COCO-Text using matched numbers of word instances. TextOCR generally delivers stronger recognition performance and broader generalization.

  • 4,055 and 38,839 TextOCR word images are sampled to match the instance counts of ICDAR15 and COCO-Text, respectively.The experiments fine-tune models using these matched-size subsets.
  • TextOCR-4055 outperforms ICDAR15 on all standard recognition benchmarks except ICDAR15 itself.The comparison supports greater diversity and generalization to other test sets than the incidental-scene-text-focused ICDAR15 dataset.
  • TextOCR-38839 outperforms COCO-Text on 5 out of 7 benchmarks.The reported comparison indicates superior quality and generalization for the matched-size TextOCR subset.

F. PixelM4C: Hyper-parameters and ST-VQA

PixelM4C training uses selected hyper-parameters, including batch size 16, while a TextOCR-trained Latin OCR model is also evaluated on ST-VQA. The ST-VQA evaluation reports improved accuracy and ANLS over prior work.

  • PixelM4C experiments use batch size 16 because it performs reasonably better or equally compared with batch size 128.The smaller batch size also eases training with fewer GPUs.
  • A TextOCR-trained Latin OCR model is trained on the ST-VQA training set and evaluated on its validation set.
  • 38.49% accuracy and 47.89% ANLS are reported as better than the results in prior work.
Loading 2105.05486v1…