PaperlayerView exact revision on arXiv

Source-linked AI summary

TAP: Text-Aware Pre-training for Text-VQA and Text-Caption

Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florencio, Lijuan Wang, Cha Zhang, Lei Zhang, Jiebo Luo

arXiv:2012.04638v1cs.CV

TL;DR

Text-VQA and Text-Caption require aligned understanding of scene text, but conventional vision-language pre-training omits that modality. TAP incorporates OCR-derived scene text through language, matching, and spatial pre-training tasks, and achieves large gains across TextVQA, ST-VQA, and TextCaps, further helped by OCR-CC.

  • Problem

    Conventional vision-language pre-training does not capture scene text or its relationship with visual and textual modalities for Text-VQA and Text-Caption.

  • Method

    TAP incorporates OCR-derived scene text and uses masked language modeling, image-text matching, and relative spatial position prediction to align text words, visual objects, and scene text.

  • Results

    +8.3% absolute accuracy on TextVQA, +8.6% on ST-VQA, and +10.2 CIDEr on TextCaps versus previous methods.

  • Takeaways & Limitations

    TAP improves scene-text vision-language performance even with identical downstream architecture and training data, with further gains from OCR-CC pre-training.

Abstract

from arXiv · show

In this paper, we propose Text-Aware Pre-training (TAP) for Text-VQA and Text-Caption tasks. These two tasks aim at reading and understanding scene text in images for question answering and image caption generation, respectively. In contrast to the conventional vision-language pre-training that fails to capture scene text and its relationship with the visual and text modalities, TAP explicitly incorporates scene text (generated from OCR engines) in pre-training. With three pre-training tasks, including masked language modeling (MLM), image-text (contrastive) matching (ITM), and relative (spatial) position prediction (RPP), TAP effectively helps the model learn a better aligned representation among the three modalities: text word, visual object, and scene text. Due to this aligned representation learning, even pre-trained on the same downstream task dataset, TAP already boosts the absolute accuracy on the TextVQA dataset by +5.4%, compared with a non-TAP baseline. To further improve the performance, we build a large-scale dataset based on the Conceptual Caption dataset, named OCR-CC, which contains 1.4 million scene text-related image-text pairs. Pre-trained on this OCR-CC dataset, our approach outperforms the state of the art by large margins on multiple tasks, i.e., +8.3% accuracy on TextVQA, +8.6% accuracy on ST-VQA, and +10.2 CIDEr score on TextCaps.

1. Introduction

Text-VQA and Text-Caption require models to read scene text in images, but conventional vision-language pre-training does not model scene text or its relationships with other modalities. TAP addresses this gap by incorporating scene text into pre-training, while OCR-CC supplies additional scene-text-related image-text pairs.

  • Text-VQA answers questions using scene text in image-question context, while Text-Caption generates captions describing visual and scene-text information.
  • Conventional vision-language pre-training omits scene text, limiting its ability to capture the scene-text modality and its relationships with visual and textual modalities.
  • TAP explicitly incorporates scene text to learn a joint representation of text words, visual objects, and scene text.
  • OCR-CC contains around 1.4 million scene text-related image-text pairs and enables further improvement when used for pre-training.
  • TAP improves TextVQA accuracy from 44.50% to 49.91% with identical network architecture and training data versus a non-TAP baseline.
  • TAP with OCR-CC outperforms previous methods by +8.3% absolute accuracy on TextVQA, +8.6% on ST-VQA, and +10.2 CIDEr on TextCaps.

2. Related Work

Prior work improves Text-VQA and Text-Caption mainly through stronger architectures, while conventional pre-training overlooks scene text. TAP extends pre-training to align text words, visual objects, and scene text.

  • Text-VQA reads and understands scene text for question answering, whereas Text-Caption generates captions describing scene text in images.
  • Prior models improve performance through OCR attention, transformer-based multimodal fusion, spatial encoding, and multi-step decoding architectures.
  • These studies directly optimize a single downstream answer or caption objective, which may be ineffective for aligned representation learning.
  • Conventional vision-language pre-training does not capture scene text, whereas TAP explicitly incorporates it to align three modalities.

3. Text-Aware Pre-training (TAP)

TAP pre-trains on text words, visual objects, and OCR-derived scene text using language, matching, and spatial tasks before downstream fine-tuning. Its design emphasizes scene-text inputs and spatial relationships.

  • TAP pre-trains a model and then fine-tunes it for a specific downstream task.
  • The framework processes text words, visual objects, scene text regions, and a special begin token during pre-training.
  • Scene-text language pre-training tasks: TAP uses masked language modeling and image-text matching to fuse scene text with text words.
  • Scene-text language pre-training tasks: The extra scene-text word input is critical because question text alone contains too few scene-text-related words for effective language alignment.
  • Scene-text visual pre-training tasks: Relative spatial position prediction randomly samples object and scene-text regions and predicts their spatial relationship.
  • Pre-training data: OCR-CC is a large-scale scene-text-related image-caption dataset built from Conceptual Caption for additional pre-training.

4. Experiments

TAP is evaluated on TextVQA, ST-VQA, and TextCaps using M4C-based models, controlled comparisons, ablations, and analyses of alignment and failures. It improves performance over non-TAP baselines and benefits further from OCR-CC and additional pre-training data.

  • Text-VQA/Text-Caption results: TAP improves TextVQA accuracy from 39.55% to 44.06% in the constrained setting and provides a 5.4% absolute gain over the corresponding non-TAP baseline in the unconstrained setting.These comparisons use identical network architecture and training data for TAP and the non-TAP baseline.
  • Text-VQA/Text-Caption results: TAP with OCR-CC improves TextVQA accuracy from 49.91% to 54.71%, showing additional benefit from large-scale pre-training data.The OCR-CC model uses OCR-CC during pre-training.
  • Text-VQA/Text-Caption results: On ST-VQA, large-scale pre-training further improves accuracy by +5.5% over the corresponding TAP result.TAP is compared with a model using TextVQA, ST-VQA, TextCaps, and OCR-CC for pre-training.
  • Text-VQA/Text-Caption results: On TextCaps, TAP improves the M4C† CIDEr score from 99.89 to 105.05, while large-scale pre-training raises it further to 109.16.The comparisons use the same training data for the baseline and TAP, followed by the large-scale pre-training result.
  • Ablation studies: Ablations show that scene-text language pre-training raises accuracy to 49.01%, scene-text visual pre-training to 46.42%, and all TAP tasks to 49.91%.The baseline accuracy is 44.50%; the evaluated tasks include MLM, ITM, and RPP.
  • Ablation studies: RPP with 12 spatial relationship classes improves accuracy from 49.91% to 50.17%, while extra data increases the result to 52.90% and larger models with extra data to 54.71%.Without extra data, the larger model drops from 49.91% to 48.78%, whereas extra data reverses this trend.
  • Alignment analysis: TAP increases scene-text language coreference scores sevenfold and scene-text visual coreference scores twofold relative to M4C†.The paper uses these attention-based coreference changes to analyze latent alignment.
  • Qualitative analysis: Qualitative analyses show higher TAP coreference scores and corrected failures involving paraphrased scene text, intermediate references, and selecting regions for multiple questions.Examples include grounding “must” and “survive” to corresponding scene-text regions and identifying editors’ names.

5. Conclusion

TAP explicitly incorporates scene text during pre-training to learn better aligned multimodal representations for Text-VQA and Text-Caption. OCR-CC further improves performance, with large margins over state-of-the-art methods.

  • TAP explicitly incorporates scene text in pre-training and learns a better aligned multimodality representation for Text-VQA and Text-Caption.
  • +5.4% absolute accuracy: TAP boosts non-TAP baselines on the TextVQA challenge with the identical framework and training data.
  • OCR-CC further improves TAP performance and supports large-margin state-of-the-art results across multiple tasks.

A. The OCR-CC Dataset

OCR-CC is constructed from Conceptual Caption images by filtering for meaningful scene text and augmenting captions with object labels and detected OCR words. The selected images predominantly contain 3–10 scene text regions.

  • Figure 5 compares detected scene-text distributions in CC and OCR-CC and shows representative discarded and selected images.
  • 1,367,170 images: OCR-CC selects Conceptual Caption images containing scene text after discarding text-free and watermark-only images.Microsoft Azure OCR processes around 3.1 million CC images; around half lack scene text and around 5% contain watermark text only.
  • Each OCR-CC caption concatenates the original caption, detected object labels, and detected scene text words.
  • 3–10 detected scene text regions: this range covers the majority of images in OCR-CC, while a small portion contains many more regions.

B. TextCaps Results

TAP substantially improves TextCaps performance over the non-TAP baseline and previous state-of-the-art methods. It also exceeds an oracle baseline using ground-truth scene-text detection, although a gap to human performance remains.

  • TAP significantly improves the non-TAP M4C† baseline across all TextCaps metrics with identical network architecture and training data.
  • TAP outperforms previous TextCaps state-of-the-art methods by large margins.
  • TAP outperforms M4C (GT OCR), which uses ground-truth scene-text detection during training and inference.
  • A performance gap remains between TAP and human performance, motivating future captioning-specific decoding pre-training.

C. Hyper-parameters

The experiments use an extended text-input length of 220 and increase the maximum scene-text length to accommodate Microsoft-OCR’s denser detections.

  • 100 maximum scene-text tokens: TAP increases the scene-text length from 50 to 100 for Microsoft-OCR experiments.
  • Microsoft-OCR detects more scene-text regions than Rosetta, motivating the increased maximum length.On TextVQA, Microsoft-OCR has mean and median counts of 23.1 and 12, versus 12.8 and 8 with Rosetta.

D. Pre-train + Fine-tune vs. Joint-train

TAP can be trained either by pre-training followed by fine-tuning or by jointly optimizing the pre-training tasks with the main answer or caption loss. Both pipelines substantially outperform the non-TAP baseline, while pre-train + fine-tune more readily incorporates weakly annotated data.

  • Pipeline comparison: The joint-train pipeline uses MLM, ITM, and RPP as auxiliary losses alongside the main answer or caption loss.This alternative pipeline treats TAP as a multi-task learning framework.
  • Pipeline comparison: 49.91% for “pre-train + fine-tune” and 49.46% for “joint train” on TextVQA show similar performance.Both methods significantly outperform the non-TAP baseline at 44.50%.
  • Practical trade-offs: Pre-train + fine-tune more easily uses extra weak data without answer or caption annotations.This is identified as an advantage over joint train.
  • Practical trade-offs: The two pipelines offer complementary choices: joint train preserves the main-task training pipeline, while pre-train + fine-tune benefits from large-scale weak pre-training data.The authors suggest these approaches may also improve other multimodal tasks when pre-training objectives are incorporated.

E. Qualitative Results

Qualitative examples show that scene-text detection can correct failures, while TAP still fails on complex reasoning and incorrect scene-text detection. The examples highlight the dependence of performance on both reasoning and OCR quality.

  • TAP failure cases: TAP occasionally fails on samples requiring complex reasoning or containing incorrect scene-text detection.These failure categories are illustrated in Figure 7.
  • TAP failure cases: In one example, TAP selects “cutfittep” on a black bag instead of the correct “aldo” on the referred white bag.The error combines incorrect scene-text selection with a failure to identify the relevant referred object.
  • OCR-correctable failures: Scene-text detection corrects failures involving key regions such as “moon bar,” “bud light,” “clemson,” and “marvel.”Figure 6 visualizes detected scene text from Rosetta-OCR and Microsoft-OCR.
  • TAP failure cases: Figure 7 highlights the key scene-text regions associated with each question in representative TAP failure cases.
Loading 2012.04638v1…