PaperlayerView exact revision on arXiv

Source-linked AI summary

LaTr: Layout-Aware Transformer for Scene-Text VQA

Ali Furkan Biten, Ron Litman, Yusheng Xie, Srikar Appalaraju, R. Manmatha

arXiv:2112.12494v2cs.CV

TL;DR

STVQA requires models to reason over scene text alongside language, layout, and visual information. LaTr pre-trains with text and spatial cues on documents, then combines OCR, question, and ViT features. It reports gains over state-of-the-art methods on three benchmarks.

  • Problem

    STVQA requires reasoning over scene text and other modalities, while prior text-aware pre-training did not align layout information with semantic representations.

    Source: §1 · pp. 1–2

  • Method

    LaTr uses a single text-and-spatial-cue pre-training objective on documents, then combines OCR tokens, questions, and ViT features for STVQA.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    +7.6% on TextVQA, +10.8% on ST-VQA, and +4.0% on OCR-VQA over state-of-the-art methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Scanned documents, despite the domain gap, provide text-rich and varied layouts that the authors report are beneficial for tying language and layout information.

    Source: §1 · p. 2, §6 · p. 8

    Reported failure cases are mostly associated with OCR errors, compositional spatial reasoning, and visual attributes.

    Source: §4 · p. 6

Abstract

from arXiv · show

We propose a novel multimodal architecture for Scene Text Visual Question Answering (STVQA), named Layout-Aware Transformer (LaTr). The task of STVQA requires models to reason over different modalities. Thus, we first investigate the impact of each modality, and reveal the importance of the language module, especially when enriched with layout information. Accounting for this, we propose a single objective pre-training scheme that requires only text and spatial cues. We show that applying this pre-training scheme on scanned documents has certain advantages over using natural images, despite the domain gap. Scanned documents are easy to procure, text-dense and have a variety of layouts, helping the model learn various spatial cues (e.g. left-of, below etc.) by tying together language and layout information. Compared to existing approaches, our method performs vocabulary-free decoding and, as shown, generalizes well beyond the training vocabulary. We further demonstrate that LaTr improves robustness towards OCR errors, a common reason for failure cases in STVQA. In addition, by leveraging a vision transformer, we eliminate the need for an external object detector. LaTr outperforms state-of-the-art STVQA methods on multiple datasets. In particular, +7.6% on TextVQA, +10.8% on ST-VQA and +4.0% on OCR-VQA (all absolute accuracy numbers).

1. Introduction

LaTr treats STVQA as a task where language and layout information can answer many questions, while some require visual features too. It combines layout-aware pre-training on documents with multimodal fine-tuning and reports gains over prior STVQA methods.

1. Introduction

  • STVQA questions vary in modality needs: some use text alone, others need text and layout, and some require visual features too.The authors report that most questions in current datasets fall into the first two categories and that LaTr can answer over 50% using only text tokens.Source: §1 · p. 1
  • Prior text-aware pre-training (TAP) did not account for aligning layout information with semantic representations.Natural images with text are difficult to scale because they are relatively uncommon and often text-sparse.Source: §1 · pp. 1–2
  • LaTr uses a single layout-aware pre-training objective with text and spatial cues, then incorporates visual features for STVQA.The authors report that scanned documents provide text-rich, varied layouts that help align language and spatial information despite the domain gap.Source: §1 · p. 2
  • +7.6% on TextVQA, +10.8% on ST-VQA, and +4.0% on OCR-VQA over state-of-the-art methods.The introduction presents these as absolute accuracy improvements.Source: §1 · p. 2
  • LaTr uses vocabulary-free decoding, performs well on answers outside the training vocabulary, and can overcome OCR errors in some cases.It also uses a vision transformer to extract visual features, replacing the extensive need for an external object detector.Source: §1 · p. 2

2. Related Work

Prior work includes text-only and multimodal pre-training, as well as a range of datasets and architectures for text-rich visual question answering. LaTr focuses on aligning language and layout through document pre-training and uses generative decoding.

Pre-training and Language Models

  • Yang et al. introduced TAP for STVQA using masked language modeling and image-text matching, while LaTr pre-trains to tie language and layout.The paper notes that TAP did not account for aligning layout information with semantic representations when designing its objectives.Source: §2 · p. 2

Vision-language tasks incorporating scene text

  • ST-VQA, TextVQA, OCR-VQA, DocVQA, and InfoVQA [44] cover text in varied visual contexts.Other cited datasets address multilingual VQA and captioning on natural images [55].Source: §2 · p. 2
  • Scene-text VQA methods include pointer-based LoRRa and M4C, graph-based MM-GNN and SMA [14], and attention-based fusion [72].SA-M4C adds supervision on self-attention weights to M4C.Source: §2 · p. 2

Pre-training and Language Models

  • LaTr enriches language representations with layout through pre-training and uses generative decoding to reduce reliance on a fixed answer vocabulary.The paper also reports greater robustness to OCR errors.Source: §2 · p. 2

3. Method

LaTr combines a T5-based language model, OCR-token spatial embeddings, and visual patch features. Its document pre-training masks text spans while exposing their rough locations, encouraging the model to use layout when predicting the masked text.

3. Method

  • LaTr combines a text-pretrained language model, OCR-token spatial embeddings, document pre-training, and ViT visual features in a multimodal encoder-decoder.Visual features are added during fine-tuning, when the image, OCR information, and question are supplied together.Source: §3 · pp. 2, 4

The Language Model and Layout-Aware Pre-Training

  • T5 provides the encoder-decoder architecture and initialization from its pre-training on cleaned English text.LaTr further pre-trains the language modality with layout information on documents.Source: §3 · p. 3

2-D Spatial Embedding

  • Each OCR token’s representation sums learned embeddings for the token and its bounding-box coordinates, height, and width.Height and width help indicate reading order for scene text that may appear at arbitrary angles or shapes.Source: §3 · p. 3

Layout-Aware Pre-Training

  • Layout-aware denoising masks text spans but gives the model the spans’ rough locations, so it predicts masked text using spatial context.The task can use weak document data without answer annotations.Source: §3 · p. 4
  • A masked span receives a loose bounding box that avoids revealing its exact token count while preserving spatial context.The model predicts the original masked text with cross-entropy loss.Source: §3 · p. 4

Visual Features and LaTr

  • During fine-tuning, image patches are projected into the visual feature sequence, which is concatenated with encoded OCR tokens and question features.The model uses a ViT and passes the combined inputs to the multimodal transformer encoder-decoder.Source: §3 · p. 4

4. Experiments

LaTr advances performance on TextVQA and ST-VQA, while experiments also examine document pre-training and qualitative capabilities. Reported remaining TextVQA failures involve OCR errors, compositional spatial reasoning, and visual attributes.

4. Experiments

  • The experiments use TextVQA, ST-VQA, and OCR-VQA benchmarks, and pre-training data that includes 13 million IDL documents comprising about 64 million pages.IDL includes varied document types and layouts; OCR was extracted with Textract.Source: §4 · p. 4
  • +7.63% on TextVQA test and +10.81% on ST-VQA are the reported state-of-the-art advances for LaTr.Table 1 reports +6.43% on TextVQA validation and +7.63% on test; Table 2 reports +10.81% on ST-VQA.Source: Table 1 · p. 5, Table 2 · p. 5
  • On TextVQA with Rosetta OCR, IDL pre-training raises LaTr-Base accuracy from 44.06% to 48.38% (+4.32%).LaTr without pre-training matches TAP at 44.06%.Source: §4 · p. 5
  • With Amazon-OCR, LaTr-Base rises from 47.84% to 52.29% (+4.45%) versus M4C, and IDL pre-training reaches 58.03%.Against TAP, LaTr reaches 58.03% validation and 58.86% test versus 54.71% and 53.97%; LaTr‡-Large reaches 61.6%, a +7.6% test gain.Source: §4 · p. 5
  • Qualitative examples show LaTr correcting OCR errors and answering questions involving language and world knowledge; reported TextVQA failures include OCR errors and compositional spatial reasoning.The authors attribute OCR correction to image-level context and generative decoding, and link language and world-knowledge abilities to extensive pre-training.Source: §4 · p. 6
  • LaTr handles relative text positions and long answers in qualitative examples, while reported TextVQA failures also include visual attributes.The authors describe document layout pre-training as yielding spatial awareness without complex architectural changes.Source: §4 · p. 6

5. Ablation Studies

The ablations examine how language, layout, vision, pre-training, vocabulary, and OCR errors affect TextVQA performance. They show strong gains from layout-aware document pre-training and little benefit from visual features on some benchmark settings.

Zero-shot Language Models on TextVQA

  • Zero-shot T5 without visual features or fine-tuning reaches 16.05% with Rosetta OCR and 21.93% with Amazon-OCR, and up to 25% with ground-truth OCR.The authors caution that dataset biases may contribute to the performance.Source: §5 · pp. 6–7

Dataset Bias or Task Definition?

  • +7.01% to 57.38% follows document-based layout-aware pre-training, while adding visual features increases performance by only around +0.7%.The authors note that at least 60% of questions do not require reasoning over all modalities, and suggest current benchmarks may not reflect vision’s importance.Source: §5 · p. 7, Table 4 · p. 7
  • With OCR tokens, fixed reading order reaches 50.37%, versus 41.77% without reading order; adding 2-D positions and vision without pre-training yields gains of only +0.85% and +1.09%.Table 4 varies LaTr’s building blocks, including pre-training, input types, and fine-tuning data.Source: §5 · p. 7, Table 4 · p. 7
  • When TextVQA and ST-VQA are used together for fine-tuning, ViT features improve performance by +0.61%, whereas FRCNN features perform worse than using no visual features.With TextVQA alone, the visual-backbone performance difference is described as very marginal.Source: §5 · p. 7, Table 4 · p. 7

The Effect of Large-Scale Pre-Training

  • TextVQA-only pre-training improves LaTr by +1.5%, while Yang et al. report +5%; TAP-dataset pre-training also shows diminished gains.The comparison varies pre-training dataset type and scale.Source: Table 5 · p. 7, §5 · p. 8
  • Pre-training on 1M IDL documents improves LaTr by almost +5%, more than the combination of all TAP-datasets, and is around 23 times faster than TAP pre-training.Increasing IDL data by an order of magnitude adds only +1% or +2%; the authors say 64M documents does not appear to be the saturation point.Source: Table 5 · p. 7, §5 · p. 8

Vocabulary Reliance and Robustness Towards OCR Errors

  • LaTr’s gap between in- and out-of-training-vocabulary answers is only -1.58%, compared with M4C’s -12.7% gap.The comparison concerns the accuracy gap between answers in and outside the vocabulary used by [20, 23, 70].Source: §5 · p. 8, Table 6 · p. 8
  • LaTr’s relative robustness over increases as OCR error probability rises, with errors simulated by randomly replacing one character in selected OCR tokens.The selection threshold is sampled from a Bernoulli distribution.Source: Figure 5 · p. 8, §5 · p. 8

6. Conclusion

The conclusion highlights language and layout, document-based layout-aware pre-training, and ViT features as contributions to STVQA. It also cautions that current benchmarks may understate the importance of vision.

6. Conclusion

  • Language and layout are essential to STVQA: language supports prior knowledge and understanding, while layout supports spatial reasoning.The authors say they demonstrated their importance methodologically.Source: §6 · p. 8
  • Scanned documents offer abundant text and varied complex layouts that can promote scene-text understanding through layout-aware pre-training.The authors describe this as a symbiosis between scanned documents and scene text.Source: §6 · p. 8
  • ViT replaces the extensive need for FRCNN feature extraction and can scale better, but the authors say current benchmarks let STVQA models use visual features only marginally.They raise dataset bias as a possible reason and call for making vision matter more in VQA.Source: §6 · p. 8

A. Implementation Details

The implementation uses text-only language pre-training, layout-aware denoising, and ViT visual features, with distinct training settings for pre-training, fine-tuning, and ablations.

A. Implementation Details

  • All models are implemented in PyTorch and pretrained on eight A100 GPUs; T5 uses a 32,000-wordpiece vocabulary encoded with SentencePiece.The paper identifies SentencePiece and WordPiece tokenization with references [30] and [29] [53].Source: A. Implementation Details · p. 12

Pre-training

  • Base pre-training runs for 2.2M steps with batch size 25 per GPU and a maximum OCR length of 512; large-model pre-training runs for 0.9M steps with batch size 28 and length 384.Both use a 100K-step warm-up to a 1e-4 learning rate followed by linear decay; denoising corrupts 15% of the text sequence.Source: A. Implementation Details · p. 12

Fine-tuning

  • Fine-tuning runs for 100K steps with AdamW, a maximum learning rate of 1e-4, and a 1,000-step warm-up; ViT features use ImageNet-pretrained weights.The ViT is fine-tuned on ImageNet for classification, following the implementation and weights from HuggingFace [60].Source: A. Implementation Details · p. 12

Ablation studies

  • Visual-backbone ablations use Faster R-CNN pretrained on Visual Genome, retain the top 100 objects, and fine-tune only the last layer.The image-text matching ablation pollutes the text 50% of the time; gradients are computed for the encoder only for that task, but for both encoder and decoder for denoising.Source: A. Implementation Details · p. 12
  • The vocabulary-reliance experiment defines its answer vocabulary as the 5,000 most frequent training-set answer words, following.Source: A. Implementation Details · p. 12

B. Datasets

The datasets span scene-text question answering, text-based image captions, book-cover questions, and scene-text image-caption pre-training data.

B. Datasets

  • TextVQA has 28k Open Images pictures and 45k questions, while ST-VQA combines several vision datasets and has 31k questions.TextVQA questions are collected through AMT with ten answers per question; ST-VQA questions are designed so the answer is scene text in the image.Source: B. Datasets · p. 12
  • TextCaps contains 28,408 images from TextVQA, with five captions per image grounded in image text but not necessarily containing OCR tokens.The dataset has 142,040 captions in total.Source: B. Datasets · pp. 12–13
  • OCR-VQA contains 207,572 book-cover images and over one million template-based question-answer pairs; OCR-CC is a 1.367-million-pair scene-text subset of Conceptual Captions.The paper recreates OCR-CC using Amazon-OCR because the original subset is not publicly released.Source: B. Datasets · p. 13

C. The Industrial Document Library dataset

The Industrial Document Library (IDL) provides a large, varied collection of text-dense documents for pre-training. The paper reports that its IDL-pretrained model improves OCR-VQA state-of-the-art accuracy by 4.0%.

C. The Industrial Document Library dataset

  • IDL contains about 13M crawled documents, representing roughly 70M pages, of which 64M are usable.Its collection includes forms, tables, and letters with varied layouts.Source: C. The Industrial Document Library… · p. 13
  • After filtering unreadable, short, or OCR-error-heavy documents, about 64M documents remained for pre-training.The filtering also used an English dictionary to screen OCR text and remove non-English content.Source: C. The Industrial Document Library… · p. 13
  • LaTr improves OCR-VQA state-of-the-art accuracy by +4.0% using an IDL-pretrained base model and Rosetta OCR.Table 7 reports results on OCR-VQA and describes this as an improvement over the state of the art.Source: Table 7 · p. 13
  • IDL retains over 91% of its original data, compared with 45.16% for OCR-CC, and has more words on average.Most IDL documents have 20 to 400 words.Source: C. The Industrial Document Library… · p. 13

D. OCR-VQA Results

In the constrained setting, LaTr-Base outperforms the previous state of the art on OCR-VQA. The evaluation retains the Rosetta OCR system rather than changing OCR.

D. OCR-VQA Results

  • LaTr-Base raises OCR-VQA accuracy from 63.5% to 67.5%, a +4.0% improvement over the previous state of the art.The evaluation uses the constrained setting and keeps the Rosetta OCR system unchanged.Source: D. OCR-VQA Results · p. 13
  • The OCR-VQA comparison uses the constrained setting, with Rosetta OCR unchanged.Source: D. OCR-VQA Results · p. 13
  • The reported baseline is the previous state-of-the-art method, which LaTr-Base surpasses by a large margin.Source: D. OCR-VQA Results · p. 13

E. Qualitative Examples

Qualitative examples show LaTr succeeding on cases where M4C fails, including natural-language and layout reasoning, while also exposing answer biases and failures involving OCR and multimodal reasoning.

E. Qualitative Examples

  • LaTr shows better natural-language understanding and layout reasoning than M4C in the displayed examples.The authors attribute this to extensive pre-training and document-based layout-aware pre-training with 2-D spatial position embeddings.Source: E. Qualitative Examples · p. 13
  • The examples expose dataset biases: over 85% of TextVQA yes/no questions have “Yes” answers, and sparse text can make guesses likely.For one car-number question, only two numbers appear in the image, giving a guess at least a 50% chance of being correct.Source: E. Qualitative Examples · p. 14
  • Both LaTr and M4C often answer analog-watch time questions with 10:10, reflecting a learned prior rather than reading the pictured time.The authors connect this prior to watch sellers' marketing practice of displaying watches set to 10:10.Source: E. Qualitative Examples · p. 14
  • LaTr failure cases mostly involve OCR errors, compositional spatial reasoning, and visual attributes.One example additionally requires reasoning about a soda imagining its reflection in a mirror.Source: E. Qualitative Examples · p. 14

F. Dataset Bias or Task Definition?

The paper categorizes TextVQA questions by the information needed to answer them, from word content alone to combined text, layout, and image reasoning. Its analysis distinguishes the contributions of reading order and spatial layout, while noting that questions requiring all modalities are uncommon in current benchmarks.

F. Dataset Bias or Task Definition?

  • Over 40% of TextVQA test questions can be answered with an order-less bag of words, including about 11% answerable from the questions alone.Source: F. Dataset Bias or Task Definition? · p. 14
  • Adding reading order makes an additional 10% of questions answerable using an ordered bag of words.The authors distinguish reading order from OCR output, though current work often treats OCR as a black box.Source: F. Dataset Bias or Task Definition? · p. 14
  • Questions requiring word tokens and their 2-D layout make up at least the category corresponding to a layout-aware pre-training boost of over 7%.The authors treat 7% as a lower bound on this category’s size based on qualitative analysis.Source: F. Dataset Bias or Task Definition? · p. 14
  • Questions requiring text, layout, and the image itself are uncommon in current benchmarks, and the authors propose screening them with words and layout shown without the image.If annotators can still answer a question from that text-and-layout view, the authors propose dropping it from the all-modality category.Source: F. Dataset Bias or Task Definition? · p. 14, Figure 8 · p. 16

Related papers

Loading 2112.12494v2…