Source-linked AI summary
Towards VQA Models That Can Read
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, Marcus Rohrbach
TL;DR
VQA models lack the ability to read and reason about text in images, despite the importance of such questions for visually impaired users. The paper introduces the TextVQA dataset and LoRRA, an OCR-aware model that reasons over image text and predicts from either a fixed vocabulary or OCR strings. LoRRA outperforms existing state-of-the-art VQA models on TextVQA and VQA 2.0, while its validation accuracy remains below human performance.
Problem
VQA models lack the ability to read and reason about text in images, a capability frequently needed for questions asked by visually impaired users.
Method
The paper introduces TextVQA and LoRRA, which reads image text, reasons over visual and textual information, and answers from a fixed vocabulary or OCR strings.
Results
LoRRA outperforms existing state-of-the-art VQA models on TextVQA and VQA 2.0.
Takeaways & Limitations
TextVQA provides a benchmark for VQA questions requiring reading and reasoning about text in images.
Takeaways & Limitations
LoRRA reaches 26.56% validation accuracy on TextVQA, below human performance of 85.01% and the LA+OCR upper bound of 67.56%.
Abstract
from arXiv · showhide
Studies have shown that a dominant class of questions asked by visually impaired users on images of their surroundings involves reading text in the image. But today's VQA models can not read! Our paper takes a first step towards addressing this problem. First, we introduce a new "TextVQA" dataset to facilitate progress on this important problem. Existing datasets either have a small proportion of questions about text (e.g., the VQA dataset) or are too small (e.g., the VizWiz dataset). TextVQA contains 45,336 questions on 28,408 images that require reasoning about text to answer. Second, we introduce a novel model architecture that reads text in the image, reasons about it in the context of the image and the question, and predicts an answer which might be a deduction based on the text and the image or composed of the strings found in the image. Consequently, we call our approach Look, Read, Reason & Answer (LoRRA). We show that LoRRA outperforms existing state-of-the-art VQA models on our TextVQA dataset. We find that the gap between human performance and machine performance is significantly larger on TextVQA than on VQA 2.0, suggesting that TextVQA is well-suited to benchmark progress along directions complementary to VQA 2.0.
1. Introduction
The paper addresses VQA models’ inability to read and reason about text in images, a capability frequently needed by visually impaired users. It introduces TextVQA and LoRRA, which combines OCR-based reading with visual and textual reasoning.
- TextVQA targets questions requiring models to read text embedded in images and reason about it alongside other visual content.
- Up to 21% of questions asked by visually impaired users involve reading and reasoning about text in images.
- LoRRA incorporates OCR regions and recognized text, then learns whether to answer from a fixed vocabulary or copy an OCR string.
- Standard VQA datasets contain few such questions, while VizWiz has limited effective size because 58% of its questions are unanswerable.
- TextVQA contains 45,336 questions on 28,408 images, with 10 human-provided ground-truth answers for each question-image pair.
- LoRRA outperforms existing state-of-the-art VQA models on TextVQA and VQA 2.0.
2. Related work
Related work covers datasets for VQA and scene-text understanding, visual representations for VQA, and copy mechanisms for generating answers from detected text. The paper extends attention-based VQA representations with OCR-derived information and a copy pathway.
- Text based VQA: Existing scene-text datasets evaluate text detection or parsing, but do not involve answering questions about images or reasoning about their text.
- Text based VQA: DVQA uses OCR to read chart-specific content while assessing automatic bar-chart understanding through question answering.
- Visual Representations for VQA Models: VQA models typically attend over object-region or spatial visual features conditioned on a question representation.
- Visual Representations for VQA Models: LoRRA extends VQA representations by reasoning over OCR tokens in addition to visual features.
- Copy Mechanism: Its copy mechanism lets the model point to an OCR token and copy it as an answer, enabling generation of out-of-vocabulary words.
3. LoRRA: Look, Read, Reason & Answer
LoRRA extends VQA with OCR-driven reading and a dynamic answer mechanism, allowing reasoning over image content and recognized text. Its dataset and analyses show why text-aware VQA requires diverse questions, answers, and OCR tokens.
- Model architecture: LoRRA combines VQA reasoning, OCR-based reading, and an answering module that predicts from a fixed space or points to recognized text.The model embeds OCR tokens, attends over them with the question, and combines OCR-question features with visual-question features.
- Answer module: A dynamic answer space adds M OCR tokens to N fixed answers, enabling the model to copy recognized out-of-vocabulary words.If the selected index exceeds N, the corresponding OCR token is copied as the answer.
- Answer module: 8.9% of TextVQA questions require combining multiple OCR tokens, which LoRRA does not support and the paper leaves for future work.When copying, the model can produce only one OCR token as the predicted answer.
- TextVQA dataset: 45,336 TextVQA questions cover diverse image-text reasoning, with 37,912 unique questions and 26,263 unique majority answers.Questions average 7.18 words, while 49.2% of majority answers are unique, complicating use of a fixed answer space.
- OCR analysis: Rosetta OCR produces a useful supply of tokens: images most commonly contain 4–5 tokens, while roughly 1.5k images contain zero detected tokens.The OCR-token word cloud overlaps with common answers such as brand names and cities.
5. Experiments
The experiments evaluate TextVQA baselines, ablations, and LoRRA variants, showing that OCR-aware modeling improves performance while a substantial gap to human accuracy remains.
- TextVQA is split into 34,602 training, 5,000 validation, and 5,734 test question-image pairs without image overlap between training and validation.
- 8.09% and 6.29% validation accuracies are obtained by question-only and image-only baselines, respectively.
- 18.41% validation accuracy is reached by BAN after adding LoRRA, indicating that the architecture improves a state-of-the-art VQA model on TextVQA.
- 26.56% validation accuracy is achieved by LoRRA, below human performance of 85.01% and the LA+OCR upper bound of 67.56%.
- 0.50 percentage points improve Pythia’s VQA 2.0 accuracy, from 68.71 to 69.21, when LoRRA is added.
6. Conclusion
The paper targets reading and reasoning about text in everyday images, introduces TextVQA and LoRRA, and reports strong gains over existing VQA models while identifying OCR limitations.
- TextVQA contains questions that require reading and reasoning about text in images, addressing the lack of datasets supporting systematic study of this skill.
- LoRRA reads image text, reasons about it using the question, and predicts from either a fixed vocabulary or text found in the image.
- LoRRA significantly outperforms current state-of-the-art VQA models on TextVQA and is agnostic to the specific OCR and VQA modules.
- The OCR model still fails to detect rotated, unstructured, or partially occluded text.
A. OCR and Answer Space Analysis
The analysis examines how LoRRA selects OCR tokens or answer-space entries, showing substantial reliance on image text and non-random copying behavior.
- 44.9% of LoRRA’s predicted answers come from OCR tokens, while 55.1% come from the short answer vocabulary.
- 30.6% of questions have answers in OCR tokens; LoRRA selects OCR predictions for 68% of these and answers 57.5% correctly.
- 48% of questions have answers in the short answer vocabulary; LoRRA selects the answer space for 66.75% and answers 38% correctly.
- 81% of validation questions contain images with at least two OCR tokens, and LoRRA copies an OCR token correctly for 24.3% of these.
- LoRRA copies from OCR tokens 49.7% of the time on images with at least two tokens, suggesting that its copying is not random.
B. TextVQA Examples and LoRRA Predictions
LoRRA demonstrates varied text-grounded reasoning abilities, including using OCR-informed answer selection and handling times, dates, places, colors, spatial relations, sizes, and shapes. Its errors expose biases, OCR dependence, and limitations in copying multi-token answers.
- LoRRA successfully answers questions about times, dates, brands, cities, and places, sometimes correcting OCR misspellings through the predetermined answer space.Examples include short-hand time, birthday date, London, and Samsung.
- LoRRA handles colors, spatial relations, basic sizes, and shapes in TextVQA examples.The examples include identifying positions, signs, numbers, shorter hands, and lowest measurements.
- LoRRA often uses OCR tokens to inform an answer from the answer space, outperforming the OCR-free Pythia prediction in one example.For Fig. 7k, Pythia predicts 3 while LoRRA predicts the correct answer, 4.
- LoRRA sometimes predicts generic answers, including “new york” for city questions, despite relevant visual or OCR information.It also predicts generic answers instead of detected tokens in examples such as “embossed” and “a-2.”
- LoRRA depends strongly on OCR quality: missed or incorrectly recognized text can produce wrong answers.The model fails when OCR tokens omit the ground-truth answer or misread text.
C. Interface Screenshots
The data collection pipeline uses staged interfaces to filter for images containing text, collect text-focused questions, and gather answers. The figures illustrate these stages and provide examples of TextVQA annotations and LoRRA predictions.
- The pipeline first identifies and removes images without text, as illustrated by the text detection interface.
- The example figure shows TextVQA images, ground-truth answers, LoRRA predictions, OCR-token attention, and whether answers came from OCR tokens or the predetermined answer space.
- The introduction-page figure presents the task addressed by the paper.
- In the second stage, workers ask questions about images whose answers require reading text in the image.Instructions and rules are provided to support high-quality questions.
- In the third stage, workers answer questions about the images.