Source-linked AI summary
Scene Text Visual Question Answering
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusiñol, Ernest Valveny, C. V. Jawahar, Dimosthenis Karatzas
TL;DR
Existing VQA datasets and models provide limited treatment of textual information in images, despite its importance for scene understanding. The paper introduces ST-VQA, text-dependent tasks, baselines, and the ANLS metric; its analyses show that textual information improves generic VQA models and that generative approaches better accommodate scene-text answers.
Problem
Existing VQA datasets and models largely ignore scene text, limiting performance on questions that require textual information from images.
Method
The paper introduces the ST-VQA dataset, three tasks with increasing contextual difficulty, baseline methods, and Average Normalized Levenshtein Similarity for evaluation.
Results
Adding textual information improves generic VQA models, while generative pipelines and ANLS better accommodate multi-word, out-of-dictionary answers and recognition errors.
Takeaways & Limitations
Scene text should be incorporated into VQA systems, and generative evaluation is appropriate when answers include arbitrary strings such as numbers, license plates, or codes.
Abstract
from arXiv · showhide
Current visual question answering datasets do not consider the rich semantic information conveyed by text within an image. In this work, we present a new dataset, ST-VQA, that aims to highlight the importance of exploiting high-level semantic information present in images as textual cues in the VQA process. We use this dataset to define a series of tasks of increasing difficulty for which reading the scene text in the context provided by the visual information is necessary to reason and generate an appropriate answer. We propose a new evaluation metric for these tasks to account both for reasoning errors as well as shortcomings of the text recognition module. In addition we put forward a series of baseline methods, which provide further insight to the newly released dataset, and set the scene for further research.
1. Introduction
ST-VQA addresses the limited treatment of scene text in VQA by requiring textual information in images to answer questions. It introduces tasks and evaluation designed to connect text recognition with visual reasoning.
- Motivation: Text in man-made environments conveys explicit high-level semantic information unavailable in other scene representations.Such information supports everyday activities including purchases, transportation, navigation, appointments, and checking store hours.
- Motivation: About 50% of images in large-scale datasets contain text, with higher prevalence in urban environments.The authors therefore frame scene text as central rather than marginal to holistic scene interpretation.
- Research gap: Existing VQA datasets and models largely ignore scene text and perform poorly on questions requiring scene-text understanding.Integrating text also requires reasoning over positions, colors, objects, semantics, and contextual information.
- Contribution: ST-VQA is a dataset whose questions can only be answered using text present in the image.The majority of images, 85.5%, come from datasets with generic question-answer pairs, enabling a more holistic VQA task.
- Contribution: The work introduces three tasks with increasing difficulty and an evaluation metric using Levenshtein distance.The metric accounts for both reasoning errors and shortcomings in text recognition.
2. Related Work
Related work spans scene-text detection and recognition, general VQA, and multimodal question answering. ST-VQA is positioned alongside TextVQA while differing in image sources and annotation criteria.
- Scene Text Detection and Recognition: Scene-text reading commonly uses separate text detection and recognition stages, with detection methods largely based on Fully Convolutional Neural Networks.Recognition approaches include word-level classification, Connectionist Temporal Classification, and end-to-end CNN encoder–LSTM attention decoder architectures.
- Visual Question Answering: General VQA answers natural-language questions about images, but existing datasets largely omit textual content except TextVQA.This omission motivates VQA datasets that exploit text in images as a semantic cue.
- Multimodal Question Answering: Related multimodal QA datasets address charts, diagrams, documents, and textbooks, but TQA provides textual information in computer-readable form.These settings differ from scene-text VQA, where text must be read from the image.
- TextVQA: TextVQA concurrently combines a standard VQA model with an independently trained OCR module and a copy mechanism that can select recognized words as answers.The copy mechanism is inspired by pointer networks.
- ST-VQA and TextVQA: ST-VQA uses multiple source image datasets and requires at least two text instances per annotated image, whereas TextVQA uses Open Images and category-based sampling.The two datasets use non-overlapping image sources and are described as complementary for transfer learning and combined training.
3. ST-VQA Dataset
ST-VQA combines scene-text and generic images with questions designed to require textual answers, then analyzes dataset characteristics and introduces progressively harder task settings. It also proposes ANLS to accommodate both reasoning and OCR errors.
- Data Collection: 23,038 images and 31,791 questions were collected from multiple scene-text and generic computer-vision datasets.The images were selected to contain at least two text instances, providing multiple possible textual answers.
- Data Collection: Crowdsourced questions were closed-ended, unambiguously answerable from image text, and verified by independent workers.Workers were prohibited from asking yes/no questions or questions answerable only from visual information.
- Data Collection: The final split contains 19,027 training images with 26,308 questions and 2,993 testing images with 4,163 questions.ST-VQA contains one answer per question because answers are explicitly grounded in scene text.
- Analysis and Comparison with TextVQA: ST-VQA questions commonly begin with “What,” “Where,” “Which,” “How,” or “Who,” while many require world knowledge to interpret scene text.Questions may ask about brands, websites, or names rather than explicitly requesting text appearing in the image.
- Tasks: Three tasks provide increasing contextual difficulty: strongly contextualised, weakly contextualised, and open vocabulary.The settings use an image-specific dictionary, a shared 30,000-word dictionary, or no extra dictionary information, respectively.
- Evaluation and Open Challenge: ANLS uses normalized Levenshtein similarity with threshold τ = 0.5, softly penalising OCR errors while assigning zero when normalized distance exceeds the threshold.This metric is intended to capture both answer reasoning and text-recognition performance.
4. Baselines and Results
The baselines compare methods using visual, question, and OCR-derived information across ST-VQA’s tasks. Results show that textual information is necessary, while ANLS accommodates recognition errors that exact-match accuracy cannot.
- Scene Text Retrieval predicts text using bounding boxes and PHOC representations while ignoring the question and other image information.
- Scene Image OCR ranks detected text by confidence and selects vocabulary matches for tasks 1 and 2, while directly returning the top detection for task 3.
- SAAA and SAN are evaluated with 1k, 5k, and 19,296 answer-class vocabularies to measure the effect of output-space size.
- Standard VQA models ignore scene text, achieving 0.085–0.102 ANLS and 6.36%–7.78% accuracy.
- SAAA improves from 1k to 5k classes but worsens at 19k, suggesting that classification over very large vocabularies is not feasible.
- ANLS generally tracks accuracy but softly penalizes near-miss OCR outputs, explaining why Scene Image OCR ranks better by ANLS than by accuracy.
- Standard VQA models perform worse or comparably at best to question-agnostic STR retrieval and Scene Image OCR, while adding PHOC features slightly improves VQA results.
- Question-type analysis finds that models fail on license plates, “who,” and “what number,” reflecting the limits of pure classification for out-of-vocabulary answers.
5. Conclusions and Future Work
The paper concludes that ST-VQA exposes major limitations in existing VQA approaches and motivates models that integrate scene text through generative answering.
- Adding textual information to generic VQA models improves performance, but ad-hoc OCR-based baselines can outperform them.
- Scene-text answers make classification impractical because the possible answer classes are intractable.
- Generative pipelines are required to represent multiple-word answers and out-of-dictionary strings such as numbers, license plates, and codes.
- Average Normalized Levenshtein Similarity better suits generative models and responds smoothly to text-recognition performance.