Source-linked AI summary
Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA
Ronghang Hu, Amanpreet Singh, Trevor Darrell, Marcus Rohrbach
TL;DR
TextVQA requires reading and reasoning over image text, but prior methods largely use pairwise fusion and one-step answer classification. M4C combines rich image-text representations with a pointer-augmented multimodal transformer and iterative decoding, outperforming previous work across three benchmark datasets.
Problem
TextVQA requires models to jointly process questions, visual content, and image text, while prior approaches rely mainly on pairwise fusion and single-step classification.
Method
M4C embeds all modalities in a common semantic space, applies homogeneous self-attention, and iteratively decodes answers with dynamic pointers that can copy image text and use the model vocabulary.
Results
M4C significantly outperforms previous work on TextVQA, ST-VQA, and OCR-VQA, with reported relative gains of 25%, 65%, and 32%, respectively.
Takeaways & Limitations
The results support handling multiple modalities through domain-specific embeddings and homogeneous self-attention, while forming complex answers through multi-step decoding.
Takeaways & Limitations
Performance remains constrained by OCR failures and by challenges resolving object-text relations or understanding large text regions such as book pages.
Abstract
from arXiv · showhide
Many visual scenes contain text that carries crucial information, and it is thus essential to understand text in images for downstream reasoning tasks. For example, a deep water label on a warning sign warns people about the danger in the scene. Recent work has explored the TextVQA task that requires reading and understanding text in images to answer a question. However, existing approaches for TextVQA are mostly based on custom pairwise fusion mechanisms between a pair of two modalities and are restricted to a single prediction step by casting TextVQA as a classification task. In this work, we propose a novel model for the TextVQA task based on a multimodal transformer architecture accompanied by a rich representation for text in images. Our model naturally fuses different modalities homogeneously by embedding them into a common semantic space where self-attention is applied to model inter- and intra- modality context. Furthermore, it enables iterative answer decoding with a dynamic pointer network, allowing the model to form an answer through multi-step prediction instead of one-step classification. Our model outperforms existing approaches on three benchmark datasets for the TextVQA task by a large margin.
1. Introduction
TextVQA addresses the need to read and reason about image text alongside questions and visual content. M4C uses richer text representations, homogeneous multimodal attention, and iterative pointer-based decoding to address limitations in prior approaches.
- TextVQA requires models to see, read, and reason over questions, visual contents, and text in images.Earlier VQA datasets and methods largely focused on visual components while ignoring image text that can carry essential scene information.
- Prior TextVQA methods use pairwise multimodal fusion and single-step classification or copying, limiting modality interactions and complex answer generation.These methods may select training-set answers or copy a single image-text token, making multiword or mixed-vocabulary answers difficult.
- Previous word-embedding-based image-text features miss cues such as font, color, and location that help distinguish related text tokens.Tokens with different fonts and spatial separation usually do not belong to the same street sign.
- M4C projects all modalities into a common semantic space and uses multimodal self-attention to model intra- and inter-modality interactions.Its transformer jointly represents questions, visual content, and image text rather than relying on pairwise fusion mechanisms.
- M4C performs iterative answer decoding with dynamic pointers, enabling permutation-invariant selection of image text and combinations with its own vocabulary.This supports answers containing both image-specific text and common vocabulary words.
- M4C outperforms previous work by 25% relative on TextVQA, 65% on ST-VQA, and 32% on OCR-VQA.The reported gains span three challenging TextVQA benchmark datasets.
2. Related work
Related work covers TextVQA systems built around OCR inputs, broader multimodal fusion methods, and dynamic copying with pointer networks. Prior TextVQA approaches commonly extend VQA models with OCR-aware attention or dynamically add OCR tokens to the answer space.
- TextVQA research introduced datasets and methods for visual question answering based on reading and understanding image text.
- LoRRA extends a VQA framework with question-conditioned OCR attention and OCR token indices in the classifier output space.It allows the model to copy a single OCR token from the image as the answer.
- A related approach groups OCR tokens into blocks and adds them to both the input features and output answer space.Other methods similarly augment existing VQA models with OCR inputs.
- Vision-and-language work has explored attention, bilinear models, self-attention, and graph networks for multimodal fusion.
- Pointer-based dynamic copying has been studied for selecting input content in tasks including text summarization, knowledge retrieval, and image captioning.The approach is relevant to TextVQA because many answers come from image text and cannot all fit in a fixed vocabulary.
3. Multimodal Multi-Copy Mesh (M4C)
M4C embeds question words, visual objects, and OCR tokens in a shared semantic space, then uses transformer self-attention to model their relations. Its iterative decoder combines fixed-vocabulary prediction with dynamic OCR copying to construct answers over multiple steps.
- Input representations: M4C extracts features from question words, detected visual objects, and OCR tokens, representing the three modalities as inputs to one model.Question features use BERT, object features come from Faster R-CNN, and OCR features combine textual, visual, character, and spatial information.
- Common embedding space: All modality-specific features are projected into a learned common d-dimensional embedding space before multimodal processing.The projections align entities from the question, image objects, and image text for joint modeling.
- Pointer-augmented decoding: At each decoding step, M4C selects the highest-scoring candidate from fixed-vocabulary words and dynamically copied OCR tokens.Vocabulary scores and OCR-copying scores are concatenated, yielding V + N candidates.
- Multimodal fusion: Transformer self-attention lets every entity attend to entities from the same or different modalities, enriching representations with intra- and inter-modality context.An OCR token can attend to another OCR token, a detected object, or a question word.
- Iterative answer prediction: The model decodes answers auto-regressively for T steps, feeding the previous prediction into the next step.The decoder reuses the transformer layers and predicts answer components sequentially.
4. Experiments
M4C is evaluated on TextVQA, ST-VQA, and OCR-VQA using rich OCR features, multimodal transformers, and iterative decoding. It significantly outperforms prior methods, while ablations show benefits from richer OCR representations and multi-step prediction.
- Experimental setup: M4C is evaluated on TextVQA, ST-VQA, and OCR-VQA, with TextVQA accuracy measured by soft voting over 10 human answers.The experiments use BERT-based question features, Faster R-CNN visual features, Rosetta OCR, and a 12-step maximum decoder.
- TextVQA results: 39.01% test accuracy is achieved without extra ST-VQA training data, while 40.46% is achieved with it, outperforming DCD ZJU by 9% absolute.The additional ST-VQA data provides a further 1% improvement.
- Ablations: Multi-step decoding substantially improves accuracy over single-step prediction across TextVQA, ST-VQA, and OCR-VQA validation sets.The experiments use 12 decoding steps by default, and OCR-VQA often contains multi-word answers such as book titles and author names.
- Ablations: 13% absolute improvement over LoRRA is obtained by the full model, while rich OCR features contribute around 4% absolute accuracy over FastText-only features.Bounding-box, appearance, and PHOC features each improve performance over the simpler OCR representation.
- Qualitative insights: OCR failures remain a major error source, alongside difficulty resolving object-text relations and understanding large text regions such as book pages.The authors identify better OCR and question-conditioned text recognition as possible directions.
- ST-VQA results: +0.18 absolute ANLS is achieved over the ST-VQA challenge winner VTA, while the restricted one-step model already outperforms prior methods.The qualitative examples show M4C combining multiple OCR tokens with fixed-vocabulary words.
5. Conclusion
M4C combines rich image-text representations with pointer-augmented multimodal transformers and iterative answer decoding. It outperforms prior TextVQA methods by a large margin on three challenging datasets.
- M4C jointly models question words, visual objects, and image text through a pointer-augmented multimodal transformer over a shared embedding space.
- M4C predicts answers through iterative decoding rather than one-step classification.
- M4C uses rich representations for text in images to support TextVQA reasoning.
- M4C outperforms previous work by a large margin on three challenging TextVQA datasets.
A. Hyper-parameters in M4C
The appendix summarizes M4C’s hyper-parameters across TextVQA, ST-VQA, and OCR-VQA. Most settings are shared, with additional training adjustments for OCR-VQA.
- Most M4C hyper-parameters are shared across TextVQA, ST-VQA, and OCR-VQA.
- OCR-VQA uses 2× the total iterations and an adapted learning-rate schedule because it contains more images.
- Table A.1 summarizes the hyper-parameters used in the M4C model.
B. Additional ablation analysis
M4C’s ablations show that answer generation requires both fixed-vocabulary words and dynamically copied OCR tokens. Removing either component substantially reduces accuracy, while multi-token copying remains valuable.
- At each decoding step, M4C selects the next answer word from either its fixed vocabulary or image OCR tokens.
- Both removing the fixed answer vocabulary and removing OCR copying causes a large accuracy drop relative to the full model.
- M4C without fixed vocabulary still outperforms LoRRA, highlighting the value of copying multiple OCR tokens to form answers.
- The ablation study identifies both fixed and dynamic vocabularies as important components of M4C.
C. Additional qualitative examples
Qualitative examples across TextVQA, ST-VQA, and OCR-VQA show M4C usually copying and combining image text with fixed-vocabulary words. Its main failures involve OCR localization, large text, and text-object relations.
- OCR localization failure is a major source of M4C errors on TextVQA.
- The examples span M4C predictions from TextVQA, ST-VQA, and OCR-VQA validation sets.
- M4C generally identifies and copies image text, then combines it with fixed-vocabulary words to answer questions.
- M4C occasionally fails when reading large text or resolving relations between text and objects.
- The qualitative figures mark OCR outputs with red boxes, using orange for OCR tokens and blue for fixed-vocabulary words.