Source-linked AI summary
Focal Visual-Text Attention for Visual Question Answering
Junwei Liang, Lu Jiang, Liangliang Cao, Li-Jia Li, Alexander Hauptmann
TL;DR
Visual question answering has mostly addressed single images, leaving questions over collections of photos or videos and their text metadata as a target setting. FVTA uses hierarchical attention to focus on relevant media and time, producing answers and supporting justifications. The paper reports state-of-the-art or comparable results across two sequential visual-text QA benchmarks.
Problem
The paper addresses how to answer questions about collections of photos or videos and associated text, where relevant supporting snippets must also be identified.
Source: §1 · pp. 1–2Method
FVTA uses a hierarchical attention process to select relevant media and temporal regions in visual-text sequences, then reasons over the selected information.
Source: §1 · p. 2, §3.2 · p. 3Results
FVTA achieves comparable or better-than-state-of-the-art results on two major sequential visual-text question answering benchmarks.
Source: §6 · p. 8Takeaways & Limitations
FVTA can provide supporting justifications alongside answers to help users verify system results.
Source: §6 · p. 8The authors identify extending FVTA to large-scale long visual-text sequences and removing answer-choice embeddings as future work.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Recent insights on language and vision with neural networks have been successfully applied to simple single-image visual question answering. However, to tackle real-life question answering problems on multimedia collections such as personal photos, we have to look at whole collections with sequences of photos or videos. When answering questions from a large collection, a natural problem is to identify snippets to support the answer. In this paper, we describe a novel neural network called Focal Visual-Text Attention network (FVTA) for collective reasoning in visual question answering, where both visual and text sequence information such as images and text metadata are presented. FVTA introduces an end-to-end approach that makes use of a hierarchical process to dynamically determine what media and what time to focus on in the sequential data to answer the question. FVTA can not only answer the questions well but also provides the justifications which the system results are based upon to get the answers. FVTA achieves state-of-the-art performance on the MemexQA dataset and competitive results on the MovieQA dataset.
1. Introduction
The paper extends visual question answering from single images to ordered collections of visual-text sequences, where answering and locating supporting evidence are both important. It proposes FVTA for this setting and reports state-of-the-art results on two VQA benchmarks.
1. Introduction
- The paper targets questions about time-ordered collections of photos or videos with associated metadata, rather than only a single image.The data may include time, GPS, captions, comments, and titles, and annotations may be inconsistent.Source: §1 · p. 1
- FVTA learns to focus on small, temporally consecutive regions and uses question words, within-sequence temporal correlation, and cross-modal interaction to infer answers.The attention tensor can also identify evidential image and text snippets for the answer.Source: §1 · p. 2
- FVTA’s localized evidence is reported as more correlated with human annotators, addressing the need to verify answers across large collections.Inspecting every image and text word can take significant time.Source: §1 · pp. 1–2
- FVTA achieves state-of-the-art results on two VQA benchmarks.The proposed attention kernel also outperforms existing attention methods.Source: §1 · p. 2
2. Related Work
Prior VQA work largely addresses single images, while video QA commonly focuses on one video. This paper distinguishes its approach by handling multiple visual-text sequences and modeling correlations across their timesteps.
2. Related Work
- Single-image VQA commonly combines question and visual features, while video QA studies use movie clips, scripts, or descriptions.The paper situates its work alongside both lines of research.Source: §2 · p. 2
- Unlike methods centered on one video, this work handles general visual-text sequences, including multiple videos or photo albums, and uses answer-choice text embeddings.The authors test on MemexQA, whose sequences include titles, timestamps, GPS, and visual content.Source: §2 · p. 2
- FVTA models correlations at every timestep across multiple sequences, unlike attention focused on image regions or a single sequence.The paper describes its attention as operating on a three-dimensional tensor rather than a vector or matrix.Source: §2 · p. 2
3. Approach
The approach formulates visual-text question answering over sequences and trains a model to maximize answer likelihood. Its architecture encodes images, text, questions, and choices, then uses FVTA attention to summarize context and question information for classification.
3.1. Problem Formulation
- Given questions and context sequences, the problem is to learn a model that maximizes the likelihood of the answers.Each context example contains an image and its corresponding text sentence.Source: §3.1 · p. 2
3.2. Network Architecture
- The architecture embeds photos with a pretrained CNN, represents text with word- and character-level embeddings, and uses separate LSTMs to encode visual and text sequences.The encoded context preserves temporal dependency within each sequence; FVTA models question-context correlations and produces summarized inputs for the output layer.Source: §3.2 · p. 3
- For multiple-choice questions, each answer choice is encoded with a bidirectional LSTM, and the attended question and context representations are combined to compute choice probabilities.The model is trained end-to-end with cross-entropy loss.Source: §3.2 · p. 3
4. Focal Visual-Text Attention
FVTA compares visual and text features, models temporal relationships within their sequences, and attends across question words and sequence timesteps. Its hierarchical attention combines localized context with visual-text and question summaries to produce a final representation.
4.1. Similarity between visual and text features
- Visual-text similarity retains a vector of comparison features, combining cosine and Euclidean information instead of summing the comparison to a scalar.The resulting vector has twice the hidden size.Source: §4.1 · pp. 3–4
4.2. Intra-sequence temporal dependency
- Temporal focal pooling uses the question to connect neighboring context hidden states when related, producing per-timestep focal representations for visual and text modalities and focusing on a few small temporal windows.The temporal windows have learnable size 2c + 1.Source: §4.2 · p. 4, Figure 3 · p. 4
4.3. Cross Sequence Interaction
- FVTA builds attention from correlations between each question word and each timestep of the focal visual-text context, then combines visual and text summaries according to their relative importance.It also computes question attention; each timestep’s attention depends on the context and question, not previous attention.Source: §4.3 · pp. 4–5
- The FVTA attention tensor models both visual-text correlations within sequences and interactions across sequences, unlike a one-dimensional soft attention vector.The model first summarizes visual and text focal contexts separately, then combines them according to inter-sequence importance.Source: Figure 3 · p. 4, §4.3 · p. 5
5 Compute S by Eq. (7) // cross-sequence
The procedure reduces F with S to obtain the FVTA representation.
5 Compute S by Eq. (7) // cross-sequence
- Reducing F with S produces the FVTA representation ˜h, which the procedure then returns.This is the stated final step of the procedure.Source: §5 · p. 5
5. Experiments
FVTA is evaluated on MemexQA and MovieQA, where it performs strongly against comparison methods. Ablations examine its attention components, while qualitative examples show how its attention can identify supporting visual and text evidence.
5.1. MemexQA
- MemexQA contains 20,860 questions about 13,591 personal photos from 101 users, with incomplete metadata and variable-length visual-text sequences.Questions have four answer choices and one correct answer; the dataset provides ground-truth evidence images.Source: §5.1 · p. 5
5.1.1 Comparison to the state-of-the-art
- FVTA achieves state-of-the-art accuracy on MemexQA and significantly outperforms previous attention methods.Table 1 compares methods by question type; its first three methods do not use attention.Source: §5.1.1 · p. 6, Table 1 · p. 6
- FVTA finds ground-truth evidence photos more effectively overall, and one example selects the corn-eating photo where soft attention selects the wrong image and answer.The MemexQA evidence photos are ground-truth photos used to answer each question.Source: §5.1.1 · pp. 6–7, Table 2 · p. 6
5.1.2 Ablation Study
- Ablations find that the proposed kernel, intra-sequence dependency, and cross-sequence interaction contribute to performance, with intra-sequence dependency more important.Question attention provides slight improvement; the no-photo result is strong, which the authors relate to dataset language bias.Source: §5.1.2 · p. 7, Table 3 · p. 7
5.2. MovieQA
- 0.410 vs 0.387 validation accuracy and 0.373 vs 0.363 test accuracy for FVTA and RWMN [15]; FVTA outperforms all baselines and is comparable to the MovieQA test-server state of the art.The test set is evaluated on the MovieQA server, and not all studies report test accuracy.Source: Table 4 · p. 7
- On MovieQA, FVTA identifies informative movie frames and subtitle sentences as fine-grained justifications, whereas most existing methods cannot provide justifications below the clip level.The qualitative analysis presents both visual and text justifications based on top attention activation.Source: §5.2 · pp. 7–8, Figure 5 · p. 8
6. Conclusions and future work
FVTA uses a hierarchical process to select modalities and snippets in sequential visual-text data. It achieves comparable or better than state-of-the-art results on two major benchmarks and identifies supporting justifications.
6. Conclusions and future work
- The network uses a hierarchical process to dynamically determine which modalities and snippets to focus on in sequential data.Source: §6 · p. 8
- FVTA achieved comparable or even better than state-of-the-art results on two major question-answering benchmarks for sequential visual-text data.Source: §6 · p. 8
- FVTA predicts answers and finds supporting justifications that help users verify the system’s results.Source: §6 · p. 8
- Future work includes extending FVTA to large-scale long visual-text sequences and removing answer-choice embeddings from the input.Source: §6 · p. 8