Source-linked AI summary
Compositional Memory for Visual Question Answering
Aiwen Jiang, Fang Wang, Fatih Porikli, Yi Li
TL;DR
Holistic visual features in prior VQA methods may miss fine-grained local information and its interaction with language. This paper sequentially fuses question-word and local-patch features into episodes for question answering. It reports improved state-of-the-art performance on DAQUAR and MSCOCO-VQA, including a 6% advance on DAQUAR.
Problem
Holistic visual features in VQA may neglect interactions with language and finer-grained local image information relevant to answering questions.
Source: §1 · p. 1Method
Compositional Memory sequentially fuses question-word and local-patch features into episodes, which are supplied to a question-answering module with contextual visual and linguistic information.
Source: §1 · pp. 1–2Results
The method outperforms state-of-the-art methods on DAQUAR and MSCOCO-VQA, advancing the DAQUAR state of the art by 6%.
Source: §1 · p. 2Takeaways & Limitations
The authors conclude that the generated episodes are effective for high-level reasoning on VQA tasks.
Source: §5 · pp. 8–9MSCOCO-VQA evaluation protocols and input handling were not standardized, with different answer selections used across evaluations.
Source: §4.5 · p. 7
Abstract
from arXiv · showhide
Visual Question Answering (VQA) emerges as one of the most fascinating topics in computer vision recently. Many state of the art methods naively use holistic visual features with language features into a Long Short-Term Memory (LSTM) module, neglecting the sophisticated interaction between them. This coarse modeling also blocks the possibilities of exploring finer-grained local features that contribute to the question answering dynamically over time. This paper addresses this fundamental problem by directly modeling the temporal dynamics between language and all possible local image patches. When traversing the question words sequentially, our end-to-end approach explicitly fuses the features associated to the words and the ones available at multiple local patches in an attention mechanism, and further combines the fused information to generate dynamic messages, which we call episode. We then feed the episodes to a standard question answering module together with the contextual visual information and linguistic information. Motivated by recent practices in deep learning, we use auxiliary loss functions during training to improve the performance. Our experiments on two latest public datasets suggest that our method has a superior performance. Notably, on the DARQUAR dataset we advanced the state of the art by 6$\%$, and we also evaluated our approach on the most recent MSCOCO-VQA dataset.
1. Introduction
The introduction argues that holistic visual features may miss region-specific information relevant to VQA. It proposes Compositional Memory to fuse local visual and linguistic features over time, reporting improved results on two public datasets.
1. Introduction
- Holistic features may be insufficient when answers depend on image regions, motivating finer-grained local visual information for VQA.The introduction notes that holistic approaches may hinder exploration of local features.Source: §1 · p. 1
- Compositional Memory fuses local visual and linguistic features over time into episodes that characterize their interaction and dynamics.Question words are traversed sequentially; attention reweights regions, and a dynamic network generates episodes for a question-answering module.Source: §1 · pp. 1–2
- During question parsing, image regions can act as candidates whose importance varies across words, connecting local visual evidence to dynamic reasoning.The introduction relates this search over local evidence to semantic facts explored in natural language processing.Source: §1 · p. 1
- The approach uses overlapping patches covering important image content, represented by features from the CNN’s last convolutional layer.Source: §1 · p. 2
- The method outperforms state-of-the-art methods on DAQUAR and MSCOCO-VQA, advancing the DAQUAR state of the art by 6%.The paper also compares variants of the method and evaluates it on MSCOCO-VQA.Source: §1 · p. 2
2. Related Work
Related work describes recurrent models for temporal sequences and LSTM mechanisms for learning their dynamics, alongside language-and-vision tasks such as image captioning and VQA. Existing LSTM-VQA systems commonly use holistic visual features, while concatenation does not explicitly extract finer-grained information.
2. Related Work
- LSTM memory cells and input, forget, and output gates regulate stored and transferred information, helping alleviate vanishing gradients across time steps.RNNs are described as suitable for nonlinear temporal sequences but difficult to train due to vanishing gradients.Source: §2 · p. 2
- VQA uses images to answer questions, and image captioning has adopted attention as a useful component in sentence generation.Source: §2 · p. 2
- Typical LSTM-VQA systems use holistic CNN features as visual words, placing them before or after the question sequence or concatenating them with question-word vectors.Figure 1 is identified as a classical LSTM-VQA model.Source: §2 · p. 2, Figure 1 · p. 2
- Concatenating feature vectors does not explicitly extract finer-grained information, motivating more effective image-language fusion for VQA.The related-work discussion notes that global visual features may neglect details of facts.Source: §2 · p. 2
3. Our Approach
The approach section first presents a model overview, then discusses technical details and training.
3. Our Approach
- The section is organized around a model overview followed by technical details and the training process.Source: §3 · p. 3
3.1. Our End-to-End VQA model
The model adds Compositional Memory and a question-parsing Language LSTM to a basic VQA LSTM. Compositional Memory processes region inputs into dynamic episodes used by the answer-generation module.
3.1. Our End-to-End VQA model
- The model adds Compositional Memory and a question-parsing LSTM to the basic VQA LSTM, with memory producing an image-level representation for answering.The question-parsing LSTM supplies input to both Compositional Memory and the question-answering LSTM.Source: §3.1 · p. 3
3.1.1 Compositional Memory
- Region-LSTMs process image regions in parallel, conditioned on prior episodes, local features, and the current question word; their parameters are shared across regions.Each Region-LSTM retains its own gate and cell state.Source: §3.1.1 · p. 3
- The α gate weights region messages for combination into episodes, which are pooled as image-level information at each time step.The pooling mechanism is a modified gated-recurrent-style mechanism.Source: §3.1.1 · p. 3
- Compositional Memory combines LSTM and attention mechanisms to generate dynamic episodic messages beyond a static image feature.The episodes represent vision-language interactions over time.Source: §3.1 · p. 3, §3.1.1 · p. 4
3.1.2 Language LSTM
- The Language LSTM captures recurrent relations among question words and stores semantic information about the question.The paper calls this strategy “factorization of questions.”Source: §3.1.2 · p. 4
3.1.3 Other components
- The answer-generation LSTM receives dynamic episodes alongside linguistic information and static visual context from the CNN.The approach uses the last convolutional layer for region selection and the last CNN layer as visual context.Source: §3.1.3 · p. 4
3.2. Training
The training section focuses on open-ended VQA, where answers may contain multiple words. It describes autoregressive answer prediction, auxiliary training losses, and end-to-end implementation details.
3.2. Training
- Open-ended VQA is selected as the most difficult task category to demonstrate the approach.The paper distinguishes single-word, open-ended multiple-word, and multiple-choice VQA.Source: §3.2 · p. 4
3.2.1 Protocol
- For open-ended VQA, the model predicts answer words recursively and stops when it emits the ⟨EOA⟩ token.During training the question sequence is augmented with the ground-truth answer; at test time it is augmented with previously predicted answer words.Source: §3.2.1 · p. 5
3.2.2 Loss function
- Two auxiliary losses—Language Only and Episode Only—are added during training and removed at testing.The model is jointly learned with cross-entropy loss, computed only for predicted answer words.Source: §3.2.2 · p. 5
3.2.3 Implementation
- The CNN models are pretrained, then fine-tuned during recurrent network training; the implementation uses Caffe2.The authors state that the CNN and LSTMs are jointly learned end to end.Source: §3.2.1 · p. 4, §3.2.3 · p. 5
4. Experiments
The experiments evaluate the method on DAQUAR and MSCOCO-VQA, where it outperforms state-of-the-art methods across recognized metrics. DAQUAR uses the full dataset rather than its restricted reduced set.
4. Experiments
- The method outperforms state-of-the-art methods on DAQUAR and MSCOCO-VQA across recognized metrics.On DAQUAR, the authors report a 6% advance over the state of the art.Source: §4 · p. 5
4.1. Datasets
- DAQUAR contains 12,468 question-answer pairs across 1,449 indoor-scene images; MSCOCO-VQA contains 369,861 questions and 3,698,610 answers across 123,287 images.DAQUAR's split is 795 training images and 654 testing images; MSCOCO-VQA uses 82,783 training and 40,504 validation images.Source: §4.1 · p. 5
- DAQUAR experiments use the full dataset rather than the reduced set, which restricts outputs to 37 object categories and covers 25 test images.The authors describe the full dataset as more challenging and its results as more meaningful in statistics.Source: §4.1 · p. 5
4.2. Evaluation criteria
DAQUAR uses WUPS scores at different thresholds, while MSCOCO-VQA uses the organizers' accuracy criterion for open-ended answers.
4.2. Evaluation criteria
- DAQUAR is evaluated with WUPS at different thresholds using Standard, Average Consensus, and Min Consensus metrics.The Standard Metric is the basic score; the consensus metrics study effects of consensus in question answering.Source: §4.2 · p. 5
- MSCOCO-VQA open-ended answers are evaluated by accuracy, the percentage that exactly agree with the human-provided ground truth.Source: §4.2 · p. 5
4.3. Experimental settings
The experiments use AlexNet and GoogLeNet region features, with shared region-LSTM and gate parameters; training runs for 100,000 iterations.
4.3. Experimental settings
- AlexNet contributes 36 flattened Pool5 regions, while GoogLeNet contributes 49 inception poo5b/output regions.Source: §4.3 · p. 5
- Region-LSTM and α-gate parameters are shared across regions, minimizing computational burden, but each region's memory state increases storage needs.Region input dimensions are 1024 for GoogLeNet and 256 for AlexNet; episodes are 200-dimensional for all LSTMs on both datasets.Source: §4.3 · pp. 5–6
- Training stops after 100,000 iterations with a base learning rate of 0.01, taking about 0.4 seconds per iteration on GTX Nvidia 780.Source: §4.3 · p. 6
4.4. Results on DAQUAR
On DAQUAR, the method surpasses state of the art, and the authors report that local features help on sophisticated, location-dependent questions. Ablations indicate that memory, language, and visual context contribute complementary information.
4.4.1 Examples
- Local features help most on sophisticated, location-dependent questions, such as identifying an object's color or counting chairs.The examples include correct cases and failures; some failure-case answers are only partially correct or wrong despite relating to nearby objects.Source: §4.4.1 · p. 6
4.4.2 Comparisons with state-of-the-art methods
- On WUPS@0.9, our method is 6% higher than the state of the art, rising from 23.31% to 29.77%; at a lower WUPS threshold, it is 5.25% superior.Source: §4.4.2 · p. 6
4.4.3 Effectiveness of Compositional Memory module
- Removing the proposed memory lowers performance; language and episodes together outperform either alone, and the full model is consistently better than other variants.Without holistic context, standard accuracy falls from 24.37% to 23.15%; Language+Episodes raises WUPS@0.9 from 25.77% to 28.73%.Source: §4.4.3 · p. 7, Table 2 · p. 7, Figure 6 · p. 7
4.5. Results on MSCOCO-VQA
MSCOCO-VQA is larger and includes more scenes and question types than DAQUAR. The section reports similar state-of-the-art performance with a larger answer dictionary, alongside accuracy patterns across answer and question types.
4.5. Results on MSCOCO-VQA
- MSCOCO-VQA is larger than DAQUAR and includes more scenes and question types not covered by DAQUAR.The section presents examples and statistics for the dataset, including answer and question categories.Source: §4.5 · p. 7, §4.5.1 · p. 8
- MSCOCO-VQA accuracy comparisons are difficult to standardize because evaluation protocols and reporting practices differ.The authors specify their tokenizer, 13,880-word answer dictionary, and results on both test-dev and the full validation set.Source: §4.5 · pp. 7–8
4.5.1 Statistical results
- Accuracy drops significantly as answers become more sophisticated, despite very high accuracy for simple types such as yes/no.The authors identify this pattern as indicating potential issues and directions for their method.Source: §4.5.1 · p. 8
- Accuracy drops significantly for more complicated questions than simple forms such as “is there?”; the validation-set bars are sorted by accuracy.Figure 8 displays accuracy for different question types in the open-ended validation task.Source: §4.5.1 · p. 8, Figure 8 · p. 8
- Similar state-of-the-art performance was achieved with a larger answer dictionary on MSCOCO-VQA open-ended tasks.The larger dictionary potentially makes correct answers more difficult; Table 3 reports overall and answer-type accuracy against the LSTM Q+I baseline.Source: Table 3 · p. 8, §4.5.1 · p. 8
5. Conclusion
The conclusion presents Compositional Memory as the core of an end-to-end VQA approach that dynamically extracts local features and uses an LSTM to fuse image regions and language into episodes. Experiments on the latest public datasets suggest superior performance.
5. Conclusion
- Compositional Memory is proposed as the core element of the VQA approach.The approach dynamically extracts local features, fuses image regions and language with an LSTM, and generates episodes for high-level reasoning.Source: §5 · p. 8
- Experiments on the latest public datasets suggest that the method has superior performance.The conclusion states this outcome without specifying dataset-level values in these passages.Source: §5 · p. 9
- The generated episodes are described as effective for high-level reasoning.They follow the LSTM-based fusion of image regions and language.Source: §5 · pp. 8–9