Source-linked AI summary
Dynamic Memory Networks for Visual and Textual Question Answering
Caiming Xiong, Stephen Merity, Richard Socher
TL;DR
The paper asks whether dynamic memory networks can perform question answering without marked supporting facts and across image and text inputs. It improves the DMN’s input and memory modules, adds an image input module, and reports state-of-the-art results on VQA and bAbI-10k. The authors conclude that the framework generalizes across input domains, although soft attention loses positional and ordering information and the model struggles on Basic Induction.
Problem
It was not established whether DMNs could achieve strong question-answering results without marked supporting facts or apply to images.
Method
The paper analyzes and improves the DMN’s input and memory modules, adding input fusion, attention-based memory updates, and a compatible image input module.
Results
The DMN+ achieves state-of-the-art results on both the VQA dataset and the bAbI-10k text question-answering dataset; it reduces mean error by 1.4% versus the end-to-end memory network on bAbI-10k.
Takeaways & Limitations
The resulting framework can generalize across input domains while learning to select important facts without supporting-fact supervision.
Takeaways & Limitations
Soft attention loses positional and ordering information, and the model has difficulty with QA16: Basic Induction.
Abstract
from arXiv · showhide
Neural network architectures with memory and attention mechanisms exhibit certain reasoning capabilities required for question answering. One such architecture, the dynamic memory network (DMN), obtained high accuracy on a variety of language tasks. However, it was not shown whether the architecture achieves strong results for question answering when supporting facts are not marked during training or whether it could be applied to other modalities such as images. Based on an analysis of the DMN, we propose several improvements to its memory and input modules. Together with these changes we introduce a novel input module for images in order to be able to answer visual questions. Our new DMN+ model improves the state of the art on both the Visual Question Answering dataset and the \babi-10k text question-answering dataset without supporting fact supervision.
1. Introduction
The paper improves the dynamic memory network for question answering without marked supporting facts and extends it to visual question answering. It introduces changes to the input and memory modules, plus a new image input module.
- The DMN combines memory and attention mechanisms and previously achieved state-of-the-art results when supporting facts were marked during training.
- The authors analyze and improve the DMN’s input and memory modules for question answering.The supplied passage introduces this analysis as the basis for the proposed improvements.
- The new DMN+ learns to select important facts from a larger set without labeled supporting facts.
- A new image input module represents images and feeds its output into the existing memory module.The module is compatible with the rest of the DMN architecture.
- The model addresses question answering over both text and images.Both tasks are illustrated in Figure 1.
2. Dynamic Memory Networks
A dynamic memory network processes input into ordered facts, represents the question, and repeatedly retrieves relevant information through attention and episodic memory updates. An answer module then uses the question and final memory to predict a single answer or decode a sequence.
- The DMN is a modular question-answering architecture whose input, question, memory, and answer components can be analyzed independently.
- Input Module: The input module converts data into an ordered set of fact vectors for later components.For text question answering, the original module uses a GRU over input words.
- Question Module: The question module represents the question with the final hidden state of a GRU over its words.
- Episodic Memory Module: The episodic memory module retrieves information relevant to the question from input facts, potentially through multiple passes for transitive reasoning.
- Episodic Memory Module: Attention summarizes relevant input using the question and previous memory, while the update mechanism produces the next episode memory.The initial memory is set to the question vector.
- Answer Module: The answer module combines the question and final memory to produce a predicted answer, using softmax for simple answers or an RNN for sequences.Training uses cross-entropy error backpropagated through the network.
3. Improved Dynamic Memory Networks: DMN+
DMN+ improves the input and memory modules for question answering without supporting-fact labels, and adds an image input module for visual question answering. Its input fusion and attention mechanisms enable information exchange across text sentences or image regions before episodic memory retrieves relevant information.
- DMN+ contributions: DMN+ compares modeling choices for input representation, attention, and memory update while targeting text and visual question answering.The final model obtains the highest accuracy on bAbI-10k without supporting facts and on the VQA dataset.
- Input Fusion Layer: The textual input module separates sentence encoding from cross-sentence fusion, using a bi-directional GRU to incorporate past and future sentence information.This design also gives distant supporting sentences a more direct interaction because gradients need not propagate through intervening words.
- Input Module for VQA: The VQA input module treats local image regions as text-like input units, embeds them into the question’s textual feature space, and applies fusion across regions.The image is split into a 14 × 14 grid, producing 196 local regional vectors, and the fusion layer propagates information between neighboring patches.
- Episodic Memory Module: The episodic memory module makes multiple passes over fused input facts, focusing attention on facts using the question and previous episode memory.Its attention gates interact with each fact, the question representation, and the previous memory state before producing a contextual vector for memory updating.
- Attention Mechanism: The attention-based GRU is selected for DMN+ because it improves performance, while soft attention can lose positional and ordering information through summation.The attention gates are scalar softmax values, enabling visualization of how attention activates over the input.
- Episode Memory Updates: Untying memory-update weights and using a ReLU formulation improve accuracy by another 0.5%.The updated memory state is computed from the previous memory, contextual vector, and question before being passed to the answer module.
4. Related Work
The paper situates DMN-based question answering at the intersection of memory and attention mechanisms, connecting textual and visual QA research. Related visual work includes attention-based models and image-description systems.
- Memory networks and DMNs use memory, input, scoring, attention, and response mechanisms for reasoning over language facts.
- Attention mechanisms let models selectively focus on inputs using a question, benefiting image classification, image captioning, and machine translation.
- Visual question answering emerged as a large-scale task with open-ended questions about general images, enabled by accurate object identification.
- The paper compares with stacked attention networks, which map image-patch features independently and cannot easily incorporate local adjacency in hidden states.
- Image description systems map images and sentences into a shared space or use recurrent networks to generate novel image-based sentences.
5. Datasets
The experiments use three datasets spanning synthetic textual QA, real-world image QA, and large-scale visual QA. They evaluate textual QA with bAbI-10k and visual QA with DAQUAR and VQA under dataset-specific protocols.
- The study uses three datasets to analyze model changes and compare performance with other architectures.
- bAbI-10k is a synthetic English dataset with 20 tasks, where examples contain facts, questions, answers, and supporting facts.
- DAQUAR contains 795 training images, 654 test images, 6,795 training questions, and 5,673 test questions.
- After excluding multiple-word answers, DAQUAR covers 90% of the original data and uses single-word classification accuracy.
- VQA uses Microsoft COCO images and contains 248,349 training questions, 121,512 validation questions, and 244,302 test questions.
- VQA test-standard allows five evaluations, test-challenge is evaluated only at competition end, and the dataset is described as the largest and most complex VQA dataset.
6. Experiments
Experiments analyze DMN variants on textual and visual question answering, finding that the combined DMN+ changes achieve the strongest overall performance while revealing task-specific strengths and limitations.
- Model analysis: DMN+ combines an input fusion layer, attention-based GRU, untied episodic-memory weights, and a ReLU memory update.DMN2 adds input fusion, DMN3 adds the attention-based GRU, and DMN+ further unties weights and uses a ReLU memory update.
- Model analysis: A large accuracy improvement on both bAbI-10k and DAQUAR results from replacing the original input module with input fusion.The fusion layer improves interaction between distant textual facts and provides neighboring-patch context for visual questions.
- Model analysis: The attention GRU helps with complex positional or ordering information, especially in textual question answering.Few visual questions are expected to require this form of logical reasoning.
- Model analysis: DMN+ achieves the highest performance across both visual and textual datasets, although its untied model overfits on some tasks while reducing average error.The reported comparison includes bAbI-10k textual and DAQUAR visual question answering.
- Text QA results: DMN+ reduces mean bAbI-10k error by 1.4% compared with the end-to-end memory network.The comparison includes E2E and the neural reasoner, neither of which uses supporting-fact supervision.
- Text QA results: A notable deficiency occurs on QA16 Basic Induction, where the complex memory update may prevent convergence on simpler tasks.The model shows the same difficulty associated with ReLU-based memory updates in the compared end-to-end memory network.
- Visual QA results: On VQA, the method outperforms baseline and state-of-the-art methods across all question domains in both test-dev and test-standard.The margin is especially wide for Other questions, where small image patches support finely detailed reasoning.
7. Conclusion
The paper introduces DMN+ modules that support question answering without supporting-fact supervision across textual and visual inputs. The model achieves state-of-the-art results on bAbI-10k and VQA, demonstrating generalization across input domains.
- Conclusion: DMN+ combines input fusion and attention-based GRU improvements to achieve strong question-answering results without supporting-fact supervision.The proposed framework is applied across textual and visual input domains.
- Conclusion: The final model obtains state-of-the-art results on both the VQA dataset and the bAbI-10k text question-answering dataset.The conclusion presents these results as evidence that the framework can be generalized across input domains.