PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Residual Learning for Visual QA

Jin-Hwa Kim, Sang-Woo Lee, Dong-Hyun Kwak, Min-Oh Heo, Jeonghee Kim, Jung-Woo Ha, Byoung-Tak Zhang

arXiv:1606.01455v2cs.CV

TL;DR

Visual question answering offers a setting for integrating vision and language, while applications of deep image-recognition methods to multimodality remain limited. The paper proposes Multimodal Residual Networks with element-wise joint residual mappings and back-propagation visualization, achieving state-of-the-art results on Open-Ended and Multiple-Choice VQA tasks. It also frames MRN as an implicit attention model without explicit attention parameters.

  • Problem

    Applications of deep neural network methods to multimodality remain limited, despite the need for integrated vision-language reasoning in visual question answering.

  • Method

    MRN extends deep residual learning to visual question answering using multimodal shortcuts, element-wise multiplication for joint residual mappings, and back-propagation visualization of attention effects.

  • Results

    MRN achieves state-of-the-art results on the VQA dataset for both Open-Ended and Multiple-Choice tasks.

  • Takeaways & Limitations

    MRN provides an implicit attention model without explicit attention parameters and enables visualization of spatial attention from collapsed visual features.

  • Takeaways & Limitations

    The paper identifies object detection, segmentation, and tracking as areas requiring further investigation of the visualization method.

Abstract

from arXiv · show

Deep neural networks continue to advance the state-of-the-art of image recognition tasks with various methods. However, applications of these methods to multimodality remain limited. We present Multimodal Residual Networks (MRN) for the multimodal residual learning of visual question-answering, which extends the idea of the deep residual learning. Unlike the deep residual learning, MRN effectively learns the joint representation from vision and language information. The main idea is to use element-wise multiplication for the joint residual mappings exploiting the residual learning of the attentional models in recent studies. Various alternative models introduced by multimodality are explored based on our study. We achieve the state-of-the-art results on the Visual QA dataset for both Open-Ended and Multiple-Choice tasks. Moreover, we introduce a novel method to visualize the attention effect of the joint representations for each learning block using back-propagation algorithm, even though the visual features are collapsed without spatial information.

1 Introduction

The paper extends deep residual learning to multimodal visual question answering through Multimodal Residual Networks, which learn joint vision-language representations. It also introduces back-propagation visualization of attention effects and reports state-of-the-art results on both VQA task formats.

  • Attention-based models commonly combine a query with contextual visual or textual information through residual-style updates.These models add the query to the attentional module output, allowing the module to learn a residual query mapping.
  • MRN extends deep residual learning to visual question answering by learning multimodal representations from visual and language inputs.The approach uses shortcuts and residual mappings for multimodality.
  • The proposed joint residual mappings use element-wise multiplication and explore alternative shortcut choices for each modality without explicit attention parameters.The multimodal design addresses the choice of shortcuts and joint mappings together.
  • The paper introduces back-propagation visualization to show attention effects for each joint residual mapping, even when visual features are spatially collapsed.The difference between visual input and joint-residual output is back-propagated to the input image.
  • MRN achieves state-of-the-art results on the Visual QA dataset for both Open-Ended and Multiple-Choice tasks.This result is reported as one of the paper’s three main contributions.

2 Related Works

Related work establishes deep residual learning as a framework for training very deep networks and describes stacked attention models for iterative visual selection conditioned on questions.

  • 2.1 Deep Residual Learning: Deep residual learning enables deeper neural network structures and has shown state-of-the-art results across multiple visual tasks.It adds a shortcut identity mapping so nonlinear layers learn residual mappings.
  • 2.1 Deep Residual Learning: A residual learning block adds the input shortcut to a nonlinear mapping, producing the block output.The supplied prose identifies x and y as the block input and output, respectively.
  • 2.2 Stacked Attention Networks: Stacked Attention Networks learn visual-feature weights from a question vector and corresponding visual features to select spatially relevant information.The selected features are linearly combined using weights associated with spatial indexes.
  • 2.2 Stacked Attention Networks: SAN transfers the combined visual feature vector plus the previous question vector to the next learning block for multi-step reasoning.Stacking narrows visual selection across successive attention networks.

3 Multimodal Residual Networks

MRN extends residual learning to visual question answering by carefully designing multimodal shortcuts and joint residual mappings. Its joint function uses element-wise multiplication so visual and language inputs directly contribute to the learned representation.

  • Motivation: Multimodal residual learning requires task-specific joint residual functions and shortcut mappings because visual and language modalities may have correlations.The paper notes that shortcuts are not predetermined in multimodal learning and must be selected for the task.
  • Background: SAN transfers the question vector through attention layers, but question information contributes to the joint representation mainly through attention coefficients p.The paper identifies this pathway as a possible bottleneck for learning the joint representation.
  • Joint residual mapping: MRN uses element-wise multiplication for the joint residual function while following SAN’s shortcut and stacking architecture.The design is intended to learn joint representations from vision and language without explicit attention parameters.
  • Model selection: Among the explored alternatives, model (b), with an extra visual-modality embedding, was selected for its best performance and relative simplicity.Models with an extra question embedding or identity shortcuts degraded performance relative to (b).
  • Joint residual mapping: The joint residual function combines embedded question and visual vectors, allowing both inputs to directly contribute to the joint representation.The nonlinear function uses tanh, and the visual and question inputs are combined element-wise.
  • Deep residual architecture: MRN stacks multiple learning blocks, replacing the question input with the preceding block’s output while transferring visual features through identity shortcuts.A linear mapping in the first block matches the question feature dimension to the joint dimension.

4 Experiments

The experiments evaluate MRN on the VQA dataset across Open-Ended and Multiple-Choice tasks using human-collected questions and answers. They examine alternative architectures, visual features, answer-set size, implementation choices, and test-standard comparisons.

  • Visual QA Dataset: The VQA dataset contains 614,163 questions and 7,984,119 answers, with ten human answers collected for each question.Questions are evaluated in Open-Ended and Multiple-Choice formats across yes/no, number, and other answer types.
  • Visual QA Dataset: The dataset uses 123,287 MS-COCO images for training and validation and 81,434 images for testing.The images were collected to contain multiple objects and natural situations.
  • Implementation: 37.5% of training time was reduced by using TrimZero to eliminate zero computations for variable-length questions.The efficiency depends on batch size, RNN model size, and the number of zeros in the inputs.
  • Implementation: The preprocessing uses 1k, 2k, or 3k most frequent answers, covering 86.52%, 90.45%, or 92.42% of questions, respectively.The corresponding vocabulary sizes are 14,770, 15,031, and 15,169.
  • State-of-the-art comparison: MRN significantly outperforms other state-of-the-art methods on both Open-Ended and Multiple-Choice VQA test-standard results.Number and Other answer types remain less satisfactory relative to human performance, motivating future counting mechanisms.
  • Experimental design: Alternative-model experiments compare architectural choices on test-dev results, including embeddings, shortcut mappings, and the number of learning blocks.The alternatives use three-block layers, VGG-19 features, and 1k answers before further exploration.

5 Results

MRN evaluation explores alternative architectures, depth, visual features, target-answer counts, state-of-the-art comparisons, and gradient-based visualization. The chosen model outperforms prior methods on both VQA tasks, while visualization exposes attention effects without explicit attention parameters.

  • Alternative Models: Model (b) is selected because it performs best with relative simplicity, while extra embeddings and identity shortcuts degrade performance in tested alternatives.Model (e) performs reasonably, but its extra shortcut is not essential; question-only and joint models achieve 50.39% and 57.75%, respectively.
  • Number of Learning Blocks: 60.53% is the highest overall accuracy among the tested depths, compared with 58.85% (L = 1), 59.44% (L = 2), and 60.42% (L = 4).The three-block model was selected after a pilot test and confirmed through this depth comparison.
  • Visual Features: ResNet-152 visual features significantly outperform VGG-19 features for the Other answer type despite using 2,048 rather than 4,096 dimensions.The model also achieves a large-margin improvement over previous work using ResNet features.
  • Number of Target Answers: The number of target answers slightly changes overall accuracy through trade-offs among answer types, so the authors choose Res, 2k using overall accuracy.The choice applies to the reported Table 2 configuration.
  • State-of-the-art comparisons: The chosen MRN significantly outperforms other state-of-the-art methods on both Open-Ended and Multiple-Choice VQA tasks.The comparison uses test-standard results.
  • Qualitative Analysis: Attention effects are visualized by back-propagating the gradient of Latt = 1/2∥V − F∥2 with respect to the input image while treating F as constant.The pretrained CNN is augmented only for visualization, and the method is applied to each learning block.
  • Qualitative Analysis: MRN uses no explicit attention mechanism, yet element-wise multiplication supports higher-resolution visualization of attention effects from collapsed visual features.The method interprets element-wise multiplication as information masking and avoids dependence on a few explicit attention parameters.

6 Conclusions

The paper applies deep residual learning to visual question answering and introduces MRN, which achieves state-of-the-art results on both VQA task formats. It also presents back-propagation visualization of spatial attention from collapsed visual features and identifies further applications for investigation.

  • Conclusions: MRN applies deep residual learning to visual question-answering and validates alternative models before proposing a three-block layered network.The paper reports state-of-the-art results on both Open-Ended and Multiple-Choice tasks.
  • Conclusions: The paper introduces back-propagation to visualize spatial attention from collapsed visual features.The authors position this as an implicit attention mechanism for attentional-model research.
  • Future Work: Object detection, segmentation, and tracking are identified as areas for further investigation using back-propagated attention effects.This is presented as a direction rather than a demonstrated result in the paper.

A.1 VQA test-dev Results

The appendix reports VQA test-dev results and examines both modeling options and shortcut connections in MRN.

  • Model options: Table 4 evaluates various options for VQA test-dev using the Figure 3a model.The experiments use VGG-19 features and 1k target answers.
  • Test-dev results: Table 5 presents the VQA test-dev results.Some reported accuracies are zero-filled because their precision differs from other results.
  • Shortcut connections: Table 6 examines the effects of MRN shortcut connections on VQA test-dev.The comparison uses ResNet-152 features and 2k target answers, with MN denoting the model without residual learning.

A.2 More Examples

Figure 5 provides additional examples corresponding to Figure 4 in Section 5.2.

  • More Examples: Figure 5 contains more examples related to Figure 4.It extends the examples presented in Section 5.2.
  • More Examples: The examples in Figure 5 are associated with Section 5.2.The figure is explicitly identified as supplementary to Figure 4.
  • More Examples: Figure 5 serves as an additional example figure for the analysis in Figure 4.Its role is to show more instances of the same analysis.

A.3 Comparative Analysis

Figure 6 compares MRN’s attention on the same images under different questions and across sequence positions.

  • Question-conditioned attention: MRN highlights the giraffe when the question concerns the giraffe, but focuses more on a background tree for “Can you see trees?”.The comparison uses panels (a1) and (a2) on the same image.
  • Comparative attention: Attention in panel (b2) is more dispersed across the background than in panel (b1).The passage suggests this may relate to recognizing the site.
  • Comparative attention: The figure illustrates that MRN’s visual focus changes with both the question and the sequence position.The examples compare question-specific focus and differences in the middle of sequences.

A.4 Failure Examples

Figure 7 documents failure cases involving counting, question-insensitive object attention, and limitations of the attentional mechanism.

  • Counting failures: MRN shows weaknesses on counting questions in examples (d) and (k).These are identified as failure examples in the figure caption.
  • Question-insensitive attention: The model sometimes attends to objects regardless of the question, such as a cat absent from question (j).The cat is nevertheless attended in the image.
  • Attentional limitations: Example (i) shows a limitation of the attentional mechanism requiring inference using world knowledge.The caption explicitly connects this case to world-knowledge inference.
Loading 1606.01455v2…