PaperlayerView exact revision on arXiv

Source-linked AI summary

Improved Fusion of Visual and Language Representations by Dense Symmetric Co-Attention for Visual Question Answering

Duy-Kien Nguyen, Takayuki Okatani

arXiv:1804.00775v2cs.CV

TL;DR

VQA requires effective fusion of visual and language features, while existing attention mechanisms considered limited image-question interactions. The paper introduces a dense, symmetric co-attention architecture with stacked multi-step interactions, achieving new state-of-the-art results on VQA and VQA 2.0.

  • Problem

    Existing attention mechanisms considered only limited interactions between image regions and question words, restricting the interactions modeled for image-question reasoning.

  • Method

    The dense co-attention network symmetrically lets each question word attend to image regions and each image region attend to question words, with layers stackable for multi-step interactions.

  • Results

    The proposed architecture achieves new state-of-the-art results on VQA and VQA 2.0 datasets.

  • Takeaways & Limitations

    Dense symmetric interactions provide a basis for improved fusion of visual and language representations in VQA.

Abstract

from arXiv · show

A key solution to visual question answering (VQA) exists in how to fuse visual and language features extracted from an input image and question. We show that an attention mechanism that enables dense, bi-directional interactions between the two modalities contributes to boost accuracy of prediction of answers. Specifically, we present a simple architecture that is fully symmetric between visual and language representations, in which each question word attends on image regions and each image region attends on question words. It can be stacked to form a hierarchy for multi-step interactions between an image-question pair. We show through experiments that the proposed architecture achieves a new state-of-the-art on VQA and VQA 2.0 despite its small size. We also present qualitative evaluation, demonstrating how the proposed attention mechanism can generate reasonable attention maps on images and questions, which leads to the correct answer prediction.

1. Introduction

VQA progress has centered on attention mechanisms and visual-language feature fusion, but these directions were largely studied separately. The paper unifies them through dense, symmetric co-attention and reports state-of-the-art results on VQA and VQA 2.0.

  • VQA performance improved rapidly through advances in attention mechanisms and visual-language feature fusion.
  • Prior work developed question-guided image attention and multiple feature-fusion methods for image-question representations.
  • The paper argues that attention and feature fusion pursue the same goal, because better attention can produce better fused image-question representations.
  • Dense co-attention computes attention from each question word to image regions and from each image region to question words.The architecture is fully symmetric and considers every image-region/question-word interaction.
  • The dense co-attention layer performs attended-feature computation, multimodal concatenation, and transformation with ReLU and a residual connection.
  • DCNs achieve new state-of-the-art results on VQA 1.0 and VQA 2.0 datasets.The network combines stacked dense co-attention layers with attention in initial feature extraction and answer prediction.

2. Related Work

Related VQA research developed attention and feature-fusion methods, but existing attention mechanisms modeled only limited image-question interactions. Dense co-attention addresses this limitation by treating the modalities symmetrically and repeatedly fusing them through attention.

  • Earlier VQA methods mainly used question-guided attention on image regions, with later methods adding image-guided attention on question words.
  • Stacked, residual, and structured attention models were introduced to support sequential reasoning, improved attention information, or cross-region relations.
  • Existing attention approaches considered limited interactions, such as attending to regions from a whole question or words from a whole image.
  • The proposed mechanism considers every interaction between any image region and any question word, potentially modeling complex image-question relations.
  • Traditional feature fusion independently extracts visual and language features before combining them with concatenation, summation, element-wise products, or bilinear pooling.
  • The paper treats attention as feature fusion and applies it repeatedly so visual and language features interact symmetrically.

3. Dense Co-Attention Network (DCN)

The DCN repeatedly fuses question and image representations using stacked dense co-attention layers. Its symmetric mechanism creates attention maps for every word–region interaction before attended-feature fusion and answer prediction.

  • Architecture: The DCN stacks dense co-attention layers between feature extraction and multi-label answer prediction.Each layer updates question and image representations for the next layer.
  • Feature Extraction: Question encoding uses a randomly initialized one-layer Bi-LSTM with a residual connection, producing word-level and whole-question representations.The question words are first converted into GloVe vectors; the model also derives sQ from the final hidden states.
  • Feature Extraction: Image encoding extracts four ResNet-152 convolutional feature levels, aligns them to d × 14 × 14, and combines them using question-guided softmax weights.The resulting d × T matrix represents T = 14 × 14 image regions.
  • Dense Co-Attention: Dense co-attention creates one attention map over image regions for each question word and one over question words for each image region.The mechanism uses row-wise and column-wise normalization of an affinity matrix, with each attention-map row corresponding to one conditioning item.
  • Dense Co-Attention: Learnable nowhere-to-attend elements augment both modalities, allowing attention maps to ignore all ordinary words or regions and providing multiple memory slots.The augmented representations contain K additional elements on each side.
  • Fusion and Prediction: The model averages multiple attended features, then fuses attended image and question representations through subsequent network operations before answer scoring.The answer-prediction alternatives include an inner-product formulation and two-layer MLP variants; the MLP uses 1024 hidden units with ReLU.

4. Experiments

Experiments evaluate DCNs on VQA and VQA 2.0 through ablations, benchmark comparisons, model-size analysis, and qualitative attention visualizations. The results show benefits from symmetric co-attention and strong performance on both datasets.

  • Experimental setup: Experiments use VQA and VQA 2.0, training on train + val splits and reporting test-dev and test-standard results under the VQA evaluation protocol.VQA includes yes/no, number, and other question types, with candidate answers selected by occurrence thresholds.
  • Ablation study: Co-attention performs best among question-guided, image-guided, and bidirectional variants, supporting the effectiveness of the symmetric co-attention implementation.The single-direction variants retain the remaining dense co-attention computations while generating attention on only one modality path.
  • Ablation study: K = 3, h = 4, and L = 3 yield the best ablation performance across memory augmentation, parallel attention maps, and stacked layers.The authors associate K > 1 with external memory accessed through attention.
  • Ablation study: Attention in the answer-prediction and image-extraction layers improves accuracy by about 1.3% and 0.5%, respectively.These results come from the final blocks of the VQA 2.0 ablation table.
  • Comparison with existing methods: On VQA 1.0, DCNs outperform MF-SIG-T3 by 0.9% ∼1.1% on both test-dev and test-standard, with gains across question types.On test-standard, the reported improvements are 1.1% for Other, 3.4% for Number, and 0.6% for Yes/No.
  • Comparison with existing methods: On VQA 2.0, the approach exceeds MF-SIG-T3 by 2.1% on test-dev and improves over Adelaide+Detector by 1.5% and 1.37% on test-dev and test-standard.The comparisons also report improvements across all question types, while the Adelaide model uses annotated-region and external-question-answering data.
  • Comparison with existing methods: DCNs achieve the best VQA performance with a parameter count comparable to or smaller than former state-of-the-art methods.Model-size counts include LSTM parameters and exclude ResNet parameters.
  • Qualitative evaluation: Visualizations show attention maps over image regions and question words, with the DCN focusing on relevant content for complementary pairs and producing correct answers in the shown cases.Image brightness and word redness indicate attention weights; additional success and failure cases appear in supplementary material.

5. Conclusion

The dense co-attention network fuses visual and language representations through dense symmetric interactions and stacked multi-step image-question interactions. Experiments on VQA and VQA 2.0 confirm the architecture’s effectiveness.

  • The dense co-attention network combines a stack of dense co-attention layers with attention-based initial feature extraction and answer prediction.The dense co-attention layers repeatedly fuse language and visual features before answer prediction.
  • Dense co-attention layers enable symmetric interactions between image and question representations and can be stacked for multi-step interactions.
  • Experiments on the VQA and VQA 2.0 datasets confirm the effectiveness of the proposed architecture.

A. More Details of the Experimental Setups

The experiments preprocess images and questions with fixed resizing, tokenization, and pretrained word-vector vocabulary choices. DCNs use three dense co-attention layers and dataset-specific training schedules.

  • Images are resized to 448 × 448 before CNN processing, while questions are tokenized using the Python Natural Language Toolkit.
  • The experiments use the CommonCrawl-840B GloVe vocabulary for English word vectors and assign out-of-vocabulary words to unk.
  • DCNs use three dense co-attention layers, selected based on preliminary experiments.The layer-stack depth is L = 3.
  • The model is trained with ADAM for 16 epochs on VQA and 21 epochs on VQA 2.0, using batch sizes of 160 and 320, respectively.
  • The initial learning rate is α = 0.001, with decay every 4 epochs for VQA and 7 epochs for VQA 2.0; β1 = 0.9 and β2 = 0.99.

B. Effects of the Employment of Contextualized Word Vectors

The study compares CoVe-pretrained and randomly initialized Bi-LSTM word features in DCNs. Performance differences are small or favor random initialization, while CoVe enables meaningful representations for unseen multiple-choice answers.

  • The study compares a two-layer CoVe-pretrained Bi-LSTM with a single-layer randomly initialized Bi-LSTM for DCNs.The initial CoVe-based approach was replaced after comparative experiments.
  • On VQA 2.0 test-dev, CoVe performs slightly better with answer prediction layer (16), while random initialization performs better with layers (17) and (18) using fewer parameters.
  • The comparison is reported on the VQA 2.0 test-dev set for CoVe-pretrained and randomly initialized LSTM variants.
  • CoVe-pretrained models with answer prediction layer (16) compute meaningful answer representations for unseen answers absent from training data.
  • DCN (16) with CoVe-pretrained features is evaluated on Multiple Choice answers, which include many unseen answers.

C. Visualization of Attention Maps in the Answer Prediction Layer

The paper provides additional examples of attention maps from the DCN answer prediction layer, covering both successful and failed cases.

  • Additional answer-prediction attention-map examples are presented for both success cases and failure cases.The examples supplement those shown in Fig. 4 of the main paper.

C.1. Success Cases

The DCN generates appropriate image and question attention maps for most successful complementary image-question pairs.

  • Most successful complementary pairs show appropriately generated image and question attention maps.

C.2. Failure Cases

The DCN’s failures fall into four categories: confusing visually similar concepts, dataset bias, incorrect region localization, and answers rejected despite being conceptually correct.

  • Type-1: The DCN can locate appropriate regions and words yet fail to distinguish objects or concepts with similar appearances.Examples include mutt versus lab and spoon versus fork.
  • Type-2: Dataset bias or missing training instances can cause incorrect answers despite appropriate localization.The training set contains many american flag examples but no dragon flag examples.
  • Type-3: The DCN sometimes fails to locate appropriate image regions when distractor regions look similar or the target region is too small.
  • Type-4: Conceptually correct answers can be judged incorrect when they are absent from the dataset’s accepted answer set.For example, the model outputs beach when the given correct answer is water.
  • Failure-case visualization: Failure visualizations mark erroneous answers with red bounding boxes and correct answers with green bounding boxes.The examples contain complementary pairs with the same question and different images, and at least one erroneous prediction per row.

D. Layer Attention in the Image Feature Extraction Step

Question-guided attention over four ResNet convolutional layers weights visual features differently by question type, with higher layers generally receiving greater weight.

  • The method computes question-conditioned weighted sums of activations from four ResNet convolutional layers.The resulting layer weights are analyzed across 55 VQA-2.0 question types using means and standard deviations.
  • Layer 1 receives some weight for Yes/No questions and no weight for other question types.
  • Layer 2 receives a small weight for Yes/No questions and no weight for other question types.
  • Layer 3 tends to receive large weights for color and object-presence questions, including “what color,” “are there,” and “how many.”
  • Layer 4 has the largest attention weights for most question types, indicating its importance for answering them.
  • Specific questions tend to have smaller standard deviations than nonspecific questions.“What color” and “what sport is” are contrasted with “is the woman” and “do you.”
Loading 1804.00775v2…