PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Modular Co-Attention Networks for Visual Question Answering

Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, Qi Tian

arXiv:1906.10770v1cs.CV

TL;DR

VQA requires fine-grained alignment between question words and image regions, while existing deep co-attention models had shown limited improvement. MCAN cascades modular layers combining self-attention and guided-attention, achieving state-of-the-art VQA-v2 performance and 70.63% test-dev accuracy for its best single model.

  • Problem

    VQA needs fine-grained joint understanding of images and questions, but existing co-attention models have limited region-word interactions and deep models show little improvement over shallow counterparts.

  • Method

    MCAN cascades Modular Co-Attention layers that combine self-attention for intra-modal interactions with guided-attention for inter-modal interactions.

  • Results

    MCAN achieves new state-of-the-art performance on VQA-v2, with its best single model reaching 70.63% overall accuracy on the test-dev set.

  • Takeaways & Limitations

    The results support jointly modeling self-attention and guided-attention, while image-region self-attention can improve object-counting performance.

  • Takeaways & Limitations

    MCAN occasionally misidentifies question keywords, including spatial or role-related words such as “left” and “catcher.”

Abstract

from arXiv · show

Visual Question Answering (VQA) requires a fine-grained and simultaneous understanding of both the visual content of images and the textual content of questions. Therefore, designing an effective `co-attention' model to associate key words in questions with key objects in images is central to VQA performance. So far, most successful attempts at co-attention learning have been achieved by using shallow models, and deep co-attention models show little improvement over their shallow counterparts. In this paper, we propose a deep Modular Co-Attention Network (MCAN) that consists of Modular Co-Attention (MCA) layers cascaded in depth. Each MCA layer models the self-attention of questions and images, as well as the guided-attention of images jointly using a modular composition of two basic attention units. We quantitatively and qualitatively evaluate MCAN on the benchmark VQA-v2 dataset and conduct extensive ablation studies to explore the reasons behind MCAN's effectiveness. Experimental results demonstrate that MCAN significantly outperforms the previous state-of-the-art. Our best single model delivers 70.63$\%$ overall accuracy on the test-dev set. Code is available at https://github.com/MILVLG/mcan-vqa.

1. Introduction

VQA requires fine-grained understanding of images and questions, making co-attention central to associating question words with image regions. MCAN addresses limitations of existing deep co-attention by jointly modeling dense intra-modal and inter-modal interactions.

  • VQA requires fine-grained semantic understanding of both images and questions, together with visual reasoning to predict accurate answers.
  • Co-attention models learn coarse multimodal interactions and cannot infer correlations between individual image regions and question words.
  • Dense co-attention models capture region-word interactions, but deep versions show little improvement over shallow or coarse co-attention models.
  • MCAN combines self-attention for word-to-word and region-to-region interactions with guided-attention for word-to-region interactions.
  • MCAN cascades Modular Co-Attention layers in depth and significantly outperforms existing state-of-the-art co-attention models on VQA-v2.

2. Related Work

Earlier VQA methods used global multimodal features or question representations, but global image representations can lose information about local regions. Visual attention was introduced to focus on image features relevant to a question before multimodal fusion.

  • Early VQA approaches fused global image and question features to predict answers.
  • Some methods improved question representations with LSTM networks or multimodal fusion with residual networks.
  • Global image features may lose critical local-region information needed to answer questions about specific image areas.
  • Visual attention adaptively learns image features for a given question before multimodal feature fusion.

3. Modular Co-Attention Layer

The MCA layer combines self-attention and guided-attention units to model intra- and inter-modal relationships in VQA. Different compositions support progressively richer image-question representations and can be cascaded in depth.

  • Attention units: MCA layers are modular compositions of self-attention and guided-attention units for processing multimodal VQA features.The self-attention unit models within-modality relationships, while guided-attention models cross-modal relationships.
  • Attention units: Multi-head attention computes attended features from queries, keys, and values by combining outputs from parallel attention heads.Each head uses an independent scaled dot-product attention function, and the head outputs are combined through an output projection.
  • Attention units: The SA unit models pairwise relationships among features within one input group, producing attended outputs through weighted aggregation of that group.For input X, each feature is reconstructed from samples in X according to normalized similarities.
  • Deep composition: All three MCA layers can be cascaded in depth, passing each layer’s outputs to the next without reducing the number of input features.The paper also explores alternative symmetric compositions, but does not report comparative performance for them.
  • MCA variants: The three illustrated MCA variants progressively add question self-attention and image self-attention to guided image attention.They are ID(Y)-GA(X,Y), SA(Y)-GA(X,Y), and SA(Y)-SGA(X,Y), with the latter modeling intra-modal interactions among image regions as well.

4. Modular Co-Attention Networks

MCAN builds VQA models by cascading MCA layers to refine attended image and question features before multimodal fusion and classification. It compares stacking and encoder-decoder strategies for deep co-attention learning.

  • Deep co-attention learning: MCANsk-L and MCANed-L denote the stacking and encoder-decoder models, with L specifying the total number of cascaded MCA layers.The two models have equal size for the same L and are equivalent when L=1.
  • Deep co-attention learning: MCAN cascades L MCA layers to recursively transform image features X^(l−1) and question features Y^(l−1) into X^l and Y^l.For the first layer, X^(0)=X and Y^(0)=Y.
  • Deep co-attention learning: The stacking model feeds each layer’s outputs into the next and uses X^(L) and Y^(L) as the final attended features.This strategy simply stacks L MCA layers in depth.
  • Deep co-attention learning: The encoder-decoder model uses L stacked self-attention units to learn Y^(L), then uses that representation to guide image-feature learning.It modifies stacking by replacing each guided-attention layer’s question input with Y^(L).
  • Multimodal fusion and output classifier: After co-attention, two-layer MLP attentional reduction models produce attended image or question features from X^(L) or Y^(L).The learned attention weights are used to obtain the attended feature for each modality.
  • Multimodal fusion and output classifier: The attended image and question features are linearly projected and fused, then passed to an N-way sigmoid classifier trained with binary cross-entropy.Layer normalization is used in the fusion function to stabilize training.

5. Experiments

Experiments on VQA-v2 evaluate MCAN variants, depth, representations, attention behavior, and state-of-the-art comparisons through quantitative and qualitative ablations.

  • Experimental Setup: MCAN experiments use VQA-v2, with overall and per-type accuracies reported for Yes/No, Number, and Other answers.The dataset includes train, validation, and test splits, while reported ablations are evaluated on the validation split.
  • MCA Variants: Self-attention for question features improves all answer types, while adding image self-attention further improves performance and becomes the default MCA variant.The default is SA(Y)-SGA(X,Y), which includes self-attention for both question and image features.
  • Depth and Architecture: Both deep co-attention models improve as layers increase and saturate at L = 6, while encoder-decoder models outperform stacking, especially at larger depths.The authors attribute saturation beyond six layers to unstable gradients and explain the encoder-decoder advantage through improved late-layer question self-attention.
  • Depth and Architecture: MCANed-2 reaches 66.2% accuracy with 27M parameters, compared with BAN-4 at 65.8% with 45M and MFH at 65.7% with 116M.These results support the reported parameter efficiency of MCAN relative to the compared approaches.
  • Question Representations: GloVe-pretrained word embeddings significantly outperform random initialization, while fine-tuning and alternative temporal modeling provide smaller additional gains.The ablation compares random initialization, positional encoding, GloVe initialization, and fine-tuned GloVe embeddings.
  • Qualitative Analysis: Later attention layers identify answer-relevant question words and image objects more clearly, with image self-attention highlighting the three sheep for counting.The visualizations also show more noise in stacking guided-attention maps than in encoder-decoder maps.
  • Qualitative Analysis: Correct predictions usually focus attention on relevant words and image regions, but the model occasionally misidentifies key question words such as “left” and “catcher.”These observed errors are presented as guidance for future improvements.
  • Comparison with State-of-the-Art: MCANed-6 outperforms BAN by 1.1 points in overall accuracy and remains 0.6 points above BAN+Counter using the same bottom-up visual features.Its object-counting performance is comparable to BAN+Counter without using auxiliary bounding-box coordinates.

6. Conclusions

The paper presents MCAN, a deep VQA model that cascades modular co-attention layers combining self-attention and guided-attention. Using an encoder-decoder strategy, MCAN achieves new state-of-the-art VQA performance.

  • MCAN cascades modular co-attention layers whose self-attention and guided-attention units model intra- and inter-modal interactions synergistically.

A. Model Ensembling

MCAN surpasses leading VQA-v2 leaderboard solutions using four MCANed-6 models with slightly varied hyperparameters, while relying on basic bottom-up attention features and fewer ensemble models.

  • Model Ensembling: MCAN surpasses the top solutions on the VQA-v2 leaderboard with an ensemble of four MCANed-6 models.The models use slightly different hyperparameters for ensembling.
  • Model Ensembling: The ensemble comparison uses accuracies on the VQA-Challenge 2018 test-standard split.Table 3 reports model-ensemble accuracies and identifies each team’s rank and number of models.

B. Comparisons of Model Stability and Computational Costs

MCANed-6 outperforms MFH and BAN-8 in accuracy and stability while requiring fewer parameters and less computation on the VQA-v2 validation split.

  • Comparisons of Model Stability and Computational Costs: MCANed-6 outperforms MFH and BAN-8 in overall accuracy and stability on the VQA-v2 validation split.Stability is assessed using accuracy ± standard deviation across three independently initialized models.
  • Comparisons of Model Stability and Computational Costs: MCANed-6 is more parameter- and computationally efficient than MFH and BAN-8.The comparison evaluates parameter counts and FLOPs, with FLOPs calculated for one testing sample.

C. More Visualized Results

Additional visualizations show MCAN attention maps across attention units, layers, and parallel heads, illustrating its reasoning process and complementary head information.

  • More Visualized Results: Figure 9 visualizes MCANed-6 attentions from three units across the first and sixth layers for two additional examples.The units are SA(X), SA(Y), and GA(X,Y), with two parallel heads shown for each unit.
  • More Visualized Results: Different attention heads provide complementary information that benefits VQA performance.The visualized attentions are reported to help explain MCAN’s reasoning process for predicting correct answers.
Loading 1906.10770v1…