Source-linked AI summary
Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering
Gao Peng, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven Hoi, Xiaogang Wang, Hongsheng Li
TL;DR
VQA requires effective fusion of visual and language features, while existing approaches largely emphasize inter-modality relations. DFAF alternates inter- and dynamically conditioned intra-modality attention flows to fuse these features, achieving state-of-the-art VQA performance with extensive ablation analysis.
Problem
Most existing VQA approaches focus on inter-modality relations, leaving complementary within-modality relations insufficiently addressed for multimodal feature fusion.
Method
DFAF alternates InterMAF and DyIntraMAF modules to pass information across modalities and dynamically conditioned information within each modality.
Results
DFAF achieves state-of-the-art VQA performance, while combining intra- and inter-modality attention flows outperforms the baseline by 2.83% and BAN-1 by 0.85%.
Takeaways & Limitations
Stacking multiple DFAF blocks improves VQA performance, and question-conditioned intra-modality flows filter unrelated information within each modality.
Takeaways & Limitations
The visual feature parameters θRCNN are fixed, while question features θGRU are learned from scratch and updated during training.
Abstract
from arXiv · showhide
Learning effective fusion of multi-modality features is at the heart of visual question answering. We propose a novel method of dynamically fusing multi-modal features with intra- and inter-modality information flow, which alternatively pass dynamic information between and across the visual and language modalities. It can robustly capture the high-level interactions between language and vision domains, thus significantly improves the performance of visual question answering. We also show that the proposed dynamic intra-modality attention flow conditioned on the other modality can dynamically modulate the intra-modality attention of the target modality, which is vital for multimodality feature fusion. Experimental evaluations on the VQA 2.0 dataset show that the proposed method achieves state-of-the-art VQA performance. Extensive ablation studies are carried out for the comprehensive analysis of the proposed method.
1. Introduction
Visual question answering depends on effective multimodal feature fusion, but existing approaches largely emphasize inter-modality relations. DFAF addresses this gap by alternating inter- and dynamically conditioned intra-modality attention flows, with experiments and ablations reporting state-of-the-art VQA performance.
- Motivation: Existing VQA methods improve representations, fusion, and cross-modal interactions, but most focus on relations between visual and language features.Bilinear fusion and co-attention capture higher-order or word-region relations, while intra-modality relations have received less emphasis.
- Proposed approach: InterMAF forms joint-modality coattention and updates each image region and word using attention-weighted information from the other modality.Visual regions select question features and question words select visual features according to the learned coattention matrix.
- Proposed approach: DyIntraMAF generates within-modality attention flows dynamically conditioned on the other modality's information.This module is presented as a core novelty for modeling intra-modality relations during multimodal fusion.
- Evaluation: Stacking multiple DFAF blocks can further improve VQA performance while gradually focusing on important regions, words, and latent alignments.The figure description associates this progressive focus with stacked DFAF blocks.
- Proposed approach: DFAF interleaves intra- and inter-modality feature fusion in a unified framework for visual question answering.Its modules combine InterMAF for cross-modal information exchange with DyIntraMAF for within-modality attention flow.
- Evaluation: Extensive experiments and ablation studies report state-of-the-art VQA performance for DFAF.The study examines the effectiveness of the framework and its components.
2. Related Work
Related VQA work advances feature representations, multimodal fusion, and attention mechanisms. These approaches include bilinear and approximate fusion, self- and co-attention, and dynamically predicted or structured attention mechanisms.
- Representation learning for VQA: Deep representation learning substantially improved VQA, with ResNet outperforming VGG and bottom-up top-down features becoming suitable for VQA and image captioning.Feature learning is described as an essential component of VQA algorithm development.
- Bilinear Fusion for VQA: Bilinear fusion captures high-level visual-language interactions, while MCB, MLB, and MUTAN approximate bilinear pooling to reduce computational cost.These approximated methods were proposed to address bilinear pooling's high computational cost.
- Self-attention-based methods: Self-attention methods selectively focus on important features by automatically ignoring irrelevant information.The approach is motivated mainly by self-attention and co-attention methods.
- Co-attention-based methods: Co-attention methods model interactions across vision and language by aggregating image-region features to each word according to co-attention weights.Dense Symmetric Co-attention achieved state-of-the-art performance on VQAv1 and VQAv2 without bottom-up and top-down features.
- Other works for language and vision tasks: Other multimodal methods use dynamically predicted parameters, question-guided hybrid convolution, adaptive attention, structured attention, or deformable neighbors.These methods target feature fusion, selective attention, spatial attention distributions, or offset and modulation-weight prediction.
3. Dynamic Fusion with Intra- and Inter-modality Attention Flow for VQA
DFAF alternates cross-modal and dynamically conditioned within-modal attention flows to fuse visual regions and question words. Its intra-modal attention is conditioned on the other modality, enabling question-dependent region relations and word-to-word relations.
- Overview: DFAF alternates InterMAF and DyIntraMAF modules to iteratively pass information between and within visual and language modalities.The modules can be stacked multiple times for further VQA improvement.
- Input Features: The visual input uses bottom-up and top-down Faster R-CNN region features, while question words are encoded by a GRU from GLoVe embeddings.The pipeline extracts 100 region proposals per image and represents questions at a fixed length of 14 words.
- Inter-modality Attention Flow: InterMAF learns region-word importance weights and aggregates value features bidirectionally to update image regions and question words.Row-wise softmax-normalized attention matrices weight information transmitted across modalities.
- Dynamic Intra-modality Attention Flow: DyIntraMAF models region-to-region and word-to-word relations after cross-modal encoding, updating each modality through weighted within-modal value flows.This complements cross-modal relations, such as linking the region above a skateboard with the skateboard region.
- Dynamic Intra-modality Attention Flow: Unlike naive IntraMAF, DyIntraMAF estimates within-modality importance conditioned on the other modality, so the same image can yield different region relations for different questions.The method specifically conditions query and key features, rather than value features, to adaptively weight intra-modal information flows.
- Dynamic Intra-modality Attention Flow: Average-pooled features from each modality are transformed into sigmoid channel-wise gates that modulate the other modality’s query and key features.Element-wise gating can activate or deactivate channels before within-modality attention is computed.
4. Experiments
Experiments use VQA 2.0 with ablations covering architecture depth, attention flow, fusion, positional features, sentinel features, and attention heads. Dynamic intra-modality flow and combined intra-/inter-modality flow improve performance, while some added features hurt or provide limited gains.
- Experimental setup: VQA 2.0 experiments use specified visual and language feature extraction, training splits, optimization settings, and validation-based ablations.Visual features come from Faster R-CNN and language features from GRU; ablations use validation data, while test-dev combines train, validation, and extra Visual Genome data.
- DFAF architecture: More stacked DFAF blocks improve performance, and a single-layer DFAF is comparable with BAN-12.The authors attribute the improvement from additional stacks to residual connections without normalization.
- Attention-flow ablations: 2.15% performance improvement follows adding dynamically conditioned DyIntraMAF, while combined intra- and inter-modality flows outperform the baseline by 2.83% and BAN-1 by 0.85%.Adding InterMAF alone improves performance by 1%, whereas adding only IntraMAF harms performance because unrelated information flows hinder learning.
- Attention-flow ablations: Sequential InterMAF updates outperform parallel updates, while the specific sequential order is evaluated in both region-to-word and word-to-region directions.The passage defines parallel updates as simultaneous feature updates and sequential updates as passing attention flow between modalities in order.
- Fusion and dimensions: 512-dimensional embeddings outperform 1024-dimensional embeddings, and feature multiplication performs slightly better than addition or concatenation.The final fusion layer uses feature multiplication in the default setting.
- Additional components: Visual sentinel and absolute or relative position features reduce performance, whereas eight attention heads perform better than one or four at the same parameter count.The visual sentinel does not improve performance, and positional features produce a performance drop.
- Attention-weight visualization: DyIntraMAF attention weights vary with input questions and filter unrelated information flow, unlike IntraMAF weights that treat different questions equally.The visualization shows flow from contextual regions to a central region, with question-conditioned weights used to generate the correct answer.
- State-of-the-art comparison: Table 2 compares the proposed algorithm trained with extra Visual Genome data against state-of-the-art VQA methods.The comparison includes Bottom Up, MFH, DCN, counting methods, BAN, and models using BERT contextualized word embeddings.
5. Conclusions
The paper concludes that DFAF alternates information flow within and across modalities for visual question answering. Stacking DFAF blocks is shown to improve VQA performance.
- DFAF is a framework for visual question answering that alternates information flow within and across modalities.
- Its inter-modality and intra-modality attention mechanisms pass information between and inside the visual and language modalities.
- The information flow inside visual features is dynamically conditioned on question features.
- Stacking multiple DFAF blocks is shown to improve VQA performance.