Source-linked AI summary
Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
Zhou Yu, Jun Yu, Jianping Fan, Dacheng Tao
TL;DR
VQA requires fine-grained understanding of images and questions, while linear fusion can be insufficient and bilinear methods can be computationally expensive. The paper develops MFB pooling and co-attention to jointly model multimodal features and attentions. The resulting model achieves new state-of-the-art performance on the real-world VQA dataset.
Problem
VQA requires fine-grained understanding of image and question semantics, while linear fusion may inadequately capture their complex associations and bilinear pooling can be computationally costly.
Method
The paper combines Multi-modal Factorized Bilinear pooling with a co-attention architecture that jointly learns image and question attentions in an end-to-end VQA network.
Results
The MFB model significantly improves VQA performance over existing bilinear pooling methods, and the MFB with co-attention model achieves new state-of-the-art performance on the real-world VQA dataset.
Takeaways & Limitations
MFB pooling and co-attention provide a unified approach for multimodal fusion and fine-grained image-question representation in VQA.
Takeaways & Limitations
The approach can fail when question attention neglects key words or when visual contents remain unrecognized or misclassified.
Abstract
from arXiv · showhide
Visual question answering (VQA) is challenging because it requires a simultaneous understanding of both the visual content of images and the textual content of questions. The approaches used to represent the images and questions in a fine-grained manner and questions and to fuse these multi-modal features play key roles in performance. Bilinear pooling based models have been shown to outperform traditional linear models for VQA, but their high-dimensional representations and high computational complexity may seriously limit their applicability in practice. For multi-modal feature fusion, here we develop a Multi-modal Factorized Bilinear (MFB) pooling approach to efficiently and effectively combine multi-modal features, which results in superior performance for VQA compared with other bilinear pooling approaches. For fine-grained image and question representation, we develop a co-attention mechanism using an end-to-end deep network architecture to jointly learn both the image and question attentions. Combining the proposed MFB approach with co-attention learning in a new network architecture provides a unified model for VQA. Our experimental results demonstrate that the single MFB with co-attention model achieves new state-of-the-art performance on the real-world VQA dataset. Code available at https://github.com/yuzcccc/mfb.
1. Introduction
VQA requires fine-grained understanding of images and questions, while existing fusion and representation strategies face expressiveness, efficiency, and attention limitations. The paper proposes MFB pooling with co-attention to address these challenges.
- VQA requires fine-grained image-question understanding and complex reasoning to predict the best-matching answer.
- Linear fusion methods may inadequately capture complex associations between visual and textual features because multimodal feature distributions can differ substantially.
- Bilinear pooling captures richer feature interactions but can produce high-dimensional representations with many parameters and high computational cost.
- Global image features can include question-irrelevant noise, while existing attention methods often overlook noise in the question itself.
- MFB pooling fuses visual and textual features, and its co-attention architecture jointly learns image and question attentions.
- The MFB with co-attention model achieves state-of-the-art performance on the VQA dataset.
2. Related Work
Related VQA methods range from coarse joint embeddings to attention-based and external-knowledge models. The paper focuses on multimodal fusion, where linear methods may be insufficient and bilinear methods trade expressive interactions against computational cost.
- VQA approaches include coarse joint embeddings, fine-grained attention-based joint embeddings, and external-knowledge models.
- Coarse joint-embedding models use global image and question features, but noisy global representations can hinder fine-grained question answering.
- External knowledge can benefit questions requiring complex reasoning or common sense, although existing approaches have dataset-specific or benchmark-effectiveness limitations.
- Linear fusion may have insufficient representation capacity because visual and textual feature distributions can vary substantially.
- MCB uses an outer product to create a high-dimensional feature and approximates its computation to reduce cost.
- MLB uses low-rank projections and a Hadamard product to obtain lower-dimensional multimodal features with fewer parameters.
- The MLB with attention model reportedly requires 250k iterations, or about 140 epochs, to converge.
3. Multi-modal Factorized Bilinear Pooling
MFB factorizes bilinear interactions between visual and textual features to retain expressive cross-modal representations while reducing the burden of full bilinear pooling. It expands both modalities, combines them element-wise, and uses sum pooling and normalization to produce compact outputs.
- Bilinear pooling captures pairwise interactions between visual and textual feature dimensions but requires many parameters, increasing computational cost and over-fitting risk.
- MFB factorizes each bilinear projection matrix into two low-rank matrices for efficient cross-modal interaction modeling.The factorized matrices use latent dimensionality k and Hadamard multiplication of projected features.
- MFB reshapes the factorized parameter tensors into two-dimensional matrices and computes the output with element-wise multiplication followed by non-overlapping one-dimensional sum pooling.The SumPooling window size is k, producing the compact output feature z.
- The MFB module can be implemented with fully connected, element-wise multiplication, pooling, dropout, power-normalization, and ℓ2-normalization layers.Dropout follows element-wise multiplication, while power and ℓ2 normalization follow the MFB output.
- MFB includes MLB as the special case k = 1 and has greater representation capacity than MLB at the same output dimensionality.MFB first expands features into a high-dimensional space before pooling them into a compact representation, whereas MLB directly projects to the low-dimensional output space.
4. Network Architectures for VQA
The VQA architectures extract image and question representations, fuse them with MFB, and classify answers. The co-attention variant additionally learns image and question attentions to capture fine-grained correlations.
- 4. Network Architectures for VQA: The VQA model extracts image and question representations, fuses them with MFB, and predicts answers through an N-way classifier.The classifier is trained with KL-divergence loss, while ResNet weights remain fixed because of GPU-memory limitations.
- 4. Network Architectures for VQA: The baseline architecture uses one MFB module for fused image-question features and supports ablation analysis against other approaches.
- 4. Network Architectures for VQA: Image attention evaluates 196 spatial image grids for question relevance before the attentional image features are merged with question features for prediction.
- 4.2. MFB with Co-Attention: The co-attention architecture first represents images and questions with fine-grained features, then jointly models question and image attention.This differs from the MFB baseline by adding jointly modeled attention modules.
- 4.2. MFB with Co-Attention: The co-attention model adds question attention after the LSTM and learns word-level attention weights without using image features in that module.The image and question modules are loosely coupled under this design.
5. Experiments
Experiments evaluate MFB on the VQA dataset, comparing bilinear variants, normalization choices, attention mechanisms, external embeddings, additional training data, and model ensembles. MFB consistently outperforms competing approaches, while co-attention and external resources further improve results.
- Experimental setup: The experiments evaluate MFB on the VQA dataset against MCB, MLB, and other state-of-the-art approaches using Open-Ended and Multiple-Choice tasks.The dataset includes train, validation, test, and test-dev splits; most settings are evaluated on test-dev because test-standard submissions are limited.
- Ablation analysis: MFB(k = 5, o = 1000) improves accuracy by about 1% over MCB with 5/6 of its parameters, while MFB(k = 5, o = 200) achieves similar results with one-third the parameters.MFB is also described as relatively robust, whereas MCB validation accuracy suffers from overfitting with high-dimensional output features.
- Ablation analysis: Increasing k from 1 to 5 yields a 0.5% performance gain, while increasing k to 10 approaches saturation; increasing o does not further improve performance when k is fixed.The experiments select k = 5 and o = 1000 for subsequent MFB models.
- Ablation analysis: Power normalization improves performance by about 0.5%, while ℓ2 normalization improves it by about 3%.The standard MFB configuration with both normalizations produces the most stable neuron distribution during training.
- Comparison with state-of-the-art: MFB outperforms comparative approaches without attention on both OE and MC tasks, and MFB+Att consistently outperforms the next-best MCB+Att model.These results are reported for single-model comparisons on the VQA dataset.
- Comparison with state-of-the-art: Co-attention improves overall OE accuracy by 0.5% over MFB+Att by jointly learning question and image attention.Adding pre-trained GloVe embeddings and Visual Genome training data further improves performance; the resulting MFB+CoAtt+GloVe+VG model surpasses the best reported single-model results on both OE and MC tasks.
- Comparison with state-of-the-art: An ensemble of MFB models outperforms the next-best approach by 1.5% on OE and 2.2% on MC.The ensemble averages prediction scores from seven MFB+CoAtt+GloVe models, four additionally trained with Visual Genome.
- Qualitative analysis: Incorrect examples reveal errors from neglected question keywords and unrecognized or misclassified visual content, including counting-related mistakes.The authors identify these weaknesses as potentially common to VQA approaches and suggest they can guide future improvements.
6. Conclusions
The paper develops MFB pooling for VQA feature fusion and combines it with co-attention learning in a unified architecture. This model achieves new state-of-the-art performance on the real-world VQA dataset and may apply to other multimodal tasks.
- MFB pooling fuses multimodal features for the VQA task.
- A co-attention architecture built on MFB jointly addresses multimodal representation and attention learning.
- The combined MFB with co-attention model achieves new state-of-the-art performance on a real-world VQA dataset.
- The proposed multimodal pooling and co-attention approaches are applicable to a wide range of multimodal tasks.