Source-linked AI summary
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, Marcus Rohrbach
TL;DR
Existing multimodal pooling methods may be less expressive than outer-product interactions, while direct bilinear pooling is computationally infeasible at high dimensions. The paper proposes MCB to efficiently combine visual and textual features, evaluates it on VQA and visual grounding, and reports improvements over non-MCB ablations and prior state of the art.
Problem
Concatenation and element-wise multimodal pooling may not capture complex cross-modal interactions, while direct bilinear pooling requires infeasibly high-dimensional representations.
Method
MCB approximates bilinear pooling using Count Sketch projections and FFT-based convolution, applying it within architectures for VQA and visual grounding.
Results
MCB consistently improves over ablations without MCB across VQA and visual grounding, with attention-based MCB improving Visual7W multiple-choice accuracy by 7.9 points overall.
Takeaways & Limitations
MCB provides a more expressive multimodal interaction mechanism that improves question answering and phrase localization accuracy within the evaluated tasks and datasets.
Takeaways & Limitations
The bilinear representation remains impractical without compression; with n1 = n2 = 2048 and z ∈ R3000, the corresponding VQA matrix would require 12.5 billion parameters.
Abstract
from arXiv · showhide
Modeling textual or visual information with vector representations trained from large language or visual datasets has been successfully explored in recent years. However, tasks such as visual question answering require combining these vector representations with each other. Approaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations. We hypothesize that these methods are not as expressive as an outer product of the visual and textual vectors. As the outer product is typically infeasible due to its high dimensionality, we instead propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and expressively combine multimodal features. We extensively evaluate MCB on the visual question answering and grounding tasks. We consistently show the benefit of MCB over ablations without MCB. For visual question answering, we present an architecture which uses MCB twice, once for predicting attention over spatial features and again to combine the attended representation with the question representation. This model outperforms the state-of-the-art on the Visual7W dataset and the VQA challenge.
1 Introduction
The paper argues that concatenation and element-wise multimodal pooling may not capture complex cross-modal associations, and proposes MCB as an expressive yet efficient alternative. MCB is evaluated across VQA and visual grounding, with VQA architectures using it for attention and answer prediction.
- Concatenation and element-wise sum or product are common ways to combine visual and textual representations.
- These operations may be insufficiently expressive for capturing complex associations between modalities.
- MCB approximates bilinear pooling to preserve multiplicative interactions between all elements while avoiding the outer product's high dimensionality.Bilinear pooling computes an outer product, but MCB uses Count Sketch projections and FFT-based convolution for efficient approximation.
- The paper evaluates MCB on visual question answering and visual grounding across two tasks, four datasets, and diverse ablations and comparisons.
- For open-ended VQA, the architecture applies MCB first to spatial attention and again to combine attended visual features with the question representation.
2 Related Work
Prior multimodal systems commonly combine visual and textual representations through concatenation, element-wise operations, attention, or learned joint embeddings. Bilinear pooling offers richer interactions, but its computational cost motivates compact approximations such as MCB.
- VQA models have used concatenation, element-wise products or sums, attention mechanisms, and dynamically predicted interactions to merge modalities.
- Visual grounding approaches concatenate phrase embeddings with visual features to predict attention or obtain segmentation across spatial locations.
- Bilinear pooling combines vectors through an outer product and has been applied to fine-grained visual recognition.
- Prior work reduces bilinear feature cost by viewing bilinear transformation as a polynomial kernel and approximating it with Count Sketches and convolutions.
- Joint multimodal embedding methods include canonical correlation analysis, ranking-loss linear models, and nonlinear deep learning models.
3 Multimodal Compact Bilinear Pooling for Visual and Textual Embeddings
MCB combines visual and textual embeddings through a compact approximation to bilinear pooling, preserving multiplicative interactions while avoiding the infeasible dimensionality of the full outer product. The paper applies MCB in VQA and visual grounding architectures, including attention and answer encoding.
- 3.1 Multimodal Compact Bilinear Pooling (MCB): MCB represents the relationship between image and question embeddings by approximating bilinear pooling without explicitly computing the outer product.Bilinear pooling permits multiplicative interactions between all elements, but the full representation can require billions of parameters.
- 3.1 Multimodal Compact Bilinear Pooling (MCB): 12.5 billion parameters would be required by the full bilinear model for VQA with n1 = n2 = 2048 and z ∈ R3000.This motivates projecting the outer product into a lower-dimensional space.
- 3.1 Multimodal Compact Bilinear Pooling (MCB): MCB projects each input with Count Sketch and computes their convolution efficiently as an element-wise product in FFT space.The resulting compact representation reduces parameters while avoiding direct outer-product computation.
- 3.2 Architectures for VQA: The VQA model pools image and question representations with MCB and treats answering as multi-class classification over 3,000 possible classes.The architecture extracts image features with a pretrained ResNet and question features with a two-layer LSTM.
- 3.2 Architectures for VQA: For attention, MCB merges each spatial visual feature with the language representation, after which convolutional layers predict normalized soft-attention weights.The attended visual representation is formed as a weighted sum of spatial vectors.
- 3.2 Architectures for VQA: For multiple-choice VQA, answer choices are encoded with shared word-embedding and LSTM weights, then merged with the original multimodal representation using an additional MCB.The resulting embedding is projected to a classification vector whose dimension equals the number of answers.
- 3.3 Architecture for Visual Grounding: For visual grounding, the model replaces GroundeR’s concatenation of visual and phrase representations with MCB to predict a bounding box for a query phrase.The approach uses a query phrase, an image, and multiple proposal bounding boxes.
4 Evaluation on Visual Question Answering
The evaluation tests MCB through ablations, dimensionality and attention studies, and comparisons on Visual7W and VQA. MCB consistently outperforms non-bilinear alternatives and achieves strong results against prior state of the art.
- Ablation Results: MCB outperforms element-wise sum, concatenation, and element-wise product in multimodal pooling comparisons.
- Ablation Results: 59.83% accuracy for MCB exceeds 57.10% for Concatenation + FC + FC despite comparable parameter budgets.The compared models have approximately 48 million parameters available in MCB and approximately 46 million in the concatenation-based model.
- Ablation Results: Attending to the MCB layer improves performance by 2.67 points, whereas attending to the Concatenation + FC layer matches the no-attention result.
- Ablation Results: A 16,000-D compact bilinear feature yields the highest accuracy among the evaluated dimensionalities.One, two, and four attention maps achieve 64.67%, 65.08%, and 64.24% accuracy, respectively, on train+val.
- Comparison to State-of-the-Art: The MCB-with-attention model outperforms the previous Visual7W state of the art by 7.9 points overall and performs better in almost every category.
- Comparison to State-of-the-Art: On VQA Test-dev, the ensemble is 1.8 points above the next best open-ended approach and 0.8 points above the next best multiple-choice approach.Without ensembles, the MCB + Genome + Att. + GloVe model reaches 65.4% versus 64.9% for the next best open-ended result.
5 Evaluation on Visual Grounding
MCB improves visual grounding over concatenation and element-wise pooling on Flickr30k Entities and ReferItGame, while using fewer parameters than the compared baseline. Grounding accuracy is measured by correct localization at more than 50% intersection over union.
- Datasets: The evaluation uses Flickr30k Entities with 31K images and 244K localized phrases, plus ReferItGame with 20K images and 120K referring expressions.The experiments follow established proposal and feature setups for each dataset.
- Evaluation: Grounding accuracy is the percentage of query phrases localized correctly, requiring predicted and ground-truth bounding boxes to exceed 50% intersection over union.The evaluation uses the same criterion across the visual grounding experiments.
- Results: 48.69% on Flickr30k Entities and 28.91% on ReferItGame are achieved by MCB, improving phrase localization over non-MCB baselines on both datasets.The MCB results are reported as state-of-the-art accuracy, and Figure 6 shows examples of improved phrase localization.
- Results: 47.86% and 27.98% are obtained by element-wise product with an additional 2048-D convolution on Flickr30k Entities and ReferItGame, respectively.This improves over element-wise product alone, which reaches 47.41% and 27.80%.
- Results: 46.5% on Flickr30k Entities and 25.48% on ReferItGame result from replacing MCB with concatenation of embedded visual and phrase features.These values are the reported concatenation baseline accuracies.
- Qualitative Results: Figure 6 presents VQA answers and attention maps above, and compares MCB grounding with Eltwise Product + Conv grounding below.The figure provides qualitative examples for both tasks.
6 Conclusion
The paper proposes MCB for combining visual and text representations and reports improvements on visual question answering and visual grounding. Its grounding experiments show improved phrase localization, while the VQA architecture uses attention and multiple MCBs.
- Contribution: MCB combines visual and text representations for multimodal tasks.The conclusion identifies multimodal combination as the central contribution.
- Visual Question Answering: Attention and multiple MCBs significantly improve results over state-of-the-art methods on two VQA datasets.The conclusion reports this improvement without specifying dataset-level scores here.
- Visual Grounding: MCB pooling improves phrase localization accuracy in visual grounding, indicating better interaction between query phrases and proposal-box representations.The conclusion connects the grounding improvement to interactions between language and visual proposal representations.
- Reproducibility: The authors provide code to replicate their experiments.The conclusion gives a repository link for reproduction.