Source-linked AI summary
MUTAN: Multimodal Tucker Fusion for Visual Question Answering
Hedi Ben-younes, Rémi Cadene, Matthieu Cord, Nicolas Thome
TL;DR
VQA bilinear fusion can model precise image-question interactions but suffers from prohibitive dimensionality. MUTAN uses Tucker and low-rank decompositions to retain expressive bilinear fusion while controlling complexity, and reports state-of-the-art VQA performance.
Problem
Bilinear VQA models capture rich image-question interactions, but fully parameterized tensors can require approximately 10^10 parameters, hindering learning and GPU memory use.
Method
MUTAN factorizes the interaction tensor with a Tucker decomposition and structures its core with constraints that control model complexity while preserving bilinear interactions.
Results
MUTAN outperforms previous methods with a large margin on test-dev and test-std and reaches state-of-the-art results on the VQA dataset.
Takeaways & Limitations
MUTAN provides controllable, interpretable fusion relations and can complement MLB, with its fusion benefit translating to the overall VQA task.
Takeaways & Limitations
MLB's operating point requires 1000–2000 projection dimensions in the reported experimental setup.
Abstract
from arXiv · showhide
Bilinear models provide an appealing framework for mixing and merging information in Visual Question Answering (VQA) tasks. They help to learn high level associations between question meaning and visual concepts in the image, but they suffer from huge dimensionality issues. We introduce MUTAN, a multimodal tensor-based Tucker decomposition to efficiently parametrize bilinear interactions between visual and textual representations. Additionally to the Tucker framework, we design a low-rank matrix-based decomposition to explicitly constrain the interaction rank. With MUTAN, we control the complexity of the merging scheme while keeping nice interpretable fusion relations. We show how our MUTAN model generalizes some of the latest VQA architectures, providing state-of-the-art results.
1. Introduction
VQA requires precise interactions between image and question representations, but bilinear fusion models face prohibitive parameter growth. MUTAN addresses this with a Tucker-based fusion scheme that preserves rich interactions while controlling complexity.
- Context: The paper positions MUTAN within a broader VQA research landscape enabled by benchmarks and large-scale multimodal datasets.These resources support the development of more powerful VQA models.
- Motivation: VQA models must encode high-level correlations between visual and textual spaces while focusing on question-relevant image regions.The task requires understanding the full scene, selecting relevant visual regions, and discarding irrelevant information.
- Motivation: Bilinear models capture full second-order interactions but become intractable as input and output dimensions increase.Existing approaches simplify or approximate bilinear fusion to reduce model complexity.
- MUTAN: MUTAN introduces a Tucker decomposition of the image-question correlation tensor to model fine and rich multimodal interactions.The architecture is presented as a new fusion strategy for VQA.
- MUTAN: The Tucker-based scheme maintains tractable model size while explicitly controlling complexity and preserving interpretable parameter organization.The decomposition supports an accurate and interpretable repartition of learnable parameters.
2. Related work
Related VQA work progresses from simple modality merging to bilinear interactions and attention-based fusion. MUTAN extends this line with a Tucker decomposition, structured sparsity, and reported gains over established bilinear methods.
- Multimodal VQA: VQA merges image and question modalities to select an answer, requiring precise correlations rather than explicit alignment between feature spaces.Attention mechanisms help identify relevant visual regions and textual features.
- Bilinear fusion: Second-order and bilinear methods provide richer interactions than element-wise products, with MCB using an outer product and MLB constraining a bilinear tensor to low rank.MLB effectively projects both modalities into a shared r-dimensional space before element-wise interaction.
- MUTAN: MUTAN introduces a Tucker tensor decomposition into three matrices and a core tensor, generalizing MCB and MLB with greater expressive power.The decomposition is proposed as a new VQA fusion scheme.
- MUTAN: Structured sparsity on the core tensor acts as a training regularizer, prevents overfitting, and provides flexibility in input and output projections.This constraint further controls the number of model parameters.
- Results: MUTAN achieves state-of-the-art results on the widely used VQA dataset and outperforms MCB and MLB in the same setting.Combining MUTAN with MLB further improves performance, supporting complementarity between the approaches.
- MUTAN: Figure 2 depicts bilinear prediction through tensor T, its Tucker factors Wq, Wv, and Wo, and the smaller constrained core tensor Tc.The figure summarizes how MUTAN controls fusion complexity structurally.
3. MUTAN Model
MUTAN uses Tucker decomposition and slice-wise low-rank structure to model bilinear interactions between question and image representations while controlling parameter complexity. Its flexible factorization generalizes existing bilinear architectures and provides an interpretable fusion scheme.
- Fusion and Bilinear models: Fully parameterized bilinear fusion encodes rich q–v interactions but becomes intractable, requiring approximately 10^10 tensor parameters for common VQA dimensions.With dv ≈ dq ≈ 2048 and |A| ≈ 2000, the tensor size also creates learning and GPU-memory problems.
- 3.2. Multimodal Tucker Fusion: MUTAN factorizes the full interaction tensor with Tucker decomposition, replacing the prohibitive tensor with learned factor matrices and a core tensor.The factor matrices project question, image, and output spaces into latent dimensions, while the core models interactions among projected modalities.
- Interpretation: Projection dimensions tq and tv control the permitted modeling complexity for the question and image modalities, while the core tensor models their interactions.Larger projected dimensions allow more complex modality-specific modeling.
- 3.3. Tensor sparsity: MUTAN constrains each core-tensor slice to rank R, expressing the fused output as a sum of R components formed by projected elementwise products.This structured decomposition balances interaction expressivity and complexity and can be interpreted as multiple AND gates combined by an OR operation.
- 3.4. Model Unification and Discussion: MLB and MCB are special Tucker cases with fixed structural constraints, whereas MUTAN learns Wq, Wv, Wo, and Tc and relaxes those constraints.MLB fixes equal projection dimensions and an identity core; MCB fixes diagonal input factors and a sparse core, learning only Wo.
- 3.4.2 Multimodal Low-rank Bilinear (MLB): MUTAN’s slice-wise core constraint differs from MLB’s global tensor-rank constraint by preserving distinct projected dimensions and avoiding reduction along the output mode.Core parameters are correlated within each mode-3 slice but independent across slices.
4. Experiments
The experiments evaluate MUTAN’s fusion scheme against bilinear and other VQA models, then analyze its complexity, rank sparsity, and latent projections. MUTAN achieves strong comparative performance while supporting controlled, interpretable multimodal interactions.
- Experimental setup: The VQA experiments use MSCOCO-based image-question pairs, ResNet152 visual features, and GRU question representations.The dataset contains 248,349 training pairs, 121,512 validation pairs, and 244,302 test pairs; images produce 14×14×2048 feature maps.
- 4.1. Fusion Scheme Comparison: MUTAN is compared with concatenation, MCB, and MLB under the same visual and linguistic features without attention.The comparison uses output dimensions of 16,000 for MCB and 1,200 for MLB; MUTAN variants use Tucker projection dimensions and, for the full model, rank R = 10.
- 4.1. Fusion Scheme Comparison: Structured sparsity gives MUTAN the best fusion results, while late-fusion MUTAN+MLB improves test-dev performance by about +1pt.The result supports a tradeoff between projection dimensions and the number of useful bilinear interaction parameters, and indicates complementarity between the two decompositions.
- 4.2. State-of-the-art comparison: MUTAN outperforms previous methods by a large margin on both test-dev and test-std in the enhanced VQA comparison.The improvement is reported as consistent with the fusion-performance gap observed without attention, and MUTAN (3) also exceeds prior state-of-the-art results using three models.
- 4.3. Further analysis: MUTAN noR outperforms an identity-tensor fusion even with very small core dimensions, showing that learned core correlations matter.The analysis varies equal projection dimensions t from 20 to 220 while comparing against an identity tensor with the same projection dimension.
- 4.3. Further analysis: Ablation results show that different latent projections contribute differently across question types, while attention projections can focus on distinct relevant objects.Some yes/no questions are nearly solvable by individual ranks, whereas questions such as “What is the man” require information from all latent projections; separate projections can attend to elephants, women, smoke, or trains.
5. Conclusion
MUTAN fuses visual and textual information through bilinear modeling, combining Tucker decomposition with a low-rank matrix constraint to control interaction complexity.
- MUTAN combines a Tucker decomposition with a low-rank matrix constraint for multimodal bilinear fusion.
- The model factorizes the interaction tensor into interpretable elements and allows control over its expressiveness.
- MUTAN generalizes competitive VQA architectures within the Tucker decomposition framework.
- MUTAN reaches state-of-the-art results on the VQA dataset.
Preprocessing details
The experiments preprocess images with fixed resizing, cropping, normalization, and ResNet-152 features, while restricting and tokenizing questions for answer prediction.
- Images are resized while preserving proportion, center-cropped to 448 × 448, ImageNet-normalized, and processed with pretrained ResNet-152.Features are extracted before the final Rectified Linear Unit (ReLU).
- Questions are restricted to those associated with the 2000 most occurring answers.
- Question text is lowercased, stripped of punctuation, split into words, and mapped to a pretrained Skip-thoughts vocabulary with unknown words replaced by UNK.
Optimization details
Optimization uses Adam without learning-rate decay, with batch sizes adjusted for attention models and early stopping based on validation accuracy or test-dev evaluation.
- Adam without learning-rate decay uses a learning rate of 10^-4 and was found more convenient while leading to the same accuracy as tested alternatives.
- Models without attention use batch size 512, whereas attention models use batch size 100 because they consume more memory.
- Early stopping is used as a regularizer by saving parameters after each epoch and selecting the best epoch by open-ended validation accuracy when available.
- For trainval models, test-dev serves as validation, with 3 to 4 submissions per model for epochs having training accuracies between 63% and 70%.
Ensemble details
The ensembles combine MUTAN and, in one case, MLB models trained with varied glimpse counts and data augmentation, then average their pre-Softmax features.
- MUTAN(3) combines three MUTAN models using 2, 3, and 2 glimpses, with the third trained using visual genome data augmentation.
- MUTAN(5) adds two MLB models to the three MUTAN models used in MUTAN(3).
- The ensemble's final results are obtained by averaging features extracted before each model's final Softmax layer.
Scores details
The section reports answer-type scores across validation and test splits, then compares fusion methods and MUTAN performance under specified evaluation setups.
- Scores are reported for each answer type on the val and test-dev splits in Table 3 and on test-dev and test-standard in Table 4.
- Table 3 compares different fusion methods under the same setup on the test-dev split.Θ denotes the number of learnable parameters, measured in millions.
- Table 4 compares MUTAN performance on the test-dev and test-standard VQA dataset splits.The notation (n) indicates an ensemble of n models.