Source-linked AI summary
BLOCK: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Relationship Detection
Hedi Ben-younes, Rémi Cadene, Nicolas Thome, Matthieu Cord
TL;DR
Bilinear multimodal fusion can capture subtle cross-modal interactions but becomes difficult to implement because its parameter count grows quadratically with input dimensions. The paper introduces BLOCK, a learnable block-term fusion model that controls this complexity while preserving rich interactions and high-dimensional monomodal representations. Across VQA and VRD experiments, BLOCK improves over prior fusion schemes and remains competitive with compact parametrization.
Problem
Bilinear fusion captures modality interactions, but its quadratically growing parameter count makes full-tensor learning impractical for high-dimensional inputs.
Method
BLOCK structures a learnable bilinear fusion tensor with block-term decomposition and uses block-term ranks to control the tradeoff between modeling capacity and complexity.
Results
BLOCK improves over previous linear, bilinear, and nonlinear fusion schemes on VQA and VRD while maintaining competitive performance with compact parametrization.
Takeaways & Limitations
BLOCK provides a practical fusion design that models rich feature-group interactions while retaining high-dimensional monomodal representations.
Takeaways & Limitations
The reported future work limits the demonstrated scope to extending BLOCK to multiple input and output modalities and to multimodal-model interpretation.
Abstract
from arXiv · showhide
Multimodal representation learning is gaining more and more interest within the deep learning community. While bilinear models provide an interesting framework to find subtle combination of modalities, their number of parameters grows quadratically with the input dimensions, making their practical implementation within classical deep learning pipelines challenging. In this paper, we introduce BLOCK, a new multimodal fusion based on the block-superdiagonal tensor decomposition. It leverages the notion of block-term ranks, which generalizes both concepts of rank and mode ranks for tensors, already used for multimodal fusion. It allows to define new ways for optimizing the tradeoff between the expressiveness and complexity of the fusion model, and is able to represent very fine interactions between modalities while maintaining powerful mono-modal representations. We demonstrate the practical interest of our fusion model by using BLOCK for two challenging tasks: Visual Question Answering (VQA) and Visual Relationship Detection (VRD), where we design end-to-end learnable architectures for representing relevant interactions between modalities. Through extensive experiments, we show that BLOCK compares favorably with respect to state-of-the-art multimodal fusion models for both VQA and VRD tasks. Our code is available at https://github.com/Cadene/block.bootstrap.pytorch.
1 Introduction
Many AI tasks require fusion across modalities, but bilinear models become impractical as input dimensions grow because their parameter count is quadratic. BLOCK addresses this challenge with a learnable block-term fusion framework that balances expressive interactions and model complexity, validated on VQA and VRD.
- Multimodal fusion supports tasks including visual question answering, visual relationship detection, cross-modal retrieval, and social-media post classification.
- Bilinear models represent interactions between two modality inputs, but their tensor parameters grow quadratically with input dimensions.For high-dimensional inputs, learning a full bilinear tensor quickly becomes intractable.
- BLOCK uses a learnable block-term tensor decomposition whose block-term ranks control the tradeoff between fusion expressiveness and complexity.The decomposition combines the concepts of tensor rank and mode ranks.
- BLOCK models rich bilinear interactions between feature groups while retaining expressive high-dimensional monomodal representations.
- BLOCK is embedded in end-to-end deep architectures for VQA and VRD, with block numbers and projection sizes adapted to balance fine interaction modeling and parameter count.
- Extensive comparisons across VQA 2.0, TDIUC, and VRD show competitive results against state-of-the-art multimodal fusion techniques.
2 BLOCK fusion model
BLOCK constrains bilinear fusion with a block-term tensor decomposition, combining block count and within-block mode ranks to balance complexity, expressivity, and interaction richness. It projects each modality, fuses corresponding chunks through block tensors, concatenates the results, and produces the prediction vector.
- BLOCK decomposition: BLOCK expresses the interaction tensor with a block-term decomposition to reduce and control the model’s complexity.The decomposition is used as a fully learnable tensor of parameters in an end-to-end bilinear model.
- Fusion computation: BLOCK projects inputs into expanded representations, then each block fuses chunks of sizes L and M to produce a vector of size N.The resulting block outputs are concatenated into z ∈ R^NR before the final prediction y = Cz ∈ R^K.
- Complexity control: A further constraint on the rank of each third-order block slice can reduce the model’s parameter count.The paper relates this constraint to approaches used in recent VQA applications.
- BLOCK decomposition: The block-term formulation combines the number of blocks with each block’s mode-n ranks, spanning CP and Tucker as extreme cases.CP uses multiple rank-(1,1,1) blocks, whereas Tucker uses a single core block.
- Expressivity–complexity trade-off: Chunked interaction modeling permits high-dimensional projections without exploding the block-term parameter count.This is intended to provide fine interactions between high-dimensional semantic representations while retaining manageable complexity.
3 BLOCK fusion for VQA task
The VQA system embeds BLOCK fusion within an attentional architecture that combines image and question representations. Experiments compare this approach with multiple multimodal fusion schemes on VQA 2.0 and TDIUC.
- Architecture: VQA combines image and question representations through a multimodal fusion module before answer classification.The compared system uses attended image features and a question embedding within an end-to-end architecture.
- Fusion schemes: The comparison includes summation, multilayer perceptron, count-sketching, CP, Tucker, slice-rank, and cascaded fusion schemes.These alternatives impose different structures or approximations on the bilinear interaction tensor.
- VQA 2.0 comparison: BLOCK gives the best results in the VQA 2.0 fusion comparison and surpasses a higher-order fusion while using 30M fewer parameters.The authors attribute this trade-off to block-term ranks and third-order slice-rank constraints.
- VQA 2.0 comparison: 64.91% is achieved by a 3M-parameter BLOCK model, which improves over 5 of 8 methods using substantially more parameters.The compact model remains competitive despite not surpassing every comparison method.
- TDIUC comparison: +1.7 accuracy, +3.95 A-MPT, +5.05 H-MPT, +16.12 A-NMPT, and +15.45 H-NMPT are reported over the best scoring TDIUC model for each metric.The harmonic and normalized metrics are used to assess question-type coverage and robustness to answer imbalance.
- VQA 2.0 comparison: +1.78 overall accuracy is obtained over Yu et al. on VQA 2.0 test-dev, with gains of +2.26 on test-dev and +2.25 on test-std over Teney et al.The comparison uses the same global VQA architecture as those baselines and the same image features as Teney et al.
4 VRD task
Visual Relationship Detection predicts subject-predicate-object triplets by fusing spatial, class, and visual information from subject and object regions. BLOCK is evaluated across VRD settings and achieves strong results while balancing interaction modeling and parameterization.
- VRD predicts subject-predicate-object triplets linking localized subject and object regions through a relationship label.
- 4.1 VRD Architecture: The architecture represents subject and object boxes using spatial coordinates, object classes, and semantic visual features.
- 4.1 VRD Architecture: Separate fusion modules process each feature type, concatenate their outputs, and feed a linear predictor trained with binary cross-entropy.Each fusion module outputs a d-dimensional vector, producing a 3d-dimensional predictor input.
- 4.2 Fusion comparison: BLOCK outperforms previous fusion methods on all three VRD tasks, while a BLOCK fusion also improves over a strong nonlinear MLP baseline.
- 4.3 Fusion analysis: R ≈20 yields the best validation performance when either the core dimensions or total parameter count is controlled.With R = 1, the core tensor contains 125M parameters; with R = 500, interactions reduce to element-wise multiplication.
- 4.4 Comparison to leading VRD methods: On VRD test results, BLOCK leads prior methods on R@50 for predicate, phrase, and relationship detection, with narrower differences at R@100.For predicate prediction, it is marginally outperformed at R@100 by a method using external data but exceeds methods without extra data.
5 Conclusion
BLOCK structures bilinear fusion with a block-term decomposition to balance modeling capacity and complexity. Experiments across VQA and VRD show improved or competitive performance, while future work targets broader modalities and interpretability.
- BLOCK structures its bilinear parameter tensor with a block-term decomposition to optimize the trade-off between complexity and modeling capacity.
- The fusion models rich interactions between feature groups while retaining high-dimensional monomodal representations.
- BLOCK improves over linear, bilinear, and nonlinear fusion schemes on VQA and VRD.
- BLOCK maintains competitive performance with very compact parametrization.
- Future work will extend BLOCK to multiple input and output modalities and investigate multimodal-model interpretation and explanation.