Source-linked AI summary
Attention Bottlenecks for Multimodal Fusion
Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, Chen Sun
TL;DR
Audiovisual models commonly rely on late fusion despite the differing structure and information content of audio and visual inputs. The paper introduces MBT, which mediates cross-modal exchange through a small set of bottleneck latents at multiple layers. MBT improves fusion at lower computational cost and achieves state-of-the-art results across several audiovisual benchmarks.
Problem
Audiovisual fusion is challenging because modalities have different learning dynamics, noise structures, information content, and input representations, while late fusion remains a common strategy.
Method
MBT uses small fusion bottleneck tokens so cross-modal attention passes through condensed modality-specific representations rather than unrestricted pairwise attention.
Results
MBT achieves state-of-the-art video classification results on AudioSet, Epic-Kitchens100, and VGGSound, including a 5.9 mAP improvement on AudioSet.
Takeaways & Limitations
Bottleneck-mediated audiovisual fusion improves performance over vanilla cross-attention while using less computation.
Abstract
from arXiv · showhide
Humans perceive the world by concurrently processing and fusing high-dimensional inputs from multiple modalities such as vision and audio. Machine perception models, in stark contrast, are typically modality-specific and optimised for unimodal benchmarks, and hence late-stage fusion of final representations or predictions from each modality (`late-fusion') is still a dominant paradigm for multimodal video classification. Instead, we introduce a novel transformer based architecture that uses `fusion bottlenecks' for modality fusion at multiple layers. Compared to traditional pairwise self-attention, our model forces information between different modalities to pass through a small number of bottleneck latents, requiring the model to collate and condense the most relevant information in each modality and only share what is necessary. We find that such a strategy improves fusion performance, at the same time reducing computational cost. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple audio-visual classification benchmarks including Audioset, Epic-Kitchens and VGGSound. All code and models will be released.
1 Introduction
The paper addresses challenges in unified audiovisual learning by proposing multimodal transformer fusion strategies that restrict cross-modal information flow. Its MBT architecture uses bottlenecks to condense modality-specific information, reducing computation while improving benchmark performance.
- Motivation: Audiovisual fusion is difficult because modalities differ in learning dynamics, noise structure, information content, and input representations.Audio often uses log-mel spectrograms, whereas video contains spatial and temporal visual structure.
- Approach: Figure 1 contrasts late fusion, later-layer mid fusion, bottleneck fusion, and their combination, Bottleneck Mid Fusion.Bottleneck Mid Fusion applies both layer restriction and bottleneck-mediated information exchange.
- Motivation: Early fusion permits unrestricted pairwise attention between visual and auditory tokens across all model layers.This allows attention across spatial, temporal, frequency, and time dimensions, but the paper hypothesizes that full attention is unnecessary.
- Approach: MBT restricts cross-modal information flow through small fusion bottlenecks that condense relevant modality-specific inputs and avoid quadratic full-attention cost.The architecture is designed to share only necessary information between modalities.
- Results: MBT achieves state-of-the-art video classification results on AudioSet, Epic-Kitchens100, and VGGSound.On AudioSet, it exceeds the previous state of the art by 5.9 mAP, a 12.7% relative improvement.
2 Related work
Prior audiovisual learning work progressed from simple early- and late-stage fusion toward learned modality-specific or joint representations. The paper situates its transformer-based approach within this broader development of supervised and unsupervised multimodal learning.
- Audiovisual learning: Early audiovisual systems used simple feature stacking, while late-stage methods combined modality predictions or scores.These approaches were motivated partly by limited data and computational resources.
- Audiovisual learning: Deep learning enabled modality-specific or joint latent representations to mediate fusion in downstream supervised audiovisual tasks.Examples include jointly trained convolutional networks whose intermediate activations are combined by summation or lateral connections.
3 Multimodal fusion transformers
The paper develops a multimodal transformer framework that compares token-fusion strategies and restricts cross-modal interaction through fusion bottlenecks. It also varies the depth at which modalities begin to interact, spanning early, mid, and late fusion.
- Fusion via vanilla self-attention: Vanilla self-attention permits every RGB token to attend to all RGB and spectrogram tokens throughout the multimodal encoder.The model concatenates modality-specific token sequences and applies standard transformer layers with unrestricted attention flow.
- Fusion with modality-specific parameters: Modality-specific cross-transformers retain separate parameters for RGB and spectrogram processing while exchanging information through generalized cross-attention.When the modality-specific parameters are equal, this computation becomes equivalent to the shared-parameter formulation.
- Fusion via attention bottlenecks: The proposed Multimodal Bottleneck Transformer introduces a small set of fusion tokens that mediate all cross-modal attention.RGB and spectrogram tokens are concatenated with bottleneck tokens, and modality-specific representations can exchange information only through them.
- Fusion via attention bottlenecks: Bottleneck fusion updates temporary modality-specific bottleneck tokens separately, then averages the resulting cross-modal updates into final fusion tokens.The bottleneck tokens are updated simultaneously with audio and visual information, and asymmetric updates were also tested.
- Where to fuse: early, mid and late: Fusion depth is controlled by applying unimodal self-attention for L_f layers before multimodal fusion in the remaining L − L_f layers.L_f = 0 denotes early fusion, L_f = L denotes late fusion, and intermediate values define mid-fusion.
4 Experiments
Experiments evaluate MBT across datasets, fusion strategies, sampling choices, data sizes, and attention behavior. MBT uses bottleneck fusion to improve multimodal performance while reducing compute, outperforming single-modality and prior fusion methods across several benchmarks.
- Datasets and evaluation: MBT is evaluated on AudioSet, Epic-Kitchens-100, and VGGSound using multimodal video-classification protocols.AudioSet evaluation reports mean average precision, while Epic-Kitchens reports Top-1 and Top-5 accuracy.
- Experimental setup: Experiments compare vanilla self-attention, separate-weight cross-attention, and bottleneck fusion as alternative within-layer token fusion strategies.Bottleneck fusion routes all cross-modal attention through dedicated fusion latents.
- Input sampling: Increasing input span improves audio and audiovisual performance, whereas visual-only performance slightly decreases; subsequent experiments use t = 8s.Performance is largely robust to synchronous versus asynchronous modality sampling, so synchronized sampling is retained.
- Results: Multimodal fusion outperforms the stronger single-modality baseline on AudioSet, Epic-Kitchens, and VGGSound, demonstrating complementary information across modalities.The relative strength of audio and visual baselines varies with dataset annotation and content.
- Results: MBT outperforms prior fusion methods on AudioSet while training on 500K samples instead of the full 2M-sample dataset.The reported comparison is based on mean average precision.
- Attention analysis: Attention maps show that MBT focuses more narrowly on moving sound-source regions, such as piano fingertips, instrument hands, and speaking mouths.The maps are computed from output CLS tokens to RGB image inputs on AudioSet test clips.
5 Conclusion
MBT uses bottleneck-mediated cross-modal fusion to improve audiovisual performance while reducing computational complexity. The authors identify tuning and supervision scope as limitations and note broader environmental concerns from transformer compute.
- MBT restricts cross-modal attention through a small set of fusion bottlenecks, achieving state-of-the-art results at lower computational cost.The bottlenecks reduce the complexity of multimodal fusion by limiting information exchange between modalities.
- Limitations: The fusion layer is a hyperparameter that may require task- and dataset-specific tuning.
- Limitations: The study explores only fully supervised fusion, leaving self-supervised extensions for future work.
- Broader impact: Transformers are compute-heavy and may have adverse environmental effects, while bottleneck token fusion helps reduce computational complexity.
A Ablations on mini-Audioset
The ablations examine asymmetric bottleneck updates on AudioSet-mini and report that performance remains robust across update-order variations.
- Ablations generally use AudioSet-mini for training and AudioSet for evaluation, with most experiments averaged over 3 runs.Reported results include mean and standard deviation for most experiments.
- Asymmetric bottleneck updates: Asymmetric updates process bottlenecks first with visual information and then with audio information.The authors also compare the reverse audio-first ordering with the symmetric update.
- Asymmetric bottleneck updates: Performance is robust to visual-first, audio-first, and symmetric bottleneck-update variations.
- Asymmetric bottleneck updates: Table 4 compares asymmetric and symmetric bottleneck updates.
A.2 Backbone architecture
Backbone size, encoder-weight sharing, and modality sampling affect audiovisual fusion differently. Performance peaks at ViT-Base, while later fusion and sampling choices show greater robustness.
- Backbone size: Performance increases from ViT-Small to ViT-Base but drops for ViT-Large on AudioSet-mini and VGGSound.The authors suggest that the smaller datasets may require more data to benefit from larger models.
- Table 5 reports performance for varying backbones on AS-mini and VGGSound, while Table 6 reports the effect of varying input clip span t.
- Encoder weight sharing: Separate encoder weights improve performance for earlier fusion layers, whereas shared and separate encoders perform similarly with later fusion.
- Modality sampling: Performance is largely robust to synchronous versus asynchronous modality sampling across multiple input span lengths.Asynchronous sampling introduces misalignment but may provide a trade-off involving independently sampled modality windows.
- Modality sampling: The AudioSet segments used in the numerical input-span experiments are 10 seconds long.
B Per class performance
Audio-visual fusion improves per-class average precision for nearly all of the examined top-ranked AudioSet classes, with especially large gains for some classes.
- 57 out of 60 top-ranked AudioSet classes improve with audio-visual fusion over audio-only or visual-only baselines.The exceptions are bagpiping, emergency vehicle, and didgeridoo, which have strong audio signatures.
- Fusion improves absolute AP over audio-only by more than 60% for classes such as bicycle and shuffling cards.
- Whistling gains almost 80% absolute AP over the visual-only baseline, the largest visual-baseline improvement reported.
C Additional Datasets
This section reports results on two additional video-classification datasets: Moments in Time and Kinetics.
- Results are reported for Moments in Time and Kinetics.
- Moments in Time is one of the additional datasets evaluated in this section.
- Kinetics is the other additional dataset evaluated in this section.
C.1 Moments In Time
Moments in Time contains short, diverse YouTube clips labeled with verb classes, and MBT provides a modest improvement over the visual-only baseline.
- Moments in Time contains 800,000 three-second YouTube clips labeled with 330 verb classes.Each class is associated with over 1,000 videos.
- Audio-only performance is much lower than visual-only performance on Moments in Time.The passage attributes this largely to the annotation procedure and the clips’ short duration.
- MBT provides a further modest 1% boost over the visual-only baseline.
C.2 Kinetics
On Kinetics, MBT modestly outperforms visual-only classification, with a larger advantage on the audio-oriented Kinetics-Sound subset.
- Kinetics contains 10-second YouTube videos, evaluated on Kinetics 400 and the 36-class Kinetics-Sound subset.
- MBT outperforms the visual-only baseline by about 1% in top 1 accuracy on the entire Kinetics test set.
- The fusion advantage exceeds 4% on Kinetics-Sound, whose classes were selected for strong audio signatures.
D Dataset Variations for MBT vs Late Fusion
MBT’s advantage over late fusion varies by dataset, is especially large on Epic-Kitchens, and depends on pretraining choices.
- MBT’s gap over late fusion is highly dataset dependent across the evaluated video-classification datasets.
- MBT provides an almost 6% advantage over late fusion on Epic-Kitchens in Top 1 action accuracy.
- Transfer learning: VGGSound pretraining gives a substantial 3% mAP boost over ImageNet initialization on AudioSet-mini.