Source-linked AI summary
Are Multimodal Transformers Robust to Missing Modality?
Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, Xi Peng
TL;DR
Real-world multimodal data often lack modalities, but whether Transformers are naturally robust to this condition has been rarely studied. The paper evaluates this robustness, develops multi-task optimization with automatic fusion-strategy search, and finds sensitivity to missing modalities alongside dataset-dependent optimal fusion strategies. Experiments on three benchmarks support the proposed method’s robustness.
Problem
Whether multimodal Transformers are naturally robust to missing modalities has been rarely investigated, despite real-world multimodal data often being incomplete.
Method
The paper jointly optimizes Transformer models on modal-complete and modal-incomplete data and automatically searches for dataset-specific fusion strategies.
Results
Transformer models are sensitive to missing modalities, while optimal fusion strategies depend on the dataset; experiments on three benchmarks support the proposed method’s superior robustness.
Takeaways & Limitations
There is no universal fusion strategy for robust multimodal Transformers under missing modalities, so fusion should be selected with respect to the input data or dataset.
Takeaways & Limitations
For a bimodal dataset, the proposed multi-task formulation considers full-modal, image-only, and text-only tasks.
Abstract
from arXiv · showhide
Multimodal data collected from the real world are often imperfect due to missing modalities. Therefore multimodal models that are robust against modal-incomplete data are highly preferred. Recently, Transformer models have shown great success in processing multimodal data. However, existing work has been limited to either architecture designs or pre-training strategies; whether Transformer models are naturally robust against missing-modal data has rarely been investigated. In this paper, we present the first-of-its-kind work to comprehensively investigate the behavior of Transformers in the presence of modal-incomplete data. Unsurprising, we find Transformer models are sensitive to missing modalities while different modal fusion strategies will significantly affect the robustness. What surprised us is that the optimal fusion strategy is dataset dependent even for the same Transformer model; there does not exist a universal strategy that works in general cases. Based on these findings, we propose a principle method to improve the robustness of Transformer models by automatically searching for an optimal fusion strategy regarding input data. Experimental validations on three benchmarks support the superior performance of the proposed method.
1. Introduction
This paper investigates whether multimodal Transformers remain robust when modalities are missing, finding substantial degradation and dataset-dependent effects of fusion strategy. It proposes multi-task optimization and automatic fusion-strategy search, supported by experiments on three benchmarks.
- Motivation: Multimodal Transformers generally require complete modalities, although real-world privacy and healthcare constraints can make modality-complete data unavailable.Examples include unavailable location information and missing medical records.
- Research gap: Robustness to missing modalities has been seldom investigated, while prior Transformer research emphasized fusion architectures, self-supervised tasks, or noisy inputs.
- Findings: Transformer performance drops dramatically with missing modalities, and multimodal performance can fall below unimodal performance when only 30% of text remains.
- Findings: Different fusion strategies significantly affect robustness, with early fusion preferred on MM-IMDb and Hateful Memes, but late fusion preferred on UPMC Food-101.
- Method: The paper proposes jointly optimizing models on modal-complete and modal-incomplete data through multi-task optimization and automatically searching for the best fusion strategy.The search for optimal fusion layers and network training is formulated as a bilevel optimization problem.
- Evaluation: Experiments and ablation studies on MM-IMDb, UPMC Food-101, and Hateful Memes validate the method against missing modality.
2. Related Work
Related work covers multimodal fusion and robustness to incomplete modalities, emphasizing that conventional explicit-fusion backbones depend on modality completeness. Transformer models instead process modalities simultaneously with a single model, while the paper also relates to dynamic neural networks.
- Multimodal learning: Multimodal learning exploits complementary information across natural language, visual, and vocal modalities, with fusion methods including concatenation and tensor fusion.
- Incomplete modalities: Existing fusion mechanisms depend heavily on complete modalities, motivating methods for robustness to modality-incomplete data.
- Conventional backbones: Conventional backbones explicitly fuse modality-specific features, so missing any modality prevents or constrains multimodal fusion.
- Transformer backbones: Transformer models process all modalities simultaneously using a single model, reducing the training load compared with independently trained modality-specific sub-models.
- Dynamic neural networks: The related work also connects this approach to dynamic neural networks, which adapt network structure to different inputs.
3. Analysis of Multimodal Transformer
The paper examines multimodal Transformers built on ViT, measuring robustness by performance differences between complete and incomplete-modal tests. Experiments show severe degradation, dataset-dependent modality importance and fusion choices, and overfitting to dominant modalities.
- Background: Multimodal Transformers use ViT layers with multi-head attention, multilayer perceptrons, and layer normalization to process token sequences.The attention layer computes dot-product similarities between tokens.
- Background: The final multimodal input sequence concatenates vision and text embeddings after modality-specific projection, positional encoding, and modality-type embeddings.Images are patchified, while text is mapped through word and position embedding codebooks.
- Robustness Against Missing Modality: 43.6%, 28.3%, and 14.2% performance drops occur when only 30% of text is observed across three datasets.Under severe missingness, multimodal performance becomes worse than unimodal performance on MM-IMDb and UPMC Food-101.
- Robustness Against Missing Modality: Text is the dominant modality on MM-IMDb and UPMC Food-101, whereas text and image are equally important on Hateful Memes.On the first two datasets, the unimodal-to-multimodal performance gaps are 10% for text and 22% for image; Hateful Memes has a gap above 20%.
- Optimal Fusion Strategy: Transformer models tend to overfit to dominant modalities, assessed through performance gaps between unimodal and multimodal testing.For MM-IMDb, text-only testing is closer to full-modal testing than image-only testing.
- Optimal Fusion Strategy: Different fusion strategies affect robustness, but the optimal strategy depends on the dataset rather than following a universal rule.Early fusion provides greater multimodal capacity at higher computation cost, while late fusion lowers cost but may limit multimodal exploitation.
4. Robust Multimodal Transformer
The method improves Transformer robustness to modal-incomplete data by jointly training on complete and incomplete modality tasks while automatically searching for a dataset-specific fusion policy. It uses masked task-specific classification tokens and differentiable policy learning to select fusion strategies.
- Improve Robustness via Multi-task Learning: Multi-task optimization jointly trains full-modal, image-only, and text-only tasks to improve robustness when testing data have missing modalities.The total loss balances the three task losses with separate hyperparameters.
- Improve Robustness via Multi-task Learning: Attention masks restrict each classification token to its corresponding modalities, producing task-specific embeddings for full-modal and unimodal predictions.The model adds three classification tokens, with masking applied so each token receives only the intended modality information.
- Search for the Optimal Fusion Strategy: Policy parameters are learned through bilevel optimization that minimizes validation loss while model weights minimize the training loss.After learning the optimal policy, the policy is fixed and the model is retrained on the whole training set.
- Search for the Optimal Fusion Strategy: The fusion search represents layerwise policies with M univariate parameters and samples a hard policy from a softmax policy using one-hot encoding with a differential trick.The reduced search space contains M policies rather than the 2^M possibilities associated with two actions at each of M layers.
- Search for the Optimal Fusion Strategy: The search method reduces the policy space and yields a differentiable, easy-to-train fusion-policy learning process.Algorithm 1 uses training and validation datasets with inner- and outer-level learning rates and iterative updates until convergence.
5. Experiments
Experiments evaluate ViLT-based Transformers on three multimodal datasets under complete and missing-text conditions. Results show strong sensitivity to missing modalities, dataset-dependent fusion preferences, and improved robustness from the proposed method.
- Datasets and setup: Experiments cover MM-IMDb, UPMC Food-101, and Hateful Memes using ViLT as the multimodal Transformer backbone.MM-IMDb is multi-label movie-genre classification, Food-101 is 101-class classification, and Hateful Memes targets hate-speech identification.
- Datasets and setup: The missing-modality evaluation trains on 100% image plus 100% text and tests with 100% image plus η% text, where smaller η means more severe missingness.When η = 0, evaluation becomes unimodal.
- Robustness to missing modality: On MM-IMDb and UPMC Food-101, baseline performance declines as text availability decreases and can fall below unimodal performance under severe missingness.At 10% text, MM-IMDb baseline F1 Macro is 23.1 versus 35.0 unimodal, while Food-101 baseline Accuracy is 58.4 versus 73.3 unimodal.
- Robustness to missing modality: At 10% observed text, the proposed method achieves MM-IMDb F1 Macro 37.3 versus 35.0 unimodal and Food-101 Accuracy 73.3 versus 71.5 unimodal.These results indicate maintained performance under severe missingness on both datasets.
- Robustness to missing modality: On Hateful Memes, multimodal performance remains above unimodal performance, and the proposed method reaches AUROC 59.6 versus the baseline’s 58.0 with 10% text.With full modalities observed, the method also outperforms the baseline by 2.3%.
- Fusion strategy analysis: Late fusion is preferred on MM-IMDb, whereas early fusion is preferred on Hateful Memes, showing that optimal fusion layers vary by dataset.The learned policy is reported as consistent with dataset characteristics.
- Ablation study: Training with missing modalities alone does not work as a baseline and performs worse than the unimodal Image-only baseline on Food-101 and Hateful Memes.The comparison uses training and testing with 100% image plus 30% text.
- Ablation study: Both multi-task learning and optimal fusion improve robustness, but multi-task learning contributes more and exceeds the optimal fusion policy by 30% when only 10% text is available.Attention masks are also important because classification tokens must avoid using information from other modalities.
6. Conclusion
The paper finds that Transformers are sensitive to missing-modal data and that no fusion strategy is universally optimal across datasets. It proposes bilevel optimization combining multi-task learning with automatic fusion-strategy search, while noting a safety-critical limitation.
- Conclusion: Transformer models are sensitive to missing-modal data, and the optimal fusion strategy is dataset-dependent rather than universal.These findings are based on experiments across multiple benchmark datasets.
- Conclusion: The proposed robust Transformer jointly uses modal-complete and modal-incomplete data through multi-task optimization and automatically searches fusion strategies for different datasets.Fusion-layer search and network training are formulated as a bilevel optimization problem.
- Conclusion: Experiments across multiple benchmark datasets verify the superior robustness of the proposed method.
- Conclusion: Multi-task learning only ensures that multimodal performance is not worse than unimodal performance, which may be insufficient for safety-critical systems such as autonomous driving.The authors identify generative methods for reconstructing missing tokens as future work.