Source-linked AI summary
SMIL: Multimodal Learning with Severely Missing Modality
Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, Xi Peng
TL;DR
The paper addresses multimodal learning when training or testing modalities are missing, including settings where up to 90% of training examples are incomplete. It proposes SMIL, which uses Bayesian meta-learning to target flexible and efficient learning, and reports state-of-the-art performance across three benchmarks against existing and generative baselines.
Problem
Existing multimodal learning often assumes complete modalities, while few methods handle incomplete training modalities, especially when missingness is severe.
Method
SMIL uses Bayesian meta-learning to uniformly address missing modalities in training, testing, or both while learning efficiently from incomplete data.
Results
SMIL achieves state-of-the-art performance over existing methods and generative baselines including autoencoders and GANs on MM-IMDb, CMU-MOSI, and avMNIST.
Takeaways & Limitations
SMIL supports multimodal learning across varied missing-modality patterns, including severely incomplete training data.
Takeaways & Limitations
The reconstruction process models ω as multivariate random variables, introducing randomness and uncertainty.
Abstract
from arXiv · showhide
A common assumption in multimodal learning is the completeness of training data, i.e., full modalities are available in all training examples. Although there exists research endeavor in developing novel methods to tackle the incompleteness of testing data, e.g., modalities are partially missing in testing examples, few of them can handle incomplete training modalities. The problem becomes even more challenging if considering the case of severely missing, e.g., 90% training examples may have incomplete modalities. For the first time in the literature, this paper formally studies multimodal learning with missing modality in terms of flexibility (missing modalities in training, testing, or both) and efficiency (most training data have incomplete modality). Technically, we propose a new method named SMIL that leverages Bayesian meta-learning in uniformly achieving both objectives. To validate our idea, we conduct a series of experiments on three popular benchmarks: MM-IMDb, CMU-MOSI, and avMNIST. The results prove the state-of-the-art performance of SMIL over existing methods and generative baselines including autoencoders and generative adversarial networks. Our code is available at https://github.com/mengmenm/SMIL.
Introduction
This section frames multimodal learning with severely missing modality as a gap in existing work and introduces SMIL, a Bayesian meta-learning approach targeting flexibility and efficiency.
- Motivation: Existing multimodal methods commonly assume full and paired modalities during both training and testing, an assumption challenged by privacy and budget limitations.Examples include restricted access to full-modality social-network data and the lower affordability of LiDARs than cameras.
- Research gap: Prior work addresses missing modalities mainly at testing time, leaving incomplete training modalities comparatively underexplored.The resulting question is whether a model trained on incomplete data can approach the performance of one trained with full modalities.
- Problem formulation: SMIL studies missing-modality ratios up to 90% and seeks flexibility across missing training, testing, or both, alongside efficient learning from incomplete data.Flexibility covers three missing patterns, while efficiency emphasizes learning from incomplete modality as fast as possible.
- Method: SMIL uses Bayesian meta-learning and perturbs latent features so single-modality embeddings can approximate full-modality embeddings.The paper contrasts this design with generative approaches that often require substantial full-modality data.
- Evaluation: Experiments on MM-IMDb, CMU-MOSI, and avMNIST report state-of-the-art SMIL performance over existing methods and generative baselines including AE and GAN.The contribution statement presents these experiments as validation of SMIL’s performance.
Related Work
Related work covers multimodal fusion, missing-modality learning, meta-regularization, and multimodal generative modeling, positioning SMIL at their intersection.
- Multimodal learning: Multimodal fusion research combines modality representations through feature concatenation, product operations, or modality-specific low-rank factors.These approaches differ in how they create interactions among modalities during fusion.
- Missing modalities: Recent missing-modality studies address testing-time omissions or unpaired training data, whereas SMIL considers missing modalities in both training and testing.The paper identifies this joint setting as more challenging and novel.
- Meta-regularization: Meta-learning research includes metric-learning and probabilistic approaches, with optimization-based methods adapted to domain generalization and knowledge distillation.SMIL extends MAML with auxiliary networks for missing-modality reconstruction and feature regularization.
- Multimodal generative models: Multimodal generative models include cross-modal generation methods and joint-model generation approaches that learn distributions over multimodal data.The related work distinguishes these two broad categories.
Proposed Method
SMIL addresses multimodal learning when training, testing, or both contain missing modalities, including settings where up to 90% of training samples are incomplete. It combines feature reconstruction, latent-feature regularization, and Bayesian meta-learning to learn efficiently from incomplete data.
- Objectives: SMIL targets flexibility across missing modalities in training, testing, or both, and efficiency when most training data are incomplete.The formulation uses modality-complete and modality-incomplete datasets and explicitly defines these two objectives.
- Feature Reconstruction: A feature reconstruction network approximates missing-modality features from available modalities, producing complete latent-space data for unified learning.Rather than conventional data reconstruction, it generates an efficient approximation of the missing-modality feature.
- Feature Reconstruction: Under severe missingness, SMIL predicts weights over modality priors instead of directly generating the missing modality.The priors are learned from modality-complete data, and the weights are modeled as multivariate Gaussian variables with variances predicted from the observed modality.
- Feature Regularization: A feature regularization network samples Gaussian regularizers and applies them to latent features, with Softplus activation weakening regularization to support convergence.The regularizer is generated from the previous layer's features and combined with the current layer through addition or multiplication.
- Bayesian Meta-Learning: A modified MAML framework meta-trains the main network on incomplete data, meta-tests it on complete data, and updates all network parameters by gradient descent.The framework jointly integrates the main, reconstruction, and regularization networks.
- Bayesian Meta-Learning: Bayesian meta-learning approximates the intractable posterior with an amortized distribution and maximizes the resulting lower bound using Monte-Carlo sampling.The generated variables are z = {ω, r}, combining reconstruction weights and regularization.
Experiments
Experiments on MM-IMDb, CMU-MOSI, and avMNIST evaluate SMIL's efficiency under severely missing modalities and flexibility across missing training or testing patterns. SMIL consistently outperforms baselines and remains close to full-modality performance in several settings.
- Experimental setup: Experiments cover MM-IMDb, CMU-MOSI, and avMNIST, assessing efficiency under severe missingness and flexibility across modality-missing patterns.The evaluation includes multi-label, binary, and digit classification settings.
- Efficiency with severely missing modality: SMIL significantly outperforms all baselines across CMU-MOSI text-modality ratios, with gains increasing from around 5.0% at η = 20% to 7.6% and 7.4% at η = 10%.The reported comparisons are against AE and GAN; improvements are also consistent on MM-IMDb.
- Efficiency with severely missing modality: SMIL improves tailed MM-IMDb genres such as Sport and Film-Noir, while text slightly decreases performance for Family and Animation.The authors attribute the latter to overlap between Family and Animation genres, which can reduce accuracy.
- Flexibility with different missing patterns: On avMNIST with image-only testing at η = 20%, SMIL is 6.7% above the generative method and 3.3% above the Lower-Bound.At η = 5%, SMIL is 1.10% higher than the Lower-Bound; with image-and-audio testing, it exceeds the Lower-Bound by 4.3% and AE/GAN by 2.1%.
- Flexibility with different missing patterns: When testing changes from two modalities to one, AE and GAN drop 5.6%, whereas SMIL drops 1.0%.The result supports SMIL's flexibility across testing patterns.
- Ablation study: Ablations show that K-Means reconstruction, feature regularization, and Bayesian inference each contribute to SMIL's performance.Removing reconstruction or regularization worsens performance, while fixed-Gaussian and deterministic variants are inferior to Bayesian inference.
Conclusion
The paper addresses multimodal learning with severely missing modality through a Bayesian meta-learning strategy targeting reconstruction and feature regularization. Experiments show that the model outperforms generative methods on three multimodal datasets and supports varied missing-modality patterns.
- SMIL addresses multimodal learning with severely missing modality using a meta-learning strategy.The framework targets missing-modality reconstruction and feature regularization.
- The framework jointly targets missing-modality reconstruction for flexibility and feature regularization for efficiency.
- SMIL significantly outperforms generative methods on three multimodal datasets.
- Further analysis indicates that modality reconstruction and feature regularization handle missing modalities across various missing patterns.