PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Multimodal Learning with Missing Modality: A Survey

Renjie Wu, Hu Wang, Hsiang-Ting Chen, Gustavo Carneiro

arXiv:2409.07825v4cs.CVcs.AIcs.LG

TL;DR

Missing modalities caused by real-world constraints can degrade multimodal performance, motivating MLMM. This survey organizes deep MLMM research, applications, and datasets through a fine-grained taxonomy and reviews reported methods and challenges. Existing approaches are diverse, but comparisons remain constrained by inconsistent settings and efficiency limitations.

  • Problem

    Real-world multimodal systems may lack one or more modalities during training or testing, creating a need to maintain performance from incomplete inputs.

  • Method

    The survey analyzes 354 papers and organizes MLMM methods by data processing and strategy design, while reviewing applications, datasets, and research directions.

  • Results

    Existing methods are diverse, with generative and distillation approaches common and intermediate-stage feature recovery representing the largest reported recovery category at 14.1%.

  • Takeaways & Limitations

    MLMM spans modality recovery and available-modality prediction across many domains, but unified benchmarks and broader real-world evaluation remain needed.

  • Takeaways & Limitations

    Evaluation protocols vary in missing rates, train–test configurations, preprocessing, dataset scale, and accessibility, complicating fair comparison and reproducibility.

Abstract

from arXiv · show

During multimodal model training and testing, certain data modalities may be absent due to sensor limitations, cost constraints, privacy concerns, or data loss, negatively affecting performance. Multimodal learning techniques designed to handle missing modalities can mitigate this by ensuring model robustness even when some modalities are unavailable. This survey reviews recent progress in Multimodal Learning with Missing Modality (MLMM), focusing on deep learning methods. It provides the first comprehensive survey that covers the motivation and distinctions between MLMM and standard multimodal learning setups, followed by a detailed analysis of current methods, applications, and datasets, concluding with challenges and future directions.

1 Introduction

Missing modalities arise throughout real-world multimodal systems and can substantially degrade performance. This survey defines MLMM, reviews its methods and applications, and proposes a fine-grained taxonomy supported by a broad literature collection.

  • Missing modalities can result from sensor failures, hardware limits, privacy concerns, environmental interference, or transmission failures, affecting data from collection through deployment.
  • MLMM addresses training or testing when one or more of N modalities are absent, aiming to exploit available modalities while retaining complete-input performance.
  • It reviews MLMM methods, applications, and datasets across information retrieval, remote sensing, pervasive computing, robotics, medicine, sentiment analysis, and multi-view clustering.
  • The paper contributes a fine-grained taxonomy, comprehensive cross-domain survey, and analysis of current challenges and future research directions.
  • The survey compiles 354 significant papers published from 2012 to August 2025 using multimodal-learning and missing-data search terms across major AI fields and venues.

2 Methodology Taxonomy: An Overview

The taxonomy organizes deep MLMM methods by where missing information is handled and how models adapt to missing-modality cases. It spans data-level imputation, representation-level processing, architecture changes, and external model combinations.

  • Data Processing: Representation-focused models address missing modalities through coordinated representations or other representation-level mechanisms.
  • Data Processing: Modality imputation fills absent data by composition or generation so downstream tasks can operate as if complete modalities were available.
  • Strategy Design: Architecture-focused models dynamically adjust fusion and processing, with attention mechanisms allowing operation with varying numbers of available modalities.
  • Strategy Design: Model combinations use dedicated training strategies, ensembles, or schedulers that select and combine specialized models for different missing-modality cases.
  • The taxonomy separates methods into data processing and strategy design, covering modality data, representations, architectures, and model combinations.
  • Data Processing: Zero/random replacement and retrieval-based composition exemplify methods that construct substitutes for missing modality data.

3 Methodologies in Data Processing Aspect

Data-processing MLMM methods address missing modalities by imputing modality data or representations, using composition, generation, coordination, and parameter-free fusion strategies.

  • 3.1 Modality Imputation: Modality imputation completes missing-modality data through composition with placeholders or retrieved samples, or generation with learned generative models.Composition uses zeros, random values, or copied data; generation uses autoencoders, GANs, diffusion models, or related architectures.
  • 3.1.1 Modality Composition Methods: Retrieved or duplicated modality data can fail on pixel-level tasks, reduce dataset diversity, and increase overfitting risk when missingness is high.Mismatched retrievals are especially problematic beyond simple classification, while repeated samples can overrepresent classes with few complete examples.
  • 3.1.1 Modality Composition Methods: KNN-based composition generally performs better than simpler retrieval methods and can handle missing modalities during testing, but incurs computational and memory costs.It remains sensitive to imbalanced data, while clustering-based lookup can enable faster runtime completion.
  • 3.1.2 Modality Generation Methods: Deep generative methods learn cross-modal relationships either with an individual generator per modality or one unified model that generates all modalities.Examples include MRI-to-PET generation, GAN-based depth synthesis, and unified autoencoder or attention-based generation.
  • 3.2 Representation-Focused Models: Representation-focused methods impose cross-modal constraints or compose representations through arithmetic operations, flexibly supporting varying modality counts without new parameters.Arithmetic composition is efficient but may lose information and poorly capture inter-modality relationships.
  • 3.2.3 Representation Generation Methods: Direct-to-task representation generation avoids reconstructing modality data and integrates more easily into existing multimodal models, whereas indirect methods use auxiliary reconstruction during training.Indirect methods can access complete samples and simulate arbitrary missing-modality cases, generally achieving better performance than direct methods.

4.1 Architecture-Focused Models

Architecture-focused MLMM methods adapt fusion, attention, transformers, distillation, and graph structures to operate when modalities are absent.

  • 4.1 Architecture-Focused Models: Architecture-focused methods modify model training or testing and include attention-based, distillation-based, graph learning-based, and multimodal large language model approaches.These categories complement modality- and representation-level data processing methods.
  • 4.1.1 Attention Based Methods: Attention fusion handles missing modalities by skipping their intra-modality attention or masking their tokens during inter-modality attention.Shared fusion representations allow available modalities to be combined sequentially, while zero or negative-infinity masks force attention to ignore absent tokens.
  • 4.1.1 Attention Based Methods: Transformer joint representation learning accepts variable numbers of modality tokens, while parameter-efficient methods fine-tune only a small number of parameters through prompts or adapters.Joint training can require substantial computation and large datasets; parameter-efficient tuning is more efficient but may have lower performance.
  • 4.1.2 Distillation Based Methods: Representation-based distillation transfers teacher logits or intermediate features to students so they can reconstruct or learn features associated with missing modalities.Intermediate distillation aligns teacher and student features, while response distillation transfers output probability distributions.
  • 4.1.3 Graph Learning-Based Methods: Graph fusion aggregates available-modality features around missing nodes while preserving the fused feature dimension.The illustrated setup treats modality 2 as absent and combines features from modalities 1 and 3.
  • 4.1.2 Distillation Based Methods: Process-based distillation uses strategies such as exponential-moving-average teachers or self-distillation, which refine students through procedural teacher–student learning.Mean Teacher Distillation uses the student’s parameter moving average as a teacher; self-distillation uses soft labels or representations from model branches.
  • 4.1.2 Distillation Based Methods: Most distillation methods assume complete multimodal training data and therefore are unsuitable when modalities are already missing during training.Only some intermediate- and self-distillation methods address incomplete training datasets under this taxonomy.

4.2 Model Combinations

Model combinations address missing modalities by selecting and aggregating predictions from multimodal, unimodal, specialized, or scheduled models. Their flexibility comes with storage, modeling, and coordination costs that limit practical deployment.

  • Ensemble Methods: Ensemble methods aggregate predictions from multiple multimodal or unimodal models, using mechanisms such as voting or weighted averaging.Multimodal ensembles contain multiple three-modal DNNs, while unimodal ensembles process each modality separately and aggregate accessible outputs.
  • Hybrid Methods: Hybrid methods dynamically select unimodal or multimodal models, while cross-attention can combine predictions from different modality configurations and reduce multimodal decoding noise.DynMM uses gating, and PME integrates box features with attention bias.
  • Dedicated Training Methods: Dedicated training assigns different missing-modality cases to specialized models, with KDNet training unimodal students from a full-modality teacher’s features and logits.The resulting unimodal models can serve different missing-modality configurations.
  • Discrete Scheduler Methods: Discrete scheduler methods use LLMs as controllers that plan and order modular downstream steps, enabling task execution across variable numbers of modalities.They rely on sufficiently diverse callable modules and accurate, fast instruction understanding in real-world scenarios.
  • Limitations: Most ensemble and dedicated approaches require additional model storage, while unimodal ensembles miss complex inter-modality relationships and schedulers depend on capable LLMs and available modules.These constraints are especially problematic for resource-constrained devices and systems with many modalities.

5 Methodology Discussion

The survey organizes MLMM methods into four methodological types and analyzes their recovery strategies, trade-offs, historical development, and unresolved evaluation challenges. Intermediate-stage recovery dominates the reviewed literature, while newer architectures and foundation models broaden the design space.

  • Taxonomy: The taxonomy divides MLMM into modality imputation, representation-focused, architecture-focused, and model-combination methods across data-processing and strategy-design aspects.The framework further subdivides these four types into twelve categories and compares their advantages and disadvantages.
  • Method Comparisons: Generative and distillation methods are common and generally strong, while Transformers gain popularity through larger receptive fields and parallelism.Most distillation and indirect-to-task generation methods still require complete training datasets, with limited exceptions.
  • Architecture-Focused Methods: Architecture-focused methods increasingly use attention, graph learning, and distillation to model inter-modal relationships, although scalability, complexity, and dataset-size constraints remain.Graph methods are currently unsuitable for large-scale datasets, and Transformer-based approaches introduce trade-offs noted in the taxonomy discussion.
  • Recovery and Non-Recovery Methods: Among 315 analyzed papers, 75.5% focus on recovering missing modalities, with intermediate-stage recovery representing the largest share of recovery methods.Intermediate features may avoid raw-data noise while retaining modality-specific and shared information.
  • Technical Evolution Discussion: Publication trends show an evolution from heuristic data-level fixes and ensembles toward representation learning, flexible architectures, and foundation-model processing or scheduling.Agentic AI may renew interest in model combinations through dynamic selection and tool integration.
  • Limitations: Performance comparisons remain difficult because missing-modality studies use varied experimental settings, leaving cross-dataset evaluation as an open research gap.The survey explicitly encourages the community to address this comparability problem.

6 Applications and Datasets

MLMM applications span affective computing, medical AI, retrieval, remote sensing, pervasive computing, and robotic vision, using diverse multimodal datasets to address incomplete inputs. However, natural missing-modality datasets remain rare, so evaluations often simulate missingness across modality combinations and rates.

  • Applications and datasets: Up to 90% of samples may have missing modalities in severe cases, challenging conventional full-modality learning methods.Privacy concerns, sensor malfunctions, and costly data collection contribute to incomplete datasets.
  • Applications and datasets: Table 3 organizes common deep-MLMM datasets by application and modality type, including vision, bio-sensors, motion sensors, physiological signals, audio, text, CT, MRI, and skeleton data.The modality categories cover sensor and biomedical sources used across the surveyed applications.
  • Robotic Vision and Sensing: Robotic-vision tasks commonly combine RGB with depth, optical flow, LiDAR, radar, infrared, event, or auditory data, which may be unavailable because of sensor damage or adverse weather.The surveyed tasks include multimodal segmentation and detection for autonomous vehicles and related systems.
  • Applications and datasets: MLMM datasets and applications cover affective computing, medical analysis, information retrieval, remote sensing, pervasive computing, and robotic vision.These areas use combinations of visual, textual, auditory, biomedical, and sensor modalities.
  • Research coverage and dataset limitations: Research is concentrated in several established domains, whereas MLMM for streaming data and scientific fields remains relatively limited.Public datasets are usually complete, and naturally occurring missing-modality datasets are rare; studies therefore evaluate simulated missing combinations and rates.

7 Open Issues and Future Research Directions

Future MLMM research must clarify when to recover missing modalities and when to predict from available ones. Recovery can improve robustness, but generated content may contain artifacts or hallucinations and may be ignored when available modalities dominate.

  • Recovery or Non-Recovery Methods: Recovered modalities can improve robustness, but generative outputs may contain artifacts or hallucinations caused by generative models or biased training data.Improving generative fidelity is identified as a future direction.
  • Recovery or Non-Recovery Methods: MLMM methods either recover missing modalities or make predictions using only the modalities that remain available.The survey frames the choice as determining when recovery is preferable to non-recovery.
  • Recovery or Non-Recovery Methods: Imbalanced missing rates can cause models to ignore recovered information and rely on available modalities instead.When a crucial modality is absent, reliance on less important available information can produce incorrect outcomes, making recovery potentially useful.

7.3 Benchmarking and Evaluations for Missing Modality Problems

MLMM evaluations are difficult to compare because studies use different missing-modality settings, dataset scales, and preprocessing choices. The survey calls for systematic benchmarks and more efficient methods suitable for constrained deployment environments.

  • Benchmarking and Evaluations: Different missing-modality settings make it difficult to compare results across MLMM studies.Existing work varies in missing rates, train–test configurations, and dataset preprocessing.
  • Benchmarking and Evaluations: Large-scale multimodal pretraining changes the learning problem because models may infer missing information from learned priors rather than explicit modality alignment.Comparisons must account for differences in dataset scale, diversity, and accessibility because they affect performance and reproducibility.
  • Benchmarking and Evaluations: The survey proposes a MultiBench-like benchmark covering common datasets and missing-modality settings for more systematic research.The goal is to support evaluation across consistent tasks and conditions.
  • Method Efficiency: Many MLMM methods are computationally heavy because they train separate models for modality combinations or missing-modality patterns.Such designs can require independent models per combination or distinct recovery models for each missing information type.
  • Method Efficiency: Resource-constrained devices such as space or disaster-response robots motivate lightweight and efficient MLMM solutions.These devices may lack high-performance GPUs and face costly sensor repair.

7.5 Multimodal Streaming/Temporal Data with Missing Modality

Missing modalities in streaming and temporal data are underexplored despite their importance for long multimodal sequences in robotics and pervasive computing. Scientific applications face related incompleteness from restricted access, acquisition costs, and heterogeneous data.

  • Streaming and temporal data: Only a small portion of MLMM research addresses multimodal streaming or temporal data.Existing examples include sentiment analysis, while broader streaming settings remain insufficiently studied.
  • Streaming and temporal data: Long RGB+X video streams and multimodal time-series signals commonly support anomaly detection and video tracking in robotic and pervasive-computing systems.Missing modalities in these settings can occur asynchronously.
  • Streaming and temporal data: Multimodal reinforcement-learning agents face sensor failures or restricted modality access in tasks such as robotic grasping, drone control, and driving decisions.These practical conditions can compromise algorithm robustness.
  • Scientific applications: Scientific MLMM applications integrate molecular structures, genomic sequences, and spectral images, but access restrictions, acquisition costs, and heterogeneous data can produce missing modalities.Research on MLMM in scientific applications remains limited.

7.8 Handling Practical Missing Modality Problems in Real-World

Real-world MLMM deployment remains insufficiently explored, especially across pervasive-computing environments where missing-modality studies are fragmented and lack holistic evaluation.

  • IoT infrastructures, wearables, smart homes, mobile sensing, and distributed environmental sensor networks remain consequential but underexplored MLMM deployment contexts.Existing pervasive-computing studies have begun addressing missing modalities, but efforts remain fragmented.

8 Conclusion

The survey synthesizes deep MLMM methods, applications, datasets, challenges, and future directions while identifying unresolved needs in benchmarking, taxonomy, efficiency, realistic evaluation, and broader applicability. It concludes that stronger theory, realistic benchmarks, efficient architectures, and cross-domain research are needed for more robust and transferable MLMM.

  • The survey presents the first comprehensive review of deep MLMM, covering motivation, a fine-grained taxonomy, methods, applications, datasets, challenges, and future directions.It also highlights the need for a unified benchmark and wider application range.
  • The taxonomy emphasizes a practical machine-learning perspective rather than theoretical analysis and may require refinement as foundation-model ecosystems evolve.New architectures, training paradigms, and modality-agnostic interfaces motivate further taxonomy updates.
  • Most MLMM approaches remain computationally heavy, limiting scalability to edge devices, resource-constrained deployments, and long-horizon streaming scenarios.The survey identifies cross-modal alignment modules and complex imputation pipelines as contributors to efficiency challenges.
  • Current evaluation uses inconsistent protocols, rarely models realistic missingness, and often lacks principled measures of modality importance, uncertainty, and redundancy.These limitations hinder fair comparison and contribute to weak cross-domain generalization.
  • MLMM remains sparse in IoT, wearables, smart homes, mobile sensing, environmental networks, climate modeling, earth observation, molecular biology, and materials discovery.Addressing these areas requires domain-aware missingness models, physics- and biology-informed constraints, and stronger theoretical understanding.
  • Future MLMM research should combine stronger theoretical frameworks, realistic benchmarks, efficiency-aware architectures, and broader cross-domain applicability.The survey links these directions to more robust, transferable, and scientifically grounded research.

A Paper Collection

The paper collection lists full names for conference and journal abbreviations used in the paper’s collection paragraph.

  • The collection includes venues such as AAAI, IJCAI, NeurIPS, and ICLR, with abbreviations expanded to their full names.
Loading 2409.07825v4…