Source-linked AI summary
M$^3$Care: Learning with Missing Modalities in Multimodal Healthcare Data
Chaohe Zhang, Xu Chu, Liantao Ma, Yinghao Zhu, Yasha Wang, Jiangtao Wang, Junfeng Zhao
TL;DR
Missing modalities are common in multimodal EHR data, while existing generative approaches pose an unstable inverse problem and may not preserve task-relevant information. M3Care instead imputes task-related latent representations using similar patients identified by task-guided, modality-adaptive similarities. Across real-world datasets, it outperforms baseline models and produces findings consistent with experts and medical knowledge.
Problem
Missing modalities are common in clinical EHR data, while existing methods often reconstruct raw inputs through an unstable inverse generative problem and may neglect task-specific information.
Method
M3Care is an end-to-end model that uses task-guided modality-adaptive similarities to find comparable patients and impute missing modalities’ task-related representations in latent space.
Results
M3Care outperforms all baseline models across experiments on real-world datasets, with relative improvements of 4.9% in micro-AUC on ODIR and 6.1% in AUPRC and 6.0% in AUROC on OV.
Takeaways & Limitations
M3Care’s findings accord with experts and medical knowledge, indicating potential to provide useful insights and explanations alongside clinical prediction and analysis.
Takeaways & Limitations
The formulation assumes irregular modality missingness but requires at least one modality to be present for every patient in training and test or validation sets.
Abstract
from arXiv · showhide
Multimodal electronic health record (EHR) data are widely used in clinical applications. Conventional methods usually assume that each sample (patient) is associated with the unified observed modalities, and all modalities are available for each sample. However, missing modality caused by various clinical and social reasons is a common issue in real-world clinical scenarios. Existing methods mostly rely on solving a generative model that learns a mapping from the latent space to the original input space, which is an unstable ill-posed inverse problem. To relieve the underdetermined system, we propose a model solving a direct problem, dubbed learning with Missing Modalities in Multimodal healthcare data (M3Care). M3Care is an end-to-end model compensating the missing information of the patients with missing modalities to perform clinical analysis. Instead of generating raw missing data, M3Care imputes the task-related information of the missing modalities in the latent space by the auxiliary information from each patient's similar neighbors, measured by a task-guided modality-adaptive similarity metric, and thence conducts the clinical tasks. The task-guided modality-adaptive similarity metric utilizes the uncensored modalities of the patient and the other patients who also have the same uncensored modalities to find similar patients. Experiments on real-world datasets show that M3Care outperforms the state-of-the-art baselines. Moreover, the findings discovered by M3Care are consistent with experts and medical knowledge, demonstrating the capability and the potential of providing useful insights and explanations.
1 INTRODUCTION
Missing modalities are common in multimodal EHR data, but conventional imputation and generative approaches do not adequately address their structured missingness and task-specific similarity. M3Care instead performs task-guided latent-space compensation using similar patients and reports stronger performance and clinically consistent findings.
- Motivation: Missing modalities arise from clinical and social factors, and each absent modality removes all of its features from a patient’s data.Examples include patient dropout, sensor damage, data corruption, safety considerations, and high cost.
- Challenges: Traditional imputation methods fit random or block-wise missingness poorly because missing modalities create column-wise consecutive gaps and rows represent permutable patients.Features within a modality are highly correlated, while the missing modality may be entirely absent.
- Challenges: Existing methods may impute task-irrelevant information because they operate in the original input space rather than a task-supervised latent space.The paper frames latent-space similarity modeling as a direct problem that is less complicated than reconstructing raw data through an inverse generative mapping.
- Challenges: Patient similarity must be task-guided, modality-adaptive, and aggregated across intra- and inter-modality feature correlations.Patients may be similar for one clinical task but not another, and local similarities alone do not capture global row relationships.
- M3Care: M3Care compensates for missing modalities in latent space using similar patients’ auxiliary information and performs clinical tasks end to end.It uses task-guided deep kernels for modality-specific similarities and self-attentive multimodal interaction to aggregate them.
- Results: M3Care outperforms state-of-the-art models across multiple incompleteness levels and evaluation metrics, while its findings accord with experts and medical knowledge.The paper positions these findings as evidence that the model can provide useful insights and explanations.
2 RELATED WORK
Healthcare multimodal learning combines diverse clinical data sources, but missing-modality handling remains difficult. Existing approaches either generate missing modalities or rely on methods that are poorly suited to modality-level missingness and may introduce noise or scaling difficulties.
- Multimodal learning for healthcare: Healthcare multimodal learning analyzes sources such as medical images, clinical notes, continuous monitoring data, clinical events, and trial-criteria text.These modalities support applications including patient feedback, physician insights, decompensation prediction, and patient-trial inference.
- Missing-modality methods: Missing modalities produce column-wise consecutive missingness, so traditional matrix-completion methods cannot be used directly.The missingness pattern differs from ordinary random-entry assumptions.
- Missing-modality methods: Generative approaches use methods such as autoencoders and GANs to generate missing modalities before downstream analysis.These approaches are presented as a second major strategy for handling incomplete multimodal data.
- Missing-modality methods: Generative imputation may add unwanted noise, harm performance when complete samples are scarce, and require many difficult-to-train generators for numerous modalities or missingness patterns.The stated risks become more pronounced as the number of modalities grows relative to complete samples.
3 PROBLEM FORMULATION
The paper formalizes multimodal EHR data as patient-level collections of observations from multiple data sources and defines incomplete patients as those missing one or more modalities. It considers irregular missingness in clinical prediction settings.
- Patient multimodal EHR data: Each patient is represented by observations from multiple modalities, including medical images, clinical notes, and lab tests.The formulation introduces M as the number of modalities and N as the number of patients.
- Patient data with missing modalities: A patient with missing modalities has observed data from fewer than all available modalities.The paper denotes the observed collection using 0 < M′ < M.
- Patient data with missing modalities: The formulation uses an irregular missingness setting in which every modality may be absent across training and test or validation data, while each patient retains at least one modality.This is described as the most relaxed setting in the problem formulation.
- Clinical tasks: Disease diagnosis is formulated as binary or multi-label classification from a patient’s multimodal EHR data with missing modalities.The label vector contains one entry for each disease category.
4 METHODOLOGY
M3Care extracts modality-specific latent representations, discovers task-guided similar patients, aggregates their information, and adaptively imputes missing modalities before multimodal clinical prediction.
- Unimodal Representation Extraction: Unimodal encoders map each modality into task-relevant latent representations, using ResNet for images, Transformer Encoders for sequential data, and MLPs for vector-based data.The framework accommodates modality-specific input structures, with missing information represented by black boxes.
- Similar Patients Discovery and Information Aggregation: M3Care computes modality-specific patient similarities with task-guided deep kernels and represents them as graphs for information propagation.The similarity computation is designed to adapt to modality semantics and clinical tasks.
- Similar Patients Discovery and Information Aggregation: A learnable threshold filters dissimilar patient pairs, while modality masks exclude similarities involving missing data.The threshold is learnable, and epsilon prevents unstable division by zero.
- Similar Patients Discovery and Information Aggregation: Graph convolutional layers aggregate auxiliary information from similar patients using the modality-specific similarity matrix as the graph adjacency matrix.The aggregated representations provide information for imputing missing modality representations.
- Adaptive Modality Imputation: For missing modalities, M3Care uses aggregated information directly; for complete modalities, learned weights combine self-information with similar-patient information.The weights are constrained so that α + β = 1.
- Multimodal Interaction Capture: The imputed and enhanced modality representations are passed to a context-aware multimodal interaction module that captures intra- and inter-modality correlations for clinical tasks.The interaction module uses stacked attention-based layers and a predictor.
5 EXPERIMENT
M3Care is evaluated on ODIR and OV multimodal EHR datasets using clinical classification tasks and comparisons with established baselines. It outperforms the baselines across reported metrics, while attention visualizations provide clinically consistent interpretive evidence.
- Experimental setup: The experiments compare M3Care with state-of-the-art multimodal and missing-modality baselines, using matched embedding layers for fair comparison.The baseline set includes MFN, MulT, ViLT, and other methods; embedding networks are upgraded to the same architectures and no pretrained parameters are used.
- Results: M3Care outperforms all baselines across the reported evaluation metrics.The paper reports results in Tables 2 and 3 for the ODIR and OV datasets.
- Results: 4.9% relative improvement in micro-AUC is reported on ODIR, while OV shows 6.1% higher AUPRC and 6.0% higher AUROC than the best baseline.ODIR uses 1,000-times bootstrap standard deviations, whereas OV uses 10-fold cross-validation standard deviations.
- Results: M3Care performs well on OV despite its 832 patients and six modalities, a setting with many modalities and missing patterns.The authors describe this as suitable for real-world scenarios involving numerous modalities or missingness patterns.
- Ablation analysis: Ablations associate the performance advantage with task-guided deep kernels, Information Aggregation, and Adaptive Modality Imputation.M3Care outperforms variants using cosine similarity or omitting the aggregation and adaptive-imputation modules.
- Clinical implications: Attention visualizations show successful predictions for two OV patients with abnormally increased intraocular pressure and highlight medications linked to this outcome.The visualized rows and columns represent query and key multimodal records.
- Clinical implications: M3Care focuses on Mydrin-P and Pred Forte, drugs whose possible association with elevated intraocular pressure agrees with medical literature and clinician experience.The model attends to these medications in both visualized patients, and the paper notes their known adverse-reaction relationship with elevated IOP.
6 CONCLUSIONS
The paper presents M3Care as an end-to-end approach for clinical prediction and analysis with missing modalities. It imputes latent missing-modality representations from similar patients and reports superior baseline performance plus medically consistent findings.
- Conclusion: M3Care compensates for missing modalities while performing clinical prediction and analysis in an end-to-end model.The approach addresses patients with incomplete multimodal EHR data.
- Conclusion: M3Care finds similar patients with a task-guided modality-adaptive metric and imputes missing-modality hidden representations in latent space.It uses auxiliary information from similar patients rather than generating raw missing data.
- Conclusion: Experiments show that M3Care outperforms all baseline models and produces findings consistent with experts and medical knowledge.The paper presents these results as evidence that M3Care can provide useful insights.
A INTUITION DISCOVERY EXPERIMENT
The intuition experiment tests whether patient similarity transfers across modalities by comparing pair-wise similarity matrices under original, noisy, and shuffled representations. The original cross-modal differences are lower than both perturbation conditions.
- Similarity construction: The experiment extracts modality-specific latent representations from complete samples and computes pair-wise similarity matrices for each modality.A unimodal classifier is trained for each modality, and its best validation model supplies test-set latent representations.
- Cross-modal similarity: The tested intuition is that patients similar in one modality are more likely to be similar in another modality with respect to the clinical task.The experiment compares normalized Euclidean distance, cosine similarity, and RBF-kernel-based similarities under different matrix norms.
- Perturbation tests: Noise and shuffle perturbations are applied to one modality’s latent representations, with average differences computed over 1,000 repetitions.The perturbed similarity-matrix differences are compared with the original cross-modal difference.
- Results: The original pair-wise similarity-matrix difference is smaller than both the Noise and Shuffle differences across tested similarity metrics and norms.This result supports cross-modal transfer of sample similarity in the evaluated OV modalities.
- M3Care pipeline: The algorithm trains on batches, detects missing modalities, imputes them, enhances representations, models multimodal dynamics, predicts, and updates parameters.The listed steps reference the paper’s imputation, representation-enhancement, interaction, prediction, and optimization equations.
C FURTHER ANALYSIS
M3Care was evaluated under increasing multimodal incompleteness and retained superior performance while its attention patterns aligned with known clinical evidence.
- C.1 Multiple levels multimodal incompleteness: M3Care was evaluated on ODIR with missing-modality rates from 30% to 60%, using 1,000 bootstrap repetitions and micro-AUC.The evaluation examines multiple levels of multimodal incompleteness.
- C.1 Multiple levels multimodal incompleteness: All models’ micro-AUCs decreased as missing rates increased, but M3Care continued to outperform every baseline.
- C.1 Multiple levels multimodal incompleteness: 0.7715 micro-AUC at 60% missingness exceeded CM-AEs’ 0.7477 and HGMF’s 0.7427, corresponding to 3.2% and 3.8% relative improvements.
- C.2 Clinical implications: For a positive case, M3Care focused strongly on Med0 and Med1 containing medications associated with elevated intraocular pressure.
- C.2 Clinical implications: For a negative case, M3Care distributed attention relatively evenly and made a correct prediction without identifying significant elevation-of-pressure signs.
D.1 Statistics of the Datasets
The datasets include modality-incomplete clinical records spanning ocular disease recognition and ophthalmic vitrectomy.
- D.1 Statistics of the Datasets: ODIR includes demographic information, bilateral clinical text, and bilateral fundus images, with incomplete modalities present.
- D.1 Statistics of the Datasets: OV includes demographic data, clinical notes, medications, admission and discharge records, and surgical consumables, with incomplete modalities present.
- D.1 Statistics of the Datasets: Table 7 presents detailed dataset statistics.
D.2 Model Implementation
The implementation uses a specified hardware and software environment with dataset-specific embedding and hidden dimensions.
- D.2 Model Implementation: Experiments use an Intel Xeon E5-2630 CPU, 256GB RAM, an Nvidia RTX8000 GPU, and PyTorch 1.5.1.
- D.2 Model Implementation: Embedding and hidden dimensions are set to 128 and 256 for the ODIR and OV datasets, respectively.
- D.2 Model Implementation: The missing-rate order corresponds to the preceding data description.