Source-linked AI summary
Multimodal Prompt Learning with Irregular EHRs for Robust Monitoring of Critical Care Patients
Yixin Yang, Yueyang Sun, Weichen Liu, Xianbing Zhao, Sicen Liu
TL;DR
Incomplete and irregular multimodal EHRs make robust ICU mortality prediction difficult when modalities are partially observed or unavailable. The paper introduces a four-prompt multimodal framework that reconstructs missing representations and conditions fusion and temporal aggregation on missingness patterns. Across two benchmark datasets and missingness settings, it reports better performance than compared methods, while ablations support complementary prompt contributions.
Problem
Missing modalities and irregular observations can substantially degrade multimodal mortality prediction from real-world EHRs.
Method
A multimodal Transformer integrates generative, missing-signal, missing-type, and temporal prompts for missingness-aware temporal, intramodal, and cross-modal representations.
Results
The method outperforms compared approaches on MIMIC-III and MIMIC-IV across missingness settings, with all prompts reaching an AUPRC of 0.5853 in ablation.
Takeaways & Limitations
The framework provides a unified approach for mortality prediction from irregular and incomplete multimodal EHRs, with prompt components contributing complementary functions.
Abstract
from arXiv · showhide
Accurate assessment of patients in intensive care units (ICUs) is essential for timely clinical intervention and improved patient outcomes. Multimodal electronic health records (EHRs), including structured physiological time series and longitudinal clinical notes, provide complementary information for critical care prediction. However, in real-world clinical settings, individual modalities may be partially observed or entirely unavailable, resulting in substantial performance degradation for existing multimodal models. To address this challenge, we propose a multimodal prompt-learning framework for robust clinical prediction under diverse missing-modality scenarios. The proposed framework introduces four complementary types of prompts: generative prompts, missing-signal prompts, missing-type prompts, and temporal prompts. Generative prompts construct surrogate latent representations for unavailable modalities, while missing-signal prompts distinguish observed representations from generated ones. Missing-type prompts condition the model on different modality-availability configurations, whereas temporal prompts perform condition-specific aggregation over temporally encoded clinical sequences. Together, these prompts enable the model to capture missingness-aware intramodal dependencies and cross-modal interactions within a unified architecture. Extensive experiments demonstrate that our method outperforms existing approaches across evaluation metrics on two missingness settings. Ablation and robustness analyses further verify the complementary contributions of the four prompt types and the effectiveness of the proposed framework for clinical prediction from incomplete multimodal EHR data.
1 Introduction
The paper targets mortality prediction from irregular, incomplete multimodal EHRs, where missing modalities and uneven temporal observations can degrade clinical risk assessment. It proposes a four-prompt multimodal framework and evaluates it on public ICU benchmarks.
- Motivation: Missing modalities and irregular observations in real-world EHRs can substantially degrade mortality prediction.Physiological measurements occur at uneven intervals, while clinical notes may be unavailable because of workflow, equipment, data, or privacy constraints.
- Approach: The framework uses generative, missing-signal, missing-type, and temporal prompts to model incomplete EHRs.These prompts reconstruct unavailable modalities, distinguish observed from surrogate features, encode availability configurations, and aggregate irregular temporal information conditionally.
- Evaluation: Experiments on MIMIC-III and MIMIC-IV evaluate the proposed method against existing approaches.The paper reports extensive experiments on two large publicly available benchmark datasets.
- Contributions: The proposed method is designed for critical-care monitoring with irregular EHR and missing modalities.The stated contributions include handling missing modalities efficiently and capturing intra- and inter-modality information.
2 Related Work
Prior work models irregular multimodal EHRs and missing clinical modalities, but generally emphasizes available streams or non-EHR settings. This paper adapts prompt-based missing-modality learning to irregular clinical data by adding temporal conditioning.
- Multimodal Modeling of Irregular EHRs: Earlier irregular-EHR methods model temporal structure using decay, interpolation, set representations, or continuous-time attention.Other approaches separately encode MISTS and clinical notes before integrating them through temporally interleaved attention or shared cross-modal patterns.
- Learning with Missing Clinical Modalities: Existing multimodal EHR approaches primarily extract information from available input streams.This leaves incomplete-data conditions as a distinct challenge for robust multimodal prediction.
- Learning with Missing Clinical Modalities: Missing-modality research has explored cross-modal translation, modality imagination, prompting, and multimodal clinical settings.Recent studies address combinations of missing modalities and labels or missing radiographs, reports, and structured variables.
- Prompt Learning for Missing Modalities: Prompt-based missing-modality methods have mainly targeted general multimodal tasks rather than irregular EHRs with heterogeneous temporal patterns.The proposed work adds temporal prompts and combines prompt-conditioned reconstruction with time-aware MISTS and clinical-note encoders for ICU outcome prediction.
- Prompt Learning for Missing Modalities: The framework overview illustrates reconstruction of an absent MISTS stream from notes-only input, followed by provenance, availability, fusion, and temporal processing.The figure describes the roles of missing-signal, missing-type, and temporal prompts in the prediction pipeline.
3 Method
The framework uses four complementary prompt types to predict ICU mortality from irregular multimodal EHRs when one modality may be unavailable. It generates surrogate representations, marks their provenance, conditions fusion on missingness, and performs availability-specific temporal aggregation.
- Modality Encoding: The model processes irregular physiological time series and longitudinal clinical notes with separate modality-specific encoders before multimodal fusion.MISTS are encoded with UTDE, while clinical notes use a Clinical-Longformer-based mTANDtxt encoder.
- Generative Prompts: Generative prompts use observed representations to construct time-aligned surrogate representations for unavailable modalities.When MISTS are missing, a learnable generative prompt queries observed clinical-note representations through cross-attention to generate the missing stream.
- Missing-Signal Prompts: Missing-signal prompts mark each modality representation as observed or generated so fusion can weight the two sources differently.Learnable provenance prompts are added to time-aligned tokens, distinguishing synthesized MISTS features from observed clinical-note features.
- Missing-Type Prompts: Missing-type prompts provide both modality streams with an explicit global condition describing which modality-availability configuration applies.The selected condition-specific prompt is appended to both cross-attended representations before separate self-attention stacks.
- Temporal Prompts: Temporal prompts perform condition-specific aggregation over concatenated time-aligned modality tokens to produce a patient-level representation for mortality prediction.The resulting summary is processed by a residual two-layer MLP and binary classifier to estimate in-hospital mortality probability.
4 Experiment
Experiments evaluate 48-hour in-hospital mortality prediction across MIMIC-III and MIMIC-IV, comparing missing-modality, full-modality, and ablation settings. The proposed prompt framework consistently improves missing-modality performance while preserving strong complete-modality performance, with complementary prompts and modality dropout supporting robustness.
- Experimental Setup: The evaluation uses MIMIC-III and MIMIC-IV cohorts for 48-hour in-hospital mortality prediction from irregular physiological data and clinical notes.Each sample uses the first 48 ICU hours, and the task is an imbalanced binary classification problem.
- Experimental Setup: The model is compared with MISTS-only, Notes-only, and full-modality baselines under corresponding modality-availability settings.Under missing-modality settings, it receives the available modality and reconstructs the missing stream through prompt-based cross-modal generation.
- Main Results: The proposed method achieves the highest AUPRC in all four single-modality settings across the two cohorts.The gains are more consistent across metrics for Notes-only inputs, suggesting that reconstructing MISTS from notes is generally more stable than reconstructing notes from MISTS.
- Main Results: Under full observations, the method achieves the best F1 and second-best AUROC on MIMIC-IV.Compared with MMIMIC, it substantially improves F1 while showing only modest decreases in AUROC and AUPRC.
- Ablation Study: With all prompts enabled, AUPRC reaches 0.5853, while removing any prompt reduces AUPRC to varying degrees.The prompts respectively reconstruct missing modalities, distinguish feature provenance, condition interactions on missingness, and aggregate temporal information by availability pattern.
- Ablation Study: Training-time modality dropout produces a non-monotonic robustness trend because insufficient dropout limits incomplete-case exposure whereas excessive dropout reduces observed supervision.The analysis suggests that moderate-to-high dropout encourages stronger missing-modality compensation and cross-modal interaction.
5 Conclusion
The paper presents a unified prompt-learning framework for mortality prediction from irregular, incomplete EHRs and reports complementary roles for its four prompt types. Future work will test broader tasks, datasets, modalities, and more complex missingness patterns.
- The framework combines generative, missing-signal, missing-type, and temporal prompts to handle missing modalities in irregular EHRs.The prompts align incomplete inputs with the full-modality decision space.
- Ablation studies confirm that the four prompts contribute complementary capabilities to robust mortality prediction.
- Future evaluations should cover broader clinical prediction tasks and datasets, more modalities, and complex real-world missingness patterns.