PaperlayerView exact revision on arXiv

Source-linked AI summary

DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency

Wenfang Yao, Kejing Yin, William K. Cheung, Jia Liu, Jing Qin

arXiv:2403.06197v1eess.IVcs.CVcs.LG

TL;DR

Clinical fusion of EHR and medical images must contend with missing modalities and patient- and target-dependent modality significance. DrFuse disentangles shared and modality-specific information and applies disease-aware attention; the paper reports that it significantly outperforms state-of-the-art models on MIMIC-IV phenotype classification.

  • Problem

    Clinical multimodal fusion must handle missing modalities and variation in each modality's significance across patients and prediction targets.

    Source: §1 · p. 1

  • Method

    DrFuse disentangles information shared across EHR and medical images from modality-specific information and uses disease-aware attention to weight modalities by patient and prediction target.

    Source: §1 · p. 2, §3.4 · p. 4

  • Results

    DrFuse significantly outperforms state-of-the-art models on phenotype classification in the real-world MIMIC-IV dataset.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support DrFuse as a clinical multimodal fusion model for settings with missing modalities and modal inconsistency.

    Source: §5 · p. 7

Abstract

from arXiv · show

The combination of electronic health records (EHR) and medical images is crucial for clinicians in making diagnoses and forecasting prognosis. Strategically fusing these two data modalities has great potential to improve the accuracy of machine learning models in clinical prediction tasks. However, the asynchronous and complementary nature of EHR and medical images presents unique challenges. Missing modalities due to clinical and administrative factors are inevitable in practice, and the significance of each data modality varies depending on the patient and the prediction target, resulting in inconsistent predictions and suboptimal model performance. To address these challenges, we propose DrFuse to achieve effective clinical multi-modal fusion. It tackles the missing modality issue by disentangling the features shared across modalities and those unique within each modality. Furthermore, we address the modal inconsistency issue via a disease-wise attention layer that produces the patient- and disease-wise weighting for each modality to make the final prediction. We validate the proposed method using real-world large-scale datasets, MIMIC-IV and MIMIC-CXR. Experimental results show that the proposed method significantly outperforms the state-of-the-art models. Our implementation is publicly available at https://github.com/dorothy-yao/drfuse.

1 Introduction

Clinicians combine EHR and medical images for diagnosis and prognosis, but real-world fusion must address missing modalities and patient-dependent differences in modality significance. DrFuse disentangles shared and modality-specific information and uses disease-aware attention; the introduction reports that it outperforms state-of-the-art models on MIMIC-IV phenotype classification.

  • Missing modalities are common in practice: fewer than 20% of MIMIC-IV patients have X-rays, and generating images from EHR is infeasible because EHR lacks detailed anatomical information.Late fusion can handle missing modalities but does not fully use cross-modal interactions.Source: §1 · p. 1
  • EHR and CXR can give conflicting mortality signals, and which modality matters can vary by a patient's condition, challenging consistent multimodal prediction.The passage gives meningitis and pneumothorax as examples of opposite modality signals.Source: §1 · pp. 1–2
  • DrFuse disentangles shared from modality-distinct information and uses disease-aware attention with a ranking loss to capture patient-specific modality significance.The shared information is intended to support prediction when one modality is unavailable.Source: §1 · p. 2
  • DrFuse significantly outperforms state-of-the-art models on phenotype classification in the real-world MIMIC-IV dataset.Source: §1 · p. 2

2 Related Work

Related work covers clinical multimodal prediction and diverse data combinations, alongside methods for missing modalities. Modal inconsistency has been studied in other domains but, according to this review, not addressed in clinical multimodal learning.

  • Clinical multimodal work spans prognosis, phenotyping, segmentation, and combinations of EHR, notes, ECG, omics, and medical imaging (Kline et al. 2022; Hayat, Geras, and Shamout 2022; Huang et al. 2020b; Venugopalan et al. 2021; Mohsen et al. 2022).Source: §2 · p. 2
  • Missing-modality approaches include late fusion, generative imputation, and shared or complementary representations, but highly heterogeneous EHR-image fusion remains an open challenge (Huang et al. 2020a; Yoo et al. 2019; Ma et al. 2021; Hayat, Geras, and Shamout 2022; Zhang et al. 2022; Chen et al. 2019; Shen and Gao 2019; Wang et al. 2023).Source: §2 · p. 2
  • Modal inconsistency has been investigated in fake-news detection and sentiment analysis, but not addressed in clinical multimodal learning.Source: §2 · p. 2

3 DrFuse: The Proposed Method

DrFuse separates shared and modality-specific information from EHR and chest X-rays, aligns shared representations for use when a modality is missing, and uses disease-aware attention to weight representations by patient and prediction target.

3.1 Notations

  • EHR time series and chest X-rays form paired data when both are present, while the full training set also includes patients with EHR alone.The method denotes these groups as paired and partial data, respectively.Source: §3.1 · pp. 2–3

3.2 Overview

  • DrFuse combines disentangled representation learning with disease-aware attention fusion to make predictions from patient- and target-specific representations.The final prediction component applies a feedforward layer to each target's fused representation.Source: §3.2 · p. 3, §3.5 · p. 5

3.3 Disentangled Representation Learning

  • Separate encoders produce shared and distinct representations for each modality; orthogonality constraints reduce redundancy between shared and modality-distinct information.EHR uses Transformer encoders and CXR uses ResNet50 encoders.Source: §3.3 · pp. 3–4
  • Jensen–Shannon divergence aligns the distributions induced by shared EHR and CXR representations, which logit pooling combines when both modalities are available.When CXR is missing, the EHR shared representation is used directly as the final shared representation.Source: §3.3 · pp. 3–4

3.4 Disease-aware Masked Attention Fusion

  • Disease-aware masked attention assigns representation weights for each prediction target, masking the CXR representation when that modality is unavailable.The module uses a query formed from available representations and target vectors for individual prediction targets.Source: §3.4 · p. 4

Attention ranking loss.

  • An attention ranking loss encourages representation weights to follow the ranking of their auxiliary predictive performance, penalizing violations by a margin.The penalty applies when one representation predicts better than another but receives no greater attention weight.Source: Attention ranking loss · p. 4

3.5 Learning Algorithms

  • Training combines distribution-alignment, disentanglement, auxiliary, attention-ranking, and final-prediction losses; for EHR-only patients, CXR-dependent terms are removed.The EHR shared representation is used directly when CXR is missing.Source: §3.5 · p. 5

4.1 Experiment settings

The experiments use MIMIC-IV and MIMIC-CXR to evaluate 25 disease phenotypes from early ICU data, with full and matched cohorts for different modality-availability settings. Performance is measured by macro-averaged PRAUC.

Datasets and preprocessing.

  • 59,344 ICU stays had EHR records, and 10,630 were associated with CXR from the same first-48-hour observation window.The cohort uses 17 routinely monitored clinical variables and the last Anterior-Posterior chest X-ray in that window.Source: Datasets and preprocessing · p. 5
  • The full dataset includes patients with or without CXR, while the matched subset contains only patients with both EHR and CXR.Data were split 7:1:2 for training, validation, and testing, with matched-subset validation and test patients also represented in the corresponding full-dataset splits.Source: Datasets and preprocessing · p. 5

Evaluation metrics.

4.2 Baseline Models

The comparison includes multimodal fusion modules, methods designed to handle missing CXR, and an EHR-only Transformer baseline. DrFuse outperforms the baselines across the reported settings.

4.2 Baseline Models

  • MMTM and DAFT exchange information across modalities; because they assume full modality, missing CXR inputs are replaced with zeros.The zero-input replacement is used during both training and testing.Source: §4.2 · p. 5
  • DrFuse consistently outperforms all baselines by a significant margin across the training and test subset combinations.Table 2 reports macro-average PRAUC over all 25 disease phenotype labels.Source: Table 2 · p. 6
  • MedFuse and MedFuse-II use LSTM-based fusion and revised ResNet50 and Transformer encoders, respectively, while Transformer alone uses EHR input.MedFuse learns a global representation for missing CXR; MedFuse-II replaces MedFuse’s encoders to align the comparison with DrFuse.Source: §4.2 · p. 6

4.3 Overall Performance of Disease Prediction

DrFuse improves disease-phenotype prediction over MedFuse in matched-subset and cross-subset evaluations. It also remains the top-performing method on the full test dataset despite a possible domain shift.

4.3 Overall Performance of Disease Prediction

  • 5.4% relative improvement over MedFuse was achieved with matched-subset training and testing, and 8% with full-dataset training and matched-subset testing.Both comparisons concern macro-average PRAUC over the 25 disease phenotype labels.Source: §4.3 · p. 6
  • All methods scored worse on the full test dataset than on the matched subset, suggesting a severe domain shift between the subsets.The authors suggest patients unable to undergo X-ray scans may have more complex health conditions and be harder to predict.Source: §4.3 · p. 6
  • DrFuse still achieves the best performance when tested on the full dataset despite the potential domain shift.The authors attribute this result to representation disentanglement.Source: §4.3 · p. 6

4.4 Disease-Wise Prediction Performance

Disease-wise results show that multimodal fusion does not help every phenotype equally, while DrFuse reduces or reverses selected performance drops. A t-SNE visualization also illustrates alignment of shared representations and separation of distinct ones.

4.4 Disease-Wise Prediction Performance

  • In the matched test subset, the t-SNE visualization shows aligned EHR and CXR shared-representation distributions and disentangled modality-distinct representations.The figure visualizes shared and distinct features for the test set.Source: Figure 3 · p. 6
  • For conduction disorders, MedFuse drops 40.5% against the best uni-modal prediction while DrFuse drops 1%; for other upper respiratory disease, DrFuse improves 24.2%.The results illustrate that the effect of combining EHR and CXR varies across diseases.Source: §4.4 · p. 6
  • DrFuse better addresses modal inconsistency by inferring patient- and disease-specific modal significance.The reported disease-wise results show that combining EHR and CXR is not always helpful for every disease.Source: §4.4 · p. 6

4.5 Visualization of Disentangled Representation

The section visualizes DrFuse’s shared and modality-distinct representations with t-SNE. The shared representations cluster together, while the distinct representations remain separated from one another and from the shared features.

4.5 Visualization of Disentangled Representation

  • t-SNE visualizes the shared and modality-distinct EHR and CXR representations.The visualization is presented in Fig. 3.Source: §4.5 · p. 6
  • The shared representations form a well-blended cluster.Source: §4.5 · p. 6
  • The distinct representations remain separated from each other and from the shared features.Source: §4.5 · p. 6

4.6 Ablation Study

The ablation study tests DrFuse components by removing or replacing them. Performance drops on missing-modality evaluation when disentanglement components or disease-wise attention are removed.

4.6 Ablation Study

  • Removing disentanglement components or replacing JSD and logit pooling causes a significant performance drop on the full dataset with missing modality.The compared models were trained on the matched subset.Source: §4.6 · p. 7
  • Capturing disease-wise modal significance is important for the disease prediction task, and the proposed method is effective in achieving this goal.Source: §4.6 · p. 7
  • The attention ablation indicates that capturing disease-wise modal significance is important for disease prediction.Source: §4.6 · p. 7

5 Conclusion

DrFuse fuses EHR and CXR by learning shared and modality-distinct representations, with disease-aware attention to capture patient-specific modality significance. Experiments show effectiveness on missing-modality and modal-inconsistency challenges; domain shift between patients with and without CXR remains a future direction.

5 Conclusion

  • DrFuse learns shared and modality-distinct representations, aligning shared features with JSD and disentangling them through orthogonal constraints.Source: §5 · p. 7
  • Logit pooling fuses shared representations, while disease-aware attention uses an attention ranking loss to capture patient-specific modality significance for each target.Source: §5 · p. 7
  • Experiments demonstrate effectiveness in representation disentanglement and addressing missing-modality and modal-inconsistency issues, with significant performance improvement.Source: §5 · p. 7
  • Future work will address domain shift between patients with and without CXR jointly with multimodal learning.Source: §5 · p. 7

Related papers

Loading 2403.06197v1…