PaperlayerView exact revision on arXiv

Source-linked AI summary

MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images

Nasir Hayat, Krzysztof J. Geras, Farah E. Shamout

arXiv:2207.07027v2eess.IVcs.CVcs.LG

TL;DR

Healthcare data are collected asynchronously, making assumptions that all modalities are available unrealistic. MedFuse processes modality representations as an LSTM sequence to accommodate missing modalities; it remains robust on unimodal samples and improves paired multimodal samples.

  • Problem

    Healthcare data are collected asynchronously, making the assumption that all modalities are available for every sample infeasible in routine clinical practice.

    Source: §1 · p. 2

  • Method

    MedFuse treats modality representations as a sequence of unimodal tokens and aggregates them with an LSTM, accommodating variable-length inputs when a modality is missing.

    Source: §1 · p. 2

  • Results

    MedFuse remains robust across unimodal samples and improves performance for paired multimodal samples, achieving state-of-the-art results without assuming correlation between modalities.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors conclude MedFuse is a better choice than baseline methods because its fusion module handles partially paired data and its architecture and training procedure provide performance gains.

    Source: §6 · p. 15

    Evaluation is limited to clinical time-series and chest X-ray integration from one data source and two benchmark tasks; settings with missing time-series data and available X-rays were not tested.

    Source: §6 · p. 16

Abstract

from arXiv · show

Multi-modal fusion approaches aim to integrate information from different data sources. Unlike natural datasets, such as in audio-visual applications, where samples consist of "paired" modalities, data in healthcare is often collected asynchronously. Hence, requiring the presence of all modalities for a given sample is not realistic for clinical tasks and significantly limits the size of the dataset during training. In this paper, we propose MedFuse, a conceptually simple yet promising LSTM-based fusion module that can accommodate uni-modal as well as multi-modal input. We evaluate the fusion method and introduce new benchmark results for in-hospital mortality prediction and phenotype classification, using clinical time-series data in the MIMIC-IV dataset and corresponding chest X-ray images in MIMIC-CXR. Compared to more complex multi-modal fusion strategies, MedFuse provides a performance improvement by a large margin on the fully paired test set. It also remains robust across the partially paired test set containing samples with missing chest X-ray images. We release our code for reproducibility and to enable the evaluation of competing models in the future.

1. Introduction

Healthcare data are often sparse, asynchronous, and not fully paired, while publicly available multimodal clinical benchmarks are limited. MedFuse uses an LSTM fusion module for variable-length modality inputs; the paper reports robust performance on unimodal samples and improvement on paired samples.

1. Introduction

  • Clinical modalities are often not paired: physiological data are collected more frequently than chest X-rays, and the inputs differ in dimensions, noise, and timing.The paper identifies these characteristics as challenges for joint fusion in healthcare.Source: §1 · p. 2
  • The proposed LSTM fusion module treats modality representations as a sequence and can handle missing data during training and inference.Unlike concatenation-based joint fusion, its recurrence aggregates modality tokens and accommodates variable-length input sequences.Source: §1 · p. 2
  • The study links MIMIC-IV clinical time-series data with MIMIC-CXR images to introduce benchmark results for mortality prediction and phenotype classification.The authors also release preprocessing and benchmark code for reproducibility and future evaluation of competing models.Source: §1 · pp. 2–3
  • MedFuse remains robust on unimodal samples and improves performance for paired multimodal samples.The authors also report state-of-the-art results without assuming correlation between modalities.Source: §1 · p. 3

Generalizable Insights about Machine Learning in the Context of Healthcare

  • The authors frame MedFuse as a flexible, encoder-agnostic approach that may extend beyond clinical time series and chest X-rays.They describe sequence processing as an alternative to conventional concatenation and characterize multimodal fusion's promise for downstream healthcare tasks.Source: Generalizable Insights about Machine… · pp. 3–4

2. Related Work

Prior multimodal research spans natural data and medical imaging, but common assumptions about shared structure or paired inputs may not fit heterogeneous clinical data. Work combining clinical data and images includes several fusion strategies, with specific constraints around incomplete samples and scalability.

2.1. Multi-modal Learning

  • Multimodal learning has addressed tasks such as visual grounding, action recognition, image captioning, and visual-question answering, often across audio, visual, and textual inputs.The passage cautions that assumptions of shared intrinsic and structural information may not hold for heterogeneous healthcare data.Source: §2.1 · p. 4

2.2. Multi-modal Fusion with Medical Images

  • Medical-image fusion spans pixel-level combinations of complementary views, feature- or prediction-level classification and segmentation, and uses of radiology reports as additional modalities.The cited examples include image captioning, report generation, and zero-shot image classification.Source: §2.2 · p. 4

2.3. Multi-modal Fusion with Clinical Data and Medical Images

  • MedFuse's overview describes independently pretrained modality-specific encoders, followed by projecting the chest X-ray representation and feeding modality representations to an LSTM fusion module.The module's last hidden state is used for the overall prediction.Source: Figure 2 · p. 5, §2.3 · p. 5
  • Clinical-image fusion studies include joint latent refinement and weighted late fusion, while late fusion requires both modalities to share labels.A related partially paired phenotype-classification method uses a separate classifier for every possible input-modality combination, which the authors describe as not scalable.Source: §2.3 · p. 5

3. Methodology

The methodology first learns modality-specific representations, then combines them with MedFuse, an LSTM fusion module that processes the available modality sequence.

3. Methodology

  • The approach first learns modality-specific latent features, then integrates them through the joint multimodal fusion module MedFuse.The method focuses on clinical time-series data and chest X-ray images.Source: §3 · p. 6

3.1. Modality-specific Encoders

  • Separate clinical time-series LSTM and chest X-ray ResNet-34 encoders produce modality-specific features and are independently pretrained with their respective labels and losses.The clinical encoder uses the last hidden state of a stacked LSTM, while the image encoder uses features after ResNet-34 average pooling.Source: §3.1 · p. 6

3.2. The MedFuse Module

  • MedFuse projects chest X-ray features to the clinical feature dimension, then an LSTM aggregates modality representations recurrently; clinical time-series is the always-present base modality and prediction target.When the chest X-ray is missing, the fusion LSTM can process a single-element sequence of clinical features.Source: §3.2 · pp. 6–7

4. Experiments

The experiments link MIMIC-IV clinical time-series data with MIMIC-CXR images to evaluate mortality prediction and phenotype classification. They describe data preparation, splits, MedFuse training, baseline comparisons, and model selection.

4.1. Datasets and Benchmark Tasks

  • The benchmarks pair ICU time-series data with the last chest X-ray from the same stay to predict 25 phenotypes or in-hospital mortality after the first 48 ICU hours.The tasks are evaluated with AUROC and AUPRC; stays shorter than 48 hours are excluded for mortality prediction.Source: §4.1 · pp. 7–8

4.1.1. Pre-processing of Clinical Time-series Data

  • Clinical inputs use 17 variables sampled every two hours, then discretized, standardized, and one-hot encoded into 76 features per time step.The variables comprise five categorical and 12 continuous measures.Source: §4.1.1 · p. 8

4.1.2. Data Splits

  • Patient-level splits allocate 70% of data to training, 10% to validation, and 20% to test, distinguishing paired samples from samples with missing chest X-rays.The partial phenotyping training set contains 7,756 samples with X-rays among 42,628 total samples.Source: §4.1.2 · pp. 8–9

4.2. Training Strategy with the MedFuse Module

  • MedFuse training pre-trains modality-specific encoders, then fine-tunes the encoders, projection layer, and fusion module on partially paired data.The image encoder uses CXRUNI and 14 radiology labels for pre-training; task-specific clinical encoders use EHRPARTIAL.Source: §4.2 · p. 9

4.3. Baseline Models

  • Baselines include early and joint fusion, MMTM, DAFT, a clinical-only LSTM, and Unified, with paired-data requirements varying by method.Early and joint fusion are also tested on partially paired data using a learnable vector for missing chest X-rays.Source: §4.3 · pp. 9–10

4.4. Model Training and Selection

  • Hyperparameters are selected over 10 runs using validation AUROC, and the MedFuse variant is chosen by validation performance on partially paired data.The selected variant, MedFuse (OPTIMAL), uses the best percentage of EHR-only samples for fine-tuning.Source: §4.4 · p. 11

5. Results

MedFuse performs best among the compared approaches on paired and partially paired test sets, while its phenotype-wise gains are larger for mixed and chronic conditions than acute ones. Results also vary by age group, with the youngest group showing a decrease in AUROC that the authors say needs further investigation.

5. Results

  • 10% of unimodal samples is optimal for in-hospital mortality prediction, versus 20% for phenotype classification.These percentages are reported for experiments varying the share of unimodal training samples.Source: §5 · p. 11

5.3. Performance Results in the Partially Paired Setting

  • 0.768 AUROC and 0.429 AUPRC for phenotype classification, and 0.874 AUROC and 0.567 AUPRC for in-hospital mortality: MedFuse (OPTIMAL) achieves the best performance on the partially paired test set.The result uses paired-sample fine-tuning with 20% unimodal samples for phenotype classification and 10% for mortality.Source: Table 3 · p. 12, §5.3 · p. 13

5.1. Performance Results in the Uni-modal & Multi-modal Settings

  • 0.770 AUROC and 0.481 AUPRC for phenotype classification, and 0.865 AUROC and 0.594 AUPRC for in-hospital mortality: MedFuse achieves the best paired-test performance when chest X-rays are used during training and inference.The comparison is to a unimodal stacked LSTM; similar but less significant trends appear on the larger partially paired test set, where 18.8% of phenotype and 26.2% of mortality samples are paired.Source: §5.1 · p. 12

5.2. Performance Results in the Paired Setting

  • 0.770 AUROC and 0.481 AUPRC for phenotype classification, and 0.865 AUROC and 0.594 AUPRC for in-hospital mortality: MedFuse (OPTIMAL) achieves the best performance across both tasks in the paired setting.The comparison includes early fusion, joint fusion, and the Unified approach.Source: §5.2 · p. 12

5.4. Phenotype-wise Analysis

  • Mixed-condition AUROC rises from 0.749 to 0.800 and AUPRC from 0.458 to 0.565; chronic-condition AUROC rises from 0.717 to 0.745 and AUPRC from 0.487 to 0.512.Acute-condition gains are smaller: AUROC rises from 0.761 to 0.772 and AUPRC from 0.432 to 0.433; improvements also occur across a number of thorax-related phenotypes.Source: §5.4 · pp. 13–14, Table 4 · p. 14

5.5. In-hospital Mortality Age-wise Analysis

  • AUROC and AUPRC improve with MedFuse for age groups 40–60, 60–80, and over 80, but AUROC decreases for ages 18–40.The youngest test group has only 11 positive samples; AUPRC increases by 24% for ages 40–60 and 1.3% for ages 60–80.Source: Table 5 · p. 14, §5.5 · p. 15

6. Discussion

MedFuse integrates partially paired clinical time-series data and chest X-rays, with better performance than the uni-modal LSTM baseline and flexibility for missing images. The discussion highlights new public benchmarks while noting that evaluation is limited in scope.

6. Discussion

  • MedFuse performs better than the uni-modal LSTM baseline and several baselines, with gains attributed to missingness handling and the architecture-training combination.The authors report that the size of the partially paired training set does not seem correlated with performance improvements.Source: §6 · p. 15
  • The method can process samples with missing chest X-ray images and does not assume correlation among input modalities.The authors note that learning cross-modal interactions is future work because clinical modalities may be sparse, asynchronous, and have different labels.Source: §6 · pp. 15–16
  • The fusion approach can scale beyond two modalities without amending the fusion loss function, but this scalability requires evaluation.Source: §6 · pp. 15–16
  • The work introduces public benchmark results and preprocessing resources, but evaluates only two tasks using time-series data and chest X-rays from a single source.The authors also identify interpretability and evaluation with more than two modalities as future work.Source: §6 · p. 16

A.1. Image Augmentations

Chest X-ray images receive random augmentations during training and center cropping during validation and testing.

A.1. Image Augmentations

  • Training images are resized to 256 × 256, randomly flipped, transformed affinely, and cropped to 224 × 224; validation and test images use resizing and center cropping.Source: §A.1 · p. 22

A.2. Hyperparameter Search Results

The hyperparameter search sampled learning rates and selected the best-performing configurations, including layer placements for MMTM and DAFT.

A.2. Hyperparameter Search Results

  • Table A1 reports the learning rates that achieved the best results after ten runs with randomly sampled rates between 10−5 and 10−3.Source: §A.2 · p. 22, Table A1 · p. 22
  • For MMTM, placing the module after the fourth ResNet layer achieved the best performance for both tasks.Two placements, after the third or fourth ResNet layer, were tested for each task.Source: §A.2 · p. 22
  • For DAFT, the third ResNet layer performed best for phenotype classification, while the fourth performed best for in-hospital mortality.The two tested versions transformed the LSTM representation after the third or fourth ResNet layer.Source: §A.2 · p. 22

A.3. Percentage of Uni-modal Samples within the Training Set

Validation AUROC was best with a small fraction of uni-modal fine-tuning samples for both tasks.

A.3. Percentage of Uni-modal Samples within the Training Set

  • 10% uni-modal samples gave the best validation AUROC for in-hospital mortality, while 20% did so for patient phenotyping.The sampling percentage with the best validation AUROC was fixed across experiments unless noted otherwise.Source: §A.3 · p. 22, Figure A1 · p. 23
  • MedFuse’s best performance gains were achieved even with a small percentage of uni-modal samples.Source: §A.3 · p. 23

A.4. Percentage of Uni-modal Samples within the Paired Test Set

Randomly dropping chest X-ray images from paired test samples reduces AUROC for both in-hospital mortality and phenotyping.

A.4. Percentage of Uni-modal Samples within the Paired Test Set

A.5. Missing Modality with Early and Joint Fusion

The appendix compares missing-modality handling, varying uni-modal sample proportions, and an ensemble alternative. Zero substitution and a learnable missing-modality vector perform comparably; AUROC trends differ by task as uni-modal samples increase, while the ensemble slightly outperforms MedFuse for phenotyping only.

A.5. Missing Modality with Early and Joint Fusion

  • Zero substitution and a learnable vector for missing chest X-rays produce comparable results, with no obvious differences.Table A2 reports AUROC and AUPRC for missing-modality handling.Source: §A.5 · p. 24, Table A2 · p. 24

A.6. Percentage of Uni-modal Samples within the Partially Paired Test Set

  • As uni-modal samples increase in the partially paired test set, mortality AUROC rises, while phenotyping AUROC is more consistent.Confidence intervals narrow across both tasks as the uni-modal proportion increases; 0% is equivalent to the fully paired test set.Source: §A.6 · p. 24, Figure A3 · p. 24

A.7. Ensemble of uni-modal and multi-modal models

  • The ensemble slightly outperforms MedFuse for phenotyping only, but requires training two models.The ensemble pairs an EHR-only LSTM for unpaired input with a MedFuse paired-input model.Source: §A.7 · p. 25, Table A3 · p. 25

Related papers

Loading 2207.07027v2…