PaperlayerView exact revision on arXiv

Source-linked AI summary

A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction

Weimin Lyu, Xinyu Dong, Rachel Wong, Songzhu Zheng, Kayley Abell-Hart, Fusheng Wang, Chao Chen

arXiv:2208.10240v2cs.CL

TL;DR

Narrative clinical notes contain complementary information but remain challenging to leverage and interpret alongside structured EHR data. The paper proposes a time-aware multimodal transformer and interpretation methods for in-hospital mortality prediction. It reports that the model outperforms other methods, with AUCPR 0.538, AUCROC 0.877, and F1 0.490.

  • Problem

    Clinical notes contain complementary information but remain challenging to leverage and interpret in predictive modeling alongside structured EHR data.

    Source: §1 · p. 2

  • Method

    A multimodal transformer jointly models time-series clinical variables and clinical-note representations, using fine-tuned Clinical BERT and interpretation methods for both modalities.

    Source: §1 · p. 2, §2.2 · p. 3, §2.3 · p. 4

  • Results

    The model outperforms other methods in in-hospital mortality classification, achieving AUCPR 0.538, AUCROC 0.877, and F1 0.490.

    Source: §3.1 · p. 6

  • Takeaways & Limitations

    Integrated gradients and Shapley methods visualize important note words and clinical features to assist interpretation of prediction outcomes.

    Source: §1 · p. 2, §3.6 · p. 9

    The reported results are not directly comparable to the Harutyunyan et al. benchmark.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Deep-learning-based clinical decision support using structured electronic health records (EHR) has been an active research area for predicting risks of mortality and diseases. Meanwhile, large amounts of narrative clinical notes provide complementary information, but are often not integrated into predictive models. In this paper, we provide a novel multimodal transformer to fuse clinical notes and structured EHR data for better prediction of in-hospital mortality. To improve interpretability, we propose an integrated gradients (IG) method to select important words in clinical notes and discover the critical structured EHR features with Shapley values. These important words and clinical features are visualized to assist with interpretation of the prediction outcomes. We also investigate the significance of domain adaptive pretraining and task adaptive fine-tuning on the Clinical BERT, which is used to learn the representations of clinical notes. Experiments demonstrated that our model outperforms other methods (AUCPR: 0.538, AUCROC: 0.877, F1:0.490).

1 INTRODUCTION

The paper addresses the challenge of combining structured clinical variables with complementary information in narrative notes. It proposes a time-aware multimodal transformer for in-hospital mortality prediction and reports promising performance alongside methods for interpreting both modalities.

1 INTRODUCTION

  • Clinical notes contain contextual information about symptoms, disease course, and treatment, while structured clinical variables such as vital signs are commonly modeled as time series.The introduction notes that domain-specific BERT variants can better handle clinical notes than normally pretrained BERT.Source: §1 · pp. 1–2
  • Prior multimodal approaches often concatenate separately extracted features, while the authors identify limited transformer use in EHR prediction and challenges modeling modality complexity and time.The introduction also notes that naive feature concatenation might reduce performance.Source: §1 · p. 2
  • AUCPR 0.538, AUCROC 0.877, and F1 0.490 are reported for the proposed multimodal transformer predicting in-hospital mortality.The authors describe these results as highly promising.Source: §1 · p. 2
  • The model adjusts note and clinical-variable representations into a shared space across time, then jointly learns their representation with a transformer and fine-tuned Clinical BERT.The paper also uses integrated gradients to visualize important note words and Shapley methods to identify important structured features.Source: §1 · p. 2

2 METHODS

The method processes MIMIC-III clinical variables and notes separately, then combines their representations with a transformer to predict in-hospital mortality.

2.1 STUDY DATASET

  • The study uses MIMIC-III ICU data, preprocessing 17 clinical variables as time series and treating each visit with notes as a sample.The dataset contains 14,068 training, 3,086 validation, and 3,107 test samples; the preprocessing step makes results not directly comparable to Harutyunyan et al.’s benchmark.Source: §2.1 · pp. 2–3

2.2 SINGLE MODEL EMBEDDING

  • Clinical notes are embedded with Clinical BERT fine-tuned on MIMIC-III mortality prediction, while clinical variables are encoded separately as a 76-dimensional time series.The model freezes MBERT weights when extracting note embeddings; categorical variables use one-hot encoding and numerical variables retain a continuous value.Source: §2.2 · pp. 3–4

2.3 MULTIMODAL EMBEDDING

  • Separate notes and time-series encoders project each modality, and a multimodal encoder maps their concatenation into a shared space for transformer processing.The first 48 ICU hours are treated as tokens with sinusoidal position embeddings; the T0 token is selected as the final multimodal representation.Source: §2.3 · pp. 4–5, Figure 3 · p. 5

2.4 OVERVIEW ARCHITECTURE

  • The final prediction concatenates the multimodal representation with the notes embedding; the architecture uses ReLU projections, a sigmoid output, cross-entropy loss, L2 regularization, and Adam.Figure 5 presents the overview architecture.Source: §2.3 · p. 5, §2.4 · pp. 5–6, Figure 5 · p. 6

3 RESULTS

The Multimodal Transformer outperformed other methods on in-hospital mortality prediction, while analyses of clinical-note words and structured variables highlighted clinically meaningful prognostic signals. The results also examine BERT pretraining and fine-tuning and discuss challenges in interpreting individual words.

3.1 PREDICTION RESULTS ANALYSIS

  • AUCPR 0.538, AUCROC 0.877, and F1 0.490; the models outperformed other methods on in-hospital mortality classification.The evaluation used the MIMIC III task results in Table 3.Source: §3.1 · p. 6, Table 3 · p. 6

3.2 DOMAIN ADAPTIVE PRETRAINING AND TASK ADAPTIVE FINE-TUNING ON BERT MODELS

  • MBERT performed best and general-purpose BERT worst among the tested note-only variants; the experiments support using properly trained language models for clinical notes.The BERT experiments used MIMIC III clinical notes without clinical variables.Source: §3.2 · p. 7, Table 4 · p. 7

3.3 CLINICAL NOTES VISUALIZATION AND INTERPRETATION

  • Pain ranked as the most important clinically meaningful note indicator; other high-importance indicators reflected pulmonary pathology, acute illness, and mental status.The authors note that “care” can be difficult to interpret in isolation because it appears in multiple contexts.Source: §3.3 · pp. 7–8
  • Figure 7 summarizes high-IG note tokens: word-cloud size represents frequency, while darker green marks greater importance in the illustrated note.The word cloud aggregates the top ten IG words from each note.Source: Figure 7 · p. 8, §3.3 · p. 8

3.4 CLINICAL VARIABLES FEATURE ANALYSIS

  • The top-ranked structured variables reflected disease severity, including mental status, respiratory status, oxygenation, and hemodynamics.Higher supplemental oxygen need increased predicted mortality, whereas higher systolic blood pressure decreased it.Source: §3.4 · pp. 8–9, Figure 8 · p. 9

3.5 DISCUSSION

  • The transformer’s attention mechanism lets multimodal fusion consider information from all other time stamps.The model projects notes and clinical variables into a shared space before jointly learning their representation.Source: §3.5 · p. 9
  • Negation can reverse a note word’s clinical meaning, and the authors identify NegEx as a future approach for handling it.“Crackles” and “not crackles” can indicate different exam findings.Source: §3.5 · p. 9

3.6 CONCLUSION

  • The conclusion presents the Multimodal Transformer as a way to fuse clinical notes with clinical variables while accounting for time-series information.The authors also report that the model outperformed other methods and describe IG and Shapley-based interpretation analyses.Source: §3.6 · p. 9

Related papers

Loading 2208.10240v2…