PaperlayerView exact revision on arXiv

Source-linked AI summary

REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models

Yinghao Zhu, Changyu Ren, Shiyun Xie, Shukai Liu, Hangyuan Ji, Zixiang Wang, Tao Sun, Long He, Zhoujun Li, Xi Zhu, Chengwei Pan

arXiv:2402.07016v1cs.AI

TL;DR

Existing EHR knowledge approaches often focus on structured data, leaving unstructured records and richer medical context less represented. REALM retrieves and aligns medical knowledge for multimodal EHR data, and reports state-of-the-art performance on MIMIC-III mortality and readmission tasks.

  • Problem

    Prior KG approaches focus mainly on structured medical data, overlooking unstructured EHR information and richer semantic medical knowledge.

    Source: §1 · p. 2

  • Method

    REALM uses LLMs to extract task-relevant entities, matches them to a professional medical KG, and fuses the retrieved knowledge with multimodal EHR representations.

    Source: §1 · p. 2

  • Results

    REALM achieves state-of-the-art performance on MIMIC-III in-hospital mortality and 30-day readmission tasks.

    Source: §7 · p. 9

  • Takeaways & Limitations

    REALM offers a framework for enhancing EHR representations with external knowledge and LLMs.

    Source: §7 · p. 9

    The formulation treats external knowledge graphs as supplemental knowledge-base references for patient representations.

    Source: §3 · p. 3

Abstract

from arXiv · show

The integration of multimodal Electronic Health Records (EHR) data has significantly improved clinical predictive capabilities. Leveraging clinical notes and multivariate time-series EHR, existing models often lack the medical context relevent to clinical tasks, prompting the incorporation of external knowledge, particularly from the knowledge graph (KG). Previous approaches with KG knowledge have primarily focused on structured knowledge extraction, neglecting unstructured data modalities and semantic high dimensional medical knowledge. In response, we propose REALM, a Retrieval-Augmented Generation (RAG) driven framework to enhance multimodal EHR representations that address these limitations. Firstly, we apply Large Language Model (LLM) to encode long context clinical notes and GRU model to encode time-series EHR data. Secondly, we prompt LLM to extract task-relevant medical entities and match entities in professionally labeled external knowledge graph (PrimeKG) with corresponding medical knowledge. By matching and aligning with clinical standards, our framework eliminates hallucinations and ensures consistency. Lastly, we propose an adaptive multimodal fusion network to integrate extracted knowledge with multimodal EHR data. Our extensive experiments on MIMIC-III mortality and readmission tasks showcase the superior performance of our REALM framework over baselines, emphasizing the effectiveness of each module. REALM framework contributes to refining the use of multimodal EHR data in healthcare and bridging the gap with nuanced medical context essential for informed clinical predictions.

1 Introduction

REALM addresses gaps in multimodal EHR knowledge integration by retrieving and aligning medical knowledge for clinical notes and time-series data. It combines this knowledge with EHR representations and reports stronger performance on MIMIC-III mortality and readmission tasks.

1 Introduction

  • Prior KG-based EHR approaches primarily extract structured concepts and relations, while neglecting unstructured modalities and richer semantic medical knowledge.The introduction also notes hallucination risk when LLMs generate KG content.Source: §1 · pp. 1–2
  • The framework applies RAG to connect unstructured EHR modalities with an external knowledge graph through LLM semantic reasoning.It is designed for clinical notes and time-series EHR data.Source: §1 · p. 2
  • The stated challenges are consistently matching entities extracted from complex EHR data without hallucinations and incorporating retrieved knowledge without sacrificing interpretability or adding complexity.The modalities include clinical notes and multivariate time-series data.Source: §1 · p. 2
  • REALM uses LLM prompting to retrieve task-relevant entities, aligns them to a professionally labeled medical knowledge graph, and encodes extended entity descriptions to capture semantic context.The process includes detailed checking and alignment intended to prevent hallucinations and preserve consistency.Source: §1 · p. 2
  • REALM demonstrates superior performance on MIMIC-III mortality and readmission tasks, with experiments also assessing module effectiveness, robustness to fewer training samples, entity quality, and offline privacy.The framework’s operations are conducted offline.Source: §1 · p. 2

2 Related Work

Prior multimodal EHR work develops methods for handling missing modalities and learning across records, but often gives limited attention to clinical background knowledge. KG-based approaches incorporate structured medical knowledge, while leaving semantic information in unstructured EHR data underused.

2.1 Multimodal EHR Learning

  • Multimodal EHR methods address missing data, irregular time intervals, visit-note learning, and modality-aware attention, but the surveyed approaches often omit clinical background knowledge.Examples include latent-space imputation, time attention, Gromov-Wasserstein contrastive learning, and modality-aware attention.Source: §2.1 · pp. 2–3

2.2 Incorporating External Knowledge for EHR

  • KG-based EHR methods use ontology hierarchies, patient-disease interactions, and visit histories, while LLM-generated graphs such as GraphCare face content hallucination challenges.These approaches predominantly draw knowledge from structured medical data, overlooking semantic information in unstructured EHRs.Source: §2.2 · p. 3

3 Problem Formulation

The formulation treats clinical prediction as binary classification using patient EHR modalities and an external knowledge graph. The two EHR modalities are represented separately, with temporal information captured alongside them.

3 Problem Formulation

  • The formulation represents structured time-series data and unstructured clinical notes as separate modalities, with a timestamp vector encoding visit times.The time-series matrix has T visits and F numeric features.Source: §3 · p. 3
  • External knowledge graphs serve as supplemental knowledge-base references for individual patient representations..Source: §3 · p. 3
  • The prediction objective is binary classification for in-hospital mortality and readmission using EHR data and an external knowledge graph.The objective combines time-series data, clinical-note data, visit times, and KG information to predict a label.Source: §3 · p. 3

4 Methodology

REALM encodes clinical notes and time-series EHR data, retrieves KG knowledge from modality-specific entities, and adaptively fuses the resulting representations.

4.1 Overview

  • REALM’s three modules extract multimodal EHR embeddings, retrieve and encode KG knowledge, and adaptively fuse representations for downstream prediction.The framework uses GRU for time series, an LLM for notes, and semantic retrieval to match extracted entities with KG nodes.Source: §4.1 · pp. 3–4

4.2 Multimodal EHR Embedding Extraction

  • GRU encodes time-series data, while an LLM encodes clinical notes and sinusoidal embeddings represent visit timestamps.The sinusoidal transformation is described as capturing visit order and cyclical continuity.Source: §4.2 · pp. 4–5

Extract Entities from Multimodal EHR Data

  • For time series, REALM extracts features whose z-scores exceed a manually selected threshold as abnormal entities, and prompts an LLM to extract disease entities from notes.The threshold is selected by inspecting entities across a manually divided subset; note entities undergo post-processing to reduce hallucinations.Source: Extract Entities from Multimodal EHR… · pp. 5–6

Match extracted entities with external KG

  • REALM embeds KG nodes and extracted entities with the same LLM, then uses cosine similarity and a threshold to identify related nodes.The threshold is chosen by examining matching across a subset and manually verifying the results.Source: Match extracted entities with external… · p. 6

Encode KG Knowledge

  • Matched KG node names, definitions, and descriptions are concatenated and encoded by an LLM as supplementary knowledge.When no entities or matching nodes are available, an instruction text is used instead of an empty string.Source: Encode KG Knowledge · p. 6

4.4 Multimodal Fusion Network

  • The fusion network applies self-attention within each modality, then cross-attention between modalities, and pools their outputs into a fused representation.Residual connections and BatchNorm are applied to attention layers and feed-forward networks; a single-layer MLP produces predictions from the fused representation.Source: §4.4 · pp. 6–7

5 Experimental Setups

Experiments use MIMIC-III mortality and readmission tasks, PrimeKG, and multimodal EHR data. Evaluation compares four binary-classification metrics across EHR, knowledge, text-embedding, and fusion baselines.

5.1 Dataset, KG and Task Description

  • MIMIC-III experiments predict in-hospital mortality and 30-day readmission using time-series records, clinical notes, and PrimeKG knowledge.The dataset split contains 10,776 training, 1,539 validation, and 3,080 test samples.Source: §5.1 · p. 7

5.2 Evaluation Metrics

  • AUROC, AUPRC, min(+P, Se), and F1 evaluate performance, with higher values indicating better results.AUROC is the primary metric; AUPRC is noted as useful for imbalanced datasets.Source: §5.2 · p. 7

5.3 Hyperparameters

  • REALM uses a hidden dimension of 312 and learning rate of 6e-4, while baseline hyperparameters are selected by grid search.Experiments use batch size 256 and report mean±std. over 10 bootstrap runs.Source: §5.3 · p. 7

5.4 Baseline Models

  • Baselines include multimodal EHR and KG models, three text-embedding approaches, and four multimodal fusion methods.Text embeddings are BERT [CLS], BGE-M3, and Qwen-7B; fusion methods are Add, Concat, Tensor Fusion, and MAG.Source: §5.4 · p. 7

6 Experimental Results

REALM outperforms the best baseline on MIMIC-III mortality and readmission tasks, and its RAG-enhanced modalities, adaptive fusion, and Qwen-7B embeddings show advantages in the reported comparisons. It also performs robustly under data sparsity, while the retrieved-entity case study reports clinically important entities.

Comparing Each Modality with RAG-Enhancement

  • RAG-enhanced time-series and text modalities each significantly improve model performance in the ablation study.The authors interpret these results as evidence that external medical knowledge captures more complex semantic background knowledge.Source: Comparing Each Modality with… · pp. 7–8

6 Experimental Results

  • Against the best baseline, REALM's mortality/readmission relative gains were AUROC 1.09%/2.06%, AUPRC 2.75%/4.75%, min(+P, Se) 0.79%/3.12%, and F1 21.90%/30.39%.Table 2 reports in-hospital mortality and 30-day readmission results on MIMIC-III.Source: §6 · p. 7, Table 2 · p. 8

Comparing Different Fusion Network

  • The adaptive self- and cross-attention fusion network outperforms all baseline fusion methods on both tasks, and REALM exceeds its reduced versions.The authors attribute the advantage to attentive integration of modality-specific features.Source: Comparing Different Fusion Network · p. 8

Comparing Text Embedding Models

  • Qwen-7B significantly outperforms BERT and BGE-M3 across all metrics in the text-embedding comparison.The passage notes Qwen-7B's extensive training data and long-context support.Source: Comparing Text Embedding Models · p. 8

Robustness to Data Sparsity

  • REALM outperforms recent SOTA models at multiple missingness levels in the MIMIC-III mortality AUPRC evaluation.The evaluation artificially reduces dataset completeness by 20%, 40%, 60%, and 80%.Source: Figure 5 · p. 8, Robustness to Data Sparsity · p. 8

Evaluation of Quality of Retrieved Entities

  • In the clinical-note case study, atrial fibrillation and sepsis have the highest entity-importance scores, followed by osteomyelitis and renal failure.The authors report that the disease entities are relatively dangerous in clinical practice and associate them with high mortality probability.Source: Evaluation of Quality of Retrieved… · p. 8, Figure 6 · p. 8

7 Conclusions

REALM combines time-series EHR, clinical notes, and external knowledge graphs for healthcare prediction. It achieves SOTA performance on MIMIC-III in-hospital mortality and 30-day readmission tasks.

7 Conclusions

  • REALM is a multimodal EHR representation-learning framework incorporating time-series data, clinical notes, and an external knowledge graph.The conclusion highlights LLM semantic reasoning, long-context encoding, and the KG's medical context.Source: §7 · p. 8
  • REALM achieves SOTA performance on MIMIC-III in-hospital mortality and 30-day readmission prediction tasks.The authors describe this as demonstrating the effectiveness of incorporating knowledge from external knowledge bases.Source: §7 · p. 9
  • The authors position REALM as a step toward more effective EHR use through representations enriched with external knowledge and LLMs.Source: §7 · p. 9

Ethical Statement

The study reports ethical safeguards for its MIMIC-based EHR analysis, including de-identification and responsible data handling.

Ethical Statement

  • The study uses a de-identified MIMIC dataset available through a data use agreement and reports steps to minimize potential harm and promote fairness.Source: Ethical Statement · p. 9

A.1 EHR Prediction Models

The section surveys multimodal EHR and KG prediction baselines, text-embedding alternatives, and fusion methods used for comparison.

A.1 EHR Prediction Models

  • Baselines span multimodal EHR models addressing missing modalities, irregular timing, and structured–unstructured synthesis, plus KG methods using ontology ancestry, graph attention, and patient-disease interactions.Named baselines include M3Care, MPIM, UMM, VecoCare, GRAM, KAME, CGL, and KerPrint.Source: §A.1 · p. 12

A.2 Text Embedding Approaches

  • Text embedding comparisons include BERT [CLS] (512-token clipping; 768 dimensions), BGE-M3 (up to 8192 tokens; 1024 dimensions), and Qwen-7B (up to 8192 tokens).BERT notes exceeding 512 tokens are clipped; BGE-M3 and Qwen-7B support contexts up to 8192 tokens.Source: §A.2 · p. 12

A.3 Multimodal Fusion Baselines

  • Fusion baselines are element-wise addition, concatenation, Tensor Fusion’s multimodal tensor representation, and MAG’s adaptive weighting of modality features.The section describes Add, Concat, Tensor Fusion (TF), and MAG as comparison methods.Source: §A.3 · p. 12

B Experimental Environment

The experiments use a specified GPU and software setup with AdamW and validation-AUROC early stopping; the prompt appendix gives disease-extraction and retrieved-knowledge templates.

B Experimental Environment

  • Models train for 30 epochs on an Nvidia RTX 3090 server with 64GB RAM, using AdamW and validation-AUROC early stopping with patience 5.The environment specifies CUDA 12.2, Python 3.11, PyTorch 2.0.1, and PyTorch Lightning 2.0.5.Source: B Experimental Environment · p. 12

C Prompt Details

  • The retrieved-knowledge prompt asks an experienced doctor to combine the patient record’s condition severity with the severity of the related diseases.The appendix presents templates for entity extraction and combining retrieved knowledge with the KG.Source: C Prompt Details · p. 13
  • The extraction prompt requests disease-only NER from a medical case, with JSON output restricted to disease names mentioned in the input.Its example uses an abdominal radiology report and an entity array that includes “small bowel obstruction.”Source: C Prompt Details · p. 13
Loading 2402.07016v1…