Source-linked AI summary
Dipole: Diagnosis Prediction in Healthcare via Attention-based Bidirectional Recurrent Neural Networks
Fenglong Ma, Radha Chitta, Jing Zhou, Quanzeng You, Tong Sun, Jing Gao
TL;DR
Diagnosis prediction from longitudinal EHRs requires modeling temporal, high-dimensional visit sequences while interpreting predictions, but existing approaches have weaknesses on long sequences and visit relationships. Dipole combines bidirectional recurrent neural networks with three attention mechanisms to weight visits and interpret predictions. Experiments on two real-world EHR datasets report improved diagnosis-prediction accuracy and meaningful medical-code representations.
Problem
Diagnosis prediction must model temporally ordered, high-dimensional EHR visits and interpret prediction results, while existing RNN approaches weaken on long sequences and may ignore relationships among visits.
Method
Dipole embeds medical codes into low-dimensional representations and applies attention-based bidirectional recurrent neural networks with three attention mechanisms.
Results
Dipole achieves higher diagnosis-prediction accuracy than state-of-the-art approaches on two real-world EHR datasets and produces meaningful, interpretable medical-code representations.
Takeaways & Limitations
Attention mechanisms assign different weights to past visits, while bidirectional modeling uses information from previous and future visits during training to support diagnosis prediction.
Takeaways & Limitations
Future visit information is used only during training; testing provides the model only with past visit information.
Abstract
from arXiv · showhide
Predicting the future health information of patients from the historical Electronic Health Records (EHR) is a core research task in the development of personalized healthcare. Patient EHR data consist of sequences of visits over time, where each visit contains multiple medical codes, including diagnosis, medication, and procedure codes. The most important challenges for this task are to model the temporality and high dimensionality of sequential EHR data and to interpret the prediction results. Existing work solves this problem by employing recurrent neural networks (RNNs) to model EHR data and utilizing simple attention mechanism to interpret the results. However, RNN-based approaches suffer from the problem that the performance of RNNs drops when the length of sequences is large, and the relationships between subsequent visits are ignored by current RNN-based approaches. To address these issues, we propose {\sf Dipole}, an end-to-end, simple and robust model for predicting patients' future health information. Dipole employs bidirectional recurrent neural networks to remember all the information of both the past visits and the future visits, and it introduces three attention mechanisms to measure the relationships of different visits for the prediction. With the attention mechanisms, Dipole can interpret the prediction results effectively. Dipole also allows us to interpret the learned medical code representations which are confirmed positively by medical experts. Experimental results on two real world EHR datasets show that the proposed Dipole can significantly improve the prediction accuracy compared with the state-of-the-art diagnosis prediction approaches and provide clinically meaningful interpretation.
1 INTRODUCTION
Diagnosis prediction must model temporally ordered, high-dimensional EHR visits while making predictions interpretable. Dipole addresses these challenges with bidirectional recurrent modeling and three attention mechanisms, and is reported to improve diagnosis prediction on two real-world datasets.
- Research challenges: Diagnosis prediction uses historical, temporally ordered visits containing high-dimensional medical codes to predict future diagnoses.Each visit may vary in importance for the prediction.
- Research challenges: Existing RNN-based models can lose predictive power on long visit sequences, while location-based attention ignores relationships among all earlier visits.These limitations motivate modeling information from both temporal directions and measuring relationships among visits.
- Proposed approach: Dipole combines bidirectional recurrent neural networks with three attention mechanisms to model visits and weight previous visits for prediction.The model learns low-dimensional visit representations from embedded medical codes and uses them for future diagnosis prediction.
- Evaluation: Experiments on two large real-world EHR datasets report higher diagnosis-prediction accuracy than state-of-the-art approaches, alongside clinical-expert analysis of code representations.The paper also studies attention weights assigned to past visits and validates the reasonableness of the bidirectional architecture.
- Proposed approach: Dipole is designed to predict future visit information while interpreting prediction results without depending on expert medical knowledge.Its learned representations are intended to make visit importance and medical-code meaning interpretable.
2 RELATED WORK
EHR mining spans several healthcare prediction and representation-learning tasks, with diagnosis prediction requiring both temporal modeling and interpretability. Dipole extends prior approaches by combining bidirectional visit modeling with attention mechanisms that weight past visits and support interpretation.
- EHR data mining: EHR mining includes tasks such as phenotyping, disease progression, adverse drug-event detection, and diagnosis prediction.Deep learning models are used across these healthcare informatics tasks.
- Diagnosis prediction: Med2Vec learns medical-code representations but ignores long-term dependencies among visits, whereas RETAIN provides interpretable reverse-time attention for binary prediction.GRAM uses graph-based attention for healthcare representation learning.
- Dipole’s distinction: Dipole differs from earlier predictive models by using bidirectional neural networks and attention mechanisms that assign different weights to past visits.The paper links these design choices to improved prediction performance and interpretable medical-code meanings.
- Attention-based models: Attention-based neural networks assign weights to input representations and combine them into a context vector for prediction.The paper notes that diagnosis prediction differs from translation because patient visits are in the same space.
3 METHODOLOGY
Dipole represents longitudinal EHR visits with code embeddings and bidirectional recurrent processing, then uses attention to select relevant prior visits for next-visit category prediction. The model is trained end-to-end while supporting interpretation of visit importance and medical-code representations.
- EHR Representation: Each visit is encoded as a binary vector over the dataset’s unique medical codes, with optional coarse-grained category representations derived from hierarchical coding systems.Diagnosis codes map to ICD-9 nodes and procedure codes to CPT nodes; category vectors represent parent categories.
- Visit and Sequence Modeling: Dipole embeds high-dimensional visit codes into low-dimensional vectors before processing them with a bidirectional recurrent neural network.The implementation uses GRU units to capture dependencies among patient visits.
- Attention Mechanism: Attention computes relative importance over previous hidden states and forms a context vector that is combined with the current hidden state for prediction.Dipole includes location-based, general, and concatenation-based attention mechanisms.
- Diagnosis Prediction: The attentional hidden state is passed through a softmax layer to predict category-level medical codes in the next visit, using cross-entropy over patient sequences for training.The proposed neural network is trained end-to-end.
- Interpretation: Interpretability is addressed by analyzing which visits receive attention and by using ReLU-based code representations intended to remain interpretable.The paper emphasizes understanding the clinical meaning of representation dimensions and identifying visits important for prediction.
- Visit and Sequence Modeling: The bidirectional network combines forward and reverse sequence information into latent visit representations, while using future visits only during training.At testing time, prediction receives only past visit information.
4 EXPERIMENTS
Experiments evaluate Dipole and related variants on Diabetes and Medicaid claims datasets, showing improved diagnosis prediction, interpretable attention, and advantages of bidirectional modeling for long sequences.
- 4.1 Data Description: Two Medicaid claims datasets contain 147,810 patients and 1,055,011 visits, and 22,820 diabetes patients with 466,732 visits, respectively.Visits were grouped or aggregated by week, with minimum-visit exclusions applied.
- 4.3 Results of Diagnosis Prediction: Dipole and its RNN variants achieve higher accuracy than the baselines on the Diabetes dataset, while RETAIN outperforms Med2Vec on Medicaid.The datasets differ in disease diversity and medical-code dimensionality, affecting baseline behavior.
- 4.3 Results of Diagnosis Prediction: RNN has the lowest accuracy on both datasets, whereas attention-based variants use prior visits more fully and perform better than RNN.The reported explanation is that standard RNN predictions depend mainly on recent visits and cannot memorize all past information.
- 4.3 Results of Diagnosis Prediction: Dipolec performs best on Diabetes and Dipolel on Medicaid, indicating benefits from bidirectional visit modeling and assigning different weights to visits.All three Dipole variants outperform baselines and RNN variants on Medicaid; Dipolel and Dipolec outperform them on Diabetes.
- 4.4 Case Study: Attention weights vary across patients and identify clinically relevant visits, including hypertension- and diabetes-related visits contributing to a sixth-visit prediction.For one patient, the reported attention vector is α = [0.2386, 0.0824, 0.3028, 0.3762].
- 4.6 Assumption Validation: Dipole− outperforms RNN across visit-count groups, with the difference increasing sharply as sequences become longer.The experiments attribute this pattern to bidirectional networks retaining more information for long patient-visit sequences.
5 CONCLUSIONS
Dipole addresses diagnosis prediction by combining bidirectional recurrent modeling with attention mechanisms that support prediction and interpretation. Experiments on two real-world EHR datasets support its effectiveness and clinically meaningful representations.
- Diagnosis prediction remains difficult because RNN-based approaches may not remember all previous visits, while location-based attention ignores relationships between current and past visits.
- Dipole uses bidirectional recurrent neural networks to retain information from previous and future visits while modeling EHR data.
- Three attention mechanisms measure relationships among visits, allowing prediction results to be interpreted through differing visit weights.
- Experiments on two large real-world EHR datasets demonstrate Dipole’s effectiveness for diagnosis prediction.
- The learned medical-code representations are meaningful, and attention mechanisms assign different weights to previous visits when predicting future visit information.