PaperlayerView exact revision on arXiv

Source-linked AI summary

An explainable Transformer-based deep learning model for the prediction of incident heart failure

Shishir Rao, Yikuan Li, Rema Ramakrishnan, Abdelaali Hassaine, Dexter Canoy, John Cleland, Thomas Lukasiewicz, Gholamreza Salimi-Khorshidi, Kazem Rahimi

arXiv:2101.11359v1cs.LG

TL;DR

Predicting complex chronic conditions from EHRs is challenging because robust evidence is limited and deep-learning models have shown modest performance and poor explainability. The authors developed and validated a model for incident heart failure using routine EHR data, finding calendar year added stronger predictive information than age.

  • Problem

    Robust prediction evidence for complex chronic conditions is limited, while deep-learning models on complex EHRs have shown modest performance and poor explainability.

    Source: Introduction · p. 3

  • Method

    The authors developed and validated an incident heart failure prediction model using large-scale routine EHR, contextualizing clinical information within a machine-learning approach.

    Source: Discussion · pp. 14–15

  • Results

    Calendar year provided additional and stronger information than age for accurate incident heart failure prediction, while diseases and medications were strong predictors.

    Source: Discussion · p. 15

  • Takeaways & Limitations

    The approach generates hypotheses rather than causal conclusions, and the authors call for additional confirmatory studies.

    Source: Discussion · p. 16

    The study used limited EHR information and validated prediction performance on only one dataset, CPRD.

    Source: Study Limitations · p. 16

Abstract

from arXiv · show

Predicting the incidence of complex chronic conditions such as heart failure is challenging. Deep learning models applied to rich electronic health records may improve prediction but remain unexplainable hampering their wider use in medical practice. We developed a novel Transformer deep-learning model for more accurate and yet explainable prediction of incident heart failure involving 100,071 patients from longitudinal linked electronic health records across the UK. On internal 5-fold cross validation and held-out external validation, our model achieved 0.93 and 0.93 area under the receiver operator curve and 0.69 and 0.70 area under the precision-recall curve, respectively and outperformed existing deep learning models. Predictor groups included all community and hospital diagnoses and medications contextualised within the age and calendar year for each patient's clinical encounter. The importance of contextualised medical information was revealed in a number of sensitivity analyses, and our perturbation method provided a way of identifying factors contributing to risk. Many of the identified risk factors were consistent with existing knowledge from clinical and epidemiological research but several new associations were revealed which had not been considered in expert-driven risk prediction models.

Introduction · Model architecture

The paper addresses the limited performance and explainability of existing heart-failure risk prediction by developing a Transformer model using longitudinal, multimodal electronic health records. It describes the cohort selection and model architecture used to predict incident heart failure.

Introduction

  • Heart failure remains a major cause of morbidity, mortality, and economic burden, while its incidence has changed little and its absolute burden has risen with population growth and ageing.Existing statistical models for incident heart failure had largely unsatisfactory predictive performance.Source: Introduction · p. 3
  • Deep-learning models had shown modest performance on complex EHR datasets, and poor explainability limited their trustworthiness and clinical use.Source: Introduction · p. 3
  • The study aimed to develop and validate a deep-sequential model for predicting incident heart failure from temporal, multimodal electronic health records.Source: Introduction · p. 3

Methods

  • The study used quality-standard CPRD primary-care records linked to secondary-care and administrative data, retaining patients with complete linkage and records from 1985 to 2015.The database was described as nationally representative by age, sex, and ethnicity.Source: Methods · p. 4
  • The prediction cohort required at least 10 GP or hospital visits, three years of records, and 10 unique codes; incident heart failure was the first heart-failure diagnosis code during the study period.The broader representation-learning cohort required at least five visits and included 1,609,024 patients.Source: Methods · p. 4
  • The prediction cohort contained 100,071 patients, including 13,050 (13%) incident heart-failure cases.Source: Methods · p. 5

Model architecture

  • The study extended BEHRT, a Transformer architecture reported by Li et al10, to capture disease and medication associations in temporal context.BEHRT was selected for sequential data and prior performance on subsequent-visit prediction tasks.Source: Model architecture · p. 5
  • Disease and medication encounters, age, and calendar year were represented with trainable embeddings and combined into a single predictor for each encounter.The encounter, age, and calendar-year layers are summed.Source: Model architecture · p. 5

Statistical analysis

The analysis evaluated heart-failure prediction using internal cross-validation and held-out validation, and assessed model interpretability by estimating how clinical encounters contributed to predictions. It also tested input-modality combinations and stratified encounter contributions by age and calendar year.

Statistical analysis

  • Dataset B was split 60%/20%/20% into training, testing, and external held-out validation cohorts; internal validation used five-fold cross-validation and AUROC and AUPRC with 95% CIs.Source: Statistical analysis · p. 5
  • Ablations tested six combinations of diagnoses, medications, age, and calendar year; the model was also compared directly with replicated RETAINEX12.The tested combinations were D, DA, DAY, DM, DMA, and DMAY.Source: Statistical analysis · pp. 5–6
  • A perturbation method inspired by Guan et al13 estimated predictor contributions from changes in predictive probability, using an asymmetric loss to prioritize encounters that enhanced HF/non-HF predictions.Large perturbations with little probability change indicated unimportant predictors, whereas minimal perturbations that greatly changed probability indicated important predictors.Source: Statistical analysis · p. 6
  • Repeated diagnoses or prescriptions counted only at first occurrence; relative contribution was the average contribution in HF patients divided by that in non-HF patients, with values above or below 1.0 indicating positive association with HF or non-HF, respectively.Predictors generally required occurrence in at least 5% of patients, while established HF risk factors from statistical models were included even below that prevalence threshold [14,15].Source: Statistical analysis · p. 6
  • Encounter contributions were stratified by five age groups and four calendar-year groups in Dataset C, a subset of external held-out testing and validation patients where BEHRT (DMAY) performed best.Source: Statistical analysis · p. 7

Dataset preparation

The dataset used encounter-level diagnoses, medications, age, and calendar year. The section reports external-validation performance, the relative value of calendar-year contextualization, and cohort characteristics.

Dataset preparation

  • Encounter-level inputs included 299 diagnostic codes, 426 medication codes, age in months, and calendar year; unmapped disease and medication codes were assigned to UNKNOWN.Source: Dataset preparation · p. 7
  • Among 100,071 patients, median baseline age was 70 years, 52% were men, 65% had hypertension, 9% prior myocardial infarction, and 5.1% ischemic stroke; median follow-up was 9 years.Source: Dataset preparation · p. 7
  • Calendar year as the contextualizing variable showed greater AUROC/AUPRC improvements than age, indicating it was more informative for contextualizing predictors.The accompanying visualized analysis is in appendix section 6.Source: Dataset preparation · p. 8, Figure 2 · p. 8
  • Quintile-AUROC was 0.84 for the first predictive-probability quintile and 0.71 for the fifth.These were the best-performing quintiles when BEHRT (DMAY) was evaluated on the combined test and external-validation dataset.Source: Dataset preparation · p. 8
  • BEHRT with all four modalities (DMAY) achieved an external-validation AUROC of 0.93 and AUPRC of 0.70, with 3%/9% absolute AUROC/AUPRC improvement over RETAINEX12 in the full cohort.Figure 2 presents the model-performance evaluation.Source: Dataset preparation · pp. 7–8, Figure 2 · p. 8

Encounter contribution results

Relative-contribution analyses identified disease and medication predictors of heart failure, with contributions generally strongest at ages 50–60 and attenuated in older groups. Untreated-subgroup and calendar-year analyses showed how treatment context and temporal change can alter the interpretation of these associations.

Encounter contribution results

  • Disease and medication contributions were generally higher at ages 50–60 and lower in older groups; bacterial diseases, lower respiratory infections, myocardial infarction, pleural effusion, corticosteroid/antibacterial drugs, bronchodilators, and acne and rosacea drugs were positively associated with HF.Source: Encounter contribution results · pp. 9–10

Discussion

  • Disease–medication pairs often reflected treatments for diagnoses, so medication associations could proxy disease risk rather than indicate an independent drug effect.The discussion cautions that relative-contribution findings generate hypotheses and require confirmatory studies; causal interpretation should be made with great caution.Source: Encounter contribution results · p. 10, Discussion · p. 16

Encounter contribution results

  • Ischemic stroke, myocardial infarction, diabetes, hypertension, and atrial fibrillation/flutter all had relative contributions above 1, with age patterns similar to those of the top diagnoses and medications.Their individual discriminatory contributions were limited in older people.Source: Encounter contribution results · p. 10, Figure 4 · p. 11
  • Treatments for hypertension, diabetes, and atrial fibrillation, including digoxin, had relative contributions below 1 and were associated with lower HF risk [23].The analyses therefore examined untreated subgroups to disentangle disease–medication relationships.Source: Encounter contribution results · pp. 11–12, Figure 5
  • Among untreated hypertensive patients, relative contributions were 1.06 (95% CI 0.82–1.37) at ages 70–75 and 0.98 (95% CI 0.78–1.23) at ages 75–80, showing no evidence of a protective effect.Across untreated subgroups, relative contribution increased for 20 of 25 disease cases.Source: Encounter contribution results · pp. 12–13, Figure 5
  • After 2000, glaucoma treatment with prostaglandin analogues had relative contribution below 1; digoxin’s relative contribution remained stable even as its prescriptions waned after 2005.The authors described these patterns as potentially preventative signals, not evidence of causation.Source: Encounter contribution results · p. 14, Figure 6 · p. 14, Figure 6A

Discussion

  • The discussion presents contextualized clinical information and relative-contribution analysis as ways to explain predictions, highlighting diseases and medications as strong predictors and calendar year as an informative context.Year-stratified medication occurrences made quantitatively different contributions to prediction, which would be missed without temporal context.Source: Discussion · p. 15

Study Limitations

The study’s conclusions are limited by coarse diagnosis phenotyping, restricted EHR information, CPRD-only validation, and cohort selection that may limit generalizability to low-risk patients with fewer encounters.

Study Limitations

  • Diagnosis codes were mapped to 299 disease categories, potentially losing detail and reflecting expert preferences; limited EHR information and CPRD-only validation also constrain assessment.Future studies could examine whether measurements, blood tests, and additional demographics improve accuracy, explainability, and transferability.Source: Study Limitations · p. 16
  • Selecting patients with sufficient records for robust predictions may compromise generalizability to low-risk groups with fewer clinical encounters.Source: Study Limitations · p. 16
  • The authors suggest routine EHR use of BEHRT could support targeted prevention or clinical-trial recruitment, while the data-driven risk-factor approach could generate hypotheses for causal exploration.Source: Study Limitations · p. 17
  • Some medications contributed negatively to HF prediction; while some treat established HF risk factors, others may warrant study for drug repurposing.The model and analysis could also be applied to more deeply phenotyped populations to investigate disease mechanisms and patterns in other complex conditions.Source: Study Limitations · p. 17

Supplementary Material · 1. Details for BEHRT model · 2. Visualised importance analysis for age and year

BEHRT represents sequential medical encounters alongside each encounter’s age and calendar year. In the visualised analysis, calendar-year variation produced greater embedding dissimilarity than age variation, suggesting year or birth-cohort context was more informative for incident heart-failure prediction.

1. Details for BEHRT model

  • BEHRT treats diagnoses and medications in a doctor or hospital visit as encounter tokens and separates visits with a SEP element.This adapts the token-and-separator structure used in NLP and BERT to sequential medical histories.Source: §1 · p. 25
  • The model orders sequential medical-history data, following an annotation approach similar to the original BERT model.Visits are separated by a SEP element.Source: §1 · p. 25
  • Each encounter is represented using three information layers: the encounter’s diagnoses and/or medications, patient age, and calendar year.Source: §1 · p. 25

2. Visualised importance analysis for age and year

  • The cosine-similarity analysis compared summed encounter and calendar-year or age embeddings, treating greater dissimilarity as evidence of greater modality importance.The analysis was conducted to validate and visualise the contribution of age and year.Source: §2 · p. 25
  • Calendar-year embeddings showed dissimilarity from 0 to 0.8, compared with 0 to 0.3 for age-in-months variation, across four common diseases (Figs S2A and S2B).The authors interpret the greater year-related dissimilarity as indicating greater sensitivity to calendar year than to patient age.Source: §2 · p. 25
  • Calendar year or birth-cohort effect was suggested to be more informative than chronological age for incident heart-failure prediction because it captures time-sensitive context.This interpretation follows the greater sensitivity of predictor representations to year variation than to age variation.Source: §2 · p. 25

3. Details of perturbation methods · 4. Discussion of Dataset C

The perturbation method adds an asymmetric loss to prioritize class-appropriate changes in predicted heart-failure probability and estimates predictor contributions. Contribution analyses used selected, correctly predicted patients from Dataset C.

3. Details of perturbation methods

  • The method adds an asymmetric loss to Guan et al.’s approach and its information-entropy-based loss term to prioritize learning perturbations.The full loss is defined over one patient’s medical history.Source: §3 · p. 25
  • For heart-failure patients, the method prioritizes perturbations that increase outcome probability; for non-heart-failure patients, it prioritizes those that decrease it.The asymmetric weighting penalizes perturbations using an alpha constant.Source: §3 · p. 26
  • The method learns allowable predictor variances, with the maximum variance set by a user-defined hyperparameter of 0.5 in this work.The learned perturbation vector contains one value per predictor.Source: §3 · p. 26
  • Predictor contribution is represented by 0.5 − ϵ1, so lower ϵ1 indicates higher contribution to heart-failure prediction at the patient-encounter level.The analysis first establishes contribution for a particular disease or medication encounter in time.Source: §3 · p. 26

4. Discussion of Dataset C

  • Dataset C contribution analyses used a subset of external testing and validation patients for whom BEHRT with DMAY performed best.The authors considered the learned latent space most effective for these patients, making their resulting contribution analyses reliable.Source: §4 · p. 26
  • Incorrectly predicted patients were excluded because outcome misclassification would invalidate population-wise and age-stratified contribution analyses.The authors selected data from the best-performing groups for these analyses.Source: §4 · pp. 26–27

5. Details of disease and medication phenotyping · 6. Details of model training

The study represented longitudinal diagnoses and prescriptions as age- and calendar-contextualized encounters, mapped codes into clinical disease and medication groups, and pretrained BEHRT before heart-failure prediction. Phenotyping used strict incident diagnosis codes, while training used Bayesian-tuned model settings and masked-language pretraining.

5. Details of disease and medication phenotyping

  • Clinical encounters combined diagnoses, medications, patient age in months, and calendar year, with prescription records included as new time-stamped encounters.Each encounter represented a recorded diagnosis or medication.Source: §5 · p. 27
  • Because CPRD records medications in six-week increments, medications made up the largest number of encounters.Source: §5 · p. 27
  • The CPRD dictionary mapped Read and ICD-10 diagnosis codes into 56,624 unique codes, then Caliber grouped them into 299 clinically meaningful disease categories.The phenotyping method was previously published and clinically validated.Source: §5 · p. 27
  • Medication codes were classified using British National Formulary sections prevalent in the population, producing 426 unique medication groups.Unknown disease and medication mappings were assigned to an “UNKNOWN” category.Source: §5 · p. 27
  • The heart-failure phenotype used strict incident Read and ICD-10 diagnosis codes from primary and secondary care, excluding historical-diagnosis codes.The phenotype was defined by Caliber.Source: §5 · p. 27

6. Details of model training

  • After 20 Bayesian-optimization iterations, the selected model had 4 layers, hidden size 120, 6 attention heads, and intermediate size 108.The search tuned the number of layers, hidden size, and intermediate size; the model was implemented in PyTorch.Source: §6 · p. 27
  • BEHRT was pretrained on dataset A with unsupervised masked language modelling, then applied to heart-failure prediction on dataset B.The pretraining task randomly masked encounters in patients’ medical histories and predicted them to learn predictors and their temporal relationships.Source: §6 · pp. 27–28

7. Temporal trends of analgesics

Analgesic prescriptions shifted from predominantly non-opioid to opioid-based after 1995 as calendar years increased. BEHRT and RC reflected this generational change, and RC attenuated in later decades.

7. Temporal trends of analgesics

  • BEHRT paralleled the generational shift in analgesic prescription from non-opioid to opioid-based drugs.Source: §7 · p. 28
  • Non-opioid analgesics, primarily NSAIDs and generally associated with increased cardiovascular-event risk12,13, contributed to quite high RC before 1995.Source: §7 · p. 28
  • RC captured the shift from predominantly non-opioid analgesic prescriptions before 1996 toward opioid-based prescriptions after 1995, attenuating in later decades as calendar years increased.Analgesic prescriptions showed a similar declining trend overall.Source: §7 · p. 28

8. Additional results

Additional analyses compared AUROC across predicted-risk strata and examined age- and year-based embeddings, alongside relative-contribution summaries for common diseases, medications, and medically validated risk factors.

Related papers

Loading 2101.11359v1…