PaperlayerView exact revision on arXiv

Source-linked AI summary

EHRAdapt: Adapting Pretrained Language Models to Electronic Health Records with Semantic Priors for Rare Clinical Events

Andre R Goncalves, Vincent Liu, Priyadip Ray

arXiv:2609.34007v2cs.AI

TL;DR

Structured EHRs encode events as codes, modalities, and times, which pretrained LLMs do not directly process and text serialization makes longer. EHRAdapt maps events into frozen LLMs using semantic priors and learned residuals; it performs best across the evaluated backbones and downstream tasks, with complementary benefits across event frequencies.

  • Problem

    Text serialization lengthens EHR histories and leaves event type and timing implicit, while long-tailed clinical event frequencies leave rare events with few observations.

    Source: §1 · pp. 1–2

  • Method

    EHRAdapt maps structured events into a frozen LLM using learned modality embeddings, temporal attention biases, and event vectors combining projected BioLORD semantics with low-rank residuals.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    EHRAdapt achieves the lowest test loss across three backbones, and outperforms evaluated baselines on downstream classification, with both components improving rare-disease prediction.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The semantic prior benefits rare events most, while the residual improves overall likelihood but degrades prediction for the rarest events, supporting evaluation across frequencies and downstream tasks.

    Source: §6 · p. 10

    The evaluation excludes laboratory and vital-sign values and demographic information, and is conducted within a single health system.

    Source: §6 · p. 11

Abstract

from arXiv · show

Electronic health records (EHRs) encode clinical histories as (time, modality, code) tuples, whereas pretrained language models expect text tokens. Serializing tuples as text inflates sequence length and repeatedly encodes the same structure. We introduce EHRAdapt, an adapter that maps tuples directly into a frozen language model's embedding space. Modality and time gaps enter through a learned embedding and attention biases, and event codes receive dedicated vectors. Learning them is challenging because clinical vocabularies are long-tailed, leaving rare events with few observations. EHRAdapt therefore represents each event vector as the sum of a semantic prior and an evidence residual. The prior is a frozen, ontology-trained biomedical embedding of the event's clinical description, mapped into the input space by a shared learned projection, so it supplies clinical meaning even when observations are scarce. The residual is a learned low-rank, event-specific correction that refines it as evidence accumulates. We perform continued pretraining on records from about 4 million patients with three frozen LLM backbones, OLMo2 1B, Llama3.2 1B, and OLMo2 7B, training only the adapter (0.1--0.6% of all parameters). The full adapter outperforms all ablations in held-out next-event prediction on every backbone. Removing the semantic pathway hurts rare events most, over ten times more than the most frequent ones, whereas removing the residual hurts overall prediction but improves it for the rarest events. On reportable infectious-disease and syndromic classification, EHRAdapt outperforms text-based LLM and count-based baselines and the same LLMs adapted to the same records through next-event prediction, and both pathways improve rare-disease discrimination. The two pathways therefore play complementary roles, visible only when results are broken down by event frequency.

1 INTRODUCTION

Text serialization of structured EHR events lengthens patient histories and leaves event type and timing implicit, while long-tailed event frequencies leave rare events with few observations. EHRAdapt maps events into a frozen LLM and combines semantic priors with learned residuals; it achieves the lowest test loss across three backbones, with complementary pathway effects and downstream gains.

1 INTRODUCTION

  • Text serialization produces about 27 tokens per event and encodes event type and timing only implicitly.This increases the length of patient histories.Source: §1 · p. 1
  • The median event type appears about 20 times in training, compared with about 87.6 million occurrences for the most common type.Rare events may carry important diagnostic information, but have few EHR observations; average loss can obscure their effects.Source: §1 · p. 2
  • EHRAdapt represents each event as one token, encodes event type with learned embeddings, and supplies temporal information through attention biases.Its semantic prior uses BioLORD embeddings projected into the LLM input space, alongside an event-specific residual.Source: §1 · p. 2
  • The authors expect semantic priors to matter more for rare events and residuals to adapt more strongly for common events, while noting the model does not enforce this behavior.They evaluate next-event prediction by ten event-frequency groups and test downstream classification for reportable infections and syndromes.Source: §1 · p. 2
  • EHRAdapt achieves the lowest test loss across all three backbones, and outperforms evaluated baselines on downstream classification.The semantic prior and event-specific residual play complementary roles, and both improve rare-disease prediction.Source: §1 · p. 2
  • Removing the semantic prior degrades prediction across event frequencies, most strongly for rare events; removing the residual improves the rarest-event predictions but worsens overall performance.The residual contributes more to common-event representations, while both components improve rare-disease prediction.Source: §1 · p. 2

2 RELATED WORK

Prior work learns reusable EHR representations through EHR-specific pretraining or adapts pretrained LLMs to structured sequences. EHRAdapt addresses sparse observations of rare events by integrating external semantic knowledge with evidence learned from EHR data.

2 RELATED WORK

  • EHR foundation models learn reusable patient representations from large longitudinal corpora, while adapting pretrained LLMs can reduce the need for EHR-specific pretraining.The paper notes that EHR-specific training must be repeated to benefit from newer architectures.Source: §2 · p. 2
  • GRAM addresses rare clinical concepts by attending over ontology ancestors, whereas EHRAdapt can use semantic priors from external sources such as biomedical language models and knowledge graphs.EHRAdapt balances priors with EHR-learned representations according to available evidence.Source: §2 · p. 3
  • EHRAdapt targets rare-event data scarcity by adaptively integrating semantic priors with EHR evidence, unlike PORTER’s primarily portability-focused fixed semantic representations.The paper describes this as a complementary problem to PORTER (Guo et al., 2026).Source: §2 · p. 3
  • Models pretrained on clinical notes can process natural language directly, whereas structured EHR codes and timestamps must first be mapped into an LLM-compatible form.The paper positions EHRAdapt’s structured-EHR focus as complementary to clinical-note models.Source: §2 · p. 3

3 METHOD

EHRAdapt continued-pretrains a frozen LLM on next-event prediction from structured patient histories, then reuses the adapted model for downstream tasks. Its method represents events with semantic and evidence pathways, ties the output head to event vectors, and adds learned biases for elapsed time.

3 METHOD

  • Continued pretraining learns EHRAdapt through next-event prediction, after which the adapted model is reused for downstream tasks with a task-specific head or, untested here, further parameter-efficient adaptation.The architecture maps structured trajectories to next-event predictions.Source: §3 · p. 3

3.1 PROBLEM SETUP AND SEQUENCE CONSTRUCTION

  • A patient history is a timestamped sequence of event codes and modalities, and continued pretraining predicts the next code from the preceding sequence using cross-entropy in nats.Each code indexes an event type or special token; each event also has a modality and timestamp.Source: §3.1 · p. 3

3.2 EVENT REPRESENTATION

  • Each event vector adds a projected frozen BioLORD semantic embedding to a low-rank event-specific residual; the input also adds a learned modality embedding.The shared semantic map projects BioLORD embeddings to the backbone hidden dimension, while the residual uses a learned event code and shared projection.Source: §3.2 · p. 3
  • The low-rank residual factorization uses fewer parameters than an unconstrained event table, but greater rank does not guarantee better held-out performance, particularly for rare events.For hidden dimension 2,048, rank 64 uses about 5.15 million residual parameters versus 161 million for an unconstrained table.Source: §3.2 · p. 4
  • A fixed semantic projection preserves BioLORD angles and cosine-neighbor rankings, while the learned projection can adapt the semantic component to the backbone.The fixed projection with and without a residual is evaluated as a control.Source: §3.2 · p. 4
  • The semantic pathway is shared across the vocabulary and reaches rare events through BioLORD geometry, whereas each event has its own residual code with a shared projection.Updates to the shared semantic map driven by frequent events can move semantically related rare events.Source: §3.2 · p. 4

3.3 OUTPUT HEAD, FREQUENCY BIAS, AND LOGIT SCALE

  • The tied output head reuses event-code vectors, learns an event bias and logit scale, and excludes modality embeddings from output representations.Its factorized scores avoid materializing a dense event table on each forward pass.Source: §3.3 · pp. 4–5

3.4 TEMPORAL ATTENTION BIAS

  • Learned attention biases add elapsed-time information using 16 buckets, including a bucket for simultaneous events.The biases are initialized to zero, leaving backbone attention unchanged at the start of training.Source: §3.4 · p. 5

4 EXPERIMENTAL SETUP

The study continued-pretrains EHRAdapt and evaluates next-event prediction and transfer to two biosurveillance-related diagnostic tasks. Its setup compares adapter components across three frozen backbones and evaluates downstream representations with shared probes and baselines.

4 EXPERIMENTAL SETUP

  • EHRAdapt is continued-pretrained and evaluated for next-event prediction and transfer to two biosurveillance-related diagnostic tasks.The full adapter and four ablations are trained on three backbones to separate component effects from backbone-family or size differences.Source: §4 · p. 5

4.1 CONTINUED-PRETRAINING DATA

  • The KPNC cohort contains approximately 4 million patients from 2010–2018, split by patient into 98% training, 1% validation, and 1% test sets.Trajectories are chunked into 16–512 events; each training epoch contains approximately 721 million prediction targets.Source: §4.1 · p. 5

4.2 BACKBONES AND CONFIGURATIONS

  • Three frozen backbones provide a within-family size comparison with OLMo 2 1B and 7B and a cross-family comparison with Llama 3.2 1B.Ablations remove or fix event-code pathways; modality embeddings, temporal biases, output bias, and logit scale remain learned. OLMo 1B also receives a within-modality permutation control and residual-rank sweep.Source: §4.2 · pp. 5–6

4.3 BIOSURVEILLANCE TASKS AND COHORTS

  • Patient-level biosurveillance tasks classify reportable infections and ICD-defined syndromes from clinical histories before the index diagnosis.The syndromic task has four classes and approximately 118 thousand examples; the infection task has 14 classes and approximately 19.1 thousand examples, including six rare-disease classes. Evaluation expands training windows from 2019 through 2021 and tests the following year.Source: §4.3 · p. 6

4.4 DOWNSTREAM MODELS AND BASELINES

  • Frozen representations are evaluated with ℓ2-regularized logistic-regression probes, while baselines isolate text input, BioLORD representations, next-event adaptation, and event counts.Native text uses the same backbones; BioLORD uses averaged event embeddings; NEP adapts the same backbones on serialized EHR histories, and random forests use event counts.Source: §4.4 · pp. 6–7
  • Backbone comparison and task-specific fine-tuning are outside the study’s scope.The authors note that medical language models could serve as backbones and that representations could be fine-tuned for individual tasks.Source: §4.4 · p. 7
  • EHRAdapt and NEP provide a data-matched comparison, both adapting on essentially all 3.95 million training patients with frozen backbones and comparable trainable parameter counts.NEP serializes histories at about 27 tokens per event and supervises one sampled event per history, whereas EHRAdapt uses one token per event and supervises every position; the CPT corpus ends before all downstream index dates.Source: §4.4 · p. 7

5 RESULTS

EHRAdapt outperforms the reported downstream baselines and has the lowest global continued-pretraining test cross-entropy across three backbones. Its semantic prior and event-specific residual contribute differently by event frequency, while the residual also changes learned representation geometry.

5.1 CONTINUED-PRETRAINING PERFORMANCE

  • The Full model has the lowest global test CE among five configurations on all three backbones; learning the semantic projection improves CE in every data stratum.Table 1 reports global CPT test CE in nats, with mean and sample SD across three seeds.Source: §5.1 · p. 7, Table 1 · p. 8
  • Removing the semantic pathway hurts rare-event prediction most, while removing the residual improves D1–D3 but hurts medium-to-high-frequency events; the curves cross around D5.Frequent events dominate aggregate CE, so large rare-event effects have little influence on total loss.Source: §5.1 · pp. 7–8, Figure 2 · p. 8
  • Shuffling BioLORD vectors raises test CE globally and in every stratum, indicating that the semantic pathway benefits from matching each event to its clinical description, not vector geometry alone.The permutation preserves the vector set and geometry while breaking event–meaning links.Source: §5.1 · p. 8

5.2 DOWNSTREAM PREDICTIVE PERFORMANCE

  • EHRAdapt outperforms every reported downstream baseline on every view and backbone, including Rare-6 AUPRC (0.190 to 0.233–0.247) and four-class syndromic AUPRC (0.732 to 0.791–0.801).The strongest baseline varies by task; the Full model also beats same-corpus next-event adaptation.Source: §5.2 · pp. 8–9, Table 2 · p. 9

5.3 REPRESENTATION GEOMETRY

  • The learned semantic projection largely retains BioLORD’s nearest neighbors while changing similarity values; adding residuals further reorganizes neighborhoods, particularly for frequent events.Residuals also reduce average angular distortion relative to BioLORD in local and global comparisons.Source: §5.3 · p. 9, Figure 3 · p. 10

6 CONCLUSION

EHRAdapt combines a semantic prior with a learned residual to adapt frozen LLMs to structured EHR events. Its components play complementary roles across next-event prediction and rare-disease classification.

6 CONCLUSION

  • Across three backbones, the full model had the best test likelihood; the semantic prior benefited rare events most, while the residual improved overall likelihood but degraded prediction for the rarest events.Both components improved rare-disease classification.Source: §6 · p. 10
  • Continued pretraining on longitudinal EHR data transferred effectively to downstream tasks and outperformed text-based inputs and LLMs adapted on the same corpus through next-event prediction.The evaluated baselines also included count-based baselines.Source: §6 · p. 10
  • The evaluation excluded laboratory and vital-sign values and demographic information, and was conducted within a single health system.The authors identify incorporating these features and assessing generalizability across healthcare systems as future work.Source: §6 · pp. 10–11

A DATA AND PREPROCESSING

The appendices describe the continued-pretraining cohort, model configuration, task construction, and additional evaluation procedures, including frequency-stratified metrics.

A DATA AND PREPROCESSING

  • The cohort table reports patient-level splits and retains all 732M clinical events; the vocabulary contains 78,384 event types plus four reserved tokens.Training-split frequency strata use exact bin memberships, with repeated singleton boundaries making D1–D10 unequal in size.Source: Table 4 · p. 14, A DATA AND PREPROCESSING · p. 14

B BACKBONES AND PARAMETER COUNTS

C TRAINING CONFIGURATION

  • The backbone receives no gradient updates, while the adapter is optimized; the semantic projection, residual factors, and modality embedding are trained during continued pretraining.The projection is initialized to match the mean norm of pretrained token embeddings, and the output bias uses training-count log probabilities.Source: C TRAINING CONFIGURATION · p. 15

D.1 CLASS COUNTS

  • Syndromic and infectious-disease evaluation windows use earlier years for training and later years for testing, retaining all positive cases and capping controls at one matched control per case.The control-to-case ratio can therefore fall below 1:1.Source: §D.1 · p. 15

D.2 CLASS DEFINITIONS

  • Syndromic classification’s Known class samples patients diagnosed outside the two syndromic code lists, while controls are matched by demographics and geographic location.Known and Control class sizes were set to keep the classes balanced.Source: §D.2 · pp. 15–16

E ADDITIONAL CPT RESULTS

  • Additional CPT results report held-out test cross-entropy and top-1 and top-10 accuracy globally and by event-frequency stratum, using validation-selected checkpoints.Frequency strata are ranked by training-split frequency, and reported standard deviations reflect variability across training seeds rather than test-patient sampling.Source: E ADDITIONAL CPT RESULTS · p. 16

F PERMUTATION CONTROL

A vector-permutation control tests whether correct event-to-description correspondence contributes beyond the semantic vectors’ distribution. The comparison is limited to OLMo2-1B at residual rank 64.

F PERMUTATION CONTROL

  • 2.847 ± 0.006 versus 2.888 ± 0.014 global test CE for Full and permuted models, respectively; the paired reduction was 0.042 ± 0.020 nats.The Full model performed better in every stratum and seed.Source: F PERMUTATION CONTROL · p. 17
  • The Full model’s advantage was larger in D1 and D2 (2.406 ± 0.098 and 3.569 ± 0.266 nats) than in D10 (0.039±0.019).D10 contains approximately 98.60% of test targets, so rare-event effects carry little weight in the global average.Source: F PERMUTATION CONTROL · p. 17
  • The comparison supports a role for correct semantic correspondence beyond vector distribution, but applies only to OLMo2-1B at rank 64.Three seeds do not measure uncertainty over possible permutations, and selected checkpoints differed between conditions.Source: F PERMUTATION CONTROL · p. 17

G RESIDUAL RANK

The residual-rank sweep shows a trade-off between aggregate validation fit, parameter use, and performance in the rarest strata.

G RESIDUAL RANK

  • The rank sweep is a validation-data capacity comparison, reported separately from held-out test results..Source: G RESIDUAL RANK · p. 17
  • Global validation CE falls from 3.224 at rank 0 to 2.709 at rank 512; rank 64 uses about 16% as many parameters as rank 512 while capturing 72% of the global improvement.Increasing rank improves aggregate fit with diminishing gains per added parameter.Source: G RESIDUAL RANK · p. 17
  • The two rarest strata do not improve consistently with rank, and rank 0 remains competitive in the extreme tail despite its worse global CE.Greater residual capacity therefore does not guarantee better prediction across the whole vocabulary.Source: G RESIDUAL RANK · pp. 17–18

H DOWNSTREAM RESULTS

Downstream component comparisons show that residual benefits vary with the semantic projection, while the evaluation reports class-view averages and seed-level variability. The reported effects require careful interpretation because ablations change multiple components and do not isolate every pathway.

H DOWNSTREAM RESULTS

  • The section presents paired Full-model advantages over ablations and absolute classification scores; Table 13 reports paired AUPRC-point differences with sample SDs.Tables 14 and 15 contain absolute infectious-disease and syndromic classification scores.Source: H DOWNSTREAM RESULTS · p. 18, Table 13 · p. 19
  • Class views average one-versus-rest metrics over selected classes from the same 14-way model, not different patient subsets or separately trained classifiers.Rare 6 is a class-metric view; GI and respiratory scores come from the four-way syndromic model.Source: H DOWNSTREAM RESULTS · p. 18
  • Metrics average equally across 2020–2022 within each seed, with reported SDs measuring variability across three training seeds rather than confidence intervals for new patients.No significance tests are computed; baselines run once and therefore have no SD.Source: H DOWNSTREAM RESULTS · p. 18
  • For Rare 6, the residual’s benefit under learned versus fixed projection differs by −3.27, −2.82, and −3.88 AUPRC points on OLMo2-1B, Llama3.2-1B, and OLMo2-7B, respectively.The residual helps less when the projection is learned, indicating that the components partly compensate for each other.Source: H DOWNSTREAM RESULTS · p. 19
  • Comparing Full model with Residual embeddings only removes shared semantic information and projection parameters; the other ablations also change which components are held fixed or removed.The permutation control was run only for CPT, and component effects need not add up.Source: H DOWNSTREAM RESULTS · p. 19

I TRAINING OF NEP AND EHRADAPT

The training comparison keeps each backbone frozen but uses different training schedules and supervision formats for NEP and EHRAdapt. NEP processes more backbone input tokens, while EHRAdapt receives far more supervised next-event targets.

I TRAINING OF NEP AND EHRADAPT

  • Both methods train only on the patient-level CPT training split, but NEP uses rank-16 LoRA for 10,000 steps while EHRAdapt trains for at most six epochs with validation-based early stopping.Every EHRAdapt run touches all training patients; NEP draws 5.12 million instruction-response examples from 3,945,492–3,945,549 distinct patients.Source: I TRAINING OF NEP AND EHRADAPT · pp. 19, 21
  • Tables 14 and 15 report absolute macro-AUPRC for infectious-disease and syndromic classification, respectively.Table 16 compares NEP and Full-model training, defining targets as supervised next-event predictions and input tokens as processed backbone positions.Source: Table 14 · p. 20, Table 15 · p. 20, Table 16 · p. 23
  • Figure 5 plots CPT test CE, top-1, and top-10 accuracy by event-frequency decile for OLMo2-1B, Llama3.2-1B, and OLMo2-7B.D1 is the rarest stratum and D10 the most frequent; bands show one sample SD across three seeds.Source: Figure 5 · p. 21
  • NEP processes 2.3–5.1 times as many backbone input tokens as EHRAdapt, whereas EHRAdapt receives 300–670 times as many supervised targets.NEP serializes an average of 58.6 context events into 1,557 BPE tokens and supervises one target per example.Source: I TRAINING OF NEP AND EHRADAPT · p. 21, Table 16 · p. 23
  • Figure 6 compares global OLMo2-1B test CE for full and permuted BioLORD assignments, then plots paired CE differences by frequency stratum.Positive permuted-minus-full differences favor the Full model.Source: Figure 6 · p. 22
  • Figure 7 relates OLMo2-1B validation CE to residual parameter capacity and compares per-stratum CE against rank 64.Negative CE-minus-rank-64 values indicate improvement over rank 64.Source: Figure 7 · p. 22
  • NEP uses 32 versus 16 GPUs for the 1B backbones and 128 versus 64 for OLMo2-7B, compared with EHRAdapt.Both ran on AMD Instinct MI300A GPUs; reported GPU-hours exclude queueing and final validation.Source: I TRAINING OF NEP AND EHRADAPT · pp. 21, 23

Related papers

Loading 2609.34007v2…