PaperlayerView exact revision on arXiv

Source-linked AI summary

ReLMem: Learning Recurrent Memory for Longitudinal EHR Modeling

Zijie Meng, Xiwei Dai, Yingying Zhang, Jian Wu, Xian Wu, Zuozhu Liu

arXiv:2609.37587v1cs.LGcs.AI

TL;DR

Longitudinal EHR modeling makes LLM computation and memory costs rise as patient histories expand, while recurrent updates must preserve useful historical evidence. ReLMem learns fixed-capacity memory updates with a frozen LLM and multi-granularity supervision; it approaches full-history performance while substantially reducing retained history.

  • Problem

    Processing expanding patient histories raises LLM computational and memory overhead, and recurrent updates must retain historical evidence needed for downstream tasks.

    Source: §1 · pp. 1–2

  • Method

    ReLMem uses lightweight adapters with a frozen LLM to update fixed-capacity memory recurrently, aligning intermediate attention outputs and supervising final-memory predictions.

    Source: §1 · pp. 2–3

  • Results

    ReLMem approaches full-history performance on two representative tasks; on medication prediction, it uses only 2.9% of average history budget and outperforms all evaluated memory compression methods.

    Source: §5.4 · p. 7, §6 · p. 10

  • Takeaways & Limitations

    The experiments demonstrate the importance of preserving patient history’s predictive value across recurrent updates, not merely compressing it.

    Source: §6 · p. 10

    Evaluation is retrospective on single-center MIMIC-IV and covers medication and diagnosis prediction, with no cross-institution or prospective validation.

    Source: LIMITATIONS · p. 34

Abstract

from arXiv · show

Longitudinal electronic health record (EHR) modeling requires integrating new visits with an expanding patient history. Yet the continual accumulation of clinical information imposes increasing computational and memory costs on large language models (LLMs) when they process and retain complete patient histories. A practical alternative is visit-wise recurrent compression, which incorporates each incoming visit into a compact, continually updated patient memory. However, under a fixed memory budget, successive updates must integrate new information without progressively losing critical historical evidence needed to subsequent tasks. To address this challenge, we introduce Recurrent Longitudinal Memory (ReLMem), a framework that learns to maintain fixed-capacity patient memory for efficient downstream prediction with a frozen LLM. ReLMem equips this LLM with lightweight compression adapters to recurrently update the memory from its previous state and each incoming visit, without rereading earlier records. Specifically, we develop a multi-granularity optimization strategy to preserve task-relevant information throughout recurrent updates and support downstream prediction from the final memory. The intermediate supervision aligns attention outputs from compressed memory and the full history under identical queries, while prediction supervision minimizes cross-entropy with ground truth answers conditioned on the final memory. On EHR-based medication prediction, ReLMem approaches the F1 scores of full-history baseline while reducing average retained historical storage by 97.1%. Under the same memory budget, it improves macro- and micro-F1 over the strongest compressed-memory baseline by 4.66 and 4.75 percentage points, respectively. These results highlight the value of learning recurrent patient memory for efficient longitudinal EHR modeling.

1 INTRODUCTION

ReLMem learns fixed-capacity patient memory by recurrently integrating each incoming visit with the previous memory. Its multi-granularity supervision targets both preservation of historical evidence during updates and prediction from the final memory.

1 INTRODUCTION

  • LLMs processing expanding patient histories face increasing computational and memory overhead; attention cost grows quadratically with sequence length and historical KV-cache size grows linearly.Reusing the cache avoids re-encoding earlier records, but each incoming visit still requires attention over accumulated history.Source: §1 · pp. 1–2
  • Recurrent compression integrates each incoming visit with the previous memory to maintain fixed capacity without rereading earlier records or appending separate visit representations.Recompressing the full history requires revisiting earlier records, while appending separately compressed visits still grows the retained context.Source: §1 · p. 2, Figure 1 · p. 2
  • Because earlier information is accessible only through prior memory, losses during one update can persist and accumulate across subsequent updates.Final-answer supervision alone does not explicitly constrain how well intermediate states retain historical evidence.Source: §1 · p. 2
  • ReLMem uses lightweight compression adapters with a frozen, task-adapted LLM and aligns intermediate attention outputs to full-history outputs under identical queries.It also supervises final-memory predictions with cross-entropy and progressively introduces training cases with more visits.Source: §1 · p. 2
  • The framework combines intermediate memory-readout alignment with supervision of clinical predictions from the final memory.These objectives target historical evidence throughout recurrent updates and downstream prediction, respectively.Source: §1 · p. 3
  • ReLMem approaches Full History using only 2.9% of its average history budget and outperforms all evaluated memory compression methods.The introduction also reports a favorable trade-off between predictive performance and retained-history storage.Source: §1 · p. 3

2 RELATED WORK

Prior work addresses longitudinal EHR prediction, context compression, and recurrent memory through distinct lines of research. ReLMem combines recurrent updates with an emphasis on retaining task-relevant history for downstream prediction.

Longitudinal EHR modeling

  • Longitudinal EHR research spans early neural models, pretrained transformers, generative forecasting, multimodal prediction, and LLM-based clinical reasoning and querying.ReLMem complements these approaches by learning recurrent patient memory for downstream prediction with a frozen LLM.Source: §2 · p. 3

Context compression

  • Context-compression methods shorten prompts, encode text as compact representations, or selectively retain KV states, but compressed representations can still accumulate across visits.Recurrent memory instead integrates each incoming visit with the previous state.Source: §2 · p. 3

Recurrent memory

  • Segment-level recurrence, compressed historical states, and recurrent compression support information transfer across segments, while ReLMem targets history preservation for downstream EHR prediction.It replaces memory at each visit and aligns attention outputs with those from an independently encoded complete historical prefix, while supervising final-memory predictions.Source: §2 · p. 3

3 PROBLEM FORMULATION

The formulation treats a patient history as a chronological sequence of completed visits and predicts an answer sequence from that history and a task query. ReLMem updates a fixed-capacity memory from the previous state and each new visit, then predicts from the final state.

3 PROBLEM FORMULATION

  • A patient history H_T contains completed visits in chronological order; the task query and history define an answer sequence to predict.For medication prediction, the query contains target-visit diagnoses and procedures and the answer is the medication set; for next-visit diagnosis prediction, the query is an instruction and the answer is the next diagnoses.Source: §3 · p. 3
  • At each visit, the recurrent update combines the previous memory M_t−1 with incoming visit v_t to produce the updated memory M_t.Memory capacity remains fixed as visits accumulate.Source: §3 · p. 3
  • At inference, processed visits are accessible only through the final memory, which conditions the LLM’s prediction of the answer given the task query.The objective is to learn updates that preserve task-relevant evidence for accurate clinical prediction.Source: §3 · p. 4, Figure 2 · p. 4

4 RELMEM

ReLMem adapts a task-specific LLM, then trains lightweight adapters to recurrently update fixed-capacity patient memory. Its optimization combines intermediate attention alignment, final-answer supervision, and a curriculum over history lengths.

4.1 TASK-SPECIFIC ADAPTATION

  • ReLMem first adapts an LLM on complete patient histories, then freezes the adapted backbone while training lightweight compression adapters for recurrent memory learning.The task adaptation uses LoRA; learned updates are merged into the backbone before it is frozen.Source: §4.1 · p. 4

4.2 RECURRENT MEMORY UPDATE

  • Each visit is encoded with the previous memory as context, then integrated with that memory into a new state of fixed capacity.Memory is represented by layer-wise KV pairs; the updated pairs replace both the previous memory and temporary visit states.Source: §4.2 · pp. 4–5

4.3 MULTI-GRANULARITY OPTIMIZATION

  • The joint objective combines intermediate attention alignment with final prediction supervision to preserve historical information and support clinical prediction.The alignment weight controls its contribution relative to prediction supervision.Source: §4.3 · pp. 5–6
  • Intermediate supervision aligns attention outputs from compressed memory and full-history references under shared queries, despite their different lengths.The alignment objective minimizes normalized squared differences across selected layers and query heads, encouraging memory to preserve information retrieved from full history.Source: §4.3 · pp. 5–6
  • Final prediction supervision minimizes cross-entropy between answers predicted from the final memory and ground-truth answers.The backbone remains fixed, so this objective trains the compression parameters; full-history references are used only during training.Source: §4.3 · p. 6

4.4 CURRICULUM LEARNING

  • Training progressively expands from short histories to examples with more visits while continuing to sample shorter histories.A nondecreasing visit-count threshold expands the eligible examples; memory capacity, update rule, and joint objective remain unchanged.Source: §4.4 · p. 6

5 EXPERIMENTS

Experiments evaluate ReLMem on medication and next-visit diagnosis prediction, comparing predictive performance, resource costs, history budgets, training components, and model backbones. ReLMem stays close to Full History while using a compact recurrent memory, with results across settings favoring its efficiency and predictive utility.

5.1 DATASETS AND EVALUATION PROTOCOL

  • Evaluation uses MIMIC-IV, 300 test patients per task, and a task-adapted Qwen3-4B backbone; comparisons include Full History and six history-compression methods.The adapted backbone is frozen while memory-token embeddings and compression adapters are trained, with default capacity B = 1,024; test patients do not overlap with training or validation patients.Source: §5.1 · p. 6, §5.2 · p. 7, §5.3 · p. 7

5.4 MAIN RESULTS

  • ReLMem approaches Full History using only 2.9% of its average history budget and outperforms all evaluated memory-compression methods on medication prediction.It also achieves the best P@5 and R@5 scores among the evaluated methods.Source: §5.4 · p. 7, Table 1 · p. 7

5.5.1 RESOURCE COSTS OF RECURRENT COMPRESSION

  • ReLMem nearly halves peak GPU memory and reduces average update latency by 26.3% compared with Full History.Full History reuses its KV cache but still attends over accumulated historical KV states for each incoming visit.Source: §5.5.1 · p. 8, Table 2 · p. 8

5.5.2 MEMORY-PERFORMANCE TRADE-OFFS

  • Reducing ReLMem’s budget eightfold, from 1,024 to 128 slots, lowers macro- and micro-F1 by only 2.81 and 3.00 percentage points, respectively.At 128 slots, ReLMem still outperforms KVzip at 2,048 positions on both metrics; Figure 4 compares medication-prediction performance across history budgets.Source: §5.5.2 · p. 8, Figure 4 · p. 8

5.5.3 EFFECT OF VISITS COUNT AND HISTORY LENGTH

  • With all available history, ReLMem remains within 0.59 percentage points of Full History in macro-F1 while reducing peak GPU memory by 48.8–49.1% and final-visit update latency by 47.2–47.9%.The analysis varies recent-visit count or history length across cohorts of 110 and 107 medication-test cases; both methods benefit from additional history.Source: §5.5.3 · p. 8, Figure 5 · p. 9

5.5.4 CONTRIBUTIONS OF TRAINING OBJECTIVES AND CURRICULUM LEARNING

  • Removing prediction supervision reduces both F1 scores to below 1%, while removing intermediate alignment lowers macro- and micro-F1 by 14.26 and 14.59 percentage points, respectively.With both objectives retained, curriculum learning further improves performance; Table 3 reports the objective and curriculum ablations.Source: Table 3 · p. 8, §5.5.4 · p. 9

5.5.5 EVALUATION ACROSS MODEL SCALES AND FAMILIES

  • Across Qwen3-4B, Qwen3-8B, and Llama-3.1-8B, the gap between ReLMem and Full History remains below 2.5 percentage points in average F1 score.Table 4 reports medication-prediction macro- and micro-F1 across model scales and families.Source: §5.5.5 · p. 9, Table 4 · p. 9

5.5.6 EVALUATION ON DIAGNOSIS PREDICTION

  • On next-visit diagnosis prediction, ReLMem’s Macro-F1 gap from Full History is only 0.87 percentage points.Prediction relies entirely on longitudinal history, without information from the target visit.Source: §5.5.6 · p. 9, Table 5 · p. 9

6 CONCLUSION

ReLMem learns fixed-capacity recurrent memory for longitudinal EHR modeling with a frozen LLM. Experiments on two representative tasks show it approaches full-history performance while reducing storage and latency as histories expand.

6 CONCLUSION

  • ReLMem combines fixed-capacity recurrent memory with multi-granularity optimization to preserve predictive value across updates while reducing storage and latency as histories expand.The conclusion reports performance approaching full history on two representative tasks.Source: §6 · p. 10

AI USE STATEMENT

The authors report using LLMs for language refinement, formatting checks, and code debugging. They state that the technical ideas, study design, implementation, experimental analysis, and scientific conclusions were developed and verified by the authors.

AI USE STATEMENT

  • LLMs were used for language refinement, formatting checks, and code debugging; the authors say they developed and verified the technical work and conclusions.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The study uses de-identified MIMIC-IV and MIMIC-IV-Note records and is retrospective, with no effect on patient care. The authors state that clinical use requires prospective validation and clinician oversight.

ETHICS STATEMENT

  • Clinical use would require prospective validation and clinician oversight because the retrospective targets may reflect documentation and treatment biases.The study uses de-identified MIMIC-IV and MIMIC-IV-Note records.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility materials direct readers to appendices covering dataset construction, algorithms, training and inference, baselines, metrics, statistical analysis, a test-set diagnosis-category word cloud, and ReLMem’s position-encoding convention. Clinical records remain restricted by dataset agreements, and code is promised upon acceptance.

REPRODUCIBILITY STATEMENT

  • The appendices document dataset construction, algorithms, configurations, baselines, metrics, resource measurements, and statistical analysis; clinical records remain governed by their original agreements, and code will be released upon acceptance.Appendices A–F cover these procedures and analyses.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 COHORT SPLIT AND TARGET CONSTRUCTION

  • Medication targets map drug identifiers and names to ATC level-3 classes and require at least 90% mapping of prescription rows and ingredient components; diagnosis prediction uses completed visits to predict subsequent-admission categories.For diagnosis prediction, patients—not visits—are split into disjoint training, validation, and test cohorts.Source: §A.1 · p. 15, §B.1 · p. 19

A.2 CLINICAL RECORD AND INPUT TEMPLATE

  • Medication prediction combines chronologically ordered completed visits with a current-visit object containing diagnoses and procedures, while diagnosis prediction uses completed visits alone.Both templates specify the task instruction and required output format.Source: §A.2 · p. 15, Table 7 · p. 16, §B.2 · p. 19

A.3 DATASET STATISTICS

  • Test examples have longer histories and inputs than training examples in both tasks, providing a more demanding assessment of whether fixed-capacity memory preserves evidence across recurrent updates.The diagnosis test set contains one target visit per patient; the medication task uses the same 6,804 training examples for task adaptation and memory learning.Source: §A.3 · p. 17, §B.3 · pp. 19–20

A.4 LABEL DISTRIBUTION

  • The medication test set spans 107 classes across 300 target visits, and the diagnosis test set spans 584 categories across 300 visits; both require variable-size predictions.Medication targets are classes, whereas diagnosis targets are categories predicted from completed history alone.Source: §A.4 · p. 18, §B.4 · p. 21

B.4 LABEL DISTRIBUTION

  • Figure 9 presents a word cloud of the most frequent diagnosis categories in the test set.The accompanying label excerpts include categories such as atherosclerosis, body mass index, and chronic ischemic heart disease.Source: §B.4 · p. 21, Figure 9

C.1 PSEUDOCODE FOR RELMEM

  • The appendix describes ReLMem’s task adaptation, recurrent memory learning, and final-query inference procedure.It freezes the task-adapted backbone while learning compression parameters with prediction and intermediate-alignment losses; one update boundary is sampled per example for alignment.Source: §C.1 · p. 22

C.2 POSITION ENCODING

  • Training and inference retain RoPE-encoded memory keys without repositioning them, while memory tokens do not advance the history-token counter.Causal attention follows physical sequence order, and intermediate alignment uses the same RoPE-encoded query vectors to read compressed memory and the full-history reference.Source: §C.2 · p. 22

D.1 TASK ADAPTATION AND MEMORY LEARNING

The appendix describes the task-adaptation and memory-learning setup, its curriculum and training trajectories, and the inference decoding settings.

D.1 TASK ADAPTATION AND MEMORY LEARNING

  • Task adapters are merged into the backbone and frozen; memory-token embeddings and compression adapters are then trained jointly, with checkpoints selected by validation answer loss.Intermediate alignment uses four layers and up to 32 query positions; compression adapters act on memory-token q/k/v/o projections.Source: §D.1 · p. 23, Table 11 · p. 23

D.2 CURRICULUM LEARNING

  • Medication training expands maximum visit counts across epochs from 4 to 6 and then unlimited; diagnosis expands from 6 to 8 and then unlimited, reserving 25% for shorter histories after epoch one.The reserved cases have at most four visits for medication prediction and six for diagnosis prediction.Source: §D.2 · p. 23

D.4 INFERENCE SETTINGS

  • Inference uses greedy decoding with at most 512 generated tokens, stopping at an end-of-sequence token or the output limit.Sampling is disabled.Source: §D.4 · p. 23

D.3 TRAINING AND VALIDATION CURVES

  • Figure 10 shows alignment loss falling rapidly while prediction loss declines more gradually, with validation loss continuing to decrease as longer histories enter training.The authors interpret this pattern as attention matching being learned early, while prediction-relevant information preservation requires longer optimization.Source: §D.3 · p. 23, Figure 10 · p. 24

E BASELINE IMPLEMENTATIONS

The appendix specifies baseline implementations and evaluation procedures, then reports ablations on alignment depth, patient-specific memory, and recurrent versus one-shot construction.

E.1 FULL HISTORY; E.2 LLM-RSUM

  • Full History retains complete selected histories; LLM-Rsum replaces a visit-wise summary using the prior summary and newly completed visit.The LLM-Rsum writer is separate from the task-adapted prediction backbone.Source: §E.1 · p. 24, Table 12 · p. 24, §E.2 · p. 25

E.3 SNAPKV; E.4 KVZIP

  • SnapKV selects high-attention entries from prior and current KV states; KVzip scores entries using reconstruction queries and caps its average retained budget at 1,024.SnapKV reserves the observation-window tail within its 1,024-slot budget.Source: §E.3 · p. 25, §E.4 · pp. 25–26

E.5 ATTENTION MATCHING; E.6 RMT; E.7 CCM-MERGE

  • Attention Matching fits selected values and attention biases to reference outputs; RMT carries learned hidden memory tokens, while CCM-merge cumulatively averages KV slots.CCM-merge trains with answer cross-entropy and lacks intermediate attention alignment; ReLMem instead replaces the prior memory at each update.Source: §E.5 · p. 26, §E.6 · p. 26, §E.7 · p. 26

F.1 PREDICTIVE PERFORMANCE

  • Macro-F1 averages case-level scores equally, Micro-F1 aggregates labels and weights larger label sets more, while P@k and R@k measure top-k correctness and coverage.P@k retains denominator k when fewer than k predictions are returned.Source: §F.1 · pp. 26–27

F.2 RESOURCE EFFICIENCY; F.3 STATISTICAL ANALYSIS

  • Resource measurements use batch size 1 on one A800 GPU and report retained-state storage, peak GPU memory, and update and prediction latency; intervals bootstrap 300 patients 10,000 times.Model loading and data I/O are excluded from resource measurements.Source: §F.2 · p. 27, §F.3 · p. 27

G.1 NUMBER OF ALIGNMENT LAYERS

  • Four-layer alignment improves macro- and micro-F1 by 7.48 and 7.58 points over final-layer-only supervision; eight layers add no gain, while step time rises 2.4%.Four layers are the reported performance-cost trade-off and default setting.Source: Table 14 · p. 28, §G.1 · p. 28

G.2 DEPENDENCE ON PATIENT-SPECIFIC MEMORY

  • Swapped patient memory lowers macro-F1 and micro-F1 by 24.80 and 25.08 percentage points; removing historical memory lowers them by 52.21 and 52.69 points.Both comparisons use patient-matched memory as the reference.Source: Table 15 · p. 28, §G.2 · p. 28

G.3 RECURRENT VERSUS ONE-SHOT MEMORY CONSTRUCTION

  • At the same 144 MiB retained state, ReLMem differs from one-shot compression by no more than 0.75 points in both F1 scores, while reducing update latency 83.0% and peak GPU memory 46.3%.The comparison uses medication prediction with Qwen3-4B.Source: §G.3 · pp. 28–29, Table 16 · p. 29

G.4 EVALUATION ON EXTENDED PATIENT HISTORIES

On extended medication histories, ReLMem’s macro- and micro-F1 remained close to Full History with a fixed 1,024-slot memory. Case analyses show retained historical evidence in medication and diagnosis prediction, while the evaluation remains retrospective and single-center.

G.4 EVALUATION ON EXTENDED PATIENT HISTORIES

  • 0.54 and 0.22 percentage points below Full History on macro- and micro-F1, respectively, ReLMem used 1,024 slots on extended histories.The evaluation selected 170 patients; the longest history contained 69 completed visits and 201,398 tokens. Full History used original benchmark histories, whereas ReLMem used extended histories.Source: Table 17 · p. 29, §G.4 · p. 29

H.1 MEDICATION PREDICTION

  • In Case A, ReLMem recovered more target medication classes with fewer additional predictions than Full History; in Case B, it matched Full History after 15 updates.Case B’s shared prediction included all target classes and the same additional antithrombotic class.Source: §H.1 · p. 30, Figure 11 · p. 30, Table 18 · p. 31

H.2 DIAGNOSIS PREDICTION

  • With 15 completed visits, ReLMem compressed 39,360 historical tokens into 1,024 slots and matched Full History on several persistent diagnoses.It predicted depressive disorder documented in earlier visits but not the latest record, omitted Full History’s heart-failure prediction, and carried forward an earlier acute myocardial infarction absent from the target set.Source: §H.2 · p. 33, Table 20 · p. 33

LIMITATIONS

  • The retrospective evaluation uses single-center MIMIC-IV and covers medication and diagnosis prediction with separately adapted backbones and memory modules.The study has not been validated across institutions or prospectively; other tasks and task-agnostic memory representations remain future directions.Source: LIMITATIONS · p. 34

Related papers

Loading 2609.37587v1…