PaperlayerView exact revision on arXiv

Source-linked AI summary

TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Records

Hejie Cui, Alyssa Unell, Bowen Chen, Jason Alan Fries, Emily Alsentzer, Sanmi Koyejo, Nigam Shah

arXiv:2503.04176v1cs.AIcs.CEcs.CLcs.LG

TL;DR

LLMs’ ability to reason across temporal dependencies in longitudinal EHRs remains underexplored, despite the demands of multi-visit care. TIMER adds controlled temporal evaluation and instruction tuning grounded in different record portions. TIMER-Instruct improves performance on MedAlign and TIMER-Bench over baselines without temporal tuning.

  • Problem

    LLMs’ ability to reason across temporal dependencies in multi-visit EHRs remains understudied, while existing benchmarks often emphasize recent records and overlook temporal distribution.

    Source: §1 · p. 1

  • Method

    TIMER combines TIMER-Bench, which evaluates longitudinal reasoning with explicit temporal evidence and controlled sampling, with TIMER-Instruct tuning on generated pairs grounded in different EHR portions.

    Source: §1 · pp. 1–2

  • Results

    7.3% on MedAlign and 9.2% on TIMER-Bench: TIMER-Instruct models improved over baselines without temporal instruction tuning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Temporal dependencies and biases in instruction distributions are important dimensions for evaluating and tuning clinical language models.

    Source: §6 · p. 9

    MIMIC-derived EHR instruction data is heavily skewed toward single ICU visits, limiting its coverage of multi-visit temporal relationships.

    Source: §3 · p. 3

Abstract

from arXiv · show

Large language models (LLMs) have emerged as promising tools for assisting in medical tasks, yet processing Electronic Health Records (EHRs) presents unique challenges due to their longitudinal nature. While LLMs' capabilities to perform medical tasks continue to improve, their ability to reason over temporal dependencies across multiple patient visits and time frames remains unexplored. We introduce TIMER (Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Records), a framework that incorporate instruction-response pairs grounding to different parts of a patient's record as a critical dimension in both instruction evaluation and tuning for longitudinal clinical records. We develop TIMER-Bench, the first time-aware benchmark that evaluates temporal reasoning capabilities over longitudinal EHRs, as well as TIMER-Instruct, an instruction-tuning methodology for LLMs to learn reasoning over time. We demonstrate that models fine-tuned with TIMER-Instruct improve performance by 7.3% on human-generated benchmarks and 9.2% on TIMER-Bench, indicating that temporal instruction-tuning improves model performance for reasoning over EHR.

1. Introduction

Longitudinal EHR reasoning requires models to connect events across visits and time frames, yet existing evaluation often underrepresents temporal complexity or favors recent records. TIMER introduces controlled temporal evaluation and instruction tuning, with TIMER-Instruct improving performance on MedAlign and TIMER-Bench.

1. Introduction

  • Existing benchmarks often overlook temporal complexity and emphasize recall of recent notes or events, limiting assessment across patient timelines.The introduction identifies temporal reasoning across long clinical records as an understudied challenge despite its relevance to multi-visit care and clinical decisions.Source: §1 · p. 1, Figure 1 · p. 2
  • TIMER-Bench uses model-generated instruction-response pairs with explicit temporal evidence and supports sampling across different timeline distributions.Its multi-time-point queries assess synthesis across patient timelines; evaluations reveal failures in temporal boundaries, trend tracking, and event ordering.Source: §1 · pp. 1–2
  • TIMER-Instruct tunes models on generated instruction-response pairs grounded in different EHR portions, testing recency-focused, edge-focused, and uniform temporal distributions.Uniformly distributed instructions are particularly strong on uniformly distributed TIMER-Bench questions that require mid-timeline reasoning.Source: §1 · p. 2
  • 7.3% on MedAlign and 9.2% on TIMER-Bench: TIMER-Instruct models improved over baselines without temporal instruction tuning.The work also reports state-of-the-art performance on physician-generated benchmarks and TIMER-Bench.Source: §1 · p. 2

2. Related Work

Prior work describes challenges in evaluating models on longitudinal records and adapting medical models through instruction tuning. It highlights the need to address long-context synthesis and limitations in existing medical instruction data.

2.1. LLM Evaluation on Longitudinal Clinical Records

  • Evaluating longitudinal records requires both medical accuracy and temporal understanding, while lengthy documentation and costly expert annotation complicate assessment.EHRs combine structured data, clinical notes, demographics, and visit dates, which clinicians synthesize to track disease progression and relationships between events.Source: §2.1 · p. 2

2.2. Instruction Tuning for Medical Applications

  • Medical instruction tuning is constrained by limited high-quality pairs, and existing approaches often use brief responses or retrieval and fact-based tasks rather than complex clinical reasoning.Synthetic data generation and instruction diversity have drawn interest, while concerns about model collapse and long-context synthesis remain.Source: §2.2 · pp. 2–3

3. Temporal Evaluation Gaps in EHR Instruction Benchmarks

Existing EHR instruction benchmarks have limited temporal reach: ICU-derived data often reflects single visits, while physician-created instructions in longer records skew toward recent events. These gaps motivate evaluation with explicit temporal evidence across patient histories.

3. Temporal Evaluation Gaps in EHR Instruction Benchmarks

  • MIMIC-derived EHR instructions are largely limited to single ICU visits, constraining coverage of temporal relationships across multiple visits and longer periods.The dataset described contains 84.5% single-visit patients, with an average visit duration of 7.2 days; this misses complexities such as chronic disease management and long-term outcomes.Source: §3 · p. 3
  • 55.3% of MedAlign instructions fall in the last quarter of patient timelines, and 71.3% are retrieval-based rather than focused on full-timeline synthesis.MedAlign records span an average of 3,895.06 days, or approximately 10.7 years; Figure 2 contrasts instruction evidence with actual patient visits across timelines.Source: §3 · p. 3, Figure 2 · p. 3
  • The identified gaps motivate benchmarks with explicit time-evidence provenance across years of records and controlled temporal distributions.This design aims to assess capabilities across comprehensive patient histories rather than only acute episodes or recent events.Source: §3 · p. 3

4. TIMER

TIMER pairs temporal-evidence benchmark construction with time-aware instruction tuning for longitudinal EHR reasoning. Clinician review scored TIMER-Bench highly for clinical relevance and factual accuracy, while generated instruction pairs underrepresented the timeline middle.

4. TIMER

  • TIMER combines TIMER-Bench, which creates evaluation sets with explicit time evidence, and TIMER-Instruct, which tunes models using temporally grounded instructions.Figure 3 presents the two components as evaluation and instruction-tuning parts of the framework.Source: §4 · p. 3, Figure 3 · p. 4

4.1. TIMER-Bench: An Evaluation Benchmark for LLM Longitudinal Reasoning

  • TIMER-Bench grounds generated instruction-response pairs in date-time evidence tied to visits and explicitly prompts reasoning across multiple visits.Patient visits are aggregated into XML-formatted longitudinal records before instruction-response pairs are generated.Source: §4.1 · p. 4
  • TIMER-Bench received average scores of 95/100 for clinical relevance, 80/100 for temporal reasoning complexity, and 98/100 for factual accuracy, confirming its clinical authenticity and long-context temporal reasoning capabilities.Source: §4.1 · p. 5

4.2. TIMER-Instruct: Instruction Tuning for Longitudinal Clinical Records

  • Using normalized temporal positions, generated instructions favor early and recent timeline edges and give middle periods significantly less attention.The normalized position metric enables comparison across records with different absolute time spans.Source: §4.2 · p. 5, Figure 4 · p. 5

Instruction tuning with various temporal distribution

  • TIMER-Instruct compares recency-focused instructions in the last timeline quartile, edge-focused instructions at context boundaries, and uniformly distributed instructions.The sets use the same patient timelines and size, varying temporal distribution.Source: Instruction tuning with various… · p. 5

5. Experiments

The experiments evaluate TIMER-Bench and TIMER-Instruct across longitudinal EHR tasks and temporal distributions. TIMER-Instruct improves benchmark performance, while the distribution study finds benefits from aligning tuning data with evaluation and from uniform temporal coverage.

5.2. Model Evaluation and Analysis

  • 7.3% average improvement on MedAlign and 9.2% on TIMER-Bench followed TIMER-Instruct tuning of Llama-3.1-8B-Instruct.The reported averages accompany improvements in correctness and completeness on both benchmarks.Source: Table 2 · p. 6, §5.2 · p. 7

5.1. Experiment Setup

  • Gemini generated 5,000 temporal-evidence instruction-response pairs; TIMER-Bench questions came from nonoverlapping timelines and were clinician-validated.Questions were selected for multiple time-stamped evidence points requiring synthesis across a patient record.Source: §5.1 · p. 6
  • TIMER-Bench combines structured data and clinical notes across visits, averages 1,294.88 days, and uniquely annotates temporal evidence.The comparison with existing benchmarks is summarized in Table 1.Source: §5.1 · p. 6, Table 1 · p. 6
  • The experiments tuned Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct, evaluating open-text correctness and completeness with GPT-4o-mini as judge.The Llama model was tuned with LoRA; the evaluation also included head-to-head comparisons and automated text metrics.Source: §5.1 · p. 6

5.2. Model Evaluation and Analysis

  • MedLM-Large achieved 41.3% correctness on MedAlign, while MedInstruct improved TIMER-Bench correctness by 1.5% and hurt MedAlign performance.The authors report that conventional short-form medical QA tuning alone was insufficient for complex temporal reasoning.Source: §5.2 · p. 7

5.3. Impact of Temporal Distribution Strategies

  • 51.01% vs 45.16% win rate favored uniform over recency tuning on TIMER-Bench (Uniform), where uniform also scored highest correctness (64.52%) and completeness (43.55%).Recency tuning led MedAlign at 55.54% correctness, while uniform slightly exceeded edge-focused tuning on TIMER-Bench (Edge), 65.17% vs 64.68%.Source: §5.3 · pp. 7–8, Table 4 · p. 8

5.4. Case Study

  • Case comparisons show TIMER-Instruct better respected specified time windows, identified measurement trends, and aligned dates with measurements than the base model.The examples also show more complete responses focused on relevant temporal information.Source: §5.4 · p. 8, Table 5 · p. 8

6. Conclusion and Discussion

The conclusion frames temporal-dependency reasoning as a key dimension for evaluating and tuning clinical language models. It presents TIMER-Bench and TIMER-Instruct as contributions and notes possible relevance to other domains involving temporal relationships.

6. Conclusion and Discussion

  • TIMER-Bench explicitly includes time evidence and enables controlled evaluation across different temporal distributions.The conclusion reports that models struggle with longitudinal temporal reasoning.Source: §6 · p. 9
  • TIMER-Instruct improves performance on physician-generated and temporal-reasoning benchmarks, according to the conclusion.The authors suggest temporal-modeling principles may also apply to other fields involving documents or event sequences.Source: §6 · p. 9
  • TIMER-Bench is to be released under a research data use agreement upon approval, and the paper states that its code is available.The reported experiments primarily focus on clinical records.Source: §6 · p. 9

Impact Statement

The impact statement positions the model as an assistive tool requiring professional oversight and notes possible bias-related performance effects. The appendices describe generation prompts, clinician verification, data statistics, tuning settings, and judge evaluation.

Impact Statement

  • The model is intended to assist healthcare professionals, not serve as a standalone medical decision system, and its outputs should be verified by qualified clinicians.The authors note potential training-data and evaluation-metric biases across demographic groups or medical conditions.Source: Impact Statement · p. 9

A. Prompt for TIMER-Bench Generation

B. TIMER-Bench Clinician Verification

  • Clinicians found sampled TIMER-Bench questions relevant and reasonably complex, but could not fully verify factual accuracy without protected patient information.Three clinicians reviewed a randomly selected subset for relevance, quality, and accuracy.Source: B. TIMER-Bench Clinician Verification · p. 13, Table 6 · p. 13

C. Prompt for Instruction-Tuning Data Generation

D. Data Statistics

E. Hyper-parameter Tuning

  • The best reported Llama-3.1-8B-Instruct tuning settings were learning rate 1e-5, gradient accumulation steps 16, and weight decay 1e-4.These values were selected through a hyperparameter search.Source: E. Hyper-parameter Tuning · p. 14, Table 8 · p. 14

F. LLM Judge Prompt

G. The Correlation of LLM-Judge with Human Annotated Model Rank

The appendix reports strong agreement between LLM-judge scores and clinician model rankings, describes the head-to-head comparison prompt, and cautions against direct proprietary-versus-open-source comparisons.

G. The Correlation of LLM-Judge with Human Annotated Model Rank

  • Spearman correlations with human ranks were −0.94 for correctness and −0.89 for completeness, with the inverse signs matching the scoring and ranking directions.The average-score correlation was −0.97, and model rankings consistently matched clinician preferences in the reported analysis.Source: Table 9 · p. 15, G. The Correlation of LLM-Judge with… · p. 15

H. Head-to-head Comparison Prompt

I. Proprietary Model Performance

  • Proprietary model scores are reference ceilings, not fair direct comparisons with open-source models, because scale, pretraining, training transparency, and cost differ.The reported proprietary-model results cover MedAlign and TIMER-Bench.Source: §I · p. 16, Table 10 · p. 16

J. Additional Metrics Results

TIMER-Instruct improves additional automatic metrics over the base model and medical QA fine-tuning across MedAlign and TIMER-Bench.

K. Additional TIMER-Bench Results

On uniformly distributed TIMER-Bench, TIMER-Instruct outperforms medical-finetuned baselines and general long-context models across all reported metrics.

L. Case Studies

Case studies contrast baseline temporal-reasoning errors with TIMER-Instruct’s improved handling of relevant information, chronology, and multi-step queries.

Loading 2503.04176v1…