PaperlayerView exact revision on arXiv

Source-linked AI summary

Structured Evidence Routing for Incident Risk Prediction from Multimodal Longitudinal EHRs

Animesh Agarwal, Meysam Ghaffari, Nina Fatehi, Carlos Morato

arXiv:2608.26191v1cs.AI

TL;DR

Incident risk prediction from longitudinal EHRs is difficult because weak signals are multimodal and distributed across irregular histories. The paper routes full-record evidence into disease-specific assessments, reviews those assessments, and pairs the routed summaries with a supervised readout; across five EHRSHOT tasks, performance reaches the range of strong supervised baselines while retaining an inspectable evidence trail.

  • Problem

    Incident risk prediction must identify future disease from weak, temporally diffuse signals distributed across heterogeneous longitudinal EHR modalities.

  • Method

    A router–predictor–reviewer workflow organizes the full pre-index EHR into a compact summary and targeted evidence slices, then produces and critiques an evidence-linked risk assessment.

  • Results

    Across five EHRSHOT tasks, routed evidence summaries improve over CLMBR, remain competitive with MoA+Cls, and reach the range of strong supervised baselines under AUROC and AUPRC.

  • Takeaways & Limitations

    The workflow provides patient-specific, inspectable evidence traces, while ablations indicate contributions from routing, laboratory evidence, task guidance, and review.

  • Takeaways & Limitations

    Evaluation is retrospective and benchmark-based, and does not measure clinician interpretation, trust, review burden, or earlier disease recognition.

Abstract

from arXiv · show

Incident risk prediction from longitudinal electronic health records (EHRs) is challenging because relevant signals are multimodal, weak in isolation, and distributed across irregular patient histories. We propose structured evidence routing, a router-predictor-reviewer workflow that separates full-record access from disease-specific assessment. The router organizes the complete pre-index EHR into a compact summary and targeted evidence slices; the predictor uses this evidence to form an evidence-linked risk assessment, which the reviewer critiques. For comparison with supervised EHRSHOT baselines, we pair the routed evidence summaries with a supervised classifier readout. Across five 1-year incident diagnosis tasks, our method reaches the AUROC range of established supervised EHRSHOT baselines and remains competitive on AUPRC, while exposing a patient-specific evidence trail. Internal pre-readout ablations further suggest that routing, laboratory evidence, task guidance, and review each contribute to performance.

1. Introduction

Incident risk prediction from longitudinal EHRs must integrate weak, temporally diffuse signals across heterogeneous modalities. The paper proposes structured evidence routing to organize full records into disease-specific evidence for prediction and review.

  • Clinically important conditions emerge gradually, with predictive signals distributed across years of care and multiple EHR modalities.Relevant information includes diagnoses, medications, laboratory trajectories, physiologic measurements, procedures, and utilization patterns.
  • The proposed router–predictor–reviewer workflow separates full-record access from disease-specific assessment.It summarizes the pre-index record, retrieves targeted evidence, produces a risk assessment, and reviews that assessment.
  • The study evaluates routed evidence summaries on five 1-year incident diagnosis tasks from EHRSHOT.The comparison includes supervised classifier readouts and baselines including count-based models, CLMBR, and MoA+Cls.
  • Within-framework ablations assess the contributions of routing, laboratory evidence, task guidance, and review.

2. Method

Structured evidence routing separates full-record organization from disease-specific assessment and review. The router summarizes and retrieves targeted evidence, while predictor and reviewer iteratively develop and check the risk assessment.

  • The framework uses three roles: Record Router, Task Predictor, and Reviewer, each with distinct access and responsibilities.All three modules are prompted LLM components.
  • The prediction task estimates whether a target disease will develop within a future horizon using only information available before prediction time.The longitudinal record may contain diagnoses, medications, procedures, laboratory results, measurements, and other structured observations.
  • The Record Router alone accesses the full structured record, organizing it into a compact summary and disease-specific evidence slices.Slices can include diagnosis history, laboratory trajectories, medication changes, and time-localized observations.
  • The Task Predictor uses the router summary, targeted evidence, and clinical task guidance to produce an initial incident-risk estimate with supporting evidence.It does not access the raw chart directly and requests additional slices when needed.
  • The Reviewer checks assessments for unsupported claims, missing evidence, and temporal inconsistencies, triggering iterative refinement when necessary.Refinement continues until the assessment is sufficiently supported or a preset iteration limit is reached.

3. Experimental Setup

The evaluation follows EHRSHOT’s 1-year incident diagnosis setting across five tasks and compares routed evidence summaries with supervised baseline readouts. Internal ablations isolate components within the same routing pipeline rather than serving as cross-method comparisons.

  • Evaluation protocol: The study follows EHRSHOT’s official 1-year incident diagnosis definitions, latest-label protocol, data partitions, and five benchmark tasks.Each patient contributes at most one prediction time per task, retaining the most recent eligible label.
  • Benchmark comparison: The main comparison embeds disease-specific routed evidence summaries with PubMedBERT and feeds them to an XGBoost classifier trained on benchmark labels.This supervised-readout setting matches the labeled training setup of leading structured-EHR baselines.
  • Workflow: Figure 1 depicts a record router organizing long, irregular EHR histories into a compact summary and targeted evidence slices, with review able to trigger refinement.The figure summarizes the workflow used for incident risk prediction.
  • Benchmark comparison: The benchmark asks whether routed evidence can support risk scores in the range of established supervised EHR baselines while preserving an auditable evidence trail.The evidence trail is patient-specific and derives from the routing workflow.
  • Ablation design: Internal ablations remove components within the evidence-routing pipeline and are not intended as cross-method comparisons with EHRSHOT baselines.Separate supervised classifiers are not trained for each ablated variant because that could test downstream compensation rather than component contribution.

4. Results

Across five EHRSHOT incident-diagnosis tasks, routed evidence summaries achieved competitive supervised-readout performance under AUROC and AUPRC. Internal ablations on pancreatic cancer indicate that routing, laboratory inputs, review, and task guidance each contribute to the framework’s direct prediction signal.

  • Routed evidence summaries were compared with coded event features, learned longitudinal embeddings, and LLM-generated summaries under supervised EHRSHOT readouts.
  • Supervised Readout Comparison: Our supervised-readout variant was competitive on AUROC across all five EHRSHOT tasks, improving over CLMBR on every task and matching or exceeding MoA+Cls on all five.
  • Supervised Readout Comparison: Against count-based models, the method matched hypertension and hyperlipidemia, exceeded lupus and acute myocardial infarction, and came within 0.01 AUROC on pancreatic cancer.
  • Supervised Readout Comparison: Under AUPRC, the method improved over CLMBR on all five tasks, exceeded MoA+Cls on hyperlipidemia and lupus, and achieved the highest AUPRC on lupus against count-based models.
  • Internal Ablations: The internal ablations were within-framework diagnostics rather than cross-method EHRSHOT comparisons, assessing component removal before supervised readout.
  • Internal Ablations: 0.86±0.03 AUROC was achieved by the full pancreatic-cancer framework; removing laboratory inputs or the Record Router reduced AUROC to 0.83, while removing review or task guidance reduced it to 0.84.

5. Discussion

Across five EHRSHOT tasks, routed evidence summaries functioned as effective patient representations, reaching the range of strong supervised baselines under AUROC and AUPRC. The framework also makes intermediate evidence inspectable, while ablations indicate that evidence organization and multimodal inputs contribute to its direct prediction signal.

  • Across five EHRSHOT tasks, routed evidence summaries were effective patient representations for incident risk prediction.
  • The method improved over CLMBR across tasks, remained competitive with MoA+Cls, and reached the range of strong supervised baselines under both AUROC and AUPRC.
  • The intermediate representation includes a compact patient summary, targeted disease-relevant evidence slices, and an evidence-linked representation before scoring.
  • Routing, laboratory evidence, task guidance, and review each contributed to the direct prediction signal in the reported ablations.

6. Limitations

The study’s clinical evaluation is retrospective and benchmark-based, so it does not yet establish how clinicians would interpret or use the surfaced evidence in practice.

  • Retrospective benchmark evaluation does not measure clinician interpretation, trust, review burden, or earlier recognition of incident disease.

A. EHRSHOT Ablation Details

The ablation study removes individual evidence-routing components while holding the rest of the pancreatic-cancer framework fixed, isolating their pre-readout effects.

  • Removing the Reviewer stops the pipeline after the initial prediction round, while removing laboratory inputs excludes lab-derived measurements and observations.
  • Removing the Record Router exposes the raw structured EHR directly to the predictor instead of providing routed evidence.
  • Removing task guidance omits guideline-derived disease-specific instructions from the predictor prompt.
  • Each variant removes one component while keeping the rest of the framework fixed, isolating component contributions before supervised readout.

B. Prompt Templates for EHRSHOT

The EHRSHOT pipeline uses a structured evidence-routing loop with separate Record Router, Task Predictor, and Reviewer modules. Prompt templates specify how each module retrieves, assesses, critiques, and refines disease-specific incident-risk evidence.

  • B. Prompt Templates for EHRSHOT: The appendix documents prompt templates for the Record Router, Task Predictor, and Reviewer used in the EHRSHOT experiments.Variable placeholders are written as {variable name}, and disease-specific guidance is injected at runtime.
  • B.1. System Overview: The Record Router alone accesses the longitudinal EHR, producing a concise patient summary and retrieving targeted evidence in response to predictor questions.It searches diagnoses, medications, procedures, observations, laboratory results, and family history, returning dates and values when available.
  • B.2. Record Router Prompts: The router’s patient summary is neutral and clinically focused, highlighting high-signal diagnoses, medications, laboratory findings, observations, and recurrent history without diagnosis or treatment recommendations.The summarization prompt takes patient metadata, clinical data, and family history as input and returns a brief contextual paragraph.
  • B.2. Record Router Prompts: Targeted retrieval returns only directly relevant EHR evidence for each predictor question, pairing structured responses with the retrieved evidence.The retrieval prompt accepts patient metadata, clinical data, family history, and predictor questions.
  • B.3. Task Predictor Prompts: Question generation prioritizes diagnostic measurements, symptoms, and major risk factors that can be answered from the structured EHR without scoring or interpretation.Questions should not repeat prior requests and may incorporate reviewer feedback in later rounds.
  • B.3. Task Predictor Prompts: The Task Predictor receives the patient summary, router responses, and task guidance to generate targeted questions and assess 12-month incident risk.Its structured risk profile includes a score, supporting evidence, countervailing evidence, and a short rationale.
  • B.3. Task Predictor Prompts: Post-review updates address hard errors and soft concerns, ask new questions only when needed, and otherwise update the structured risk assessment directly.The final summary reports the active disease’s score and a concise evidence-based rationale.
  • B.4. Reviewer Prompts: The Reviewer checks assessments for evidence grounding, temporal appropriateness, and alignment with incident prediction, then may recommend a score adjustment or additional evidence request.It identifies unsupported claims, temporal mismatches, and incident-versus-prevalent errors before final prediction summarization.

C.1. Benchmark Protocol and Test-Set Selection

The benchmark protocol follows official EHRSHOT definitions and held-out patient splits while retaining the latest eligible prediction time per patient. Preprocessing preserves pre-prediction longitudinal structure and enriches coded events with human-readable descriptions.

  • C.1. Benchmark Protocol and Test-Set Selection: The evaluation follows official EHRSHOT task definitions and patient-ID train, validation, and test splits using the latest-label setting.Each patient contributes at most one retained prediction time per task.
  • C.1. Benchmark Protocol and Test-Set Selection: Held-out test construction selects positive and negative examples from official test patients and retains the latest prediction row for each patient-task pair.Training patients are used only for supervised-readout training and model selection, with test labels excluded from both.
  • C.1. Benchmark Protocol and Test-Set Selection: Only EHR events observed on or before each benchmark prediction time enter the model input.This establishes the pre-index information boundary for prediction.
  • C.1. Benchmark Protocol and Test-Set Selection: Patient records are converted into chronological encounter-style representations grouped by calendar date across clinically interpretable modalities.Duplicate same-day concept occurrences are collapsed while temporal structure is preserved.
  • C.1. Benchmark Protocol and Test-Set Selection: Preprocessing outputs patient-level JSON records containing demographics, task metadata, and time-ordered pre-prediction encounters.The format preserves the available structured longitudinal history for router-mediated summarization and targeted retrieval.
  • C.1. Benchmark Protocol and Test-Set Selection: Structured EHR codes spanning SNOMED CT, LOINC, RxNorm, and CPT4 are enriched with human-readable terminology descriptions before inference.Both original codes and interpretable descriptors are presented for diagnoses, medications, procedures, laboratory tests, and observations.

D. Supervised Readout over Routed Evidence

The supervised-readout variant converts routed evidence summaries into patient representations and trains a classifier under the same supervised task-adaptation setting used for benchmark comparisons. The classifier is evaluated on held-out test data using AUROC and AUPRC.

  • D. Supervised Readout over Routed Evidence: The supervised-readout variant applies a classifier to routed evidence representations to compare patient representations under shared supervised task adaptation.This aligns the method with task-label-trained count-based and CLMBR baselines.
  • D. Supervised Readout over Routed Evidence: Framework outputs are condensed into clinical evidence summaries, embedded with PubMedBERT, and used to train a supervised classifier head.The extracted summary includes the disease-specific prediction, final task-specific reasoning, and top evidence drivers.
  • D. Supervised Readout over Routed Evidence: The selected classifier produces held-out test scores for AUROC and AUPRC, with test labels excluded from readout training and model selection.Evaluation is performed on the benchmark test set after classifier selection.

E. Clinical Guideline Sources for Task Guidance

Disease-specific task guidance is derived from established clinical guidelines and summarized reference sources. These sources define relevant risk factors, supporting evidence, and temporal patterns for retrieval and assessment.

  • E. Clinical Guideline Sources for Task Guidance: Each incident diagnosis task receives disease-specific guidance derived from established clinical guidelines summarized in Table 4.The guidance informs evidence retrieval and risk assessment.
  • E. Clinical Guideline Sources for Task Guidance: The reference sources define relevant risk factors, supporting evidence, and temporal patterns for disease-specific prediction.These criteria guide both what evidence is retrieved and how risk is assessed.
Loading 2608.26191v1…