PaperlayerView exact revision on arXiv

Source-linked AI summary

EHR2Trace: Auditable EHR Data Infrastructure for Patient World Models and Clinical Agents

Xinye Yang, Yuli Wang, Cheng Ting Lin, Harrison Bai

arXiv:2609.38193v1cs.LGcs.AIcs.DBq-bio.QM

TL;DR

EHR histories can differ in how they represent clinical events and when information becomes available, complicating their use for patient models and clinical agents. EHR2Trace converts records into traceable events with explicit timing and medication stages, then validates reusable OMOP and MEDS exports. Across three datasets it converted 846.4 million events and detected all 28 injected faults; a controlled experiment found higher AUROC and AUPRC for availability-filtered training than training with backdated diagnoses.

  • Problem

    Patient models and clinical agents need longitudinal records that distinguish patient condition, care provided, and information available at each decision, but EHR records vary in how events are recorded.

    Source: §1 · p. 2

  • Method

    EHR2Trace converts source records into linked canonical events with separate event and availability times and medication action types, then exports and validates shared representations.

    Source: §1 · p. 2, §3.1 · p. 3, §5 · p. 10

  • Results

    EHR2Trace converted 846.4 million events across three clinical datasets and detected all 28 injected faults; availability-filtered training achieved higher AUROC and AUPRC than training with backdated diagnoses.

    Source: §5 · p. 10, §7 · p. 11

  • Takeaways & Limitations

    The shared event representation lets researchers inspect source records and conversion rules and reuse records across cohort construction, trajectory modeling, and agent evaluation.

    Source: §5 · p. 10

    The evaluation covers three clinical datasets and a retrospective mortality-proxy task with logistic regression, not evaluations with patient world models, clinical agents, or prospective deployment.

    Source: §6 · p. 11

Abstract

from arXiv · show

Patient world models and clinical agents aim to predict changes in patients' health and support clinical work. Developing these systems requires reliable histories of patient conditions, treatments, and the information available at each decision. Electronic health records (EHRs) contain these histories, but differences in how events are recorded make them difficult to use consistently. We present EHR2Trace, a system that converts EHRs from different sources into traceable patient events for model training and evaluation. It links events to source records, separates event time from information availability, and distinguishes medication orders, dispensing, and administration. A shared event representation supports both OMOP and MEDS exports, with automated validation and reproducible builds. Across three clinical datasets, EHR2Trace converted 846.4 million events, with every applicable check passing except one unit-consistency check on MIMIC-IV, and detected all 28 injected faults. A controlled prediction experiment showed that assigning later diagnoses to admission time substantially inflated measured performance, and that a model trained on such data lost accuracy when deployed on histories filtered by availability. EHR2Trace provides a reusable data foundation for patient world models and clinical agents, helping researchers inspect patient histories, check conversion decisions, and evaluate models with explicit data rules.

1 Introduction

EHR2Trace addresses inconsistent EHR records by preserving clinical timing, information availability, and medication stages in auditable patient histories. It combines traceable event conversion, automated validation, and empirical evaluation for downstream modeling.

1 Introduction

  • Patient world models predict changes in patient state following clinical actions [16] [20], while clinical agents retrieve information and perform tasks in EHR environments [12].Both depend on longitudinal records that distinguish patient condition, care provided, and information available at each decision.Source: §1 · p. 2
  • EHR conversion can conflate event and availability times or medication orders, dispensing, and administration, introducing future information or misrepresenting treatment.Temporal leakage inflates measured performance; offline reinforcement learning also requires explicit action, reward, and episode definitions and methods for addressing confounding.Source: §1 · p. 2
  • EHR2Trace converts source records into shared patient events for OMOP and MEDS exports while preserving source links, timestamps, and medication action types.These links and timing metadata support inspection of conversion decisions and downstream trajectory construction.Source: §1 · p. 2
  • Automated checks compare saved outputs with source and audit records to assess patient identity, temporal consistency, terminology mappings, and dataset splits.The work also evaluates conversions of three clinical datasets, injected faults, repeated builds, and timestamp errors in a controlled prediction experiment.Source: §1 · p. 2

2 Related work

Related work spans patient trajectory models, clinical agents, EHR data standards, and data-quality assessment. EHR2Trace prepares and validates source data for use with existing tools while comparing evidence across source records, canonical events, and exports.

2 Related work

  • ETHOS and EHRWorld study patient trajectories and states [16] [20], while PhysicianBench evaluates clinical agents performing EHR tasks [12].Other work integrates large language models with structured EHR data, and controlled clinical vision-language model evaluations show sensitivity to workflow, prompt, and benchmark design [24].Source: §2 · p. 2
  • OMOP organizes clinical data in tables with standard concepts, while MEDS organizes data as patient event streams.MEDS-Transforms, ACES, and MEDS-DEV provide preprocessing, cohort and task definitions, and model-training workflows; EHR2Trace prepares and validates source data for these tools [23].Source: §2 · p. 2
  • EHR2Trace extends data-quality approaches by checking consistency across source records, canonical events, and exports, including MEDS event ordering and dataset splits.Prior frameworks organize assessment around conformance, completeness, and plausibility, and OHDSI’s DQD implements checks for OMOP data.Source: §2 · p. 2

3 System design and validation

EHR2Trace converts source records into linked canonical events, applies explicit policies for histories and data interpretation, and validates saved exports against source and reference evidence. Its design retains event provenance and supports task-specific trajectory construction.

3.1 Conversion with source links

  • Canonical events retain patient and encounter identifiers, event and availability times, codes, values, reference ranges, and quality flags, with links to source rows.Merge rules preserve medication dose, unit, route, status, and end time; unresolved conflicts are flagged and fail validation unless a configured rule resolves them.Source: §3.1 · p. 3
  • Dataset-specific YAML configurations define input patterns, aliases, identity and merge rules, time policies, unit declarations, and terminology normalization.Preparation scripts record row counts and hashes, and document transformations such as splitting combined blood-pressure values into separate measurements.Source: §3.1 · p. 4

3.2 Data for patient trajectories

  • Trajectory histories use information available at decision time, with transitions representing current observation, recorded action, elapsed time, and next observation.Availability timestamps support decision-time filtering, medication action types distinguish orders, dispensing, and administration, and task-specific rewards and episode boundaries remain downstream choices.Source: §3.2 · p. 4

3.3 Time, missing data, and terminology

  • Unit normalization preserves source values and units, applies only exact conversions, and flags documented corrections or values outside plausible ranges.When a code appears in units that cannot be converted exactly, it is separated into one code per unit; terminology mapping records ambiguity and sends unresolved terms for review.Source: §3.3 · pp. 4–5
  • Event time and availability time are stored separately; missing availability can be replaced by event time and flagged, while duplicate results retain the earliest availability time and receive a merge flag.For MIMIC-IV laboratory results, specimen collection defines event time and storetime defines availability time; Table 1 illustrates withholding events unavailable by decision time while keeping medication stages distinct.Source: §3.3 · pp. 4–5, Table 1 · p. 5

3.4 Validation of saved outputs

  • The 55-check validation suite assesses saved outputs for source accounting, identity and timing, output structure, mappings, and review decisions.Cross-layer checks compare exported data with separately stored evidence and reference information; checks also report applicability, status, and diagnostics, with unavailable inputs marked skipped.Source: §3.4 · pp. 5–6

4 Results

Across three clinical datasets, EHR2Trace produced large-scale event conversions with validation and information-preservation checks. A controlled prediction experiment quantified how diagnosis backdating inflated measured performance and how evaluation results changed across time rules.

4.1 Conversion scale and concept coverage

  • 846.4 million canonical events were produced, with matching MEDS event counts; all applicable checks passed for JHU-CTPE and CU-CTPA, while MIMIC-IV had a unit-consistency finding.The MIMIC-IV finding is described in Section 6.Source: §4.1 · p. 6, Table 2 · p. 7
  • 129 of 139 reviewed free-text medication terms mapped to RxNorm, and 93.8% of resolved terms matched reviewer choices; MEDS retained source codes for events awaiting standard mappings.The drug matcher used ingredient, strength, and dose form.Source: §4.1 · p. 6

4.2 Preserving time and medication actions

  • The audit kept dispensing distinct from administration, retained administration dose and route, and converted death to dated events only when a date was available.Table 4 also reports each dataset’s share of timed events with assumed availability times.Source: §4.2 · p. 6, Table 4 · p. 8

4.3 Comparison with published conversions

  • In comparison with published conversions, EHR2Trace produced OMOP and MEDS together and stored availability separately from event time, which neither baseline did.It linked every exported event to its source file and row through 972,567 links; the published OMOP conversion lacked a source-row field, while MEDS keys lacked table names.Source: §4.3 · p. 7, Table 5 · p. 8

4.4 Fault detection

  • All 28 injected faults were detected; removing 21 cross-layer checks reduced detection to 13 of 28, while DQD detected five of nine faults reaching OMOP.Eight faults affected data outside OMOP and were outside DQD’s scope.Source: §4.4 · p. 7, Figure 2 · p. 9

4.5 Effect of time errors on model evaluation

  • 0.136 AUROC (0.829 under availability filtering vs 0.965 under diagnosis backdating) at six hours; backdating increased measured performance at every horizon.The MIMIC-IV experiment held outcome labels, patient splits, model class, and fitting procedure fixed; event-time-only filtering had a smaller effect.Source: §4.5 · pp. 7–8, Figure 3 · p. 10
  • For the same fitted model, changing the test rule reduced AUROC by 0.323; backdated training’s deficit was 0.188 under a fixed test rule.AUPRC was 0.071 for C→A, 0.222 for A→A, and 0.654 for C→C; across four horizons, availability-filtered AUROC deficits ranged from 0.169 to 0.188.Source: §4.5 · pp. 8–9, Table 6 · p. 9

4.6 Reproducibility and runtime

  • Four synthetic builds produced 60 byte-identical artifacts and passed validation; a staged MIMIC-IV rebuild reproduced the identity map, canonical events, MEDS outputs, and clinical OMOP tables.On a 48-core, 251 GB machine, the MIMIC-IV build took 111 minutes for ingestion, 191 for canonical conversion, 47 for OMOP export, and 128 to write MEDS shards; peak canonical-conversion memory was 167.2 GB.Source: §4.6 · p. 9

5 Discussion

EHR2Trace makes clinical histories inspectable from source records through model inputs while preserving timing and medication-action distinctions. Explicit availability rules improve history construction and expose inflated evaluation, while validation and reproducible builds support reuse across downstream tasks.

5 Discussion

  • Source links, temporal metadata, and medication action types let researchers trace exported events back to source rows and inspect the rules that produced them.The shared records can be reused across cohort construction, trajectory modeling, and agent evaluation.Source: §5 · p. 10
  • Availability-filtered training achieved higher AUROC and AUPRC than training with backdated diagnoses on a common availability-filtered test representation.Cross-rule evaluation also quantified inflation from backdated test histories.Source: §5 · p. 10
  • Availability times and medication action types provide explicit records for constructing world-model histories and actions at each decision.Separating conversion from task construction supports direct comparison of episode definitions, rewards, time rules, and action detail on a shared event base.Source: §5 · p. 10
  • Cross-layer checks compare outputs with source rows, reference tables, dataset declarations, and repeated builds to support auditable, reusable artifacts.Injected faults assess check coverage, while repeated builds assess reproducibility under execution-setting changes.Source: §5 · p. 10

6 Limitations and future work

The evaluation covers three datasets and a retrospective mortality-proxy task, while source interpretation and temporal availability impose boundaries on what the converted histories establish. Further datasets, error collections, downstream evaluations, and timing methods remain areas for development.

6 Limitations and future work

  • The evaluation uses three clinical datasets and a retrospective mortality-proxy task with logistic regression; independent error collections and prospective or agent evaluations remain future work.Treatment-effect estimation and policy learning also require methods addressing confounding and censoring.Source: §6 · p. 11
  • Validation checks consistency with declared rules and reference information, but source completeness and clinical interpretation remain the responsibility of data owners and domain experts.MIMIC-IV full and demonstration conversions fail UNIT_HOMOGENEOUS_PER_CODE; broader terminology coverage and unit harmonization are priorities.Source: §6 · p. 11
  • When source timestamps are absent, availability is assumed; individual-field availability and revisions are not tracked, and untimed attributes or interval ends may exceed a prediction cutoff.Field-level timing, revision tracking, and task-specific history construction are identified as extensions.Source: §6 · p. 11

7 Conclusion

EHR2Trace provides shared, auditable EHR infrastructure for patient world models and clinical agents. Its conversions, fault detection, reproducible builds, and temporal experiment demonstrate scale and validation capabilities, while the paper also reports ethics approvals and data-access constraints.

7 Conclusion

  • 846.4 million events across three datasets, all 28 injected faults detected, and reproducible builds demonstrate EHR2Trace’s scale and validation capabilities.The temporal experiment found that explicit availability rules improve model training and expose inflated evaluation.Source: §7 · p. 11
  • The study reports IRB approval for the JHU-CTPE extract under protocol IRB00424745 and for CU-CTPA under protocol 24-1825.OpenAI Codex and Anthropic Claude assisted with language editing, consistency checks, and figure-generation code.Source: §7 · p. 11
  • EHR2Trace is available under Apache-2.0 with configurations, synthetic tests, validation and reproducibility tests, and experiment scripts and records; JHU-CTPE and CU-CTPA are private.MIMIC-IV access follows PhysioNet credentialing and data-use conditions.Source: §7 · p. 11

A Converter comparison details

The converter comparison uses published MIMIC-IV demonstration baselines and reports EHR2Trace output counts alongside measures of source recovery, availability, medication-action representation, and concept coverage. Counts and coverage are interpreted within matching source modules and model-specific denominators.

A Converter comparison details

  • EHR2Trace produced 971,545 events on the 100-patient MIMIC-IV demonstration subset, compared with published baselines of 423,057 clinical rows for OMOP and 916,166 events for MEDS.The OMOP baseline used CDM 5.3.1 and a September 2020 vocabulary; the MEDS baseline used MEDS 0.3.3 and MEDS-Transforms.Source: A Converter comparison details · p. 14
  • Source recovery requires both a source table name and row identifier, while availability measures the share of timed events whose availability time follows event time.Medication action types distinguish ordering, dispensing, and administration; the MEDS baseline retains order and administration keys on 99.8% of medication events.Source: A Converter comparison details · p. 14
  • Concept coverage uses clinical rows as the OMOP denominator and events as the MEDS denominator, so counts and coverage should be compared within matching source modules.Coverage also depends on the vocabulary release.Source: A Converter comparison details · p. 14
  • The comparison table is generated from measurements recorded by the repository’s converter-comparison script.The script writes the measurements to experiments/results/converter_comparison.json.Source: A Converter comparison details · p. 14

B Temporal experiment details

The temporal experiment used first qualifying MIMIC-IV admissions and a mortality-proxy label, then compared availability rules with paired held-out estimates. Cross-rule tests retained each training rule’s code vocabulary and represented absent test-rule codes as zeros.

B Temporal experiment details

  • The cohort contained 131,007 patients, with 4,185 positive mortality-proxy cases (3.2%) based on the earliest recorded death date relative to discharge.Each patient contributed their first qualifying admission lasting at least 72 hours; features were log-transformed code counts, including prior history and untimed attributes.Source: B Temporal experiment details · p. 14
  • Paired percentile bootstrap intervals used 2,000 held-out-patient resamples shared across arms to estimate AUROC increases relative to availability filtering.Each arm built its code index and fitted its model using only its training split; codes found only in held-out patients were excluded.Source: B Temporal experiment details · p. 14
  • Cross-rule evaluation kept each fitted model’s training-rule code vocabulary, assigning zero to codes absent under the test rule.At each horizon, all nine training–test combinations shared bootstrap resamples; matched-rule results agreed with single-rule experiments within 0.001. Availability-filtered training achieved AUROC 0.721 on backdated test histories at six hours, a combination omitted from Table 6 as unrealistic.Source: B Temporal experiment details · p. 14
Loading 2609.38193v1…