Source-linked AI summary
M4Bench: Evaluating Procedural Specification for Clinical EHR Derivation Agents
Hannes Ill, Rafi Al Attrach, Rajna Fani, Ahram Han, Ryohei Yamamoto, Marlene Louisa Moerig, Dukyong Yoon, Leo Anthony Celi
TL;DR
Clinical EHR derivations rely on procedural rules that are not captured by schema structure alone, yet benchmarks have not represented that knowledge as an auditable agent artifact. M4Bench evaluates task-specific procedural specifications and finds that they improve Codex agreement with declared references, with a directionally consistent descriptive Claude replication.
Problem
EHR derivations depend on clinical rules, identifiers, time windows, and missingness conventions that may not be visible from schema structure alone.
Source: §1 · p. 2Method
M4Bench evaluates 28 clinical EHR derivation tasks using clinician-reviewed procedural skills and a controlled comparison of task-and-schema access against access plus a task-specific skill.
Source: §1 · pp. 2–3Results
Targeted skills improve Codex reward by +0.136 [0.085, 0.191] on 28 tasks, with 26 of 28 tasks improving, and the Claude replication is directionally consistent but descriptive.
Source: §1 · p. 3Takeaways & Limitations
M4Bench provides an artifact-backed way to inspect residual derivation errors that require human audit.
Source: §7 · p. 10The results measure agreement with declared reference derivations, not clinical correctness, and the 26 MIMIC-IV and two eICU tasks do not establish dataset or hospital generality.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Clinical research over electronic health records depends on derived variables whose definitions sit outside the schema, in item codes, time windows, missingness rules, and output-grain choices that change cohorts and downstream conclusions. We introduce M4Bench, 28 clinical derivation tasks across 15 task families on MIMIC-IV and eICU, to test whether AI coding agents can reproduce such derivations and how clinician-reviewed agent skills change their results. Its controls decompose the value of skills into specification completeness, delivery format, and schema familiarity. In a Docker-isolated Codex campaign (GPT 5.4 Mini and GPT 5.5), skills raise reward by 0.136 over the no-skill baseline, with 26 of 28 tasks improving. Specification completeness accounts for most of this gain: on an eight-task control subset, an operational spec alone raises reward by 0.162, and skills add only 0.024 on top, indistinguishable from zero. On a ten-task subset, decoy skills score like no skill and raw reference SQL nearly matches targeted skills, arguing against generic context or the skill format as the explanation. Schema obfuscation and restructuring preserve 86% and 70% of the skill gain (GPT 5.4 Mini, 26 MIMIC-IV tasks), a descriptive robustness check rather than a bound on memorization. A descriptive Claude Code replication on all 28 tasks (Opus 4.7 and Sonnet 4.6, 560 runs) is directionally consistent (+0.120, 21 of 28 tasks improving). These gains are partial agreement with the reference, not completed tasks: only 25 of 280 with-skill and 13 of 280 no-skill Codex runs pass every strict check, and skills barely change which rows agents return (key F1 +0.003). Skills are released as a versioned, clinician-reviewable markdown delivery format, with a family-level reference-validity audit. M4Bench releases tasks, skills, evaluation code, and sanitized per-run evidence as an open-source benchmark.
1 Introduction
M4Bench evaluates whether coding agents can reproduce clinical EHR derivations whose procedural rules are not visible from schemas alone. It tests clinician-reviewed procedural specifications and controls that separate specification completeness from other possible sources of improvement.
1 Introduction
- Clinical derivations depend on rules distributed across item identifiers, units, documentation practices, time windows, and dataset conventions, so implementation choices can change cohorts and downstream conclusions.Procedural knowledge may not be recoverable from schema structure alone.Source: §1 · p. 2
- M4Bench asks whether coding agents reproduce declared EHR research derivations and how explicit procedural specifications change their behavior in supervised research workflows.Its central question is how human domain expertise should be represented and evaluated.Source: §1 · p. 2
- Each task-family skill is a clinician-reviewed markdown artifact summarizing clinical rules, item identifiers, time windows, and output grain; the benchmark estimates changes in agreement with declared references.The NO-SKILL condition provides task instructions and database access, while WITH-SKILL adds the task-specific procedure; submitted CSVs are evaluated against hidden reference outputs.Source: §1 · p. 2
- M4Bench comprises 28 clinical EHR derivation tasks across 15 task families on MIMIC-IV and eICU, with clinician-reviewed skills and auditable evaluation artifacts.The benchmark includes OS-level isolation, contamination lint, sanitized per-run audit logs, and a family-level reference-validity audit.Source: §1 · p. 3
- Decoy-skill, raw-SQL, operational-spec, and schema-perturbation controls separate specification completeness from delivery format, reference-code structure, and schema familiarity.These controls are designed to distinguish possible sources of skill effects.Source: §1 · p. 3
- Targeted skills improve Codex reward by +0.136 [0.085, 0.191] on 28 tasks, with 26 of 28 tasks improving; the Claude replication is directionally consistent but descriptive.Only 25 of 280 with-skill Codex runs pass every strict diagnostic, and key F1 moves only +0.003.Source: §1 · p. 3
2 Related work
M4Bench evaluates reproduction of declared clinical derivations, distinguishing that task from EHR question answering, medical calculators, and harmonization efforts. It also treats isolation, reproducibility, and contamination as relevant concerns for coding-agent benchmarks.
2 Related work
- M4Bench isolates reusable clinical derivations and scores reproduction against declared references, unlike EHR benchmarks focused on questions or end-to-end observational-study protocols [Lee et al., 2022, Yang et al., 2026, Jiang et al., 2025, Li et al., 2026a].Examples include SOFA, suspected infection, ventilation, vasopressor-equivalent dose, and rolling urine output.Source: §2 · p. 3
- Unlike medical-calculator benchmarks or harmonization efforts, M4Bench tests whether agents reproduce fixed derivations that depend on item identifiers, time windows, unit conversion, missingness conventions, and output grain [Khandekar et al., 2024, Krohn-Grimberghe, 2026, Rojas et al., 2025, Hripcsak et al., 2015].The cited calculator benchmarks test formula recall, extraction, or arithmetic; CLIF and OHDSI/OMOP standardize data and definitions across settings.Source: §2 · p. 3
- M4Bench releases task-specific procedural knowledge in an auditable form without claiming a unique benefit for one packaging format, while addressing isolation, reproducibility, and contamination concerns in coding-agent benchmarks [Zhang et al., 2025a, Li et al., 2026b, Khattab et al., 2024, Zhuo et al., 2024, Jimenez et al., 2024].Schema perturbations are treated as probes of schema familiarity only.Source: §2 · pp. 3–4
3 Benchmark
M4Bench packages clinical derivation tasks, procedural skills, and evaluation tools to assess agreement with declared EHR reference derivations. Its benchmark distinguishes reconstruction from intermediate features from reconstruction using source tables, while the references remain retrospective public-code conventions rather than clinically validated standards.
3 Benchmark
- M4Bench covers 28 tasks across 15 clinical families on MIMIC-IV and eICU, with released specifications, clinician-reviewed skills, evaluation tools, and sanitized per-run evidence.Task databases derived from credentialed data are not redistributed; users must obtain source-data credentials and rebuild them.Source: §3 · pp. 4, 6
- Each task asks the agent to produce a CSV with required identifiers and values, checked for valid format, key coverage, and agreement with reference values.The evaluator also reports key precision, row coverage, pass/fail diagnostics, key F1, and normalized MAE; unordered categorical columns require exact matches.Source: §3 · pp. 4–5
- The benchmark targets retrospective concept derivations, not real-time prediction, because some adapted reference implementations use information available only after a decision point.One example is baseline creatinine derivation using discharge ICD codes for chronic-kidney-disease patients.Source: §3 · p. 4
- The primary comparison is NO-SKILL versus WITH-SKILL on the native schema; NO-SKILL is schema-plus-task access, not a deployed-system baseline.The operational-spec control tests how much of the effect comes from specification completeness rather than the skill format.Source: §3 · p. 4
- Task-specific skills provide markdown implementation guidance adapted to declared reference conventions, with paired task specifications reviewed by two clinicians.Skills may include item identifiers, time windows, missingness rules, unit conversions, output-grain reminders, and edge-case cautions; they contain no executable SQL or hidden reference outputs.Source: §3 · p. 4
- MIT-LCP and eICU reference conventions are widely used research implementations, not clinically validated gold standards, and may diverge from clinical definitions or omit edge cases.Future versions should add clinician-adjudicated targets alongside public-code targets to distinguish agreement with community convention from agreement with clinical intent.Source: §3 · p. 5
- Standard tasks retain selected intermediate features, whereas raw tasks require reconstruction from source tables.The distinction tests whether procedural guidance matters when item identifiers, unit conversions, and temporal joins cannot be bypassed through derived features.Source: §3 · p. 4, Table 2 · p. 6
4 Experimental protocol
The protocol estimates task-paired skill effects under controlled execution and reports uncertainty across tasks. Codex is the primary full-coverage campaign; the full-coverage Claude comparison is descriptive because its harness was not uniform.
4 Experimental protocol
- The primary estimand is the task-paired change in mean per-column match rate between WITH-SKILL and NO-SKILL on native-schema tasks.The headline aggregate averages model-level effects within an arm and then across tasks, so each task contributes equally regardless of trial count or output size.Source: §4 · p. 6
- Uncertainty is estimated over tasks with task-cluster bootstrap intervals, family-level sensitivity analyses, and a Monte Carlo sign-flip test.The 95% percentile bootstrap intervals are over task-level deltas and are conditional on the task set.Source: §4 · p. 6
- The final primary Codex analysis uses GPT 5.4 Mini and GPT 5.5 across all 28 native tasks, with five trials per cell and matched-context controls.These paper-quality runs execute inside Docker.Source: §4 · pp. 6–7
- The 560-run Claude Code replication covers all 28 native tasks, but its skill contrast is descriptive because CLI versions and routing profiles were not uniform.A sensitivity analysis restricts comparisons to task cells sharing a recorded CLI version and routing profile per model.Source: §4 · p. 7
- All 760 canonical Codex runs and 560 canonical Claude replication runs passed the stated isolation and contamination checks.Audits fail closed on access to hidden references or evaluator internals and flag traces suggesting answer copying.Source: §4 · p. 7
- Schema perturbations probe familiarity rather than fully decontaminating data values; decoy-skill and raw-SQL controls distinguish generic context, reference-code exposure, and task-specific guidance.For raw tasks, reference SQL may use intermediate tables absent from the task database, making it an SQL-adaptation control rather than an executable answer-copying control.Source: §4 · p. 7
5 Results
Targeted skills improve Codex agreement with reference derivations, but controls attribute most of the subset gain to specification completeness. Strict success remains rare, and effects vary across tasks; a full-coverage Claude replication is directionally consistent.
5 Results
- +0.136 [0.085, 0.191] reward gain for targeted skills on 28 Codex tasks, with 26/28 improving.Reward is mean value-column match rate; the comparison is native-schema WITH-SKILL versus NO-SKILL.Source: §5 · p. 7, Table 3 · p. 8
- On the eight-task subset, operational specifications add +0.162 over NO-SKILL and targeted skills add +0.024 more; on ten tasks, decoys match NO-SKILL and raw SQL nearly matches targeted skills.The control subsets were purposive and over-represented tasks with larger targeted-skill effects; the operational-spec subset had no eICU task.Source: §5 · p. 7, Table 4 · p. 8
- Only 25/280 WITH-SKILL Codex runs pass every strict diagnostic, while task-balanced key F1 changes by just +0.003.The headline gain is concentrated in value-column agreement on rows where agents already produced a matching key.Source: §5 · p. 8
- The largest gains occur on implementation-heavy tasks, while several high-baseline scores remain near zero or negative and grain errors cluster in raw Sepsis-3, ventilation, and suspected infection.Raw KDIGO and standard APS III retain value errors despite skills.Source: §5 · p. 8
- On 26 MIMIC tasks, transformed-schema gains retain 0.862 of the native gain under obfuscation and 0.704 under restructuring, as descriptive robustness checks.These probes do not establish bounds on schema-name memorization or test value-level memorization and broader database generalization.Source: §5 · p. 8
- Across all 28 tasks, Claude's WITH-SKILL reward difference is +0.120 [0.067, 0.177] with 21/28 positive; strict passes rise from 25/280 to 37/280.This is a descriptive replication, not a provider ranking; key F1 increases by +0.008.Source: §5 · pp. 8–9, Table 5 · p. 9
- An exploratory local gpt-oss-20b arm has a near-zero task-paired delta of +0.008 [-0.072, 0.088], suggesting procedural specifications do not substitute for base coding capability.The arm is non-publishable under the authors' release criteria.Source: §5 · p. 9
6 Discussion and limitations
M4Bench evaluates whether procedural specifications help agents reproduce declared clinical EHR derivations, rather than answer EHR questions. Its findings are bounded by reference agreement, limited dataset coverage, and residual implementation failures.
Scope and contribution
- M4Bench evaluates declared clinical derivations, not bedside reasoning, diagnosis, treatment planning, real-time prediction, or downstream study validity.Its tasks span severity scores, organ dysfunction, comorbidity, infection, respiratory support, medications, and rolling windows over MIMIC-IV and eICU.Source: §6 · p. 9
- Procedural artifacts encode item identifiers, time windows, and tie-breaking rules, but help some derivations while others remain flat or negative when grain, timing, or reconstruction fail.This evaluates derivation reproduction rather than answers to EHR text-to-SQL, structured-QA, or interactive-agent tasks.Source: §6 · p. 9
Interpretation limits
- The estimand is agreement with declared references, not clinical correctness, hidden-knowledge transfer, or decontaminated reasoning; public code and schema probes leave memorization unresolved.With 26 MIMIC-IV and two eICU tasks, results are not dataset- or hospital-general; operational specifications without item identifiers already capture most of the gain.Source: §6 · p. 9
Statistical and external validity
- The pilot-informed campaign is not a balanced confirmatory design, and the Claude replication differs from Codex in harness uniformity.A causal Claude skill contrast requires rerunning affected cells under one frozen CLI version and routing policy; broader eICU and cross-site coverage is the stated external-validity priority.Source: §6 · p. 9
Implications and safety
- Agent builders should pair procedural guidance with executable checks because skills barely move key F1 or strict pass, and outputs can retain consequential grain, timing, or missingness errors.The paper recommends independent clinical review before consequential use because incorrect derivations can propagate into cohorts, risk adjustment, and downstream conclusions, while references may contain contestable choices.Source: §6 · pp. 9–10
7 Conclusion
M4Bench evaluates procedural specifications for clinical EHR derivation agents, finding improved agreement with declared references alongside persistent implementation failures. Its task files and paired artifacts are designed to support extensions under shared evaluation and evidence standards.
7 Conclusion
- M4Bench evaluates procedural specifications for clinical EHR derivation agents; explicit specifications improve agreement with declared references, while failures remain in temporal joins, output grain, and raw intermediate reconstruction.The descriptive Claude replication is directionally consistent with the Codex result; the benchmark provides artifacts for inspecting residual errors requiring human audit.Source: §7 · p. 10
- Schema exploration alone is often insufficient, and artifact-backed inspection can help identify residual errors that require human audit.The conclusion frames this as a way to inspect errors, not as evidence that derivations are clinically correct.Source: §7 · p. 10
- Researchers can extend M4Bench through task.toml files pairing reference implementations, evaluation configuration, and clinician-reviewed skills, using shared comparisons, isolation, and evidence standards.Future versions should add clinician-adjudicated targets, multiple accepted derivations, broader eICU and cross-site tasks, and care-process families.Source: §7 · p. 10
A Benchmark details
The benchmark spans 15 task families and evaluates both agreement with reference values and output validity. Its documented implementation materials also describe campaign evidence and reproducibility procedures.
Task families
- Ten of the 15 task families derive from published scores and definitions, including SOFA, APS III, SAPS II, OASIS, GCS, Charlson, SIRS, MELD, KDIGO AKI, and Sepsis-3.The full inventory covers severity scores, neurological status, renal and liver dysfunction, comorbidity, infection, respiratory support, medications, and rolling-window data engineering.Source: A Benchmark details · p. 14
Evaluation
- The reported reward is mean value-column match rate, alongside row coverage, required-column validity, key precision/F1, and ordered-column normalized MAE.Ordered-column nMAE is calculated on matched keys, normalized by each scored column’s observed reference range, and clipped at one per-value error.Source: A Benchmark details · p. 14
Evidence and campaign procedure
- The final Codex analysis uses 760 canonical runs, while the retained archive has 800 directories because it preserves 40 superseded early runs for provenance.The passage also describes retained per-run outputs and manifests, plus a preflight, isolation-canary, trace-audit, and output-hash checklist for fresh campaigns.Source: A Benchmark details · p. 14
B Worked derivation example: suspected infection
The suspected-infection example operationalizes culture–antibiotic pairing and evaluates one row per antibiotic order. On the standard task, the skill improves value agreement primarily by changing which culture is paired.
Clinical definition and mapping
- Suspected infection pairs systemic antibiotic administration with culture collection up to 72 hours before or 24 hours after an antibiotic start.For date-only culture collections, the task uses day-level windows; when both directions match, the earlier culture takes precedence, then the earliest eligible culture within a direction is selected.Source: B Worked derivation example: suspected… · p. 14
Output and scoring
- The output contains one row per antibiotic order, keyed by subject_id and ab_id, with 949,901 reference rows.Reward averages match rates for suspected_infection, positive_culture, and specimen; strict passing additionally requires required columns, full key coverage, and task-specific value agreement.Source: B Worked derivation example: suspected… · p. 14
Diagnostics
- The standard task’s reward rises from 0.720 to 0.820, while the culture_time diagnostic match rises from 0.46 to 0.69; neither condition passes strict checks.Key precision is 1.000 in both conditions and mean key F1 is at least 0.999, indicating the standard-task output grain is correct in both conditions.Source: B Worked derivation example: suspected… · p. 14
Caveats
- The reference inherits broad culture inclusion, which can overcall suspected infection, and a plausible rule with the wrong window or tie-break still produces incorrect rows.Table 6 is identified as per-task evaluation details, with reference-row counts defined before task-specific evaluation joins.Source: B Worked derivation example: suspected… · p. 15, Table 6 · p. 15
C Control subset selection
The control subsets were selected before their runs using earlier task lists and campaign evidence, so their results describe purposive subsets rather than the full benchmark. Each task was reviewed by two clinicians.
Control subset selection
- The decoy-skill/raw-SQL control used a precommitted 10-task list spanning 6 of 15 families; the operational-spec control used a pre-existing 8-task sentinel list spanning 8 of 15 families.The first list was committed before its first control run; the sentinel list predated its first operational-spec run by twelve days.Source: C Control subset selection · p. 15
- The subsets were purposively selected using earlier campaign evidence, and their targeted-skill deltas exceeded those of their complements.The 10-task subset deltas were +0.280 and +0.186, versus +0.056 and +0.116 for their complements.Source: C Control subset selection · p. 15
- Each task received review by two clinicians covering its definition, schema, and output.Reviewers are identified as C1–C4.Source: Table 7 · p. 16
D Release, verification, and provenance
The release provides code and evidence artifacts with documented reconstruction and verification procedures. Reproducibility is bounded by missing per-run provenance and by materials that require private archives or credentialed reference outputs.
Release and reconstruction
- The tagged release includes benchmark code, skills, harness, evaluator, tests, run manifests, and table scripts, while reconstruction requires credentialed access to MIMIC-IV and eICU.The documented process also calls for schema-equivalence verification, preflight, and a Docker leak canary before a fresh campaign.Source: D Release, verification, and provenance · p. 16
Evidence packaging
- Evidence packaging verifies source files against manifests, sanitizes outputs and identifiers, and retains agent messages, SQL, and code.A separate scanner checks for residual tool output, credentials, identifiers, and shifted datetimes; both packages pass it, though agent messages can still describe what agents saw.Source: D Release, verification, and provenance · p. 16
- Packaging-time redaction malformed 322 result files and 2,513 trace events; derived result files are documented, while affected trace events are withheld whole.Evaluator fields were unaffected, and original bytes remain in the private archive.Source: D Release, verification, and provenance · pp. 16–17
Verification
- Packaged evidence regenerates all four Claude replication tables and 18 of 19 historical result tables, but not Table 23.Some task-detail, artifact, and compute tables need private archives or credentialed reference outputs; re-evaluating agent outputs needs both.Source: D Release, verification, and provenance · p. 17
Provenance
- Runs omit source commit, working-tree state, and container image digest, limiting per-run provenance.The Claude collaborator declares a base commit and uncommitted changes; the separate authentication-failure archive was not retrieved, so failed attempts are missing from the inventory.Source: D Release, verification, and provenance · pp. 17–18
E Additional result details
Additional results detail campaign integrity, task-level heterogeneity, and output-grain diagnostics. Follow-up and exploratory arms have distinct evidentiary boundaries.
Primary Codex campaign integrity
- The primary Codex analysis contains 760 runs across 152 task/condition/schema/model cells, with 170 clinician-review rerun replacements.For four task variants, primary Codex and Claude WITH-SKILL runs saw different skill text; publishability establishes analysis eligibility, not clinical correctness.Source: E Additional result details · p. 18
Codex heterogeneity
- Figure 2 and Tables 16–18 detail where the primary effect is largest, flat, or negative, and its split between standard and raw tasks.Source: E Additional result details · p. 18
Output-grain diagnostics
- Output-grain diagnostics report aggregate key F1 and key precision; the largest extra-key cells are dominated by interval-style derivations.Source: E Additional result details · p. 18
Validity follow-up
- The May 6 validity follow-up supplies validity evidence rather than replacing the primary native-schema estimand; its 340 canonical runs pass publishability, isolation, canary, and contamination-lint checks.The canonical matrix contains 80 operational-spec runs and 260 transformed-schema with-skill runs.Source: E Additional result details · p. 19
Historical Claude and exploratory OSS arms
- Claude schema-perturbation evidence remains inconclusive without matched native, obfuscated, and restructured cells; the local OSS arm is non-publishable.All 110 local OSS runs are marked non-publishable, and none passes every diagnostic.Source: E Additional result details · p. 19
F Full-coverage Claude replication details
This section documents the 28-task, 560-run full-coverage Claude replication and its nonuniform harness, framing cross-provider findings as descriptive. It also summarizes reproducibility materials, clinical-data access limits, source assets and releases, and the checklist’s statements on crowdsourcing, human-subject interventions, and evaluated LLM coding agents.
F Full-coverage Claude replication details
- The canonical Claude replication combines 560 publishable runs across all 28 tasks; substituting local runs for two duplicated tasks changes the reported delta from +0.1196 to +0.1205.The substitution is a weak numerical-stability check: both duplicated tasks have near-zero skill deltas, and the comparison does not establish harness equivalence.Source: F Full-coverage Claude replication… · p. 19, Table 29 · p. 25
- Eight task comparisons mix Claude Code versions, and raw SAPS II has model-slot overrides only in NO-SKILL, so nine tasks are excluded from the matched subset.All runs requested Opus or Sonnet and used medium reasoning effort, but Claude Code’s native Haiku routing means they were not pure single-model executions.Source: F Full-coverage Claude replication… · p. 20
NeurIPS Paper Checklist
- The paper frames M4Bench as measuring agreement with reference EHR derivations, not autonomous clinical correctness, and presents the cross-provider Claude replication as descriptive because its harness was not uniform.It also discusses risks of incorrect derivations propagating into cohorts, risk adjustment, or downstream analyses.Source: NeurIPS Paper Checklist · pp. 27, 31
- The tagged release provides code, task specifications, skills, manifests, scripts, and sanitized per-run evidence, while reproducing data-layer results requires credentialed access to MIMIC-IV and eICU.Exact agent trajectories may vary because coding agents and hosted model APIs are nondeterministic; released packages include metadata and hash lineage.Source: NeurIPS Paper Checklist · pp. 28–29
- The paper credits MIMIC-IV, eICU, and MIT-LCP MIMIC-Code and documents dataset versions, usage terms, release licensing, and access restrictions.It also documents task specifications, skills, evaluator code, and result artifacts, including provenance, retention, rerun requirements, checksums, and metadata.Source: NeurIPS Paper Checklist · p. 32
- The paper reports no crowdsourcing or prospective human-subject experiments; clinician review covers benchmark artifacts, and the study uses de-identified databases without recruitment or interventions.Source: NeurIPS Paper Checklist · p. 33
- LLM coding agents are evaluated systems, and the paper describes model-provider arms, conditions, and scoring of agent outputs.Source: NeurIPS Paper Checklist · p. 33