PaperlayerView exact revision on arXiv

Source-linked AI summary

Asclepius: An Adaptive Harness for Long-Horizon Clinical Agents

Grace Chang Yuan, Xiaoman Zhang, Sung Eun Kim, Luyang Luo, Pranav Rajpurkar

arXiv:2609.13543v1cs.AI

TL;DR

Long-horizon clinical agents must sustain execution across interleaved patients, yet current agents often diagnose correctly without delivering complete, timely actions. Asclepius combines an evolving operating manual, clinical skills library, and isolated subagents; it improves critical actions and timeliness on CES while preserving diagnostic accuracy.

  • Problem

    Short-horizon, single-task benchmarks provide limited evidence about sustained agent execution across hours of interleaved tasks under contention.

    Source: §1 · p. 1

  • Method

    Asclepius combines a trace-feedback-driven manual that evolves between shifts, an externalized clinical skills library, and three isolated subagents for per-turn decisions.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    +25% on critical actions (p < 0.001) and +13% on timeliness (p < 0.01) across ten batches, while preserving diagnostic accuracy; held-out critical-action gain is +0.63 (p = 0.024).

    Source: §1 · p. 2

  • Takeaways & Limitations

    The three failure modes behave as a coupled bottleneck: the largest joint reduction appears only when the manual, skills library, and subagents act together.

    Source: §1 · p. 2

    CES results are a proxy rather than evidence of real clinical competence; evaluation uses one default ED configuration and one backbone, with transfer to other settings untested.

    Source: §8 · p. 10

Abstract

from arXiv · show

LLM agents are predominantly benchmarked on short, single-task trajectories, yet real deployments run for hours under contention, surfacing a different class of failures. We use the Clinical Environment Simulator (CES), in which an agent manages an entire emergency-department shift under continuous time and resource pressure, as a testbed: long-horizon execution failures manifest measurably in a single rollout under structured, multi-dimensional grading. On CES, current agents reach the correct diagnosis in most cases yet fail to deliver complete and timely critical actions, revealing an execution gap. We attribute this gap to three long-horizon failure modes, each operationalized as a per-trace counter: instruction-adherence drift, treatment incompleteness, and a severity-equity gap in timeliness. We then introduce Asclepius, an adaptive agent scaffolding with a self-evolving harness that rewrites the operating manual between shifts from trace-level feedback, an externalized clinical skills library for high-stakes regimen knowledge, and three isolated subagents that partition per-turn decisions across the patient queue. On held-out batches never observed during harness evolution, Asclepius improves critical-action correctness by 22% (p = 0.024) over a strong baseline agent framework while preserving diagnostic accuracy, with consistent gains across five LLM judges from three model families; on the full ten-batch set, improvements reach 25% on critical actions and 13% on timeliness. The three failure modes form a coupled bottleneck: decisive reductions appear only when all three components act together.

1 Introduction

On CES, agents often diagnose correctly but fail to deliver complete, timely actions during sustained emergency-department shifts. Asclepius targets three measured failure modes and improves critical actions and timeliness, with its largest joint counter reductions appearing when all three components act together.

From Short-Horizon to Long-Horizon Agent Deployment

  • CES tests sustained emergency-department execution with staggered patient arrivals, evolving vitals, time-sensitive interventions, and contention for limited beds and staff.Performance is graded on diagnosis, critical actions, timeliness, and disposition.Source: §1 · p. 1

Gaps in Current Agents

  • Current agents show an execution gap: diagnostic accuracy is 4.39/5, versus 2.94/5 for critical actions and 3.34/5 for timeliness.The gap is failure to translate correct diagnosis into complete and timely action under multi-patient load.Source: §1 · p. 1

Failure Modes

  • Three per-trace counters operationalize instruction-adherence drift, treatment incompleteness, and a severity-equity gap in timeliness.The measures track rising naked-disposition rates over a shift, zero-treatment rates, and timeliness gaps between non-severe and severe patients.Source: §1 · pp. 1–2

Asclepius

  • Asclepius combines a self-evolving manual, an on-demand clinical skills library, and three isolated subagents for triage, diagnosis, and treatment completion.The proposer uses trace-level feedback to rewrite the operating manual between shifts.Source: §1 · p. 2, Figure 1 · p. 2

Results

  • +25% on critical actions (p < 0.001) and +13% on timeliness (p < 0.01) across ten batches, while diagnostic accuracy is preserved.On four held-out batches, the critical-action gain remains significant (+0.63, p = 0.024).Source: §1 · p. 2
  • The failure modes form a coupled bottleneck: no single component reduces all three counters, and the largest joint reduction appears when all three act together.Five judges from three model families reproduce the improvement pattern.Source: §1 · p. 2

2 Related Work

Prior clinical-agent evaluations largely focus on isolated cases or diagnosis, while agent harness research includes tool use, skill libraries, and prompt optimization. Asclepius adapts a natural-language manual across multi-patient clinical shifts and characterizes execution failures with continuous per-trace counters.

2.1 Clinical LLM Agents and Evaluation

  • Earlier clinical evaluation ranges from static questions to multi-turn single-patient agents; simulated environments and multi-agent variants remain single-patient and untimed, with diagnosis-dominated evaluation.The section positions continuous, multi-patient simulators that grade time-sensitive treatment alongside diagnosis as a newer direction.Source: §2.1 · pp. 2–3

2.2 Agent Harnesses

  • Prior harnesses use tools and skill libraries, while Asclepius curates condition-indexed clinical regimens, dosage tables, and condition-to-action mappings for incomplete treatment selection.ReAct interleaves tool calls with reasoning (Yao et al., 2023), Toolformer fine-tunes API use (Schick et al., 2023), and Voyager introduces a growable skill library (Wang et al., 2024).Source: §2.2 · p. 3

2.3 Prompt Optimization and Self-Evolving Harnesses

  • Unlike single-prompt optimization, Asclepius rewrites a natural-language operating manual across multi-hour, multi-patient rollouts using structured feedback; its three failure modes are continuous per-trace counters.It adapts across shifts rather than within a single-patient encounter, and targets execution under multi-patient load rather than diagnostic accuracy.Source: §2.3 · p. 3

3 Preliminaries

The paper evaluates agents in CES, a turn-based emergency-department simulator with evolving patient states and four graded dimensions. It compares raw frontier models with a stronger tool-using framework that serves as Asclepius’s starting point.

3.1 Clinical Environment Simulator

  • CES simulates multi-hour, multi-patient emergency-department shifts with staggered arrivals, evolving vitals, delayed tests, and developing complications.An LLM judge scores diagnosis, critical actions, timeliness, and disposition on a 1–5 scale.Source: §3.1 · p. 3

3.2 Baseline Agents

  • The baseline framework wraps Claude Code in an MCP clinical interface and hand-crafted manual, but has no specialized knowledge resources or subagents.Raw-agent baselines receive simulator access without a manual or skill resources; the framework maintains the full multi-patient state in one context window.Source: §3.2 · pp. 3–4

4 Asclepius

Asclepius wraps a baseline agent in coupled within-shift and across-shift loops. The harness evolves the manual between shifts, while skills and isolated subagents support treatment selection and per-turn decisions.

4 Asclepius

  • Asclepius has an outer loop that rewrites the manual between shifts and an inner loop with a skills library and three subagents.The loops wrap a baseline agent.Source: §4 · p. 4

4.1 Outer Loop: Self-Evolving Harness

  • A proposer reviews candidate manuals and shift artifacts, edits the current best-scoring manual, and evaluates each revision on search batches.It must base edits on patterns seen across at least three patients in multiple batches, avoid patient- or batch-specific hard-coding, and make substantive changes; six batches are held out from evolution.Source: §4.1 · p. 4

4.2 Inner Loop: Skills Library

  • The skills library supplies on-demand clinical references, including disease-specific drugs, doses, routes, nonpharmacologic interventions, and treatment-completeness guidance.It also contains clinical reasoning and simulation mechanics modules, loaded when their descriptions match the context.Source: §4.2 · p. 4

4.3 Inner Loop: Subagents

  • Three isolated LLM subagents handle triage prioritization, diagnosis and disposition recommendations, and treatment completion.In the acting variant, they have scoped MCP access; prompts use only the hand-crafted manual, so decomposition gains come from focused context, structured outputs, and coordination rather than added medical knowledge.Source: §4.3 · pp. 4–5

5 Experimental Setup

The evaluation uses CES to test agents managing multi-hour emergency-department shifts, with patient batches split between harness evolution and held-out validation. It compares raw agents, a baseline framework, and Asclepius configurations under multidimensional grading.

5.1 The Clinical Simulator

  • CES simulates multi-hour emergency-department shifts with staggered arrivals, evolving patients, delayed test results, complications, and actions through an EHR interface.An LLM judge scores diagnosis, critical actions, timeliness, and disposition on a 1-to-5 scale; additional judges re-grade the baseline and full Asclepius configurations.Source: §5.1 · p. 5

5.2 Patient Batches and Evaluation Protocol

  • Ten batches of twelve patients are split into six harness-evolution batches and four held-out validation batches never observed by the proposer.Each configuration runs once per batch, with scores averaged across batches.Source: §5.2 · p. 5

5.3 Configurations

  • Comparisons include three raw single-agent variants, a single-agent baseline with a hand-crafted manual, full Asclepius, and component ablations.The ablations add Asclepius components individually to the baseline.Source: §5.3 · p. 5

5.4 Implementation Details

  • Claude Opus 4.6 powers both the main and proposer agents, and the harness is evolved for ten iterations on the six search batches.The reported visuals cover all-batch main results, component ablations, and held-out significance comparisons.Source: §5.4 · p. 5, Table 1 · p. 6, Figure 2 · p. 6

6 Results

Asclepius improves critical actions and timeliness while preserving diagnosis and disposition, but the held-out critical-action gain is the primary quantitative result. Ablations and per-trace counters indicate that decisive reductions across the three execution failures appear only when all components are combined.

6.1 Main Results

  • On held-out batches, critical actions rose +0.625 (+22%, p = 0.024); across ten batches, critical actions rose +0.73 (+25%) and timeliness +0.45 (+13%), while diagnosis and disposition stayed within ±0.03.The ten-batch aggregates cover 120 patients; comparisons use patient- and batch-level random effects with BH-FDR correction.Source: §6.1 · pp. 6–7, §6 · p. 6

6.2 Component Ablations

  • The harness was strongest on pooled batches (+0.18 overall, +0.38 critical actions, +0.34 timeliness), but held-out scores favored +SA (3.97) over +H (3.82) and baseline (3.78).The full stack reached 4.10 overall, with critical actions and timeliness at least 0.10 points above any single-component variant.Source: §6.2 · p. 7
  • The evolved manual added +0.12 overall over S+SA on all batches (p = 0.36) and +0.02 on held-out batches (p = 0.88), neither statistically significant.Diagnosis and disposition remained within ±0.11 of baseline across variants and never reached significance.Source: §6.2 · p. 7

6.3 Failure-Mode Analysis

  • Drift tracks early-to-late naked-disposition rates, Incomplete tracks zero treatment after correct diagnosis, and Neglect tracks the timeliness gap by severity; single components reduce multiple counters.The harness reduced drift from +41pp to +32pp, skills reduced zero-treatment from 19% to 14%, and subagents narrowed the equity gap from 1.13 to 0.98; skills and subagents reduced drift further than the harness.Source: §6.3 · pp. 7–8
  • Only the full stack makes decisive reductions across all three counters: drift falls to a +2pp rise, the timeliness gap from 1.13 to 0.23, and zero-treatment from 19% to 9%.For the same patients, baseline naked dispositions rose from 16% to 57%, versus 10% to 12% for Asclepius.Source: §6.3 · p. 8

6.4 Harness Search Trajectory

  • Search scores plateaued at v6–v8, and v7 was selected for overall search-batch score; the full configuration retained a +0.19 overall gain on held-out batches.The v0-to-v7 manual edits added a treatment-enumeration routine and clinical-content updates; inference used 1.21× the baseline LLM calls and 1.52× the wall-clock time per batch.Source: Figure 4 · p. 8, §6.4 · pp. 8–9

6.5 Subagent Variants

  • Acting subagents outperformed advisory variants by +0.10 overall, +0.26 on critical actions, and +0.10 on timeliness, without degrading diagnosis or disposition.Direct MCP ordering avoids Main Agent paraphrasing and re-resolving long structured recommendations, removing a class of vocabulary-resolution failures observed in advisory traces.Source: §6.5 · p. 9, Table 4 · p. 9

6.6 Qualitative Analysis

  • In a representative atrial-flutter case, a treatment-completion subagent flagged missing rate control and anticoagulation; the completed regimen accompanied clinical improvement and a critical-actions score increase from 1 to 5.The baseline trace had reached the correct diagnosis but lacked a rate-control strategy and durable oral anticoagulation plan before disposition.Source: §6.6 · p. 9

7 Conclusion

The conclusion frames long-horizon emergency-department work as an execution challenge: agents often diagnose correctly but fail to complete timely critical actions. Asclepius targets three linked failure modes, and its reported gains support evaluating sustained execution rather than isolated diagnostic accuracy.

7 Conclusion

  • On held-out batches, Asclepius improves critical-action correctness by 22% over a strong baseline while preserving diagnostic accuracy; full-set gains reach 25% on critical actions and 13% on timeliness.The three failure modes form a coupled bottleneck: decisive reductions across all three counters appear only when the harness, skills library, and isolated subagents act together.Source: §7 · p. 9
  • Current agents often reach correct diagnoses yet fail to deliver complete and timely critical actions under sustained shift-long load.The conclusion uses the Clinical Environment Simulator as a testbed for measuring this execution gap in long-horizon, multi-task settings.Source: §7 · p. 9
  • Asclepius pairs one component with each failure mode: an evolving manual, an externalized clinical skills library, and isolated subagents that divide per-turn queue decisions.The targeted modes are instruction-adherence drift, treatment incompleteness, and a severity-equity gap in timeliness.Source: §7 · p. 9
  • The conclusion argues that clinical-AI evaluation should measure sustained, complete, and equitable execution across a patient queue, not diagnostic accuracy on isolated cases.Source: §7 · p. 9

8 Limitations

The paper identifies boundaries on its evidence and describes the simulator’s time, severity, resource, and treatment dynamics. CES models a six-hour emergency-department shift with interleaved patients and constrained clinical resources.

8 Limitations

  • Results are limited to automated grading, one stochastic rollout per configuration and batch, a single default CES setting, and a single backbone; harness evolution is costly and the hand-curated skills library may omit rare presentations.CES performance is a proxy rather than evidence of real clinical competence; varied shift lengths, staffing ratios, acuity mixes, weaker models, other environments, and other language or dosing regimes remain untested. Per-batch differences range from −0.12 to +0.71, and search-set adaptation compresses the overall gain from +0.36 to +0.19 on held-out batches.Source: §8 · p. 10

A.1 Temporal Model and Shift Structure

  • A default CES scenario is a six-hour, 72-turn shift with twelve staggered arrivals; each turn advances five simulated minutes through a fixed-order event pipeline.Patients are based on de-identified emergency-department records and have hidden deterioration thresholds and required actions.Source: §A.1 · p. 13

A.2 Patient Severity and Deterioration

  • Patients are grouped as mundane, dynamic moderate, or dynamic severe, with hidden deterioration thresholds of 90–120 minutes and 30–60 minutes for the latter two classes.Without required critical actions, pathological drift begins after the threshold; completing those actions stops drift and vital signs recover, while only severe patients can trigger code blue.Source: §A.2 · p. 13

A.3 Action Costs and Resource Constraints

  • Clinical actions consume constrained time and resources: for example, an H&P takes 10 minutes, standard medication administration 5 minutes, and transfusions or large-volume infusions 20 minutes.The default setting has four beds, six nurses, one physician, specialist consultants, and capacitated imaging and laboratory queues.Source: §A.3 · p. 13

A.4 Medication and Treatment Effects

  • Treatment effects have different onset profiles: fluids and oxygen act in 5–10 minutes, antibiotics and bronchodilators in 15–30 minutes, and antipyretics in at least 30 minutes.The simulator’s temporal treatment model penalizes delayed initiation and rewards early, appropriate intervention.Source: §A.4 · p. 14

A.5 Engine Architecture

CES coordinates patient physiology and hospital resources through separate engines, while the baseline agent uses a tool-based interface and operating manual. The appendix details its tool categories, vocabulary resolution, workflow guards, and released Asclepius artifacts.

A.5 Engine Architecture

  • The Patient Engine maintains clinical state and generates patient updates, while the Hospital Engine allocates and releases resources through priority-based scheduling; an orchestrator coordinates their turn-level exchanges.Agents use a browser interface with patient charts, an order catalog, and real-time event notifications.Source: §A.5 · p. 14

B Baseline Agent Configuration Details

B.1 Tool Categories

  • The tools provide structured patient observations, encapsulate clinical actions such as orders and dispositions, and control simulator turns and blocking states.The three groups comprise five observation tools, eight action tools, and four control tools.Source: §B.1 · p. 14

B.2 Medical Vocabulary Resolution

  • Free-text orders pass through deterministic abbreviation lookup, semantic search over a clinical catalog, and interface-level verification against available simulator options.The semantic stage accepts matches above a cosine-similarity threshold of 0.3; the pipeline addresses shorthand mismatches with the 6,888-item order catalog.Source: §B.2 · p. 14

B.3 Pre-Action Guards and the Operating Manual

  • Interface guards and the operating manual encode workflow rules, including acuity-based triage, a treatment-completeness sweep, and pre-disposition verification.The manual begins as M0 and is the starting point for the self-evolving harness.Source: §B.3 · pp. 14–15

C Asclepius Component Artifacts

  • The released artifacts include prompts, skill modules, subagent definitions, and proposer prompts, organized by baseline, component ablations, and full Asclepius configurations.Source: C Asclepius Component Artifacts · p. 15

D Evaluation Design

The evaluation reports higher held-out overall scores for full Asclepius and improvement on critical actions across all ten batches. It uses LLM grading, ten patient batches split between search and held-out sets, and a four-dimension rubric.

D Evaluation Design

  • Each shift is graded by an LLM judge using a 1–5 rubric for diagnosis, critical actions, timeliness, and disposition; overall score is the unweighted mean of those dimensions.The judge receives the answer key and action record alongside clinical context, including vital-sign trajectories and test results.Source: D Evaluation Design · p. 15
  • Diagnosis uses clinical-equivalence matching; critical actions score the fraction completed, timeliness checks completion before deterioration thresholds, and disposition checks final placement against acceptable options.Incomplete critical actions cap the maximum timeliness score at 3; overall score is the unweighted mean of the four dimension scores.Source: D Evaluation Design · p. 15

D.1 Judge Reliability

  • The judge’s rubric has prior expert-agreement results, but the authors did not run an expert-agreement study on Asclepius traces; re-grading under four further judges reproduced the gains.The cited reliability study reports Gwet’s AC2 values of 0.93 for diagnosis, 0.86 for critical actions, 0.86 for timeliness, and 0.88 overall.Source: §D.1 · p. 16

D.2 Patient Batches

  • The ten batches each contain twelve patients; batches 1–6 are used for harness search, while batches 7–10 are held out and never seen by the proposer.Each configuration is run once per batch, with per-dimension scores averaged within each split.Source: §D.2 · p. 16

E.1 Search-Batch and Held-Out Means

  • 3.97 vs. baseline 3.78 on held-out overall score (∆= +0.19); the overall gain compresses from +0.36 on search batches to +0.19 on held-out batches.The authors interpret the persistent held-out gain as a lower bound on the generalizable component, while noting adaptation to recurring search-batch patterns.Source: §E.1 · p. 16

E.2 Per-Batch Overall Scores

  • Full Asclepius beats the baseline on critical actions in every batch, with the smallest per-batch margin at +0.17; it beats the baseline on overall score in 8 of 10 batches.The two overall-score exceptions are B3 and B6, with differences of −0.12 and −0.06.Source: §E.2 · p. 16

E.3 Component Ablation Means and Significance Tests

The ablations show that full Asclepius improves execution-related dimensions, while diagnosis and disposition do not significantly change. The evolved manual's marginal effects are not statistically separable from zero, and the appendix also reports per-trace counters and harness search scores.

E.3 Component Ablation Means and Significance Tests

  • Full Asclepius was significant at p < 0.01 or stricter on overall, critical actions, and timeliness; no configuration significantly changed diagnosis or disposition.The patient-level mixed-effects model used the baseline framework as reference, Satterthwaite degrees of freedom, and BH-FDR correction across 20 system-by-dimension comparisons; Table 8 reports corrected results.Source: §E.3 · pp. 16–17

E.4 Marginal Contribution of the Evolved Manual

  • Across all batches, the evolved manual adds +0.12 overall, +0.23 on critical actions, and +0.08 on timeliness, but neither split is statistically separable from zero.On held-out batches, the overall difference is +0.02; claims therefore remain at the system level rather than attributing an independently significant effect to the harness.Source: §E.4 · p. 17

E.5 Per-Trace Failure-Mode Counters

  • Table 10 reports per-configuration values for the three per-trace failure-mode counters across all ten patient batches.Source: §E.5 · p. 17

E.6 Harness Search Trajectory (Per-Iteration Scores)

  • Table 11 presents per-iteration search-batch scores for the harness search trajectory.Source: §E.6 · p. 17

F Additional Qualitative Traces

Three additional rollouts illustrate Asclepius components intercepting baseline failures: two close treatment gaps, while one broadens diagnosis in an atypical presentation. The trauma example comes from a held-out batch.

Trauma resuscitation completion

  • In the held-out trauma case, the treatment-completion subagent flags three omitted interventions, and the agent completes five of six before transfer.Both systems identify the fracture and arterial injury and route the patient to the operating room; the baseline omits tetanus prophylaxis, early TXA, and hemostatic resuscitation.Source: F Additional Qualitative Traces · pp. 17–18

Sepsis antibiotic completion

  • For probable urinary-source sepsis, the treatment-completion subagent flags absent empiric antibiotics, which the agent starts before admission.The baseline admits the patient without antimicrobial therapy.Source: F Additional Qualitative Traces · p. 18

Atypical NSTEMI recognition

  • For an atypical NSTEMI presentation, the diagnosis-formation subagent keeps ACS in the differential, prompting ECG and serial troponins that support NSTEMI.The baseline anchors on renal/metabolic encephalopathy and does not order cardiac tests; the Asclepius trace proceeds to cardiology-directed treatment and monitored admission.Source: F Additional Qualitative Traces · pp. 17–18
Loading 2609.13543v1…