PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments

Ruoqi Liu, Imran Q. Mohiuddin, Austin J. Schoeffler, Kavita Renduchintala, Ashwin Nayak, Prasantha L. Vemu, Shivam C. Vedak, Kameron C. Black, John L. Havlik, Isaac Ogunmola, Stephen P. Ma, Roopa Dhatt, Jonathan H. Chen

arXiv:2605.02240v1cs.AI

TL;DR

Existing medical benchmarks rarely test whether LLM agents can execute long-horizon clinical workflows in real EHR environments. PhysicianBench evaluates such workflows with physician-reviewed tasks and execution-grounded checkpoints, finding that GPT-5.5 completes only 46% of tasks at pass@1.

  • Problem

    Existing medical benchmarks rarely evaluate long-horizon, multi-step agent behavior through verified execution in real EHR environments.

  • Method

    PhysicianBench evaluates 100 physician-reviewed tasks adapted from real consultation cases in a real EHR environment with execution-grounded checkpoints.

  • Results

    46% pass@1: GPT-5.5 completes only 46% of tasks, while remaining models cluster below 32% pass@1.

  • Takeaways & Limitations

    Current LLM agents remain far from autonomously and reliably handling real-world clinical tasks.

  • Takeaways & Limitations

    PhysicianBench currently targets EHR-mediated, e-consult-style physician work rather than broader health workflows, multimodal inputs, or expanded tool ecosystems.

Abstract

from arXiv · show

We introduce PhysicianBench, a benchmark for evaluating LLM agents on physician tasks grounded in real clinical setting within electronic health record (EHR) environments. Existing medical agent benchmarks primarily focus on static knowledge recall, single-step atomic actions, or action intent without verifiable execution against the environment. As a result, they fail to capture the long-horizon, composite workflows that characterize real clinical systems. PhysicianBench comprises 100 long-horizon tasks adapted from real consultation cases between primary care and subspecialty physicians, with each task independently reviewed by a separate panel of physicians. Tasks are instantiated in an EHR environment with real patient records and accessed through the same standard APIs used by commercial EHR vendors. Tasks span 21 specialties (e.g., cardiology, endocrinology, oncology, psychiatry) and diverse workflow types (e.g., diagnosis interpretation, medication prescribing, treatment planning), requiring an average of 27 tool calls per task. Solving each task requires retrieving data across encounters, reasoning over heterogeneous clinical information, executing consequential clinical actions, and producing clinical documentation. Each task is decomposed into structured checkpoints (670 in total across the benchmark) capturing distinct stages of completion graded by task-specific scripts with execution-grounded verification. Across 13 proprietary and open-source LLM agents, the best-performing model achieves only 46% success rate (pass@1), while open-source models reach at most 19%, revealing a substantial gap between current agent capabilities and the demands of real-world clinical workflows. PhysicianBench provides a realistic and execution-grounded benchmark for measuring progress toward autonomous clinical agents.

1 Introduction

PhysicianBench evaluates whether LLM agents can reliably complete long-horizon clinical workflows in real EHR environments, addressing limitations of benchmarks centered on knowledge, conversation quality, or single-step actions. Its physician-validated, execution-grounded tasks expose substantial gaps in current agents, with the best model completing only 46% of tasks.

  • Motivation: Existing medical benchmarks rarely test long-horizon, multi-step behavior or reliable workflow completion in real EHR environments.They instead emphasize question answering, clinical conversation and response quality, or single-step atomic actions.
  • Benchmark: PhysicianBench comprises 100 physician-curated tasks from real consultations, spanning 21 specialties and diverse clinical workflows.Tasks were authored from primary-care and subspecialty consultations and independently reviewed by physicians.
  • Benchmark: 27 tool calls per task are required on average to retrieve patient data, reason across encounters, execute clinical actions, and produce documentation.Tasks run against EHR instances accessed through standard FHIR APIs.
  • Results: 46% of tasks are completed by the best-performing agent, GPT-5.5, at pass@1, versus 32% for Claude Opus 4.6, 29% for Claude Opus 4.7, and 19% for DeepSeek V4-Pro.The strongest open-source model reaches 19%, while the reported pass^3 for the best model is only 28% across three independent runs.
  • Evaluation: Execution-grounded verification compares agent actions with the postexecution EHR state, while structured checkpoints diagnose distinct stages of task completion.The benchmark uses structured per-task checkpoints to support analysis of agent failure modes.

2 Related Work

Prior medical benchmarks largely assess static clinical knowledge, conversational diagnosis, retrieval, or isolated EHR actions rather than composite workflows with consequential execution. PhysicianBench addresses this gap through long-horizon tasks in a real FHIR-based EHR, execution-grounded verification, and independently validated consultation cases.

  • Medical agent benchmarks: Traditional benchmarks such as MedQA, PubMedQA, and MedMCQA test clinical knowledge with static multiple-choice questions without tool use, EHR interaction, or action execution.HealthBench instead evaluates 5,000 open-ended health conversations using physician-designed rubrics.
  • Medical agent benchmarks: AgentClinic evaluates interactive patient-doctor conversations, but remains diagnosis-centric and omits EHR systems, structured data manipulation, and clinical action execution.Its diagnostic accuracy can drop to one-tenth of static performance.
  • Medical agent benchmarks: FHIR-AgentBench grounds 2,931 clinical questions in FHIR data but focuses exclusively on retrieval precision and answer correctness without workflow execution or action creation.Other systems use site-specific EHR schemas or target diagnosis and treatment planning, according to the supplied passage.
  • Medical agent benchmarks: MedAgentBench provides a FHIR-compliant interactive environment with 300 clinician-authored tasks across 10 templates, but predominantly evaluates single-step atomic operations.Examples include retrieving one lab value or placing one order, rather than composite multi-step workflows.
  • PhysicianBench: PhysicianBench combines long-horizon clinical workflows, a real EHR accessed through standard FHIR APIs with execution-grounded verification, and independently validated consultation cases.Its workflows span information retrieval, reasoning, action execution, and documentation.

3 Benchmark Environment and Evaluation

PhysicianBench evaluates agents in a FHIR-compliant EHR environment modeled as a deterministic POMDP with structured tool interactions. Its checkpoint-based framework assesses clinically meaningful milestones through execution-grounded grading across 670 checkpoints and 100 tasks.

  • Benchmark pipeline: PhysicianBench connects task instructions to a FHIR-compliant EHR through structured tool calls and evaluates agents against clinically grounded checkpoints.The integrated pipeline formalizes task structure, EHR interaction, and checkpoint-based evaluation.
  • Task formulation: Each task is modeled as a POMDP whose state contains EHR contents, whose action space has 14 structured tool calls, and whose transitions are deterministic.Write actions modify EHR state, whereas read actions leave it unchanged; interaction ends after a final response or the maximum step limit.
  • EHR environment: The environment uses FHIR R4 with an open-source HAPI FHIR JPA server and persistent H2 database to mirror production healthcare systems.Each task instance runs in an isolated Docker container with a dedicated FHIR server.
  • Checkpoint evaluation: 670 checkpoints across 100 tasks decompose evaluation into discrete, clinically meaningful milestones that diagnose which workflow stages challenge agents.Checkpoint types include data retrieval, clinical reasoning, and documentation, with documentation judged for completeness, clinical soundness, guideline adherence, and safety.
  • Checkpoint evaluation: Three grader types verify checkpoint completion: code graders inspect created FHIR resources, while hybrid graders combine deterministic computation with LLM-assisted extraction.The grading framework selects the grader type according to each criterion’s nature.

4 Task Design and Curation

PhysicianBench tasks are adapted from real e-consult cases and realistic de-identified EHR records, then assembled into structured task artifacts and validated through iterative physician review. The benchmark contains 100 physician-validated tasks spanning 21 clinical subspecialties and diverse workflow types.

  • Source material: Tasks are adapted from real asynchronous EHR-based e-consults in which primary care physicians seek subspecialty assessment, workup, treatment, or referral recommendations.The source cases come from clinical practice and are intentionally reframed beyond the original e-consult.
  • Patient records: Patient records derive from the de-identified STARR clinical data warehouse and receive privacy-preserving perturbations that preserve original clinical meaning.These records retain realistic co-morbidity patterns, data irregularities, and longitudinal health trajectories that fully synthetic records typically lack.
  • Task construction: Each task instance includes a natural-language instruction, a FHIR patient record with standardized terminologies, an evaluation test suite, and reference solution materials.The artifacts specify clinical context, multi-step requirements, expected deliverables, checkpoint definitions, grader logic, and review criteria.
  • Benchmark composition: 100 physician-validated tasks span 21 clinical subspecialties and four high-level clinical workflow types.Workflow subtypes include disease workup, abnormal lab evaluation, incidental finding followup, and infection evaluation.
  • Physician validation: Validation uses three iterative stages: expert checklist review, LLM-assisted artifact revision with structured diffs, and physician approval or rejection of individual changes.Rejected edits trigger repeated revision and approval cycles until all changes are approved and the task is finalized.

5 Experiments

PhysicianBench evaluates proprietary and open-source LLM agents with standardized FHIR tools using task-completion, reliability, and efficiency metrics. GPT-5.5 leads overall and across most specialties, while failures primarily reflect incomplete clinical reasoning and insufficiently grounded execution.

  • Experimental setup: Agents span proprietary and open-source LLM families and use the same minimal tool-calling loop with access to the full FHIR tool set.The evaluation includes 13 models across both model families.
  • Metrics: Pass@1 measures single-attempt task completion, Pass@3 measures success across three independent attempts, Pass^3 measures consistency, and #Turns measures tool-calling efficiency.Pass@3 estimates whether at least one of three runs succeeds, whereas Pass^3 estimates whether all three succeed.
  • Overall results: 46.3% pass@1: GPT-5.5 leads Claude Opus 4.6 by 14.6 points, while DeepSeek V4-Pro is the strongest open-source model at 18.7%.GPT-5.5 reaches 46.3% pass@1 versus 31.7% for Claude Opus 4.6; DeepSeek V4-Pro surpasses several proprietary models.
  • Specialty variation: GPT-5.5 leads seven of eight specialty groups, but the Claude family leads Nephrology/Urology at 33.3% versus GPT-5.5 at 29.2%.Within specialties, performance ranges from close to zero for weaker models to nearly 60% for GPT-5.5 in Endocrinology.
  • Task-type variation: Treatment Planning is consistently hardest, with the best non-GPT-5.5 model at 23.5% versus 43.6% on Diagnosis; GPT-5.5 peaks at 54.9% on Workup & Risk Stratification.GPT-5.5’s Workup & Risk Stratification score is 17 points above Claude Opus 4.6.
  • Failure analysis: Clinical reasoning accounts for most failures across nearly all models, while GPT-5.5’s failures concentrate less on reasoning as stronger models resolve more reasoning checkpoints.Fine-grained reasoning failures commonly involve incomplete reasoning or near-miss details such as doses, monitoring plans, or safety caveats; outright wrong conclusions are rare.

6 Discussion · Appendix

PhysicianBench evaluates LLM agents on physician workflows in real EHR environments using physician-reviewed long-horizon tasks and fine-grained, verifiable checkpoints. Its results show substantial performance limitations and motivate broader workflow coverage, collaborative evaluation, and multimodal extensions.

  • 6 Discussion: PhysicianBench contains 100 long-horizon tasks sourced from real consultation cases and authored and independently reviewed by physicians.
  • 6 Discussion: 670 fine-grained checkpoints support verifiable evaluation of task completion.
  • 6 Discussion: 12 LLM agents were comprehensively evaluated on PhysicianBench.
  • 6 Discussion: 46% of tasks were completed by the best-performing agents.
  • 6 Discussion: 28% of tasks were reliably solved across multiple attempts.
  • 6 Discussion: PhysicianBench currently targets EHR-mediated, e-consult-style physician work.
  • Appendix: Future iterations may extend coverage to discharge summaries, medication management, and other inpatient settings.
  • Appendix: Future work may evaluate agent-user collaboration with simulated physicians or patients under clinical constraints and shared decision-making.

A Task Taxonomy

PhysicianBench organizes 100 clinician-validated tasks across 21 subspecialties into 8 specialty groups, 4 task types, and 14 fine-grained subtypes. The benchmark inventory is presented through complementary tables and a figure summarizing specialty and task-type distributions.

  • Taxonomy structure: 100 tasks span 21 subspecialties organized into 8 specialty groups and 4 task types subdivided into 14 fine-grained subtypes.Specialty groups represent each task’s primary clinical domain, while task types reformulate the DICTUM clinical decision-making taxonomy.
  • Task types: 4 task types include Diagnosis & Interpretation, Medication Prescribing, Treatment Planning, and a fourth type not fully specified in the supplied passage.The supplied passage names Diagnosis & Interpretation and Medication Prescribing subtypes, including result and imaging interpretation, new therapy, regimen adjustment, and acute treatment.
  • Inventory presentation: 2 complementary enumerations present the full task inventory: Table A1 groups tasks by type and subtype, while Table A2 expands them across 21 subspecialties.Figure 3 summarizes the marginal distributions over specialty and task type.

B Checkpoint Grader Specifications · C Data Annotation Protocol · C.1 Data Collection

PhysicianBench uses three checkpoint-grading approaches to verify executable actions, objectively defined values, and clinical reasoning or documentation. Its scenarios come from de-identified STARR e-consults spanning diverse specialties and clinical task types.

  • B Checkpoint Grader Specifications: Three grader types are selected according to the nature of each checkpoint criterion.The specified types are code graders, hybrid graders, and LLM-judge graders.
  • B Checkpoint Grader Specifications: Code graders query FHIR resources created after the task timestamp and validate types, statuses, coded values, and dosage parameters.Medication-order checks can require a matching drug name pattern, dose range, and frequency.
  • C Data Annotation Protocol: Hybrid graders deterministically compute ground-truth values, extract corresponding values from free-text outputs, and compare them using deterministic logic.This accommodates variable reporting formats while retaining objective scoring criteria.
  • C Data Annotation Protocol: LLM-judge graders assess clinical reasoning and documentation with rubrics specifying required elements, sequential verification steps, and PASS / PARTIAL / FAIL thresholds.Documentation rubrics can require correct laboratory values and appropriate citations.
  • C.1 Data Collection: 185 candidate cases formed the initial pool drawn from de-identified STARR e-consults submitted by primary care providers to subspecialists.The provided passage indicates that cases were subsequently filtered, but the filtering criteria are truncated.
  • C.1 Data Collection: The e-consult cases span diagnostic workup, treatment planning, medication management, referral coordination, and clinical documentation.They were selected to cover diverse subspecialties and task types.

C.2 Patient Data De-identification · C.3 Annotation Guidelines and Quality Control · D Annotator Details

The benchmark protects patient privacy through plausibility-preserving structured-data perturbation and PHI-safe rewriting, while clinicians iteratively review task quality, safety, and EHR fidelity. Validation involved 11 clinicians acting as second-tier reviewers of subspecialist-authored cases.

  • C.2 Patient Data De-identification: Structured EHR values were shifted within assay reference ranges, while categorical fields were anonymized by dropping entries and inserting clinically plausible distractors.These procedures preserved clinical plausibility while preventing trivial identification of target conditions from the record.
  • C.2 Patient Data De-identification: Each task artifact underwent a four-stage clinician validation pipeline: draft generation, checklist review, agentic revision, and approval review iterated to convergence.The approval stage presented proposed edits for clinician acceptance or rejection.
  • C.2 Patient Data De-identification: Unstructured clinical notes were rewritten by PHI-safe SecureGPT models inside a secure institutional enclave while preserving clinical meaning and key findings.SecureGPT was prompted to remove HIPAA Safe Harbor PHI categories and produce substantially different text.
  • C.3 Annotation Guidelines and Quality Control: Reviewers assessed instruction clarity, clinical reasoning validity, completeness, patient safety, and consistency with the de-identified EHR record.Safety concerns triggered task revision or exclusion, while cited values, dates, and findings were checked against the reference panel.
  • C.3 Annotation Guidelines and Quality Control: Reviewers recorded categorical checklist responses and justified non-positive assessments using the full de-identified EHR and curated summary panel.Each review session took approximately 20 minutes per task, and tasks required at least two review rounds on average.
  • C.3 Annotation Guidelines and Quality Control: The annotation application supported initial checklist review and diff-based approval, showing reviewer comments alongside before/after edits for clinician decisions.Figures A2 and A3 depict the two principal interactive screens.
  • D Annotator Details: 11 clinicians recruited through Stanford institutional networks validated tasks as second-tier reviewers of cases whose clinical answers were authored by board-certified subspecialists.They verified fidelity to the original answer, clinical coherence, and absence of newly introduced safety issues.
  • D Annotator Details: The annotator cohort’s demographic characteristics were summarized in Table A3.The table is identified as the demographic summary of the 11 expert annotators.

E Agent Framework Implementation

The framework is a compact Python implementation built directly on the OpenAI-compatible chat completions API, exposing a standardized EHR interaction interface without third-party agent SDKs. It adds safeguards for tool failures and output size while using fixed experimental configuration and isolated task environments.

  • Framework design: The agent framework is implemented as a ∼300-line Python module wrapping the OpenAI-compatible chat completions API and avoiding third-party agent SDKs.This design aims to attribute performance differences to model capability rather than model-specific scaffolding.
  • Tool interface: 14 registered tools comprise 13 FHIR-backed EHR tools and one workspace file-writing tool.Search tools issue GET requests, while create tools issue POST requests that persist on the task-local server.
  • Robustness: Tool errors and malformed arguments are returned to the model as structured error messages, while outputs exceeding 10,000 characters are truncated with recovery guidance.The safeguards allow recovery from transient or semantic errors and help prevent large FHIR bundles from consuming the context window.
  • Configuration: Experiments use max_steps=100, parallel_tool_calls=True, provider-default temperature, 10,000-character tool-output truncation, and provider-specific reasoning effort set to high.Each task runs against a fresh FHIR server container, eliminating state leakage across tasks.

F Additional Results · G Representative Failure Trajectory · Task: Adrenal Insufficiency Management

The representative Adrenal Insufficiency Management trajectory shows an agent retrieving extensive clinical data but misclassifying secondary adrenal insufficiency as primary disease, recommending inappropriate treatment, and failing to execute part of its written plan. The task involved a 69-year-old woman with symptoms despite glucocorticoid replacement and ended after 44 tool calls with a saved management plan.

  • G Representative Failure Trajectory: A 69-year-old woman’s secondary adrenal insufficiency was attributed to chronic ophthalmic corticosteroid exposure and opioid use, with preserved aldosterone secretion.The task asked an endocrinologist agent to evaluate replacement adequacy.
  • Task: Adrenal Insufficiency Management: The agent retrieved Condition, MedicationRequest, Observation, and clinical-note resources across the patient record before composing its plan.Retrieved data included medications, laboratory results, vital signs, procedures, diagnoses, and endocrinology notes.
  • G Representative Failure Trajectory: The agent incorrectly reclassified the condition as primary adrenal insufficiency based on elevated renin and recommended fludrocortisone, which is not indicated in secondary adrenal insufficiency.The written plan proposed fludrocortisone 0.1 mg PO daily.
  • G Representative Failure Trajectory: The agent named an urgent cardiology referral in its written plan but never created the referral, illustrating a failure to execute a consequential action.The trajectory explicitly contrasts the written recommendation with the absent referral action.
  • Task: Adrenal Insufficiency Management: The patient record showed labile blood pressure, persistent tachycardia, sodium 135.1 mmol/L, and potassium 4.95 mmol/L during the evaluation.Blood pressure ranged from 118/90–164/86, while heart rate ranged from 105–127 bpm.
  • Task: Adrenal Insufficiency Management: 44 tool calls ended the trajectory after the agent wrote a 14,372-byte management plan file.The output file was reported as successfully written.
  • Task: Adrenal Insufficiency Management: The written plan also increased hydrocortisone from 15 mg/day to 20 mg/day, proposed confirmatory testing and sick-day education, and conditioned cardiology referral on persistent blood-pressure control problems.It specified 15 mg AM plus 5 mg PM and cardiology referral if blood pressure remained uncontrolled.
Loading 2605.02240v1…