Source-linked AI summary
MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents
Yixing Jiang, Kameron C. Black, Gloria Geng, Danny Park, James Zou, Andrew Y. Ng, Jonathan H. Chen
TL;DR
Medical LLM agents lack a standardized benchmark for complex, interactive medical-record tasks. MedAgentBench addresses this gap with clinician-authored tasks, realistic patient profiles, and a FHIR-compliant environment; evaluations show meaningful but incomplete performance across task categories.
Problem
A standardized benchmark for evaluating LLM agent capabilities in complex medical applications is lacking, despite the need to assess interactive healthcare tasks.
Method
MedAgentBench combines 300 clinician-authored tasks from 10 categories, 100 patient profiles with over 700,000 records, and a FHIR-compliant interactive environment.
Results
Models show non-trivial performance, with performance generally better on query-based than action-based tasks and substantial variation across categories.
Takeaways & Limitations
MedAgentBench provides a standardized framework for systematically assessing and advancing medical LLM agents in interactive settings.
Takeaways & Limitations
The environment should not be used directly in production because it lacks security implementation and enterprise logging.
Abstract
from arXiv · showhide
Recent large language models (LLMs) have demonstrated significant advancements, particularly in their ability to serve as agents thereby surpassing their traditional role as chatbots. These agents can leverage their planning and tool utilization capabilities to address tasks specified at a high level. However, a standardized dataset to benchmark the agent capabilities of LLMs in medical applications is currently lacking, making the evaluation of LLMs on complex tasks in interactive healthcare environments challenging. To address this gap, we introduce MedAgentBench, a broad evaluation suite designed to assess the agent capabilities of large language models within medical records contexts. MedAgentBench encompasses 300 patient-specific clinically-derived tasks from 10 categories written by human physicians, realistic profiles of 100 patients with over 700,000 data elements, a FHIR-compliant interactive environment, and an accompanying codebase. The environment uses the standard APIs and communication infrastructure used in modern EMR systems, so it can be easily migrated into live EMR systems. MedAgentBench presents an unsaturated agent-oriented benchmark that current state-of-the-art LLMs exhibit some ability to succeed at. The best model (Claude 3.5 Sonnet v2) achieves a success rate of 69.67%. However, there is still substantial space for improvement which gives the community a next direction to optimize. Furthermore, there is significant variation in performance across task categories. MedAgentBench establishes this and is publicly available at https://github.com/stanfordmlgroup/MedAgentBench , offering a valuable framework for model developers to track progress and drive continuous improvements in the agent capabilities of large language models within the medical domain.
1 Introduction
MedAgentBench addresses the lack of standardized medical benchmarks for LLM agents by evaluating complex, interactive medical-record tasks. It combines clinician-authored tasks, realistic patient profiles, and a FHIR-compliant environment to support systematic assessment beyond traditional question answering.
- Agent Capabilities: Agentic healthcare workflows can support clinical and administrative work by interpreting high-level instructions, planning actions, integrating patient data, and interacting with external systems.Examples include risk assessment, clinical triage, EHR configuration, and insurance prior authorization.
- Motivation: Medical LLM benchmarks lack standardized evaluation of agent capabilities in complex interactive healthcare environments.Medical records involve specialized coding systems, abbreviations, and longitudinal data, while robust evaluation is important for safe deployment.
- Contribution: The benchmark is intended to help advance LLMs from chatbot-style systems toward sophisticated clinical agent systems.Physicians spend roughly 27 percent of their time on direct clinical care, with the remainder involving documentation and administrative tasks.
- Dataset: MedAgentBench contains 300 clinically relevant, verifiable tasks from 10 categories written by licensed clinicians.The benchmark targets agent capabilities beyond traditional question answering and requires autonomous interaction with medical-record environments.
- Interactive Environment: The benchmark provides a FHIR-compliant environment with 100 realistic patient profiles, over 700,000 records, and standard API-based interaction.Its environment is designed so benchmark tasks can migrate toward real-world EMR settings.
2 MedAgentBench
MedAgentBench models realistic clinician-facing medical-record work as high-level instructions paired with hospital-specific context. Its tasks cover broad categories and can be executed through interactions with an EHR environment.
- Workflow: A typical workflow converts a clinician’s high-level task into planned FHIR function calls and returns a summary of completed actions.The orchestrator interprets the request, executes requests against the FHIR server, and reports the performed tasks to the user.
- Task Design: Two internal medicine physicians submitted 300 clinically derived tasks selected for complexity and clinical relevance.The scope emphasizes inpatient and outpatient scenarios suited to computer interaction rather than surgical or procedural interventions.
- Task Structure: Task instructions commonly specify a patient MRN, a time window, and the data to record, such as a blood pressure value.Tasks may also include NDC, LOINC, base names, and SNOMED codes where applicable.
- Task Categories: The benchmark groups ten specific task categories into seven broad categories, each illustrated with a user instruction and hospital-specific EHR context.Curly-bracket placeholders such as {MRN} represent patient-specific values.
- Task Context: Hospital-specific context is managed by EHR administrators because formularies and preferred medication formulations can differ across hospitals.Instructions are written by users such as clinicians, whereas context reflects local EHR configuration.
2.2 Patient profiles
Patient profiles are built from deidentified clinical warehouse data and sampled to provide realistic longitudinal medical-record contexts. The environment represents selected clinical observations and orders as structured FHIR resources.
- Source Data: Patient profiles use deidentified, patient-level timestamp-jittered data covering five years of labs, vital signs, procedures, diagnoses, and medication orders.The source is the STARR clinical data warehouse, with November 13, 2018 used as the cutoff date.
- Patient Cohort: 100 patients were randomly sampled from a cohort with an inpatient sodium test ordered on November 13, 2023.The sodium test served as an anchor because it is common and clinically significant in inpatient settings.
- Deidentification: Protected health information was removed, and synthetic MRNs, names, phone numbers, and addresses were generated for the profiles.Synthetic MRNs use seven random digits prefixed with S to match Stanford Hospital formatting.
- Observations: Lab results and selected vital signs are uploaded as Observation resources in the environment.Vital signs include heart rate, SpO2, respiratory rate, FiO2, blood pressure, and temperature, with measurement values and timestamps.
- Clinical Resources: Procedure orders, diagnoses, and medication orders are transformed into structured Procedure, Condition, and MedicationRequest resources.The extraction process removes records with missing required fields, removes PRN medication orders, and imputes missing procedure quantities as one.
2.3 Environment setup
MedAgentBench uses a FHIR-compliant environment to simulate live EMR interaction, while remaining unsuitable for production because it lacks security implementation and enterprise logging.
- Infrastructure: The environment uses the open-sourced HAPI FHIR JPA with a persistent H2 database and parallel patient-profile uploads.A Docker image supports easy setup.
- Deployment boundary: The environment can facilitate direct migration to live EMR systems but should not be used in production without security implementation and enterprise logging.Its realistic infrastructure does not provide production safeguards.
- Deployment: The simulation runs on a Google Cloud c2d-standard-2 virtual machine and supports HTTP-based retrieval and modification of patient data.A web frontend also permits users to retrieve or modify data.
2.4 Evaluation setup
The evaluation benchmarks LLM agents through constrained FHIR interactions, manually curated grading, and single-attempt success rates across multiple state-of-the-art models.
- Evaluation: Task success is the main metric, with reference comparisons for query tasks and rule-based payload checks for action tasks.Invalid actions or exceeding the interaction-round limit count as failures.
- Evaluation: The benchmark exclusively uses pass@1 to reflect healthcare’s stringent accuracy requirements and single-attempt deployment constraints.Repeated-sampling metrics such as pass@k are not used.
- Models: The study evaluates state-of-the-art models from multiple providers and sizes, generally using temperature zero.o3-mini is the exception to the temperature setting.
- Agent orchestrator: The orchestrator exposes nine FHIR functions and requires each round to select a GET request, POST request, or finish action.The functions cover searches and record creation for conditions, labs, vitals, medications, procedures, and patients.
- Agent orchestrator: The orchestrator is a high-level abstraction that permits compound agents with hierarchical reasoning, specialized sub-agents, planning, or retrieval-augmented reasoning.Advanced systems remain subject to limited function access and an 8-round interaction cap.
- Reported metrics: Table 3 reports overall success rate, query SR, and action SR for the evaluated LLMs, highlighting the best value in each column.These metrics distinguish general performance from query and action task performance.
2.5 Main results
MedAgentBench shows meaningful but insufficient LLM-agent performance, with substantial differences between retrieval and record-modification tasks and recurring syntax and formatting errors.
- Overall performance: 69.67% overall success rate was achieved by Claude 3.5 Sonnet, the best-performing model among 11 evaluated LLMs.Most models showed non-trivial performance.
- Limitations: Current state-of-the-art LLMs remain unable to serve as highly reliable agents in high-stakes healthcare settings.The authors identify a remaining gap between closed and open-weights models.
- Task-type analysis: Most models performed better on query-based tasks than action-based tasks, except Gemini 1.5 Pro and Qwen2.5.The benchmark separates 150 GET-only tasks from 150 tasks requiring POST-based record modification, often after GET requests.
- Common error patterns: Gemini 2.0 Flash produced invalid actions in 54% of cases, commonly violating the required response syntax.The model often wrapped code in tool_code or JSON blocks despite instructions forbidding additional text.
- Common error patterns: Agents also failed by returning textual or incorrectly structured answers when the benchmark required an exact numerical or structured output.One example changes the expected [5.4] into a value-labeled structure.
3 Discussion
MedAgentBench targets complex, specific medical-agent tasks and provides a benchmark for evaluating progress in interactive medical applications. Results indicate stronger performance on query-based than action-based tasks, while the benchmark remains limited in its coverage and realism.
- Medical-agent tasks are more specific and intricate than general agent tasks, motivating a dedicated medical benchmark.
- MedAgentBench contains 300 tasks and 100 patient profiles, balancing performance-estimation accuracy against evaluation cost.
- Many leading LLMs perform better on query-based than action-based tasks, indicating weaker navigation of complex decision-making.
- The benchmark does not capture full real-world coordination across medical teams and may contain population bias because profiles derive from Stanford Hospital records.
- MedAgentBench offers medical-specific tasks, an interactive environment, and standardized evaluation, but current models are not yet reliable on the full complexity of clinically relevant tasks.
A.1 Screenshot of the interactive environment
The appendix presents a screenshot of the frontend for MedAgentBench’s FHIR-compliant interactive environment.
- Figure 3 shows the frontend of the FHIR-compliant interactive environment.
A.2 Prompts for the agent system
The agent prompt instructs models to interpret questions, invoke FHIR functions through strictly formatted GET or POST calls, and signal completion with a finish command.
- The agent must use FHIR functions to answer questions through one or more tool calls.
- GET calls must use a URL with query parameters, while POST calls must include a JSON payload.
- After completing all requested tasks, the agent must return answers in the finish([answer1, answer2, ...]) format.
- Responses must follow one of the three specified formats without additional text.
- The prompt supplies the available functions, API base, contextual information, and user question to the agent.
A.3 Subgroup analysis based on difficulty level
MedAgentBench groups tasks by the number of required steps and reports model success rates separately for easy, medium, and hard tasks.
- Tasks are categorized as easy for one step, medium for two steps, and hard for at least three steps.
- Table 4 reports state-of-the-art LLM success rates across the three difficulty levels, highlighting the highest rate in each column.