GPAgentBench-2K: Benchmarking Large Language Model Agents in Complex Clinical Action Space

• Static-prediction benchmarks assume structured patient data is available upfront, bypassing sequential evidence gathering; existing MDPs also use coarse action sets and treat safety as a soft penalty.Real-world clinical decisions must optimize diagnostic performance within strict safety constraints, since unsafe actions cannot be compensated for by downstream task performance.Source: §1 · pp. 1–2
• GPAgentBench-2K uses expert-validated GP encounter records to model six actions—ask, body_exam, test, diagnose, treat, and refer—with workflow masking and diagnose-and-manage or abstain-and-refer outcomes.It supports evaluation of diagnosis and treatment accuracy, safety, and efficiency.Source: §1 · p. 2
• C-GRPO improves over unconstrained RL baselines, but absolute safety-failure rates remain high, leaving clinically acceptable safety in complex action spaces unresolved.C-GRPO augments GRPO with a Lagrangian relaxation to explicitly penalize safety violations.Source: §1 · p. 2

Non-MDP Clinical Agent Environment

• Static-prediction frameworks assume complete patient data upfront, while persona-based multi-agent designs divide work into narrow roles (Zhu et al., 2025).Although multi-agent designs can exceed a single model’s capability (Wang et al., 2024), narrow roles bypass sequential evidence gathering and safety-informed abstention under uncertainty.Source: §2 · p. 2

MDP Clinical Agent Environment

• GPAgentBench-2K is the first clinical-agent CMDP benchmark, requiring full-spectrum clinical actions while optimizing diagnosis within strict safety boundaries.Unlike unconstrained MDPs with diagnosis as the sole terminal outcome, it restricts how agents trade safety for diagnostic accuracy.Source: §2 · p. 3
• Prior MDP environments simulate evidence gathering through dialogue or test selection, but DoctorAgent reduces actions to asking and diagnosing (Feng et al., 2025).These environments remain unconstrained and require diagnosis as the determined terminal outcome, allowing safety to be traded for diagnostic accuracy.Source: §2 · p. 3

Data Collection

• The benchmark draws on real-world GP visits spanning 9 primary and 62 secondary departments, with demographics, symptoms, history, exams, test results, and management plans.Source: §3 · p. 3
• Its cases support both diagnose-and-manage and safety-informed abstain-and-refer trajectories, enabling agents to select actions as evidence evolves rather than follow one fixed outcome.Source: §3 · p. 3
• Two senior physicians validated cases against trajectory-specific criteria, yielding 2,428 retained cases with an average of 4.9 symptoms each.Cases averaged 4.1 physical exams and 2.9 diagnostic tests; the dataset includes 1,050 unique symptoms and 584 unique test items, and all cases were de-identified.Source: §3 · pp. 3–4

Data Preprocessing

• Gemini-3-Flash converted records to JSON using only explicit findings, with absent fields set to null and diagnoses and treatment recommendations removed to prevent label leakage.Processed files were manually cross-referenced against original records, and clinical experts inspected a randomly selected 20% subset.Source: §3 · p. 4

4.1 Persona-Driven Patient Simulator.

• Patient personas vary across personality, language proficiency, history-recall fidelity, and cognitive disorientation, requiring agents to elicit reliable evidence despite communication hurdles.For example, a disoriented patient with low anamnestic fidelity may omit critical history, prompting targeted clarification.Source: §4.1 · p. 4
• Physician audits found high simulator fidelity across factual consistency, clinical realism, and persona adherence, while evaluation audits showed strong agreement and robust frontier-model rankings.Two senior physicians achieved Cohen’s κ = 0.92 for Dx accuracy and κ = 0.89 for Tx score; alternative-judge re-scoring preserved rankings within reported standard errors.Source: §4.1 · p. 4, §4.2 · p. 5

4.2 Constrained Markov Decision Process.

• The CMDP state accumulates dialogue and revealed findings, while its objective maximizes expected return subject to safety and diagnostic-cost feasibility thresholds.Safety costs capture missed referrals and unnecessary referrals; efficiency costs track dialogue turns and test expenditure.Source: §4.2 · pp. 4–5
• Six clinical actions follow an expert-defined workflow: agents gather evidence, then refer to abstain or diagnose and treat, with admissible actions enforced at each state.Body exams and tests use record-derived targets; unavailable requests return unavailable results rather than fabricated evidence.Source: §4.2 · p. 5
• Clinical-quality rewards comprise diagnosis, treatment, and episode-level management accuracy, with Dx and Tx judged against ground-truth labels and management scored by terminal pathway.Correct terminal management means treating diagnose-and-manage cases and referring abstain-and-refer cases.Source: §4.2 · p. 5

5.1 Zero-Shot Evaluation Results

• As the action set expands from two to six actions, Dx accuracy and Tx score decline monotonically across models, with Dx accuracy drops reaching 21 pp.Adding ask produces an average 4.1 pp Dx-accuracy drop, adding test produces 7.0 pp, and adding refer produces an average 2.9 pp Tx-score drop; cases and personas stay fixed, with evidence from omitted actions supplied upfront.Source: §5.1 · p. 6
• Scaling Qwen2.5 from 7B to 72B raises Dx accuracy from 40.4% to 54.1% and Tx score from 18.4% to 25.4%, but does not reliably improve safety or cost.Qwen2.5 (72B) misses 54.7% of high-risk patients and incurs $473 in diagnostic costs, while Kimi K2.6 spends $34; larger open-source models also show no consistent safety or cost gains over models with over 10 times fewer parameters.Source: §5.1 · pp. 6–7
• Top Dx accuracy and Tx score do not ensure safe triage: Claude Opus 4.7 leads Dx accuracy at 70.0%, while GPT-5.4 leads Tx score at 53.2%.Models cluster as overconfident or overcautious: Sonnet 4.6 has a 13.4% Miss GP rate but misses 77.1% of required referrals, while Kimi K2.6 has 56.7% Dx accuracy and 71.8% Miss GP.Source: §5.1 · p. 6, Table 1 · p. 7
• Clinical QA fine-tuning and RL in oversimplified action spaces do not close the safety-accuracy gap; Huatuo-o1 (7B) management accuracy falls to 20.6%, with both safety-failure rates above 77%.Huatuo-o1 (72B) improves management accuracy over its base by only 2.1 pp while its Miss GP rate rises 7.4 pp; DoctorAgent-RL (7B), trained with only ask and diagnose, reaches 41.0% Dx accuracy versus 40.4% for the base and averages 9.44 turns.Source: §5.1 · p. 7

5.2 Reinforcement Learning Results

• C-GRPO exceeds GRPO by 11.8 pp in Dx accuracy, 2.1 pp in Tx score, and 3.6 pp in management accuracy, yet misses 82.1% of required referrals.C-GRPO also lowers Miss-GP to 26.1%, but its Lagrangian relaxation does not provide formal safety guarantees; in a high-risk case, GPAgent refers after detecting a rectal mass while the base model manages locally.Source: Table 2 · p. 8, §5.2 · pp. 8–9

5.3 Step-Level Decision Quality

• Step-level quality tracks outcome rankings: DoctorAgent-RL (7B) duplicates known questions for 36.7% of turns and duplicates exams and tests for 48.9%.GPAgent (C-GRPO) duplicates questioning for 1.0% of turns, on par with GPT-5.4, while duplicating fewer exams and tests than its Qwen2.5 (7B) base; exact trajectory matching is not evaluated because multiple evidence paths may be clinically reasonable.Source: Table 3 · p. 9, §5.3 · p. 9

6 Conclusion

• GPAgentBench-2K is a CMDP-based clinical-agent benchmark spanning six foundational actions and making safety abstention a first-class clinical outcome.The benchmark is designed to drive progress toward safe clinical agents.Source: §6 · p. 9
• Frontier models with high diagnostic accuracy violate safety constraints in over half of high-risk cases, amid performance drops as the action space becomes complex.The evaluation covers 16 LLMs.Source: §6 · p. 9
• C-GRPO improves diagnostic accuracy and reduces the safety trade-off of scalarized RL, but a substantial residual safety gap remains.The conclusion identifies safe clinical agents as an open challenge the benchmark is designed to drive progress on.Source: §6 · p. 9

Limitations

• The English-only benchmark, simulated patients, institution-specific referral norms, and expert-designed action masks limit generalizability and require further validation against real-world practice.Transfer to other languages and healthcare systems remains open; the benchmark is not a set of clinical guidelines.Source: Limitations · p. 9

A.1 Examples of Raw Clinical Records

• The raw-record examples contrast a routine headache encounter discharged with follow-up against urgent hematuria requiring same-day emergency and urology referral and hospital admission.The routine encounter has a normal physical examination and conservative management.Source: §A.1 · pp. 11–12, Figure 6 · p. 13, Figure 7 · p. 14

A.2 Prompt for Data Processing

• The processing pipeline uses Gemini-3-Flash with a two-part prompt: system instructions define persona and constraints, while the user prompt supplies the schema, extraction rules, and raw clinical text.Case ID and raw-text placeholders are populated during execution.Source: §A.2 · p. 12
• The prompt requires source-only extraction, prohibits unsupported facts, uses null or empty lists for missing or uncertain fields, and requests valid JSON without markdown.It instructs the model to produce the JSON only.Source: §A.2 · pp. 12, 15
• The target JSON schema organizes records into patient and encounter fields, vital signs, physical examination, test results, referral details, and treatment and follow-up fields.The listed fields include age and sex, diagnosis and disposition, body-system findings, and tests such as ECG, CBC, and imaging.Source: §A.2 · p. 12
• Extraction rules map source sections directly to corresponding fields and require missing or unstated findings to remain null rather than be inferred.Diagnosis is left null for external population; vitals, physical examination, and test results map from their named record sections.Source: §A.2 · p. 12
• Referral extraction records whether referral or transfer is requested, its urgency, and its destination using the visit summary or management/disposition text.The prompt gives urgency examples from none to emergent and destinations including the Emergency Department, Urology, and GP.Source: §A.2 · p. 15
• Treatment extraction captures medications, non-pharmacologic advice, and follow-up instructions or monitoring parameters from the record’s medication and management sections.The prompt examples include supportive care, hydration, rest, and lifestyle advice.Source: §A.2 · p. 15

A.3 Examples of Processed Data

• The two processed encounter outputs demonstrate the structured representation and filtering of diagnostic labels described in the extraction process.Both examples have diagnosis set to null.Source: §A.3 · p. 15
• One example records a discharged patient with no referral needed and routine GP follow-up if symptoms persist or worsen.Source: §A.3 · p. 15
• The other records an admitted patient with elevated white blood cells and urinalysis findings, urgent referral to the Emergency Department, and same-day assessment.Source: §A.3 · pp. 15–16

A.4 Statistics of the Dataset

• Specialist referrals span 29 specialties, from frequent respiratory, orthopaedic, and oncology cases to fields with one case each; the dataset contains 756 unique disease categories.The specialty distribution is long-tailed.Source: §A.4 · p. 16
• 4.9 symptoms per case versus 2.2 in initial complaints, alongside 4.1 examinations and 2.9 tests per patient, characterize iterative evidence gathering.The dataset includes 1,050 symptom descriptions and 584 unique diagnostic items.Source: §A.4 · pp. 16–17
• 45.1% (1,097 cases) are manageable by general practitioners, while 54.9% (1,337 cases) require specialist referral.The referral cases span 29 specialties, with respiratory medicine, trauma and orthopaedic surgery, and oncology most frequent.Source: §A.4 · p. 16, Figure 9 · p. 17

B Extended Related Work

• Static-note classifiers and role-partitioned pipelines omit autonomous evidence gathering or leave GP agents to classify or route, avoiding the full safety-management trade-off.Some frameworks also treat diagnostic cost as a soft penalty rather than a strict resource constraint.Source: B Extended Related Work · pp. 17, 20
• Interactive and EHR benchmarks add dialogue, tools, or execution-grounded tasks, but omit parts of the unified primary-care decision process, such as examinations, treatment, or referral.The paper distinguishes EHR-centered record actions from recursively gathering evidence and choosing treatment or referral under uncertainty.Source: B Extended Related Work · p. 20

C.1 Experiment Setting

• Each zero-shot turn uses the same system prompt, reconstructed with the round, case-specific examination and test targets, transcript, workflow state, and admissible actions.The prompt specifies the GP role, six-action schema, and clinical workflow guidelines.Source: §C.1 · pp. 20–21
• The six-action schema uses ask, body_exam, and test to gather evidence, with tests requested when they can change management.The prompt calls for one concise patient-facing question per ask action and recommends gathering history before exams or tests.Source: §C.1 · p. 21
• After enough evidence, diagnose precedes treat; refer is terminal when primary-care management is unsafe or insufficient, and both refer and treat end the consultation.Treat is valid only after a previous diagnose action, while diagnosis itself is non-terminal.Source: §C.1 · p. 21

C.2 Reward, Cost, and Evaluation Metrics

• Zero-shot metrics come from sampled policy rollouts and are averaged across test cases, whose terminal action is treat with a diagnosis and plan or refer.Ground-truth labels indicate whether a case is safely manageable in primary care or requires specialist referral.Source: §C.2 · p. 22
• Diagnosis accuracy is a raw binary zero-shot score from GPT-5.4 for clinical equivalence to the ground truth, allowing synonyms and minor specificity differences but penalizing vague diagnoses.For RL, the score is scaled by the diagnosis-reward weight.Source: §C.2 · p. 22
• Management accuracy scores correct terminal actions, while binary safety cost penalizes incorrect actions as a CMDP constraint; its multiplier is capped at 6 without guaranteeing constraint satisfaction.During RL, terminal-action correctness is driven by safety cost rather than a separate reward term.Source: §C.2 · p. 23
• Safety cost covers missed referrals when specialist escalation is needed and missed local management when referral is unnecessary.These correspond respectively to false negatives and false positives when referral is treated as the positive class.Source: §C.2 · p. 23
• Diagnostic test cost is a thresholded constraint: exams and vital checks are free, while laboratory and imaging tests incur monetary penalties based on 2024–2025 US fee schedules.Costs are normalized so 0.2 corresponds to $100 USD per consultation, penalized adaptively during RL, and reported raw per episode in evaluation (Centers for Medicare & Medicaid Services, 2025; AFC Urgent Care, 2024).Source: §C.2 · p. 23

C.3 Automatic Judge Validation

• Judge–physician agreement was κ = 0.92 for Dx accuracy and κ = 0.89 for Tx score across a 200-output audit.Two senior physicians independently scored sampled diagnose-and-manage outputs using the judge’s rubric.Source: §C.3 · p. 23
• Cross-judge re-scoring changed frontier-model scores by less than 1 pp, preserved rankings, and showed no judge preference for its own model family.Three frontier models were re-scored with Claude Opus 4.7 using the same prompts and rubric; GPT-5.4 had the lowest Dx accuracy among them when serving as judge.Source: §C.3 · pp. 23–24, Table 5 · p. 24

C.4 Patient Simulator Fidelity Audit

• The simulator’s patient turns were highly source-consistent and clinically plausible, and its persona-adherence ratings indicated it reflected challenging personas.Physicians audited 32 trajectories, two per evaluated model, contrasting standard and challenging personas; agreement was strong across all dimensions.Source: §C.4 · p. 24, Table 6 · p. 24

C.5 Constrained Group Relative Policy Optimization Details

• C-GRPO masks its policy to admissible actions, steering sampled trajectories toward valid clinical actions and focusing relative-advantage signals on meaningful clinical variation.The mask is applied at each step over the environment’s admissible action set.Source: §C.5 · p. 25
• C-GRPO combines independently standardized reward and cost components using Lagrangian weights, then updates policy and dual multipliers through iterative primal-dual optimization.The policy uses a PPO-style clipped surrogate; dual updates respond to empirical constraint violations.Source: §C.5 · p. 25
• Because safety multipliers could destabilize training, C-GRPO caps λ_safety at p_mgmt = 6, which does not guarantee constraint satisfaction.The method updates multipliers for safety and diagnostic-cost constraints and reports achieved safety levels directly.Source: §C.5 · p. 25

C.6 Hyperparameter

• For fair comparison, baseline GRPO and C-GRPO were each tuned over diagnosis and treatment reward weights and the safety multiplier cap, with each reported at its best-performing combination.The selected values are listed in Table 7.Source: §C.6 · p. 25, Table 7 · p. 26
• The environment applies fixed penalties for inadmissible workflow actions and for failing to reach a terminal action within the 10-turn limit.These are the workflow-violation and timeout penalties, respectively.Source: §C.6 · pp. 25–26

C.7 Additional Results

• Appendix results cover remaining off-the-shelf models, dialogue turns, RL comparisons with standard errors, and GRPO/C-GRPO return trajectories.The supplied passages identify these materials but do not provide their numerical results.Source: §C.7 · p. 26, Table 8 · p. 27, Table 9 · p. 27

D.1 Potential Risks

• Real-EHR records carry residual re-identification risk despite de-identification, and audited management plans may be misread as universal advice rather than localized decisions.The dataset is for academic evaluation only and must not guide patient care or real-world clinical decision-making.Source: §D.1 · p. 26

D.2 Intended Use Specifications

• The benchmark supports academic research on clinical-agent workflows and safe decisions, but is not a medical device or standalone diagnostic system.Real-world deployment and high-stakes medical or commercial use are prohibited absent extensive clinical validation, regulatory approval, and ethical review.Source: §D.2 · p. 26

D.3 Terms of Use

• Users may not use the dataset for commercial purposes without written consent, clinical decision-making, patient care, or re-identification, and must attribute the original authors.These terms apply to use or reference of the code, environment, and datasets.Source: §D.3 · p. 26
• The project is provided without warranties, limits author liability for resulting damages, and reserves the right to change terms or remove cases over privacy risks.The disclaimer includes clinical accuracy and fitness for purpose; the liability clause covers direct and consequential damages.Source: §D.3 · pp. 26–27

D.4.1 Compliance with Existing Artifact Usage

• Clinical encounter records were processed under applicable data-governance requirements, de-identified, structured into cases, and manually checked against source notes.These steps aim to protect patient privacy and reduce extraction errors and hallucinations.Source: §D.4.1 · p. 27
• External APIs, open-source checkpoints, and clinical cost references were used only for benchmarking, simulation, and evaluation under their licenses and terms; artifacts are not redistributed unless permitted.The passage states that proprietary model outputs and third-party artifacts are not redistributed absent explicit license permission.Source: §D.4.1 · p. 27

D.4.2 Specification of Intended Use for Created Artifacts

• GPAGENTBENCH-2K is a non-commercial research benchmark, and its accompanying evaluation code, simulator, and RL scripts support reproducible research rather than clinical deployment.The dataset permits academic use, redistribution, and adaptation with attribution, while commercial use requires additional permission; the code and scripts are released under Apache License 2.0.Source: §D.4.2 · p. 27

GPAGENTBENCH-2K: Benchmark Dataset and CMDP Environment

• The benchmark is not a standalone clinical decision-support system and must not guide real-world diagnosis, treatment, triage, referral, or other patient-facing care.High-stakes healthcare use requires clinical validation, institutional review, regulatory assessment, and expert oversight; results should be treated as research findings rather than evidence of readiness, given possible source, selection, and annotation biases.Source: GPAGENTBENCH-2K: Benchmark Dataset and… · p. 28

Evaluation Code, Simulator, and RL Scripts

• The codebase supports agent-environment interaction, structured-action parsing, metric computation, and safety and efficiency evaluation, but users must not use it with the restricted dataset to deploy clinical products.Users should document model settings, prompts, protocols, and environment changes, and distinguish benchmark performance from real-world clinical safety.Source: Evaluation Code, Simulator, and RL… · p. 28

D.5 Data Collection and Anonymization Procedures; D.6 Artifact Documentation

• The records target primary-care diagnosis, management, and referral, use English clinical shorthand and structured XML-style agent actions, and include age and gender where available but not ethnicity.Direct identifiers and high-risk quasi-identifiers were removed or generalized, dates and locations replaced with synthetic placeholders, and exact birth dates and specific demographic intersections generalized or redacted.Source: §D.6 · p. 28, §D.5 · p. 28

D.7.2 Study Procedure; D.7.3 Evaluation Interface; D.7.5 Instructions for Clinical Validation

• Physicians evaluated cases online after standardized briefing and two practice cases, using a three-column interface and case-specific criteria for diagnosis-and-management or abstain-and-refer cases.They graded individual criteria Pass or Fail, then retained cases only when all criteria were fundamentally correct; discards required a brief explanation.Source: §D.7.2 · p. 29, §D.7.5 · p. 29, §D.7.3 · p. 29

D.7 Human Evaluation Details; D.7.1 Physician Demographics; D.7.2 Study Procedure

• Only cases unanimously passed on every criterion and retained by both senior physicians entered the dataset, following ethical approval and written volunteer consent.The study recruited two board-certified senior physicians with 35 and 41 years of experience, who participated voluntarily and without compensation.Source: §D.7 · p. 28, §D.7.2 · p. 29, §D.7.1 · p. 29

D.7.4 Disclaimer for Annotators; D.7.6 Data Consent

• Annotators reviewed anonymized records, could skip uncomfortable cases or withdraw without penalty, and were told their responses would be confidential and used only for academic research.They could request that their data not be used after withdrawal to the extent permitted; direct identifiers would be removed and results reported in aggregate where possible.Source: §D.7.4 · p. 29, §D.7.6 · pp. 29–30

D.8 Use of AI Assistants in Research

• The researchers used ChatGPT sparingly for basic polishing and grammar checks, as a supplement to rather than a replacement for human expertise.This use was described as aligned with ARR’s Policy on AI Writing Assistance.Source: §D.8 · p. 30