CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?

Motivation

• Prior authorization and care management are inefficient, long-horizon operations involving queues, extensive outreach, and coordination across roles, making each handoff a potential point of failure.These workflows represent daily managed-care operations rather than edge cases, and AI agents are increasingly proposed to assist or partially automate them.Source: §1 · p. 2

Challenges

• Healthcare workflows combine policy density, multi-role composition with irreversible handoffs, and multilateral multi-turn interactions requiring agents to gather information and carry results back into workflow.Policies vary across providers, payers, and time; roles include clinicians, coordinators, UM nurses, medical directors, and RN care managers; interactions include peer-to-peer review and patient outreach.Source: §1 · p. 2, Figure 2 · p. 2

Benchmark

• χ-Bench evaluates provider prior authorization, payer utilization management, and RN care management in a high-fidelity simulator spanning 20 healthcare apps exposed via MCP.Agents drive cases to terminal status through tool calls and role artifacts, guided by a 1,279-markdown managed-care operations handbook and scored by deterministic checks plus rubric-based LLM judgment.Source: §1 · p. 3

Results

• 28.0% of tasks are resolved at pass@1 by the best configuration, while no agent exceeds 20% on pass^3, marathon performance falls to 3.8%, and provider–payer prior authorization reaches 0%.These results indicate that long-horizon capabilities shown on coding-style benchmarks do not generalize well to realistic healthcare workflows.Source: §1 · p. 3, Figure 3 · p. 3

2 Related Work

• Prior healthcare benchmarks target factual medical knowledge, broad clinical proficiency, EHR querying, short-horizon clinical agents, or narrower administrative interactions.General-purpose long-horizon benchmarks cover GUI control, long-horizon code, and tool use, while τ/τ 2-Bench [59] and TheAgentCompany [57] model simulated stakeholders but not healthcare’s policy-dense, information-asymmetric workflows.Source: §2 · pp. 3–4
• χ-Bench is the first healthcare benchmark to combine long-horizon tool calls, dense policy retrieval, irreversible workflow state, hidden multilateral interaction, and in-situ simulator verification.HealthAdminBench [8] instead focuses on GUI interaction over a payer portal, whereas χ-Bench uses structured MCP tools, an explicit policy handbook skill, and adds care management with patient outreach.Source: §2 · p. 4

3.1 χ-World Engine: Simulated Worlds for Clinical Healthcare In-Situ Workflows

• χ-Bench simulates provider prior authorization, payer utilization management, and RN care management across 20 healthcare applications accessed through 87 MCP tools.The workflows require policy-grounded actions such as evidence gathering and appeals, utilization review and determinations, or patient outreach and care-plan authoring.Source: §2 · p. 4, §3.1 · p. 4

3.1.1 Realistic Healthcare Software Environments

• The simulator models 29-status case state machines, reviewer independence, channel-specific submissions, document authorship and signing, FHIR-grade linkage, and atomic cross-application effects.It contains approximately 5,000 chart activities for 50 simulated patients and approximately 90 healthcare workers, with agents operating through MCP servers, a local database, and the file system.Source: §3.1 · p. 4, §3.1.1 · p. 5

3.1.2 Encoding Healthcare Workflows with Managed-Care Operations Handbook Skill

• The Managed-Care Operations Handbook contains 1,279 markdown documents organized into role-specific workflows, medical-policy libraries, and MCP platform tutorials, developed with clinicians and operations leaders at Johns Hopkins Medicine.The handbook is presented as a progressive-disclosure onboarding manual for PA specialists, UM reviewers, and care managers, and is described as the first evaluation of agent skills at real healthcare operational-library scale.Source: §3.1.2 · pp. 5–6

3.2.2 Task Construction and Composition

• The task pipeline produced 523 candidates and retained 75 representative long-horizon tasks, requiring humans an average of 21 steps and at most 40 steps to complete.Candidates were retained only when expected actions resolved to cited policies and the chart and rubric mutually entailed without leaking the chosen path; trajectories underwent healthcare-worker and author review plus PHI and realism checks.Source: §3.2.2 · p. 7

3.2.3 Reward

• Each trial receives a binary reward only when both deterministic contract verification and rubric-based LLM judging pass against the simulator’s persisted world store, event log, and transcripts.The verifier also produces a fractional scorecard for diagnostics.Source: §3.2.3 · p. 7, Figure 10 · p. 7

4.1 Experiment Setup

• The evaluation covers 30 agent harness/model configurations across proprietary and open-source stacks, using 3 independent trials per task and reporting pass@1, pass@3, and pass^3.The configurations span first-party CLIs with closed-weight models, four open-source frameworks with five open-weight models, and an additional OpenClaw + Claude Opus 4.7 reference cell.Source: §4.1 · p. 7

4.2 χ-Bench Results

• 28.0% Overall pass@1 is highest for Claude Code + Claude Opus 4.6, while pass^3 falls to 18.7%, revealing strong run-to-run unreliability.Other leading Overall pass@1 results are Sonnet 4.6 at 26.2%, Opus 4.7 at 24.4%, and Codex + GPT-5.5 at 20.9%.Source: §4.2 · p. 8, Figure 11 · p. 8

4.3 χ-Bench-Arena: Can Prior Authorization Workflows be Automated End-to-End?

• 0% pass@1 results when payer participation and cross-role checks join the provider agent, versus 30.4% for the provider-only baseline.Across 23 prior-authorization tasks, 2 were not submitted, 18 did not finish the medical-director decision, and 5 failed the final judge.Source: §4.3 · p. 9, Table 3 · p. 9

4.4 χ-Bench-Marathon: Can Long-Running Agents Stay on Track Across All 25 Tasks?

• 3-8 of 25 cases reach finalized determinations or care plans in marathon sessions, while neither evaluated agent submits any prior authorization.Both agents fan out across queued cases, save partial work, and fail to drive most cases to terminal actions; Codex auto-compacts 4-6 times per PA session and 1-2 times on UM, whereas Claude Code never compacts.Source: §4.4 · p. 9, Table 4 · p. 9

4.5 Effects of Handbook Skills Components

• 32.0→17.3 pass@1 after removing the domain handbook in UM, whereas PA modestly improves with −Both and CM remains near its floor.The handbook can support policy-heavy reviews but also induce exhaustive verification, refusal, or cognitive overload; CM complexity is conversation driving rather than policy.Source: §4.5 · pp. 9–10, Figure 12 · p. 9

4.6 MCP vs. CLI for Healthcare Agent Workflows

• MCPorter-style CLI tool resurfacing causes a small PA regression, a clear UM drop, and a small CM gain, making it neutral-to-worse overall.The probe reran Codex + GPT-5.5 across 75 tasks with 3 trials per task.Source: §4.6 · p. 10, Table 5 · p. 10

4.7 Failure Mode Analysis

• 35.4% of failed trials are Clinical-Reasoning errors and 23.3% are Workflow-Completion failures, followed by Abstain-or-Stuck at 15.6% and Policy-Compliance at 13.2%.Harness-Fault accounts for 1.0%, Tool-Use-Error 10.7%, and Hallucination 0.8% of the 5,886 failed trials.Source: §4.7 · p. 10, Figure 13 · p. 10
• 18.7% of failed trials skip required steps, 13.2% misread policy criteria, and 337 failures (5.7%) involve illegitimate consent in care management.Criteria misapplication means applying correct rules or evidence incorrectly, whereas policy criteria misreading means misunderstanding the rule text itself.Source: §4.7 · pp. 10–11, Figure 14 · p. 11

5 Conclusion

• 28.0% of tasks were resolved by the strongest agent at pass@1, while no agent exceeded 20% at passˆ3.The benchmark covers prior authorization, utilization management, and care management using a 1,279-document managed-care operations handbook.Source: §5 · p. 11
• Clinical-Reasoning failures accounted for 35.4%, Workflow-Completion for 23.3%, and Policy-Compliance for 13.2%, while illegitimate consent showed completion can violate autonomy.Examples include criteria misapplication, skipped required steps, policy criteria misreading, and advancing care-management workflows without valid consent.Source: §5 · p. 11
• χ-Bench evaluates language-only agents, omits multimodal reasoning and many long-tail workflows, and uses Opus 4.7 as its only judge model.The authors identify multimodal and workflow expansion, alongside studying alternative judge models, as immediate next steps.Source: §5 · p. 11
• 28% pass@1 on a static benchmark may be risky for live patient care because failures can create clinical, financial, and regulatory harm.The benchmark is released to encourage caution before deploying agents on irreversible workflows affecting patients.Source: §5 · p. 11

A Ethical Statement

• χ-Bench contains no real patient data or PHI, uses fictional clinician-authored composites, required no IRB review, and can be redistributed without privacy-preserving transformations.The study is explicitly characterized as not being human-subjects research.Source: A Ethical Statement · p. 17
• Claude Opus 4.7 supported rubric judging with V =3 independent votes and strict-majority aggregation, post-hoc failure analysis, and patient and peer simulators.Prompts for these three uses are reproduced in Appendix H.Source: A Ethical Statement · p. 17

B Extended Related Work

• χ-Bench compares 29 healthcare and long-horizon agent benchmarks across nine axes, emphasizing multilateral interaction, multi-role composition, and policy density.The appendix justifies the non-trivial comparison-cell assignments and defines the axes used in Table 1.Source: B Extended Related Work · p. 17

B.1 Axis Definitions

• The axis definitions distinguish conventional capabilities from three discriminators: required live dialogue, irreversible switching among at least two role tool surfaces, and policy handbooks exceeding approximately 10K tokens.Conventional axes include healthcare focus, API tools, long horizons, hidden state, in-situ verification, and LLM judging; each discriminator also specifies partial and absent cases.Source: §B.1 · pp. 17–18

• Static QA, Text2SQL, coding, staged doctor, and FHIR benchmarks generally omit combinations of policy navigation, dialogue, role switching, hidden information, and workflow-state evaluation.These benchmarks respectively rely on closed prompts, single queries, single coder roles, templated patient exchanges, or programmatically exposed resources and code state.Source: §B.2 · pp. 18–19
• MedHELM [7] spans 35 healthcare benchmarks, yet evaluates single prompt-to-response tasks without tools, long horizons, policy documents, role switching, dialogue partners, hidden information, or persisted state.It does use an LLM jury for open-ended benchmarks, with agreement to clinician ratings.Source: §B.2 · p. 18
• HealthBench offers multi-turn patient–clinician dialogue, but remains natural-language Q&A without tools, policy libraries, role switching, hidden-world elicitation, or persisted simulator state.Its scoring is rubric-based on the transcript alone.Source: §B.2 · p. 19
• AgentClinic supports multi-party dialogue and hidden-information elicitation, but the agent remains a doctor, follows no formal policy document, uses limited test actions, and is graded mainly on diagnosis.The other parties are fixed simulators rather than alternative role surfaces the agent must inhabit.Source: §B.2 · p. 19
• HealthAdminBench [8] combines multi-role GUI traversal, long-horizon tasks, persisted-state checks, and LLM judging, but lacks live dialogue, general tool calling, and a structured policy library.Its policy conditions are embedded in portal flows and per-task instructions, while payer-side information is surfaced through navigation rather than deliberately withheld in dialogue.Source: §B.2 · p. 19

B.3 Long-Horizon Agent Benchmarks

• Unlike prior suites, χ-Bench combines healthcare scope, policy density, multi-role composition, multilateral interaction, hidden state, and long-horizon execution in one benchmark.Existing benchmarks variously lack policy handbooks, role switching, simulated counterparties, typed APIs, hidden state, or rubric-based evaluation.Source: §B.3 · pp. 19–20
• χ-Bench satisfies all nine benchmark axes through 87 MCP tools, 20 healthcare apps, multi-role handoffs, hidden information, multi-turn dialogues, and policy-grounded decisions.Cases move across clinician, utilization-management nurse, medical director, and care-manager roles, with deterministic gates and multi-rubric LLM judging.Source: §B.3 · p. 20

C.1 Healthcare Workflows

• χ-World interlocks provider prior authorization, payer utilization management, and care management through shared cases, documents, and messages.Figures 15–17 represent the three workflows as phased state machines over the MCP tools.Source: §C.1 · p. 20

C.1.1 Provider Prior Authorization

• Eight provider prior-authorization phases gather evidence, submit cases, handle payer responses, conduct peer-to-peer review, and pursue internal or external appeals.The case ends when an authorization number is issued or every appeal route is exhausted.Source: §C.1.1 · p. 21, Figure 15 · p. 22

C.1.2 Payer Utilization Management

• Six payer utilization-management stages normalize intake, route cases by urgency and coverage rules, review clinical criteria, determine outcomes, and deliver compliant letters.Denials can proceed through independent review and, if upheld, an external IRO decision.Source: §C.1.2 · p. 21, Figure 16 · p. 23

C.1.3 Care Management

• Five care-management phases progress from referral intake and chart review through patient outreach, four-domain assessment, and a hierarchical care plan.Cases may be disenrolled at any phase for refusal, unreachability, relocation, safety concerns, or duplication.Source: §C.1.3 · p. 21, Figure 17 · p. 24

C.2 Tool Manifest

• The simulator exposes 87 MCP tools across Provider PA, Payer UM, and Care Management servers containing 30, 41, and 16 tools, respectively.The tools cover case management, documents, peer-to-peer dialogues, letters, assessments, and care plans.Source: §C.2 · p. 21, Table 6 · p. 25, Table 7 · p. 26

C.3 Handbook Detail

• The managed-care handbook is a structured skill library organized into provider-pa, payer-um, care-manager, platform, and medical-library sections.It supplies workflow standards, tool tutorials, clinical guidelines, medical policies, pharmacy requirements, and utilization-review criteria.Source: §C.3 · p. 28

D.1 Task Construction Details

• Provider PA difficulty counts required document kinds, payer UM difficulty counts policy-path criteria, and care-management difficulty maps directly to consent stance.PA difficulty is Easy at 0, Moderate at 1–3, and Hard at ≥4 document kinds; UM is Easy at ≤5, Moderate at 6–9, and Hard at ≥10 criteria; care management maps Engaged, Hesitant, and Refusing to Easy, Moderate, and Hard.Source: §D.1 · pp. 28–29, §D.2 · p. 31, §D.3 · p. 35
• Tasks are generated from policy or NNN state graphs and handbook sections, then pass citation, entailment, dry-run, clinical-realism, PHI, and expert-review gates.Rejected candidates receive diagnostic retries for citation or entailment failures, while invalid dry runs are discarded; reviewers include credentialed clinicians, simulator operators, and paper authors.Source: §D.1 · pp. 29–30

D.2 Prior Authorization and Utilization Management

• Provider PA tasks distinguish complete, insufficient, and absent documentation, producing Submitted to payer (8), Returned for docs (12), or Gather more evidence (5).These terminal states test packet completeness, document retrieval and resubmission, and upstream coordination with the ordering clinician.Source: §D.2 · pp. 31–32
• Payer UM tasks encode each policy criterion as text, citation, and chart evidence, requiring named policy references across staged reviews and rollup rules.Tasks span five clinical categories, two to five review stages, 0–50 policy criteria with mean 9.2, and four terminal outcomes: Approved (9), Denied (10), Pended (5), and Coverage not required (1).Source: §D.2 · pp. 31–32
• Four UM tasks use peer-to-peer calls to test sustained adversarial reasoning, requiring agents to defend or revise determinations as attending physicians present new evidence.All four P2P tasks resolve to Approved (2) or Denied (2), with Pended structurally excluded because the call must resolve the case.Source: §D.2 · p. 32

D.3 Care Management

• A two-layer verifier checks terminal state, workflow progression, structured artifacts, evidence grounding, policy or NNN alignment, completeness, and mutation scope across all three domains.CM additionally requires forward-only five-phase progression, consent before assessment, complete instruments, and evidence-supported NNN plans; both deterministic and LLM-judge layers must pass.Source: §D.2 · p. 32, §D.3 · p. 35
• Care management follows five RN phases from intake through closure, with outreach adapting to opaque consent and disclosure states rather than scripted questioning.Disenrollment can occur at any phase, and sensitive disclosures require explanations, permission to skip or stop, and applicable safety limits.Source: §D.3 · pp. 32–35
• Care plans require at least two NANDA-I diagnoses linked to NOC targets and NIC interventions, while consent thresholds and anti-triggers can block assessment or permanently close disclosure paths.Engaged patients require 1–2 triggers, whereas Refusing patients require all triggers with zero anti-triggers; patient profiles comprise Engaged (3), Hesitant (7), and Refusing (15) tasks.Source: §D.3 · pp. 33–35

E Task Catalog

• 75 tasks span provider prior authorization, payer utilization management, and care management, with PA tasks at referral and UM tasks across five workflow states.PA Provider and UM tasks sharing a service slug represent the same patient case at different workflow stages.Source: E Task Catalog · p. 35
• Difficulty is structurally grounded: PA counts required document kinds, UM counts clinical criteria along the policy path, and CM maps to consent profile.These definitions produce Easy, Moderate, and Hard labels across the task catalog.Source: E Task Catalog · p. 35, Table 16 · p. 37

E.1 PA: Prior Authorization (Provider Side)

• 25 provider-side prior-authorization tasks cover surgical, diagnostic, imaging, pharmacy, and other services, all invoked at the referral state.Difficulty ranges from Easy to Hard, reflecting the number of distinct document kinds required in each submission packet.Source: Table 14 · p. 35, §E.1 · p. 36

E.2 UM: Utilization Management (Payer Side)

• Payer utilization-management tasks cover intake, triage, nurse review, physician review, and peer-to-peer stages across varied clinical services.Examples include cardiac PET denial peer-to-peer, MRI and surgical physician review, and nurse review for infusion, imaging, and procedures.Source: Table 15 · p. 36, §E.2 · p. 37

E.3 CM: Care Management

• Care-management tasks are longitudinal cases spanning metabolic, cardiovascular, behavioral, neurological, and post-acute conditions.Difficulty follows persona consent: Engaged is Easy, Hesitant is Moderate, and Refusing is Hard.Source: Table 16 · p. 37, §E.3 · p. 38

F Experiment Detail

• Each trial uses an isolated containerized sandbox with its own database, file system, and tool-server processes, and is shut down after completion.The shared χ-World is combined with a task manifest to pin the task’s initial state, while trial state is not shared across runs.Source: §F.1 · p. 38

F.1 Sandbox Environment and Trial Orchestrator

• Harbor [33] orchestrates trials by spawning sandboxes, connecting agent and simulator through MCP over streamable HTTP, recording trajectories, and enforcing wall-clock and retry budgets.At trial start, the χ-World backend, role-scoped MCP servers, and agent process launch in parallel.Source: §F.1 · p. 38

F.2 Pass@k and pass^k Metrics

• Three independent trials per task produce task-level pass@k and pass^k metrics, with c counting the number of passes among n = 3 trials.The metrics are computed from the same trial pool.Source: §F.2 · p. 38
• pass^3 is the operational reliability measure because long-horizon healthcare workflows must succeed on every individual case; n = k = 3 requires 6,750 trials per benchmark pass.The budget is 30 cells × 75 tasks × 3 trials.Source: §F.2 · pp. 38–39

F.3 Judge, Container, and Harness Configuration

• Each rubric is judged by Claude Opus 4.7 using three independent votes with strict-majority aggregation, while trials share fixed resources and an 1,800-second agent wall-clock cap.Ties and trials without verifier-resolvable terminal states are counted as failures; containers use 2 CPU cores, 4 GB RAM, and 10 GB ephemeral disk.Source: §F.3 · p. 39

F.4 OpenAI Agents SDK Harness Customization

• The OpenAI Agents SDK harness preserves the upstream Runner.run loop and exposes reproducibility-relevant bounds for turns, retries, and tool-return size.Defaults are max_turns = 50, max_retries = 10, and max_tool_return_chars = 100K, forwarded through environment variables.Source: §F.4 · p. 39
• The SDK wrapper adds role-scoped MCP integration, oversized-output spillover, transient-error retries, trajectory conversion, and compatibility with the first-party CLIs’ tool and prompt interfaces.It sanitizes tool names, stores oversized outputs in overlong_tool_outputs/, uses SDK 0.13.6 retry settings, and preserves stop-on-tool-error behavior and the verbatim Toolathlon template.Source: §F.4 · p. 40

F.5 Infrastructure and Harness Versions

• Table 17 pins the harness and runtime versions used for all 6,750 trials, with exact package resolutions and container SHAs recorded in the reproducibility bundle.Open-source harness versions correspond to upstream releases installed by each Harbor bootstrap script at the experiment date.Source: Table 17 · p. 41, §F.5 · p. 41

By difficulty band

• PA Hard pass^3 is roughly an order of magnitude below PA Easy, while CM Refusing tasks collapse to near-zero pass^3 across every cell.The comparison is reported in Table 18, which aggregates results across all 30 harness–model rows.Source: Table 18 · p. 41, §F.6 · p. 41

By workflow coverage

• Gather more evidence is the hardest PA terminal-status bucket because it requires coordination with the ordering clinician, while UM Peer-to-peer pickup is hardest because it requires adversarial multi-turn dialogue.These workflow-coverage results are reported in Tables 19 and 20.Source: Table 19 · p. 41, §F.6 · p. 41, Table 20 · p. 42

By clinical category (Payer UM)

• Surgical procedures comprise 12 of 25 UM tasks and impose the longest policy decision chains, whereas specialty pharmacy tests a distinct combination of clinical-condition and drug-specific criteria.Table 21 groups payer UM tasks by clinical category.Source: Table 21 · p. 42, §F.6 · p. 42

By patient engagement profile (Care Management)

• Refusing personas comprise 15 of 25 CM tasks and drive the dominant failure mode, as agents over-extract reluctant ambivalence as same-call consent.This pattern is linked to consent fabrication and summarized by persona consent stance in Table 22.Source: Table 22 · p. 42, §F.6 · p. 42

F.7 Judge Inter-rater Reliability

• V = 3 independent judge votes are aggregated by strict majority, with ties resolved as failures for every binary rubric.Cohen’s κ is averaged across vote pairs within rubrics and then across rubrics within each domain.Source: §F.7 · p. 42
• 4,780 of 6,750 canonical task-runs produced parseable judge verdicts, while 1,970 terminated before scoring and were excluded from reliability statistics.Source: §F.7 · p. 42
• 65.6% of trials with both verdict layers agreed overall, and judges concurred with deterministic failures in 90.3% of cases.DeterministicPass ∧JudgePass prevents the judge from awarding a positive verdict alone; failure-direction concurrence was 98.2% for CM, 89.6% for UM, and 76.7% for PA.Source: §F.7 · p. 42
• Table 23 reports per-domain mean Cohen’s κ and the rate at which strict-majority verdicts differ from at least one individual vote.Source: Table 23 · p. 43
• Figure 21 shows per-rubric Cohen’s κ distributions across 199 binary rubrics, with κ = 0.6 marking substantial agreement.Source: Figure 21 · p. 43
• The three judgment, completion, and policy axes combined accounted for 71.9% of the agent-attributable bucket, indicating observed failures reflected agent execution and policy interpretation rather than verifier brittleness.Source: §F.7 · p. 43

F.8 Per-domain Trial Cost and Wall-clock Statistics

• Table 24 reports per-(harness, model) median and 95th-percentile wall-clock and token usage to complement Table 2’s means.Figure 22 visualizes per-trial distributions as horizontal violins pooled by harness; Hermes costs and tokens are aggregated from the OpenRouter dashboard when harness usage is unavailable.Source: §F.8 · p. 43

F.9 Task Level Results

• Rows in the domain heatmaps are (harness, model) configurations sorted by overall pass@1, while columns are tasks sorted by per-task pass@1.This ordering supports comparison across configurations and tasks within each domain.Source: §F.9 · p. 43
• Red x markers identify tasks that no configuration passed on any of three trials; these unresolved tasks are listed in Table 26.The markers appear above the corresponding task columns.Source: §F.9 · p. 43

G Failure-Mode Analysis

• The supplied material for G Failure-Mode Analysis contains no substantive passage describing failure modes.No failure-mode finding can be reported from the provided evidence.Source: §F.9 · p. 43

F.9 Task Level Results

• Table 24 reports median and 95th-percentile wall-clock, token, and cost for each harness–model configuration.The table summarizes resource use at the per-configuration level.Source: Table 24 · p. 44
• Figure 22 shows per-trial wall-clock and total-token distributions by domain, with colors denoting harnesses and a 1800 s wall-clock cap.The distributions are faceted by domain and pooled by harness.Source: Figure 22 · p. 44
• Per-task pass@1 heatmaps cover PA, UM, and CM, with each cell showing pass@1 and each row annotated by mean pass@1.Figures 23–25 provide one heatmap for each domain.Source: Figure 23 · p. 45, Figure 24 · p. 45, Figure 25 · p. 46

G.1 Analysis Methodology

• The labeling pipeline analyzes all 5,886 failed trials with three ordered tiers, assigning each trial a first-level category and second-level mode without later re-entry.Tier 1 uses runtime logs and verifier outcomes; Tier 2 classifies analyzer evidence by substring rules; Tier 3 verifies candidate hallucinations end-to-end.Source: §G.1 · p. 47
• Tier 1 deterministically identifies crashes, zero-step outputs, timeouts, fatal tool-call cascades, rubric successes, and ambiguous non-finalized workflows from logs and verifier outcomes.Fatal DeepAgents ToolException cases are classified as Tool-Use-Error because malformed agent calls, rather than infrastructure, trigger escalation.Source: §G.1 · p. 47
• Tier 2 routes analyzer-written summaries by evidence phrases, separating criterion misreading, illegitimate consent, and ignored-condition or action–narrative mismatch cases.Claude Opus 4.7 generates one JSON summary per trial from the full extract, rubric outcome, and six taxonomy examples; substring rules consume its supporting-evidence string.Source: §G.1 · p. 47

G.2 Taxonomy Definitions

• The taxonomy assigns one of eight first-level categories and a second-level mode, with definitions covering harness, tool-use, policy, clinical, workflow, hallucination, and abstention failures.Canonical kebab-case identifiers support cross-reference with per-trial labels, while humanized italics are used in text.Source: §G.1 · p. 47, §G.2 · pp. 47–48
• Hallucination requires an unsupported or input-contradicting assertion, whereas Clinical-Reasoning references relevant facts but reaches an incorrect medical or protocol judgment.Policy-Compliance concerns policy or citation-surface errors; criterion misreading misreads rule text, while criteria misapplication applies correct evidence incorrectly.Source: §G.2 · pp. 47–48
• CM hard-refuse success requires autonomy-first engagement, explicit acknowledgment of refusal, and affirmative enrollment; concern-mining with explicit yes is illegitimate consent, otherwise structured consent is fabricated.Disenrollment, abandonment, and partial engagement do not count as success under interpretation B.Source: §G.2 · pp. 48–49

G.3 Failure-Mode Statistics

• 35.4% of failures are Clinical-Reasoning, 23.3% Workflow-Completion, 15.6% Abstain-or-Stuck, 13.2% Policy-Compliance, 10.7% Tool-Use-Error, 1.0% Harness-Fault, and 0.8% Hallucination across 5,886 failures.Clinical-Reasoning reaches 50.0% of CM failures, while Tool-Use-Error reaches 14.8% of UM failures; analysis success counts match verifier outcomes across all 90 row-domain cells.Source: §G.3 · p. 49, Table 25 · p. 49

Workflow-Completion

• 23.3% of failures were Workflow-Completion errors, dominated by 1,098 skipped required steps (79.9%) in which agents narrated completion without executing terminal MCP actions.These “completion theatre” failures included leaving care plans in draft, omitting authorization submission, and failing determination finalization or letter validation.Source: §G.4 · p. 53

Clinical-Reasoning

• 35.4% of failures were Clinical-Reasoning errors, chiefly misapplied clinical criteria (1,647; 79.1%), while 25% of hard-refuse trials produced illegitimate consent through repeated reframing.The category included 2,083 trials and also covered ignored clinical conditions, illegitimate consent, action-reasoning mismatch, and wrong-protocol pathways.Source: §G.4 · p. 54

Hallucination

• 0.8% of failures were Hallucination cases, including fabricated consent, unsupported clinical facts, and claimed tool results that were never persisted or returned.The analysis distinguishes fabricated consent without transcript support from illegitimate consent, where repeated reframing eventually produced an explicit affirmative.Source: §G.4 · p. 55

Abstain-or-Stuck

• 15.6% of failures were Abstain-or-Stuck cases, including 449 wall-clock timeouts, 206 abandoned trials, 147 refusals to act, and 119 agent loops.Some agents stopped after parameter errors or requested user confirmation, while others continued looping after substantively completing the workflow.Source: §G.4 · pp. 56–57

Successes

• 12.8% of trials succeeded through direct-pathway or recovery patterns, with strong rows finalizing through documented MCP calls and recovering by abandoning incorrect narratives.Direct successes used at most one backtrack, while recovery successes recognized empty or non-authoritative responses and returned to sanctioned MCP execution.Source: §G.4 · p. 57

Case 19

• Case 19 succeeded on the first pass (reward=1) after P2P concessions established failed alternatives and weak evidence, supporting the finalized adverse decision.The agent identified the policy classifying virtual upper GI endoscopy as unproven, recorded the intended adverse decision, finalized denial, and dispatched compliant correspondence with appeal rights.Source: §G.4 · p. 57

Case 20

• Case 20 recovered a hard refusal into verbal enrollment consent through autonomy-first, non-judgmental outreach that surfaced medication-adherence barriers and enabled a finalized complex-care plan.The agent respected limited factual clarification boundaries, disclosed program limitations, and sustained grounded execution across intake, review, outreach, assessment, and planning.Source: §G.4 · p. 57

G.5 Unresolved Tasks

• 17 of 75 tasks scored pass3=0 across every harness–model row, and all 17 also scored pass@1=0 across the full 30 × 3 matrix.Table 26 identifies this set through the success-set complement in Section G.1.Source: §G.5 · p. 57, Table 26 · p. 58
• Nine provider prior-authorization tasks were new-referral cases requiring a specific submission bundle that no row produced.Source: §G.5 · p. 58
• The remaining unresolved tasks comprised six payer utilization-management intake, triage, nurse, or physician-review cases and two care-management hard-refusal cases.Source: §G.5 · p. 58

G.6 Skill Invocation: Did the Agent Actually Read the Policy?

• 29.3% mean policy-read recall across 6,750 canonical trials shows agents accessed fewer than one-third of ground-truth-required handbook policies.Recall measures the fraction of GT-cited policies accessed through Read, Grep, or Bash cat calls.Source: §G.6 · p. 58
• Policy-Compliance failures had 18.5% recall, while Clinical-Reasoning and Hallucination failures had 44.0% and 66.4%, respectively, separating policy neglect from misapplication.The failure-mode pattern indicates agents often skipped policies in compliance failures but read them before applying them incorrectly or contradicting them in other failures.Source: §G.6 · p. 58
• r = +0.77 rank correlation between cell-mean policy-read recall and pass@1 across 30 harness-model cells indicates handbook access is necessary but insufficient.Within-domain Pass-vs-Fail recall gaps were CM 82.3% vs 62.9%, UM 26.3% vs 21.0%, and PA-Provider 2.5% vs 1.5%.Source: §G.6 · p. 58, Figure 29 · p. 59

H Prompts

• The appendix documents every LLM-use point and provides fully rendered examples for the verifier judge and failure-mode analyzer.Long templates are split at natural boundaries, with inline elisions in shortened examples.Source: H Prompts · p. 59

H.1 Verifier Rubric Judge

• The verifier judge uses three Claude Opus 4.7 votes per rubric with strict-majority passage, dispatching to PA, UM, or CM role-specific analyzers.Shared scaffolding covers workspace contents, verdict discipline, citation rules, and output schema, while role sections assess chart-submission consistency, policy-determination alignment, or clinical documentation quality.Source: §H.1 · p. 59
• Verifier grading requires rubric-first investigation, artifact-based file-and-line citations, explicit accounting of named items, literal quotations, and incremental verdict persistence.The judge must not add deterministic checks beyond the rubric and must fail unsupported fabricated or omitted evidence.Source: §H.1 · pp. 59–60
• CM outreach verdicts hard-fail listed anti-triggers, unrepaired privacy violations, and treating non-enrollment statements such as “Send me information” as consent.Unsupported promises are anti-triggers when forbidden, while privacy repair requires stopping, acknowledging the issue, explaining source and scope, allowing refusal, and requesting permission.Source: §H.1 · p. 61

H.2 Failure-Mode Analyzer

• The Stage 2 failure-mode analyzer labels approximately 225 trial extracts per row with outcome, hierarchical mode, evidence, confidence, and a row summary across seven failure and three success categories.Low-confidence or OTHER labels are routed to cleanup, while the analyzer’s taxonomy and six worked examples support consistent category assignment.Source: §H.2 · p. 61

H.3 Analyzer Cleanup Pass (Stage 2.6)

• The Stage 2.6 cleanup pass reanalyzes low-confidence or OTHER trials using head-and-tail evidence plus pattern, location, context, and ground-truth cross-check steps.It retires OTHER, defaults to medium confidence, and reserves high confidence for decisive judge breakdowns, exceptions, or tool calls.Source: §H.3 · pp. 63–64

H.4 Patient Persona (CM Outreach Simulator)

• The CM outreach simulator combines global communication rules, a seeded patient persona contract, and stage-conditioned scenario information, optionally adding one of seven voice styles.The simulated patient discloses information progressively, avoids hallucination, and responds at scenario-defined health-literacy and emotional levels.Source: §H.4 · p. 64
• Patient personas continue outreach until objectives are met or the character ends the call, then emit the ###STOP### token.Persona contracts specify identity, consent boundaries, anti-triggers, and end-of-call conditions while requiring realistic, character-consistent conversation.Source: §H.4 · p. 64
• The patient persona receives chart-grounded medical history, reason-for-call facts, known and unknown information, and behavioral instructions as scenario inputs.Unknown facts are explicitly distinguished from facts the patient knows about their own care.Source: §H.4 · p. 65

H.5 Peer-to-Peer Counterpart Simulator

• P2P sessions use an LLM-played clinical counterpart whose role, goals, knowledge, forbidden arguments, and style are generated from the benchmark side.The simulator defaults to Claude Opus 4.7 and withholds position, required_resolution_points, and primary_issue because they are verifier ground truth.Source: §H.5 · p. 65
• P2P counterpart responses stay in character and use first-person, 2–4-sentence turns resembling a real phone call.The provider-side template frames the interaction as addressing a payer reviewer’s denial concern using the counterpart’s complete clinical knowledge.Source: §H.5 · p. 65
• P2P counterparts may cite only chart documents and revealed facts supplied in “What You Know,” and must acknowledge unsupported medical necessity claims honestly.They cannot introduce clinical specifics absent from that knowledge block and must defer questions requiring information outside it.Source: §H.5 · p. 65

H.6 Per-Task Agent Instruction (instruction.md)

• Per-task instruction.md renders a deterministic scenario, workspace map, and rules block conditioned on the case’s initial workflow state.Claude Sonnet 4.6 generates stage-keyed narratives from seeded clinical facts, with per-stage fallback templates if generation fails.Source: §H.6 · p. 66
• Agent instructions expose role-specific workspaces, MCP resources, handbook locations, incoming documents, shared handoffs, and tool references.Provider and payer templates use different private/shared file paths and namespace sets, while CM reuses the provider skeleton with CM-specific tools.Source: §H.6 · p. 66
• Rules prohibit simulator-source inspection and fabricated evidence while requiring agent-authored summaries, correspondence, and rationale-rich clinical-review notes.The payer template restricts tools to payer namespaces and instructs agents to create only workflow-required artifacts.Source: §H.6 · p. 66