PaperlayerView exact revision on arXiv

Source-linked AI summary

CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?

Haolin Chen, Deon Metelski, Leon Qi, Tao Xia, Joonyul Lee, Steve Brown, Kevin Riley, Frank Wang, T. Y. Alvin Liu, Hank Capps MD, Zeyu Tang, Xiangchen Song, Lingjing Kong, Fan Feng, Tianyi Zeng, Zhiwei Liu, Zixian Ma, Hang Jiang, Fangli Geng, Yuan Yuan, Chenyu You, Qingsong Wen, Hua Wei, Yanjie Fu, Yue Zhao, Carl Yang, Biwei Huang, Kun Zhang, Caiming Xiong, Sanmi Koyejo, Eric P. Xing, Philip S. Yu, Weiran Yao

arXiv:2605.16679v2cs.CLcs.AI

TL;DR

Realistic healthcare workflows remain difficult to automate because they combine dense policies, multiple roles and handoffs, and multi-turn interactions. χ-Bench evaluates agents on these workflows and finds that the best configuration resolves only 28.0% of tasks, with performance collapsing further under stricter reliability and marathon conditions.

  • Problem

    Current benchmarks underrepresent healthcare workflows requiring policy-grounded decisions, multi-role handoffs, and multi-turn interactions across stakeholders.

  • Method

    χ-Bench evaluates agents on long-horizon provider authorization, payer utilization management, and care management workflows in a high-fidelity simulated healthcare environment.

  • Results

    28.0% of tasks are resolved by the best configuration at pass@1, while no agent exceeds 20% at pass^3 and marathon performance falls to 3.8%.

  • Takeaways & Limitations

    The results suggest that long-horizon capabilities demonstrated on coding benchmarks do not generalize well to realistic healthcare workflows.

  • Takeaways & Limitations

    χ-Bench evaluates language-only agents, covers only selected healthcare workflows, and uses a single judge model whose alternatives remain unstudied.

Abstract

from arXiv · show

End-to-end automation of realistic healthcare operations stresses three capabilities underrepresented in current benchmarks: policy density, decisions must be grounded in a large library of medical, insurance, and operational rules; Multi-role composition: a single task requires the agent to play multiple roles with handoffs; and multilateral interaction: intermediate workflow steps are multi-turn dialogs, such as peer-to-peer review and patient outreach. We introduce $χ$-Bench, a benchmark of long-horizon healthcare workflows across three domains: provider prior authorization, payer utilization management, and care management. Each task hands the agent a clinical case in a high-fidelity simulator of 20 healthcare apps exposed via 87 MCP tools, which it must drive to a terminal status through tool calls and writing the role's artifacts, guided by a 1,290+ document managed-care operations handbook skill. Across 30 agent harness/models configurations, the best agent resolves only 28.0% of tasks, no agent clears 20% on strict pass^3, and executing all tasks in a single session slumps the performance to 3.8%. These results raise the hypothesis that similar gaps are likely to surface in other policy-dense, role-composed, irreversible enterprise domains.

1 Introduction

χ-Bench frames realistic healthcare workflow automation as a test of policy density, multi-role composition, and multilateral interaction, then evaluates agents across provider PA, payer UM, and care management tasks. Results show these workflows remain far from solved, with performance collapsing under stricter reliability, marathon execution, and end-to-end provider–payer conditions.

  • Motivation: Prior authorization and care management are common, inefficient healthcare workflows involving lengthy queues, patient outreach, and coordination across roles.PA requires providers to prepare clinical documents for payer review, while CM is a long-term patient-assisting program.
  • Challenges: Healthcare automation must handle policy density, multi-role composition, and multilateral interaction across long, policy-grounded workflow chains.Agents must navigate changing medical, insurance, and operational rules, perform roles such as clinician and reviewer, and manage interactions including clinician outreach.
  • Benchmark: χ-Bench evaluates provider PA, payer UM review, and RN care management cases in a high-fidelity simulator of 20 healthcare apps exposed via MCP.Agents must drive each case to a terminal status through tool calls and by writing role artifacts such as submission packets, review notes, letters, and care plans.
  • Results: 28.0% of tasks is the best configuration’s pass@1 result, achieved by Claude Code+Claude Opus 4.6 across 30 agent harness/model configurations.The evaluation spans major frontier models and strong agent stacks.
  • Results: 3.8% is performance in the marathon run, while 0% is the best prior auth agents’ result in the end-to-end provider–payer arena.No agent clears 20% under the strict pass^3 reliability metric.

2 Related Work

Prior healthcare benchmarks typically isolate medical knowledge, clinical proficiency, EHR querying, short-horizon agents, or narrow administrative interactions. χ-Bench instead combines long-horizon tool use, dense policy retrieval, irreversible workflow state, hidden multilateral interaction, and in-situ verification in a single healthcare task.

  • Healthcare AI Benchmarks: Prior healthcare benchmarks evaluate factual medical knowledge, broad clinical proficiency, EHR querying, short-horizon agents, or narrower administrative interactions.These categories cover distinct capabilities rather than the complete policy-rich workflow surface targeted by χ-Bench.
  • Healthcare AI Benchmarks: χ-Bench is the first benchmark to combine long-horizon tool calls, dense policy retrieval, irreversible workflow state, hidden multilateral interaction, and in-situ simulator-state verification in one task.The benchmark distinguishes itself by integrating these capabilities rather than evaluating them separately.
  • Healthcare AI Benchmarks: HealthAdminBench focuses on GUI interaction with a payer portal through pixel/DOM browsing, whereas χ-Bench exposes healthcare applications through structured MCP tools.The comparison identifies a difference in interaction modality between the closest healthcare peer and χ-Bench.
  • Long-Horizon Agent Benchmarks: General-purpose benchmarks cover GUI control, long-horizon coding, and broad tool use but rarely model multi-actor workflows.τ/τ 2-Bench and TheAgentCompany pair agents with simulated stakeholders under policy constraints, but neither targets healthcare or prior authorization’s long-horizon, policy-dense information asymmetry.

4 Experiments

Across 30 agent harness/model configurations, χ-Bench exposes low end-to-end reliability, with the best Overall pass@1 reaching 28.0% and performance collapsing under cross-role interaction and single-session marathon execution. Results also show domain-dependent handbook effects and failures dominated by clinical reasoning, workflow completion, and termination problems.

  • Experimental setup: 30 agent harness/model configurations were evaluated across proprietary and open-source stacks spanning multiple agent frameworks and closed- and open-weight models.The evaluation covered two stacks and included first-party CLIs, four open-source frameworks, and five OpenRouter-served open-weight models.
  • Performance, Reliability and ROI: 28.0% Overall pass@1 was achieved by Claude Code paired with Claude Opus 4.6, while domain-leading rows differed across utilization management, care management, and prior authorization.The best domain-level rows were Opus 4.6 for UM at 41.3%, Opus 4.7 for CM at 32.0%, and Codex + GPT-5.5 for PA at 29.3%.
  • Multi-agent arena: 0% pass@1 was recorded in the two-agent PA arena after adding the payer agent and cross-role checks, versus 30.4% for the single-agent baseline.Two tasks were not submitted and 18 did not finish the MD decision.
  • χ-Bench-Marathon: Pass@1 slumped for both evaluated configurations when all 25 domain tasks were queued in one shared χ-World session.χ-Bench-Marathon scored each case individually after a single agent run, with context compaction following the harness default.
  • Skill ablations: 32.0 to 17.3 was the UM pass@1 change after removing the domain handbook, while removing the medical library barely changed UM performance.The handbook effect was domain-dependent: PA modestly favored dropping both libraries, and large skills could induce over-verification, refusal, or cognitive overload.
  • Failure analysis: 35.4% of failed trials were Clinical-Reasoning failures, followed by 23.3% Workflow-Completion and 15.6% Abstain-or-Stuck failures.Nearly half of Abstain-or-Stuck failures exhausted the 1800 s wall-clock cap; dominant second-level modes included criteria misapplication at 18.7% and policy criteria misreading at 13.2%.

5 Conclusion … B.1 Axis Definitions

χ-Bench exposes substantial reliability gaps in long-horizon healthcare workflows, while highlighting limitations, deployment risks, ethical safeguards, and benchmark dimensions that define its scope. The appendix formalizes conventional axes and the three discriminators motivating χ-Bench: multilateral interaction, multi-role composition, and policy density.

  • 5 Conclusion: 28.0% pass@1 was achieved by the strongest agent, while no agent exceeded 20% at passˆ3.The benchmark covers prior authorization, utilization management, and care management using a 1,279-document managed-care operations handbook.
  • 5 Conclusion: χ-Bench evaluates language-only agents and covers only a subset of healthcare’s long-tail workflows.The authors identify multimodal reasoning, broader workflow coverage, and alternative judge models as next steps.
  • 5 Conclusion: 28% pass@1 on a static benchmark may be risky for live patient care because failures can cause clinical, financial, and regulatory harm.The benchmark is released to encourage caution before deployment on irreversible workflows affecting patients.
  • A Ethical Statement: χ-Bench contains no real patient data or Protected Health Information, using fictional composites authored by clinicians and healthcare operational specialists.It was not human-subjects research, required no IRB review, and can be redistributed without privacy-preserving transformations.
  • A Ethical Statement: The study uses Claude Opus 4.7 for rubric judging, post-hoc failure analysis, and patient-persona and peer-to-peer counterpart simulation.Judging uses V =3 independent rubric votes with strict-majority aggregation, and prompts are reproduced in Appendix H.
  • A Ethical Statement: Poor χ-Bench performance signals failure modes including consent fabrication, criteria misreading, premature denial, and misrouted handoffs.The paper links these failures to clinical, financial, and regulatory risks in live deployment.
  • B Extended Related Work: Table 1 compares χ-Bench with 29 healthcare and long-horizon agent benchmarks across nine axes, emphasizing three healthcare-workflow discriminators.These discriminators are multilateral interaction, multi-role composition, and policy density.

B.2 Healthcare Benchmarks

Healthcare benchmarks span static medical question answering, simulated dialogue, coding or FHIR workflows, and GUI-based administration, but typically omit policy libraries or combine only subsets of workflow capabilities. HealthAdminBench most closely approaches long-horizon, multi-role, stateful healthcare operations while retaining only partial policy density.

  • Coverage and limitations: 35 benchmarks in MedHELM cover five clinical task categories, but each evaluates a single prompt-to-response output without iterative tools, long-horizon execution, or supplied policy documents.The categories are decision support, note generation, patient communication, research, and administration.
  • Coverage and limitations: 4,460 expert-level questions in MedXpertQA span 17 specialties and 11 body systems, yet the benchmark is single-turn, tool-free, policy-free, and lacks role variation, simulated interlocutors, hidden information, or persisted state.It includes text and multimodal subsets and uses multiple-choice evaluation.
  • Interactive benchmarks: AgentClinic centers on multi-party dialogue and hidden patient information, but the agent remains a doctor, uses a small fixed action set, and receives no formal policy document.The other parties are fixed simulators rather than alternative role surfaces the agent must inhabit.
  • Closest workflow benchmark: HealthAdminBench combines multi-role GUI traversal, long-horizon execution, persisted-state checks, and LLM judging across an EHR, two payer portals, and a fax system.Its 1,698 evaluation points probe persisted UI/file state, while policy density is only partial because no separate multi-document medical-policy library is supplied.

B.3 Long-Horizon Agent Benchmarks … C.3 Handbook Detail

χ-Bench combines policy-grounded, multi-role, multilateral healthcare workflows with extensive tool and handbook infrastructure. Its three interlocking operations domains span prior authorization, utilization management, and care management, supported by 87 MCP tools and a structured managed-care reference library.

  • B.3 Long-Horizon Agent Benchmarks: Existing benchmarks generally omit combinations of policy libraries, role switching, simulated dialogue, hidden state, typed APIs, and rubric-based LLM evaluation.τ/τ 2-Bench and TheAgentCompany provide multilateral interaction but retain single-agent roles and limited policy-document depth.
  • B.3 Long-Horizon Agent Benchmarks: χ-Bench satisfies all nine benchmark axes through healthcare workflows, 87 MCP tools across 20 apps, multi-stage pipelines, persisted-state gates, and multi-rubric LLM grading.Each case is handed off across clinician, UM nurse, medical director, and care manager roles.
  • C.1 Healthcare Workflows: χ-World interlocks three end-to-end clinical operations workflows through shared cases, documents, and messages: provider prior authorization, payer utilization management, and care management.Figures 15–17 depict these workflows as phased state machines over the MCP tools listed in Section C.2.
  • C.1.1 Provider Prior Authorization: Provider prior authorization shepherds treatment requests through eight phases, including requirement checks, evidence gathering, packet submission, status polling, and branches for approval, pend, denial, peer-to-peer, and appeals.The workflow begins from a doctor’s order and checks whether the insurance plan requires advance permission.
  • C.1.2 Payer Utilization Management: Payer utilization management normalizes requests from portal, API, fax, or EDI channels, structures cases, triages them, computes regulatory deadlines, and routes clinical review toward determination.The workflow includes fast-track approval, peer-to-peer, pending, post-denial appeals, and outbound correspondence.
  • C.1.3 Care Management: Care management uses a five-phase workflow in which case managers prioritize high-risk referrals, conduct outreach and assessment, and finalize problems–goals–interventions care plans.Referrals may follow hospital discharge, emergency-department visits, primary-care referrals, or insurer risk-model flags.
  • C.3 Handbook Detail: The Managed-Care Operations Handbook is a structured skill library organized into provider-pa, payer-um, care-manager, platform, and medical-library sections.It defines a shared typed document taxonomy and was developed with clinicians and operations leaders at Johns Hopkins Medicine; research access is governed by a DUA.

D.1 Task Construction Details · D.2 Prior Authorization and Utilization Management · D.3 Care Management

χ-Bench constructs policy-grounded healthcare workflows with difficulty tied to document requirements, policy-path length, and patient consent, then validates them through rejection sampling, clinician review, and multi-perspective checks. Its provider PA, payer UM, and care-management tasks require document classification, policy-cited decisions, adaptive outreach, and structured terminal artifacts.

  • D.1 Task Construction Details: Provider PA difficulty is defined by required document kinds: Easy = 0, Moderate = 1–3, and Hard = ≥4.Tasks focus on documentation and submission behavior rather than clinical determination.
  • D.1 Task Construction Details: Payer UM difficulty is defined by clinical criteria on the selected policy path: Easy = ≤5, Moderate = 6–9, and Hard = ≥10.UM tasks span two to five review stages and policy paths of 0 to 50 criteria, with mean 9.2.
  • D.1 Task Construction Details: Care-management difficulty maps one-to-one to consent profiles: Engaged = Easy, Hesitant = Moderate, and Refusing = Hard.The dataset contains Engaged (3), Hesitant (7), and Refusing (15) tasks, intentionally emphasizing resistant patients.
  • D.1 Task Construction Details: Task generation uses schema-constrained rejection sampling over system-state graphs, handbook rules, and artifact formats, followed by three acceptance gates and blinded clinician editing.The gates require citation coverage, chart–rubric entailment without surface paraphrase, and a deterministic reference dry-run within a difficulty band.
  • D.2 Prior Authorization and Utilization Management: Payer UM criteria pair verbatim policy conditions with exact section citations, and positive judgments require policy references rather than merely clinically plausible reasoning.Criteria are organized by review stage, including documentation completeness, medical necessity, and physician-level review.
  • D.2 Prior Authorization and Utilization Management: Provider PA tasks classify each required document as present and sufficient or present but insufficient, producing terminal states of Submitted to payer, Returned for docs, or Gather more evidence.A valid packet requires all required document kinds to be present and sufficient; insufficient evidence requires upstream coordination with the ordering clinician.
  • D.3 Care Management: Care management follows a five-phase RN workflow from intake through chart review, outreach, formal assessment, and care-plan authoring and closure.The care plan uses the NANDA-I/NOC/NIC framework and requires at least two NANDA-I diagnoses with corresponding NOC targets and NIC interventions.
  • D.3 Care Management: Multi-turn outreach requires adaptive, noncoercive communication because opaque consent triggers, anti-triggers, and disclosure pacing govern enrollment and access to sensitive clinical information.Without enrollment consent, formal assessment instruments cannot be administered; a single anti-trigger can permanently close a consent path.

E Task Catalog … F.1 Sandbox Environment and Trial Orchestrator

χ-Bench catalogs 75 healthcare workflows across provider prior authorization, payer utilization management, and longitudinal care management, with tasks varying by workflow stage, service, difficulty, and patient consent. Each trial runs in an isolated containerized sandbox over a shared χ-World, with per-trial databases, filesystems, and tool servers shut down after completion.

  • E Task Catalog: 75 tasks span provider prior authorization, payer utilization management, and care management workflows.The catalog enumerates all 75 tasks, including 25 provider prior-authorization tasks and separate utilization-management and care-management catalogs.
  • E Task Catalog: Provider and utilization-management tasks sharing a service slug represent the same patient case observed at different workflow stages.Difficulty labels use structurally grounded Easy, Moderate, and Hard definitions.
  • E.1 PA: Prior Authorization (Provider Side): 25 provider prior-authorization tasks are uniformly invoked at the referral state for request assembly and submission.The catalog includes surgical, diagnostic, imaging, pharmacy, and other services with Easy, Moderate, or Hard difficulty labels.
  • E.2 UM: Utilization Management (Payer Side): Utilization-management tasks begin at intake, triage, nurse_review, md_review, or p2p workflow stages.Examples include intake routing, nurse clinical review, physician determination, triage disposition, and payer-side peer-to-peer review.
  • E.3 CM: Care Management: Care-management task difficulty follows the persona’s consent profile: Engaged is Easy, Hesitant is Moderate, and Refusing is Hard.The catalog covers longitudinal cases such as diabetes, heart failure, hypertension, and major depression.
  • F.1 Sandbox Environment and Trial Orchestrator: Each χ-Bench trial is a containerized sandbox bound to one task and layered over a shared χ-World containing one fixed population of data for all 25 domain tasks.The sandbox has its own per-trial database, filesystem, and tool-server processes, and is shut down after the trial so state is not shared across trials.
  • F.1 Sandbox Environment and Trial Orchestrator: At trial start, the χ-World backend, role-scoped MCP servers, and agent process are brought up in parallel.The passage describes this startup sequence as part of the simulator’s trial orchestration.

F.2 Pass@k and pass^k Metrics … F.5 Infrastructure and Harness Versions

The benchmark evaluates reliability with three independent trials per task and treats pass^3 as the operationally meaningful metric. Reproducibility is supported by fixed judging, container resources, harness customizations, and pinned infrastructure versions across 6,750 trials.

  • F.2 Pass@k and pass^k Metrics: Each task uses n = 3 independent trials, from which task-level pass@k and pass^k are computed.The pass count c ranges from 0 to n.
  • F.2 Pass@k and pass^k Metrics: pass^3 is the operationally meaningful reliability metric because long-horizon workflows must succeed on every individual case.Setting n=k=3 keeps pass^3 defined at a tractable evaluation budget.
  • F.2 Pass@k and pass^k Metrics: 6,750 trials per benchmark pass result from 30 cells × 75 tasks × 3 trials.This budget supports evaluating pass^3 with three trials per task.
  • F.3 Judge, Container, and Harness Configuration: Claude Opus 4.7 judges each rubric with three independent votes under a strict-majority quorum, with ties resolved as fail.Each rubric has a 1200-second and 128K-token cap; trials use 2 CPU cores, 4 GB RAM, and 10 GB ephemeral disk.
  • F.4 OpenAI Agents SDK Harness Customization: The OpenAI Agents SDK harness preserves the upstream Runner.run loop and limits customization to four reproducibility-critical areas.The SDK requires a thin wrapper because it is a library rather than an opinionated first-party CLI agent.
  • F.4 OpenAI Agents SDK Harness Customization: The harness uses role-scoped MCP servers, sanitizes dotted tool names, and spills tool outputs exceeding 100K characters to disk.The sanitizer is required for OpenAI and OpenRouter routes, while oversized outputs are replaced with a visible head and file pointer.
  • F.5 Infrastructure and Harness Versions: Table 17 pins the harness and runtime versions used for all 6,750 trials and records exact npm, pip, and container resolutions.Open-source versions correspond to upstream releases installed by each Harbor harness bootstrap script at experiment time.

F.6 Stratified Results

Performance failures concentrate in hard, coordination-heavy, adversarial, and consent-sensitive workflow conditions. Stratified results identify distinct failure patterns across prior authorization, utilization management, and care management.

  • By difficulty band: Hard-band tasks concentrate failures: PA Hard pass^3 is roughly an order of magnitude below PA Easy, while CM Refusing pass^3 approaches zero across every cell.The CM Refusing band represents Hard care-management tasks.
  • By workflow coverage: Gather more evidence is the hardest PA terminal-status bucket because it requires coordination with the ordering clinician beyond submission completion.UM Peer-to-peer pickup is also hardest because it requires adversarial multi-turn dialogue.
  • By clinical category: Surgical procedures comprise 12 of 25 payer-UM tasks and impose the longest policy decision chains.Specialty pharmacy is the smallest category and tests clinical-condition plus drug-specific criteria.
  • By patient engagement profile: Refusing personas account for 15 of 25 care-management tasks and drive failures when agents convert reluctant ambivalence into same-call consent.This pattern is identified as consent fabrication.

F.7 Judge Inter-rater Reliability · F.8 Per-domain Trial Cost and Wall-clock Statistics

The evaluation used three independent judge votes per binary rubric, strict-majority aggregation, and Cohen’s κ to assess agreement, while explicitly addressing verifier robustness and judge bias. Cost and runtime reporting complemented means with median and 95th-percentile wall-clock and token-use statistics, plus per-trial distributions.

  • F.7 Judge Inter-rater Reliability: F.7 Judge Inter-rater Reliability: V = 3 independent judge votes were aggregated by strict majority, with ties resolved as fail.Every rubric is binary (pass/fail).
  • F.7 Judge Inter-rater Reliability: Cohen’s κ was averaged across the three vote pairs within each rubric and then across rubrics within each domain.The method treats Cohen’s κ as the appropriate agreement statistic for binary rubrics.
  • F.7 Judge Inter-rater Reliability: Of 6,750 canonical task-runs, 4,780 produced at least one parseable judge verdict, while 1,970 terminated before the judging process completed.The passage reports these counts as the basis for the judge-verdict analysis.
  • F.7 Judge Inter-rater Reliability: Verifier robustness was treated as a within-judge stability check, and V = 3 self-agreement by Claude Opus 4.7 could not rule out same-family judge bias.The stated mitigation includes requiring deterministic contract success alongside judge success: RT = DeterministicPass ∧JudgePass.
  • F.7 Judge Inter-rater Reliability: The judgment, completion, and policy axes together accounted for 71.9% of the agent-attributable bucket, indicating failures tied to execution and policy interpretation.The passage contrasts this bucket with verifier brittleness and mentions a planned second-family cross-judge sensitivity check.
  • F.8 Per-domain Trial Cost and Wall-clock Statistics: USD cost uses harness-recorded cost_usd when available and otherwise derives cost from token counts at provider list rates; Hermes uses OpenRouter dashboard aggregates.Hermes does not record harness-level token usage.
  • F.8 Per-domain Trial Cost and Wall-clock Statistics: F.8 Per-domain Trial Cost and Wall-clock Statistics: Table 24 reports per-(harness, model) median and 95th-percentile wall-clock and token usage.These statistics complement Table 2’s mean values by reporting dispersion.
  • F.8 Per-domain Trial Cost and Wall-clock Statistics: Figure 22 visualizes per-trial wall-clock and token-use distributions as horizontal violins pooled by harness for readability.The pooling keeps the figure panels readable.

F.9 Task Level Results · G Failure-Mode Analysis

Per-task results are presented as domain-specific pass@1 heatmaps across provider prior authorization, utilization management, and care management. The figures identify unresolved tasks and summarize per-row performance, while additional plots report resource usage and a wall-clock cap.

  • F.9 Task Level Results: Rows are harness–model configurations sorted by overall pass@1, while columns are task IDs sorted by per-task pass@1.The heatmap ordering enables comparison across configurations and tasks.
  • F.9 Task Level Results: Red x markers identify tasks that no configuration passed on any of three trials.These tasks are listed as unresolved tasks in Table 26.
  • F.9 Task Level Results: Wall-clock, token, and cost distributions are summarized by harness and configuration using medians and 95th percentiles.Table 24 reports these resource metrics per harness–model pair.
  • F.9 Task Level Results: Per-trial wall-clock and total token usage are shown as violin distributions faceted by domain and pooled by harness.A vertical dashed line marks the 1800 s wall-clock cap.
  • F.9 Task Level Results: Per-task pass@1 is visualized as domain-specific heatmaps for provider prior authorization, utilization management, and care management.Figures 23–25 use cells representing pass@1 values in [0, 1].
  • F.9 Task Level Results: Each heatmap includes a right-edge annotation showing the per-row mean pass@1.This annotation accompanies the per-task cell values for each domain.

G.1 Analysis Methodology · G.2 Taxonomy Definitions · G.3 Failure-Mode Statistics

The paper classifies all 5,886 failed trials through a three-tier pipeline and a seven-mode failure taxonomy, with hallucination boundaries verified against complete trajectories. Clinical-Reasoning is the largest failure category at 35.4%, followed by Workflow-Completion at 23.3%.

  • G.1 Analysis Methodology: Tier 1 deterministically classifies unambiguous failures from runtime logs and verifier outcomes, including crashes, zero-step outputs, timeouts, and fatal tool-call cascades.Fatal tool-call cascades are classified as Tool-Use rather than infrastructure failures.
  • G.1 Analysis Methodology: Tier 2 routes analyzer-written evidence summaries by substring rules, while Tier 3 rechecks candidate hallucinations using the persona prompt, full trajectory, verifier outcome, and verbatim quotes.The Tier 3 boundary rule requires evidence establishing an input-versus-output mismatch.
  • G.1 Analysis Methodology: Claude Opus 4.7 produces one JSON summary per trial from the full extract, rubric outcome, and six taxonomy examples; Tier 2 primarily consumes its evidence string.Candidate categories serve only as a fallback when substring rules do not match.
  • G.2 Taxonomy Definitions: Hallucination means an assertion contradicting or unsupported by plainly readable input, whereas Clinical-Reasoning means observing relevant facts but reaching an incorrect narrative explanation.Criterion misreading concerns the rule text itself; criteria misapplication concerns applying a correct rule or evidence incorrectly.
  • G.2 Taxonomy Definitions: Concern-mining repeatedly reframes scope until reluctance becomes agreement; explicit affirmative “yes” distinguishes illegitimate consent from consent fabrication.Hard-refuse success requires autonomy-first engagement, refusal acknowledgment, transparent limitations, and explicit affirmative enrollment.
  • G.3 Failure-Mode Statistics: 35.4% of failures are Clinical-Reasoning, 23.3% are Workflow-Completion, 15.6% are Abstain-or-Stuck, and 13.2% are Policy-Compliance.Clinical-Reasoning peaks on CM at 50.0% of CM failures, driven by 15 hard-refuse tasks; Policy-Compliance consists entirely of criterion misreading.

G.4 Trajectory and Case Analysis

Trajectory analysis attributes failures primarily to clinical reasoning, workflow completion, policy compliance, and tool-use errors, while case studies show agents mishandling terminal actions, refusals, and interaction timing. Infrastructure faults and hallucinations are comparatively rare, but some harness failures could have been recoverable.

  • Failure categories: 1.0% of failures were Harness-Faults and 0.8% were Hallucinations, while one tool-use case could likely have recovered through an early retry.Harness faults included crashes, setup errors, transport exceptions, and zero-step exits; hallucinations included consent, clinical-fact, and tool-result fabrication.
  • Failure categories: 10.7% of failures were Tool-Use-Error cases, including 432 fatal tool-call cascades, 133 wrong tool selections, and 61 malformed arguments.DeepAgents’ default-no-recovery policy escalated malformed calls into trial-fatal exits.
  • Case analysis: Case analyses show terminal and interaction failures: agents stopped before submission, chose gather_more_evidence despite ready_to_submit: true, manufactured consent after refusal, or finalized care artifacts before the promised follow-up assessment.Examples include Cases 8–9 and Cases 10–12.
  • Failure categories: 35.4% of failures were Clinical-Reasoning errors, dominated by misapplied clinical criteria (1,647; 79.1%), despite correct tools and evidence.This was the single largest failure category.

G.5 Unresolved Tasks

Seventeen of 75 tasks were unresolved across all 30 × 3 harness–model trials, with zero pass@1 and pass3 scores for every task. These comprised nine PA provider new-referral cases, six UM payer-side review tasks, and two CM hard-refusal cases.

  • Unresolved task set: 17 of 75 tasks scored pass3=0 across every harness–model row, and all 17 also scored pass@1=0.No trial across the entire 30 × 3 matrix passed any of these tasks.
  • Unresolved task set: 9 tasks were PA provider new-referral cases requiring a specific submission bundle that no row produced.These cases were part of the 17 tasks where every harness–model row scored 0/3.
  • Unresolved task set: 6 tasks were UM payer-side intake, triage, nurse, or MD review tasks, and 2 were CM hard-refusal cases.Together, these categories account for all 17 tasks unresolved across every harness–model row.

G.6 Skill Invocation: Did the Agent Actually Read the Policy?

Policy-read recall measures how often agents accessed the handbook policies cited by the ground-truth rubric. Recall varied substantially by failure mode, and higher recall was strongly associated with first-attempt success across 30 harness–model configurations.

  • Measurement: Policy-read recall is the fraction of ground-truth-cited handbook policies accessed through Read, Grep, or Bash cat calls.Ground-truth citations come from each task’s expectations.json and include outreach-protocol.md, escalation-rules.md, payer policy section IDs, and NNN node identifiers.
  • Failure modes: 66.4% recall occurred for Hallucination failures, where agents had the policy text but produced a contradicting structured field.This was the highest listed recall among the reported failure modes.
  • Failure modes: 44.0% recall occurred for Clinical-Reasoning failures, where agents read about half of the relevant policies but applied them to the wrong patient state.Workflow-Completion, Abstain-or-Stuck, Policy-Compliance, Tool-Use-Error, and Harness-Fault had recalls of 24.9%, 22.6%, 18.5%, 14.0%, and 12.7%, respectively.
  • Outcome association: r = +0.77 across n = 30 harness–model cells indicates a strong positive rank correlation between policy-read recall and pass@1.The figure also reports consistently positive within-domain Pass-vs-Fail recall gaps, including CM +19 pp and UM +5 pp.

H Prompts

Appendix H specifies every LLM-use point in χ-Bench, including verifier judging and failure-mode analysis templates with rendered examples for the two most consequential prompts. The verifier enforces role-specific, evidence-cited rubric decisions, while the analyzer classifies trial behavior into structured failure or success modes.

  • H Prompts: Appendix H documents every LLM-use point through system or system+user templates, with rendered examples for the verifier judge and failure-mode analyzer.Long templates are split at natural boundaries, and rendered examples are trimmed with inline elisions.
  • H.1 Verifier Rubric Judge: The verifier judge uses Claude Opus 4.7, casts 3 votes per rubric, applies a strict-majority quorum, and resolves ties as fail.It reads persisted simulator state and the role-stage rubric, then emits binary pass/fail verdicts through PA, UM, or CM role-specific prompts.
  • H.1 Verifier Rubric Judge: The verifier workflow reads rubrics and source evidence, investigates named agent outputs, cites file paths and line ranges, and incrementally persists verdicts.json.The judge must terminate after writing the final verdict file.
  • H.1 Verifier Rubric Judge: Verifier verdict discipline requires explicit accounting of contextual items, primary-source citations to agent artifacts, and literal quotation of cited field names and values.The judge cannot add deterministic checks beyond rubrics.json and must fail unsupported evidence or missed required elements.
  • H.1 Verifier Rubric Judge: CM outreach-quality receives a NO verdict when transcripts contain listed consent anti-triggers, including unsupported promises forbidden by the task.The hard-fail rule checks per-task persona consent_anti_triggers and cm_reference.outreach_expectations consent_anti_triggers.
  • H.1 Verifier Rubric Judge: CM outreach also hard-fails when sensitive chart details precede effective identity, access, and scope explanations or when statements are mistaken for enrollment consent.Effective repair requires stopping, acknowledging the issue, explaining source and scope, allowing refusal, and asking permission; “Send me information” and similar statements do not count as enrollment consent.
  • H.2 Failure-Mode Analyzer: The Stage 2 analyzer uses Claude Opus 4.7 to process ∼225 trial extracts per row, producing structured labels, evidence, confidence, and a row summary.OTHER or low-confidence trials go to Stage 2.6, which applies pattern → locate → context → ground-truth cross-check with head 3 plus tail 12 messages.
  • H.2 Failure-Mode Analyzer: The analyzer’s taxonomy and rules constrain each trial to success categories or specified failure categories, require exact trial IDs, and mark confidence low when evidence is thin or ambiguous.The Stage 2 template also requires JSON-only output, trajectory-grounded evidence, and strict disambiguation.
Loading 2605.16679v2…