Source-linked AI summary
HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks
Suhana Bedi, Ryan Welch, Ethan Steinberg, Michael Wornow, Taeil Matthew Kim, Haroun Ahmed, Peter Sterling, Bravim Purohit, Qurat Akram, Angelic Acosta, Esther Nubla, Pritika Sharma, Michael A. Pfeffer, Sanmi Koyejo, Nigam H. Shah
TL;DR
Healthcare administration is underexplored by LLM evaluations despite accounting for over $1 trillion in annual U.S. spending and requiring multi-step, cross-system workflows. HEALTHADMINBENCH evaluates seven agent configurations on 135 expert-designed tasks across four simulated healthcare-administration environments, using 1,698 fine-grained verifiers. End-to-end task success remains low despite stronger subtask performance, with Claude Opus 4.6 CUA achieving 36.3% full-task success while GPT-5.4 CUA attains the highest subtask success rate.
Problem
Healthcare administration is underexplored by LLM evaluations despite accounting for over $1 trillion in annual U.S. spending and requiring multi-step, cross-system workflows.
Method
HEALTHADMINBENCH evaluates seven agent configurations on 135 expert-designed tasks across four simulated healthcare-administration environments, using 1,698 fine-grained verifiers.
Results
End-to-end task success remains low despite stronger subtask performance, with Claude Opus 4.6 CUA achieving 36.3% full-task success while GPT-5.4 CUA attains the highest subtask success rate.
Takeaways & Limitations
The benchmark reveals a substantial gap between interface-level competence and reliable deployment-ready automation of complete healthcare administrative workflows.
Takeaways & Limitations
The benchmark uses static, retrospective environments that simplify production systems by omitting interface drift, evolving payer policies, deployment heterogeneity, and human-like recovery behavior.
Abstract
from arXiv · showhide
Healthcare administration accounts for over $1 trillion in annual spending, making it a promising target for LLM-based computer-use agents (CUAs). While clinical applications of LLMs have received significant attention, no benchmark exists for evaluating CUAs on end-to-end administrative workflows. To address this gap, we introduce HealthAdminBench, a benchmark comprising four realistic GUI environments: an EHR, two payer portals, and a fax system, and 135 expert-defined tasks spanning three administrative task types: Prior Authorization, Appeals and Denials Management, and Durable Medical Equipment (DME) Order Processing. Each task is decomposed into fine-grained, verifiable subtasks, yielding 1,698 evaluation points. We evaluate seven agent configurations under multiple prompting and observation settings and find that, despite strong subtask performance, end-to-end reliability remains low: the best-performing agent (Claude Opus 4.6 CUA) achieves only 36.3 percent task success, while GPT-5.4 CUA attains the highest subtask success rate (82.8 percent). These results reveal a substantial gap between current agent capabilities and the demands of real-world administrative workflows. HealthAdminBench provides a rigorous foundation for evaluating progress toward safe and reliable automation of healthcare administrative workflows.
1 INTRODUCTION
Healthcare administration is costly, error-prone, and poorly represented by existing computer-use benchmarks. HEALTHADMINBENCH addresses this gap with realistic multi-step environments, expert-designed tasks, fine-grained verification, and evaluations of agent reliability.
- Motivation: Healthcare administration exceeds $1 trillion in annual U.S. spending, yet remains underexplored relative to clinical LLM applications and involves multi-step work across legacy systems without API support.These workflows include prior authorization, claims submission, and patient intake, and can contribute to preventable medical errors.
- Benchmark gap: Existing benchmarks do not capture healthcare administration’s gated access, limited documentation, sensitive data, and dynamic cross-system interactions.Prior healthcare-specific evaluation is limited to static, text-based assessment rather than computer-use workflows.
- Benchmark design: HEALTHADMINBENCH introduces four deterministic web environments and 135 expert-designed tasks spanning Prior Authorization, Appeals and Denials Management, and DME Order Processing.The environments include an EHR, two payer portals, and a fax system, with tasks derived from real-world administrative work and aligned with MedHELM’s Administration and Workflow category.
- Evaluation: The benchmark decomposes tasks into 1,698 verifiable subtasks evaluated with deterministic checks and LLM-based judges.The released package includes environments, task definitions, evaluation code, and model outputs to support reproducible benchmarking.
- Headline finding: The strongest evaluated agent achieves only 36.3% full-task success despite stronger performance on individual subtasks.This result motivates measuring end-to-end reliability rather than subtask performance alone.
2 BACKGROUND AND RELATED WORK
Prior benchmark families cover web, enterprise, operating-system, and clinical tasks, but do not jointly represent healthcare administration’s cross-application, schema-constrained workflows. HEALTHADMINBENCH is positioned to fill that evaluation gap with healthcare-specific environments and objective success checks.
- Computer-use benchmarks: Existing web and enterprise benchmarks evaluate increasingly realistic multi-step interactions but generally omit domain-specific healthcare workflows spanning multiple systems and organizations.These benchmarks include synthetic UI tasks, multi-website tasks, CRM and ticketing systems, and unified browser frameworks.
- OS and multi-application workflows: Operating-system benchmarks expose failures in perception-action grounding and operational knowledge, challenges amplified by healthcare workflows requiring coordination across environments and schema-constrained execution.Healthcare administration adds rigid constraints and cross-application dependencies beyond ordinary GUI interaction.
- Benchmark comparison: Table 1 compares benchmarks using cross-application workflows, rigid schema validation and terminal commits, verifiable success criteria, and healthcare grounding.These dimensions identify the structural properties most relevant to administrative work.
- HEALTHADMINBENCH: HEALTHADMINBENCH uses four simulated applications—an EHR, two payer portals, and an eFax—to model commonly used administrative interfaces.The websites are implemented following the REAL framework (Garg et al., 2025).
- Healthcare evaluation gap: Clinical evaluation suites such as MedHELM include healthcare administration, but their largely text-based assessments do not capture payer-facing navigation, form completion, and submission.Earlier healthcare-agent benchmarks mainly focus on clinical workflows within single EHR environments.
3 HEALTHADMINBENCH
HEALTHADMINBENCH models healthcare administration as partially observable browser-based decision problems over deterministic, synthetic environments. Its 135 workflow tasks span three revenue-cycle categories and are decomposed into fine-grained subtasks with deterministic or LLM-based verification.
- Task formalization: Administrative tasks are modeled as POMDPs in which agents receive partial screenshots or structured interface views and execute browser actions that deterministically transition environment state.Episodes end at task completion or a fixed interaction budget, with binary terminal reward based on successful completion.
- Environments: The benchmark provides four sandboxed web systems: an EHR-like viewer, two payer authorization portals, and a fax submission portal designed from over 100 hours of observed administrative work.Synthetic patient data, typed inputs, schema validation, policy checks, and realistic failure modes enforce task logic while abstracting deployment-specific access constraints.
- Action space and observations: Agents use natural-language keyboard and mouse actions, while accessibility-tree observations permit symbolic interaction through stable data-testid identifiers instead of pixel-coordinate grounding.Screenshot-only agents must additionally infer each action’s spatial location, whereas structured observations expose indexed interactive elements.
- Task suite: The 135 tasks cover Prior Authorization, Appeals and Denials Management, and DME Order Processing, with tasks requiring information retrieval, structured interaction, reasoning, documentation handling, and valid stopping decisions.The task counts are 60 prior-authorization tasks, 60 appeals-and-denials tasks, and 15 DME tasks.
- Verification: Task completion requires satisfying every fixed subtask, yielding a strict end-to-end reliability measure across 1,698 subtasks.Of these subtasks, 1,177 use deterministic checks and 521 use LLM-based evaluation; 419 are information-retrieval subtasks.
4 RESULTS
Across HEALTHADMINBENCH, native computer-use systems outperform standardized-harness agents on end-to-end reliability, but strong subtask performance does not translate into reliable full-task completion. Performance varies by task and subtask type, prompting and observation choices matter, and limited domain-specific fine-tuning substantially improves results.
- Overall performance: Claude Opus 4.6 CUA achieves the highest task success rate, while GPT-5.4 CUA attains the highest subtask success rate, revealing a gap between system-level reliability and individual-step execution.Task success requires passing every subtask, making it a stricter end-to-end measure.
- Performance by task type: DME Order Processing has the highest success rates, Prior Authorization shows mixed performance, and Appeals and Denials Management remains consistently challenging.Prior Authorization and Appeals and Denials Management involve longer action sequences and more Clinical Reasoning and Information Retrieval subtasks than DME.
- Performance by subtask type: Information Retrieval is the easiest subtask type, whereas Clinical Reasoning and Task Resolution are the most challenging.CUA agents lead across subtask types, with GPT-5.4 CUA strongest on Documentation, Document Handling, and Clinical Reasoning, and Claude Opus 4.6 CUA strongest on Information Retrieval, Form Completion, and Task Resolution.
- Ablation experiments: prompting and observation: Portal Guidance improves task success over Task Description alone, and Accessibility Tree observations generally outperform screenshot-only observations.The gains are larger for end-to-end performance, although accessibility-tree identifiers may be unavailable in real deployments.
- Fine-tuning on domain-specific data: Qwen-3.5-Kinetic-SFT reaches 40% task success on the held-out test set, gaining +23% over its base model and exceeding Claude Opus 4.6 by over +14%.The adaptation uses supervision from trajectories generated on 100 training tasks and is evaluated on 35 held-out tasks.
- Qualitative analysis of failure modes: Agents commonly fail because they avoid file operations, lose information over long horizons, and struggle with hidden dependencies across systems.These failures affect document-handling, downstream data use, and recovery during multi-step workflows.
5 DISCUSSION
HEALTHADMINBENCH evaluates healthcare agents on realistic, long-horizon workflows rather than static text-only tasks. Results show low end-to-end reliability despite stronger subtask performance, while fine-tuning suggests domain-specific workflow data may improve outcomes; however, the benchmark remains a simplified proxy for production systems.
- Discussion: HEALTHADMINBENCH moves beyond static text-only evaluation to realistic administrative workflows requiring long-horizon, cross-system interaction.The benchmark targets document handling, cross-portal coordination, and sustained state tracking.
- Discussion: Claude Opus 4.6 CUA reaches only 36.3% task success despite much higher subtask success rates, indicating persistent difficulty completing full administrative workflows.The discussion links these difficulties particularly to document handling, cross-portal coordination, and sustained state tracking.
- Discussion: Fine-tuning results suggest that failures reflect not only model capability but also the scarcity of domain-specific workflow data and training environments.This interpretation is supported by the improvement observed after limited domain-specific supervision.
- Limitations: The benchmark is a simplified proxy: it omits interface drift, changing payer policies, deployment heterogeneity, human recovery, and constraints such as MFA, CAPTCHAs, and session timeouts.Accessibility-tree evaluation also uses clean, stable identifiers that real-world portals typically do not expose.
6 CONCLUSION
HEALTHADMINBENCH evaluates computer-use agents on realistic healthcare administrative workflows and finds that current frontier agents remain unreliable end to end despite promising subtask performance.
- HEALTHADMINBENCH reveals a substantial gap between interface-level competence and deployment-ready workflow automation.The benchmark is intended to measure progress toward safer and more effective administrative AI systems in healthcare.
A PROMPTING STRATEGY DETAILS
The benchmark uses standardized prompts, action formats, and observation settings to support reproducible computer-use evaluation, while testing both minimal and domain-informed guidance. Task-specific procedural prompts are reserved for validation and debugging rather than reported benchmark performance.
- A.1 BASE SYSTEM PROMPTS: Accessibility-tree agents use symbolic data-testid identifiers, whereas screenshot-only agents select actions using screen coordinates.Table 5 enumerates the action space, including clicks, text entry, selection, scrolling, and file transfer.
- A.1 BASE SYSTEM PROMPTS: The fixed system prompt defines the agent role, admissible actions, required syntax, structured output, executable identifiers, and standardized done() termination.The structured output records the next action and newly observed task-relevant information as a task trace.
- A.1 BASE SYSTEM PROMPTS: Harness-based agents use standard chat or completion endpoints, while native CUA configurations rely on vendor SDKs to manage the interaction loop.The THINKING field is an explicit action rationale and task trace, not a special reasoning-mode API.
- A.1 BASE SYSTEM PROMPTS: The prompt requires agents to respond with one action plus concise key information summarizing newly observed, task-relevant page content.Examples include scrolling, downloading and uploading documents, and emitting done() after completion.
- A.2 TASK DESCRIPTION PROMPT AMENDMENTS: Minimal prompting provides only the base system prompt and task goal, testing whether agents can infer administrative strategies from the environment.
- A.3 PORTAL GUIDANCE PROMPT AMENDMENTS: Portal guidance adds modular instructions for action syntax, document transfer, multi-portal navigation, workflow modules, and relevant payer or fax portals.The amendments are assembled per task from workflow-specific hint blocks.
- A.3 PORTAL GUIDANCE PROMPT AMENDMENTS: Portal guidance specifies custom dropdown interaction and MM/DD/YYYY date entry for the accessibility-tree setting.Dropdowns require opening the list and then selecting the option; select() is disallowed.
- A.4 TASK-SPECIFIC PROMPT AMENDMENTS: Task-specific prompt amendments are excluded from reported metrics because such detailed procedural guidance is unlikely in realistic deployments.They are used solely to validate task correctness, environment consistency, and successful task completion.
B LLM JUDGE EVALUATION
The benchmark validates its GPT-5.4 LLM judge against independent human reviewers using a stratified sample of Claude Opus 4.6 submissions. Human-judge agreement is high, supporting the reliability of the evaluation rubrics and judge.
- The validation sample contained 60 Claude Opus 4.6 submissions, stratified into 30 judge-labeled failures and 30 judge-labeled successes.Reviewers assessed outcomes without seeing the LLM judge’s result or reasoning.
- Human reviewers unanimously agreed on 30 successes and 26 failures, while disagreeing about four remaining subtasks.Human-human agreement was also 93.3%, with a 95% confidence interval of 83.8% to 98.15%.
- 93.3% human-judge agreement supports GPT-5.4 as an adequate evaluator for LLM-judged subtasks.Humans agreed with the judge on 112 of 120 reviews, with a 95% confidence interval of 87.4% to 97.1%.
C RESOURCES USED: STEPS TAKEN AND COST
Resource analysis compares agents by interaction steps and API cost, showing that fewer steps may reflect giving up or batching actions rather than greater efficiency. Cost comparisons exclude native CUAs and identify Claude Opus 4.6 as unusually expensive.
- Steps Taken: Gemini 3.1 Pro uses the fewest steps, partly because it tends to give up when struggling while other models continue until termination.GPT-5.4 also uses fewer steps because it often performs multiple actions per turn.
- Steps Taken: A benchmark step is one interaction turn in which the model receives input and is asked to take actions.
- Cost: Most agents have comparable API costs, except Claude Opus 4.6, which is very expensive to run.Figure 6 reports agent API costs by task difficulty, but native CUA agents are omitted because of inconsistent cost accounting.
- Cost: Anthropic CUA and OpenAI CUA are excluded from cost plots because their harnesses use inconsistent cost accounting, preventing fair comparison.
D.1 SUBTASK SUCCESS RATE BY PROMPTING & OBSERVATION
Subtask success is consistently higher with Portal Guidance than Task Description alone and with Accessibility Tree observations than Screenshots. This stronger step-level performance does not translate into comparable end-to-end reliability.
- Subtask Success Rate: 89.7% subtask success for Claude Opus 4.6 with Portal Guidance + Accessibility Tree contrasts with 51.9% task success.The same agent under native CUA reaches 78.4% subtask success versus 36.3% task success.
- Prompting Strategy: Portal Guidance improves subtask success over Task Description alone across the prompting grid.
- Observation Modality: Accessibility Tree observations outperform Screenshot observations across the evaluated settings.Figure 7 reports these comparisons across prompting strategies and observation modalities, with 95% bootstrap confidence intervals.
D.2 HEAD-TO-HEAD AGENT COMPARISONS
Head-to-head comparisons use pairwise performance differences with confidence intervals to test whether HEALTHADMINBENCH distinguishes agents. The benchmark detects many statistically significant differences between agents.
- Head-to-Head Comparisons: Pairwise head-to-head comparisons reveal statistically significant differences between many evaluated agents.The comparisons report pairwise performance differences together with confidence intervals.
- Task Success: Table 6 compares agents using task-success differences, with positive values indicating that the column agent outperforms the row agent.It reports 95% test-set bootstrap confidence intervals and marks statistically significant rows at p < 0.05.
- Subtask Success: Table 7 applies the same pairwise comparison framework to subtask success rates.Positive values favor the column agent, with 95% bootstrap confidence intervals and significance markings at p < 0.05.
E FINE-TUNING ON DOMAIN-SPECIFIC DATA
Domain-specific fine-tuning evaluates Qwen-3.5-Kinetic-SFT and Claude Opus 4.6 on 35 held-out tasks under accessibility-tree observations and task-description prompting. Fine-tuning improves task success over the base Qwen model, although the task-level comparison with Claude is not statistically significant.
- Experimental Setup: Fine-tuning uses accessibility-tree observations and Task Description prompting, with 100 training tasks and 35 held-out test tasks.Training trajectories use task-specific step-by-step prompting, while Claude is evaluated without further fine-tuning.
- Results: 40% task success for Qwen-3.5-Kinetic-SFT is a +23% absolute improvement over base Qwen 3.5 27B and exceeds Claude Opus 4.6’s 25.7%.These results use 35 held-out examples under the fine-tuning evaluation configuration.
- Statistical Comparison: The subtask-performance increase over Claude Opus 4.6 is statistically significant, but the task-performance increase is not.Head-to-head bootstrap comparisons address statistical uncertainty in the 35-task evaluation.
- Implications: Only 100 domain-specific examples substantially improve performance on complex administrative workflows.The result highlights the potential for rapid adaptation and the importance of domain-specific workflow data.
- Limitations: Results may vary because the experiment uses one benchmark configuration and a relatively small task subset.The stated scope boundary includes Task Description prompting, accessibility-tree observations, and sensitivity to different splits or larger-scale fine-tuning.