ERPBench: A State-Grounded Evaluation Paradigm for Computer-Use Agents in Enterprise Software

• General GUI benchmarks such as OSWorld, Windows Agent Arena, and WebArena primarily target desktop and web environments, while enterprise software spans business workflows with higher failure costs.The passage cites these benchmarks as,, and, respectively.Source: §1 · p. 1
• Enterprise interactions can appear successful while leaving a business record stale, empty, or wrong, with errors potentially propagating into operational and financial processes.Existing enterprise benchmarks use proprietary platforms or simulated environments and do not decompose runs into workflow stages.Source: §1 · p. 1
• ERPBench instantiates state-grounded evaluation on a live, self-hosted ERPNext system, grading screenshot-only agents against ground-truth database fields.Field-level grading helps distinguish failures such as unrecoverable interaction loops that task-level metrics can conflate.Source: §1 · p. 1
• ERPBench combines an open, reproducible benchmark, a production-grade harness with human approval for irreversible actions, and systematic analysis of enterprise-specific failures.The harness runs autonomously for benchmarking, while gating actions behind human approval in deployment.Source: §1 · p. 2

• General GUI benchmarks evaluate desktop and web tasks, but do not model enterprise business state; ERPBench extends WebArena’s execution-based verification to database-field correctness.The related work identifies OSWorld and Windows Agent Arena as examples of general GUI benchmarks.Source: §2 · p. 2
• Prior enterprise evaluations vary in modality and verification: SCUBA uses application APIs, CRMArena-Pro uses text interaction, and EntWorld [15] exposes an accessibility tree alongside screenshots.UI-CUBE [10] uses simulated web applications, DOM information, and application-state snapshots; ERPBench instead uses screenshot-only control on a live, self-hosted ERP and database-field grading.Source: §2 · pp. 2–3
• Prior safety work studies approval, intent verification, and risk awareness [16] [17] [18] [19], while ERPBench integrates risk-tiered approval into a production-oriented harness also used for autonomous benchmarking.The paper positions its benchmark as complementary to computer-use agent systems, evaluating their enterprise reliability rather than improving GUI grounding.Source: §2 · p. 3
• ERPBench is the only benchmark in the comparison to satisfy every row, spanning observation modality, deployment, grounding, and grading.The comparison includes general GUI benchmarks OSWorld and Windows Agent Arena, and enterprise benchmarks WorkArena, SCUBA, CRMArena-Pro, and UI-CUBE [10].Source: §2 · p. 3

3. ERPBENCH

• Agents operate ERPNext through browser screenshots and pixel-coordinate mouse and keyboard actions, with no DOM or accessibility tree exposed.The pixel-only channel is intended to mirror remote-desktop or VDI settings without API or accessibility hooks.Source: §3 · p. 3

3.1. Task Design

• Tasks progress from T1 single-record edits to T2 multi-field records and T3 chains of two to four dependent ERPNext documents, all judged by final database state.Tasks specify prompts, start URLs, fixtures, expected values, and grader configuration; UUID-suffixed record names isolate runs without a full database reset.Source: §3.1 · pp. 3–4

3.2. Harness

• The harness gates commit or irreversible actions for human approval in deployment, while evaluation auto-approves actions and scores runs from task-specific start URLs.The operator interface includes the live ERPNext workspace, planned-action log, risk-gated approval panel, and chat panel.Source: Figure 1 · p. 3, §3.2 · p. 4, Figure 2 · p. 4

3.3. Grading

• The database grader checks post-run fields against expected values using type-specific equivalence, while stage grading tracks navigation, interaction, commit, and database correctness.The log grader records action and turn counts, latency, and, where available, token cost; final database state remains the definitive success criterion.Source: §3.3 · p. 4

3.4. Human Baseline

• Three annotators provide human references scored by the same database grader, spanning experienced ERPNext users and a first-time user.Humans use the browser directly, so the paper reports their task success and completion time rather than agent-specific metrics such as token usage.Source: §3.4 · pp. 4–5

4.1. Experimental Setup

• Six agents complete five runs per task under a shared configuration and database grader, across 100 T1, 20 T2, and 30 T3 runs per model.Three human annotators complete the same tasks using the same database grader; stage measures distinguish navigation, interaction, commit, and database correctness.Source: §4.1 · p. 5

4.2. Results

• Claude completes all six T3 workflows (30/30) at 36 actions and 323 seconds per run; OpenCUA-32B chain depth falls from 22% warm-start to 9% blank-start.Open models score 22–47% on exact free-text entry versus 31–54% on select/toggle choices, except Holo3.Source: §4.2 · p. 5, Table 5 · p. 7
• A first-time user achieves 96%, 90%, and 87% success on T1–T3, while open-model maxima are 34% on T1 and 3% on T3.Experts reach 99–100% on T1, 95% on T2, and 97–100% on T3.Source: §4.2 · p. 5
• Claude reaches 94%, 100%, and 100% success across T1–T3, while the strongest open models reach 34% and 32% on T1 and 0–3% on T2–T3.Holo3-35B-A3B outscores Claude on OSWorld-Verified, 77.8 to 72.5, yet reaches 34% on T1 where Claude reaches 94%.Source: §4.2 · p. 5, Table 2 · p. 6
• Open models navigate to target records in 90–95% of T1 runs but reach database correctness in only 0–34%; on T2, they reach records in every run but are correct in at most 5%.UI-TARS commits in 68% of T1 runs but leaves the correct database value in only 9%.Source: §4.2 · p. 5, Table 3 · p. 6

4.3. Failure Analysis

• The five failure modes distinguish recovery, planning, grounding, save-step, and perception; open-weight models fail predominantly after navigation, including 40 grounding versus 10 planning failures for Qwen3-VL-32B.Perception and save-step failures can leave the persistent database wrong or unchanged despite an apparently successful save.Source: §4.3 · pp. 6–7

• ERPBench evaluates screenshot-only agents on a real, open, self-hosted ERP by grading the database state they leave and tracing failures across navigation, interaction, commit, and correctness.Source: §5 · p. 7
• Across six agents, strong general GUI performance did not carry over to enterprise work, and database-grounded grading exposed failures invisible at the GUI level.Source: §5 · p. 7
• Because failures localize to interaction, commit, and persistence, the authors point to stage-targeted finetuning on state-grounded trajectories, while human approval supports deployment meanwhile.Source: §5 · p. 7