PaperlayerView exact revision on arXiv

Source-linked AI summary

SCUBA: Salesforce Computer Use Benchmark

Yutong Dai, Krithika Ramakrishnan, Jing Gu, Matthew Fernandez, Yanqi Luo, Viraj Prabhu, Zhenyu Hu, Silvio Savarese, Caiming Xiong, Zeyuan Chen, Ran Xu

arXiv:2509.26506v1cs.AI

TL;DR

Enterprise computer-use agents lack benchmarks that capture the complexity of CRM workflows and evaluate performance beyond success rates. SCUBA introduces an interactive Salesforce benchmark for these workflows. Its experiments find differences across agent designs and model types, while demonstrations improve task success and reduce time and cost.

  • Problem

    Existing benchmarks do not fully capture enterprise-grade platforms and workflows, where latency and costs also matter alongside task success.

    Source: §1 · p. 2

  • Method

    SCUBA evaluates computer-use agents on interactive Salesforce CRM workflows, pairing tasks with initialization scripts and rule-based evaluation.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Browser-use agents outperform specialized computer-use agents at the current stage, while demonstrations improve success and reduce task time and cost.

    Source: §4.3 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    SCUBA aims to accelerate progress toward reliable computer-use agents for complex business software ecosystems.

    Source: §1 · p. 2

    Planning and grounding remain major causes of computer-use task failure; demonstrations partly address planning, while grounding persists.

    Source: §4.3 · p. 9

Abstract

from arXiv · show

We introduce SCUBA, a benchmark designed to evaluate computer-use agents on customer relationship management (CRM) workflows within the Salesforce platform. SCUBA contains 300 task instances derived from real user interviews, spanning three primary personas, platform administrators, sales representatives, and service agents. The tasks test a range of enterprise-critical abilities, including Enterprise Software UI navigation, data manipulation, workflow automation, information retrieval, and troubleshooting. To ensure realism, SCUBA operates in Salesforce sandbox environments with support for parallel execution and fine-grained evaluation metrics to capture milestone progress. We benchmark a diverse set of agents under both zero-shot and demonstration-augmented settings. We observed huge performance gaps in different agent design paradigms and gaps between the open-source model and the closed-source model. In the zero-shot setting, open-source model powered computer-use agents that have strong performance on related benchmarks like OSWorld only have less than 5\% success rate on SCUBA, while methods built on closed-source models can still have up to 39% task success rate. In the demonstration-augmented settings, task success rates can be improved to 50\% while simultaneously reducing time and costs by 13% and 16%, respectively. These findings highlight both the challenges of enterprise tasks automation and the promise of agentic solutions. By offering a realistic benchmark with interpretable evaluation, SCUBA aims to accelerate progress in building reliable computer-use agents for complex business software ecosystems.

1 Introduction

Enterprise workflows remain difficult to evaluate because existing benchmarks do not capture the full complexity of enterprise platforms. SCUBA addresses this gap with Salesforce CRM tasks and multidimensional evaluation.

1 Introduction

  • Enterprise computer-use evaluation needs benchmarks that reflect complex workflows and measure latency and cost as well as success.The introduction identifies a gap in benchmarks for enterprise-grade platforms and tasks.Source: §1 · p. 2
  • WebArena and OSWorld cover web navigation and generic desktop tasks, while WorkArena focuses on ServiceNow service use cases and CRMArena emphasizes Salesforce information retrieval.The introduction notes that CRMArena tasks do not write or change environment records and settings.Source: §1 · p. 2
  • SCUBA evaluates Salesforce CRM workflows drawn from user interviews across platform administration, sales, and customer service.Its tasks cover UI navigation, data manipulation, workflow automation, information retrieval, and troubleshooting.Source: §1 · p. 2, Table 1 · p. 2
  • SCUBA pairs tasks with rule-based success and milestone evaluation, reports latency and cost, and supports asynchronous evaluation that can finish in under 90 minutes.The benchmark also provides knowledge articles and human demonstrations; demonstrations can improve success while lowering latency and costs.Source: §1 · p. 2

2 Related Work

Prior work develops computer-use agents through several model and framework paradigms, while benchmarks vary in platform, task scope, and evaluation. The related work also notes a throughput constraint for API-based models.

2.1 Computer-Use Agents

  • Computer-use agents include foundational VLMs with computer-use capabilities, native GUI agents, and agentic frameworks that delegate capabilities to specialized components.The related work describes toolboxes for executable actions in the first paradigm and notes that specialized grounding or coding/API designs can help agentic frameworks on OSWorld.Source: §2.1 · pp. 2–3
  • API-based models are bottlenecked by API rate limits, unlike the self-served models referenced in the footnote.The footnote specifies that the stated qualification applies to self-served models.Source: §2.1 · p. 2

2.2 Computer-Use Benchmarks

  • The benchmark landscape spans interactive browser and desktop tasks, including simulated or replicated websites, production-grade website sandboxes, and desktop applications.Examples include WebArena and VisualWebArena, REAL, WorkArena and Amazon-Bench, and OSWorld and its Windows and macOS follow-ups.Source: §2.2 · p. 3

3 SCUBA Benchmark

SCUBA is an interactive Salesforce benchmark for CRM workflows, combining a sandbox environment, tasks based on user workflows, and rule-based evaluation. Its construction covers task templates, varied queries, initialization, scoring, and human review.

3 SCUBA Benchmark

  • SCUBA is an interactive Salesforce environment for automation-focused tasks, each paired with an initialization script and rule-based evaluation.The platform is described as a widely used cloud-based enterprise platform for CRM workflows.Source: §3 · p. 3

3.1 Environment

  • The Salesforce sandbox uses a production-identical UI, while task-level resets revert changed configuration files rather than recreating the org.Parallel browser sessions or OS desktops allow agents to run asynchronously in dedicated environments.Source: §3.1 · p. 3

3.2 Tasks

  • SCUBA contains 300 test cases sourced from interviews with Salesforce users, covering platform administrators, sales roles, and service roles.Each use case is associated with a Salesforce Trailhead knowledge article.Source: §3.2 · p. 4

3.2.1 Core Abilities

  • The tasks test enterprise UI navigation, information retrieval and textual reasoning, Salesforce data manipulation, workflow building and execution, and troubleshooting Q&A.Examples include configuring a territory model, updating Opportunity fields, and answering questions grounded in org settings.Source: §3.2.1 · p. 4

3.2.2 Task Construction Details

  • Task construction starts from user-research workflows, pairing each template with a Trailhead article and assigning difficulty based on UI-action complexity and required abilities.Templates are instantiated into five queries with varied values, paraphrased for language diversity, and manually checked for wording accuracy and ethics compliance.Source: §3.2.2 · pp. 4–5
  • Task initialization prepares required data and settings, and manually crafted API-based evaluators return binary success and milestone scores with rubrics.Figure 3 depicts task configuration and evaluation; the difficulty distribution is also shown by split.Source: §3.2.2 · p. 5, Figure 3 · p. 5
  • Human annotation checks that tasks are valid and completable and that evaluator results match human expectations.Annotated trajectories may also be used to improve agent performance.Source: §3.2.2 · p. 5

4 Experiments

SCUBA experiments compare browser-use and computer-use agents across zero-shot and demonstration-augmented settings, using distinct observation/action spaces and measures of accuracy, efficiency, and cost. Demonstrations generally improved task success, while results also expose trade-offs between agent paradigms and persistent computer-use grounding challenges.

4.1 Setups

  • Browser-use agents observe tagged browser screenshots and DOM text, while computer-use agents receive full-desktop screenshots; their differing action spaces are mapped to a unified space.The tested browser-use backbones are GPT-5, o3, Claude-4-sonnet, and Gemini-2.5-pro; computer-use agents are selected from several design categories.Source: §4.1 · p. 6

4.2 Implementation Details

  • SCUBA supports asynchronous parallel evaluation by assigning browser tasks to dedicated browsers and computer-use tasks to available Docker containers reset to fresh snapshots.Containers are reused as tasks become available, supporting resource utilization when the task count exceeds the container count.Source: §4.2 · pp. 6–7
  • Evaluation allows up to 50 prediction steps and 90 minutes per task, and records milestone score, task success, time, steps, costs, and token use.The step metric does not distinguish predictions that produce multiple actions from those producing one action.Source: §4.2 · p. 7

4.3 Main Results

  • Human demonstrations generally improve task success while reducing time, steps, token consumption, and costs, though some computer-use agents see increases in other metrics.UI-TARS-1.5-7B and OpenCUA-7B are exceptions; Gemini-2.5-Pro also shows increased steps and costs, attributed to shortcuts absent from its demonstration.Source: §4.3 · p. 7
  • Demonstration benefits vary by agent, and task-success gains may be limited when zero-shot success is already high.Demonstrations can also increase latency or costs when they are inefficient or agents could take shortcuts in zero-shot runs.Source: §4.3 · p. 8
  • Browser-use agents have higher task success but rely on Salesforce-specific DOM customization; Gemini-2.5-pro with demonstrations balances success, latency, and costs.The authors also report close Claude-4-sonnet performance under browser-use and computer-use settings.Source: §4.3 · p. 9

5 Conclusion

SCUBA evaluates computer-use agents on realistic Salesforce CRM tasks and exposes gaps across agent designs and model types. Demonstrations improve success rates while reducing task time and cost.

5 Conclusion

  • Figure 6 compares performance on OSWorld (50 steps) with SCUBA without demonstrations.Its caption identifies the comparison as a performance drop but gives no numeric values.Source: Figure 6 · p. 9
  • Demonstrations improve task success rates while simultaneously reducing task time and cost.The conclusion describes these improvements without specifying their magnitudes in this passage.Source: §5 · p. 9
  • Browser-use agents’ observation and action space designs can better leverage foundation VLMs and outperform specialized computer-use agents at the current stage.Source: §5 · p. 9
  • A substantial performance gap exists between current open-source and closed-source models.Source: §5 · p. 9

A Agent Observation Space

The appendix illustrates browser-use observations using SOM and DOM text and shows screenshots and Salesforce Setup interface content.

A Agent Observation Space

B.1 Browser-Use Agents

The browser-use agents are implemented with the browser-use framework, with modifications to its DOM parser; their action space is documented in Table 4.

B.1 Browser-Use Agents

B.2 Computer-Use Agents

The computer-use agent implementation extends OSWorld’s implementation and retains its action space, which Table 5 repeats for completeness.

B.2 Computer-Use Agents

  • The computer-use agent implementation builds upon and extends OSWorld’s implementation.Source: §B.2 · p. 13
  • The computer-use agents use the same action space as OSWorld.Source: §B.2 · p. 13
  • Table 5 repeats the computer-use agents’ action space for completeness.Source: Table 5 · p. 13, §B.2 · p. 13

C Construction of human-demonstration

The section illustrates a successful queue-creation trajectory and its corresponding plan, then describes how demonstrations are selected to test agents’ use of exact and similar examples.

C Construction of human-demonstration

  • A recorded successful trajectory logs the sequence of UI actions used to create the Shoe Case Support queue, from login through saving.The listed actions include entering the queue label and email, adding Customer, and selecting the queue-member role.Source: C Construction of human-demonstration · p. 14
  • The successful queue-creation plan covers setup navigation, queue details, supported objects, members, and saving.The example queue is labeled “Shoe Case Support” and includes the Customer object and a Customer Support, North America role.Source: C Construction of human-demonstration · p. 14
  • Two samples use exact demonstrations and three use similar demonstrations to test whether agents can follow an exact example or transfer from a similar one.The evaluation uses two crafted examples for each task template.Source: C Construction of human-demonstration · p. 14

D Metrics breakdown for the demonstration-augmented setting

Demonstration-augmented results vary with whether an exact or similar example is available and with the agent’s ability to use demonstrations. The appendix also describes the annotation qualification and review process.

D Metrics breakdown for the demonstration-augmented setting

E Additional details on the annotation process

  • Annotators pass onboarding tests, learn Salesforce and the annotation tools, complete tasks using prescribed actions, and submit trajectories for review.Rejected trajectories are sent to different annotators for rework.Source: E Additional details on the annotation… · p. 15

F Qualitative Analysis

The analysis identifies grounding errors and history management as persistent challenges for computer-use agents, while describing reflection as a recovery mechanism. It also notes that demonstrations may prescribe less efficient routes than agents find without them.

F Qualitative Analysis

  • Computer-use grounding can miss controls: Claude-4-sonnet clicked the “Add” text rather than its button, while element-index selection succeeded.The authors suggest making multiple coordinate predictions and combining them by voting or weighted averaging.Source: F Qualitative Analysis · p. 15, Figure 8 · p. 16
  • SOM is not a complete fix: the browser-use DOM parser missed the Add button in Figure 9(b), and universal parsers across websites are challenging.Source: F Qualitative Analysis · p. 16, Figure 9 · p. 16
  • Because history concatenates actions with the latest screenshot, a failed Add click can be read as success; reflection checks prior goal completion and recovered this task.Figure 10 illustrates incorrect progress tracking.Source: F Qualitative Analysis · pp. 16–17, Figure 10 · p. 16
  • Demonstrations can add latency or cost, or replace a zero-shot shortcut: the Reports task used a longer App Launcher search path learned from Trailhead.The figure caption notes that agents take different actions in zero-shot and demonstration-augmented settings.Source: F Qualitative Analysis · p. 17, Figure 11 · p. 17

G Budget-Constrained Test Setting

The budget-constrained tests examine method performance as available budgets change. Success rates rise with larger budgets, while average cost under the default step limit remains below $1.

G Budget-Constrained Test Setting

Loading 2509.26506v1…