PaperlayerView exact revision on arXiv

Source-linked AI summary

ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents

Xuhao Hu, Xi Zhang, Haiyang Xu, Kyle Qiao, Jingyi Yang, Xuanjing Huang, Jing Shao, Ming Yan, Jieping Ye

arXiv:2605.12481v1cs.AI

TL;DR

Computer-use agents struggle to choose efficiently between GUI actions and tool calls because high-quality interleaved training trajectories and trajectory-level supervision are scarce. ToolCUA scales hybrid trajectories from GUI data and trains switching decisions with reinforcement learning, achieving 46.85% OSWorld-MCP accuracy, about 66% above baseline and 3.9% above GUI-only settings.

  • Problem

    Agents struggle to select efficient, reliable GUI-Tool paths, while high-quality interleaved trajectories and tool-calling supervision remain scarce.

  • Method

    ToolCUA synthesizes grounded tools and interleaved trajectories from GUI corpora, then optimizes GUI-Tool switching through staged reinforcement learning.

  • Results

    46.85% accuracy on OSWorld-MCP establishes state-of-the-art performance among similar-size models, with a 3.9% improvement over pure GUI settings.

  • Takeaways & Limitations

    The results suggest that hybrid GUI-Tool training is a promising paradigm for computer-use agents requiring effective path orchestration.

  • Takeaways & Limitations

    Synthesized trajectory diversity and quality remain constrained by the coverage and fidelity of source GUI demonstrations and the synthesis model's capability.

Abstract

from arXiv · show

Computer Use Agents (CUAs) can act through both atomic GUI actions, such as click and type, and high-level tool calls, such as API-based file operations, but this hybrid action space often leaves them uncertain about when to continue with GUI actions or switch to tools, leading to suboptimal execution paths. This difficulty stems from the scarcity of high-quality interleaved GUI-Tool trajectories, the cost and brittleness of collecting real tool trajectories, and the lack of trajectory-level supervision for GUI-Tool path selection. In this paper, we propose ToolCUA, an end-to-end agent designed to learn optimal GUI-Tool path selection through a staged training paradigm. We first introduce an Interleaved GUI-Tool Trajectory Scaling Pipeline that repurposes abundant static GUI trajectories and synthesizes a grounded tool library, enabling diverse GUI-Tool trajectories without manual engineering or real tool-trajectory collection. We then perform Tool-Bootstrapped GUI RFT, combining warmup SFT with single-turn RL to improve decisions at critical GUI-Tool switching points. Finally, we optimize ToolCUA with Online Agentic RL in a high-fidelity GUI-Tool environment, guided by a Tool-Efficient Path Reward that encourages appropriate tool use and shorter execution paths. Experiments on OSWorld-MCP show that ToolCUA achieves 46.85% accuracy, a relative improvement of approximately 66% over the baseline, establishing a new state of the art among models of comparable scale. It also improves by 3.9% over GUI-only settings, demonstrating effective GUI-Tool orchestration. The results further suggest that training in a hybrid action space is a promising paradigm for real-world digital agents. Open-sourced here: https://x-plug.github.io/ToolCUA/

1. Introduction

ToolCUA addresses confusion in hybrid GUI-Tool action spaces by scaling interleaved training data from GUI corpora and optimizing trajectory-level orchestration through staged reinforcement learning. It achieves 46.85% on OSWorld-MCP [12], approximately 66% above Qwen3-VL-8B-Instruct, while improving hybrid-action performance over pure GUI actions.

  • Motivation: Hybrid action spaces can confuse agents: Qwen3VL-235B-A22B averages 6.10 tool-calls while success falls from 41.14% to 38.14%, whereas Qwen3VL-8B averages 0.003 tool calls.These behaviors reflect overuse and underuse of tools rather than effective orchestration.
  • Method: ToolCUA repurposes existing pure GUI corpora into scalable interleaved GUI-Tool data by synthesizing trajectory-aware tools, avoiding manual environment construction and tool-trajectory collection.This pipeline addresses the scarcity and cost of high-quality interleaved GUI-Tool trajectories.
  • Method: Its staged training combines tool-bootstrapped RFT for hybrid-action foundations and GUI-Tool switching optimization with online agentic RL using tool-efficient path rewards.The paradigm targets trajectory-level decisions rather than only local imitation or final task-completion rewards.
  • Results: 46.85% accuracy on OSWorld-MCP [12] establishes ToolCUA as state of the art among similar-size models, approximately 66% above Qwen3-VL-8B-Instruct.The result also rivals leading proprietary models.
  • Results: +3.9% improvement over pure GUI actions demonstrates that hybrid-action training improves ToolCUA’s performance.ToolCUA reaches 42.9% accuracy even in pure GUI action settings.

2. ToolCUA

ToolCUA frames computer use as decision-making over a hybrid GUI-and-tool action space and trains path orchestration through scaled interleaved trajectories, targeted switching optimization, and online reward shaping. Its pipeline combines grounded tool synthesis, critical-step calibration, and trajectory-level optimization for appropriate and efficient tool use.

  • 2. ToolCUA: ToolCUA models computer use as an MDP with multimodal GUI observations and a hybrid action space combining atomic GUI interactions with structured tool invocations.The state includes desktop screenshots and previously invoked tool results, while actions come from A_GUI ∪ A_Tool.
  • 2. ToolCUA: The interleaved trajectory scaling pipeline transforms successful GUI-only trajectories into diverse GUI-Tool data using grounded tools across functionality, granularity, and switching contexts.It filters and balances source trajectories, synthesizes callable tools from observed GUI procedures, and varies tool replacement with partial tool availability to require GUI fallback.
  • 2. ToolCUA: Tool-Bootstrapped GUI RFT first applies warmup SFT on all interleaved data, then uses GRPO on critical switching steps to calibrate GUI-versus-tool decisions.The SFT phase teaches tool usage, parameters, and resulting states, while single-turn RL provides feedback at decision boundaries.
  • 2. ToolCUA: Online Agentic RL optimizes complete GUI-Tool trajectories with a Tool-Efficient Path Reward that combines standard success signals with tool appropriateness and path efficiency.The tool reward favors calls on tool-beneficial tasks and abstention otherwise, while the length reward compares trajectories with rollout-group averages and rewards shorter successful paths.
  • 2. ToolCUA: ToolCUA is optimized with multi-turn GRPO in an online GUI-Tool environment, retaining rollout groups containing both successful and failed trajectories for informative updates.Dynamic filtering reduces unnecessary computation while preserving relative-policy learning signals.

3. Experiments

ToolCUA achieves state-of-the-art OSWorld-MCP performance by learning efficient GUI-Tool path selection, with gains over GUI-only training and generalization across tasks and platforms. Ablations show that interleaved trajectory supervision and the Tool-Efficient Path Reward are both important for reliable, efficient tool use.

  • Overall Results: 46.85% accuracy makes ToolCUA-8B state of the art among 8B-class models on OSWorld-MCP, surpassing GUI-Owl-1.5-8B at 43.84% and Claude-4-Sonnet at 43.54%.It also exceeds Gemini-3.1-Pro at 41.14% and trails Claude-4.5-Sonnet by less than 2%.
  • Overall Results: 24.32% TIR and 14.93 average completion steps improve over the baseline’s 8.41% and 19.34, respectively, with ToolCUA achieving the fewest steps overall.These results indicate more appropriate tool invocation and shorter GUI-Tool execution paths.
  • Generalization: 23.9% accuracy on held-out multi_apps tasks improves over 18.5% before online RL and 9.8% for the baseline, while WindowsAgentArena reaches 33.8%.On WindowsAgentArena, ToolCUA surpasses the baseline by 7.4 percentage points and exceeds Qwen3-VL-235B-A22B at 32.1%.
  • Ablation Studies: Without interleaved data, TIR reaches only around 15% and tool calls remain near zero, showing that online RL with path rewards alone cannot overcome GUI-centric bias.Offline synthetic interleaved supervision is therefore necessary for reliable tool-calling behavior.
  • Ablation Studies: Without the Tool-Efficient Path Reward, accuracy becomes unstable and ends about 7 percentage points lower, while TIR, tool calls, and trajectory length lack consistent improvement.Vanilla multi-turn GRPO does not reliably teach when tool invocation is beneficial.
  • Hybrid GUI-Tool Training: 46.85% accuracy after full hybrid training exceeds 42.05% after pure GUI agentic RL, while interleaved-trajectory RFT alone reaches 38.13%.Pure GUI training improves the baseline from 29.03% to 34.93% after SFT and then 42.05% after agentic RL, but both stages remain below GUI-Tool counterparts.

4. Related Work

Prior CUA research centers on multi-agent and end-to-end multimodal agents, while hybrid GUI-tool action spaces remain insufficiently addressed. ToolCUA extends this work with staged training for complete GUI-Tool execution paths.

  • Multimodal Agents for Computer Use: CUAs generally follow multi-agent systems that decompose tasks across specialized modules or end-to-end models that unify planning and grounding.Most existing agents still rely primarily on GUI interaction and do not directly address hybrid action spaces combining raw GUI actions with structured tool invocation.
  • Hybrid GUI-Tool Actions for CUAs: ToolCUA uses staged training to move beyond GUI-only control and optimize complete GUI-Tool execution paths.This approach builds on emerging hybrid-action CUA research that combines raw GUI interaction with structured tool invocation.
  • Hybrid GUI-Tool Actions for CUAs: Structured tools can reduce repetitive low-level GUI operations and improve efficiency when reliable interfaces are available.Tool-augmented LLM research has advanced autonomous coding, deep research, and multimodal visual search, while recent CUA studies explore MCP tools for hybrid-action evaluation.

5. Conclusion · Appendix

ToolCUA is presented as an end-to-end agent for orchestrating GUI and tool actions through learned GUI-Tool path selection. The paper argues that exposing agents to both action types alone is insufficient because models may overuse tools or remain overly GUI-centric, producing inefficient and brittle trajectories.

  • 5. Conclusion: ToolCUA orchestrates GUI and tool actions as an end-to-end computer-use agent.
  • 5. Conclusion: The system learns GUI-Tool path selection rather than relying only on access to both action types.
  • 5. Conclusion: Simply exposing agents to GUI actions and tool calls is insufficient for effective orchestration.
  • 5. Conclusion: Current models may overuse tools when operating in the hybrid action space.
  • 5. Conclusion: Current models may instead remain overly GUI-centric when choosing actions.
  • 5. Conclusion: These mismatched action preferences lead to inefficient and brittle execution trajectories.

A. Limitations and Future Works · B. Broader Impact and Ethics Statement

ToolCUA’s synthesis-driven training is limited by the coverage and fidelity of its source GUI trajectories, while broader deployment raises both practical and ethical considerations. Future work should extend hybrid action spaces across platforms and develop lighter, more robust training environments.

  • A. Limitations and Future Works: ToolCUA’s synthesized hybrid trajectories remain constrained by the state frames and domain coverage of the original GUI-only demonstrations.Their diversity and quality therefore depend on the breadth, fidelity, and task distribution of the source trajectories.
  • A. Limitations and Future Works: Future research should examine hybrid GUI-Tool action spaces across desktop, mobile, and web environments.The appropriate balance between atomic GUI operations and high-level tools may vary across interface structures and task types.
  • A. Limitations and Future Works: Future work should reduce agentic RL’s dependence on heavy sandbox infrastructure through lighter, more diverse, and more robust hybrid-action environments.These environments should support both GUI actions and tool calls.
  • B. Broader Impact and Ethics Statement: ToolCUA could benefit real-world digital automation by coordinating GUI actions and tool calls more efficiently.Potential benefits include productivity, accessibility, and assistance with repetitive workflows.
  • B. Broader Impact and Ethics Statement: More capable computer-use agents also create risks involving unauthorized operation, accidental user-data modification, and privacy leakage from desktop observations.These risks arise alongside the system’s potential benefits for digital automation.
  • B. Broader Impact and Ethics Statement: Computer-use agents could be misused to automate harmful or deceptive online activities.This risk is explicitly identified as part of the paper’s broader-impact and ethics discussion.

C. Implementation Details · C.1. Preliminary study: Optimal Path Confusion in Hybrid Action Spaces

Table 1 shows that adding tools to GUI-capable agents does not reliably improve performance: hybrid action spaces can confuse agents into choosing suboptimal execution paths. The diagnostic identifies tool underuse, tool overuse, and trajectory-level path selection as central challenges.

  • C.1. Preliminary study: Optimal Path Confusion in Hybrid Action Spaces: The diagnostic compares pure GUI and hybrid GUI-Tool execution across multiple agents, with hybrid models receiving the same tool interface and documentation through the system prompt.Accuracy is measured over full tasks, while ACS reports average completion trajectory steps as an execution-efficiency measure.
  • C.1. Preliminary study: Optimal Path Confusion in Hybrid Action Spaces: The evaluation uses official verified GUI trajectories for EvoCUA-32B and Claude baselines, a 50-step maximum horizon, feasible tasks only, and aligned implementations for Qwen3VL models.Qwen3-VL-8B-Instruct and Qwen3-VL-235B-A22B-Thinking are abbreviated as Qwen3VL-8B and Qwen3VL-235B in Table 1.
  • C.1. Preliminary study: Optimal Path Confusion in Hybrid Action Spaces: Hybrid GUI-Tool execution often fails to improve accuracy because agents deviate from the most effective execution path when both action spaces are available.The study frames this as a “forked road” problem in which access to both GUI actions and tools can confuse otherwise strong models.
  • C.1. Preliminary study: Optimal Path Confusion in Hybrid Action Spaces: 29.0% to 28.2%: Qwen3VL-8B’s accuracy drops after tool access, while Tool-Calls remain 0.00 per trajectory and ACS rises from 19.2 to 19.3.This exemplifies tool underuse: the model remains almost entirely on the GUI branch despite the availability of a short tool call.
  • C.1. Preliminary study: Optimal Path Confusion in Hybrid Action Spaces: 41.1% to 38.1%: Qwen3VL-235B’s accuracy falls despite Tool-Calls increasing to 6.10 and completion steps decreasing from 25.9 to 17.4.The shorter trajectory does not translate into better task success, illustrating that frequent tool usage alone is insufficient.
  • C.1. Preliminary study: Optimal Path Confusion in Hybrid Action Spaces: Tool overuse is also observed in EvoCUA-32B, which invokes tools 7.49 times on average without the supplied passage establishing improved task success.This supports the broader finding that aggressive tool invocation is not itself a reliable solution.
  • C.1. Preliminary study: Optimal Path Confusion in Hybrid Action Spaces: The core difficulty is optimal GUI-Tool path selection rather than merely expanding the available action space.Agents must decide whether to continue GUI grounding, invoke a tool immediately, or switch after additional GUI setup.

C.2. Data Statistics of the Scaling Pipeline

The scaling pipeline combines diverse GUI trajectories with sandbox rollouts to synthesize 10,000 interleaved GUI-Tool trajectories, 180k SFT steps, and 5k critical switching steps. Its 4,350-tool inventory spans multiple granularities and application domains, supporting diverse tool-selection decisions.

  • Data volume: 10,000 synthesized interleaved GUI-Tool trajectories yield 180k warmup SFT steps and 5k critical switching steps for offline single-turn RL.The source data combines OpenCUA [40], ScaleCUA [18], and 1,200 curated sandbox-rollout trajectories after filtering.
  • Tool inventory: 4,350 unique tools span fine-, mid-, and coarse-grained operations, with each trajectory averaging 19.75 candidate tools and 7.89 executed tools.The inventory is hierarchically structured and primarily anchored by fine- and mid-grained operations, complemented by coarse-grained skills.
  • Tool-space diversity: Figure 7 shows distinguishable tool regions across LibreOffice, Chrome, and VSCode, with all three granularity tiers coexisting within regions.The projected tool space indicates that the synthesized inventory does not collapse into a narrow set of routines.

C.3. Training Details

ToolCUA uses staged training that combines full-parameter warmup SFT, offline single-turn RL, and online agentic RL with explicit tool-appropriateness supervision. Training scales through distributed GUI-tool rollouts in an OSWorld-based environment with MCP integrations.

  • Tool-Bootstrapped GUI RFT: Warmup SFT fine-tunes Qwen3-VL-8B-Instruct for 3 epochs across the vision tower and LLM backbone, using 8 × 8 GPUs.Offline single-turn RL then uses rollout group size 32, learning rate 1 × 10^-6, and batch size 128.
  • Online Agentic RL Training: Online agentic RL uses λ = 0.4, β = 0.2, and Smax = 30, with rollout size 32, learning rate 1 × 10^-6, and batch size 32.These settings produce the final ToolCUA model and cap the maximum execution horizon at 30.
  • Online Agentic RL Training: The Tool Appropriateness Reward uses OSWorld-MCP [12] task labels indicating whether tool invocation is beneficial, with manual verification before reward integration.Labels are represented as tb ∈{1, −1}.
  • CUA Agentic RL in a GUI-Tool Environment: Distributed training separates policy optimization on GPU clusters from rollouts on ECS servers using verl [31] and an OSWorld [45] QEMU sandbox with OSWorld-MCP [12] and AutoGLM [17] tools.The tool-calling interface is further optimized with an agent-readable return design.

C.4. Benchmark Evaluation

ToolCUA is evaluated primarily on OSWorld-MCP [12], a hybrid GUI-Tool benchmark, using accuracy, tool-use alignment, and execution efficiency metrics. Cross-platform transfer is assessed on WindowsAgentArena [5] under a comparable 50-step limit.

  • C.4. Benchmark Evaluation: OSWorld-MCP [12] directly measures hybrid GUI-Tool execution through task accuracy, Tool Invocation Rate, and Average Completion Steps.Results use average@3 evaluation with a maximum of 50 execution steps per task.
  • C.4. Benchmark Evaluation: Task accuracy measures successful completion of the target instruction, including visual grounding, GUI interaction, and tool use.It is the primary success metric in OSWorld-MCP.
  • C.4. Benchmark Evaluation: Tool Invocation Rate measures whether agents use tools on Tool-Beneficial Tasks and avoid them on Non-Tool-Beneficial Tasks.It captures alignment between tool usage and task-level utility rather than invocation frequency alone.
  • C.4. Benchmark Evaluation: Average Completion Steps measures execution efficiency through the average number of environment interaction steps required across tasks.Shorter tool-conditioned paths and fewer redundant GUI operations generally reduce ACS.
  • C.4. Benchmark Evaluation: WindowsAgentArena [5] evaluates cross-platform transfer using accuracy@avg3 as the main metric with max_steps=50 for each task.This evaluation tests ToolCUA beyond OSWorld-MCP.

C.5. Ablation Details

Three ablation studies assess the staged training components and the hybrid GUI-Tool action space. The studies compare offline interleaved-data bootstrapping, online agentic RL, and pure-GUI training under controlled protocols.

  • C.5. Ablation Details: Three ablation studies evaluate each training component and the effect of the hybrid GUI-Tool action space.The supplied passage identifies the overall ablation scope but does not provide the corresponding result values.
  • Staged training ablations: Figure 6 examines staged training by removing offline interleaved GUI-Tool data bootstrapping and directly applying online agentic RL with the Tool-Efficient Path Reward.This ablation uses Qwen3-VL-8B-Instruct as the baseline and tests whether online exploration alone can learn tool calling and GUI-Tool switching.
  • Action-space ablation: Table 3 compares hybrid GUI-Tool training with pure-GUI training using comparable-scale fine-tuning data and online agentic RL in the corresponding action space.In the pure-GUI setting, tool actions are unavailable and the reward comprises Racc + R_fmt + Rlength, including the Path Efficiency Reward Term.
  • Evaluation protocol: All online agentic RL ablations report average@3 results with max_steps=30, roughly 25 optimization steps, batch size 32, dynamic filtering, and approximately 1,200 effective samples per run.These settings follow Section C.4 and reflect the computational constraints of sandbox-based online RL.

D. More Case Study … Message Construction

The case studies illustrate how ToolCUA combines semantic tool calls with GUI actions across office, coding, and multi-application tasks, while the data pipeline and message construction provide grounded trajectory generation, tool synthesis, state prediction, and multimodal execution context.

  • D. More Case Study: ToolCUA combines direct tool execution with GUI actions to complete LibreOffice Calc, VS Code, multi-application, and Impress tasks efficiently while handling interface states tools cannot fully cover.The cases include pivot-table creation, workspace-folder management, terminal screenshot capture, and slide text-color changes; GUI actions remain useful for dialogs, confirmations, and finalization.
  • E. Prompts Used in Data Pipeline; SCREENSHOT DESCRIPTION PROMPT; JOINT GENERATION PROMPT; TOOL GENERATION PROMPT; FIX TOOL PROMPT; PREDICT SCREENSHOT PROMPT; DESCRIBE AND LOCATE PROMPT; MERGE TREE PLANNING PROMPT; BOTTOM UP MERGE PROMPT: The offline trajectory-scaling pipeline uses screenshot descriptions, grounded tool generation and repair, joint step generation, screenshot prediction and matching, and merge-tree planning.Its prompts require semantic tools grounded in visible state transitions, valid schemas, one-step trajectories, and coherent multi-granularity merges.
  • JOINT GENERATION PROMPT; TOOL GENERATION PROMPT; FIX TOOL PROMPT: The joint-generation prompt produces exactly one grounded step containing an observation, thought, action, tool call, and tool response, while constraining progress, valid schemas, and non-invented effects.Tool-generation instructions further require application-grounded semantic tools, fine and coarse granularity, standardized return fields, and exactly one terminate tool.
  • PREDICT SCREENSHOT PROMPT; DESCRIBE AND LOCATE PROMPT: The prediction and matching prompts reconstruct post-action desktop states from tool results and prior screenshots, selecting the best candidate or returning null when no candidate matches.They require concrete descriptions of visible windows, dialogs, elements, data, and status changes, with confidence and evidence for the selected match.
  • MERGE TREE PLANNING PROMPT; BOTTOM UP MERGE PROMPT: The merge-tree and bottom-up merge prompts preserve leaf order while synthesizing semantically cohesive coarse tools that summarize complete grounded chunks without inventing effects.Merged tools must be broader than their constituent actions, use coarse granularity, and remain tied to the final UI state reached by the chunk.
  • F. Messages for ToolCUA; System Prompt: ToolCUA’s system prompt always supplies GUI-action specifications and conditionally appends MCP tools and reminders, distinguishing screenshot feedback from textual JSON tool results.The hybrid prompt warns that some MCP actions may not visibly change the screenshot, requires GUI screenshots for state decisions, and discourages repeated failing calls.
  • Message Construction: Message construction retains up to five prior screenshots, tool results, actions, and assistant responses, then appends the current instruction and screenshot in a multimodal user message.At the initial step it provides the current screenshot directly; later steps include the current tool result and screenshot within tool-response delimiters.
Loading 2605.12481v1…