PaperlayerView exact revision on arXiv

Source-linked AI summary

HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use Agents

Tongbo Chen, Junbo Niu, Zhengxi Lu, Niu Lian, Fei Tang, Yuchen Yan, Yike Hong, Yong Du, Yizhou Liu, Bofan Chen, Yongliang Shen

arXiv:2609.38008v1cs.CV

TL;DR

GUI-based computer-use agents can be slow and error-prone, while application-specific tools sacrifice generality; existing agents also lack reliable guidance on when and how to use the CLI. HybridCUA builds GUI-only, CLI-only, and interleaved trajectories and trains with CLI-aware rewards. HybridCUA-9B reaches 53.6% OSWorld accuracy, 14.8 percentage points above its base model, and improves on WindowsAgentArena.

  • Problem

    Existing agents can lack guidance on when and how to use the CLI, while GUI-only interaction can be inefficient and application-specific tools are difficult to scale across applications.

    Source: §1 · p. 2

  • Method

    HybridCUA constructs GUI-only, CLI-only, and interleaved trajectories plus verified RLVR tasks, then applies supervised fine-tuning and reinforcement learning with CLI-aware rewards.

    Source: §1 · pp. 2–3

  • Results

    53.6% OSWorld accuracy, 14.8 percentage points above the base model; WindowsAgentArena performance improves by 4.0 percentage points.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results support GUI–CLI orchestration as a promising direction for accurate, efficient, and generalizable computer-use agents across benchmarks.

    Source: §5 · p. 9

    Benefits depend on available, stable CLIs, and OSWorld-focused data and evaluated benchmarks do not cover the full diversity of applications, operating systems, permissions, or long-horizon workflows.

    Source: A LIMITATIONS · p. 13

Abstract

from arXiv · show

Computer use agents (CUAs) have demonstrated strong capabilities in completing digital tasks. However, existing CUAs either rely solely on graphical user interface (GUI) interactions, which are often inefficient and error prone, or augment GUI interactions with application specific APIs or tools, which require substantial engineering effort and are difficult to scale across applications. We argue that the next generation of CUAs should combine GUI interactions with the command line interface (CLI), leveraging the generality of the GUI and the efficiency of shell commands. A critical challenge, however, is that current models do not know when or how to use the CLI during task execution. To address this challenge, we develop a data construction pipeline that produces three types of trajectories: GUI only, CLI only, and interleaved GUI and CLI trajectories. This pipeline results in HybridCUA-8K, containing 5K hybrid trajectories and 3K verified RLVR tasks. Building on these data, we propose a training framework with two stages: supervised fine tuning on the constructed trajectories, followed by reinforcement learning with our CLI aware rewards that encourages agents to use the CLI selectively and reliably. Experiments show that HybridCUA-9B achieves 53.6% accuracy on OSWorld, improving over the base model by 14.8 percentage points, and improves performance on WindowsAgentArena by 4.0 percentage points. These results demonstrate the effectiveness and cross platform generalizability of the hybrid GUI and CLI paradigm for computer use agents.

1 INTRODUCTION

HybridCUA addresses the challenge of combining GUI generality with CLI efficiency: existing agents often do not know when or how to use the shell. It pairs hybrid trajectories with CLI-aware reinforcement learning and reports gains on OSWorld and WindowsAgentArena.

1 INTRODUCTION

  • GUI actions are general but can be slow and error-prone, while application-specific APIs can improve efficiency at the cost of portability.The paper proposes pairing GUI generality with the efficiency of shell commands.Source: §1 · p. 2
  • Exposing a CLI alone can reduce OSWorld accuracy; the identified bottleneck is knowing when and how to use it, not shell access itself.The four representative agents lose 2.5 to 11.5 percentage points when given CLI access.Source: §1 · p. 2
  • HybridCUA constructs 5,000 GUI-only, CLI-only, and interleaved trajectories, using existing GUI sequences that can be rewritten as equivalent shell commands.It also constructs 3,000 verified RLVR tasks labeled by whether CLI use offers a clear execution advantage.Source: §1 · pp. 2–3
  • 53.6% OSWorld accuracy, 14.8 percentage points above the base model; HybridCUA-9B improves WindowsAgentArena performance by 4.0 percentage points.The OSWorld result is also the best among comparably sized models.Source: §1 · p. 3
  • Training uses supervised fine-tuning on trajectories followed by RLVR with CLI-aware rewards that separately target interface routing and command execution.The two reward signals address when to invoke the CLI and how to improve command reliability using terminal feedback.Source: §1 · p. 3

2 RELATED WORK

GUI-based agents operate across applications and platforms, but some tasks still require long sequences of atomic actions. Hybrid GUI–tool approaches can shorten repetitive interactions, while their portability is limited by application-specific interfaces.

2.1 GUI-BASED COMPUTER-USE AGENTS

  • GUI agents use screenshots and mouse or keyboard actions across applications, but file manipulation, text editing, and repeated operations may require long sequences despite concise programmatic alternatives.The paper notes that long GUI sequences can reduce efficiency and increase the risk of cascading errors.Source: §2.1 · p. 3

2.2 HYBRID COMPUTER USE

  • Hybrid tools can compress repetitive interactions, but application-specific interfaces bound their coverage and portability; HybridCUA instead uses executable verifiers and CLI-aware rewards for routing and command reliability.The cited approaches differ in supervision: CUA-Universe uses VLM scoring, while RecreationWorld supervises task outcomes without making interface choice explicit.Source: §2.2 · p. 3

3 HYBRIDCUA

HybridCUA formulates computer use with a unified GUI–CLI action space and constructs trajectories and verified tasks to train interface selection. It then combines supervised fine-tuning with online reinforcement learning and CLI-aware rewards.

3.1 HYBRID COMPUTER-USE FORMULATION

  • The agent’s observation includes the current screenshot and, when the preceding action was a direct CLI command, its stdout or stderr.The formulation treats computer use as partially observable.Source: §3.1 · p. 4
  • Executable interactions share one bash action format for direct shell commands and Python heredocs containing GUI code, alongside wait, terminate, and answer actions.The command may include an optional timeout.Source: §3.1 · p. 4

3.2 SCALABLE HYBRID DATA GENERATION PIPELINE

  • The data pipeline creates GUI-only trajectories from open-source demonstrations, successful CLI-only rollouts, and interleaved trajectories from free interface switching or replayed GUI-to-CLI rewrites.Rewritten segments are retained only when replay succeeds.Source: §3.2 · pp. 4–5
  • The pipeline constructs 3,000 verified tasks and labels whether CLI use offers a clear execution advantage by comparing success rates across GUI-only, CLI-only, and hybrid rollouts.Tasks require an executable verifier and a reachable target state; near-ties are resolved using step count.Source: §3.2 · p. 5

3.3 TWO STAGE TRAINING PARADIGM

  • Supervised fine-tuning combines all three trajectory types to teach GUI control, shell commands, and interface routing before online reinforcement learning in a live GUI–CLI environment.The trajectory types provide complementary supervision for those behaviors.Source: §3.3 · p. 5
  • Because task-completion rewards do not distinguish efficient interface choices or penalize failed CLI commands on ultimately successful tasks, CLI-aware rewards supervise routing and execution separately.The task-level signal rewards successful rollouts whose CLI use matches the task-specific label; terminal feedback supplies localized command-execution supervision.Source: §3.3 · p. 5
  • The execution signal penalizes shell-level command failures, while the routing signal is incorporated at trajectory level and execution feedback is assigned at action level.The two CLI-aware signals operate at complementary granularities.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

HybridCUA-9B outperforms comparably sized models on OSWorld, and ablations and transfer tests examine its data, rewards, interface choices, and cross-environment performance.

4.1 EXPERIMENTAL SETTINGS

  • The evaluation uses OSWorld as its primary benchmark and tests transfer on OSWorld-MCP and WindowsAgentArena, with a 50-step maximum per task.The study initializes from Qwen3.5-9B and uses supervised warm-up followed by online agentic RL.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • 53.6% accuracy with 14.0 average steps makes HybridCUA-9B best on both metrics among comparably sized models; it also exceeds GUI–API baselines by 4.7–9.9 percentage points.The matched GUI–CLI branch also outperforms GUI-only after SFT and RL, with fewer average steps at both stages.Source: §4.2 · p. 7

4.3 ABLATION STUDY

  • Mixing all three trajectory types yields 46.0% accuracy, 2.8, 14.3, and 5.0 percentage points above GUI-only, CLI-only, and hybrid-only SFT, respectively.Interleaved trajectories alone do not match the mixed corpus, suggesting single-interface demonstrations and hybrid trajectories provide complementary supervision.Source: §4.3 · p. 7, Table 2 · p. 7
  • Removing task-level RCLI limits CLI use to 58.9% versus 64.0%; removing step-level rexec raises step-120 errors to 16.5% versus 11.5%.The task-level reward governs when to use the CLI, while the step-level reward governs command reliability.Source: §4.3 · p. 7, Figure 4 · p. 7

4.4 OUT-OF-DISTRIBUTION GENERALIZATION

  • HybridCUA-9B reaches 47.1% accuracy on OSWorld-MCP, 9.1 points above Qwen3.5-9B, and 36.0% on WindowsAgentArena, 4.0 points above it.It issues PowerShell commands despite training only on Linux shells.Source: §4.4 · p. 8, Table 3 · p. 8

4.5 ANALYSIS

  • CLI dominates OS tasks (84%), content editing (85.8%), and result verification (98.4%), while GUI dominates Chrome tasks (74%) and spatial adjustment (58.4%).The reported patterns associate shell use with direct data manipulation and state inspection, and GUI use with visual positioning and layout.Source: §4.5 · p. 8, Figure 5 · p. 8, Figure 6 · p. 9
  • In essay formatting, GUI steps focus the Writer window and open menus, while CLI steps edit formatting and verify the saved document.One hybrid step combines dismissing a menu through pyautogui with reopening the file through the CLI in a single bash call.Source: §4.5 · p. 8, Figure 7 · p. 9
  • The unified bash action schema achieves 46.0% accuracy with 19.8 average steps, 7.2 percentage points above the separate-tool schema at a cost of 3.0 additional steps.Both variants use the same base model and trajectories under matched settings.Source: §4.5 · p. 9

5 CONCLUSION

The conclusion presents HybridCUA as a framework for GUI–CLI orchestration, combining three trajectory types and verified RLVR tasks with supervised fine-tuning and CLI-aware online reinforcement learning.

5 CONCLUSION

  • HybridCUA constructs three trajectory types and verified RLVR tasks, then trains a unified policy with supervised fine-tuning and CLI-aware online reinforcement learning.The framework teaches agents when and how to combine GUI and CLI actions.Source: §5 · p. 9
  • Across multiple computer-use benchmarks, HybridCUA improves task completion and interaction efficiency over its base model while generalizing across interfaces and operating systems.Source: §5 · p. 9
  • The paper concludes that GUI and CLI provide complementary capabilities and that learning to orchestrate them is a promising direction for computer-use agents.Source: §5 · p. 9

AI USE STATEMENT

The authors describe limited supporting use of AI assistants and state that they reviewed outputs and retained responsibility for the paper’s content and results.

AI USE STATEMENT

  • AI assistants supported language polishing, LATEX formatting, coding, and debugging.Source: AI USE STATEMENT · p. 10
  • Large language models also supported data collection by sampling CLI-only and interleaved trajectories through an execution harness.Source: AI USE STATEMENT · p. 10
  • The authors reviewed AI-assisted outputs and made the final decisions about methodology, experiments, analyses, and presentation.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The study uses sandboxed virtual machines and excludes human participants and private or personally identifiable information. Real-world deployment safety is outside its scope.

ETHICS STATEMENT

  • The study involved no human participants and collected no private or personally identifiable information.The work focuses on hybrid GUI and CLI orchestration in benchmark computer environments.Source: ETHICS STATEMENT · p. 10
  • Trajectory collection, training, and evaluation run in resettable sandboxed virtual machines without real user accounts or credentials.Environments are reset between episodes.Source: ETHICS STATEMENT · p. 10
  • Real-world deployment safety is beyond scope and would require further evaluation, safeguards, and human oversight.Shell access can amplify what an agent does in a single action.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The paper documents its data, action protocol, training, and evaluation materials and plans to release the trajectories, tasks, pipelines, and model. Its limitations include CLI availability and coverage concentrated on OSWorld applications.

REPRODUCIBILITY STATEMENT

  • The planned release includes trajectories, RLVR tasks, data-generation and training pipelines, and the HybridCUA-9B model.The reproducibility statement points to appendices covering action protocol, trajectory construction, task labeling, statistics, hyperparameters, compute, benchmarks, and prompts.Source: REPRODUCIBILITY STATEMENT · p. 10

A LIMITATIONS

  • Transfer depends on usable, stable CLIs, while OSWorld-focused data and benchmark coverage leave broader applications and environments underrepresented.The stated gaps include operating-system-specific command semantics, permission settings, specialized applications, and out-of-distribution workflows.Source: A LIMITATIONS · p. 13

B ACTION SPACE AND INTERACTION PROTOCOL

  • PyAutoGUI interactions remain GUI actions even when serialized through the shared bash wrapper; direct shell commands are CLI actions.The wrapper standardizes action representation rather than changing the interface category.Source: B ACTION SPACE AND INTERACTION PROTOCOL · p. 13
  • At each decision step, the agent receives a screenshot and, after a direct CLI command, its prior stdout/stderr; execution returns a new screenshot and CLI output when available.Task instructions and interaction history also condition the next action.Source: B ACTION SPACE AND INTERACTION PROTOCOL · p. 13
  • The bash action accepts a direct shell command or a quoted Python heredoc; wait, terminate, and answer control settling, episode ending, and response submission.A model-generated completion declaration is distinct from the task verifier’s assessment.Source: B ACTION SPACE AND INTERACTION PROTOCOL · pp. 13–14

C DATASET CONSTRUCTION AND STATISTICS

  • HybridCUA-8K distinguishes 5,023 supervised execution trajectories from 3,000 verified RL tasks that can each generate multiple rollouts.A trajectory records one execution; an RL task specifies an environment and verifiable objective.Source: C DATASET CONSTRUCTION AND STATISTICS · p. 14

C.1 SUPERVISED FINE-TUNING DATA CONSTRUCTION

The pipeline constructs supervised trajectories across GUI-only, CLI-only, and interleaved modes, then labels verified tasks according to which mode offers a CLI advantage. Interleaved data combines interface switching with replay-checked GUI-to-CLI rewriting.

C.1 SUPERVISED FINE-TUNING DATA CONSTRUCTION

  • The SFT corpus contains 5,023 trajectories across 11 domains and three modalities, averaging 10.4 logical steps per trajectory.The median is 7 steps, the 90th percentile 22, and the maximum 50; Figure 8 summarizes domain, modality, and length composition.Source: §C.1 · p. 14, Figure 8 · p. 14
  • GUI-only trajectories from UI-MOPD are translated into equivalent PyAutoGUI calls, with predicted coordinates mapped from a shared 1000 × 1000 space to display coordinates.The calls are serialized as Python heredocs in the shared bash wrapper, without changing their GUI classification.Source: §C.1 · p. 14, Figure 9 · p. 16
  • CLI-only trajectories are sampled with application-specific skills and a Claude Code harness; the Impress example uses Python/UNO to edit and verify selected slide backgrounds.The example checks live UNO state against saved XML and confirms the other four slides remain unchanged.Source: §C.1 · p. 16, Figure 10 · p. 16
  • Interleaved trajectories come from free interface switching or semantics-preserving command rewriting, and rewritten trajectories are retained only after successful replay.Rewriting reconstructs whole terminal commands from GUI actions and preserves genuine GUI interactions; the example uses CLI for extension setup and saved-setting inspection, with GUI for theme selection.Source: §C.1 · pp. 16–17, Figure 11 · p. 17

C.2 RLVR TASK CONSTRUCTION AND CLI ADVANTAGE LABELING

  • For each of 3,000 verified tasks, 16 rollouts per mode are ranked by success rate, with close ties resolved using successful-rollout step counts.Residual ties favor a single-interface mode, then GUI-only; labeling retains every task.Source: §C.2 · pp. 17–18
  • A task is labeled CLI-advantaged when CLI-only ranks first, or when GUI–CLI ranks first and more than half of its successful rollouts use a direct CLI command.Otherwise its label is zero.Source: §C.2 · p. 18

D TRAINING DETAILS

Training uses supervised fine-tuning on successful trajectories followed by online reinforcement learning from verified tasks. The RL setup combines task success, CLI-preference rewards, and step-level execution feedback, with filtering and rollout controls.

D.1 SUPERVISED FINE-TUNING CONFIGURATION

  • SFT expands 5,023 successful trajectories into 52,227 step-level samples, retaining 46,876 after the 12,000-token filter.The current screenshot remains higher resolution for coordinate actions, while history is downsampled; only the final assistant turn contributes to supervised loss.Source: §D.1 · p. 18
  • The Qwen3.5-9B initialization is trained for two epochs with batch size 256, yielding 366 optimizer updates; the final checkpoint is reported without validation-based selection.Training uses two nodes with eight GPUs each.Source: §D.1 · p. 18

D.2 ONLINE REINFORCEMENT LEARNING CONFIGURATION

  • Online GRPO starts from the SFT checkpoint and trains on 1,000 sampled tasks from the 3,000 verified-task pool.Tasks use custom definitions, initial states, and success criteria in an OSWorld-compatible desktop framework.Source: §D.2 · p. 18
  • The reward combines binary task success with a success-gated CLI-preference reward, then adds an execution term after group normalization.The combined reward uses λCLI = 0.1, the execution term uses λexec = 0.3, and groups with negligible reward range are filtered before normalization.Source: §D.2 · p. 18
  • Step-level credit assignment broadcasts each trajectory’s group-relative advantage to its steps before adding local execution feedback, preventing repeated group counts for long trajectories.Aborted trajectories receive zero reward and are excluded from gradient computation; rollout groups over two policy updates stale are discarded.Source: §D.2 · p. 19

E EVALUATION PROTOCOLS AND METRICS

The evaluation uses separate benchmarks and reports task accuracy and average environment steps, with interface access and score aggregation defined as part of the protocol.

E EVALUATION PROTOCOLS AND METRICS

  • OSWorld covers 361 tasks; transfer is evaluated separately on 361 OSWorld-MCP tasks and 154 WindowsAgentArena tasks, with one run per model and benchmark.Each task has a maximum budget of 50 environment steps; OSWorld-MCP shares OSWorld's task set, while WindowsAgentArena changes the operating system.Source: E EVALUATION PROTOCOLS AND METRICS · p. 20
  • Acc. is the mean evaluator score, retaining fractional verifier scores rather than thresholding them; it is therefore an upper bound on strict all-or-nothing success.A model-generated terminate declaration does not contribute to the task score.Source: E EVALUATION PROTOCOLS AND METRICS · p. 21
  • Avg. Steps is the mean environment interactions across all tasks, not only successful ones; waits and termination count, and budget-exhausted tasks contribute 50 steps.One step is one environment interaction.Source: E EVALUATION PROTOCOLS AND METRICS · p. 21
  • Interface access is part of the evaluation configuration, and pooled CLI-step and CLI-error rates use executable steps while excluding control actions.A zero-denominator rate is undefined, and pooled step shares weight longer trajectories more heavily than an unweighted mean of per-task shares.Source: E EVALUATION PROTOCOLS AND METRICS · p. 21

F MORE CASE STUDIES

This appendix presents examples of agents combining GUI and CLI, along with the OSWorld evaluation setup and system-prompt configurations for their interaction. It also clarifies that domain-specific CLI skills guide teacher sampling only, not student training or evaluation.

F MORE CASE STUDIES

  • The Vim case scored 1.0: GUI browsing supplied instructions for absolute line numbers, then CLI editing appended set number to ~/.vimrc and read it back; no subsequent Vim launch is shown.The handoff made the setting persistent rather than applying it only for one session.Source: F MORE CASE STUDIES · p. 22
  • The VLC case scored 1.0: after GUI preferences exploration failed to establish the requested 200% maximum, CLI editing changed the backing configuration from 125 to 200, which the evaluator checked after relaunch.The rollout does not show the updated volume control; this fallback does not establish that VLC’s GUI cannot support the setting.Source: F MORE CASE STUDIES · p. 24
  • The terminal-screenshot case scored 1.0: CLI setup, GUI execution of ls /home/user, and CLI capture and inspection produced an 876 × 533 PNG, with evaluation checking for ls or an OCR variant.The final observation displayed the saved image; the evaluator did not check every directory entry.Source: F MORE CASE STUDIES · pp. 26, 28

G.1 OSWORLD EVALUATION: UNIFIED ACTION SCHEMA

  • OSWorld evaluation uses one bash action for direct shell or file operations and GUI control through PyAutoGUI in a quoted Python heredoc, with effects shown in the next screenshot.The prompt presents the latest screenshot and previous command output at each step.Source: §G.1 · p. 30, System Prompt: OSWorld Evaluation… · pp. 30–31

System Prompt: OSWorld Evaluation (Unified Bash)

  • OSWorld’s unified interface routes GUI interactions through pyautogui heredocs in bash and also supports shell commands, waiting, and termination.The prompt specifies quoted heredoc delimiters and uses a subsequent screenshot to observe GUI effects.Source: System Prompt: OSWorld Evaluation… · pp. 31–32

System Prompt: Separate GUI/CLI Tools

  • The separate-tool prompt assigns terminal and file operations to a CLI on the same machine and reserves GUI tools for actions that must be seen or clicked on screen.It directs the agent to consult screenshots for click coordinates and take successive screenshots to check changes.Source: System Prompt: Separate GUI/CLI Tools · pp. 33–34

G.3 TRAJECTORY CONSTRUCTION: DOMAIN CLI SKILLS

  • Eight domain-specific CLI skills guide teacher sampling during trajectory collection but are absent from student supervised context, RL rollouts, and evaluation prompts.The skills are reproduced as used for Qwen3.8-27B in CUA-Gym.Source: §G.3 · p. 35

G.2 ACTION-SCHEMA ANALYSIS: SEPARATE GUI AND CLI TOOLS

The separate-tools variant exposes GUI interaction and CLI operations through distinct tools, unlike HybridCUA's single bash action.

G.2 ACTION-SCHEMA ANALYSIS: SEPARATE GUI AND CLI TOOLS

  • The separate-tools variant uses one computer-use tool for GUI interaction and another CLI tool for terminal and file operations, while HybridCUA exposes both through bash.The unified bash action does not imply that GUI interactions themselves are CLI operations.Source: §G.2 · p. 33

Related papers

Loading 2609.38008v1…