Source-linked AI summary
CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents
Haoting Shi, Wenhao Wang, Weicheng Fang, Yaozhong Liang, Tian Jin, Pengxiang Zhao, Guangyi Liu, Siheng Chen, Yanfeng Wang
TL;DR
Computer-use agents need to combine GUI perception with efficient CLI operations, but scalable real-software environments for shared-state hybrid interaction remain limited. CUA-Universe builds such environments through application adaptation, task synthesis, and efficient trajectory harvesting. Its 9B model improves success and efficiency on CUA-Verse, OSWorld, and OSWorld-MCP, with transfer to an unseen tool interface.
Problem
Scalable hybrid environments for real applications are limited, while GUI-dominant agents can be inefficient and CLI-dominant agents lack visual perception for interface state and layout.
Method
CUA-Universe uses App-Forge to adapt applications and construct CLI surfaces, Task-Weave to synthesize seed-conditioned hybrid tasks, and Path-Steer to harvest efficient verified trajectories.
Results
The 9B model improves success and efficiency across CUA-Verse, OSWorld, and OSWorld-MCP, including a +16.8-point OSWorld success-rate gain over GUI-only execution and +7.84 Score points on OSWorld-MCP over its base.
Takeaways & Limitations
The results show that hybrid GUI+CLI interaction skills learned from CUA-Universe transfer across tasks, benchmarks, and tool interfaces.
Takeaways & Limitations
The study focuses on single-application tasks, uses supervised fine-tuning, relies on a VLM judge, and targets open-source or scriptable desktop Linux software.
Abstract
from arXiv · showhide
Computer-use agents have advanced on benchmarks like OSWorld and AndroidWorld, but still act mostly through the GUI, often producing inefficient trajectories. Real-world computer work is hybrid, combining visual-state inspection with precise, high-throughput command-line operations, so capable agents must coordinate both modalities over shared application state. Yet scalable hybrid environments remain scarce because supporting both GUI and CLI over real applications typically requires substantial manual engineering for each application. Existing agents also struggle to use the two interfaces complementarily: CLI-native agents lack visual perception for tasks involving interface state or layout, while GUI-native agents are inefficient for operations better executed through commands. We introduce CUA-Universe, a scalable environment-to-data pipeline that turns real desktop software into hybrid GUI+CLI environments. App-Forge adapts applications into reproducible VMs and command-line surfaces it discovers, wraps, or generates, scaling to 16 applications; Task-Weave synthesizes diverse hybrid tasks of controllable difficulty from reusable operations over seed files; and Path-Steer steers rollouts along efficient hybrid paths and harvests verified trajectories for post-training. Training on this data shifts behavior from inefficient GUI interaction and brittle CLI scripting toward effective GUI+CLI orchestration. Our 9B model improves both success and efficiency on CUA-Verse (Score +39.3 pts; -37% steps, -60% tokens), OSWorld (SR +16.8 pts; -57% steps, -44% tokens), and OSWorld-MCP (Score +7.84 pts; -27% steps, -30% tokens). CUA-Universe provides a scalable path toward more capable and efficient computer-use agents.
1 Introduction
CUA-Universe addresses the need for computer-use agents to coordinate GUI and CLI actions over shared application state without costly per-application engineering. Its three-part pipeline scales hybrid environments, tasks, and trajectories, improving success and efficiency across CUA-Verse, OSWorld, and OSWorld-MCP.
- Motivation: Real computer use combines visually grounded GUI interaction with precise, high-throughput command-line operations over shared application state.GUI-centric agents can produce unnecessarily long trajectories, while CLI-only approaches lack visual perception for interface state and layout.
- Approach: CUA-Universe turns real desktop software into hybrid GUI+CLI environments through App-Forge, Task-Weave, and Path-Steer.The components cover application adaptation, task synthesis, and efficient trajectory generation for post-training.
- Scale: 16 real desktop applications span creative, office, media, and technical domains, each exposed through GUI and application-specific CLI interfaces.The pipeline uses reproducible VMs, discovered or generated command-line surfaces, reusable operations, seed files, and harvested trajectories.
- Results: CUA-Universe improves performance within its constructed environments and transfers hybrid interaction skills across tasks, benchmarks, and tool interfaces.The evaluation reports gains on CUA-Verse, OSWorld, and the unseen tool interface of OSWorld-MCP.
- Evaluation: CUA-Verse provides 160 held-out hybrid tasks across eight applications to evaluate coordination of GUI and CLI actions over shared state.The benchmark directly tests hybrid orchestration rather than isolated interface use.
- Results: The trained 9B model achieves roughly 3× the success of its base model on CUA-Verse while using 60% fewer tokens.It also transfers to OSWorld with a +16.8-point gain over GUI-only execution and generalizes to OSWorld-MCP.
2 Related Work
Prior work advances GUI agents, CLI agents, and hybrid action models, but typically relies on fixed tools or artificial GUI environments. CUA-Universe instead constructs application-grounded CLI surfaces and synthesizes hybrid data for real desktop software.
- Environment synthesis: Automatically synthesized environments reduce benchmark construction costs but often target artificial web pages confined to GUI interaction.This leaves a sim-to-real gap relative to actual desktop software.
- GUI and CLI agents: GUI agents span desktop, mobile, and heterogeneous platforms but can remain inefficient on long-horizon and cross-application workflows.Related benchmarks include OSWorld, AndroidWorld, and FedGUI.
- Hybrid agents: Hybrid systems combine visual actions with CLI tool calls, while hybrid benchmarks measure GUI+CLI efficiency trade-offs over fixed tools and tasks.Examples include CoAct-1 and UltraCUA, alongside recent hybrid benchmarks.
- CUA-Universe: CUA-Universe differs by constructing application-grounded tool layers and synthesizing environments and data rather than training over a fixed generic tool set.Its CLI surfaces are tied to each real desktop application.
3 Method
The method converts real desktop applications into reproducible shared-state GUI+CLI environments, composes validated tasks from reusable operations and seed files, and harvests efficient verified trajectories. App-Forge handles environment and tool construction, Task-Weave handles task generation, and Path-Steer handles rollout collection.
- 3.1 Overview and Problem Formulation: CUA-Universe models each hybrid task as a POMDP whose GUI and CLI actions read and modify shared persistent application state.Tasks include an instruction, seed initial state, and verifier scoring the trajectory.
- App-Forge: App-Forge addresses application-specific installation and tool-construction bottlenecks by producing reproducible environments with CLI surfaces aligned to GUIs.The pipeline uses agentic adaptation across heterogeneous applications.
- Task-Weave: Task-Weave abstracts reusable operations, composes seed-conditioned tasks, and validates them through real execution.Tasks range from single-operation edits to multi-step hybrid workflows, with invalid tasks discarded and fixable tasks revised.
- Path-Steer: Path-Steer converts synthesized tasks into training trajectories through hybrid execution, modality-aware path steering, and harvesting of high-scoring rollouts.The verifier scores completed trajectories, and retained records include observations, actions, returns, screenshots, and final scores.
- Training data: 4,923 episodes and approximately 235K training steps are generated across all 16 applications.The training-data composition demonstrates the pipeline’s scale across CUA-Universe extensions and OSWorld applications.
4 Experiments
Experiments evaluate CUA-Universe across held-out hybrid tasks, OSWorld transfer, MCP generalization, and rollout efficiency. The trained model improves success while reducing interaction cost, and Path-Steer improves trajectory quality and efficiency across backbones.
- Evaluation on CUA-Verse: CUA-Verse contains 160 held-out hybrid tasks across eight desktop applications, exposing GUI and CLI interfaces over shared application state.The applications include Blender, Draw.io, Zotero, Godot, QGIS, Kdenlive, OBS, and Audacity.
- Evaluation on CUA-Verse: 37% fewer steps and 60% fewer tokens accompany roughly 3× higher CUA-Verse performance than the identical Qwen3.5-9B baseline.Average tokens decrease from 643K to 255K per episode.
- Transfer to OSWorld: 40.2% OSWorld SR with GUI+CLI is 16.8 points above GUI-only execution, while using 28.6 steps and 286.5k tokens per task.Adding CLI access converts 41 previously failed GUI tasks into successes for Ours, compared with only 3–8 additional successes for other models.
- Generalization to OSWorld-MCP: OSWorld-MCP transfer raises SR from 20.90% to 28.69% and reduces steps by 27% and tokens by 30% relative to the base model.The model was trained with application-specific CLI tools but evaluated on the unseen MCP action space.
- Rollout Efficiency: Path-Steer improves rollout quality and efficiency for Kimi K2.5 and Seed2.1 Pro, increasing acceptance and mean score while reducing steps, tokens, and cost.For Kimi K2.5, Accept Rate rises from 0.44 to 0.51 and Mean Score from 0.63 to 0.71, while steps fall from 26.68 to 22.75.
5 Conclusion
CUA-Universe scales computer-use training by providing shared-state GUI+CLI environments and shows that hybrid interaction can transfer across benchmarks. Its results emphasize coordinated modality selection rather than maximizing CLI usage alone.
- CUA-Universe scales agent learning by expanding environments and interaction spaces instead of relying only on larger models or GUI-only trajectories.
- CUA-Universe frames hybrid environment construction as a new scaling axis for training more capable and efficient computer-use agents.The conclusion describes a shift from collecting larger amounts of single-modality behavior toward continuously generated hybrid supervision.
- CLI usage alone is insufficient: GPT-5.5 executes entirely through CLI yet does not fully solve the visually grounded CUA-Verse benchmark.
- 37% fewer mean steps and a Score increase from 0.189 to 0.582 follow hybrid training of the same backbone on CUA-Verse.CLI usage rises to 25.3%, while the base model remains GUI-only and uses nearly the full 60-step budget.
- Out-of-domain transfer on OSWorld is evaluated under a controlled 244-task GUI+CLI scope using applications disjoint from training applications.
A.2 Out-of-Domain Training Transfer
Out-of-domain training transfers hybrid GUI+CLI interaction skills across applications, although gains vary by application. Combining transferable skills with in-domain coverage produces substantially larger improvements.
- The experiment tests zero-shot transfer under the same controlled 244-task GUI+CLI scope rather than application-specific evaluation conditions.
- The 8-app OOD-only LoRA raises overall OSWorld success from 24.2% to 27.4% (+3.2 points) despite disjoint training applications.It improves over the untuned base on seven of eight applications.
- Transfer is heterogeneous: Thunderbird gains +13.4 points and VS Code +8.7, while Impress declines by 1.9 points.
- The full 16-app LoRA reaches 40.2% overall success, a +13.2-point gain over the untuned base across all eight OSWorld applications.Its training mixture includes the OSWorld application domains in addition to CUA-Universe extensions.
B Adapted Applications
CUA-Universe provides adapters for 16 desktop applications, including eight inherited from OSWorld and eight added by the framework. These adapters expose GUI, command-line, and programmatic interfaces over shared application state.
- CUA-Universe supports 16 desktop applications spanning diverse software domains, with eight inherited from OSWorld and eight newly introduced.
- Each application exposes graphical, command-line, and programmatic interfaces operating on the same underlying state.
- The tool layer converts native automation surfaces into bounded subcommands that return JSON, while the rollout runner returns exit code, stdout, and stderr.
- Application adapters cover operations such as projects, layers, filters, scenes, objects, media, documents, timelines, and exports across the supported software.
- The VLC example shows Path-Steer producing an optimal hybrid trajectory, whereas the unguided trajectory becomes stuck during filter configuration and fails.
- The trajectory judge achieves 99.0% precision on accepted trajectories, with judge–human agreement of 97.0% and Cohen’s κ = 0.94.The rejection set has a 5.2% false-negative rate, which reduces data yield without inflating reported Score.
G Rollout Harness and Budget
The rollout harness runs tasks in isolated OSWorld virtual machines under fixed resource and step budgets. Although trajectory collection dominates cost, parallel CPU execution makes the data-generation pipeline practical on one commodity server.
- Each task runs in an isolated OSWorld VM with 4 vCPU and 4 GB of guest RAM on a copy-on-write overlay disk.
- A typical VLC task combines CLI video conversion, media-file opening for verification, and GUI navigation to the Audio Effects panel.
- All agents share a 60-step budget, with a 3-second post-action settle before each observation.
- Approximately 10^4 rollouts at about 5 minutes each require roughly 50,000 VM-minutes before parallelization.
- A 128-core CPU server runs 32 environments concurrently, reducing the idealized collection time to about 1,560 minutes, or just over one day.Observed turnaround is approximately 1.3–1.5 calendar days because of reset overhead and occasional VM stalls.
H.1 Data
The data combines balanced trajectory records, LoRA fine-tuning, and separate CLI and GUI evaluation modes. Evaluation removes Path-Steer guidance so policies operate without task-specific hints.
- Data construction: The OSWorld pool contains 11,915 records per application across eight applications, while the CUA-Verse pool contains 17,511 records per application across eight applications.All records satisfy VLM score ≥0.75; OSWorld records use exactly four images per record.
- Training: Fine-tuning uses Qwen3.5-9B with LoRA while freezing the base weights, vision tower, and multimodal aligner.Only language-model linear modules are trainable.
- Evaluation modes: Each backbone is evaluated in CLI mode with execute cli exposed and GUI mode following the standard OSWorld screenshot-and-pyautogui protocol.Each evaluated backbone uses its own actor prompt.
- Illustrative comparison: In a VS Code example, GUI+CLI completes the settings change with 5 CLI calls and 12 GUI actions, whereas GUI-only exhausts 60 actions without success.The hybrid trajectory edits settings.json directly and verifies persistence; the GUI-only trajectory receives reward 0.
- Evaluation modes: Evaluation defaults the path-hint field to none, reserving Path-Steer priors for data-generation rollouts.No task hint or solution guidance is appended during evaluation.
I.2 CUA-Verse Benchmark
CUA-Verse is a held-out benchmark of 160 hybrid tasks spanning eight applications. Its task solutions require both GUI and CLI operations, reflecting the benchmark’s focus on hybrid competence.
- Benchmark composition: CUA-Verse contains 160 hybrid GUI+CLI tasks, arranged as 20 tasks for each of eight applications.The applications are in-domain because the benchmark measures competence on software covered by the pipeline.
- Task demands: Every application requires both modalities, with task compositions reporting mean abstract-tool counts separately for CLI and GUI operations.GUI operations include keyboard, click, move, drag, scroll, and vision; CLI operations are application command groups.
J OSWorld Example
The VS Code example illustrates why combining modalities can outperform GUI-only interaction on a precise configuration task. The hybrid agent uses GUI for locating the setting and CLI for writing and verifying it.
- Task: The task requires changing VS Code so the cursor remains focused on the debug console during debugging.The target configuration is debug.focusEditorOnBreak: false.
- Judge validation: Human evaluation samples 30 judge-accepted and 30 judge-rejected trajectories per application across all 16 applications.The table reports acceptance rates, false positives, false negatives, precision, and false-negative rates.
- GUI+CLI: GUI+CLI completes the task with 5 CLI calls and 12 GUI actions, earning reward 1.The agent locates the setting through the interface, writes settings.json through CLI, and verifies persistence.
- GUI-only: GUI-only repeatedly searches and clicks through the settings interface, exhausts the 60-action budget, and earns reward 0.The correct configuration is never committed.
K Limitations
The study leaves several scope and evaluation boundaries open. Tasks are single-application and desktop-Linux oriented, training uses supervised fine-tuning, and VLM judging can introduce residual label noise.
- Scope: The current tasks are grounded in one application’s shared GUI+CLI state rather than cross-application workflows.Cross-application state transfer is identified as future work.
- Training: Using harvested trajectories for supervised fine-tuning bounds the learned policy by its data-generation backbone.The paper identifies verifier-based reinforcement learning as a next step toward surpassing the teacher.
- Evaluation: VLM-based task scoring may introduce label noise because it is not based on per-task programmatic checks.Human validation reports 99.0% acceptance precision and Cohen’s κ = 0.94, but the paper does not claim the issue is eliminated.
- Platform boundary: Application adaptation assumes software that is open-source or scriptable enough to expose a command-line surface, and the environments target desktop Linux.Broadening to closed-source software or non-desktop platforms remains open.