Source-linked AI summary
OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution
Le Zhang, Yixiong Xiao, Xinjiang Lu, Jingjia Cao, Yusai Zhao, Jingbo Zhou, Lang An, Zikan Feng, Wanxiang Sha, Yu Shi, Congxi Xiao, Jian Xiong, Yankai Zhang, Hua Wu, Haifeng Wang
TL;DR
GUI agents still face bottlenecks in data quality, training, and evaluation across diverse digital environments. OmegaUse combines a MoE model with synthesized training data, decoupled SFT and GRPO training, and OS-Nav evaluation, achieving competitive results across established and new benchmarks.
Problem
GUI agents lack consistently high-quality training data and comprehensive evaluation across diverse mobile and desktop environments.
Method
OmegaUse combines a parameter-efficient MoE backbone, a data pipeline integrating curated datasets with autonomous exploration and taxonomy-guided synthesis, and decoupled SFT-GRPO training.
Results
OmegaUse is competitive across GUI benchmarks, including 96.3% on ScreenSpot-V2, 79.1% step success on AndroidControl, and strong OS-Nav performance.
Takeaways & Limitations
The results support OmegaUse as a cross-platform GUI agent for autonomous task execution on mobile and desktop systems.
Takeaways & Limitations
The work focuses on end-to-end model training rather than the full GUI-agent system, which may include external tools.
Abstract
from arXiv · showhide
Graphical User Interface (GUI) agents show great potential for enabling foundation models to complete real-world tasks, revolutionizing human-computer interaction and improving human productivity. In this report, we present OmegaUse, a general-purpose GUI agent model for autonomous task execution on both mobile and desktop platforms, supporting computer-use and phone-use scenarios. Building an effective GUI agent model relies on two factors: (1) high-quality data and (2) effective training methods. To address these, we introduce a carefully engineered data-construction pipeline and a decoupled training paradigm. For data construction, we leverage rigorously curated open-source datasets and introduce a novel automated synthesis framework that integrates bottom-up autonomous exploration with top-down taxonomy-guided generation to create high-fidelity synthetic data. For training, to better leverage these data, we adopt a two-stage strategy: Supervised Fine-Tuning (SFT) to establish fundamental interaction syntax, followed by Group Relative Policy Optimization (GRPO) to improve spatial grounding and sequential planning. To balance computational efficiency with agentic reasoning capacity, OmegaUse is built on a Mixture-of-Experts (MoE) backbone. To evaluate cross-terminal capabilities in an offline setting, we introduce OS-Nav, a benchmark suite spanning multiple operating systems: ChiM-Nav, targeting Chinese Android mobile environments, and Ubu-Nav, focusing on routine desktop interactions on Ubuntu. Extensive experiments show that OmegaUse is highly competitive across established GUI benchmarks, achieving a state-of-the-art (SOTA) score of 96.3% on ScreenSpot-V2 and a leading 79.1% step success rate on AndroidControl. OmegaUse also performs strongly on OS-Nav, reaching 74.24% step success on ChiM-Nav and 55.9% average success on Ubu-Nav.
1 Introduction
OmegaUse addresses GUI-agent bottlenecks in data quality, training, and cross-platform evaluation with a general-purpose MoE model and supporting pipelines. It achieves competitive performance across grounding, navigation, and newly introduced offline benchmarks.
- 1 Introduction: OmegaUse is a general-purpose GUI agent using a parameter-efficient MoE architecture for autonomous computer-use and phone-use task execution.The model is designed to preserve large-model reasoning capacity while activating only a subset of parameters.
- 1 Introduction: Noisy labels, misaligned bounding boxes, incorrect trajectories, and redundant actions limit GUI-agent spatial perception and long-horizon planning.These data-quality problems can degrade grounding and decision-making.
- 1 Introduction: OmegaUse constructs training data from curated open-source datasets, autonomous exploration, taxonomy-guided synthesis, and cross-terminal expert demonstrations.The pipeline targets data diversity, coverage, fidelity, and reduced reliance on manual annotation.
- 1 Introduction: SFT establishes interaction syntax and basic task logic, while GRPO refines spatial grounding and sequential planning using specialized rewards.The rewards include an Inside-of-Bounding-Box grounding reward and stepwise coordinate-based navigation rewards.
- 1 Introduction: OS-Nav evaluates cross-platform generalization and planning consistency through ChiM-Nav for Chinese Android systems and Ubu-Nav for Ubuntu desktop interactions.Both sub-benchmarks provide expert-verified reasoning trajectories.
- 1 Introduction: 96.3% on ScreenSpot-V2, 79.1% step success on AndroidControl, 74.24% step success on ChiM-Nav, and 55.9% average success on Ubu-Nav.OmegaUse is reported as competitive or superior across multiple GUI benchmarks and platforms.
2 Related Work
Prior GUI-agent research has developed grounding benchmarks, modular systems, and increasingly unified end-to-end models. The literature focuses on UI localization and architectures for perception, planning, memory, and action.
- 2 Related Work: GUI-agent research primarily addresses UI grounding and GUI-agent architectures, including modular pipelines and native end-to-end models.These models perceive screenshots or structured UI representations and execute actions such as clicking, typing, and scrolling.
- 2 Related Work: Grounding methods localize instruction-relevant UI elements across mobile, web, and desktop interfaces, but remain sensitive to resolution, layout diversity, and domain shift.ScreenSpot and ScreenSpot-V2 were introduced to characterize cross-platform grounding performance.
- 2 Related Work: Reinforcement-learning-style grounding methods use spatial correctness rewards to improve generalization and reduce dependence on dense annotations.Related work includes UI-R1, GUI-R1, InfiGUI-R1, GUI-Actor, GUI-G2, and InfiGUI-G1.
- 2 Related Work: Modular GUI agents separate perception, planning, memory, and execution through components such as planners, memory buffers, verifiers, and tool wrappers.Agent-S, Agent-S3, and Cradle exemplify framework-centric designs.
- 2 Related Work: Native end-to-end agents unify perception, reasoning, and action within one model, aiming to leverage large-scale data and reinforcement-learning signals.AutoWebGLM and UI-TARS represent this shift, while AutoGLM introduces an intermediate interface separating planning from grounding.
3 Methodology
OmegaUse combines decoupled grounding and navigation models with curated data, staged training, unified actions, and cross-platform deployment. Its methodology refines noisy supervision, uses SFT followed by GRPO, and represents UI interaction through explicit perception, reasoning, and executable actions.
- Model and training design: OmegaUse separates grounding and navigation models to optimize precise visual perception and sequential decision-making with reduced interference.The grounding model targets spatial coordinates, while the navigation model handles sequential decisions.
- Grounding data pipeline: 1.66 million raw grounding instances were distilled into 111K high-quality samples through filtering, manual box correction, and instruction refinement.The pipeline removed redundant, simplistic, blurred, and ambiguous samples before producing reliable supervision.
- Model and training design: SFT establishes coordinate-formatting syntax, while GRPO refines spatial precision using relative group rewards and a clipped policy objective.GRPO reduces the need for a separate critic model by estimating baselines from within-group rewards.
- Grounding data pipeline: Grounding rewards combine executable-format validation with an inside-of-bounding-box criterion for localization accuracy.The format reward checks syntax, while the positional reward gives credit when the predicted center lies inside the ground-truth region.
- Navigation representation: Navigation uses a unified cross-platform action space spanning shared primitives and terminal-specific operations such as hotkeys, clicks, long presses, and system navigation.The action representation supports desktop and mobile interaction within one cohesive operational schema.
- Navigation data pipeline: The navigation pipeline compresses semantically redundant UI states into a transition graph, while each agent step follows observation, reasoning, and action.MLLM-based clustering merges functionally equivalent states, and the O → T → A structure grounds executable actions in perception and planning.
4 Offline Benchmarks for Real-World GUI Navigation
OS-Nav is an offline benchmark for realistic GUI navigation across Chinese Android and Ubuntu desktop environments, built with expert-labeled and human-refined trajectories.
- OS-Nav comprises ChiM-Nav for Chinese Android systems and Ubu-Nav for routine Ubuntu desktop interactions.
- Expert-labeled execution traces and human refinement support reliable state transitions and authentic user-behavior evaluation.MLLM-generated intermediate reasoning descriptions provide a semantic bridge between linguistic goals and raw actions.
- ChiM-Nav: ChiM-Nav contains 142 trajectories across 69 applications and 991 operational steps, averaging 6.98 steps per trajectory.It emphasizes daily Chinese mobile usage, distinctive UI layouts, and multi-step workflows.
- Ubu-Nav: Ubu-Nav contains 101 trajectories and 641 steps, with tasks averaging 6.35 steps in Ubuntu environments.Its scenarios cover routine desktop operations and common PC interactions requiring multi-step reasoning.
5 Experiments
Experiments evaluate OmegaUse across grounding and navigation benchmarks spanning mobile, web, and desktop environments. The results show strong performance on standard and specialized benchmarks, with particular strengths in ScreenSpot-V2, AndroidControl, ChiM-Nav, and Ubu-Nav.
- GUI Grounding: 96.3% average success on ScreenSpot-V2 establishes OmegaUse-G as the benchmark’s state-of-the-art model.It outperforms UI-Venus-Ground-72B at 95.3% and Seed1.5-VL at 95.2%.
- Standard Navigation: 87.6% Type Accuracy and 79.1% Step Success Rate on AndroidControl give OmegaUse first place on both evaluated metrics.These results exceed UI-Venus-Navi-72B’s 85.9% Type Accuracy and 77.2% Step Success Rate.
- Standard Navigation: 55.7% success rate on AndroidWorld shows competitive online mobile interaction without external planners or Accessibility trees.OmegaUse outperforms UI-TARS-72B at 46.6% and Aria-UI at 44.8%, while remaining below UI-Venus-Navi-72B at 65.9%.
- Specialized Offline Benchmarks: 74.24% Step Success Rate on ChiM-Nav surpasses all existing open-source baselines in Chinese Android workflows.OmegaUse reaches 87.78% Type Accuracy and exceeds UI-Venus-72b’s 67.51% Step Success Rate.
- Specialized Offline Benchmarks: 55.9% average performance on Ubu-Nav leads Holo2-30B-A3B at 50.0% across routine Ubuntu desktop operations.Non-coordinate tasks reach 48.6%, above UI-Venus-Navi-72B at 40.0% and Holo2-30B-A3B at 34.3%.
6 Conclusion
OmegaUse combines a Mixture-of-Experts backbone, high-fidelity data construction, and decoupled SFT-plus-GRPO training for autonomous GUI task execution across mobile and desktop platforms. Across multiple platforms, it establishes strong benchmark results, including 96.3% on ScreenSpot-V2 and 79.1% Step Success Rate on AndroidControl.
- OmegaUse is an autonomous GUI agent for complex mobile and desktop tasks, supporting phone-use and computer-use scenarios.
- Its framework combines curated datasets, automated hierarchical trajectory synthesis, and decoupled SFT-plus-GRPO training with specialized reward mechanisms.
- 96.3% on ScreenSpot-V2 and 79.1% Step Success Rate on AndroidControl are representative benchmark records for OmegaUse.