Source-linked AI summary
ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents
Fei Tang, Zhiqiong Lu, Boxuan Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen
TL;DR
GUI-agent progress is constrained by fragmented training infrastructure, drifting evaluation protocols, and limited deployment to real users and devices. ClawGUI unifies online RL, standardized evaluation, and deployment in one open-source framework, and its end-to-end model achieves 17.1% MobileWorld GUI-Only Success Rate, 6.0 points above same-scale MAI-UI-2B.
Problem
GUI-agent research lacks coherent infrastructure spanning reproducible RL training, standardized evaluation, and deployment to real devices and users.
Method
ClawGUI integrates virtual and physical-device RL with GiGPO and Process Reward Models, standardized evaluation, and cross-platform deployment with hybrid CLI-GUI control and personalized memory.
Results
17.1% MobileWorld GUI-Only Success Rate lets ClawGUI-2B outperform the same-scale MAI-UI-2B baseline by 6.0 absolute points, while evaluation reproduces 95.8% of official baselines.
Takeaways & Limitations
ClawGUI provides a unified open-source pipeline for training, comparing, and deploying GUI agents across virtual environments, physical devices, and user-facing platforms.
Takeaways & Limitations
ClawGUI-Agent is an early instance of persistent on-device system agents, while the full vision remains to be realized.
Abstract
from arXiv · showhide
GUI agents drive applications through their visual interfaces instead of programmatic APIs, interacting with arbitrary software via taps, swipes, and keystrokes, reaching a long tail of applications that CLI-based agents cannot. Yet progress in this area is bottlenecked less by modeling capacity than by the absence of a coherent full-stack infrastructure: online RL training suffers from environment instability and closed pipelines, evaluation protocols drift silently across works, and trained agents rarely reach real users on real devices. We present \textbf{ClawGUI}, an open-source framework addressing these three gaps within a single harness. \textbf{ClawGUI-RL} provides the first open-source GUI agent RL infrastructure with validated support for both parallel virtual environments and real physical devices, integrating GiGPO with a Process Reward Model for dense step-level supervision. \textbf{ClawGUI-Eval} enforces a fully standardized evaluation pipeline across 6 benchmarks and 11+ models, achieving 95.8\% reproduction against official baselines. \textbf{ClawGUI-Agent} brings trained agents to Android, HarmonyOS, and iOS through 12+ chat platforms with hybrid CLI-GUI control and persistent personalized memory. Trained end to end within this pipeline, \textbf{ClawGUI-2B} achieves 17.1\% Success Rate on MobileWorld GUI-Only, outperforming the same-scale MAI-UI-2B baseline by 6.0\%.
1. Introduction
ClawGUI addresses three infrastructure gaps in GUI agents: closed and emulator-only RL training, irreproducible evaluation, and disconnected real-user deployment. Its unified pipeline integrates training, standardized evaluation, and deployment, with end-to-end validation from ClawGUI-2B.
- GUI agents require realistic training, comparable evaluation, and deployment to real devices for users to benefit from them.
- Closed training infrastructure and emulator-only validation leave open-source GUI agent RL difficult to reproduce and real-device training largely unexplored.
- Evaluation choices such as prompts, coordinates, resolution, and sampling can shift reported accuracy by several points, undermining cross-paper comparison.
- ClawGUI integrates scalable RL, standardized evaluation, and deployment through three modules in one coherent system.ClawGUI-RL supports virtual and physical devices; ClawGUI-Eval standardizes evaluation; ClawGUI-Agent connects agents to real devices and users.
- 17.1% Success Rate on MobileWorld GUI-Only lets ClawGUI-2B outperform the same-scale MAI-UI-2B baseline by 6.0 absolute points.
2. Related Work
Related work advances GUI grounding, long-horizon navigation, online RL, benchmarking, and deployment, but open infrastructure and comparable evaluation remain limited. ClawGUI-RL targets these gaps while supporting real-device training.
- GUI agent research progressed from cascaded grounding pipelines toward unified end-to-end policies for long-horizon navigation.
- Online RL reduces reliance on costly demonstrations but still faces sparse rewards, difficult credit assignment, infrastructure costs, and limited openness.
- ClawGUI-RL releases open infrastructure for environment management, dense rewards, and training across parallel emulators and physical devices.
- Reported benchmark numbers are rarely directly comparable because evaluation configurations vary across papers.
- CLI harnesses provide efficient, precise control but cannot cover applications without programmatic interfaces and can reduce user observability.
3.1. System Overview
ClawGUI covers the GUI-agent lifecycle with three integrated modules: online RL training, reproducible evaluation, and real-device deployment with human interaction.
- ClawGUI is introduced as a unified framework covering the complete GUI-agent development lifecycle.
- ClawGUI-RL provides scalable online RL training, ClawGUI-Eval provides standardized evaluation, and ClawGUI-Agent supports real-device deployment and human interaction.
3.2. ClawGUI-RL: Scalable Online RL Training
ClawGUI-RL provides end-to-end online RL infrastructure with unified virtual and physical environments, reward shaping, and hierarchical advantage estimation for long-horizon GUI tasks.
- ClawGUI-RL supports end-to-end GUI RL from environment management and reward design through policy optimization.
- Environment Manager: Virtual and physical devices can be used through a unified interface within the same training loop.
- Virtual Environment: Parallel Docker-based Android emulators reset tasks and combine system-level verification with MLLM judging for evaluation.
- Real Device Training: Physical-device training uses human-authored executable tasks and MLLM judging because root-level state verification is unavailable.
- Reward Design: A binary episode-end reward assigns 1 for success and 0 for failure but provides sparse guidance for intermediate actions.
- Reward Design: A Process Reward Model scores each action using screenshots and action history, supplying dense step-level supervision alongside the outcome reward.
- Advantage Estimation: GiGPO combines episode-level trajectory advantages with step-level anchor-state grouping for finer-grained credit assignment.
3.3. ClawGUI-Eval: Reproducible GUI Evaluation
ClawGUI-Eval standardizes GUI-agent evaluation through a reproducible Infer → Judge → Metric pipeline spanning multiple benchmarks and models. It pins evaluation choices, supports independent stage reruns, and achieves high reproduction against official results.
- 95.8% reproduction rate against official results across 6 benchmarks and 11+ models demonstrates ClawGUI-Eval’s reproducibility.
- ClawGUI-Eval pins evaluation choices per model and uses a strict three-stage pipeline to reduce cross-paper comparability problems.
- The framework covers 6 benchmarks and 11+ models, with inference results publicly released alongside evaluation code.
- Infer, Judge, and Metric: The evaluation pipeline separates inference, judging, and metric aggregation, with each stage having a defined input and output.
- Infer, Judge, and Metric: Benchmark-specific judges produce per-sample correctness labels for GUI grounding, OSWorld-G, and AndroidControl before metric aggregation.
- Infer, Judge, and Metric: Decoupled stages allow individual components to be rerun, such as re-judging existing predictions without repeating expensive inference.
3.4. ClawGUI-Agent: Personal GUI Assistant
ClawGUI-Agent connects GUI agents to users through chat platforms and real devices, combining CLI efficiency with GUI coverage. It adds persistent personalized memory and supports remote, local, and natural-language-triggered operation.
- ClawGUI-Agent bridges trained GUI agents to real users across real devices through a production-ready deployment system.
- The hybrid approach uses CLI control where interfaces permit and GUI control otherwise, combining operational speed with broader application coverage.
- Persistent personalized memory stores structured facts about users and retrieves semantically similar memories for subsequent task contexts.
- Users can control target devices remotely through 12+ chat platforms or locally from a chat application running on the phone itself.
- A single natural-language command can trigger environment verification, multi-GPU inference, judging, metric computation, and baseline comparison.
4. Experiments
Experiments evaluate ClawGUI-2B on MobileWorld GUI-Only, test dense reward design, and assess reproducibility across GUI grounding benchmarks. The results show gains from GiGPO-based training and high reproduction under standardized evaluation choices.
- Main Results: 17.1% Success Rate: ClawGUI-2B surpasses the same-scale MAI-UI-2B baseline by 6.0% on MobileWorld GUI-Only.It also exceeds Qwen3-VL-32B at 11.9% and UI-Venus-72B at 16.4%.
- Reward Ablation: 2.6% improvement: replacing episode-level GRPO with step-level GiGPO raises Success Rate from 14.5% to 17.1%.The corresponding relative gain is 17.9%, supporting finer-grained credit assignment for GUI reinforcement learning.
- Evaluation Protocol: ClawGUI-Eval pins evaluation choices and uses an Infer → Judge → Metric pipeline to improve comparability across benchmark results.For closed-source frontier models, a two-stage crop-then-ground Zoom strategy recovers official performance without model-internal access.
- Benchmark Reproduction: 95.8% reproduction rate: ClawGUI-Eval reproduces 46/48 cells with official baselines across 6 benchmarks and 11+ models.Successful reproduction requires matching or exceeding the official value or staying within an absolute difference of 2%.
5. Discussion
The discussion positions GUI agents as moving toward unified research-and-deployment systems, with world models and persistent on-device agents as important future directions.
- GUI agents are increasingly framed as full-stack systems whose reliability depends on the surrounding harness as much as the model.The discussion highlights permission pipelines, tool dispatch, context compaction, and multi-turn recovery as determinants of reliable agency.
- Emulator-only RL training misses real app behavior and authenticated commercial applications, motivating mock applications and privacy-preserving on-device RL.Both directions require infrastructure that can manage environment instability at scale.
- ClawGUI-RL is designed to support the environment infrastructure needed to scale online RL beyond emulator sandboxes.The framework is presented as addressing instability across large-scale training environments.
- Persistent system agents remain an open vision requiring tighter operating-system integration, on-device policy training, and local-first privacy guarantees.ClawGUI-Agent’s hybrid CLI-GUI control and personalized memory are described as early instances rather than a complete realization.
- GUI-specific world models could enable model-based planning, counterfactual rollouts, and early dead-end detection from screen-action trajectories.ClawGUI-RL’s dense step-level trajectory logging is identified as a substrate for training such models at scale.
6. Conclusion
ClawGUI unifies online RL training, standardized evaluation, and real-device deployment in an open-source pipeline for GUI agents. The resulting ClawGUI-2B reaches 17.1% MobileWorld Success Rate and outperforms a same-scale baseline by a relative margin of 54%.
- ClawGUI unifies online RL training, standardized evaluation, and real-device deployment into one open-source GUI-agent pipeline.Its three modules are ClawGUI-RL, ClawGUI-Eval, and ClawGUI-Agent.
- ClawGUI-RL supports parallel virtual environments and real physical-device training while combining GiGPO with a Process Reward Model.The Process Reward Model provides dense step-level reward supervision.
- 95.8% reproduction rate was achieved across 6 benchmarks and 11+ models against official baselines.
- ClawGUI-Agent enables natural-language automation across Android, HarmonyOS, and iOS through 12+ chat platforms with personalized memory.
- 17.1% MobileWorld SR was achieved by ClawGUI-2B, surpassing the same-scale baseline by a relative margin of 54%.The model was trained end-to-end within the pipeline.