PaperlayerView exact revision on arXiv

Source-linked AI summary

ZeroGUI: Automating Online GUI Learning at Zero Human Cost

Chenyu Yang, Shiqian Su, Shi Liu, Xuan Dong, Yue Yu, Weijie Su, Xuehui Wang, Zhaoyang Liu, Jinguo Zhu, Hao Li, Wenhai Wang, Yu Qiao, Xizhou Zhu, Jifeng Dai

arXiv:2505.23762v1cs.AIcs.CLcs.CV

TL;DR

Offline GUI-agent training relies on manual annotations and has limited adaptability to dynamic environments, while online learning lacks scalable tasks and verifiers. ZeroGUI combines VLM-generated tasks and rewards with two-stage online reinforcement learning. It improves performance across OSWorld and AndroidLab, including relative gains on OSWorld for both evaluated agents.

  • Problem

    Offline GUI-agent approaches rely on high-quality human annotations for grounding and action supervision and have limited adaptability to dynamic, interactive environments.

    Source: §1 · p. 1

  • Method

    ZeroGUI generates GUI tasks and estimates success rewards with VLMs, then trains agents on generated tasks and adapts them at test time through online reinforcement learning.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    ZeroGUI significantly improves task success rates across OSWorld and AndroidLab; on OSWorld, ZeroGUI-Aguvis-7B achieves a 63% relative improvement.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Training on generated tasks extends capability coverage, while test-time training helps agents adapt to target test tasks.

    Source: §1 · p. 2

    On OSWorld, UI-TARS-7B-DPO's full-test-set improvement is smaller than on the feasible subset (+2.5 versus +4.5 in average success rate), indicating reduced infeasibility detection that may reflect software-knowledge gaps or noisy rewards.

    Source: §4.2.1 · p. 8

Abstract

from arXiv · show

The rapid advancement of large Vision-Language Models (VLMs) has propelled the development of pure-vision-based GUI Agents, capable of perceiving and operating Graphical User Interfaces (GUI) to autonomously fulfill user instructions. However, existing approaches usually adopt an offline learning framework, which faces two core limitations: (1) heavy reliance on high-quality manual annotations for element grounding and action supervision, and (2) limited adaptability to dynamic and interactive environments. To address these limitations, we propose ZeroGUI, a scalable, online learning framework for automating GUI Agent training at Zero human cost. Specifically, ZeroGUI integrates (i) VLM-based automatic task generation to produce diverse training goals from the current environment state, (ii) VLM-based automatic reward estimation to assess task success without hand-crafted evaluation functions, and (iii) two-stage online reinforcement learning to continuously interact with and learn from GUI environments. Experiments on two advanced GUI Agents (UI-TARS and Aguvis) demonstrate that ZeroGUI significantly boosts performance across OSWorld and AndroidLab environments. The code is available at https://github.com/OpenGVLab/ZeroGUI.

1 Introduction

Offline GUI-agent training depends on costly human-annotated trajectories and struggles with dynamic settings, while online learning is difficult to scale without generated tasks and verifiers. ZeroGUI automates task generation and reward estimation and uses two-stage online reinforcement learning; evaluations report gains across desktop and mobile environments.

1 Introduction

  • Offline methods require costly human labels for GUI grounding and action trajectories, while agents trained on static or narrow tasks struggle to generalize to dynamic GUIs.Source: §1 · pp. 1–2
  • Online learning remains hard to scale because interactive environments lack diverse training tasks and success verifiers, and novel scenarios may have no ground-truth annotations.Source: §1 · p. 2
  • ZeroGUI uses VLMs to generate tasks from random initial states and estimate binary success rewards from agent trajectories, avoiding hand-crafted task verifiers.Source: §1 · p. 2
  • On OSWorld, ZeroGUI-UI-TARS-7B achieves 14% relative improvements and ZeroGUI-Aguvis-7B achieves 63% relative improvements, with gains reported across OSWorld and AndroidLab and generalization across base models.Source: §1 · pp. 2–3
  • Its two-stage reinforcement learning first trains on generated tasks for general capabilities, then adapts the agent to target test tasks at test time.Source: §1 · pp. 2–3

2 Related Work

Related work addresses GUI grounding, long-horizon planning, and adaptation in interactive environments through datasets, specialized models, and reinforcement learning. ZeroGUI positions automated task generation and reward estimation as a route to scalable RL in environments without curated training tasks.

2 Related Work

  • GUI grounding research ranges from structured inputs and specialized parsers to manually labeled or automated datasets used for supervised fine-tuning.Source: §2 · p. 3
  • Planning work uses expert trajectories and annotated demonstrations, while studies report that agents trained on static trajectories can struggle to generalize in dynamic real-world environments.Source: §2 · p. 3
  • Interactive environments support autonomous perception and action in more realistic settings, but their evaluation often relies on complex, hardcoded rules or scripts.Source: §2 · p. 3
  • Prior post-training RL work often targets settings with accessible labels; ZeroGUI applies test-time RL to interactive GUI environments, adapting GRPO to multi-step interaction and tuning KL regularization for stability.Source: §2 · pp. 3–4

3 ZeroGUI

ZeroGUI replaces offline GUI-agent training with online interaction, using VLM-generated tasks and rewards to train agents in two stages. Its components automate task generation and reward estimation, then adapt training from generated tasks to test tasks.

3 ZeroGUI

  • ZeroGUI combines automatic task generation, VLM-based reward estimation, and two-stage online reinforcement learning to train GUI agents through environment interaction.The first stage uses generated tasks; the second trains on test tasks without ground-truth labels.Source: §3 · p. 4, §3.3 · p. 6

3.1 Automatic Task Generation

  • Example-guided prompts use exemplars and random initial screenshots, while multi-candidate generation encourages diverse, environment-aligned tasks.Generated tasks can also include intentionally infeasible goals that require the agent to respond “FAIL.”Source: §3.1 · p. 4

3.2 Automatic Reward Estimation

  • The VLM evaluates trajectories for binary task-success rewards; multiple evaluations use majority or unanimous voting to reduce false positives.All trajectory screenshots are included, while agent responses are excluded because claims of success could mislead the evaluator.Source: §3.2 · p. 5

3.3 Two-stage Online Reinforcement Learning

  • Two-stage training first builds capabilities on generated tasks, then adapts to test tasks using rewards from the same evaluator without ground-truth labels.The approach adapts GRPO to multi-step GUI trajectories by assigning normalized group rewards to action-prediction sequences.Source: §3.3 · p. 6
  • Replacing GRPO’s k3 KL estimator with a per-token MSE k2 estimator improves training stability and avoids numerical overflow.The KL constraint is retained because removing it may cause policy-distribution drift.Source: §3.3 · p. 6

4 Experiment

ZeroGUI is evaluated on OSWorld and AndroidLab using screenshot-only settings, with results showing gains over base models and improvements from both online-training stages. Ablations examine task generation, reward estimation, online RL, and modified KL loss.

4.1.1 Evaluation Environments and Metrics

  • OSWorld includes 369 tasks, including 30 infeasible tasks; results are reported for the full test set and the feasible subset.Evaluation uses Ubuntu screenshot-only mode, a 15-step limit, and mean and standard deviation over four runs; pass@k measures capacity coverage and all-pass@k consistency.Source: §4.1.1 · pp. 6–7

4.1.2 Implementation Details

  • The setup samples 725 Ubuntu and 175 Android training tasks from pools exceeding 4,000 and totaling 225 tasks, respectively.GPT-4o generates tasks; Qwen2.5-VL-32B estimates rewards using four queries and unanimous voting. Training uses UI-TARS-7B-DPO and Aguvis-7B with one epoch for generated and test-time tasks.Source: §4.1.2 · p. 7

4.2.1 OSWorld

  • Both generated-task training and test-time training contribute to OSWorld gains; test-time training alone underperforms the two-stage setup.Generated-task training improves pass@4, while test-time training mainly improves all-pass@4.Source: §4.2.1 · p. 7
  • UI-TARS-7B-DPO gains +2.5 (14%) on all OSWorld tasks and +4.5 (40%) on feasible tasks; Aguvis-7B gains +1.9 (63%) and +2.1 (88%), respectively.The reported success-rate results are mean±std; improvements are relative to each base model.Source: §4.2.1 · p. 7, Table 1 · p. 8

4.2.2 AndroidLab

  • On AndroidLab, ZeroGUI improves SR by +2.8 on operational tasks and +1.8 on the full set, and improves operation-subset Sub-SR by +2.9.The full set includes query-detecting tasks, whose GPT-based evaluation is not fully reliable; the operation subset uses predefined rules.Source: §4.1.1 · p. 7, Table 2 · p. 8, §4.2.2 · p. 8

4.3 Ablation Study

  • Task generation without examples or with only one task at a time lowers test performance; the ablations are run on OSWorld’s Daily domain.Table 3 covers task-generation and reward-estimation designs, while Table 4 covers online RL training and modified KL loss.Source: §4.3 · p. 8, Table 3 · p. 9, Table 4 · p. 9

5 Conclusion

ZeroGUI removes manually collected and labeled offline training data through VLM-generated tasks and rewards, then applies two-stage reinforcement learning. The paper reports relative improvements on OSWorld for both evaluated agents.

5 Conclusion

  • Figure 3 compares training accuracies for k3-KL (GRPO) and k2-KL (ours).Source: Figure 3 · p. 9
  • ZeroGUI uses VLMs to generate training tasks and reward signals, eliminating manual offline data collection and human cost for task design and evaluation.Source: §5 · p. 9
  • 63% relative improvement for ZeroGUI-Aguvis-7B and 14% for ZeroGUI-UI-TARS-7B are reported on OSWorld.Source: §5 · p. 10

A.1 Task Generation

Task generation samples environment states to produce training tasks, while a VLM evaluator scores execution trajectories. The appendix describes task-generation scale and constraints, and a repeated-vote reward procedure.

A.1 Task Generation

  • Over 4,000 Ubuntu-based and 225 Android-based tasks were generated, with 725 and 175 respectively sampled for the final training set.Prompts use screenshots of randomly sampled initial states; each prompt generates 10 candidates for OSWorld and 5 for AndroidLab.Source: §A.1 · p. 10
  • Generated instructions should be diverse, achievable from the current state, objectively evaluable, and require at least three steps.The prompt also asks for clear instructions and a distribution of task difficulty and complexity.Source: §A.1 · p. 10

A.2 Reward Estimation

  • A locally deployed QwenVL-2.5-32B-Instruct assigns binary success or failure rewards from the task instruction and trajectory screenshots, using unanimous agreement across repeated queries.The prompt asks the evaluator to assess the final state after analyzing the task requirements and screenshots.Source: §A.2 · pp. 10–11

A.3 RL Training

The appendix directs readers to Table 5 for ZeroGUI’s RL training hyperparameters.

A.3 RL Training

  • ZeroGUI’s RL training hyperparameters are listed in Table 5.Source: §A.3 · p. 12

B.1 Theoretical Derivation and Analysis

The analysis replaces GRPO’s k3 KL estimator with a per-token k2 estimator and examines the resulting KL gradients during training. It reports a stable gradient coefficient and avoids exponentiation-related overflow or underflow in practical loss computation.

B.1 Theoretical Derivation and Analysis

  • The method replaces GRPO’s per-token k3 KL estimator with a k2-estimator expressed as per-token MSE loss.Source: §B.1 · p. 13
  • The analysis plots KL gradient coefficients against log πθ − log πref and records KL loss plus token-wise maximum and minimum log-probability differences during training.Figures 4 and 5 present these quantities.Source: §B.1 · p. 13, Figure 4 · p. 14, Figure 5 · p. 14
  • The authors report that when some tokens have log πθ much smaller than log πref, GRPO’s gradient coefficient can become very large, while theirs remains stable and avoids large gradients.They note that concurrent works reached similar derivations and conclusions.Source: §B.1 · p. 14
  • The proposed KL loss avoids the exponentiation required by GRPO’s KL loss, which the authors say is prone to overflow or underflow.Source: §B.1 · p. 14

B.2 Supplementary Ablation Study

The ablation study finds that test performance depends on the KL loss coefficient β: both removing the penalty or making it small, and making it large, lead to worse results.

B.2 Supplementary Ablation Study

  • A KL-coefficient sweep identifies a setting of β with the best test performance.Figure 6 presents test success rates for different KL loss coefficients.Source: §B.2 · p. 14, Figure 6 · p. 14
  • Removing the KL penalty (β = 0) or using a small coefficient such as 0.01 degrades performance, likely through policy drift and overfitting to current tasks.The authors attribute this degradation as likely caused by policy distribution drift.Source: §B.2 · pp. 14–15
  • A large KL coefficient such as 1 also worsens results by imposing excessive constraints on optimization.Source: §B.2 · p. 15

C.1 Task Example

Generated tasks extend common operations with diverse, feasible, and well-grounded examples. Their instructions align with target domains while also reaching regions beyond the original OSWorld instruction set.

C.1 Task Example

  • Generated tasks extend common operations with diverse examples, including Android auto-rotate and screen-timeout settings and Ubuntu video-playback speed changes absent from the original test tasks.Source: §C.1 · p. 15
  • Examples such as copying a package list, averaging sales data, setting alarms, and editing calendars are described as feasible, well-grounded, and practically applicable.Source: §C.1 · p. 15

C.2 Visualization

  • The instruction visualization encodes original OSWorld test instructions in orange and generated instructions in blue, using Sentence-BERT embeddings reduced to two dimensions with UMAP.The visualization follows OS-Genesis [59] and uses Sentence-BERT [52] and UMAP [42].Source: §C.2 · p. 15, Figure 7 · p. 16
  • Generated instructions align with their target application domains while extending into previously unexplored regions of instruction space.The paper describes this as evidence of semantically valid and diverse tasks beyond the original dataset.Source: §C.2 · pp. 15–16

D Qualitative Comparisons

In the qualitative cases, the base model fails to complete the tasks, while the ZeroGUI-trained model handles them more reliably, including detailed requirements and long-horizon actions. The authors report that training on generated tasks yields more generalizable interaction strategies.

D Qualitative Comparisons

Loading 2505.23762v1…