Source-linked AI summary
UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience
Zichuan Lin, Feiyu Liu, Yijun Yang, Jiafei Lyu, Yiming Gao, Yicheng Liu, Zhicong Lu, Yangbin Yu, Mingyu Yang, Junyou Li, Deheng Ye, Jie Jiang
TL;DR
Mobile GUI agents struggle to learn from failed long-horizon trajectories because rewards are sparse and do not identify the responsible step. UI-Voyager combines RFT with GRSD, using fork points and successful peer trajectories to generate denser supervision. On AndroidWorld, its 4B model reaches 81.0% Pass@1 success across 116 tasks, exceeding baselines and reported human-level performance.
Problem
Long-horizon mobile GUI tasks provide trajectory-level success or failure rewards, leaving agents without clear signals about which actions caused failure or what alternatives were correct.
Method
UI-Voyager combines Rejection Fine-Tuning for autonomous data–model co-evolution with Group Relative Self-Distillation, which detects fork points and distills successful actions into failed trajectories.
Results
81.0% Pass@1 success was achieved by the 4B model on 116 AndroidWorld tasks, surpassing all baselines and reported human-level performance.
Takeaways & Limitations
UI-Voyager provides a self-evolving mobile GUI-agent training framework that reuses failed trajectories and supplies dense step-level supervision without manual annotation.
Takeaways & Limitations
Evaluation uses AndroidWorld’s predefined high-level action space, which may underestimate failures involving gesture dynamics, timing, actuation noise, and transfer to finer-grained controls.
Abstract
from arXiv · showhide
Autonomous mobile GUI agents have attracted increasing attention along with the advancement of Multimodal Large Language Models (MLLMs). However, existing methods still suffer from inefficient learning from failed trajectories and ambiguous credit assignment under sparse rewards for long-horizon GUI tasks. To that end, we propose UI-Voyager, a novel two-stage self-evolving mobile GUI agent. In the first stage, we employ Rejection Fine-Tuning (RFT), which enables the continuous co-evolution of data and models in a fully autonomous loop. The second stage introduces Group Relative Self-Distillation (GRSD), which identifies critical fork points in group rollouts and constructs dense step-level supervision from successful trajectories to correct failed ones. Extensive experiments on AndroidWorld show that our 4B model achieves an 81.0% Pass@1 success rate, outperforming numerous recent baselines and exceeding human-level performance. Ablation and case studies further verify the effectiveness of GRSD. Our method represents a significant leap toward efficient, self-evolving, and high-performance mobile GUI automation without expensive manual data annotation.
1 Introduction
Mobile GUI agents aim to operate diverse, dynamic interfaces autonomously, but UI-Voyager addresses inefficient learning from failures and sparse credit assignment through a self-evolving training pipeline. On AndroidWorld, its 4B model achieves 81.0% Pass@1 success and surpasses reported baselines and human-level performance.
- Motivation: Mobile interfaces challenge autonomous agents through diverse layouts, rich interactions, limited visual context, and dynamic state transitions.These properties motivate studying mobile GUI agents as mobile phones become increasingly important in daily life.
- Approach: UI-Voyager uses a two-stage self-evolving pipeline combining Rejection Fine-Tuning and Group Relative Self-Distillation.RFT supports autonomous data–model co-evolution, while GRSD addresses learning from failed trajectories and sparse credit assignment.
- Approach: GRSD identifies shared fork points in group rollouts and transfers dense step-level supervision from successful trajectories to failed ones.This reuses failed trajectories while replacing sparse trajectory-level rewards with more precise self-distillation signals.
- Results: 81.0% Pass@1 success was achieved by UI-Voyager’s 4B model on AndroidWorld, surpassing all baseline methods and reported human-level performance.The benchmark contains 116 diverse tasks spanning real-world apps and varying complexities.
2 Related Work
Prior GUI-agent training commonly relies on static datasets or interactive reinforcement-learning and behavior-cloning approaches. UI-Voyager instead uses lightweight intra-group fork-point detection and self-distills successful peer actions without external models.
- Static-data methods: Large-scale static interaction datasets provide GUI knowledge such as action grounding, icon functionality, and task decomposition.Their static training nature limits the agent’s ability to learn continually from interaction outcomes.
- Interactive agents: Interactive-agent research often uses reinforcement learning to optimize policies through environment interaction.Earlier UI-operating agents also used reinforcement learning or behavior cloning for mouse-click interactions.
- UI-Voyager: UI-Voyager differs from methods that use external VLMs to synthesize correction traces by detecting divergent states with lightweight intra-group SSIM matching.Its GRSD mechanism distills correct actions from successful peer trajectories into failed-rollout contexts.
3 Method
UI-Voyager trains mobile GUI agents through a two-stage self-evolving pipeline: Rejection Fine-Tuning improves data and models iteratively, while Group Relative Self-Distillation provides step-level corrections from successful trajectories. Its fork-point mechanism matches screen states across successful and failed rollouts to address sparse-reward credit assignment.
- Overview: UI-Voyager combines iterative Rejection Fine-Tuning with Group Relative Self-Distillation in a self-evolving mobile GUI training pipeline.RFT generates and filters trajectories for iterative supervised updates, while GRSD learns from discrepancies between successful and failed trajectory groups.
- 3.2 Rejection Fine-Tuning: RFT generates diverse trajectories, retains verifier-approved successes, and uses each iteration’s model to produce new data for subsequent updates.New seed tasks perturb parameters such as temporal constraints, quantities, and file entities to maintain task novelty and prevent overfitting.
- 3.2 Rejection Fine-Tuning: After three iterations, RFT improves Pass@1 from 37% to 73%, with consistent gains across all Pass@K metrics.The reported gains support the effectiveness of iterative co-evolution between trajectory quality and model capability.
- 3.3 Group Relative Self-Distillation: Trajectory-level rewards create ambiguous credit assignment because every token shares the same success or failure signal, even in tasks lasting up to 30 steps.A single wrong action can make the entire trajectory receive zero reward, leaving other correct actions without credit.
- 3.3 Group Relative Self-Distillation: GRSD identifies fork points where failed and successful trajectories share a screen state but diverge in their next action, enabling targeted self-correction.It uses SSIM-based cross-trajectory state matching and distills successful self-generated transitions without an external teacher policy.
- 3.3.2 Step-Level Self Distillation: GRSD trains on failed-context prompts paired with successful responses using an autoregressive next-token loss computed only over response tokens.The constructed samples preserve the failed trajectory’s contextual history while replacing its response with the aligned successful trajectory response.
4 Experiment
Experiments on AndroidWorld evaluate UI-Voyager’s self-evolving training pipeline, showing strong performance from RFT initialization, GRSD correction, and fork-point analysis. The 4B model achieves 81.0% success and exceeds reported human-level performance while learning corrective supervision from failed trajectories.
- 4.2 Main Results: 81.0% success rate: UI-Voyager (4B) surpasses all baselines and the reported 80.0% human-level performance on AndroidWorld.The result is averaged over 64 independent runs with randomized task parameters.
- 4.3 Analysis: RFT steadily improves Pass@1 and Pass@k across four iterative rounds, with the third-round checkpoint reaching Pass@1=73.2%.This checkpoint is used for subsequent GRSD, GRPO, and PPO training.
- 4.3 Analysis: Fork-point detection identifies shared states where failed and successful trajectories diverge, then transfers the successful action as supervision.In BrowserMaze, both trajectories share the state at Step 12; the failed Right action is replaced by the successful Down action.
- 4.3 Analysis: Fork points may occur at the initial state, as in SystemBluetoothTurnOff, enabling corrective supervision from the first action.The successful trajectory uses a downward swipe to open the notification shade, unlike the failed upward swipe.
- 4.3 Analysis: Self-corrective samples combine successful reasoning, actions, and tool calls to turn failed trajectories into supervised data without expensive human annotation.This provides dense step-level guidance for improving long-horizon mobile GUI decisions.
- 4.3 Analysis: GRSD reaches 81% success from the same 73.2% RFT model, while GRPO and PPO plateau at 76%.The comparison attributes the gap to GRSD’s fork-point detection and self-distillation for correcting decisive errors.
5 Discussion
The discussion identifies robustness boundaries for GRSD’s SSIM-based fork-point matching and limited action space, then proposes practical extensions for real-time transfer.
- Real-time execution and SSIM-based matching: SSIM can miss equivalent states or falsely match transiently different screens during asynchronous execution.Temporal misalignment lowers similarity, while cursor blinking, notifications, loading indicators, and clock updates can perturb pixels without changing semantic state.
- Practical implications for GRSD: Time-window matching, temporal smoothness, region masking, OCR/layout tokens, and accessibility cues could improve GRSD’s robustness efficiently.These additions target asynchronous execution artifacts and are especially relevant when transferring from offline traces to real-time deployment.
- Limited action space: AndroidWorld’s predefined high-level actions simplify exploration and stabilize training but may underestimate failures involving low-level gesture dynamics.Potentially omitted factors include gesture duration, trajectory shape, release timing, and actuation noise.
- Limited action space: Policies trained with limited actions may transfer less robustly to finer-grained controls or different action wrappers.The discussion suggests hierarchical action modeling that adds low-level gestures and perturbations during post-training.
6 Conclusion
UI-Voyager addresses inefficient learning from failed trajectories and ambiguous credit assignment through a two-stage self-evolving framework. On AndroidWorld, its 4B model reaches 81.0% pass@1 and exceeds reported baseline and human-level performance, while future work remains beyond AndroidWorld.
- UI-Voyager combines Rejection Fine-Tuning and Group Relative Self-Distillation to address failed-trajectory learning and sparse-reward credit assignment.
- RFT enables automatic data–model co-evolution, while GRSD uses fork-point detection to provide dense step-level supervision from successful trajectories.
- 81.0% pass@1 success across 116 AndroidWorld tasks surpasses all baselines, including larger models, and reported human-level performance.
- Ablation studies and case studies validate the effectiveness of UI-Voyager’s core designs.
- The paper considers only AndroidWorld and identifies extension to other GUI tasks, adaptive reasoning, self-correction, and real-world efficiency as future directions.