Source-linked AI summary
ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay
Fanbin Lu, Zhisheng Zhong, Shu Liu, Chi-Wing Fu, Jiaya Jia
TL;DR
Training GUI agents requires optimizing long-horizon, multimodal interactions despite sparse rewards, delayed feedback, and expensive rollouts. The paper proposes ARPO, combining GRPO with task selection and successful-trajectory replay, and evaluates it on OSWorld, where reinforcement learning improves in-domain performance but generalization remains limited.
Problem
End-to-end reinforcement learning for vision-language GUI agents remains underexplored because GUI tasks provide sparse, delayed rewards and costly rollouts.
Method
ARPO augments GRPO with baseline-informed task selection and a per-task replay buffer that reuses successful trajectories during training.
Results
Reinforcement learning substantially improves in-domain GUI-agent performance on OSWorld, while offering limited benefit on out-of-domain agentic tasks.
Takeaways & Limitations
Rule-based rewards, careful task selection, and experience replay can support end-to-end policy optimization for multi-turn, multimodal GUI agents.
Takeaways & Limitations
Strong generalization still depends on broader task diversity, carefully designed reward signals, and larger-scale training compute.
Abstract
from arXiv · showhide
Training large language models (LLMs) as interactive agents for controlling graphical user interfaces (GUIs) presents a unique challenge to optimize long-horizon action sequences with multimodal feedback from complex environments. While recent works have advanced multi-turn reinforcement learning (RL) for reasoning and tool-using capabilities in LLMs, their application to GUI-based agents remains relatively underexplored due to the difficulty of sparse rewards, delayed feedback, and high rollout costs. In this paper, we investigate end-to-end policy optimization for vision-language-based GUI agents with the aim of improving performance on complex, long-horizon computer tasks. We propose Agentic Replay Policy Optimization (ARPO), an end-to-end RL approach that augments Group Relative Policy Optimization (GRPO) with a replay buffer to reuse the successful experience across training iterations. To further stabilize the training process, we propose a task selection strategy that filters tasks based on baseline agent performance, allowing the agent to focus on learning from informative interactions. Additionally, we compare ARPO with offline preference optimization approaches, highlighting the advantages of policy-based methods in GUI environments. Experiments on the OSWorld benchmark demonstrate that ARPO achieves competitive results, establishing a new performance baseline for LLM-based GUI agents trained via reinforcement learning. Our findings underscore the effectiveness of reinforcement learning for training multi-turn, vision-language GUI agents capable of managing complex real-world UI interactions. Codes and models:https://github.com/dvlab-research/ARPO.git.
1 Introduction
GUI agents have mainly learned to imitate trajectories through supervised fine-tuning, but sparse feedback and costly long-horizon interaction motivate end-to-end reinforcement learning. The paper introduces GRPO-based training with task selection and experience replay, reporting gains on OSWorld.
- Prior GUI agents typically use supervised fine-tuning on trajectory datasets, which can cause error accumulation and limit self-correction.The model predicts the next action from the current screenshot and interaction history while imitating human behavior.
- GRPO provides a value-function-free policy optimization method suited to long sequences and multimodal vision-language agents.It estimates token-level advantages through group-wise reward normalization.
- GUI reinforcement learning must address sparse, delayed rewards and costly desktop rollouts, which complicate early policy optimization.Agents may receive feedback only at task completion, while operating-system delays slow data collection.
- ARPO combines GRPO with baseline-informed task selection and a per-task replay buffer storing successful trajectories.Replay trajectories are injected when all sampled rollouts fail, preserving reward variance for advantage estimation.
- On OSWorld, reinforcement learning improves in-domain performance but provides limited benefit on out-of-domain agentic tasks.The paper identifies task selection, experience replay, and end-to-end GRPO optimization as core contributions.
2 Related Works
Prior GUI-agent research has advanced multimodal perception, executable action generation, and interface pretraining, while reinforcement learning has mainly targeted symbolic or static tool-use settings. The paper addresses the remaining gap of end-to-end policy optimization for vision-language agents in dynamic GUI environments.
- Recent GUI agents use vision-language models for interactive interface tasks, while related systems improve screen understanding through web and desktop data.Examples include SeeClick, ScreenAgent, CogAgent, and UI-Tars.
- End-to-end direct policy optimization for GUI agents remains unexplored in the cited research landscape.
- Existing reinforcement-learning agent work largely emphasizes symbolic tasks or static tool use rather than dynamic multimodal GUI interaction.The paper instead targets rule-based rewards from live desktop environments for multi-turn policy optimization.
3 Method
The method trains a vision-language GUI agent with GRPO in a multi-turn MDP, using full interaction histories, parallel rollouts, task filtering, and replayed successful trajectories to address sparse rewards and costly exploration.
- Multi-turn GUI Agent: The GUI agent models each trajectory as screenshot observations, mouse or keyboard actions, and a terminal scalar reward in a multi-turn MDP.Its policy is optimized to maximize trajectory rewards.
- Multi-turn GUI Agent: The model tokenizes the entire screenshot-and-action history, using up to 15 images and a 64K context to reason over long-term dependencies.This avoids truncating trajectories to only the most recent screenshots.
- Trajectory Collection: Distributed rollout workers interact with live environments in parallel and batch visual observations through a centralized inference server to reduce decision latency.Parallelization addresses OS-level delays in environments such as OSWorld.
- GRPO: GRPO optimizes vision-language GUI agents without a value function by computing token-level advantages from group-normalized rewards.The agent uses GRPO to support multiple image inputs and extended context lengths.
- Valuable Tasks Selection: The method filters OSWorld tasks by retaining those solved at least once in 16 UI-Tars-1.5 rollouts, producing 128 tasks with informative early rewards.This targets sparse feedback from tasks that current agents cannot reliably solve.
- Experience Replay: ARPO stores successful trajectories per task and replaces one all-failure rollout with a replayed success, preventing zero advantages and preserving learning signals.Dynamic sampling alone is less effective when successful GUI rollouts are rare and expensive to obtain.
4 Experiments
The experiments use OSWorld, a real-computer benchmark spanning diverse open-ended GUI tasks, with rule-based rewards and a specified evaluation protocol.
- Training Details: Training samples 128 OSWorld tasks, uses 256 parallel virtual environments with eight rollouts per task, and runs for 15 epochs.Rollouts use batch size 32 and temperature 1.0 to encourage exploration.
- Datasets and Benchmarks: OSWorld contains 369 tasks across office productivity, web browsing, system management, and multi-application workflows in virtual machines.Tasks are evaluated with execution-based scripts while agents interact through real applications.
- Evaluation Metrics: Each trajectory receives a rule-based scalar reward between 0 and 1.0 from the environment under the standard OSWorld evaluation protocol.The paper notes that replacing the final action with FAIL at the step limit can distort rewards for impossible tasks.
4.2 Experimental Results
ARPO improves OSWorld performance over the base model and shows that replay benefits in-domain results, while the supplied figure references identify GRPO and ARPO ablations.
- Overall Evaluation: 29.9% on OSWorld and 23.8% on OSWorld Hard are achieved by ARPO with UI-Tars-1.5, improving over the base by 6.4% and 5.6%.All models are evaluated with a maximum trajectory length of 15 steps.
- Overall Evaluation: 20.4% is achieved by UI-Tars-7B-DPO with ARPO, up from 15.6% without ARPO.The paper reports consistent gains across earlier model versions.
4.3 Ablation on the Replay Buffer
The replay buffer improves reward and in-domain success, while out-of-domain gains remain modest and broader generalization depends on task diversity, reward design, and compute.
- Replay-buffer impact: 0.75 vs. 0.65: Replay produces a higher average trajectory reward by retaining successful trajectories for later updates.The buffer maintains reward diversity and non-zero advantages, supporting more stable optimization and faster convergence in sparse-reward GUI environments.
- Replay-buffer impact: 81.25% vs. 68.8%: ARPO raises in-domain task success by 12.5 percentage points over GRPO.The improvement is reported on the in-domain evaluation in Fig. 4.
- Generalization: 56.3% vs. 55.2%: ARPO slightly exceeds the base UI-Tars-1.5 model on out-of-domain tasks, after GRPO reaches 52.08%.The in-domain comparison is 81.25% for ARPO, 68.8% for GRPO, and 43.8% for the base model.
- Generalization: Strong generalization depends on broader task diversity, carefully designed reward signals, and larger-scale training compute.The paper contrasts strong in-domain improvement with limited out-of-domain benefit.
4.5 Valuable Task Selection for GRPO Training
Selecting valuable tasks improves GRPO training by increasing reward quality, convergence speed, and within-group reward diversity.
- Task selection: Training on a curated subset of 128 valuable tasks yields higher average trajectory rewards and faster convergence than training on the full task set.The advantage appears from the early stages of training.
- Task selection: Higher within-group reward variance in the curated task set supplies the diversity GRPO needs to compute token-level advantages.The full task set produces flatter reward distributions with reduced variance.
4.6 Comparison with Offline Preference Optimization
ARPO and GRPO outperform offline preference optimization methods under equal task and rollout budgets, with ARPO achieving the highest score.
- Interpretation: Direct trajectory-level optimization with rule-based rewards provides stronger learning signals than offline preference modeling in this comparison.ARPO’s replay buffer further enhances stability and sample efficiency in sparse-reward GUI settings.
4.7 Rollout Efficiency Analysis
Parallel environments substantially reduce epoch-level rollout time despite increasing the duration of each trajectory batch.
- Rollout efficiency: Over 6 hours to around 1.2 hours: Increasing parallel environments from 8 to 256 sharply reduces total epoch sampling time.Rollout time per batch simultaneously increases from 3 minutes to 19 minutes.
- Rollout efficiency: Scaling to 256 environments enables high-throughput rollouts for reinforcement learning in real desktop settings.The speedup is attributed to more efficient GPU inference and overlapping OS-level delays across environments.
4.8 Qualitative Analysis: Self-Correction Behavior in GUI Agent
ARPO-trained GUI agents can detect and correct interaction mistakes, improving success on the illustrated task from 25% to 62.5%.
- 25% vs. 62.5%: ARPO raises the specific task’s success rate before and after training.The agent also recognizes an incorrect superscript-button choice and uses Ctrl-Z to revert it.
5 Conclusion
The paper presents ARPO as reinforcement-learning-based end-to-end policy optimization for multimodal GUI agents, using longer context and multi-turn screenshot processing. It reports improved learning stability from task selection and identifies broader task coverage, longer contexts, and learned reward models as future directions.
- ARPO applies reinforcement learning to vision-language GUI agents using longer input context and multi-turn, multi-modal screenshot processing.
- Careful task selection improves learning stability and reward variance during training.
- Future work includes broader real-world task coverage, longer agent context, and learned reward models to reduce reliance on manually crafted rewards.