Source-linked AI summary
Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation
Pengxiang Li, Zechen Hu, Zirui Shang, Jingrong Wu, Yang Liu, Hui Liu, Zhi Gao, Chenrui Shi, Bofei Zhang, Zihao Zhang, Xiaochuan Shi, Zedong YU, Yuwei Wu, Xinxiao Wu, Yunde Jia, Liuyu Xiang, Zhaofeng He, Qing Li
TL;DR
GUI-agent RL is limited by slow multi-turn environment interaction and insufficient high-quality learning trajectories. DART decouples rollout, environment, data, and training operations while adaptively curating experiences, achieving 42.13% task success on OSWorld with improvements over the base model and open-source state of the art.
Problem
GUI-agent RL faces slow multi-turn rollouts and insufficient high-quality interactions for policy learning.
Method
DART uses four asynchronous modules and adaptive curation across tasks, trajectories, steps, and tokens to improve training efficiency and experience quality.
Results
42.13% task success on OSWorld yields a 14.61% absolute gain over the base model and a 7.34% improvement over open-source state of the art.
Takeaways & Limitations
DART-GUI-7B demonstrates improved GUI-agent performance alongside higher rollout GPU, training-throughput, and environment utilization.
Abstract
from arXiv · showhide
Vision-language model (VLM) based GUI agents show promise for automating complex desktop and mobile tasks, but face significant challenges in applying reinforcement learning (RL): (1) slow multi-turn interactions with GUI environments for policy rollout, and (2) insufficient high-quality agent-environment interactions for policy learning. To address these challenges, we propose DART, a Decoupled Agentic RL Training framework for GUI agents, which coordinates heterogeneous modules in a highly decoupled manner. DART separates the training system into four asynchronous modules: environment cluster, rollout service, data manager, and trainer. This design enables non-blocking communication, asynchronous training, rollout-wise trajectory sampling, and per-worker model synchronization, significantly improving the system efficiency: 1.6*GPU utilization for rollout, 1.9* training throughput, and 5.5* environment utilization. To facilitate effective learning from abundant samples, we introduce an adaptive data curation scheme: (1) pre-collecting successful trajectories for challenging tasks to supplement sparse success in online sampling; (2) dynamically adjusting rollout numbers and trajectory lengths based on task difficulty; (3) training selectively on high-entropy steps to prioritize critical decisions; (4) stabilizing learning via truncated importance sampling for policy mismatch between policy rollout and updating. On the OSWorld benchmark, DART-GUI-7B achieves a 42.13% task success rate, a 14.61% absolute gain over the base model, and 7.34% higher than open-source SOTA. We will fully open-source our training framework, data, and model checkpoints via computer-use-agents.github.io/dart-gui, which we believe is a timely contribution to the open-source community of agentic RL training.
1 INTRODUCTION
DART addresses modest RL gains for GUI agents by decoupling training components and curating informative experiences. On OSWorld, DART-GUI-7B improves task success while substantially increasing system utilization and throughput.
- Framework: DART separates environment, rollout, data-management, and training functions into asynchronous modules, enabling non-blocking execution and continuous policy updates.Distributed rollout workers and rollout-level trajectory sampling support the decoupled design.
- Efficiency: The framework improves rollout GPU utilization by 1.6×, training throughput by 1.9×, and environment utilization by 5.5× compared with coupled baselines.These gains target the idle time created by sequential action prediction, environment interaction, data management, and model updates.
- Data curation: Adaptive curation supplements difficult tasks with successful trajectories and adjusts rollout counts and trajectory lengths according to task difficulty.It also prioritizes high-entropy decision steps and uses truncated importance sampling to stabilize updates under policy mismatch.
- Results: 42.13% task success on OSWorld gives DART-GUI-7B a 14.61% absolute gain over the base model and a 7.34% improvement over open-source state of the art.The model is initialized from UI-TARS-1.5-7B.
2 RELATED WORK
Related work traces GUI agents across structured, visual, and hybrid architectures, while training has progressed from supervised learning and offline RL toward online feedback-based methods. Asynchronous RL systems address computational bottlenecks by decoupling rollout generation from training.
- GUI Agents: Structured, visual, and hybrid GUI agents trade metadata robustness, visual generality, and multimodal coverage differently.Structured agents depend on metadata quality, visual agents are sensitive to visual variation, and hybrid approaches combine both modalities.
- GUI-Agent Training: GUI-agent training has shifted from supervised fine-tuning toward reinforcement learning that learns from environmental feedback.Offline RL methods struggle with distribution shift and multi-turn reasoning, motivating recent online approaches.
- Agentic RL: Agentic RL has evolved from preference-based supervision toward outcome-based training and reinforcement learning with verifiable rewards.This shift reduces reliance on indirect human-preference signals.
- Asynchronous RL: Asynchronous RL architectures decouple training components to improve computational efficiency, including systems that separate rollout generation from model training.Recent frameworks also use mechanisms such as staleness-aware optimization and multi-model pipelines.
3 DART: DECOUPLED AGENTIC RL TRAINING FRAMEWORK
DART formulates GUI interaction as sequential decision-making and implements training through decoupled rollout, environment, data, and trainer components. Its asynchronous sampling and synchronization design targets the long, heterogeneous trajectories that otherwise underutilize resources.
- 3.1 FORMULATION: GUI tasks repeatedly map screenshots, interaction history, and task instructions to thoughts and executable actions until termination or a maximum step limit.The policy model πθ generates the thought and action, and each action produces an updated visual state.
- 3.2 DART ARCHITECTURE: The framework uses an environment cluster, rollout service, data manager, and trainer that operate without blocking one another.Hundreds of desktop environments generate N trajectories per task, while idle rollout workers are dynamically assigned to parallel executions.
- 3.3 ASYNCHRONOUS TRAINER: Asynchronous training lets the trainer consume filtered trajectories and update the policy while new trajectories are sampled, then synchronizes updated weights to the rollout service.This avoids blocking between training and rollout.
- 3.3 ASYNCHRONOUS TRAINER: Step-wise GRPO decomposes each trajectory into state-thought-action steps and groups steps from the same task for advantage computation.Each step is paired with its history and the trajectory-level reward.
- 3.3 ASYNCHRONOUS TRAINER: Heterogeneous task difficulty and trajectory lengths make conventional batch-wise sampling inefficient because early-completing environments remain idle.DART addresses this setting with rollout timelines that expose parallel execution across tasks and environments, plus a shared model-service pool for load balancing.
- 3.3 ASYNCHRONOUS TRAINER: Per-worker model updates avoid system-wide downtime by refreshing rollout-worker weights in a staggered manner while other workers continue serving inference requests.This keeps environments from experiencing a complete service block during synchronization.
4 MULTI-LEVEL ADAPTIVE DATA CURATION FOR GUI TASKS
The adaptive data curation strategy reallocates sampling toward difficult GUI tasks, preserves positive learning signals, focuses updates on uncertain steps, and corrects policy-distribution mismatch.
- 4.1 PERFORMANCE-AWARE TASK ROLLOUT: Rollout frequency decreases for tasks exceeding a 0.6 success rate, while low-success tasks retain maximum sampling.This reallocates computation from well-learned tasks to challenging ones.
- 4.1 PERFORMANCE-AWARE TASK ROLLOUT: Task-specific trajectory limits use historical successful-completion lengths to avoid hopelessly long rollouts while preserving exploration where longer solutions are needed.Unlike a fixed global limit, each task receives its own threshold.
- 4.1 PERFORMANCE-AWARE TASK ROLLOUT: Figure 5 relates dynamic rollout N to task success rate, illustrating the performance-aware adjustment of sampling frequency.The strategy reduces rollout frequency for already successful tasks and maintains maximum sampling for low-success tasks.
- 4.2 EXPERIENCE POOL OF TRAJECTORIES: Guaranteeing at least one positive trajectory per task addresses sparse successes and training instability on challenging GUI tasks.Successful trajectories are pre-collected and inserted when all current rollouts fail.
- 4.3 HIGH-ENTROPY-DRIVEN STEP OPTIMIZATION: Training prioritizes the top 80% highest-entropy steps because low-entropy GUI steps are treated as non-critical and potentially destabilizing.Step entropy averages token entropies across each concatenated thought-and-action sequence.
- 4.4 DISTRIBUTION ALIGNMENT FOR OOD TOKENS: Truncated importance sampling reweights gradient contributions using probability ratios to mitigate distribution gaps between rollout and training policies.The mismatch arises from differing quantization strategies and pre-collected trajectories.
5 EXPERIMENT
DART is evaluated on OSWorld through a decoupled single-VLM RL setup and adaptive data curation, achieving strong task performance and substantial efficiency gains.
- 5.2 MAIN RESULTS: 42.13% overall success rate establishes DART-GUI-7B as open-source state of the art on OSWorld with only 30 maximum steps.This exceeds UI-TARS-1.5-7B’s 27.52% with 100 steps and is comparable to Claude-4-Sonnet’s 41.39% with 100 steps.
- 5.2 MAIN RESULTS: 31.25% higher OS-task performance, 21.73% higher LibreOffice Writer performance, and 20.00% higher Thunderbird performance show gains on complex applications.The reported improvements are 62.50% versus 31.25%, 60.86% versus 39.13%, and 60.00% versus 40.00%, respectively.
- 5.3 EFFICIENCY ANALYSIS: 1.9× training throughput, 5.5× environment utilization, and 1.6× rollout GPU utilization demonstrate major efficiency improvements over the non-decoupled baseline.Training throughput rises from 22.6 to 43.6 actions/min, environment utilization from 12.2% to 67.7%, and GPU utilization from 29.6% to 46.7%.
- 5.4 ABLATION: Dynamic rollout and trajectory-length controls reduce redundant computation as accuracy improves while retaining exploration on challenging tasks.Average rollout frequency decreases from 8.0 to 5.0 per task, while average trajectory length drops from 30 to fewer than 10 steps.
- 5.4 ABLATION: The experience pool raises success on initially unsolved challenging tasks from 0% to 46% by inserting successful trajectories when online rollouts fail.The evaluated set contains 22 tasks with an initial success rate of 0%.
- 5.4 ABLATION: High-entropy step selection improves success rates from 28.67% to 68.33%, while distribution alignment reaches 70.55% and prevents baseline collapse.With alignment, accuracy remains around 70%, peaks at 78% versus 55%, and avoids the baseline’s decline to near 0% after step 60.
6 CONCLUSION
The paper presents DART as an efficient RL approach for VLM-based GUI agents, combining decoupled training with curated interaction data to improve utilization and performance.
- 6 CONCLUSION: DART addresses slow GUI interactions and limited high-quality training data through decoupled RL and multi-level data curation.The conclusion reports improved GPU and environment utilization alongside better agent performance.
REPRODUCIBILITY STATEMENT
The paper describes its architecture, experiments, ablations, and implementation details, and plans to release code, configurations, and pretrained models for reproducibility.
- REPRODUCIBILITY STATEMENT: The paper supports reproducibility with architectural and algorithmic descriptions, experimental protocols, implementation details, ablations, appendices, and planned public artifacts.Source code, configuration files, and pretrained models will be made available.
A.1 BROADER IMPACT
DART-GUI is positioned as a tool for accessibility, productivity, and UI validation, with open release intended to lower adoption barriers while recognizing security and privacy risks.
- A.1 BROADER IMPACT: The authors identify benefits for users with disabilities, repetitive-task automation, UI validation, and research access through open release, while acknowledging unauthorized-access and privacy concerns.The passage also connects efficient training with reduced environmental impact.
A.2 LLM USAGE STATEMENT
The authors report using LLMs only for language improvement during manuscript preparation, while human authors conducted the scientific work.
- LLMs were used exclusively to improve language quality, including grammar correction.
- The stated assistance also included enhancing the manuscript’s clarity.
- Human authors conducted the conceptualization, methodology, experiments, analysis, and interpretation.
A.3 SYSTEM PROMPT AND ACTION SPACE
This appendix describes DART-GUI’s system prompt, distributed training infrastructure, data-management design, task visualizations, and representative failure cases.
- A.3 SYSTEM PROMPT AND ACTION SPACE: The system prompt specifies DART-GUI’s action space and output format for GUI interaction.
- A.4 MORE IMPLEMENTATION DETAILS: Training uses FSDP through verl across 8 NVIDIA H100 GPUs, with specified learning-rate, KL, clipping, and rollout-scaling settings.
- A.4 MORE IMPLEMENTATION DETAILS: The rollout service uses vLLM, load balancing, worker-wise model syncing, two GPUs per worker, temperature 1.0, and 30-step episodes.
- A.4 MORE IMPLEMENTATION DETAILS: The environment cluster orchestrates 180 independent Ubuntu Docker containers through Kubernetes, while the dashboard visualizes the distributed setup.
- A.4 MORE IMPLEMENTATION DETAILS: A centralized MySQL Data Manager coordinates 11 interconnected tables covering model management, trajectory records, rewards, and task-linked data.
- A.4 MORE IMPLEMENTATION DETAILS: The Data Manager balances training samples by ensuring each task includes successful and failed trajectories, retrieving positives when all sampled trajectories fail.
- A.5 VISUALIZATION: Task visualizations compare baseline and DART-GUI decisions at pivotal steps, including successful behavior on extremely difficult tasks using trajectory-pool data.
- A.6 FAILURE CASES: Failure cases include incorrect privacy navigation and limitations in executing simultaneous Ctrl-click actions, alongside difficult-task examples involving Impress and bookmarks.