Source-linked AI summary
Where to Learn and How Far to Roll Out: Efficient Adaptive Training for GUI Agents
Yuxiang Chai, Quyu Kong, Panrong Tong, Steven Hoi, Yue Wang
TL;DR
Multi-step GUI-agent training can waste interaction replaying mastered prefixes and provides coarse task-level supervision about which decisions need improvement. This paper combines action-disagreement pivots, reusable checkpoints, and progress-adaptive Local, Suffix, and Root training; across two benchmarks, it reduces training costs while maintaining or improving performance.
Problem
Trajectory-level GUI-agent training gives coarse supervision over which intermediate decisions need improvement and repeatedly re-executes mastered prefixes before reaching difficult states.
Source: §1 · p. 1Method
The framework uses action-disagreement pivots and reusable checkpoints, then expands from local one-step updates to suffix and root-level training as task reliability improves.
Source: §1 · p. 2, §3.1 · p. 3, §3.3 · p. 5Results
Across MobileGym and CUA-Gym-Hub, the framework reduces training samples and environment interaction while maintaining or improving task performance; on MobileGym it reaches 63.86 Train Avg@4 with 40.8% fewer training samples and 50.2% fewer environment interactions than Action→Vanilla.
Source: §4.3 · p. 8, §5 · p. 9Takeaways & Limitations
Selectively allocating training across intermediate states and rollout ranges is an effective way to make GUI-agent reinforcement learning more efficient.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Trajectory-level GRPO is a common approach for training multi-step GUI agents, but repeatedly rolling out complete trajectories from the task root can be inefficient: task-level rewards provide coarse supervision over which intermediate decisions require improvement, while already-mastered prefixes are repeatedly re-executed before reaching difficult states. We introduce a two-stage adaptive training framework that jointly addresses where to learn and how far to roll out. We first construct reusable intermediate checkpoints from successful expert trajectories and select informative pivots using action disagreement, which measures uncertainty directly from sampled GUI actions. Stage I performs one-step RL at these pivots to improve uncertain local decisions. Stage II then progressively expands policy responsibility through adaptive Local, Suffix, and Root training according to task progress. Recoverable checkpoints enable repeated training from intermediate states without replaying the preceding trajectory. Across MobileGym and CUA-Gym-Hub, our method reduces environment interaction by 56-61% relative to Vanilla GRPO while improving task performance, demonstrating that jointly allocating training across states and rollout ranges can substantially improve GUI-agent RL efficiency.
1 INTRODUCTION
Trajectory-level GUI-agent training is inefficient because task-level outcomes provide coarse supervision and root-started rollouts replay prefixes the policy may already handle. The framework jointly chooses where to learn and how far to roll out using action-disagreement pivots, reusable checkpoints, and progress-aware training.
1 INTRODUCTION
- Trajectory-level training provides coarse supervision over intermediate decisions and repeatedly re-executes mastered prefixes before reaching states that still need learning.These inefficiencies make both credit assignment and environment interaction costly for multi-step GUI tasks.Source: §1 · p. 1
- Reusable checkpoints from successful expert trajectories provide reliable downstream states and local next-action references even when the current policy cannot consistently reach them.They support repeated intermediate-state training without replaying preceding trajectory prefixes.Source: §1 · pp. 1–2
- Action disagreement selects informative intermediate states by measuring uncertainty directly from sampled GUI actions.Stage I performs one-step RL at selected pivots to focus early optimization on uncertain local decisions.Source: §1 · p. 2
- The framework jointly adapts training location and rollout extent, using checkpoint-supported learning to improve efficiency and task performance across GUI environments.It progresses from local pivot updates to suffix and root-level training as policy capability improves, without replaying full trajectory prefixes.Source: §1 · p. 2
2 RELATED WORK
Prior GUI-agent work applies trajectory-level reinforcement learning and explores intermediate-state optimization, but often relies on complete interaction trajectories or chooses either training states or rollout branches. This paper studies joint adaptation of training location and rollout extent.
2.1 TRAINING GUI AGENTS WITH MULTI-TURN REINFORCEMENT LEARNING
- Multi-turn GUI reinforcement learning methods use complete environment trajectories and task-level outcomes, leaving environment sampling and fine-grained credit assignment costly as tasks lengthen.The paper frames budget allocation across intermediate decisions as complementary to improving the trajectory-level objective.Source: §2.1 · p. 3
2.2 LEARNING AND RECOVERING INTERMEDIATE AGENT STATES
- Intermediate-state methods select pivots or branch suffix rollouts using signals such as local reward variance, policy entropy, or identified errors.The paper distinguishes its focus by jointly adapting training location and rollout extent as policy capability changes.Source: §2.2 · p. 3
- Recovering GUI states by replaying action prefixes incurs repeated interaction cost and can be fragile under dynamic behavior.Existing approaches vary between snapshot-and-restore, resettable environments, and replay-based recovery.Source: §2.2 · p. 3
3 EFFICIENT ADAPTIVE TRAINING
The framework organizes training around where to learn and how far to roll out, using recoverable intermediate states and progressively longer training as task-solving ability improves. It combines action-disagreement pivot selection with a two-stage training procedure.
3.1 REUSABLE INTERMEDIATE STATES
- Each checkpoint bundles the task instance, interaction history, current observation, and recoverable executable environment state for repeated intermediate-state training.This avoids replaying the preceding trajectory to recover the policy context and environment.Source: §3.1 · p. 3
- 81.1% on MobileGym and 71.6% on CUA-Gym-Hub: checkpoint restoration reduces recovery latency versus replaying a 10-step prefix.Checkpoints can be restored repeatedly to generate independent rollout branches without re-executing the preceding trajectory.Source: §3.1 · pp. 3–4, Table 1 · p. 4
3.2 WHERE TO LEARN: SELECTING PIVOTS BY ACTION DISAGREEMENT
- Action disagreement selects expert-trajectory pivots where sampled GUI actions differ, including states whose samples all receive the same reward.States with disagreement above a threshold become pivots for local updates and suffix rollouts.Source: §3.2 · pp. 4–5
3.3 HOW FAR TO ROLL OUT: ADAPTIVE HYBRID TRAINING
- Root-level success initializes tasks in Local, Suffix, or Root training, then rollout responsibility expands as the current segment becomes reliable.Suffix training moves its frontier earlier after sufficient suffix success; Root tasks remain in the Root regime.Source: §3.3 · p. 5
- Local training uses one-step RL at selected pivots when root-level success is low, then re-evaluates tasks from the root as success improves.The authors motivate this choice by noting that long rollout groups can collapse to the same failure outcome while incurring substantial interaction cost.Source: §3.3 · p. 5
3.4 TWO-STAGE TRAINING
- The final method first warms up uncertain pivots with one-step updates, then applies adaptive hybrid training from the Stage-I policy.Stage II assigns tasks to Local, Suffix, or Root training and progressively moves suffix rollouts toward earlier pivots.Source: Figure 2 · p. 4, §3.4 · pp. 5–6
4 EXPERIMENTS
Experiments evaluate pivot selection and adaptive training on MobileGym and CUA-Gym-Hub. Action Disagreement offers a favorable accuracy-efficiency trade-off, while adaptive hybrid training reaches comparable or better performance with lower training and interaction costs than Vanilla GRPO.
4.1 EXPERIMENTAL SETUP
- The evaluation uses disjoint training and held-out task sets, begins from an SFT-initialized policy, and tests from task roots without expert prefixes or checkpoints.MobileGym uses 83 training tasks and 128 held-out tasks; CUA-Gym-Hub uses 80 and 60, respectively. Each task receives four evaluation rollouts, with Avg@4 and Pass@4 reported.Source: §4.1 · p. 6
4.2 WHERE TO LEARN: EVALUATING PIVOT SELECTION
- Action Disagreement provides a stronger overall accuracy-efficiency trade-off than Reward Variance on both benchmarks, and reaches 55.00% Held-Out Avg@4 and 65.00% Pass@4 on CUA-Gym-Hub with 3.1k rollout samples.On MobileGym, it reaches 48.83% Held-Out Avg@4 with 8.2k samples, compared with All Pivots’ 50.00% after 38.4k samples.Source: Figure 3 · p. 7, §4.2 · p. 7
4.3 FULL TRAINING: EFFECTIVENESS AND EFFICIENCY
- Base→Hybrid reaches comparable or better task performance than Base→Vanilla with substantially fewer training samples and rollout steps across both benchmarks.On MobileGym, it reaches 58.43 Train Avg@4 with 17.9k samples and 25.0k steps, versus 43.3k samples and 71.5k steps for Vanilla; on CUA-Gym-Hub, it reaches 63.75 Train Avg@4 with 9.0k samples and 9.2k steps, versus 33.3k samples and 40.5k steps.Source: Figure 4 · p. 8, §4.3 · p. 8
- Action→Hybrid improves on Action→Vanilla in training and held-out results, while using fewer resources on MobileGym and reaching stronger held-out performance on CUA-Gym-Hub.On MobileGym, it reaches 63.86 Train Avg@4 using 40.8% fewer samples and 50.2% fewer environment interactions than Action→Vanilla; on CUA-Gym-Hub, it reaches 57.92 Held-out Avg@4 versus 53.33.Source: §4.3 · p. 8, Table 3 · p. 9
- For Action→Hybrid, Root training grows while Suffix training declines over training samples on both benchmarks, consistent with a progression toward full-task execution.The Root share rises from 31.96% to 82.29% on MobileGym and from 53.78% to 86.97% on CUA-Gym-Hub; the Suffix share falls from 36.12% to 6.65% and from 27.66% to 0%, respectively.Source: Figure 5 · p. 9, §4.3 · p. 9
- On CUA-Gym-Hub, Action→Hybrid and SFT both reach 70.00 Train Avg@4, but Action→Hybrid attains higher Held-Out Avg@4 and Pass@4.Held-Out Avg@4 is 57.92 for Action→Hybrid versus 50.83 for SFT, and Pass@4 is 66.67 versus 58.33.Source: §4.3 · p. 9
5 CONCLUSION
The framework jointly adapts where GUI-agent training starts and how far rollouts extend. Across MobileGym and CUA-Gym-Hub, it reduces training costs while maintaining or improving task performance.
5 CONCLUSION
- Across MobileGym and CUA-Gym-Hub, the framework reduces optimization samples and environment interaction while maintaining or improving task performance.Source: §5 · p. 9
- The framework combines action-disagreement pivot selection, reusable intermediate checkpoints, and adaptive Local, Suffix, and Root training.Source: §5 · p. 9
- The approach selectively allocates training across intermediate states and rollout ranges to make GUI-agent RL more efficient.Source: §5 · p. 9
A.1 ENVIRONMENT CHECKPOINTING
Container-level checkpointing preserves runtime state and supports branching rollouts from intermediate GUI states. Restoration checks detect observable failures but do not establish complete state equivalence or guarantee identical future execution.
A.1 ENVIRONMENT CHECKPOINTING
- Podman and CRIU capture running process state, including browser memory and the JavaScript heap, alongside container filesystem changes.This preserves runtime information beyond screenshots or browser storage alone.Source: §A.1 · p. 11
- A saved checkpoint can initialize multiple independent rollout branches without repeating earlier actions.Checkpoints are captured between actions; restoration includes execution metadata and agent context, and closes existing TCP connections.Source: §A.1 · p. 11
- Health checks, screenshot capture, and consistency tests identify observable restoration failures but do not prove complete state equivalence or identical future execution.Asynchronous events or external services can prevent identical future execution.Source: §A.1 · p. 11
A.2 ACTION DISAGREEMENT AND LOCAL RL
The method groups GUI actions by task-specific matching rules, then uses sampled action disagreement to identify uncertain pivots. At those pivots, local one-step rewards compare sampled actions with the expert next action.
A.2 ACTION DISAGREEMENT AND LOCAL RL
- Coordinate actions match within 5% of screen width and height, swipes by discretized direction, and other actions by type, with text-entry content ignored.The rule treats minor coordinate or textual variation as the same GUI decision.Source: §A.2 · p. 11
- With N = 8 samples and τdis = 0.45, disagreement scores of 0.429 for a 6–2 split are filtered out, while 0.464 for 6–1–1 and 0.536 for 5–3 are retained.Table 4 lists representative action-count patterns and empirical disagreement scores.Source: §A.2 · p. 11, Table 4 · p. 12
- At a pivot, local one-step reward compares a policy sample with the expert next action using the action-matching function.The expert prefix conditions the policy at the pivot; optimization is performed on the newly sampled response.Source: §A.2 · p. 11
A.3 ADAPTIVE HYBRID TRAINING ALGORITHM
Hybrid training expands its rollout frontier monotonically as task progress improves, moving from the deepest pivot toward the task root.
A.3 ADAPTIVE HYBRID TRAINING ALGORITHM
- Suffix training begins at the deepest pivot and moves one pivot earlier when the current suffix success rate exceeds τsuffix.In the experiments, τlow = 0.25, τhigh = 0.75, and τsuffix = 0.5.Source: §A.3 · p. 12
- Once the rollout frontier reaches the task root, training remains in the Root regime rather than returning to intermediate pivots.Source: §A.3 · p. 12
- The method orders pivots by their positions in the expert trajectory, with cK denoting the deepest pivot.Source: §A.3 · p. 12
A.4 AI ASSISTANCE STATEMENT
The authors used generative AI for writing and language polishing, literature retrieval, and synthetic-data generation, while reviewing and validating its contributions and taking responsibility for the paper’s final content and claims.
A.4 AI ASSISTANCE STATEMENT
- Generative AI assisted with writing and language polishing, literature retrieval and discovery, and synthetic-data generation in the research workflow.Source: §A.4 · p. 12
- Authors reviewed and revised AI-assisted text and manually verified retrieved literature against original sources.Source: §A.4 · p. 12
- AI-generated experimental data was inspected and validated, and the authors take full responsibility for the final content, results, and claims.Source: §A.4 · p. 12