PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Milestone Reward for GUI Agents

Congmin Zheng, Xiaoyun Mo, Xinbei Ma, Qiqiang Lin, Yin Zhao, Jiachen Zhu, Xingyu Lou, Jun Wang, Zhaoxiang Wang, Weiwen Liu, Zhuosheng Zhang, Yong Yu, Weinan Zhang

arXiv:2602.11524v1cs.LGcs.AIcs.CL

TL;DR

Long-horizon Mobile GUI tasks make temporal credit assignment difficult because outcome rewards are sparse while process rewards can be biased and hacked. ADMIRE uses adaptive, verifiable milestones with asymmetric credit assignment to denoise successful trajectories and scaffold failed ones. Experiments report over 10% improvement in success rate across base models on AndroidWorld, with generalization across algorithms and heterogeneous environments.

  • Problem

    Long-horizon Mobile GUI tasks require credit assignment over sparse, delayed feedback, while outcome and process rewards trade off fidelity against density and resistance to bias or reward hacking.

  • Method

    ADMIRE dynamically distills verifiable milestones from successful explorations and uses asymmetric credit assignment to denoise successful trajectories and provide partial-credit scaffolding for failed ones.

  • Results

    Over 10% improvement in success rate is reported across different base models on AndroidWorld, with gains across diverse RL algorithms and environments including ALFWorld and WebShop.

  • Takeaways & Limitations

    ADMIRE provides dense, high-fidelity feedback and generalizes across base models, RL algorithms, and cross-domain tasks within the reported evaluations.

  • Takeaways & Limitations

    ADMIRE depends on the reasoning quality of the VLM used to generate milestones, and its verifiable milestones are currently used only for auxiliary rewards rather than proxy outcome signals in open-ended environments.

Abstract

from arXiv · show

Reinforcement Learning (RL) has emerged as a mainstream paradigm for training Mobile GUI Agents, yet it struggles with the temporal credit assignment problem inherent in long-horizon tasks. A primary challenge lies in the trade-off between reward fidelity and density: outcome reward offers high fidelity but suffers from signal sparsity, while process reward provides dense supervision but remains prone to bias and reward hacking. To resolve this conflict, we propose the Adaptive Milestone Reward (ADMIRE) mechanism. ADMIRE constructs a verifiable, adaptive reward system by anchoring trajectory to milestones, which are dynamically distilled from successful explorations. Crucially, ADMIRE integrates an asymmetric credit assignment strategy that denoises successful trajectories and scaffolds failed trajectories. Extensive experiments demonstrate that ADMIRE consistently yields over 10% absolute improvement in success rate across different base models on AndroidWorld. Moreover, the method exhibits robust generalizability, achieving strong performance across diverse RL algorithms and heterogeneous environments such as web navigation and embodied tasks.

1 Introduction

Mobile GUI agents face temporal credit assignment in long-horizon tasks because outcome rewards are sparse while process rewards can be biased and hacked. ADMIRE addresses this tension with adaptive, verifiable milestones and asymmetric credit assignment, and shows broad empirical gains.

  • Motivation: RL training for long-horizon Mobile GUI tasks struggles to attribute final outcomes to specific actions in sparse, delayed-feedback sequences.This is the temporal credit assignment problem motivating the proposed reward mechanism.
  • Reward limitations: Outcome rewards preserve fidelity but collapse complex trajectories into sparse binary feedback, hindering recognition of near-successful explorations.Process rewards are dense but vulnerable to systemic bias and reward hacking, potentially rewarding executable yet futile actions.
  • ADMIRE: ADMIRE identifies key state transitions in successful explorations as adaptive milestones that trigger rewards through verifiable rule-based matching.Milestones update dynamically to reflect superior behaviors discovered during exploration.
  • ADMIRE: ADMIRE restricts positive incentives to milestone-triggering steps on successful trajectories while assigning dense partial credit through intermediate milestones on failed trajectories.This asymmetric strategy filters process noise from successes and scaffolds exploration from failures.
  • Results: Over 10% improvement in success rate is reported on AndroidWorld across different base models, with gains also reported across RL algorithms and cross-domain environments.The reported cross-domain environments include ALFWorld and WebShop.
  • Contributions: The paper presents ADMIRE, asymmetric credit assignment, and extensive evaluations as contributions aimed at dense, high-confidence feedback and generalization in long-horizon scenarios.Evaluations include AndroidWorld, MobileMiniWob++, and cross-domain tasks.

2 Preliminary

The mobile GUI task is modeled as a partially observable Markov decision process in which agents act from observations and interaction history. Episodes proceed through environment transitions until completion or a step limit, producing a trajectory and binary outcome.

  • POMDP formulation: The mobile GUI task is formulated as a partially observable Markov decision process defined by a tuple of state, action, observation, transition, terminal, and reward components.The formulation captures incomplete access to underlying system state and reward-based evaluation.
  • Observations and history: The agent cannot fully access latent system states, so it receives interface observations consisting of the high-level instruction and current screenshot.Interaction history is also used to mitigate partial observability when selecting actions.
  • Interaction dynamics: At each step, the agent executes an interface action, the environment transitions to a new latent state, and a new observation is emitted.The policy maps interaction history to an action distribution.
  • Trajectories: An episode ends when the task is completed or a predefined maximum step limit is reached, forming a complete trajectory.The trajectory is the accumulated interaction history at the terminal step.
  • Outcome evaluation: Each trajectory receives a binary outcome score, where 1 indicates success and 0 indicates failure.The reward function evaluates agent performance using this outcome.

3 Methodology

ADMIRE builds an objective, adaptive reward system around milestones distilled from successful interactions, then assigns credit asymmetrically to denoise successes and scaffold failures. Semantic matching and sequential verification trigger milestone rewards, which are integrated into policy optimization with curriculum decay.

  • Framework: ADMIRE comprises adaptive milestone generation, reward assignment, and policy optimization for long-horizon mobile GUI tasks.The framework is designed to address credit assignment through an objective, adaptive reward system.
  • Adaptive Milestone Generation: Milestones are dynamically distilled from successful trajectories and refined when newly discovered paths represent more optimal strategies.An LLM-based abstraction function extracts critical checkpoints, while refinement updates milestones to accommodate superior behaviors such as shortcuts.
  • Semantic Matching and Verification: Semantic matching compares action and milestone embeddings, while a sequential pointer allows rewards only for the next uncompleted milestone above threshold δ.The pointer advances only when s(a_t,m_p_t) > δ, preventing out-of-order milestone skipping.
  • Asymmetric Credit Assignment: Successful trajectories receive rewards only at milestone-hit steps, filtering redundant actions and emphasizing critical decisions.This denoising strategy uses milestone-hit steps T_mil to filter non-essential actions.
  • Asymmetric Credit Assignment: Failed trajectories receive dense scaffolding through continuous progress rewards plus milestone bonuses, validating partial successes.The scheme uses k/K progress and milestone bonuses to break the all-or-nothing reward pattern and lower exploration barriers.
  • Policy Optimization: ADMIRE aggregates outcome success, format validity, and milestone guidance in step-level policy optimization, while dense milestone rewards decay with training epoch.The curriculum coefficient λ0 · γ^E gradually shifts optimization toward outcome reward.

4 Experiment

The experiments evaluate ADMIRE on mobile GUI benchmarks using Qwen2.5-VL agents and Success Rate, with comparisons against outcome/process rewards and strong proprietary and open-source baselines. ADMIRE improves performance across settings, transfers to MobileMiniWob++, and achieves competitive results against larger models.

  • Experimental Setup: Experiments evaluate Success Rate on AndroidWorld and MobileMiniWob++ using Qwen2.5-VL-3B and Qwen2.5-VL-7B agents.The agents use Android Virtual Device workers running the AndroidWorld sandbox for scalable online trajectory collection.
  • Experimental Setup: ADMIRE is compared with outcome and process reward baselines, GPT-4o, UI-TARS-1.5-7B, and GLM-4V-9B-Thinking.The evaluation includes both reward-mechanism baselines and prominent proprietary and open-source models.
  • Overall Performance: 9.2% average success rate gain is achieved by ADMIRE with Qwen2.5-VL-3B, exceeding outcome reward at 6.1% and process reward at 5.8%.With Qwen2.5-VL-7B, ADMIRE reaches the highest average success rate of 52.6%.
  • Overall Performance: ADMIRE consistently outperforms base models and traditional reward mechanisms across all reported settings and improves on both AndroidWorld and MobileMiniWob++.The cross-benchmark gains are reported as evidence that the method does not overfit to a specific task distribution.
  • Overall Performance: 44.0% success rate is achieved by ADMIRE with Qwen2.5-VL-7B on AndroidWorld, surpassing GLM-4.1V-9B-Thinking and the larger Qwen2.5-VL-72B.ADMIRE with Qwen2.5-VL-3B reaches 31.0%, competitive with 7B-scale MobileGUI-7B.

5 Analysis

ADMIRE improves performance across task difficulties, reward designs, models, algorithms, and environments by combining adaptive milestones with asymmetric reward assignment. Experiments show benefits for long-horizon tasks, transfer across models, and cross-domain generalization.

  • Performance Across Varying Task Difficulties: ADMIRE reaches 19.0% success on Hard AndroidWorld tasks, compared with 9.5% for Outcome Reward and 14.3% for the base model.Process Reward shows no improvement over the base model on these complex tasks.
  • Performance Across Varying Task Difficulties: ADMIRE achieves 60.3% on Easy tasks and 28.1% on Medium tasks, exceeding the reported Outcome and Process Reward results.Outcome and Process Reward reach 52.4% and 47.6% on Easy tasks, while comparative methods plateau at 21.9% on Medium tasks.
  • Effect of Adaptive Milestones: ADMIRE consistently outperforms static milestone approaches because milestones co-evolve with the agent policy and remain aligned with emerging efficient strategies.The analysis contrasts adaptive milestones with human-annotated and fixed 7B-derived alternatives.
  • Effect of Adaptive Milestones: Fixed milestones derived from a converged 7B model produce a 5.9% gain when training a 3B model, indicating portability across model sizes.The result supports transfer of functional states and task structures captured by the stronger model.
  • Impact of Reward Components: Removing the base progress reward from failed trajectories causes the largest degradation, showing that milestone signals alone provide insufficient guidance for failures.The base reward supplies partial-success information when the final outcome remains zero.
  • Generalizability Analysis: ADMIRE generalizes across ALFWorld and WebShop and remains compatible with GRPO, RLOO, and DAPO, including 87.5% on ALFWorld with DAPO.With GRPO, ADMIRE reaches 81.9% on WebShop and 78.1% on ALFWorld; DAPO-ADMIRE reaches 78.1% on WebShop.
  • Generalizability Analysis: ADMIRE incurs negligible computational overhead, produces human-validated milestones, and rapidly achieves milestone coverage for most tasks.The reported auxiliary analyses cover hyperparameter robustness, milestone quality, and coverage.

6 Related Work

Mobile GUI agents have advanced in perception and interaction but still struggle with complex long-horizon tasks. Existing RL training is limited by sparse binary outcomes, while process reward methods introduce dense supervision with potential reliability concerns.

  • Mobile GUI Agents: Mobile GUI agents continue to struggle with complex, long-horizon tasks despite improved human-like gesture capabilities.The related-work discussion motivates post-training approaches that improve robustness and generalization.
  • Reward Design: Sparse binary outcomes make it difficult for agents to learn efficient policies without dense, high-quality feedback.This limitation motivates adaptive, verifiable reward shaping for GUI-agent training.
  • Reward Design: Process Reward Models evaluate intermediate steps to provide granular supervision, and GUI-specific methods apply them to navigation.The passage cites GUI-Shepherd for step-by-step feedback and GUI-PRA for process-reward generation.

7 Conclusion

ADMIRE addresses temporal credit assignment by combining adaptive, verifiable milestones with asymmetric credit assignment. It improves learning on AndroidWorld and generalizes across algorithms and domains, including results where 7B models outperform 72B baselines.

  • Conclusion: ADMIRE bridges sparse outcome signals and noisy process supervision through dynamic milestone generation and asymmetric credit assignment.The method denoises successful trajectories while scaffolding failed attempts with dense, high-fidelity feedback.
  • Conclusion: ADMIRE significantly enhances learning efficiency on AndroidWorld and enables 7B-parameter models to outperform 72B baselines.The conclusion also reports robustness across ALFWorld and WebShop.

Limitations

ADMIRE’s effectiveness depends on the quality of the VLM generating milestones and is currently limited to auxiliary reward assignment, leaving open-ended outcome signaling unexplored. The reported setup also uses substantial distributed infrastructure and fixed training configurations.

  • Limitations: ADMIRE depends on the reasoning capabilities of the VLM used to generate milestones, despite milestone extraction being framed as easier than precise execution.The paper identifies generator quality as an inherent limitation.
  • Limitations: ADMIRE uses verifiable milestones only for auxiliary reward assignment, leaving their use as proxy outcome signals unexplored in open-ended environments without rule-based success detection.This limits the current framework’s applicability beyond settings with programmatic success checks.
  • Implementation: Training runs online across 32 remote Android emulators and uses 8 NVIDIA A800 GPUs to maximize data throughput.The implementation describes a distributed infrastructure requirement for online training.
  • Implementation: Each iteration samples 4 tasks and collects 8 trajectories, while trajectory length, prompt length, and response length are capped at 20 steps, 6500 tokens, and 512 tokens.These caps constrain the reported online training configuration.
  • Implementation: The reward configuration fixes δ = 0.75, ζ = 0.5, η = 0.5, λ0 = 0.3, γ = 0.99, and the process-reward weight at 0.3.These values define the reported reward setup rather than a general guarantee across environments.
  • Implementation: Optimization uses a 1 × 10−6 learning rate, PPO clip ratio 0.2, two epochs per iteration, mini-batches of 128, advantage normalization, and no KL or entropy regularization.The reported optimization recipe is specific to the described training setup.

A.2.1 Benchmarks

The paper evaluates ADMIRE across mobile GUI, web, embodied, and model-baseline settings, using benchmarks that vary in task structure, interaction scope, and success criteria. AndroidWorld provides the primary dynamic mobile evaluation with difficulty categories based on human assessment.

  • Benchmarks: AndroidWorld contains 116 multi-step tasks across 20 Android applications, with programmatic success checks and Easy, Medium, and Hard difficulty levels.It uses a Pixel 6 Android 13 emulator and seeded task templates for reproducibility.
  • Benchmarks: MobileMiniWoB++ contains 92 tasks in one simulated application and emphasizes localized interaction among densely arranged UI elements.Its structure differs from multi-page navigation benchmarks.
  • Benchmarks: ALFWorld evaluates multi-step language reasoning and decision-making in household environments across 3,827 task instances with text-based goals.Success depends on translating instructions into grounded actions that complete state-based objectives.
  • Benchmarks: WebShop evaluates language grounding and decision-making across 1.18 million products and 12,087 instructions using automated attribute-level reward alignment.Agents must navigate web interfaces, reformulate queries, and select matching products.
  • Reward Baselines: Outcome reward provides a binary terminal signal, whereas Process Reward assigns a scalar score at every trajectory step through an LLM-as-a-Judge.These are the principal reward baselines used for comparison.
  • Model Baselines: The comparison set includes GUI agents and multimodal models such as UI-TARS-1.5-7B, Qwen2.5-VL-72B, GUI-Shepherd, GLM-4.1V-9B-Thinking, and MobileGUI-7B.The listed baselines span GUI agents, process-reward systems, and vision-language models.

B.2 Dynamic and Static Milestones Hit Count Analysis

The milestone hit-count analysis contrasts fixed and adaptive milestones during training. Static milestones become concentrated in early stages, whereas dynamic milestones remain consistently activated across all four stages.

  • Analysis Setup: The case study tracks intermediate milestone hits on the MarkorDeleteNote task to examine how milestone adaptivity influences training dynamics.The analysis provides a microscopic view of dynamic versus static milestone behavior.
  • Static Milestones: Static milestone hits concentrate heavily in the first two milestones and decline sharply for later milestones during training.The pattern is attributed to policy trajectories drifting away from a fixed milestone path.
  • Dynamic Milestones: Dynamic milestones maintain consistently high activation across all four task stages, including late training phases.Updating milestones to match efficient trajectories keeps supervision dense throughout the long-horizon episode.
  • Dynamic Milestones: ADMIRE’s dynamic milestone updates bridge the gap between the reward structure and evolving agent behavior.The reported mechanism is intended to prevent signal attenuation as exploration trajectories change.

B.3 Definitions of Ablation Variants

The ablation and auxiliary analyses examine asymmetric reward components, reward-weight scheduling, matching quality and efficiency, training overhead, and milestone reliability. Together, they characterize how ADMIRE’s supervision design behaves under alternative configurations and operational constraints.

  • Ablation Variants: The successful-trajectory ablation adds continuous base progress reward k/K to test whether dense supervision helps when outcome feedback is already available.Standard ADMIRE instead uses sparse milestone rewards for successful trajectories to reduce noise.
  • Ablation Variants: The failed-trajectory ablation removes the base reward k/K, eliminating continuous partial-success feedback and leaving only sparse milestone hits.Standard ADMIRE uses this component as scaffolding for failed trajectories.
  • Ablation Variants: The no-decay ablation sets γ = 1, keeping milestone-reward weight constant instead of shifting emphasis toward outcome reward over training.This tests the necessity of the time-dependent curriculum coefficient λ(t).
  • Reward Weight: 44.0% is the peak AndroidWorld success rate at λ0 = 0.3, while 63.3% is the peak MobileMiniWoB++ success rate at λ0 = 0.45.The reward-weight sweep follows an inverted U-shaped trend, with excessive weights reducing performance.
  • Matching Quality: 81.3% is the embedding matcher’s peak accuracy at δ = 0.75, versus 85.39% for the LLM Judge.The validation used 500 manually annotated milestone-action pairs.
  • Matching Efficiency: 11 seconds versus approximately 780 seconds makes embedding matching over 70× faster than the LLM Judge on the validation set.The paper presents this efficiency as important because reward calculation occurs at every reinforcement-learning step.
  • Training Overhead: 187.99 s/epoch is ADMIRE’s average training time, only +12.7% over Outcome Reward’s 166.83 s/epoch, while Process Reward averages 388.44 s/epoch (+132.8%).The comparison covers 150 training epochs.
  • Milestone Quality: ADMIRE’s generated milestones average 4.42 on a 5-point scale, with 87.7% rated Good or Perfect and fewer than 5% receiving the lowest score.Human evaluation assesses factuality, logical coherence, and granularity.

B.6 Milestone Coverage Analysis

ADMIRE’s milestone coverage expands rapidly during online training, establishing supervision for most tasks early and continuing to include harder long-horizon scenarios. Milestones are matched to agent progress to provide more comprehensive guidance during online reinforcement learning.

  • Coverage Definition: Milestone initialization coverage is defined as the percentage of tasks with at least one successful trajectory available to initialize milestone memory.This metric tracks how ADMIRE adapts throughout online training.
  • Coverage Growth: 92.7% milestone initialization coverage is ultimately achieved as the agent’s capability evolves.The coverage rises from 83.8% in Epoch 0-100 to 89.7% at Epoch 200 before reaching its peak.
  • Coverage Growth: 83.8% milestone initialization coverage is reached by Epoch 0-100, indicating rapid establishment of supervision signals for most tasks.Coverage later reaches 89.7% at Epoch 200 and peaks at 92.7% as policy optimization proceeds.
  • Milestone Alignment: Milestones are matched according to the agent’s progress, providing more comprehensive guidance during online reinforcement learning.The case study examines milestone alignment for trajectories generated by the Mobile GUI agent.
Loading 2602.11524v1…