PaperlayerView exact revision on arXiv

Source-linked AI summary

GSAR: Goal-State-Anchor Rewards for Mobile GUI Agents with Self-Evolving Data Synthesis

Long Zhang, Yuhan Chen, Chaoran Zhang, Wanxia Cao, Kun Huang, Pengzhi Gao, Wei Liu, Jian Luan, Chenliang Li, Lixin Zou

arXiv:2608.22847v1cs.AI

TL;DR

GUI-agent RL training is bottlenecked by insufficiently diverse executable data and reward evaluators that are either difficult to scale or unreliable. GSAR combines self-evolving data synthesis with state anchors from successful goal states to provide scalable reward supervision. It achieves over 90% accuracy in trajectory verification, improves online-RL performance over the base model, and has limitations for tasks requiring multiple solutions or nonvisual completion criteria.

  • Problem

    GUI-agent RL requires diverse environment-task pairs and reliable reward verification, but existing data synthesis depends on manually configured environments and current judges trade scalability against accuracy.

  • Method

    GSAR combines self-evolving data synthesis, which evolves environments and generates tasks and trajectories, with state anchors that annotate task-relevant UI elements in successful goal states.

  • Results

    Over 90% accuracy in trajectory verification was achieved, while online RL improved performance over the base model and produced reward curves closely aligned with rule-based rewards.

  • Takeaways & Limitations

    GSAR provides a scalable approach for GUI-agent training with reward feedback that closely approximates rule-based evaluation without relying on handcrafted rules.

  • Takeaways & Limitations

    GSAR uses a single final state as the success criterion, limiting handling of alternative solutions and tasks whose completion cannot be fully expressed visually.

Abstract

from arXiv · show

Vision-Language Models (VLMs) based GUI agents stand to benefit significantly from online reinforcement learning (RL). However, their training is bottlenecked by two fundamental issues: current data synthesis methods for GUI Agents rely on specific environments and struggle to generate diverse data, while existing evaluators either suffer from limited scalability or provide inaccurate and unreliable reward signals. To overcome these challenges, we introduce GSAR (Goal-State-Anchor Reward), a RL reward framework that supports scalable task generation and delivers reliable reward signals for stable and efficient policy optimization. Our approach features self-evolving data synthesis, which produces multiple environments through task execution and generates diverse tasks and goal states. Complementing this, a state-anchor mechanism automatically annotates task-relevant UI elements in successful goal states as reference anchors. During RL training, these reference anchors provide accurate, scalable reward signals that substantially enhance efficiency. Extensive evaluations demonstrate that our framework achieves over 90% accuracy on offline trajectory verification and performs closest to rule-based methods. Furthermore, agents trained using our reward framework exhibit strong performance on both AndroidWorld and our constructed benchmark, establishing a scalable approach for GUI agent training.

1 Introduction

GUI agents can benefit from online RL, but scalable training requires diverse executable tasks and reliable reward verification. GSAR addresses both bottlenecks through self-evolving data synthesis and goal-state anchors, while reward accuracy critically affects agent performance.

  • Motivation: Online RL for GUI agents depends on diverse task-environment instances and strictly verified reward signals.Open-ended task descriptions still require corresponding initial environments and reliable outcome evaluation.
  • Motivation: Manually designed environment configurations prevent automatic generation of truly diverse environment-task pairs at scale.This dependency makes environment setup and data collection labor-intensive.
  • Reward Verification: Rule-based judges are precise but require manually written execution functions, whereas model-based judges can be inaccurate under limited context or capacity.The two approaches therefore face opposing scalability and reliability constraints.
  • GSAR: GSAR combines self-evolving data synthesis with a state-anchor mechanism to generate diverse trajectories and provide semantically grounded reward signals.Successful trajectories yield task-relevant UI anchors in final states for online training.
  • Contribution: GSAR introduces a scalable Goal-State-Anchor reward framework intended to overcome prior judges’ limitations for online reinforcement learning.The framework is designed to deliver accurate, generalizable signals.
  • Contribution: Reward design critically affects GUI-agent performance during reinforcement learning.The paper explicitly analyzes reward accuracy as a determinant of training outcomes.

2 Related Works

GUI-agent research has moved from supervised datasets and narrow grounding rewards toward RL and outcome verification, but scalable data and reliable rewards remain difficult. GSAR combines environment-interaction-based synthesis with a reward framework designed to improve accuracy while maintaining scalability.

  • GUI Dataset Synthesis: Manually curated GUI datasets are costly and inefficient to scale for supervised fine-tuning.Prior synthesis efforts generate training data, but their manual curation limits scalability.
  • Reinforcement Learning: Recent GUI-agent research integrates reinforcement learning to reduce supervision requirements and improve generalization and long-term decision-making.Reward signals guide policy optimization and shape learning dynamics.
  • GSAR: GSAR uses self-evolving data synthesis to generate diverse tasks, trajectories, and initial environments through interaction-driven evolution.The approach integrates environment evolution with automated data generation.
  • Reward Design: Grounding rewards based on click positions or IoU are limited in scope and lack validation on complex high-level tasks.These designs target localized grounding rather than broader task completion.
  • GSAR: GSAR is designed to provide more accurate and reliable RL reward signals while maintaining scalability.This objective addresses limitations associated with model-based reward methods.

3 Method

GSAR combines self-evolving data synthesis with state-anchor reward annotation to generate diverse task environments and provide scalable, reference-based supervision for GUI-agent reinforcement learning.

  • Self-Evolving Data Synthesis: Self-evolving synthesis jointly evolves task distributions, environment states, and execution trajectories through repeated interaction with GUI environments.The process is designed as an automatic and scalable alternative to static or one-shot generation pipelines.
  • Self-Evolving Data Synthesis: The framework explores app interfaces, generates tasks from visual observations, preserves initial environments, and uses resulting task–environment pairs to initialize RL episodes.A vision-language model generates potential tasks conditioned on explored observations, while preserved environments define episode starting states.
  • Trajectory Collection: Trajectory filtering removes task–trajectory mismatches, yielding triplets containing a task, its initial environment, and a successful trajectory.Difficult tasks that fail during automated collection may instead receive manually executed goal states for reinforcement-learning references.
  • Self-Evolving Data Synthesis: Successful source triplets are complexified through inheritance, composition, or rewriting to produce progressively more challenging and diverse workflows.Inheritance creates follow-up tasks, composition adds subtasks, and rewriting changes task parameters while preserving overall structure.
  • State-Anchor Reward: The state-anchor mechanism identifies task-relevant UI elements in successful terminal screenshots and uses their anchored regions as structured goal-state references.Element indices are overlaid using the accessibility tree before relevant elements are selected and their bounding boxes anchored in the goal screenshot.
  • State-Anchor Reward: During evaluation, the current GUI state, action history, and goal-state-anchor representation are combined to produce a binary completion reward at each step.The resulting human-free annotation process provides scalable supervision for reinforcement learning and supports more accurate outcome judgments.

4 Experiments

The experiments evaluate synthesized-data quality, offline trajectory verification, online RL performance, and component contributions. GSAR improves synthesized-data benchmarks, exceeds 90% accuracy and F1 in offline verification, and supports scalable online RL training.

  • Experiment Settings: GSAR combines task, environment, and trajectory evaluation settings across AndroidControl, GUI-Odyssey, AndroidWorld traces, and a self-built benchmark.The experiments report TM, EM, Accuracy, F1, and Success Rate using model-based and rule-based comparisons.
  • Synthesized Data Quality: Synthesized data improves AndroidControl and GUI-Odyssey performance, including EM gains of 5.2% and 3.2% and a 6.3% TM gain.Scaling to additional open-source apps maintains strong performance, though results are slightly lower because of dataset distribution differences.
  • Offline Evaluation: GSAR is the only evaluated method exceeding 90% in both accuracy and F1 for open-source and closed-source judge models.Reference states significantly improve model-based evaluation, while rule-based accuracy and F1 are theoretically 100%.
  • Online RL Training: GSAR training improves success rate by 23.2% on the training split and 8.1% on the full task set compared with UI-TARS.GUI-Owl shows corresponding gains of 18.6% and 8.2%; goal-state anchors and action history further improve execution.
  • Ablation Study: Combining action history with the anchored reference screenshot yields the best ablation performance, while removing either component causes noticeable degradation.Removing the reference loses final-state information, and removing anchors impairs detection of subtle differences between incomplete and completed states.

5 Analysis Experiments

Additional analyses examine goal-state annotation, rollout efficiency, reward-signal errors, and comparison with rule-based rewards. GSAR reaches 91.5% annotation accuracy, lowers rollout latency relative to rule-based verification, and improves AndroidWorld performance over the base model.

  • Goal-State Annotation: 91.5% overall accuracy is achieved for automatic goal-state annotation, with Delete tasks reaching 100%.Answer and Normal tasks are harder because relevant elements can be small, scattered, or numerous.
  • Reward-Signal Analysis: StepCritic’s high false-positive rate produces overly optimistic rewards and less stable, less reliable online policies.Its relatively low false-negative rate does not prevent reward errors from limiting online performance.
  • AndroidWorld Comparison: GSAR raises AndroidWorld success rate from the UI-TARS-7B-DPO baseline of 26.7% to 30.2%, while rule-based rewards reach 32.8%.The corresponding improvements are +3.5% for GSAR and +6.1% for rule-based rewards.
  • Rollout Efficiency: GSAR achieves lower overall rollout latency than the rule-based baseline despite adding model-inference latency after each action.Repeated ADB-based device access creates greater overhead for most tasks.

6 Conclusion

GSAR integrates self-evolving data synthesis with state anchoring to generate diverse training data and reliable reward supervision. Experiments report over 90% trajectory-verification accuracy and improved online RL performance.

  • Conclusion: GSAR integrates self-evolving data synthesis with a state-anchor mechanism for scalable GUI-agent training.The framework evolves environments and tasks through execution, then anchors task-relevant UI elements in successful final states.
  • Conclusion: GSAR achieves over 90% trajectory-verification accuracy and improves performance over the base model in online RL training.Its reward curve closely aligns with rule-based rewards while avoiding handcrafted rules.

Limitations

GSAR has several scope and data-quality limitations despite addressing scalability and reward-accuracy issues. Its success criterion, task coverage, and dependence on VLM-generated trajectories constrain applicability.

  • Limitations: GSAR evaluates success using only a single final state, although tasks may have multiple valid solutions.This limits the framework's ability to represent alternative task-completion paths.
  • Limitations: For question-answering tasks, GSAR may assign a correct reward before the actual correct answer is produced.
  • Limitations: Tasks whose completion cannot be fully expressed visually, such as deleting certain items, are difficult to annotate with task-relevant elements.Natural language or other modalities may be needed to describe these final states.
  • Limitations: VLM-based trajectory collection and filtering eliminate manual annotation, but generated trajectory quality is sometimes constrained by model capabilities.Tasks that cannot be completed automatically still require manual execution to obtain a correct goal state for reinforcement learning.

A Benchmarks

The evaluation uses AndroidControl and GUI-Odyssey, which cover large-scale Android control and cross-app navigation. Their benchmark settings include removing steps associated with the “open_app” action type.

  • AndroidControl: AndroidControl contains 15,283 task demonstrations across 833 apps and covers 14,548 unique tasks.Its tasks include high-level and low-level human-generated instructions.
  • AndroidControl: AndroidControl supports evaluation across varying task complexity, including in-domain and out-of-domain settings.It highlights generalization challenges involving unseen apps and higher-level tasks.
  • GUI-Odyssey: GUI-Odyssey consists of 8,334 episodes averaging 15.3 steps across 6 devices, 212 apps, and 1,357 app combinations.Each step includes semantic reasoning annotations for complex cross-app navigation.
  • Evaluation setting: The benchmark evaluations remove steps related to the “open_app” action type.

B Self-Evolving Data Synthesis

Self-evolving data synthesis iteratively generates tasks, evolves GUI environments, and collects trajectories. Its algorithm maintains task and trajectory sets while executing newly generated tasks under bounded exploration parameters.

  • Self-Evolving Data Synthesis: The framework iteratively collects tasks, evolves GUI environments, and generates increasingly complex instructions from prior trajectories.This workflow is designed to produce diverse, high-quality reinforcement-learning data.
  • Iteration control: GPT-4o performs task generation, task complexification, and task execution, with AndroidWorld capped at 6 iterations.For additional open-source apps, exploration determines an iteration range based on unique pages in trajectory screenshots.
  • Algorithm inputs and outputs: Algorithm 1 takes a device, package, exploration limits, and a maximum task-step budget as inputs, producing task and trajectory sets.The stated limits include maximum tasks per page, breadth, depth, and steps per task.
  • Environment evolution: Each iteration pulls the environment from the device, updates the task set, and pushes the evolved environment back to the device.
  • Trajectory collection: The procedure executes newly collected tasks with a step limit, gathers resulting trajectories, and updates the trajectory set for subsequent iterations.

C Details of Experiments

The experiments specify separate supervised fine-tuning, offline verification, and online reinforcement-learning configurations. Evaluation uses constructed positive and negative trajectories, while online training relies on parallel Android emulators for interaction sampling.

  • Supervised Fine-Tuning: The SFT training dataset contains 6,331 steps spanning low-level and high-level instruction types.The detailed SFT settings are reported in Table 6.
  • Offline Verification: Offline GSAR verification uses 315 AndroidWorld trajectories, comprising 164 positive and 151 negative samples.Goal states are selected from final-step screenshots and anchored using GPT-4o.
  • Online Reinforcement Learning: Online RL configurations include the backbone agent, reward evaluation model, optimization algorithm, and other training parameters.These settings are summarized in Table 7.
  • Online Reinforcement Learning: 128 Android emulators provide parallel trajectory sampling during online RL.Two machines each launch 64 emulators, decoupling GUI interaction from the training process.

D Supplementary SFT Results

The supplementary SFT experiments examine whether synthesized data generalizes across base models, increases task and page complexity, and yields usable training trajectories. Results show transfer across UI-TARS variants, later-iteration complexity growth, longer complexified tasks, and substantial trajectory utilization.

  • Cross-model generalization: Synthesized data transfers across UI-TARS-7B-SFT and UI-TARS-7B-DPO, matching or surpassing the UI-TARS baseline across all three benchmarks.Scaling to additional open-source apps also produces comparable results, although gains are narrower than with Qwen2.5-VL-7B.
  • Self-evolving complexity: Later self-evolving iterations produce pages with slightly higher edge density, image entropy, and average UI-element counts.The analysis samples 300 tasks from early and later iterations and randomly selects one trajectory state per task for page-complexity measurement.
  • Task complexity: Task complexification increases both average natural-language query length and average trajectory length.The comparison uses 339 samples for each setting.
  • Data utilization: 45.7% of trajectories in the aw-app dataset are utilized for fine-tuning when trajectories with reward score R ≥4 are retained.The aw-app and extra-app datasets contain 993 and 960 tasks, respectively.

E.2 Offline Evaluation by Task Category

Offline evaluation compares reward methods across task categories and illustrates their effects in representative GUI interactions. GSAR performs consistently across categories, while task structure determines whether action history, goal-state references, or semantic understanding are most important.

  • Category-wise evaluation: GSAR demonstrates consistently strong offline trajectory-evaluation performance across all task categories.The category-wise results highlight differences in completion-judgment difficulty and GSAR’s effectiveness across categories.
  • Delete tasks: StepCritic outperforms GSA-Only and GS-Only on Delete tasks, indicating that action history is critical for judging completion in this category.GSA-Only performs worse than GS-Only, potentially because large anchor boxes draw attention to irrelevant screen regions.
  • Answer tasks: DigiRL obtains an F1 of zero on Answer tasks because its frame-similarity filter labels trajectories with minimal final-frame change as incomplete.Answer tasks typically show little visual difference between the final two frames, producing zero true positives for precision and recall.
  • Normal tasks: Normal tasks can be judged strongly without action history because successful and incomplete trajectories differ in key UI elements.
  • Online behavior examples: Representative cases contrast a failed StepCritic trajectory with a successful GSAR trajectory on complex mobile GUI tasks.The examples are presented in Figures 8 and 9.
Loading 2608.22847v1…