Source-linked AI summary
GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents
Run Luo, Lu Wang, Wanwei He, Longze Chen, Jiaming Li, Xiaobo Xia
TL;DR
GUI agents trained mainly with supervised fine-tuning require extensive data and struggle with screenshot understanding and generalization to unseen interfaces. GUI-R1 addresses this gap with rule-based reinforcement fine-tuning, unified action-space rewards, GRPO, and curated cross-platform data, achieving superior performance across eight benchmarks with 3K examples versus millions for prior methods.
Problem
Existing GUI-agent methods largely rely on supervised fine-tuning, which demands extensive high-quality data and struggles with screenshot comprehension and generalization to unseen interfaces.
Method
GUI-R1 generates multiple action responses, evaluates action type, click point, and input text with unified verifiable rewards, and updates LVLMs through policy optimization.
Results
GUI-R1 achieves superior performance to OS-Atlas across eight benchmarks and three platforms using 0.02% of the data, 3K versus 13M examples.
Takeaways & Limitations
The results support unified action-space rule modeling as a promising reinforcement-learning framework for improving LVLM execution in real-world GUI tasks.
Takeaways & Limitations
Single-image training in a non-interactive environment may prevent autonomous tracing of incorrect action sequences and the emergence of an “aha moment.”
Abstract
from arXiv · showhide
Existing efforts in building Graphical User Interface (GUI) agents largely rely on the training paradigm of supervised fine-tuning on Large Vision-Language Models (LVLMs). However, this approach not only demands extensive amounts of training data but also struggles to effectively understand GUI screenshots and generalize to unseen interfaces. The issue significantly limits its application in real-world scenarios, especially for high-level tasks. Inspired by Reinforcement Fine-Tuning (RFT) in large reasoning models (e.g., DeepSeek-R1), which efficiently enhances the problem-solving capabilities of large language models in real-world settings, we propose \name, the first reinforcement learning framework designed to enhance the GUI capabilities of LVLMs in high-level real-world task scenarios, through unified action space rule modeling. By leveraging a small amount of carefully curated high-quality data across multiple platforms (including Windows, Linux, MacOS, Android, and Web) and employing policy optimization algorithms such as Group Relative Policy Optimization (GRPO) to update the model, \name achieves superior performance using only 0.02\% of the data (3K vs. 13M) compared to previous state-of-the-art methods like OS-Atlas across eight benchmarks spanning three different platforms (mobile, desktop, and web). These results demonstrate the immense potential of reinforcement learning based on unified action space rule modeling in improving the execution capabilities of LVLMs for real-world GUI agent tasks.
1 Introduction
GUI-R1 applies rule-based reinforcement fine-tuning to GUI agents to address SFT’s data demands and limited screenshot understanding and generalization. Using unified action-space rewards and a small curated dataset, it achieves superior performance across broad evaluations.
- Prior GUI agents predominantly use supervised fine-tuning, which requires extensive high-quality data and struggles with screenshot comprehension and unseen-interface generalization.
- Rule-based reinforcement fine-tuning generates multiple reasoning-and-action responses, evaluates them with unified action-space rewards, and updates the model through policy optimization.
- 0.02% of the data (3K vs. 13M) yields superior GUI-R1 performance to OS-Atlas across eight benchmarks spanning mobile, desktop, and web platforms.
- GUI-R1 introduces a unified action-space reward function, constructs the GUI-R1-3K dataset, and evaluates agents across platforms and task granularities.
2 Related Work
GUI-agent research has expanded from specialized grounding toward broader action prediction, but existing methods rely mainly on supervised fine-tuning. Rule-based reinforcement fine-tuning offers verifiable rewards yet remains underexplored for complex high-level GUI tasks.
- GUI agents use visual screens to simulate mouse and keyboard interactions, offering flexibility when commercial software restricts access to internal APIs or code.
- UGround specializes in GUI element localization, OS-Atlas predicts function-call actions, and UITars combines GUI pretraining with task-level reasoning fine-tuning.
- Rule-based reinforcement fine-tuning uses verifiable signals such as symbolic correctness, execution feedback, IoU, or click-position accuracy instead of traditional supervision.
- Complex high-level GUI tasks remain challenging because of diverse layouts, implicit semantics, and long-horizon action dependencies.
3 GUI-R1 Framework
GUI-R1 maps high-level GUI instructions and execution history to atomic actions in a unified action space, then trains the policy with verifiable rewards. Its curated data combines diverse platforms and balanced high- and low-level examples.
- 3 GUI-R1 Framework: The framework takes high-level instructions, action history, and interface images, generates multiple reasoning responses, and updates the policy using verifiable rewards.
- 3.1 Preliminaries: High-level tasks require predicting actions from task objectives and execution history because they lack explicit fine-grained instructions.
- 3.1 Preliminaries: Given Q, I, and H, the model generates candidate responses containing an action type, input text, and click point, which receive unified action-space rewards.
- 3.2 Verifiable Rewards in Unified Action Space: The unified action space extracts categories across platforms so high-level instructions decompose into atomic actions without cross-platform action conflicts.
- 3.2 Verifiable Rewards in Unified Action Space: Accuracy reward combines action-type, click-point, and input-text rewards to provide correctness signals for all predicted actions.
- 3.2 Verifiable Rewards in Unified Action Space: Response reward combines format and accuracy rewards, while format rewards encourage structurally valid reasoning processes and final answers.
- 3.3 Training Data Curation: Approximately 14M examples were collected across Windows, Linux, MacOS, Android, and Web, including grounding, low-level, and high-level GUI data.
- 3.3 Training Data Curation: Filtering and balancing reduce the training pool to 3K high-quality examples by combining 1.5K low-level samples with 1.5K high-level samples.
4 Experiments
Experiments evaluate GUI-R1 across grounding, low-level, and high-level tasks on mobile, desktop, and web platforms, using zero-shot comparisons and ablations. GUI-R1 shows strong performance with small-scale RFT data, while filtered data and reward-weight choices affect training and outcomes.
- Experimental Setup: The evaluation covers eight benchmarks across Android, desktop, and web platforms, spanning grounding, low-level execution, and high-level tasks.The experiments use test splits and compare models under a unified zero-shot prompt.
- Grounding Capability: 0.2% of the data (3K vs. 14M) enables RFT on the 3B Qwen2.5-VL model to outperform the previous SOTA Os-Atlas-7B on GUI grounding.GUI-R1 also outperforms base and same-dataset SFT models on grounding tasks.
- Grounding Capability: 80.08 vs. 63.55 on ScreenSpot and 25.23 vs. 13.80 on ScreenSpot-Pro are the 3B-scale GUI-R1 gains over SFT models.These correspond to reported improvements of 26.3% and 82.8%, respectively.
- Low-Level Task Capability: 55.65 to 80.88 is the average low-level task success-rate increase at the 3B scale.GUI-R1 also achieves a 10-point improvement over concurrent RFT-based UI-R1 at that scale.
- High-Level Task Capability: GUI-R1 achieves SOTA on high-level GUI tasks, with a 21.06 absolute improvement over GPT-4o at the 3B scale.It also averages 3.4 points over UI-R1 at 3B scale and leads by 27.2% in step success rate on GUI-Odyssey.
- Ablation Study: Filtered GUI-R1-3K data reaches high rewards after only a few updates, whereas unfiltered low-quality data converges more slowly and has a lower performance ceiling.The ablation examines image resolution and data quality; reward-coefficient experiments report improvements when accuracy rewards receive greater relative weight.
- Training Visualization: Format rewards converge quickly, while accuracy rewards provide the main differentiated rewards later in training.Mean response length first decreases and then increases; the reported “aha moment” does not occur in the single-image, non-interactive setting.
5 Conclusion
GUI-R1 is a GUI reinforcement learning framework grounded in unified action space rule modeling, integrating reinforcement fine-tuning with LVLMs for contextual action prediction and verifiable reward-driven learning.
- GUI-R1 is grounded in unified action space rule modeling for GUI reinforcement learning.
- The framework integrates reinforcement fine-tuning with LVLMs to support contextual action prediction in GUI environments.
- GUI-R1 uses verifiable, reward-driven learning to train GUI agents.
- Extensive experiments show that GUI-R1 consistently outperforms baselines across various tasks.