Source-linked AI summary
Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning
Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng-Tao Jiang, Bo Li
TL;DR
GUI agents still struggle to ground natural-language instructions to precise interface elements, while supervised fine-tuning requires large, diverse datasets and generalizes weakly in complex, high-resolution settings. The paper addresses this with curated data, dense point rewards, and self-evolutionary reinforcement fine-tuning, achieving state-of-the-art grounding results with only 3k samples.
Problem
GUI agents face difficulty grounding natural-language instructions to correct interface elements, while supervised fine-tuning requires large diverse datasets and generalizes weakly in complex, high-resolution professional settings.
Method
The framework curates high-quality training data, uses dense point rewards based on prediction accuracy, and iteratively refines the model with attention maps from preceding models.
Results
State-of-the-art performance across three grounding benchmarks was achieved, including 47.3% accuracy on ScreenSpot-Pro, surpassing UI-TARS-72B by 24.2% with a 7B model and only 3k training samples.
Takeaways & Limitations
The results support reinforcement learning with clean data and self-evolutionary refinement as an effective approach for GUI grounding, including with small-scale training datasets.
Takeaways & Limitations
Experiments cover only 3B and 7B models, and the 7B model is limited to 2 million input pixels, potentially losing fine-grained details in high-resolution images.
Abstract
from arXiv · showhide
Graphical User Interface (GUI) agents have made substantial strides in understanding and executing user instructions across diverse platforms. Yet, grounding these instructions to precise interface elements remains challenging, especially in complex, high-resolution, professional environments. Traditional supervised finetuning (SFT) methods often require large volumes of diverse data and exhibit weak generalization. To overcome these limitations, we introduce a reinforcement learning (RL) based framework that incorporates three core strategies: (1) seed data curation to ensure high quality training samples, (2) a dense policy gradient that provides continuous feedback based on prediction accuracy, and (3) a self evolutionary reinforcement finetuning mechanism that iteratively refines the model using attention maps. With only 3k training samples, our 7B-parameter model achieves state-of-the-art results among similarly sized models on three grounding benchmarks. Notably, it attains 47.3\% accuracy on the ScreenSpot-Pro dataset, outperforming much larger models, such as UI-TARS-72B, by a margin of 24.2\%. These findings underscore the effectiveness of RL-based approaches in enhancing GUI agent performance, particularly in high-resolution, complex environments.
1 Introduction
GUI agents still struggle to ground instructions precisely in complex interfaces, while SFT requires large datasets and generalizes weakly. The paper addresses these issues with curated data, dense rewards, and self-evolutionary attention supervision.
- GUI agents face persistent difficulty mapping natural-language instructions to precise interface elements in complex, high-resolution settings.
- Automated grounding datasets can contain vague instructions, inaccurate annotations, and boxes for elements that are not visually rendered.These issues motivate explicit training-sample quality assessment.
- 3,018 samples are curated by filtering vague, inaccurate, or overly simple tasks to improve linguistic consistency and task-complexity balance.
- Dense point rewards evaluate prediction proximity to ground truth, providing continuous feedback that lets the model learn from near-misses.
- Self-evolutionary fine-tuning uses attention maps as intermediate supervision so successive models improve their focus on relevant interface elements.The best-performing model generates attention maps for the next training iteration.
- 47.3% accuracy on ScreenSpot-Pro lets the 7B SE-GUI model surpass UI-TARS-72B by 24.2% using only 3k training samples.The evaluation covers six grounding and agent benchmarks across desktop, mobile, and web environments.
2 Related Work
GUI agents use visual interaction rather than programmatic APIs to automate tasks across platforms. Recent work has extended reinforcement learning and reinforcement fine-tuning toward GUI and multimodal tasks.
- GUI agents simulate human-like interaction through mouse clicks, keyboard inputs, and visual perception across diverse platforms.They differ from traditional programmatic agents that rely on APIs or internal code access.
- Reinforcement learning uses rule-based rewards to guide GUI-agent behavior, building on successful applications in reasoning, code generation, and multimodal vision tasks.
- UI-R1 and GUI-R1 are identified as pioneering applications of reinforcement fine-tuning to GUI tasks.
3 Method
SE-GUI combines seed-data curation, dense point rewards, and attention-guided self-evolutionary reinforcement fine-tuning to improve GUI grounding. The framework filters noisy samples, supplies feedback for near-miss predictions, and iteratively uses attention maps to refine training.
- 3.1 Seed Data Curation: The framework generates multiple candidate responses and optimizes them with GRPO using verifiable rewards, then reuses the updated model to identify higher-quality training samples.This creates an iterative loop that refines both predictions and training data.
- 3.1 Seed Data Curation: SE-GUI constructs a clean training set through instruction-quality, bounding-box, and task-difficulty filtering.The resulting SE-GUI-3k dataset contains 3,018 high-quality samples.
- 3.2 Group Relative Policy Optimization with Dense Point Reward: GRPO normalizes format and point rewards within each candidate group to compute relative advantages, while KL regularization constrains policy updates.The group comparison helps capture differences between candidate responses and supports stable reinforcement learning.
- 3.2 Group Relative Policy Optimization with Dense Point Reward: Dense point rewards evaluate prediction proximity to the ground-truth box, providing smoother feedback for near-miss clicks than binary rewards.Points inside the box receive a base reward of 1.0 with distance-based decay, while outside points are scored by normalized distance to the box center.
- 3.3 Self-Evolutionary Reinforcement Fine-Tuning: SE-RFT uses attention maps from the previous model iteration to filter and supervise subsequent training samples.Samples whose attention fails to focus on the target can receive zero loss, preventing incorrect guidance from affecting optimization.
4 Experiments
SE-GUI is evaluated across grounding and agent benchmarks, showing strong performance with limited data and improvements from data quality, dense rewards, and self-evolutionary training. Ablations also examine resolution and professional-environment generalization.
- Benchmark Results: SE-GUI-7B achieves an 8.5% improvement over UI-TARS-72B using only 3K versus 14M training examples.It also reaches 88.2% on ScreenSpot and 90.25% on ScreenSpot-v2.
- Benchmark Results: SE-GUI-7B outperforms baselines on AndroidControl, reaching 52.8% low-level and 68.2% high-level step accuracy.The high-level result is a +5.8% improvement over UGround-V1.
- Benchmark Results: SE-GUI achieves an action score of 34.4 on OmniACT, surpassing GPT-4o at 29.6 and UGround-V1 at 32.8.The evaluation covers desktop and web environments under zero-shot settings.
- Ablation Study: 4.45% higher accuracy results from filtered, higher-quality data, whereas unfiltered data reduces accuracy to 31.31%.The ablation evaluates data quality on ScreenSpot-Pro.
- Ablation Study: Dense rewards improve performance by 4.21% over sparse rewards by providing more reward signals during early training.The point reward guides the model toward correct UI-element positions.
- Ablation Study: Self-evolutionary training raises performance from 39.97% in stage one to 46.55% in stage three, while stage-four gains are minimal.Later stages use attention maps generated by the preceding model, and reward curves indicate near convergence.
5 Conclusions and Limitations
The paper combines curated data with self-evolutionary reinforcement fine-tuning to improve GUI grounding, achieving state-of-the-art results across three grounding benchmarks. Its 7B model is evaluated under a 2-million-pixel input-resolution limit, which may remove fine-grained visual details.
- Conclusions: The method filters training data and progressively improves the model through self-evolutionary reinforcement fine-tuning.Attention-guided iterations build on a filtered dataset to enhance performance progressively.
- Conclusions: The approach achieves state-of-the-art results across three grounding benchmarks.
- Limitations and future work: The 7B experiments are limited to a maximum input resolution of 2 million pixels because of hardware constraints.This limit may cause loss of fine-grained details in high-resolution images.
- Limitations and future work: The paper proposes that larger models such as 32B or 72B could yield better performance, but this remains future work.
A GRPO preliminary
GRPO evaluates multiple candidate responses for each task and compares their rewards relative to the candidate group. Reward normalization captures relative quality while KL regularization constrains policy updates for stable learning.
- GRPO procedure: GRPO compares candidate responses using normalized rewards instead of relying on a critic model.This differs from PPO, which uses a single reward signal and a critic to estimate the value function.
- GRPO procedure: GRPO generates N candidate responses and evaluates each by executing its corresponding actions and computing a reward.
- Reward normalization: The relative advantage A_i is computed from each response reward after subtracting the group mean and dividing by the group standard deviation.
- Policy stability: KL-divergence minimization constrains updates between the new and reference policies to maintain stable reinforcement learning.
B More training details.
The training details include a 2-million-pixel resolution limit, scoring prompts for cropped UI elements and instruction quality, and a computer-use tool interface. Figure 4 presents the SE-GUI-7B training curve.
- Training setup: Training on the 7B model uses a maximum input resolution of 2 million pixels because of resource constraints.The paper states that increasing this limit could improve performance.
- Scoring prompts: The bounding-box scoring prompt checks whether a crop contains exactly one visually complete UI element.It rejects crops containing background, empty space, meaningless fragments, or significant cut-offs.
- Scoring prompts: The instruction-quality prompt tests whether wording matches visible text or function, avoids confusion, and specifies an unambiguous action.
- Tool interface: The computer-use interface supports mouse and keyboard interaction with screenshots and requires screenshot-based coordinate selection before clicking.The described desktop screen resolution is 1876 x 1036.
- Training visualization: Figure 4 shows the training curve of the SE-GUI-7B model.
C Attention visulization.
The paper visualizes attention behavior and reports an example in which the model attends to the target region for the instruction “pin Jack’s conversation.” Ablation tables are identified for SE-GUI-3B and SE-GUI-7B on ScreenSpot-Pro.
- Attention visualization: For “pin Jack’s conversation,” the model correctly attends to the image’s upper-right region.
- Attention visualization: The visualization extracts attention matrices, locates visual tokens, and computes generated-token attention weights over those visual tokens.
- Attention visualization: Figure 5 is described as showing model attention on a specific training sample.
- Attention visualization: Figure 6 is described as showing model attention on a specific training sample.
- Ablation studies: Tables 5 and 6 report ablations for SE-GUI-3B and SE-GUI-7B, respectively, on ScreenSpot-Pro.
D Hyper parameters
The experiments tune reward weighting, KL regularization, and attention-map filtering to balance prediction accuracy, exploration, and supervisory guidance.
- A 1:1 format-to-point reward ratio slightly reduces performance, so training uses a 1:2 ratio to emphasize prediction accuracy.The authors attribute this adjustment to the relative simplicity of learning format consistency.
- A smaller KL divergence coefficient improves results, leading to a final coefficient of 0.004.The authors hypothesize that a large KL term overly aligns the model with the reference policy and limits exploratory learning.
- An attention-map filtering threshold of 0.2 balances retaining reasoning-potential samples against providing effective supervisory guidance.Higher thresholds may remove useful samples, whereas lower thresholds may provide insufficient guidance.