Source-linked AI summary
POINTS-GUI-G: GUI-Grounding Journey
Zhongyin Zhao, Yuan Liu, Yikun Liu, Haicheng Wang, Le Tian, Xiao Zhou, Yangxiu You, Zilin Yu, Yang Yu, Jie Zhou
TL;DR
GUI grounding requires building precise interface-element localization into agents, including when starting from a model lacking native grounding capabilities. POINTS-GUI-G addresses this through unified data engineering, optimized training, and reinforcement learning with verifiable rewards, achieving strong results across GUI-grounding benchmarks.
Problem
GUI agents need precise GUI grounding to locate interface elements for accurate operations, but POINTS-1.5 lacks native grounding capabilities.
Method
POINTS-GUI-G builds grounding through unified data engineering, vision-encoder and resolution-focused training strategies, and reinforcement learning with verifiable rewards.
Results
POINTS-GUI-G ranks first among similarly sized models across multiple benchmarks and outperforms several significantly larger counterparts.
Takeaways & Limitations
The open-sourced model and evaluation suite provide a foundation for future work on optimizing end-to-end GUI-agent task execution.
Abstract
from arXiv · showhide
The rapid advancement of vision-language models has catalyzed the emergence of GUI agents, which hold immense potential for automating complex tasks, from online shopping to flight booking, thereby alleviating the burden of repetitive digital workflows. As a foundational capability, GUI grounding is typically established as a prerequisite for end-to-end task execution. It enables models to precisely locate interface elements, such as text and icons, to perform accurate operations like clicking and typing. Unlike prior works that fine-tune models already possessing strong spatial awareness (e.g., Qwen3-VL), we aim to master the full technical pipeline by starting from a base model with minimal grounding ability, such as POINTS-1.5. We introduce POINTS-GUI-G-8B, which achieves state-of-the-art performance with scores of 59.9 on ScreenSpot-Pro, 66.0 on OSWorld-G, 95.7 on ScreenSpot-v2, and 49.9 on UI-Vision. Our model's success is driven by three key factors: (1) Refined Data Engineering, involving the unification of diverse open-source datasets format alongside sophisticated strategies for augmentation, filtering, and difficulty grading; (2) Improved Training Strategies, including continuous fine-tuning of the vision encoder to enhance perceptual accuracy and maintaining resolution consistency between training and inference; and (3) Reinforcement Learning (RL) with Verifiable Rewards. While RL is traditionally used to bolster reasoning, we demonstrate that it significantly improves precision in the perception-intensive GUI grounding task. Furthermore, GUI grounding provides a natural advantage for RL, as rewards are easily verifiable and highly accurate.
1 Introduction
GUI grounding is presented as a prerequisite for GUI agents because precise localization of interface elements supports task execution. POINTS-GUI-G builds this capability from a model with limited native grounding through data engineering, training changes, and reinforcement learning with verifiable rewards.
- GUI grounding enables precise localization of interface elements needed for operations such as clicking and typing.
- POINTS-GUI-G builds grounding from the ground up using refined data engineering, precise training strategies, and reinforcement learning with verifiable rewards.
- Improved Training Strategies: Over 10 points of performance improvement on ScreenSpot-Pro followed unfreezing the vision encoder and aligning training and inference resolution constraints.Training resolution increased to 3072 × 3072, while inference resolution was constrained below 2000 × 2000 pixels.
- Reinforcement Learning with Verifiable Rewards: Reinforcement learning is applied to GUI grounding because its point and bounding-box outputs support precise, objective, and highly verifiable rewards.
- The authors position POINTS-GUI-G as a state-of-the-art GUI grounding model and seek to provide a foundation for future GUI agent development.
- Refined Data Engineering: Unified data engineering standardizes diverse GUI datasets and introduces cleaning and complexity-enhancement strategies.
2 Related Works
Related work situates GUI grounding within broader vision-language model progress and GUI-agent research. Prior studies have begun applying reinforcement learning to GUI grounding, reporting gains over supervised fine-tuning alone.
- Vision-Language Models: Vision-language model research has progressed through advances in architectural efficiency, scaling, and modality alignment.
- GUI Grounding: GUI grounding maps natural-language instructions to precise interface coordinates and serves as a prerequisite for higher-level GUI-agent task execution.
- GUI Grounding: Several studies report substantial performance gains from integrating reinforcement learning into GUI grounding compared with supervised fine-tuning-only baselines.
3 Methods
The method builds GUI grounding capability through a three-stage data engineering pipeline, specialized training strategies, and reinforcement learning with verifiable rewards. It standardizes and filters heterogeneous data, increases task complexity, adapts resolution and vision encoding, and uses layout difficulty and coordinate-based rewards to improve learning.
- Data Engineering: The data engineering pipeline standardizes heterogeneous GUI datasets, filters unreliable annotations, and enhances task complexity.It includes preprocessing, filtering, and complexity enhancement stages.
- Data Engineering: All instructions are reformatted as bounding-box prediction or center-point localization tasks, with coordinates normalized to [0, 1] at three-decimal precision.This unified formulation prioritizes spatial localization across source datasets.
- Data Engineering: Filtering retains samples whose ground-truth annotations achieve coverage score S ≥τ against UI elements detected by OmniParser-v2.Point annotations are first expanded into square bounding boxes, and the procedure prunes hallucinated or misaligned coordinates.
- Data Engineering: GUI-CodeGen and GUI-Overlay increase complexity using dense rendered software interfaces and overlapping application windows with visual distractors and occlusions.These strategies synthesize high-resolution interfaces and simulate more challenging desktop scenarios.
- Training Strategies: Training improves by adapting the vision encoder and aligning training and inference resolution, including caps of 2000 × 2000 or 2500 × 2500 pixels.Both resolution adjustments significantly improve performance by mitigating train-test inconsistency.
4 Experiments Settings
The experiments use a POINTS-1.5-based model optimized through supervised and reinforcement learning, with GUI data drawn from synthesized and diverse open-source sources.
- Training: POINTS-GUI-G replaces the Qwen2.5-7B-Instruct backbone with Qwen3-8B and optimizes GUI grounding in supervised and reinforcement learning stages.The reinforcement learning stage uses 8 rollouts per sample, a global batch size of 64, and a learning rate of 1 × 10^-5.
- Datasets: Figure 6 summarizes GUI grounding data by image shape distribution and dataset composition.The figure provides the experiment section’s overview of the geometric and source composition of the datasets.
- Datasets: The training data includes synthesized samples, a text-centric GUI-DataComp subset, 13 additional open-source datasets, and general-purpose corpora.GUI-DataComp uses PaddleOCR text boxes as ground-truth answers, while synthesized samples overlay application windows on desktop backgrounds.
5 Experiment Results
The experiments analyze which training factors influence performance and track reinforcement learning dynamics across the evaluation setup.
- Benchmark Results: Table 1 reports performance comparisons on ScreenSpot-V2, while Table 2 reports comparisons on OSWorld-G.The table captions identify bold entries as best results and underlined entries as second-best results.
- Influential Factors: Figure 7 compares key factors using average performance across five evaluation benchmarks.The comparison includes a naive baseline, data engineering, an unfrozen vision encoder, image-resolution consistency, and reinforcement learning.
- RL Dynamics: Reinforcement learning reward rises consistently before reaching a stable plateau.Reward is treated as a proxy for policy improvement during training.
- RL Dynamics: Entropy loss decreases with fluctuations, indicating continued exploration alongside a progressively sharper output distribution.The paper describes entropy loss as characterizing the exploration–exploitation trade-off.
6 Comparison with Other Models
POINTS-GUI-G performs robustly across desktop, mobile, and web environments, ranking first on three of five benchmarks and leading or matching comparable-scale state-of-the-art models.
- Overall Comparison: POINTS-GUI-G ranks first on three of five benchmarks and achieves either a significant lead or comparable performance against current state-of-the-art models.The evaluation covers desktop, mobile, and web environments using models of comparable scale.
- ScreenSpot-Pro: 9.8 points: POINTS-GUI-G outperforms GTA1-7B on ScreenSpot-Pro.It also exceeds GUI-Owl-7B by 5 points and surpasses the larger OpenCUA-32B, trailing only MAI-UI-8B on this metric.
- Cross-Benchmark Comparison: Approximately 6 points: POINTS-GUI-G outperforms MAI-UI-8B on OSWorld-G and secures the top rank among state-of-the-art models.It also exceeds competitors by margins often above 10 points on UI-Vision and remains top on ScreenSpot-v2 while matching MAI-UI-8B on MMBench-GUI.
7 Conclusion
POINTS-GUI-G is presented as a compact, high-performing GUI grounding model supported by a comprehensive optimization methodology and released evaluation resources.
- POINTS-GUI-G achieves state-of-the-art GUI grounding performance with a compact 8B-parameter architecture.It is reported to rank first among similarly sized models and outperform several larger counterparts.
- Table 3 compares methods on the UI-Vision grounding dataset.
- The paper details optimization across data engineering, training strategies, and grounding-accuracy techniques.
- The evaluation suite is open-sourced to address inconsistent evaluation standards in GUI grounding.
- Table 4 compares methods on the MMBench-GUI L2 benchmark.
- Table 5 compares methods on the ScreenSpot-Pro benchmark.
A.1 Unified System Prompt for Grounding Task
The unified grounding prompt supports locating either a UI element’s bounding box or its center point, using normalized coordinates and strict output formatting.
- Bounding-box grounding requires returning only normalized (x0, y0, x1, y1) coordinates rounded to three decimals.The format must contain no additional text.
- Center-point grounding requires returning the normalized (x, y) coordinates of the target element’s center.The coordinates are rounded to three decimals and returned without additional text.
- The prompt frames grounding as locating the exact UI element matching the user’s instruction in a provided screenshot.
A.2 GUI Grounding Examples
The examples show GUI grounding predictions across desktop, mobile, and web screenshots.
- Figure 8 presents a prediction on a desktop screenshot.
- Figure 9 presents a prediction on a mobile screenshot.
- Figure 10 presents a prediction on a web screenshot.