Source-linked AI summary
GUI-G$^2$: Gaussian Reward Modeling for GUI Grounding
Fei Tang, Zhangxuan Gu, Zhengxi Lu, Xuyang Liu, Shuheng Shen, Changhua Meng, Wen Wang, Wenqi Zhang, Yongliang Shen, Weiming Lu, Jun Xiao, Yueting Zhuang
TL;DR
GUI grounding requires precise spatial mapping, but binary reinforcement-learning rewards provide sparse hit-or-miss feedback that ignores continuous clicking behavior. GUI-G2 models interactions with Gaussian point and coverage rewards plus element-size-adaptive variance, and reports accuracy gains up to 24.7% over state-of-the-art methods across three benchmarks.
Problem
Binary GUI-grounding rewards treat coordinates as hit-or-miss outcomes, producing sparse signals that ignore continuous click quality, spatial distributions, and element-scale differences.
Method
GUI-G2 models GUI elements as two-dimensional Gaussian distributions using point rewards for localization, coverage rewards for overlap, and adaptive variance based on element dimensions.
Results
GUI-G2 reports accuracy gains up to 4.1%, 3.3%, and 24.7% on ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro, respectively, over state-of-the-art methods.
Takeaways & Limitations
Continuous spatial modeling provides dense feedback and is reported to improve robustness to interface variations and generalization to unseen layouts.
Takeaways & Limitations
The Gaussian model assumes independence between the x and y dimensions through a diagonal structure.
Abstract
from arXiv · showhide
Graphical User Interface (GUI) grounding maps natural language instructions to precise interface locations for autonomous interaction. Current reinforcement learning approaches use binary rewards that treat elements as hit-or-miss targets, creating sparse signals that ignore the continuous nature of spatial interactions. Motivated by human clicking behavior that naturally forms Gaussian distributions centered on target elements, we introduce GUI Gaussian Grounding Rewards (GUI-G$^2$), a principled reward framework that models GUI elements as continuous Gaussian distributions across the interface plane. GUI-G$^2$ incorporates two synergistic mechanisms: Gaussian point rewards model precise localization through exponentially decaying distributions centered on element centroids, while coverage rewards assess spatial alignment by measuring the overlap between predicted Gaussian distributions and target regions. To handle diverse element scales, we develop an adaptive variance mechanism that calibrates reward distributions based on element dimensions. This framework transforms GUI grounding from sparse binary classification to dense continuous optimization, where Gaussian distributions generate rich gradient signals that guide models toward optimal interaction positions. Extensive experiments across ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro benchmarks demonstrate that GUI-G$^2$, substantially outperforms state-of-the-art method UI-TARS-72B, with the most significant improvement of 24.7% on ScreenSpot-Pro. Our analysis reveals that continuous modeling provides superior robustness to interface variations and enhanced generalization to unseen layouts, establishing a new paradigm for spatial reasoning in GUI interaction tasks.
1 INTRODUCTION
GUI-G2 reframes GUI grounding rewards as continuous spatial optimization, replacing sparse binary feedback with Gaussian signals for localization and coverage. Across three benchmarks, it reports substantial accuracy improvements and robustness to interface variations.
- Motivation: Binary rewards assign 1 inside target boxes and 0 elsewhere, giving near-misses the same reward as complete failures.This creates sparse learning signals and overlooks continuously varying click quality and element size.
- Motivation: Human clicks form Gaussian distributions centered on target elements, with click density decreasing smoothly toward edges.The observed pattern is consistent with Fitts’ Law and motivates continuous spatial reward modeling.
- Method: GUI-G2 combines Gaussian point rewards for precise localization with Gaussian coverage rewards for spatial overlap.Together, these mechanisms provide dense feedback for policy optimization.
- Method: An adaptive variance mechanism calibrates reward distributions according to element dimensions, preserving consistent signals across GUI components.The mechanism accounts for differing geometric properties and spatial uncertainty among interface elements.
- Contribution: GUI-G2 is presented as a geometrically aware transformation from discrete binary signals to continuous feedback for GUI interactions.The framework captures the planar nature of interface elements while providing fine-grained spatial information.
2 RELATED WORK
The related-work context situates GUI agents within natural-language interface interaction and reinforcement-learning approaches, while Figure 3 introduces GUI-G2’s continuous reward pipeline.
- GUI Agents: GUI agents understand and interact with graphical interfaces through natural-language instructions to automate complex computer tasks.The passage describes GUI-agent approaches as spanning multiple paradigms, including expert-designed workflows.
- GUI-G2: Figure 3 depicts task and screenshot inputs producing multiple policy predictions evaluated by Gaussian point and coverage rewards.Point rewards assess localization precision, whereas coverage rewards measure spatial overlap and jointly provide dense learning signals.
- Reinforcement Learning: Rule-based reward reinforcement learning has been applied to GUI tasks through systems including GUI-R1, UI-R1, and InfiGUI-R1.These works are presented as pioneering applications demonstrating the potential of verifiable reward paradigms for GUI tasks.
3 METHOD
GUI-G2 reformulates GUI grounding rewards as continuous Gaussian feedback over predicted and target element locations. It combines center precision, spatial coverage, adaptive scale calibration, and GRPO optimization to provide dense signals for policy learning.
- Problem Formulation: The predicted bounding box is evaluated against the ground truth by comparing predicted and target centers, with binary inside-box success transformed into continuous spatial feedback.The standard criterion checks whether the predicted center lies within the ground-truth box, whereas GUI-G2 varies reward smoothly with prediction quality.
- Gaussian Representation: GUI-G2 models GUI elements as smooth 2D Gaussian distributions, replacing discrete bounding boxes with representations of spatial uncertainty and rich gradient information.The Gaussian representation uses a diagonal covariance matrix, assuming independence between horizontal and vertical dimensions.
- Gaussian Point Rewards: Gaussian point rewards reach 1 at perfect center alignment and decrease exponentially with distance, while their decay rate is controlled by variance parameters.This provides continuous gradients across the spatial domain and supports adaptation to different element characteristics.
- Gaussian Coverage Rewards: Gaussian coverage rewards measure spatial overlap between predicted and target Gaussian distributions using the Bhattacharyya coefficient.Coverage complements center alignment by representing successful interaction across an element’s regional extent.
- Adaptive Variance Mechanism: Adaptive variance scales standard deviations with element dimensions, allowing larger elements greater spatial uncertainty while preserving precise targeting for small icons.The mechanism applies to both point and coverage rewards and uses σx = α · (x2 − x1) and σy = α · (y2 − y1).
- Reinforcement Learning with GUI-G2: GUI-G2 combines point and coverage rewards with weights ν and γ, then supplies normalized advantages to GRPO for policy optimization.The resulting Gaussian reward surface provides directional guidance even when predictions are far from targets, unlike binary rewards with discontinuities at box edges.
4 EXPERIMENTS
GUI-G2 is evaluated across three GUI grounding benchmarks and reward-design analyses. It consistently outperforms reinforcement-learning baselines, while continuous Gaussian rewards improve convergence, component complementarity, and adaptive handling of element scale.
- Main results: 92.0%, 93.3%, and 47.5% accuracy are achieved on ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro, respectively.
- Main results: 9.4% separates GUI-G2-7B from UI-TARS-72B on ScreenSpot-Pro, with 47.5% versus 38.1% using 10× fewer parameters.
- Main results: 93.3% on ScreenSpot-v2 exceeds LPO-8B and SE-GUI-7B, which reach 90.5% and 90.3%, respectively.The analysis attributes the gap to modeling both point localization and spatial coverage rather than distance alone.
- Training dynamics: Gaussian rewards converge monotonically from 290px to 150px, whereas sparse methods fluctuate between 200-400 pixels without consistent progress.Sparse training is especially unstable for IoU rewards because of restrictive activation thresholds.
- Training dynamics: 5.9% separates GUI-G2 from the best sparse baseline on ScreenSpot-v2, reaching 93.3% versus Point at 87.4%.Gaussian feedback supplies informative gradients at every spatial position, unlike binary rewards that lose gradient information at bounding-box edges.
- Reward design: 4.9% separates GUI-G2 from IG, with 93.3% versus 88.4%, showing the value of Gaussian feedback across the entire interface plane.
- Reward design: 92.1% without coverage rewards and 90.2% without point rewards trail 93.3% with both components.Point rewards emphasize center localization, while coverage rewards account for successful clicks across element boundaries.
- Reward design: 5.5 percentage points separate adaptive variance at α = 0.5 from fixed variance, with 93.3% versus 87.8%.The 1σ setting is overly restrictive, while 3σ dilutes localization precision.
5 CONCLUSION
The conclusion presents GUI-G2 as a continuous spatial reward framework for GUI grounding. Across three benchmarks, GUI-G2-7B outperforms state-of-the-art models, including a reported 24.7% improvement over UI-TARS-72B on high-resolution professional interfaces.
- GUI-G2 models GUI elements as 2D Gaussian distributions using point and coverage rewards plus adaptive variance.These mechanisms provide dense, geometrically aware feedback for localization precision and spatial coverage.
- 24.7% is the reported maximum improvement over UI-TARS-72B on high-resolution professional interfaces.
- The authors characterize GUI-G2 as a robust solution for spatial reasoning in GUI interaction tasks.
A.1 ANALYSIS OF SPURIOUS REWARDS
The spurious-reward experiments test whether arbitrary feedback can support GUI grounding. Both random reward strategies progressively degrade performance, supporting the need for spatially meaningful rewards.
- The experiments use random U(0, 1) rewards and binary random rewards as two artificial reward strategies.
- Both continuous uniform U(0, 1) and binary random rewards produce progressive performance degradation.
A.2 EVALUATION DETAILS
The evaluation details describe the compared reinforcement-learning methods, training configuration, and auxiliary analyses. These materials frame comparisons across sparse, continuous distance-based, random, and Gaussian reward strategies.
- Compared methods: The compared methods span sparse point, sparse IoU, continuous distance-based, and combined reward formulations.
- Random rewards: ScreenSpot-v2 comparisons include continuous uniform random and binary random reward strategies, both of which progressively degrade.
- Random rewards: Reward variance analysis contrasts continuous and binary random mechanisms during reinforcement-learning training.
- Compared methods: The evaluation covers diverse reward paradigms to validate GUI-G2 across different methodological frameworks.
- Evaluation protocol: The training prompt requests only [x1,y1,x2,y2] coordinates for the instructed GUI position.
A.3 ERROR ANALYSIS
Icon recognition is the dominant failure mode in GUI grounding, substantially exceeding text-related errors across platforms.
- 76.9% of ScreenSpot-V2 failures were icon errors, compared with 19 text errors across all platforms.The analysis counted 63 icon errors versus 19 text errors.
A.4 FUTURE WORK
Future work should address computational overhead as GUI agents scale to complex, high-resolution interfaces and practical deployment settings.
- Computational overhead may limit practical deployment as GUI agents scale to complex, high-resolution interfaces.Suggested directions include model compression and acceleration frameworks for large vision-language models.