Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learning

1 Introduction

• Existing SFT- and RL-trained GUI grounding models often produce confidence estimates poorly aligned with correctness, making prediction reliability difficult to judge.On ScreenSpot-Pro, evaluated models show higher confidence than accuracy, for both probabilistic and verbalized confidence.Source: §1 · p. 1, Figure 1 · p. 2
• In dynamic GUI tasks, a single intermediate error can result in overall task failure, making unreliable confidence particularly consequential.HyperClick exposes uncertainty alongside click predictions to reduce high-confidence errors and enable confidence-based abstention.Source: §1 · p. 2
• HyperClick uses self-critiqued reinforcement learning to produce a targeted UI element alongside an explicit verbalized confidence statement.It focuses on verbalized rather than probabilistic confidence, which the paper describes as model-agnostic and directly exposable to downstream agents.Source: §1 · p. 2
• A binary correctness reward and a confidence-alignment reward based on a bounded spatial target jointly optimize click precision and confidence alignment.The target is constructed over the annotated element region.Source: §1 · p. 2
• HyperClick maintains strong grounding accuracy while producing better-aligned confidence estimates that support selective abstention on uncertain clicks.The paper reports these outcomes across challenging GUI grounding benchmarks.Source: §1 · p. 2

2.1 GUI Agents and GUI Grounding

• GUI grounding research progressed from supervised fine-tuning on large corpora to RL rewards tailored to spatial localization, while largely overlooking confidence alignment.Examples include visual-only grounding, diverse-environment fine-tuning, and reward designs based on box size, continuous rewards, or Gaussian modeling.Source: §2.1 · p. 3

2.2 Confidence in LLMs

• Confidence signals include token probabilities, agreement across multiple outputs, and explicit verbalized certainty.Probabilistic and consistency-based confidence can correlate better with performance but may require token likelihoods, repeated sampling, or in-domain calibration.Source: §2.2 · p. 3
• Verbalized confidence is model-agnostic and easier to expose to downstream agents, but is prone to weak alignment and overconfidence.The paper aligns it with a spatial confidence distribution over the screenshot for selective abstention.Source: §2.2 · p. 3

3.1 Problem Formulation

• The policy predicts a single point at the target element's center, a click formulation chosen to match executable actions and reduce the action space.Source: §3.1 · p. 3

3.2 Confidence Modeling

• A Gaussian-kernel target scores clicks highest at the annotated center, zeros clicks outside the box, and scales its spread to element size.Larger elements receive broader high-confidence regions, while smaller elements produce sharper peaks.Source: §3.2 · pp. 3–4

3.3 Training Objective

• GRPO combines format validity, binary in-box correctness, and confidence alignment to a bounded spatial target, rewarding low confidence for incorrect clicks.Source: §3.3 · p. 4

4.1 Implementation Details

• Training uses one epoch on 16 H100 GPUs, with a global batch size of 16 and temperature 0 at inference.The implementation uses Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct, with a cosine learning-rate schedule and KL constraint coefficient β = 0.04.Source: §4.1 · p. 5

4.2 Evaluation Benchmarks

• Evaluation spans ScreenSpot, ScreenSpot-V2, ScreenSpot-Pro, and MMBench-GUI, covering common interfaces, professional settings, and varied instruction complexity.ScreenSpot-Pro includes 23 applications across five industries and three operating systems.Source: §4.2 · p. 5

4.3 Main Results

• HyperClick maintains competitive open-source accuracy across mobile, desktop, and web settings, with consistent gains especially on challenging ScreenSpot-Pro and MMBench-GUI benchmarks.The reported comparison covers 3B and 7B model categories and pairs grounding performance with explicit confidence-quality alignment.Source: §4.3 · p. 5, Table 1 · p. 6, Table 2 · p. 6

4.4 Ablation Study

• Combining format, correctness, and confidence-alignment rewards performs best; confidence alignment alone can cause reward hacking by favoring incorrect answers with confidence zero.Strict format verification improves accuracy from 91.0% to 93.7%.Source: §4.4 · p. 7, Table 4 · p. 7
• All tested confidence-reward formulations improve over the no-confidence baseline; L2 performs best at 93.7%, with KL comparable at 93.6%.The study compares L1, L2, and KL formulations.Source: §4.4 · p. 7, Table 5 · p. 7
• Without a bounded spatial target, binary confidence alignment reaches 92.7%; Gaussian-target variants perform better, while overly diffuse or concentrated targets are suboptimal.The paper sets α according to the Gaussian 3σ principle; large variance weakens center-sensitive supervision, whereas small variance is too strict for minor deviations.Source: Table 6 · p. 7, §4.4 · p. 8
• Applying HyperClick training to MiMo-VL raises its reported result from 91.4% to 94.0%, supporting transfer of the framework to another model.The paper presents this as evidence of generality and scalability across models.Source: Table 7 · p. 7, §4.4 · p. 8

4.5 Visualization

• Predicted confidence concentrates around ground-truth elements and stays low in irrelevant regions, with smaller elements producing tighter peaks and larger ones broader heatmaps.The visualization forms heatmaps by requiring confidence output for possible click positions.Source: §4.5 · p. 8, Figure 4 · p. 8

• HyperClick addresses overconfidence in GUI grounding, which undermines autonomous GUI agent reliability.Source: §5 · p. 8
• Its SCRL framework combines a binary correctness reward with an l2 confidence alignment reward derived from a bounded spatial target.Source: §5 · p. 8
• Experiments show improved grounding accuracy and better-aligned confidence estimates, suggesting a path toward trustworthy grounding in broader multimodal agents.Source: §5 · p. 8

• Confidence alignment has been verified for GUI grounding but not extended to GUI planning tasks.Source: Limitation · p. 9
• The authors identify planning reliability as critical because inaccurate or overconfident planning decisions can propagate errors across multiple steps and lead to task failure.Source: Limitation · p. 9
• Future work will investigate confidence alignment in planning modules so agents can expose uncertainty during high-level decisions in complex interactions.Source: Limitation · p. 9

• The research focuses on building a policy model for reliable GUI grounding.Source: Ethics Consideration · p. 9
• The authors report no discrimination, bias, or fairness issues requiring attention and do not expect the models to generate potentially harmful content.Source: Ethics Consideration · p. 9
• Experimental and data details are provided in Section 4 and appendices, with source code and model checkpoints planned for release to support reproducibility.Source: Ethics Consideration · p. 9

• The grounding dataset contains 50K samples balanced across Mobile, Web, and Desktop, combining public datasets with in-house data.Table 8 gives per-source statistics.Source: §A.1 · p. 13, Table 8 · p. 13
• For RL data, Qwen2.5-VL-7B generates predictions at temperature 0, and eight temperature-0.9 inferences supply correct predictions for initially incorrect cases.Source: §A.1 · p. 13
• An equal number of correctly predicted Stage 1 samples are added before RL to stabilize training and help the model follow the target output format.Confidence is rounded to three decimal places.Source: §A.1 · p. 13

A.2 Evaluation Prompts

• The section presents replicated ScreenSpot-Pro evaluation prompts, including prompts labeled for UI-TARS and UI-TARS-1.5.The authors state that they follow the original instructions to reproduce and analyze the results.Source: §A.2 · p. 13
• Grounding prompts request a point, JSON coordinates, labeled coordinates, or a bounding box, depending on the variant.Some variants ask the model to point to the element without specifying a JSON format.Source: §A.2 · p. 13
• For UI-TARS and UI-TARS-1.5, grounding is prompted first, followed by a second round that elicits verbalized confidence.A confidence prompt requests a float between 0 and 1.Source: §A.2 · p. 13

A.3 Training Dynamics

• The mean reward rises and plateaus while reward standard deviation stays above zero, indicating stable optimization with continued rollout diversity.The authors interpret the nonzero standard deviation as preserving a usable advantage signal rather than policy collapse.Source: §A.3 · pp. 13–14

A.4 Stability of Confidence

• Repeated confidence sampling shows low variance for HyperClick-3B and HyperClick-7B, with the larger model slightly more stable.Confidence is sampled eight times at temperature 1.0 after coordinates are predicted without sampling.Source: §A.4 · p. 14, Table 9 · p. 14

A.5 More Evaluation Benchmarks and Detailed Experimental Results

• CAGUI evaluates Chinese mobile GUI grounding, emphasizing textual elements and functional operations in Chinese-language apps.Table 10 reports accuracy across Fun2Point, Text2Point, and Bbox2Text.Source: §A.5 · p. 14, Table 10 · p. 15
• UI-I2E-Bench tests implicit instructions requiring semantic understanding and spatial reasoning across mobile, desktop, and web platforms.Table 11 covers tasks with varying degrees of implicitness.Source: §A.5 · p. 14, Table 11 · p. 15
• UI-Vision evaluates generalization across diverse desktop environments using previously unseen applications and subtasks spanning six categories.Table 12 lists Education, Browsers, Development, Productivity, Creativity, and Entertainment.Source: §A.5 · p. 14, Table 12 · p. 16