PaperlayerView exact revision on arXiv

Source-linked AI summary

SafeGround: Know When to Trust GUI Grounding Models via Uncertainty Calibration

Qingni Wang, Yue Fan, Xin Eric Wang

arXiv:2602.02419v2cs.AIcs.SE

TL;DR

GUI grounding models typically provide no indication when point predictions are unreliable, despite the risk of costly, hard-to-reverse actions. SafeGround calibrates spatial uncertainty from stochastic grounding samples to support selective execution and deferral, improving system-level accuracy by up to 5.38% percentage points over Gemini-only inference.

  • Problem

    GUI grounding models typically provide deterministic point predictions without uncertainty estimates, leaving unreliable outputs uncharacterized despite potentially costly, hard-to-reverse actions.

  • Method

    SafeGround estimates spatial uncertainty from stochastic grounding samples and calibrates a decision threshold with finite-sample false discovery rate guarantees for selective prediction.

  • Results

    Across multiple GUI grounding models, SafeGround improves system-level accuracy; on ScreenSpot-Pro, Holo1.5-7B reaches 58.66% accuracy at risk level 0.34, improving over Gemini-only inference by 5.38% points.

  • Takeaways & Limitations

    SafeGround supports risk-controlled GUI grounding through reliable uncertainty discrimination, FDR control, selective prediction, and cascading inference.

  • Takeaways & Limitations

    For highly deterministic models with limited sampling diversity, SafeGround’s sampled spatial distributions may be less informative.

Abstract

from arXiv · show

Graphical User Interface (GUI) grounding aims to translate natural language instructions into executable screen coordinates, enabling automated GUI interaction. Nevertheless, incorrect grounding can result in costly, hard-to-reverse actions (e.g., erroneous payment approvals), raising concerns about model reliability. In this paper, we introduce SafeGround, an uncertainty-aware framework for GUI grounding models that enables risk-aware predictions through calibrations before testing. SafeGround leverages a distribution-aware uncertainty quantification method to capture the spatial dispersion of stochastic samples from outputs of any given model. Then, through the calibration process, SafeGround derives a test-time decision threshold with statistically guaranteed false discovery rate (FDR) control. We apply SafeGround on multiple GUI grounding models for the challenging ScreenSpot-Pro benchmark. Experimental results show that our uncertainty measure consistently outperforms existing baselines in distinguishing correct from incorrect predictions, while the calibrated threshold reliably enables rigorous risk control and potentials of substantial system-level accuracy improvements. Across multiple GUI grounding models, SafeGround improves system-level accuracy by up to 5.38% percentage points over Gemini-only inference.

1. Introduction

GUI grounding enables autonomous agents to translate natural-language instructions into executable coordinates, but incorrect predictions can trigger costly, hard-to-reverse actions. SAFEGROUND addresses this risk with distribution-aware uncertainty quantification and calibrated selective prediction, achieving reliable FDR control and improved system-level accuracy on ScreenSpot-Pro.

  • Motivation: GUI grounding supports autonomous agents but can cause costly, hard-to-reverse errors such as erroneous payment approvals or irreversible system configurations.Vision-language models translate natural-language instructions into executable screen coordinates across diverse GUI environments.
  • Motivation: Existing uncertainty methods are poorly suited to GUI grounding because they may require model internals, reliable verbalized confidence, ground-truth regions, or provide scores without decisions.These limitations impede use with black-box vision-language models and inference-time safety decisions.
  • Method: SAFEGROUND calibrates a test-time uncertainty threshold for risk-aware selective prediction, executing low-uncertainty cases while abstaining from or deferring high-uncertainty cases.Selective deferral enables cascading inference to stronger models or external resources when the primary model is uncertain.
  • Method: SAFEGROUND estimates uncertainty from the spatial dispersion and concentration of multiple stochastic grounding samples without accessing model internals.The framework applies distribution-aware uncertainty quantification to outputs from existing GUI grounding models.
  • Results: SAFEGROUND consistently outperforms existing uncertainty baselines in distinguishing correct from incorrect predictions and improves system-level accuracy while reliably controlling FDR on ScreenSpot-Pro.The evaluation covers multiple state-of-the-art GUI grounding models and validates selective cascading for high-stakes interaction scenarios.

2. Related Work

Prior GUI grounding methods primarily predict actionable interface locations from screenshots and natural-language instructions, while uncertainty estimation for GUI grounding remains underexplored. SafeGround builds on Learn Then Test calibration to apply uncertainty-based calibration to spatial GUI actions.

  • GUI Grounding: GUI grounding maps natural-language instructions to actionable interface elements or click locations, commonly as text-conditioned point-coordinate prediction.Models generate point locations conditioned on the input screenshot and instruction.
  • Uncertainty Estimation: Uncertainty estimation is widely used for reliable AI decision making, but remains largely underexplored in GUI grounding.Large language models derive uncertainty from probabilistic measures, semantic entropy, or verbalized self-reports.
  • Statistical Calibration: Learn Then Test separates model learning from statistical risk control by framing decisions as hypothesis testing over a low-dimensional decision space.Held-out calibration data identifies risk-control decisions, while split conformal prediction uses data splitting and concentration-based confidence bounds.
  • Statistical Calibration: SafeGround extends the Learn Then Test paradigm to GUI grounding through uncertainty-based calibration of spatial action decisions.The paper presents this as the first such extension to GUI grounding.

3. Methodology

SafeGround addresses deterministic GUI grounding without explicit confidence by estimating predictive uncertainty from sampled spatial outputs and combining complementary dispersion measures. It calibrates a decision threshold from held-out data to control the proportion of incorrect admitted predictions at a user-specified risk level.

  • Framework and risk control: SAFEGROUND integrates with diverse GUI grounding models without internal-state access and uses a user-specified risk level α to govern conservative deployment.The risk level α quantifies the maximum tolerable proportion of incorrect predictions.
  • Uncertainty Measurement: Three uncertainty measures capture top-candidate ambiguity, global informational dispersion, and concentration deficit as complementary failure modes.Higher uncertainty indicates localized confusion, fragmented confidence, or insufficient spatial focus, depending on the measure.
  • Sampling-Based Spatial Distribution Construction: K stochastic forward passes are aggregated onto a discretized screen grid to estimate a normalized density map and clustered into ranked candidate regions.Region scores are average probability densities, with S(1) ≥ S(2) ≥ · · · ≥ S(M).
  • Combined Uncertainty Score: The combined score is UCOM(ŷ) = wCD · UCD + wIE · UIE + wTA · UTA, using one fixed weight set across all models.The fixed weights preserve a plug-and-play interface without model-specific tuning.
  • Selective Prediction and Calibration: The calibrated threshold τ controls the proportion of incorrect accepted predictions at the desired α level, while maximizing acceptance or minimizing abstention.Calibration uses a high-probability upper confidence bound derived from the Clopper–Pearson interval to provide finite-sample FDR guarantees.

4. Experiment

SafeGround is evaluated on six GUI-grounding models using ScreenSpot-Pro, where its distribution-aware uncertainty improves error discrimination and selective prediction over probabilistic confidence. Calibration provides high-probability FDR control, while uncertainty-aware cascading and component analyses demonstrate practical gains and robustness.

  • Models and Dataset: Experiments evaluate six GUI-grounding models on the challenging ScreenSpot-Pro benchmark.The models include Holo1.5, GUI-Actor, UI-TARS-1.5, and GTA1 variants.
  • Uncertainty Quality: SafeGround improves AUROC over probabilistic confidence when PC is available, including Holo1.5-7B from 0.6983 to 0.7526.For models without directly applicable PC, it achieves AUROC values up to 0.8155.
  • Uncertainty Quality: AUARC improves on Holo1.5-3B from 0.6444 to 0.6576 compared to PC, with consistent gains across models.AUARC measures accuracy gains as high-uncertainty predictions are progressively rejected.
  • FDR Control Guarantee: Clopper–Pearson calibration produces decision thresholds whose test-time FDR stays below the theoretical upper bound across tested models and risk levels.The resulting guarantee controls the error rate among accepted predictions at the specified level with high probability.
  • Power Comparison: UCOM consistently outperforms PC in power, retaining a larger fraction of correct predictions while satisfying the same FDR constraint.Higher power indicates that more reliable predictions are retained without violating the target FDR.
  • System-Level Accuracy: 58.66% accuracy is achieved by uncertainty-aware cascading with Holo1.5-7B at risk level 0.34, improving system accuracy over Gemini-only inference and base models.The approach consistently improves system accuracy across a wide range of feasible risk levels.
  • Sampling Efficiency: Increasing sampling from K = 5 to K = 10 improves AUROC and AUARC, while increasing K from 10 to 15 produces only marginal changes.The results indicate that effective uncertainty estimates can be obtained with a small number of samples.
  • Ablation of Uncertainty Components: The dominant uncertainty component depends on the model, and removing it causes clear AUROC and AUARC drops, supporting complementary cue combination.UT A is strongest on GTA1, whereas UCD is more effective for GUI-Actor-2VL and Holo1.5.

5. Conclusion

SAFEGROUND is an uncertainty-aware framework for reliable, risk-controlled GUI grounding under limited model access. It models spatial uncertainty from stochastic grounding samples and calibrates decision thresholds with finite-sample guarantees.

  • SAFEGROUND enables reliable and risk-controlled GUI grounding under limited model access.
  • Modeling spatial uncertainty from stochastic grounding samples captures distributional signals beyond point predictions.
  • The resulting uncertainty estimates effectively discriminate between correct and incorrect predictions.
  • SAFEGROUND calibrates decision thresholds with finite-sample guarantees based on uncertainty estimation.

Impact Statement

SAFEGROUND is presented as a framework for improving the reliability and safety of autonomous GUI agents. It addresses high-stakes, irreversible GUI grounding errors through principled uncertainty quantification with finite-sample statistical guarantees.

  • Impact Statement: SAFEGROUND significantly enhances the reliability and safety of autonomous GUI agents.The passage frames these improvements as the paper’s central impact.
  • Impact Statement: The framework provides the first principled method for uncertainty quantification in GUI grounding with finite-sample statistical guarantees.This contribution targets a critical bottleneck in deploying visual agents in the real world.
  • Impact Statement: SAFEGROUND addresses the risk of high-stakes, irreversible errors, including erroneous financial transactions, in real-world visual-agent deployment.The passage identifies such errors as a critical deployment bottleneck.

Limitation

SafeGround’s uncertainty estimation depends on variability across sampled predictions to characterize spatial ambiguity. For highly deterministic models with limited sampling diversity, its spatial distributions may be less informative, although the framework remains general and principled.

  • Limitation: The method estimates uncertainty from variability in sampled predictions to characterize spatial ambiguity.This approach relies on diversity among stochastic samples of model outputs.
  • Limitation: For highly deterministic models, limited sampling diversity can make the resulting spatial distributions less informative.The limitation arises when sampled predictions vary too little to represent ambiguity usefully.
  • Limitation: Despite this limitation, SafeGround provides a general and principled foundation for uncertainty-aware GUI grounding.

A. Proofs · B. Details of Experimental Settings

The proofs establish that the calibrated upper confidence bound provides a statistically guaranteed upper bound on system FDR. The guarantee applies at test time as a marginal FDR-control result under exchangeability.

  • A. Proofs: The section provides a complete proof that the upper confidence bound satisfies the statistical guarantee in Eq. (9).
  • A. Proofs: The bound is defined as sup{R : Pr(Bin(n, R) ≤ X) ≥ δ}, with n equal to the number of accepted calibration samples.
  • A. Proofs: Bin(n, R) models the number of errors among n accepted samples when the system FDR is R under threshold τ.
  • A. Proofs: The proof defines the CDF of the binomial error-rate random variable ˆR(τ) = Bin(n;R(τ))⁄n.
  • A. Proofs: The empirical error rate over accepted calibration samples is used with the inverse CDF to derive the upper confidence bound.
  • A. Proofs: At least 1 − δ confidence is obtained for the system FDR bound at threshold τ, yielding marginal test-time FDR control by exchangeability.

B.1. Dataset · B.2. Evaluation Metrics · B.3. Spatial Region Construction

ScreenSpot-Pro provides a challenging GUI grounding setting with 1,581 spatially annotated screenshots, while evaluation measures uncertainty discrimination, selective prediction, risk, and retained correctness. SafeGround constructs object-level spatial regions from stochastic grounding samples by discretizing coordinates, filtering density, and grouping adjacent patches.

  • B.1. Dataset: 1,581 UI screenshots pair natural-language instructions with spatial-region annotations for target elements in ScreenSpot-Pro.The benchmark has higher visual complexity, denser layouts, and finer distinctions between neighboring elements than earlier GUI grounding benchmarks.
  • B.2. Evaluation Metrics: Four metrics—AUROC, AUARC, FDR, and power—evaluate uncertainty estimation and selective prediction using the admission function.The admission function indicates whether a grounding prediction is admissible.
  • B.2. Evaluation Metrics: AUROC measures how well uncertainty scores distinguish inadmissible from admissible grounding predictions, with higher values indicating stronger discrimination.The metric is computed from the receiver operating characteristic curve obtained by thresholding uncertainty.
  • B.2. Evaluation Metrics: AUARC measures selective-prediction accuracy as high-uncertainty samples are rejected, with acceptance defined by U(ŷ_i) ≤ τ.In practice, AUARC is computed over accuracy as a function of the rejection rate.
  • B.2. Evaluation Metrics: FDR measures the proportion of inadmissible predictions among accepted predictions and is SafeGround’s primary controlled risk metric.Power measures the proportion of correct predictions retained under a risk constraint; higher power means retaining more correct predictions while satisfying the FDR constraint.
  • B.3. Spatial Region Construction: Stochastic decoding yields K point predictions, which are mapped onto an H × W patch grid and normalized into an empirical spatial predictive distribution.Patch counts C_u,v are normalized to estimate the model’s predictive density over the output space.
  • B.3. Spatial Region Construction: β = 0.3 density filtering followed by 4-connected grouping produces disjoint regions representing plausible grounding targets.Patches are retained when P_u,v > βP_max, using an instance-adaptive threshold; region scores average density within each region and are normalized for uncertainty metrics.

C. Threshold Calibration with Finite-Sample Guarantees · E. Case Study

SafeGround calibrates prediction thresholds with finite-sample selective-risk guarantees using Clopper–Pearson confidence bounds. Qualitative case studies illustrate how its uncertainty score reflects GUI grounding reliability in practice.

  • C. Threshold Calibration with Finite-Sample Guarantees: SafeGround uses a threshold calibration procedure to obtain finite-sample guarantees on selective prediction risk.The procedure is summarized in Algorithm 1.
  • C. Threshold Calibration with Finite-Sample Guarantees: The calibration procedure is based on Clopper–Pearson confidence bounds.These bounds support the stated finite-sample guarantees.
  • C. Threshold Calibration with Finite-Sample Guarantees: Algorithm 1 summarizes how SafeGround calibrates the threshold for selective prediction risk guarantees.The algorithm provides the procedural overview for this section.
  • E. Case Study: Qualitative examples examine how the proposed uncertainty score reflects GUI grounding prediction reliability.The examples are presented as practical illustrations rather than quantitative evaluations.
  • E. Case Study: Figure 10 illustrates the uncertainty score’s relationship to GUI grounding reliability in practice.The case study includes qualitative examples.
  • E. Case Study: Figures 11, 12, 13, and 14 provide additional qualitative examples of prediction reliability.Together with Figure 10, these figures comprise the reported case study.

F. Additional Experimental Results

Additional experiments show that SafeGround’s uncertainty measures benefit from higher sampling temperatures, while its combined uncertainty remains robust to moderate changes in component weighting. These findings support practical use of the framework across models.

  • Sensitivity to Sampling Temperature: Higher sampling temperatures consistently improve AUROC for UIE and UCD, whereas margin-based uncertainty changes relatively little.The temperature-sensitivity study evaluates Holo1.5-3B using AUROC and AUARC.
  • Sensitivity to Uncertainty Weighting: Across evaluated models, AUROC and AUARC show only minor fluctuations under alternative uncertainty-weighting schemes.The analysis varies the normalized contributions of the three uncertainty components in UCOM.
Loading 2602.02419v2…