Source-linked AI summary
UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding
Fei Tang, Bofan Chen, Zhengxi Lu, Tongbo Chen, Songqin Nong, Tao Jiang, Wenhao Xu, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen
TL;DR
GUI grounding remains challenging for small icons and dense layouts, and existing zoom-in methods use uniform triggers and fixed crop sizes. UI-Zoomer uses uncertainty-aware gating and variance-based adaptive cropping, improving three benchmarks by up to +13.4%, +10.3%, and +4.2% without additional training.
Problem
GUI grounding models still fail on small icons and dense layouts, while existing zoom-in methods do not adapt whether or how much to zoom to each instance.
Method
UI-Zoomer is a training-free framework that fuses spatial consensus with token-level confidence and uses variance decomposition to adapt crop size for uncertain instances.
Results
+13.4%, +10.3%, and +4.2% gains are achieved on ScreenSpot-Pro, UI-Vision, and ScreenSpot-v2, respectively, across four model architectures.
Takeaways & Limitations
UI-Zoomer supports zooming only when uncertain and scaling the crop according to prediction disagreement as a test-time scaling principle for GUI grounding.
Takeaways & Limitations
UI-Zoomer struggles with multiple similar-looking icons, strong visual distractors, and extremely small targets in cluttered layouts.
Abstract
from arXiv · showhide
GUI grounding, which localizes interface elements from screenshots given natural language queries, remains challenging for small icons and dense layouts. Test-time zoom-in methods improve localization by cropping and re-running inference at higher resolution, but apply cropping uniformly across all instances with fixed crop sizes, ignoring whether the model is actually uncertain on each case. We propose \textbf{UI-Zoomer}, a training-free adaptive zoom-in framework that treats both the trigger and scale of zoom-in as a prediction uncertainty quantification problem. A confidence-aware gate fuses spatial consensus among stochastic candidates with token-level generation confidence to selectively trigger zoom-in only when localization is uncertain. When triggered, an uncertainty-driven crop sizing module decomposes prediction variance into inter-sample positional spread and intra-sample box extent, deriving a per-instance crop radius via the law of total variance. Extensive experiments on ScreenSpot-Pro, UI-Vision, and ScreenSpot-v2 demonstrate consistent improvements over strong baselines across multiple model architectures, achieving gains of up to +13.4\%, +10.3\%, and +4.2\% respectively, with no additional training required.
1 Introduction
GUI grounding remains difficult for small icons and dense layouts, while existing zoom-in methods crop uniformly without adapting the trigger or scale to instance uncertainty. UI-Zoomer addresses this with uncertainty-aware routing and adaptive cropping, improving results across three benchmarks without additional training.
- GUI grounding models still fail systematically on small icons and dense layouts in complex interfaces.
- Existing zoom-in methods leave open which instances need zoom-in and how much to zoom.
- Unconditional or error-triggered cropping ignores instance uncertainty, while fixed crop ratios disregard candidate disagreement and can increase latency or remove useful global context.
- UI-Zoomer fuses spatial consensus and token-level confidence to route confident cases to consensus voting and uncertain cases to refinement.
- UI-Zoomer improves over strong baselines across ScreenSpot-Pro, UI-Vision, and ScreenSpot-v2, with gains up to +13.4%, +10.3%, and +4.2%, respectively.The framework is training-free and uses adaptive crop sizing based on prediction uncertainty.
2 Related Work
GUI grounding has progressed from pipeline-based systems to end-to-end and reinforcement-trained VLMs, while test-time scaling has introduced zoom-in and prediction-consistency approaches.
- Early GUI grounding systems chain OCR, icon detectors, and language models for planning and element selection.
- End-to-end VLMs trained on large GUI corpora demonstrate strong cross-platform generalization.
- Reinforcement fine-tuning uses coordinate-accuracy rewards and can match or exceed SFT models trained on much larger datasets.
- Test-time GUI scaling includes fixed-ratio iterative zooming, error-triggered cropping, density-based crop selection, iterative narrowing, and prediction-consistency reliability signals.
3 Method
UI-Zoomer samples multiple localization hypotheses, combines spatial agreement with token confidence for reliability gating, and adaptively crops uncertain cases before deterministic re-inference.
- Problem Setup: The method predicts a normalized click location from a screenshot and instruction, representing each hypothesis as a bounding box whose center defines the click.
- Pipeline: UI-Zoomer proceeds through global multi-sampling, reliability gating, and adaptive crop-and-zoom stages.
- Adaptive Crop and Zoom: For uncertain cases, candidate boxes are filtered and variance-decomposed to construct an adaptive crop window before zoomed deterministic re-inference.
- Global Multi-Sampling: The model samples N=8 candidate boxes at temperature T=0.9, discards invalid parses, and records each valid box with a token-probability confidence.
- Reliability Gating: Spatial consensus and average token confidence are complementary reliability signals: high scores return a consensus-voted prediction, while lower scores trigger refinement.
3.4 Stage 3: Uncertainty-Driven Adaptive Crop
UI-Zoomer derives each zoom-in crop from candidate uncertainty rather than a fixed crop ratio. It combines positional disagreement and predicted element extent, then refines the target with a single deterministic pass.
- When S ≤τ, UI-Zoomer triggers zoom-in because the candidates are considered unreliable.
- Outlier filtering: The method discards outliers and computes subsequent uncertainty statistics over the K retained candidates.It retains the K = ⌊0.75N⌋ candidates whose centers are closest to the median center.
- Variance decomposition: UI-Zoomer applies the law of total variance to decompose crop uncertainty into inter-sample positional spread and intra-sample box extent.The inter-sample term captures disagreement across draws, while the intra-sample term represents predicted element scale.
- Variance decomposition: The two variance terms have complementary effects: positional disagreement expands the crop, while box extent prevents it from being smaller than the predicted element.
- Adaptive crop sizing: The crop radius is set as r = γσ, with σ determined from both variance components, then constrained by a minimum side length and squarified.
- Refinement: The cropped image is resized to the model’s resolution budget, processed with one deterministic pass, and mapped back to global normalized coordinates.Invalid refined boxes fall back to the most confident global candidate.
4 Experiments
Experiments evaluate UI-Zoomer across three GUI grounding benchmarks and four model architectures, finding consistent gains over strong baselines. Results also show larger benefits for icon targets and improvements over naive sampling and prior zoom-in methods.
- Evaluation Setup: Evaluations span ScreenSpot-Pro, UI-Vision, and ScreenSpot-v2, using general-purpose and GUI-specific vision-language models.The model set includes Qwen2.5-VL-7B, UI-Venus-7B, UI-Venus-72B, and GUI-G2-7B.
- Main Results: UI-Zoomer consistently improves all four models across all three benchmarks, with average gains of up to +13.4%, +10.3%, and +4.2% on ScreenSpot-Pro, UI-Vision, and ScreenSpot-v2 respectively.The reported results cover the main benchmark tables and the full multi-model evaluation.
- Main Results: Icon targets gain more than text targets on ScreenSpot-Pro, improving by +12.5% versus +11.1% across models.The paper associates this pattern with compact, semantically ambiguous elements being more limited by resolution in a single forward pass.
- Comparisons: UI-Zoomer reaches 61.8% at a comparable inference budget, versus 54.84% and 58.19% for UI-Venus-7B naive sampling at pass@4 and pass@8.It also substantially outperforms RegionFocus at 32.1% on ScreenSpot-Pro.
- Comparisons: The method surpasses UI-S1-7B by +31.2% and GTA1-7B by +11.7% on ScreenSpot-Pro, indicating gains relative to RL-trained methods.The comparison uses UI-Venus-7B with UI-Zoomer against the reported RL-trained baselines.
5 Ablation Study
Ablations show that combining complementary confidence signals and adapting crop geometry improves grounding, while moderate gating selectively zooms uncertain cases without the cost of indiscriminate cropping.
- Gating signals: 61.80% accuracy results when spatial consistency and average token confidence are combined, versus 60.81% and 61.10% when used alone.The two signals capture spatial variability and token-level certainty, respectively.
- Crop sizing: 61.80% accuracy is achieved by Gaussian adaptive cropping, exceeding the 61.35% of the best reported fixed-ratio alternative.A 0.8 ratio retains too much background and achieves 55.22%, while 0.3 risks removing context.
- Boundary handling: 61.80% accuracy is achieved by shifting out-of-bounds crop windows inward, outperforming clipping at 60.25% and shrinking at 58.47%.Shift preserves crop size while avoiding changes to the effective crop area.
- Candidate filtering: ρ = 75% candidate retention achieves 61.80% accuracy by balancing outlier removal against retaining predictions.Keeping 50% reaches 60.37%, while unfiltered candidates include noisy outliers that inflate the crop window.
- Crop geometry: +1.24 percentage points results from enforcing a square crop, improving accuracy from 60.56% to 61.80%.The square crop better preserves visual context across varying UI-element aspect ratios.
- Gating threshold: Moderate gating thresholds outperform both CROP%=0 and CROP%≈100% across three models, while full cropping nearly doubles inference time from ∼5:50 to ∼10:20.Very high thresholds crop nearly everything, whereas very low thresholds approach the baseline.
- Sampling: 61.80% accuracy occurs at T=0.9, rising from 54.46% at T=0.1 before marginally declining at T=1.0.The ablation links higher candidate diversity with better consensus-based crop estimation.
- Case studies: Successful cases use scattered candidate boxes to identify a crop and refine the target, while dense layouts with similar tiny icons remain failure cases.The case study attributes failures to strong distractors and ambiguous cues.
6 Conclusion
UI-Zoomer frames both zoom-in triggering and scale as prediction-uncertainty problems, using adaptive gating and variance-based cropping. Across three benchmarks and four architectures, it reports gains of up to +13.4%, +10.3%, and +4.2%.
- Conclusion: UI-Zoomer treats zoom-in triggering and scale as prediction uncertainty quantification problems.It uses a reliability gate and variance-derived crop window.
- Conclusion: The method selectively routes uncertain instances to adaptive cropping after fusing spatial consensus with token-level confidence.The crop window is derived from a variance decomposition.
- Conclusion: +13.4%, +10.3%, and +4.2% are the reported maximum gains on ScreenSpot-Pro, UI-Vision, and ScreenSpot-v2, respectively.The conclusion presents these as improvements across the evaluated benchmarks.
A Appendix
The appendix provides additional implementation and evaluation details complementing the main paper.
- Appendix: The appendix supplements the main paper with additional implementation and evaluation details.
A.1 Prompt Template
The experiments use one unified prompt template across settings to support fair comparisons among models and evaluation scenarios.
- Prompt Template: A unified prompt template is used consistently across models and evaluation settings.The stated purpose is to ensure fair comparisons.
A.2 Comprehensive Comparison on ScreenSpot-v2 and UI-Vision
Tables 10 and 11 provide comprehensive evaluations of UI-Zoomer on ScreenSpot-v2 and UI-Vision against a broad range of existing baselines. Across these benchmarks, UI-Zoomer consistently improves localization performance over corresponding base models.
- UI-Zoomer is evaluated on ScreenSpot-v2 and UI-Vision against a broad range of existing baselines.The results cover different models, environments, and task settings.
- The experiments provide a comprehensive evaluation across different models, environments, and task settings.
- UI-Zoomer consistently improves localization performance over the corresponding base models.
A.3 More Ablations
The gating ablation shows that samples routed to Gating Pass achieve higher accuracy than samples routed to Crop, supporting the gating score as a confidence signal. Tables 10 and 11 define the evaluated model configurations for ScreenSpot-v2 and UI-Vision.
- Gating mechanism: Samples routed to the Gating Pass branch consistently exhibit significantly higher accuracy than those sent to the Crop branch.
- Gating mechanism: The gating score S reliably reflects prediction confidence.
- Benchmark configurations: ScreenSpot-v2 comparisons evaluate UI-Zoomer across Qwen2.5-VL-7B, GUI-G2-7B, and UI-Venus-7B.
- Benchmark configurations: UI-Vision comparisons evaluate UI-Zoomer across Qwen2.5-VL-7B, GUI-G2-7B, UI-Venus-7B, and UI-Venus-72B.
- Ablation setting: The gating-threshold ablation uses UI-Venus-7B on ScreenSpot Pro with σ set to 2.5.