PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaGround: Token-Normalized Adaptive Zooming with Verifier-in-the-Loop Refinement for GUI Grounding

Bingchao Wang, Yongcheng Jing, Chengyu Wang, Jun Huang, Jie Yang, Wei Liu, Dacheng Tao

arXiv:2610.36874v1cs.CVcs.AIcs.CL

TL;DR

Small targets make GUI grounding fragile on high-resolution screens, and existing training-free zooming often neither sizes zooms to targets nor checks answers. AdaGround adapts zooms, verifies and revisits predictions, and reports stronger benchmark results than prior training-free zooming methods, with gains on held-out UI-Vision.

  • Problem

    Training-free GUI refinement often uses zooms unrelated to target size, accepts zoomed predictions unconditionally, and does not revisit wrong answers.

    Source: §1 · p. 1

  • Method

    AdaGround combines target-size-based Token-Normalized Adaptive Zooming, verifier-guided zoom selection and re-grounding with abstention, and spreadsheet-cell composition from headers.

    Source: §1 · p. 2

  • Results

    AdaGround exceeds published MVP and ZoomClick results on OSWorld-G and ScreenSpot-Pro, with ScreenSpot-Pro gains over single pass of +14.9 and +20.6 points.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    Differently zoomed views fail on different samples, and a grounder asked about a marked click can choose between them, recovering more than a third of ScreenSpot-Pro headroom above the better chain.

    Source: §5 · p. 8

    Conclusions cover two backbones and four benchmarks, not agentic tasks; AdaGround also uses 6 to 12 times the visual tokens of a single pass.

    Source: Limitations · p. 9

Abstract

from arXiv · show

GUI grounding on high-resolution screens fails mostly on small targets, and zooming in is a common training-free remedy. However, most existing zooming pipelines do not match the zoom to the target's size, accept the zoomed answer unconditionally and never revisit a wrong answer. To address these issues, we propose AdaGround, a training-free framework with three components: (1) Token-Normalized Adaptive Zooming, which sets the magnification, field of view and zoom depth of each sample from the model's own estimate of the target size; (2) Verifier-in-the-Loop Refinement, which uses one marked-click Yes/No verifier to keep or reject each zoom, choose between zoom chains, trigger re-grounding and, with an existence query, decide when to abstain; and (3) Compositional Structure Grounding, which composes spreadsheet cells from their headers. On Qwen3-VL models, AdaGround surpasses state-of-the-art training-free zooming methods and many trained GUI models, improving accuracy over the backbone by 7.4 to 20.6 points on OSWorld-G and ScreenSpot-Pro, and its gains carry over to held-out UI-Vision. Code is available at https://github.com/bcwang-sjtu/AdaGround

1 Introduction

GUI grounding remains fragile on small targets in high-resolution software, while training-free zooming refines coarse predictions without retraining. AdaGround adapts zooms to target size, verifies and revisits answers, and reports gains on its evaluated benchmarks.

1 Introduction

  • Training-based specialists and learned focusing can improve grounding but require annotated data and backbone-specific retraining, motivating training-free test-time refinement.The refinement crops around a coarse prediction and re-grounds at higher resolution.Source: §1 · p. 1
  • AdaGround exceeds published MVP and ZoomClick results and trained GUI specialists; its largest ScreenSpot-Pro gains over single pass are +14.9 and +20.6 points.The comparison covers OSWorld-G and ScreenSpot-Pro, with gains also carrying over to held-out UI-Vision; ScreenSpot-v2 improvements are not significant.Source: Figure 1 · p. 1, §1 · pp. 2–3
  • Existing refinement uses scale-agnostic zooms, accepts zoomed answers unconditionally, and lacks answer checks for retrying errors or abstaining.Other cited shortcomings include reliance on model internals and voting, while targets absent from the screen or defined by spreadsheet structure pose additional challenges.Source: §1 · pp. 1–2
  • TNZ derives magnification, crop size, and zoom depth from the model’s coarse target box, aiming for a fixed visual-token span while covering localization error.AdaGround’s adaptive and fixed-crop views can be correct on different samples, motivating per-sample selection.Source: §1 · p. 2
  • VLR uses one marked-click Yes/No verifier to accept or reject zooms, choose between chains, trigger re-grounding with the rejected click crossed out, and support abstention.CSG handles structural targets by composing spreadsheet cells from separately grounded row and column headers; the framework uses the VLM as a black box.Source: §1 · pp. 2–3

2 Related Work

Related work spans GUI grounding models, test-time zooming, attention or vote-based aggregation, and verification or learned focusing. AdaGround instead changes how an off-the-shelf grounder is queried, using training-free zoom sizing and verification.

GUI grounding

  • GUI specialists fine-tuned on corpora or improved through post-training change the grounder itself, whereas AdaGround changes how an off-the-shelf grounder is queried.Examples include fine-tuned specialists (Gou et al., 2025; Wu et al., 2025b; Hong et al., 2024; Lin et al., 2025; Wu et al., 2026) and post-training or self-distillation (Liu et al., 2026b; Zhou et al., 2026; Qiu et al., 2026; Zhang et al., 2026c).Source: §2 · p. 3

Test-time zooming

  • Recent training-free zoom methods use fixed shrink ratios, size triggers, or candidate variance, rather than deriving zoom from the target’s visual-token span.ZoomClick (Jiang et al., 2025), AdaZoom-GUI (Pei et al., 2026), and UI-Zoomer (Tang et al., 2026a) accept zoomed predictions unconditionally.Source: §2 · p. 3

Attention-guided and vote-based aggregation

  • Attention-guided methods such as MVP use model internals, while spatial voting aggregates predictions; AdaGround uses generated text and first-token probabilities for verification instead.Re-Prefill (Lin et al., 2026a) also requires model internals; GUI-RC (Du et al., 2026) votes over sampled predictions on a spatial grid.Source: §2 · p. 3

Verification and feedback; learned focusing and agent memory

3 The AdaGround Framework

AdaGround maps screenshot–instruction pairs to a click or abstention using two zoom chains, a verifier, and compositional grounding for spreadsheet cells. Its zoom geometry is normalized to the target’s estimated visual-token span, while the verifier governs acceptance, chain choice, re-grounding, and abstention.

3.3 Verifier-in-the-Loop Refinement

  • The framework runs a TNZ chain and a fixed 4× chain, using one Yes/No verifier to keep zooms, choose between chains, trigger re-grounding, and support abstention.Rejected answers are crossed out before re-grounding, and a replacement is accepted only when its verifier score is higher; abstention is enabled only on OSWorld-G and does not override a CSG-composed answer.Source: §3 · p. 4, §3.3 · pp. 5–6

3.1 Problem Formulation

  • GUI grounding predicts a point from an instruction and screenshot, with correctness defined by whether the point falls inside the referenced element’s ground-truth region.AdaGround uses the backbone as a black box, reading generated text and first-token probabilities rather than attention maps, hidden states, or gradients.Source: §3.1 · p. 4

3.2 Token-Normalized Adaptive Zooming

  • TNZ uses the model’s estimated target extent to decide whether to zoom and how much, how wide, and how often to zoom.Its adaptive stages stop when the estimated span reaches the specified threshold or the maximum depth is reached.Source: §3.2 · pp. 4–5
  • TNZ estimates target size from a coarse-view box, then chooses magnification and crop geometry to render the target at a token-span target while covering context and localization error.The crop is centered on the current point and shifted inside the screenshot; a missing box prevents zooming.Source: §3.2 · p. 4

3.4 Compositional Structure Grounding

  • CSG identifies a spreadsheet cell by separately grounding its column and row headers, then combines their coordinates when the column header lies above and to the right of the row header.Otherwise, it falls back to the VLR answer.Source: §3.4 · p. 6

3.5 One Configuration

  • One configuration of constants is used across both backbones and all benchmarks, with the encoder determining only the token-pitch value.The listed settings include τ = 3, a maximum zoom factor of 4, and a maximum of two TNZ stages.Source: §3.5 · p. 6

4 Experiments

Across OSWorld-G and ScreenSpot-Pro, AdaGround outperforms published training-free zooming results, and it also performs competitively on held-out UI-Vision. Ablations attribute gains to token-based zoom sizing, verifier arbitration, spreadsheet-cell recovery, and abstention.

4.1 Experimental Setup

  • The evaluation covers 564 OSWorld-G, 1,581 ScreenSpot-Pro, and 1,272 ScreenSpot-v2 instructions, with 5,479 UI-Vision instructions held out.OSWorld-G credits absent-target instructions only when the system abstains; the other benchmarks use point-in-box scoring.Source: §4.1 · pp. 6–8

4.2 Main Results

  • AdaGround exceeds MVP’s published results by 3.4 and 7.3 points on OSWorld-G and 0.9 and 1.9 on ScreenSpot-Pro, respectively, across Qwen3-VL-8B/32B.It also exceeds ZoomClick’s published Qwen3-VL-32B ScreenSpot-Pro result; paired leads over MVP’s per-sample predictions are significant in every test.Source: Table 2 · p. 7, §4.2 · p. 8
  • On held-out UI-Vision, AdaGround scores 32.0% against MVP’s 31.9% with Qwen3-VL-8B and 47.6% against 44.1% with Qwen3-VL-32B.It also exceeds ZoomClick’s 39.2% result; UI-Vision was not used to develop or tune the configuration.Source: Table 3 · p. 7, §4.2 · p. 8, §4.1 · p. 8

4.3 Ablation: Where the Gains Come From

  • Against the fixed 4× crop, full AdaGround is significantly better on ScreenSpot-Pro, held-out UI-Vision, and OSWorld-G with Qwen3-VL-8B; crop gains stop rising once targets span two tokens.A crop pass raises accuracy by +7.8 and +10.8 points when the target spans under one token, motivating TNZ’s τ = 3 target.Source: Table 4 · p. 7, §4.3 · p. 8
  • On ScreenSpot-Pro, chain arbitration raises cTNZ by 5.9 and 3.2 points, recovering 39% and 36%, respectively, of the distance from the better chain to the oracle.The two chains disagree on 45% and 34% of samples, respectively.Source: §4.3 · p. 8
  • CSG recovers 2 and 5 OSWorld-G spreadsheet-cell samples net, while abstention adds 2.5 and 1.4 points and withdraws at most one correct answer.The two chains with arbitration alone, at about 10 calls, already beat the fixed crop on ScreenSpot-Pro.Source: §4.3 · p. 8

5 Conclusion

AdaGround is a training-free, black-box framework that addresses shortcomings in GUI test-time refinement. The conclusion is that zoom geometries fail on different samples, so a grounder can use a marked-click check to choose between them.

5 Conclusion

  • The conclusion frames AdaGround as a black-box approach to five shortcomings of test-time refinement for GUI grounding.Its stated components include token-normalized zooming, verifier-in-the-loop refinement, and compositional structure grounding.Source: §5 · p. 8
  • AdaGround uses one configuration across Qwen3-VL-8B and Qwen3-VL-32B, outperforming MVP’s published results on OSWorld-G and ScreenSpot-Pro and many trained GUI models, including 72B models.It is also significantly above its fixed 4× crop chain on ScreenSpot-Pro and held-out UI-Vision.Source: §5 · p. 8
  • Different zoom geometries fail on different samples, and the grounder can choose between them when asked about a marked click.On ScreenSpot-Pro, that choice recovers more than a third of the headroom above the better chain.Source: §5 · p. 8

Limitations

AdaGround’s evaluation is limited to two Qwen3-VL backbones and four benchmarks, not longer agentic episodes. Its added compute has weaker value on large-target tasks, while several components have specific assumptions or limitations.

Scope

  • The conclusions cover Qwen3-VL-8B and Qwen3-VL-32B on OSWorld-G, ScreenSpot-Pro, ScreenSpot-v2, and held-out UI-Vision, not agentic tasks where grounding is one step of a longer episode.The Qwen3-VL-32B OSWorld-G comparison with MVP uses its published number and is not paired.Source: Limitations · p. 9

Cost

  • AdaGround reads 3.5–3.9 times the visual tokens of the fixed 4× crop and has 3.4–4.2 times its latency; on large-target ScreenSpot-v2, it matches that crop at higher cost.It also reads 6–12 times the single-pass visual tokens.Source: Limitations · p. 9

Components

  • TNZ assumes the model returns a box for its box query; a point-only response receives no TNZ zoom.A target outside an over-tight view is rejected by verification and left to the fixed 4× chain and re-grounding.Source: Limitations · p. 9

Verifier and abstention

  • A verifier that favors a plausible but wrong element can pass that preference to acceptance, re-grounding, arbitration, and abstention decisions.The abstention rule is tuned for few false refusals and catches 15 and 8 of OSWorld-G’s 54 absent-target instructions.Source: Limitations · p. 9

Structure

  • CSG covers spreadsheet cells but not slider positions or text-cursor placement; on spreadsheet cells, single-pass answers perform as well as composed answers.Source: Limitations · p. 9

Ethical Considerations

AdaGround is an inference-time procedure that uses no model training or new data, but more capable GUI agents can still cause harm or be misused. The appendix contents include implementation details, prompts, and analyses.

Ethical Considerations

  • A wrong click can trigger an unintended, possibly irreversible action, and capable agents may be misused to automate interactions intended for humans.Abstention removes one failure mode but does not replace user confirmation before consequential actions.Source: Ethical Considerations · p. 9
  • AdaGround trains no model and collects no new data; experiments use screenshots and instructions from four public GUI-grounding benchmarks, with no human subjects.Source: Ethical Considerations · p. 9

A Implementation Details and Prompts. . 13

A Implementation Details and Prompts

The appendix specifies AdaGround’s inference procedure, prompts, parsing, and evaluation controls. It also reports how development choices, reproducibility, and compute costs shape interpretation of the results.

A Implementation Details and Prompts

  • The fixed 4× baseline re-grounds in a 1288×728-pixel window centered on the single-pass answer, while the single-pass baseline uses the whole screenshot.Both baselines come from the same runs as AdaGround.Source: A Implementation Details and Prompts · p. 13
  • All components and constants were developed on OSWorld-G, ScreenSpot-Pro, and ScreenSpot-v2, while UI-Vision was evaluated with final constants and not tuned for this study.The three development benchmarks were not held out, and the reported results include the effect of those choices.Source: A Implementation Details and Prompts · p. 13

A.2 Prompt Template and Output Parsing

  • Point outputs are mapped from the rendered-view coordinate grid to screenshot pixels using the crop’s actual corner and size; a missing point leaves the prior answer unchanged.The box query’s shorter side, rescaled to original pixels, supplies the target-size estimate.Source: §A.2 · p. 14
  • Outputs are reproducible from public model weights, public benchmark inputs, and the reported settings, but continuous batching can prevent bit-exact outputs across runs.Source: §A.2 · p. 14
  • The overall procedure runs TNZ and a fixed-crop chain, groups their answers, verifies competing representatives, and may re-ground rejected locations or compose spreadsheet-cell clicks from headers.A cell answer is composed only when the column header lies right of and above the row header; otherwise the VLR answer remains.Source: §A.2 · p. 15, §A.3 · p. 16
  • TNZ estimates target size from a box query, adapts zoom depth and crop geometry, and accepts a zoomed answer only when its verifier score does not decrease.The procedure stops when the target’s estimated token span reaches its threshold or a depth condition is met.Source: §A.2 · p. 15

A.3 Verbatim Prompts

  • The appendix provides prompts for grounding, size estimation, marked-click verification, negative-feedback re-grounding, and instruction-to-program rewriting for structural targets.The rewrite routes spreadsheet-cell instructions to CELL and ordinary element instructions to GROUND.Source: §A.3 · pp. 15–16

A.4 Compute and Cost

  • AdaGround takes 3.4–4.2 times the fixed 4× crop’s latency and reads 3.5–3.9 times its visual tokens; Table 6 reports accuracy, tokens, calls, and latency.Latency is measured with sequential calls on 150 random samples per benchmark.Source: §A.4 · p. 16, Table 6 · p. 16

A.1 Implementation Details

Implementation uses Qwen3-VL backbones, a staged rendering and zoom procedure, and shared constants. Abstention is limited to the benchmark that permits refusal; the ablation intervals are computed as described in the appendix.

A.1 Implementation Details

  • Qwen3-VL-8B and Qwen3-VL-32B run with greedy decoding on vLLM; views are resized to a 32-pixel grid, and grounding passes generate at most 32 tokens.Source: §A.1 · p. 13
  • TNZ uses adaptive crop stages and updates the target-size estimate after each zoom; a zoomed answer replaces the current answer only when its verifier score is at least as high.The verifier uses a marked candidate in an independent window, and the existence query is whole-screen.Source: §A.1 · p. 13
  • One set of constants serves both backbones and all benchmarks, with only the token pitch π read from the encoder.The constants include TNZ thresholds, verifier-window geometry, arbitration settings, re-grounding, and abstention parameters.Source: Table 5 · p. 14
  • Only OSWorld-G admits refusal, and the pipeline never abstains on a CSG-composed answer.Source: §A.1 · p. 14

B Ablations and Component Analyses

  • Ablation intervals are computed using the procedure described in Appendix D.2, with the full benchmarks and the Section 3.5 configuration except for the varied quantity.Source: B Ablations and Component Analyses · p. 16

B.1 Components

Component analyses show that abstention and chain arbitration contribute under different conditions, while target-size estimates and design choices qualify how TNZ behaves. The fixed-geometry controls and ScreenSpot-v2 results also show that refinement gains depend on benchmark and target scale.

B.1 Components

  • On ScreenSpot-Pro, chain arbitration raises cTNZ by +5.9 and +3.2 points, while its gains over cTNZ are smaller on OSWorld-G and ScreenSpot-v2.On ScreenSpot-Pro, arbitration recovers 38.8% and 35.9% of the headroom between the better chain and the oracle.Source: §B.1 · p. 17
  • Abstention adds 2.5 and 1.4 points on OSWorld-G, accounting for about half or more of AdaGround’s gain over the fixed crop.Without abstention, that lead is significant only on Qwen3-VL-8B.Source: §B.1 · p. 17

B.2 Abstention

  • The joint abstention rule recalls 27.8% and 14.8% of absent OSWorld-G targets at false-refusal rates of 0.8% and 0.4%.The existence query alone withdraws many answers whose target is present; applying the rule to fixed-crop or MVP answers also raises accuracy in some settings.Source: §B.2 · p. 17

B.3 Quality of the Size Estimate

  • TNZ’s size estimates under-magnify on average on ScreenSpot-Pro and over-magnify slightly on ScreenSpot-v2, yet using true target size changes arbitrated accuracy by only −0.6 to −0.2 points on Qwen3-VL-8B.A target outside an over-tight view can be rejected by verification but not recovered.Source: §B.3 · pp. 17–18

B.4 Design Choices

  • Removing a component causes significant losses only for abstention on OSWorld-G and re-grounding on Qwen3-VL-8B ScreenSpot-Pro; negative feedback changes re-grounded answers but not overall accuracy significantly.Accuracy changes by at most 0.6 points at τ = 2 or τ = 4, and the first TNZ zoom carries most of its gain over the coarse point.Source: §B.4 · p. 18

B.5 Chain Arbitration

  • When the two chains disagree, the verifier selects the right chain on 78.0% of 227 Qwen3-VL-8B ScreenSpot-Pro cases, versus 63.4% for always choosing the fixed crop.The corresponding Qwen3-VL-32B rates are 72.4% and 56.9% across 174 cases.Source: §B.5 · p. 18

B.6 Fixed-Geometry Controls

  • The fixed-geometry control uses 41–47% fewer visual tokens on OSWorld-G and ScreenSpot-Pro without significant accuracy differences, but its 540-pixel window is too narrow for ScreenSpot-v2 targets.It is significantly worse after arbitration on ScreenSpot-v2 and held-out UI-Vision, though these losses are no longer significant in the full pipeline.Source: §B.6 · p. 18

C Reproduced Baselines and ScreenSpot-v2

D.1 MVP Setup

The appendix describes the MVP reproduction and scoring procedures, including paired screenshot-bootstrap tests. It also provides baseline and fixed-geometry-control tables, but the supplied passages do not report their detailed values.

D.1 MVP Setup

  • Table 9 compares alternative second-chain geometries with cTNZ while leaving the rest of the pipeline unchanged.Its caption states that values are accuracy percentages and gives paired differences to cTNZ, with 95% confidence intervals for arbitration.Source: Table 9 · p. 19
  • The tables report overall benchmark accuracy and category-level reproduced baselines, with UI-Vision averaged over three tasks.MVP values may come from official-code reproduction, released predictions, or published results, as indicated in Table 10’s caption.Source: Table 10 · p. 19, Table 11 · p. 20
  • The MVP reproduction scores below published results, as does MVP’s own single pass, while reproducing its gain over the single pass.For OSWorld-G and ScreenSpot-Pro, the reproduced single pass scores 65.1% and 51.5%, versus published 68.8% and 55.0%; MVP’s gains are +4.8 and +10.4 points versus published +3.9 and +10.3.Source: §D.1 · p. 19

D.2 Scoring and Paired Tests

  • OSWorld-G credits an absent-target instruction only when the system abstains; other benchmarks use point-in-box scoring with abstention off.For paired differences, 95% intervals come from 4,000 screenshot-bootstrap resamples, with samples from a screenshot resampled together.Source: §D.2 · p. 20

E Token-Span Analysis

The token-span analysis defines target legibility in the rendered view and distinguishes it from an area-based measure. For a square target, up-sampling a crop increases token span even when the target’s area ratio stays unchanged.

E.1 Definition, Relation to Area Measures, and Pitch Sensitivity

  • Token span σ = sk/π measures target size in the rendered view, whereas effective tokens reff = T(AI) · At/AI is defined on the whole screenshot.For a square target at k = 1, reff ≈ σ^2; a 4× up-sampled crop leaves At/AI unchanged but doubles σ.Source: §E.1 · p. 21
  • A 24-pixel target on a 3840×2160 screenshot spans 0.4 tokens at k ≈ 0.5, 0.75 at k = 1, and 1.5 after a 4× crop; Eq. (2) selects k = 4 to reach 3.0.The example uses π = 32 for Qwen3-VL and the constants of Table 5.Source: §E.1 · p. 21
  • The pitch π fixes the unit, but the target size must be measured after rendering because the same target appears at different scales on different screens.Source: §E.1 · p. 21

E.2 Accuracy versus Token Span

Across views containing the target, accuracy rises steeply below about three visual tokens, while crops can also discard useful context. The analysis motivates TNZ’s token-span criterion, verification of zooms, and separate handling for targets without visual evidence.

E.2 Accuracy versus Token Span

  • Below about three tokens, pooled accuracy climbs steeply; crops gain +7.8 and +10.8 points below one token, with no significant gain once span reaches two tokens or more.The analysis covers 15,488 view–sample pairs and calls the trend on three benchmarks with Qwen3-VL-8B the token-span regularity.Source: §E.2 · p. 21, Table 14 · p. 23
  • Four-times up-sampling of the same crop changes token span alone and gains 2.6, 6.6, and 0.9 points on ScreenSpot-Pro, OSWorld-G, and ScreenSpot-v2, respectively.The ScreenSpot-v2 interval includes zero; one ScreenSpot-Pro bin with four to six tokens runs against the trend but contains only 56 views from 41 screenshots.Source: §E.2 · p. 21

E.3 The 2×2 Control and ScreenSpot-v2

  • Whole-screen rendering at four times the area grows median span 1.7× without significant accuracy change, while native-scale cropping changes ScreenSpot-Pro and ScreenSpot-v2 accuracy by +8.1 and −3.5 points.The control indicates that added span without restricting the view brings no gain; cropping combines higher span with lost context and candidate locations.Source: §E.3 · p. 22

E.4 What a Well-Placed Crop Leaves Behind

  • On ScreenSpot-Pro, an oracle crop centered on the ground truth resolves only 287 of 787 single-pass failures and breaks 50 correct answers; in 52% of remaining failures, the target is still inside the crop.These cases motivate signals to accept or reject zoomed answers, push the model off a wrong choice, and choose between disagreeing views.Source: §E.4 · p. 22

E.5 Targets without Visual Evidence

  • Spreadsheet cells depend on row and column headers outside a tight crop, while OSWorld-G absent targets earn credit only when the system abstains.The 54 absent-target instructions correspond to 9.6 points that an always-answering pipeline forfeits.Source: §E.5 · pp. 22–23

E.6 From the Analysis to the Design

  • TNZ uses a looser first-stage threshold because crops help most below one token; every zoom is verified because span does not capture lost context.Negative-feedback re-grounding addresses errors with the target in view, while CSG and abstention address targets without visual evidence.Source: §E.6 · pp. 23–24

Related papers

Loading 2610.36874v1…