RankGround: Efficient High-Resolution GUI Grounding via Lightweight Reranker-Guided Crop Selection

• Direct inference is fast but loses small-element detail after screenshot resizing; multi-crop methods expose detail but require repeated VLM calls and can yield conflicting predictions.Figure 1 contrasts these approaches with RankGround, which matches or exceeds multi-crop accuracy using one VLM call.Source: §1 · pp. 1–2, Figure 1 · p. 2
• RankGround ranks dense candidate crops with GroundRanker, then uses one GUI-specialist VLM call to predict coordinates from the top-ranked crop.Source: §1 · p. 2
• Strict containment labels require each positive crop to fully enclose the target, while boundary-aware augmentation spreads targets across crop positions.Source: §1 · p. 2
• GroundRanker first learns containment with a pointwise objective, then uses a listwise objective to distinguish near-miss crops from valid ones.Source: §1 · p. 2
• 5.5% higher average localization accuracy than the second-best method and 1.4× faster inference across all tested backbones and scales.Source: §1 · p. 2

2.1 GUI Grounding

• Early GUI grounding adapted natural-image methods [12] [16] [19] [20] [27], but tiny resized elements and different scale distributions limited transfer to screenshots.Source: §2.1 · p. 2
• SeeClick [2] introduced ScreenSpot and demonstrated the value of GUI pre-training; UGround [5], UI-TARS [21], MAI-UI [32], and UI-Venus-1.5 [24] scaled this approach.Source: §2.1 · p. 2
• ZoomIn [32], ScreenSeekeR [11], ZoomClick [7], and GOLD revisit regions; MVP [31] aggregates views, while CoG [10] refines coordinates.Source: §2.1 · p. 2
• GUI-ARP [29] and Focus [22] learn staged inference, but still spend extra forward passes on selection or verification; RankGround assigns selection to a reranker and calls the grounding VLM once.Source: §2.1 · p. 2

2.2 Multimodal Reranking

• RankGround uses multimodal reranking to score crop relevance for GUI region selection, which the authors describe as the first multimodal-reranker use for this task.Source: §2.2 · p. 2

3.1 Overview

• RankGround narrows the VLM input to one selected crop, leaving coordinate prediction to a single high-resolution VLM pass.The candidate set is generated through coverage-preserving tiling, and the selected crop's local coordinates are mapped back to the original image.Source: §3.1 · pp. 2–3, Figure 2 · p. 3, §3.3 · p. 4

3.2.1 Overlap Tiling-based Candidate Generation.

• Overlap Tiling uses a G × G grid with overlap ratio ρ and (2G − 1)^2 candidates; ρ = 0.5 guarantees full containment for targets smaller than a crop.Candidates must contain the full target because otherwise the reranker cannot recover it.Source: §3.2.1 · p. 3
• For targets smaller than a crop, ρ = 0.5 yields 99.75% Oracle coverage under the default G = 2 layout, while wide targets may cross every crop boundary.Each crop covers about 25% of the screenshot; the reported 90th-percentile target areas are 0.156% for text and 0.034% for icons.Source: §3.2.1 · p. 3

3.2.2 Data Preparation for GroundRanker.

• Crop-level ranking labels are derived from annotated screenshots, and a crop is positive only when it fully contains the target box.Boundary-aware augmentation creates up to 12 variants placing the target at corners, edge midpoints, and nearby boundary positions; listwise samples pair a positive or variant with every same-image negative crop.Source: §3.2.2 · pp. 3–4

3.2.3 GroundRanker Training.

• GroundRanker is a cross-encoder that jointly scores each crop and instruction, using the affirmative-token logit as its ranking score.LoRA adapts query and value projections in the vision encoder's self-attention and language model's cross-attention while preserving pretrained features.Source: §3.2.3 · p. 4
• The pointwise-to-listwise curriculum first learns spatial containment, then contrasts valid crops with overlapping near-misses to sharpen their boundaries.Boundary-aware positives reduce position bias, and the listwise stage uses normalized labels.Source: §3.2.3 · p. 4

3.2.4 Region Selection.

• At inference, GroundRanker scores the candidate crops and passes the selected crop to coordinate prediction, so the VLM processes one region rather than the full candidate set.The VLM predicts a local coordinate within the upsampled crop, which is mapped back to the original image.Source: §3.2.4 · p. 4, §3.3 · p. 4

3.3 Coordinate Prediction and Mapping

• Region selection costs O(T_R) rather than O(NT_R), and coordinate prediction brings total latency to O(T_R + T_V), with T_R ≈ 145 ms and T_V ≈ 210–810 ms.RankGround is slower than Direct but avoids repeated VLM calls while preserving local crop detail.Source: §3.3 · p. 4

Dataset and Evaluation Metrics

• ScreenSpot-Pro evaluation covers 1,581 tasks from 23 professional applications, five industries, and three operating systems, with UI-Vision used for cross-benchmark generalization.Click Accuracy is micro-averaged over 977 Text and 604 Icon/Widget targets; reranker ablations report Hit Rate@k.Source: §4 · p. 4

Baselines

• The baselines are Direct full-screenshot inference, ZoomIn's two-call coarse-to-refined prediction, and MVP's attention-guided region aggregation.The study tests Qwen3-VL, MAI-UI, and UI-Venus-1.5 at both 8B and 2B scales.Source: §4 · pp. 4–5

Implementation

• GroundRanker is trained on 6,789 ViSurf instances, with data construction yielding about 295K pairwise, 124K pointwise, and 78K listwise samples.Latency is measured per sample at batch size 1, including preprocessing and post-processing but excluding data loading.Source: §4 · p. 5

5.1 Main Results

• RankGround matches or exceeds MVP on every backbone; MVP takes 2,342–4,648 ms per sample, or 4–12× longer.Peak memory is 21.2 GB at 8B and 8.6 GB at 2B with co-resident execution; sequential execution lowers these to 16.94 and 4.38 GB.Source: §5.1 · p. 5
• Published ScreenSpot-Pro scores offer wider context, but differ in backbones and training protocols: RankGround reaches 64.7% at 2B and 71.0% at 8B with one grounding-VLM call plus one reranker pass.The cited comparison reports 48.1% for ScreenSeekeR, 60.8% for GUI-ARP, and 68.4% for CoG.Source: §5.1 · p. 5

5.2.1 Partitioning Strategy.

• Overlap 2×2 retains 99.75% Oracle with nine crops, 43.3% Gain@1, and 145 ms latency; larger layouts improve some ranking scores but increase crop counts and latency.Finer uniform grids reduce Oracle accuracy, while overlapping grids and sliding windows keep it above 98.8%.Source: §5.2.1 · p. 5

5.2.2 Reranker Design.

• The reranker-design analysis reports that the pointwise-to-listwise curriculum reaches 88.2% Hit@1 across three seeds and improves 2B Text accuracy by 0.62 points over pointwise-only training.The base reranker remains more than 10 points below Oracle accuracy.Source: §5.2.2 · p. 5

5.1 Main Results

• RankGround leads all six backbone-scale settings with one GUI-specialist VLM call, improving on Direct by 12.08–19.36 points and averaging 2.3 points higher at 8B and 5.4 at 2B.It also uses a 145 ms reranker pass per query.Source: §5.1 · p. 5, Table 2 · p. 6

5.2.3 Prompt Design.

• The optimized prompt consistently improves Hit@1 across partitioning strategies, with its largest gain of 5.4 points on Overlap 2×2, without changing reranker weights.It asks directly whether the crop contains the target.Source: §5.2.3 · pp. 5–6, Table 4 · p. 6

5.3 Reranker Generalization

• RankGround reaches 78.70% Hit Rate on UI-Vision without benchmark-specific fine-tuning.The reported setup uses Overlap 2×2 and UI-Venus-1.5-2B.Source: §5.3 · p. 6, Table 5 · p. 6
• On UI-Vision, RankGround reaches 49.00% end-to-end accuracy, compared with 18.88% for Direct.The table reports oracle coverage and accuracy on out-of-distribution benchmarks using Overlap 2×2 and Qwen3-VL-2B.Source: §5.3 · p. 6, Table 6 · p. 7
• UI-Vision differs from the reranker’s ViSurf training distribution and ScreenSpot-Pro, with 2,786 Icon and 758 Text samples and about 2× larger relative element areas.Element sizes are similar in pixels, while the relative-area difference indicates lower-resolution screenshots on average.Source: §5.3 · p. 7, Table 7 · p. 7

5.4 Broader Benchmarks

• RankGround beats ZoomIn by 1.18–5.49 points on all five benchmarks and exceeds Direct by 2.36–4.91 points on four.The comparison uses UI-Venus-1.5-8B with the same grounder for Direct, ZoomIn, and RankGround within each benchmark.Source: §5.4 · p. 7, Table 8 · p. 7
• On ScreenSpot-V2, Direct reaches 96.31%, while RankGround stays within one point of Direct and remains above ZoomIn.The authors identify ScreenSpot-V2 as the exception to RankGround’s gains over Direct.Source: §5.4 · p. 7
• Gains on the other four benchmarks indicate that learned crop selection transfers beyond ScreenSpot-Pro and UI-Vision.Source: §5.4 · p. 7

5.5 Curriculum Analysis

• The curriculum comparison holds data, LoRA configuration, optimizer, and evaluation protocol fixed, with results averaged over three random seeds.Pointwise-only uses L1 throughout, whereas Listwise-only starts directly from L2 without containment pre-training.Source: §5.5 · p. 7
• The curriculum achieves the best Hit@1, Hit@3, Hit@5, and Text Hit@1, while Pointwise-only remains strongest on icons.The pointwise stage supports binary containment decisions for small icons, while listwise training helps distinguish semantically similar text-bearing crops.Source: §5.5 · p. 7, Table 9 · p. 7
• The curriculum reaches 90.00% validation AUROC and completes its listwise stage at step 260, 30 steps before Listwise-only.Source: §5.5 · p. 7, Table 9 · p. 7
• At 2B, Two-Stage improves overall accuracy by 0.28 points and Text accuracy by 0.62 points over Pointwise-only.Two-Stage gives the highest overall and Text accuracy at both model scales; gains are smaller at 8B.Source: §5.5 · p. 7
• In the wide-target example, GroundRanker selects a useful crop with a clear score margin even though the target crosses crop boundaries.Source: Figure 4 · p. 7

5.6 Failure Modes and Fallbacks

• Overlap Tiling covers 99.75% of ScreenSpot-Pro targets, but GroundRanker reaches 88.2% Hit@1; reranker errors are 19.9% for icons versus 5.5% for text.Coverage failures account for only 0.25% of data; most first-stage errors select the wrong candidate among valid crops.Source: §5.6 · p. 8
• Origin has a 41.9% error rate, nearly twice Windows Common’s 21.0%, amid similar chart icons and instructions with little local text.Table 11 lists the ten hardest ScreenSpot-Pro applications.Source: §5.6 · p. 8, Table 11 · p. 8
• Among incorrect selections, 54.6% have a Top-1 minus Top-2 margin below 0.05, while 7.5% exceed 0.2.Figure 5 shows errors concentrated near zero margin.Source: §5.6 · p. 8, Figure 5 · p. 8
• Blender failures include plausible competing crops with a 0.001 top-two score gap, while Origin’s similar chart icons receive scores below 0.42.Figure 6 depicts these two failure patterns.Source: §5.6 · p. 8, Figure 6 · p. 8
• Top-1 reaches 71.0% accuracy with 1.0 VLM call, versus 64.0% with 2.0 calls for Top-2 and 56.9% with 3.0 calls for Top-3.A confidence-triggered fallback reaches 68.4% with 1.47 calls on average; the authors retain Top-1 as the best measured accuracy and lowest-cost option.Source: §5.6 · p. 8

6 Limitations

• RankGround depends on one selected crop; its 99.75% Oracle coverage assumes targets are smaller than a crop, so wide controls lack a formal containment guarantee.Hit@1 is 88.2%, and Top-k experiments show coordinate-token confidence cannot reliably recover a wrong selection.Source: §6 · p. 8
• GroundRanker’s training data are 79% desktop and 21% web, while mobile-heavy, multilingual, and changing interfaces remain untested.Pure icons have a 19.9% selection error rate versus 5.5% for text.Source: §6 · p. 8
• The fixed 145 ms reranker pass is more visible at 2B than beside an 8B VLM, and RankGround is faster than repeated-call baselines, not Direct.The authors characterize RankGround as an accuracy–efficiency trade-off.Source: §6 · p. 8

• On ScreenSpot-Pro, the Direct–Oracle gap exceeds 18 points, with region selection accounting for most of that gap.Source: §7 · p. 8
• An Oracle selector using one VLM call beats ZoomIn and MVP by more than 7 points across the 8B backbones.Source: §7 · p. 8
• 5.5% over the best multi-crop baseline and 1.4× faster across backbones and scales, RankGround improves accuracy while reducing inference time.Source: §7 · p. 8