Source-linked AI summary
PreView: Enhancing GUI Grounding with Preprocessed Views and Cues
Wenjun Hou, Kaishuai Xu
TL;DR
GUI grounding must locate small controls and identify which control an instruction refers to. PreView prepares views and candidate-specific cues for a frozen grounder, improving average accuracy across five benchmarks and three models by 1.5–13.9 points.
Problem
GUI grounding must locate precise controls and identify the intended target, while a clearer image alone may leave ambiguity about which control an instruction refers to.
Source: §1 · pp. 1–2Method
PreView uses an auxiliary model to prepare center-guided views and candidate-specific visual and textual cues for a frozen grounding model.
Source: §1 · p. 2Results
Takeaways & Limitations
PreView’s gains require only one call to the frozen grounding model, while analyses identify target exclusion and misleading descriptions as failure modes.
Source: §6 · p. 9The evaluation uses desktop-trained models and offline single-step grounding, adds auxiliary inference cost and latency, and reports one run per configuration.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
GUI grounding requires locating small controls in cluttered screenshots and identifying which control a natural-language instruction refers to. Cropping and enlargement can expose local detail, but the resulting image may still leave ambiguity about the intended target. We propose PreView, an auxiliary preprocessor that improves the inputs to an existing grounding model without further training that model. PreView prepares two complementary types of input features: task-relevant views for localization and candidate-specific cues for target understanding. A trained lightweight auxiliary model predicts and refines a center, crops the screenshot into a local view, and enlarges that view for zoomed-in grounding. It also prepares a visual cue (i.e., a red-circle marker with a location hint) and a textual cue describing the candidate's identity or function. The frozen grounding model uses the prepared inputs to predict the final click. Across five benchmarks, PreView improves the average grounding accuracy of Qwen3-VL-2B, OS-Atlas-4B, and Qwen3-VL-4B without further training these grounding models. Extensive analyses examine the contributions of views and cues, center refinement, and visual input budgets.
1 Introduction
GUI grounding must locate precise controls and identify which one an instruction names. PreView addresses both with prepared views and cues, without modifying the downstream grounder.
1 Introduction
- Grounding requires both resolving a control’s precise position and determining whether it fulfills the instruction, including its role and relationship to nearby elements.A clearer image may resolve small controls but not which adjacent control is intended; identifying the control also does not guarantee an accurate click.Source: §1 · pp. 1–2
- Prior approaches use training, zooming, search, or semantic clarification, but tighter crops can discard context and added searches consume inference resources.PreView is motivated by preparing useful views and cues while limiting lost context and misleading descriptions.Source: §1 · p. 2
- PreView is an auxiliary preprocessing framework that improves GUI grounding without training or modifying the downstream grounding model.It prepares center-guided views for localization and candidate-specific visual and textual cues for target understanding.Source: §1 · p. 2
- Across five benchmarks and three grounding models, PreView improves reported average accuracy by 1.5–13.9 points.Reported gains include SSPro, OSW-G, MMB-GUI, and UI-V.Source: §1 · p. 2
2 Related Work
Related work improves GUI grounding through specialized training, architectural design, zooming and search, and semantic augmentation. PreView instead prepares views and cues before a single call to the retained grounding model.
2 Related Work
- PreView assigns center refinement and cue generation to an auxiliary model before one downstream grounding call, while retaining the grounder’s parameters and click-prediction interface.This contrasts with zooming and search approaches that add inference steps, which consume resources.Source: §2 · pp. 2–3
- Visual prompting and semantic augmentation connect interface regions with language or function; PreView prepares a task-relevant view with paired visual and textual cues.Candidate verification and center consistency control which cues reach the frozen model.Source: §2 · p. 3
- The five-benchmark suite isolates single-step grounding, separating input preparation from planning and multi-step recovery.This differs from interactive agent evaluations of task completion.Source: §2 · p. 3
3 Method
PREVIEW prepares views and candidate-specific cues for a frozen grounding model, using a shared auxiliary model to localize and verify candidates. At inference, the model refines a crop center, generates cues, routes inputs based on center consistency, and maps the final click back to the screenshot.
3.1 Training
- The auxiliary model shares parameters across center prediction and candidate verification, trained to predict target-box centers and describe matching candidates or reject mismatches.Training minimizes loss over target-response tokens while keeping the grounding model frozen.Source: §3.1 · p. 3
3.2 Inference
- At inference, the auxiliary model predicts a center from the full screenshot, refines it using a local view, and uses the refined center to set the crop.A third refinement is used only for analysis; the pipeline stops at the second center.Source: §3.2 · p. 4
- A red-circle marker anchors the visual cue, while a generated description conveys candidate identity, function, or relationships; rejected candidates receive no textual cue.The visual cue remains available when center consistency passes, even if the auxiliary model rejects the candidate.Source: §3.2 · p. 4
- Center-consistency routing compares initial and refined centers, selecting a marked view with available cues or the same unmarked view with the original instruction.The provided passages describe the displacement measure and routing alternatives but do not specify the threshold value.Source: §3.2 · p. 4
- The grounding model’s raw click is normalized within the input view and mapped to full-screenshot coordinates using that view’s boundaries.The model’s click prediction is distinct from the auxiliary model’s center predictions.Source: §3.2 · p. 4
4 Experiments
Experiments train the auxiliary model on annotated desktop-application screenshots and evaluate grounding across five benchmarks. They compare full-screenshot and PREVIEW inputs across model-size groups, with implementation details and a separate visual-budget ablation.
4.1 Datasets
- Training examples use 50,000 GroundCUA desktop screenshots, combining center-prediction, positive-description, and candidate-rejection examples.The mixture contains 50,000 center-prediction, 49,311 positive-description, and 49,284 rejection examples.Source: §4.1 · p. 4
4.2 Evaluation Metrics
- Grounding accuracy measures clicks inside target boxes, with OSW-G also scoring correct refusals and MMB-GUI reporting the mean of Basic and Advanced accuracy.Auxiliary evaluation separately measures direct center hits and whether the 12.5%-area view contains the target-box center.Source: §4.2 · pp. 4–5
4.3 Baselines
- Table 1 groups published results by model size and compares Full, PREVIEW, and their percentage-point difference across benchmark columns.The table caption identifies grounding accuracy, original and refined OSW-G instruction conditions, and five-benchmark averages based on OSW-GR.Source: §4.3 · p. 5, Table 1 · p. 5
4.4 Implementation Details
- The auxiliary model is jointly trained for two epochs in BF16, using LoRA in language layers, a trainable multimodal merger, and a frozen vision encoder.The visual-budget ablation varies crop area and upsampling on three benchmarks while holding refined centers and grounding settings fixed.Source: §4.4 · pp. 5–6
- At inference, grounding models use τ = 0.02 to select between marked views with available cues and the same unmarked view.The candidate-verification output is a description or REJECT; auxiliary inference allows 128 output tokens.Source: §4.4 · pp. 5–6
5 Results
PreView improves grounding across benchmark and model families through input preparation rather than further training the grounder. Results also show that views and cues help under different conditions, while refinement and cue reliability have practical limits.
5.1 Grounding Performance
- Average accuracy gains are 7.4, 13.9, and 1.5 percentage points for Qwen3-VL-2B, OS-Atlas-4B, and Qwen3-VL-4B, respectively.Qwen3-VL-2B with PreView matches Qwen3-VL-4B on SSPro at 59.3% and exceeds it on UI-V, 33.7% versus 25.3%; input preparation also partly bridges model-size gaps.Source: §5.1 · p. 6
- On SSPro, PreView gains 16.2, 25.4, and 4.4 percentage points over training-free baselines for Qwen3-VL-2B, OS-Atlas-4B, and Qwen3-VL-4B, respectively, with one grounding-model call.The auxiliary model handles localization refinement and cue generation; integration preserves the grounder’s input interface and decoding procedure, adding preprocessing and coordinate mapping.Source: §5.1 · p. 6, Table 3 · p. 6
5.2 Auxiliary Model Performance
- One refinement raises mean direct hit from 60.83% to 64.02%; the analysis-only second refinement reaches 63.93%, so inference stops after the first refinement.Refine 1 retains the target center in 80.14% of views despite directly hitting the target in only 53.83% of cases. Rejected candidates have lower weighted direct hit than accepted ones, 26.1% versus 66.5%, but can still benefit from visual cues.Source: §5.2 · pp. 6–7
5.3 View and Cue Ablation
- On SSPro, View raises Qwen3-VL-2B accuracy from 43.1% to 56.9%, while cues contribute more on UI-V, adding 7.7 points beyond View versus 2.5 on SSPro.Combined cues improve average accuracy beyond View for all three models, but their effects vary by task and model: Qwen3-VL-4B falls from 65.0% with View to 63.7% with both cue types on SSPro, yet rises from 35.9% to 40.6% on UI-V.Source: §5.3 · p. 7
5.4 Visual Budget Ablation
- At the same 25% pixel budget, enlarging a 12.5%-area crop beats a native 25%-area crop on all three benchmarks; doubling crop dimensions adds 6.5–10.8 points.At a fixed 50% visual budget, smaller crops perform better on SSPro and UI-V, while the wider crop improves OSW-GR by 0.71 points.Source: §5.4 · p. 7, Figure 4 · p. 7
5.5 Cue Reliability and Failure Analysis
- For 2B examples passing the consistency filter, cues inside the target correct 71.0% of View errors and disrupt 1.9% of correct predictions; outside it, rates are 3.2% and 57.8%.The analysis covers 8,288 target-present examples and groups them by whether the cue lies inside the annotated target; passing the consistency filter does not guarantee a correct cue location.Source: §5.5 · pp. 7–8
- Cue filtering cannot repair errors caused by cropping out the target or necessary context, and misleading cues can override a correct prediction.Examples include a crop that excludes target cell U42 and an accepted description that points to a neighboring notification icon instead of the date-and-time control.Source: §5.5 · p. 8
5.6 Transfer with Overlapping Training Data
- On SSPro, adding View raises GroundNext accuracy from 52.9% to 59.0%; on OSW-GR it rises from 67.7% to 68.3%.PreView uses 148,595 examples from a subset of GroundCUA, compared with 700,000 examples used for GroundNext supervised fine-tuning; the results indicate GroundNext can still benefit from task-relevant views after training on the same source.Source: §5.6 · p. 9
6 Conclusion
PREVIEW improves GUI grounding without updating the downstream model, combining center-guided views with visual and textual cues. Across five benchmarks and three grounding models, it improves average accuracy, with ablations showing complementary benefits from views and cues.
6 Conclusion
- Across five benchmarks and three grounding models, PREVIEW improves reported average accuracy by 1.5–13.9 points with one frozen-grounder call.Source: §6 · p. 9
- Center-guided views support localization, while visual and textual cues support target understanding.Source: §6 · p. 9
- Ablations show complementary benefits from views and cues, while error analysis identifies target exclusion and misleading descriptions as failure modes.Source: §6 · p. 9
- Future work should adapt view selection and cue use based on estimated target coverage and cue reliability.Source: §6 · p. 9
Limitations
The authors identify risks from cropping and incorrect cues, limits to generalization and interaction scope, added inference cost, and single-run variability.
Limitations
- Cropping may discard the target or necessary context, and incorrect cues may mislead the grounding model.Source: Limitations · p. 9
- Training on desktop applications limits generalization to mobile interfaces, while offline evaluation does not establish reliability in multi-step interaction.Source: Limitations · p. 9
- Auxiliary inference adds computational cost and latency, and single-run results may be influenced by initialization noise.Source: Limitations · p. 9
Ethical Considerations
The paper advises reviewing grounding outputs and using human confirmation for consequential operations. Its prompt templates specify candidate verification and how cues are presented to the grounder.
Ethical Considerations
- Incorrect grounding can trigger unintended GUI operations, including modifying or deleting information, so consequential operations warrant human confirmation.Source: Ethical Considerations · p. 9
A Prompt Templates
- The auxiliary prompts request normalized center coordinates and candidate verification with either a description or REJECT.Source: A Prompt Templates · p. 11
- For accepted candidates, the grounder receives a marked view with a fixed location hint and generated textual description.Source: A Prompt Templates · p. 11
- If verification rejects a candidate, the marker and location hint may remain; failed consistency filtering falls back to an unmarked view and original instruction.Source: A Prompt Templates · p. 11
B Evaluation Details
The evaluation details report benchmark sizes and compare crop-area and enlargement-scale settings across three grounding-evaluation sets.
B Evaluation Details
- OSW-G Original and Refined each contain 564 tasks and share screenshots; MMB-GUI has 1,787 Basic and 1,807 Advanced examples.Source: B Evaluation Details · p. 11
- The visual-budget evaluation compares 12.5%-area crops enlarged at linear scales 1 and 2, and evaluates 25%-area crops at scale 1.Source: B Evaluation Details · p. 11
- Table 7 summarizes sample counts for the evaluation benchmarks.Source: Table 7 · p. 12
- Each visual-budget configuration uses the full grounding-evaluation sets for OSW-GR, SSPro, and UI-V.Source: B Evaluation Details · p. 12
C Qualitative Case Studies
The case studies compare a frozen grounding model across full screenshots, preprocessed views, and views with cues. Their examples illustrate distinct outcomes, while Section 5.5 analyzes visibility, target interpretation, and information loss.
C Qualitative Case Studies
- Figure 6 compares Full, view-only, and view-plus-cue inputs to the same frozen Qwen3-VL-2B-Instruct model, holding the predicted center and view preprocessing constant.Both view conditions use a 12.5% crop and 2× enlargement; all four cases have accepted descriptions.Source: C Qualitative Case Studies · p. 12
- The selected examples illustrate distinct outcomes rather than estimating how frequently those outcomes occur.They show inputs before consistency filtering, and the detail panels in (a–c) use the same region.Source: C Qualitative Case Studies · p. 12
- Section 5.5 analyzes how the cases distinguish visibility, target interpretation, and information loss.Source: C Qualitative Case Studies · p. 12
D Detailed Benchmark Results
Tables 8–11 break down Full and PREVIEW accuracies for Qwen3-VL-2B and Qwen3-VL-4B across four benchmarks. The breakdowns use benchmark-specific groupings and definitions, and report results on the same evaluated samples as the main experiments.
D Detailed Benchmark Results
- Tables 8–11 report accuracy (%) and example counts for Full and PREVIEW on Qwen3-VL-2B and Qwen3-VL-4B, using the main-experiment configuration.Full is the reproduced baseline using the original screenshot; PREVIEW includes the 2% center-consistency filter and unmarked-view fallback, and these are breakdowns of the same evaluated samples.Source: D Detailed Benchmark Results · p. 12, Table 8 · p. 12, Table 9 · p. 12
- SSPro groups results by application domain and Text/Icon, while MMB-GUI groups by platform and Basic/Advanced difficulty.MMB-GUI overall accuracy is the mean of Basic and Advanced accuracies.Source: D Detailed Benchmark Results · p. 12, Table 9 · p. 12, Table 11 · p. 13
- UI-V reports Basic, Functional, and Spatial instruction categories, with Text/Icon as a separate partition of the same examples.Its overall accuracy is computed over examples.Source: D Detailed Benchmark Results · p. 12, Table 8 · p. 12
- OSW-G reports refined instructions only; its overall score covers all 564 examples, including 54 refusal tasks by target-presence annotations.The released GroundCUA category-matching code assigns only nine examples to its Refusal bucket; GUI task labels overlap, so category counts cannot be summed for the overall sample count.Source: D Detailed Benchmark Results · p. 12, Table 10 · p. 13