PaperlayerView exact revision on arXiv

Source-linked AI summary

VISTA: View-Consistent Self-Verified Training for GUI Grounding

Xinyu Qiu, Yunzhu Zhang, Heng Jia, Shuheng Shen, Changhua Meng, Linchao Zhu

arXiv:2606.14579v1cs.AI

TL;DR

GUI-grounding GRPO can produce uninformative all-failure or all-success groups when rollouts share one screenshot view. VISTA compares rollouts across target-preserving views and adds a self-verified cross-view anchor, improving accuracy across benchmarks, including ScreenSpot-Pro gains from 55.5/52.7/53.7 to 63.4/65.8/67.0 for Qwen3-VL 4B/8B/30B-A3B.

  • Problem

    GUI-grounding GRPO needs comparison groups that preserve target semantics while varying view geometry to provide informative relative advantages.

  • Method

    VISTA builds GRPO groups from multiple target-preserving views and adds a self-verified oracle anchor activated by maximum-reward model rollouts.

  • Results

    63.4/65.8/67.0 versus 55.5/52.7/53.7 on ScreenSpot-Pro for Qwen3-VL 4B/8B/30B-A3B, with improvements reported across benchmarks, scales, and backbones.

  • Takeaways & Limitations

    VISTA improves GUI-grounding accuracy and crop robustness across benchmarks, scales, and backbones.

  • Takeaways & Limitations

    VISTA requires conservative cropping schedules because aggressive cropping can increase reinforcement-learning update variance.

Abstract

from arXiv · show

When applying Group Relative Policy Optimization (GRPO) for GUI Grounding, rollouts are sampled from a single screenshot view; groups often become either all failures on difficult instances or all successes on easy ones, yielding no useful relative advantage. We propose VISTA (View-Consistent Self-Verified Training), a GRPO-based training framework that constructs each comparison group from multiple target-preserving views of the same GUI instance.Each view is generated by a crop that keeps the target element visible and remaps its box exactly, so model rollouts are compared across semantically equivalent but geometrically different inputs. To stabilize short coordinate generation without turning reinforcement learning into unconditional imitation, VISTA further adds a self-verified cross-view anchor: an oracle answer optimized with an advantage-weighted loss, excluded from the group baseline and activated only when the model has produced a maximum-reward rollout. Across five GUI-grounding benchmarks and multiple Qwen backbones, VISTA consistently improves grounding accuracy.On ScreenSpot-Pro, it raises Qwen3-VL 4B/8B/30B-A3B from 55.5/52.7/53.7 to 63.4/65.8/67.0. Robustness analyses further show higher worst-view accuracy and lower prediction flip rates.

1 Introduction

VISTA addresses reward degeneracy in GRPO-based GUI grounding by comparing rollouts across target-preserving views and conditionally using oracle coordinates for stabilization. It improves ScreenSpot-Pro accuracy across Qwen backbones and model scales.

  • Motivation: GUI grounding requires precise click localization because small, densely arranged, visually similar interface elements make slight spatial errors disruptive.A misplaced click may activate the wrong element and disrupt the subsequent workflow.
  • Problem: Single-view GRPO suffers all-fail degeneracy on difficult screenshots and all-success degeneracy on easy screenshots, collapsing the group-relative advantage.Sparse binary point-in-box rewards make homogeneous rollout outcomes uninformative.
  • Method: VISTA constructs each GRPO group from multiple target-preserving views of one GUI instance, preserving instruction and target semantics while varying screenshot geometry.The model therefore faces the same action requirement with different coordinate predictions, increasing informative cross-view reward variation.
  • Method: VISTA adds a self-verified cross-view anchor that supplies an oracle coordinate only after a maximum-reward model rollout, without altering the GRPO baseline or supervising all-fail groups.Model-generated rollouts remain the basis for group statistics, avoiding unconditional oracle injection.
  • Results: 55.5/52.7/53.7 to 63.4/65.8/67.0: VISTA improves Qwen3-VL ScreenSpot-Pro accuracy at the 4B/8B/30B-A3B scales.On Qwen3.5 initialized backbones, it further improves over standard GRPO by +2.0/+0.9/+1.2 points at the 4B/9B/35B-A3B scales, with 35B-A3B reaching 72.9.

2 Related Work

GUI grounding models map natural-language instructions to precise screen coordinates, evolving from supervised fine-tuning toward reinforcement learning with rule-based click rewards. GRPO can lose its learning signal when rollout groups are uniformly correct or incorrect, motivating oracle guidance.

  • GUI Grounding: GUI grounding models translate natural-language instructions into precise screen coordinates across mobile, web, and desktop interfaces, with early progress driven by large-scale supervised fine-tuning.The cited works established GUI grounding through dataset-based SFT across multiple interface types.
  • GUI Grounding: Reinforcement learning with rule-based click rewards has emerged to improve grounding accuracy beyond SFT, but existing approaches lose effectiveness as optimization progresses.Recent RL methods are described as promising while exhibiting diminishing training effectiveness during optimization.
  • Oracle Guidance in GRPO Training: GRPO degenerates when rollout groups are uniformly correct or incorrect, collapsing relative advantages to zero and eliminating the learning signal; recent math and reasoning work introduces oracle guidance.Oracle guidance is intended to ensure informative training updates.

3 Method

VISTA is a GRPO-based GUI-grounding framework that compares rollouts across multiple target-preserving views of one screenshot rather than repeated completions from a fixed rendering. It stabilizes this training with a self-verified cross-view oracle anchor that is conditionally activated and excluded from model-only group statistics.

  • Multi-view group construction: VISTA constructs each GRPO group from multiple target-preserving views of the same screenshot, making reward comparisons measure localization across renderings.This addresses fixed-view limitations and low reward variance from near-identical short coordinate outputs.
  • Multi-view group construction: Target-preserving crops fully contain the target, use constrained scale and contextual margins, and remap its box exactly into each resized crop.The resulting views preserve target geometry without truncating the target UI element.
  • Multi-view group construction: Each group contains G = Kn model-generated rollouts, with n = G/K completions sampled per view; advantages compare predictions across views of the same semantic target.When n > 1, groups retain within-view stochasticity; when K = G, they maximize cross-view diversity.
  • Self-verified cross-view anchor: The self-verified anchor activates only when at least one model rollout reaches maximum reward Rmax = 1, indicating successful grounding in a target-preserving view.If no rollout reaches Rmax, the oracle anchor is a zero-advantage no-op, while non-identical model rewards can still produce standard GRPO updates.
  • Self-verified cross-view anchor: The anchor selects an unsolved view when available, uses the remapped box center as its oracle coordinate sequence, and computes advantage against a model-only baseline.Excluding the oracle from group statistics prevents implicit supervision when the model has never succeeded.
  • Self-verified cross-view anchor: VISTA therefore differs from unconditional ground-truth injection or fixed-weight SFT mixing because its anchor is self-verified, baseline-excluded, and conditional.The design avoids turning reinforcement learning into unconditional supervised fine-tuning on unsolved or ambiguous groups.

4 Experiments

Experiments show that VISTA improves GUI grounding across Qwen scales, benchmarks, and initializations, with especially large gains on ScreenSpot-Pro. Ablations attribute the gains to combining view-consistent dynamic cropping with adaptive self-verified supervision, while inference-time multi-view aggregation adds further improvements.

  • Main results: VISTA raises Qwen3-VL average scores from 71.1/69.0/73.6 to 75.5/76.3/77.6 at 4B, 8B, and 30B-A3B, respectively.On ScreenSpot-Pro, gains are +7.9, +13.1, and +13.3 points at the three scales.
  • Main results: Adding MVP further raises average scores to 77.3, 77.8, and 79.4 at the 4B, 8B, and 30B-A3B scales.The corresponding ScreenSpot-Pro + MVP rows reach 71.6, 72.0, and 74.1, indicating complementarity between VISTA training and inference-time aggregation.
  • Training dynamics: Compared with standard GRPO, multi-view rollout reduces all-zero degeneracy and improves content reward, but destabilizes format reward; the self-verified anchor restores stable format reward while improving content reward and ScreenSpot-Pro accuracy.The diagnostic supports combining view diversification with model-verified anchoring rather than relying on rollout diversification alone.
  • Cross-backbone generalization: VISTA transfers beyond Qwen3-VL initialization, improving ScreenSpot-Pro at 4B, 9B, and 35B-A3B and OSWorld-G-R for each reported Qwen3.5 backbone.At 9B, it also improves ScreenSpot-V2.
  • Component ablations: Combining dynamic cropping and adaptive supervision gives the best 8B result, reaching 65.8 on ScreenSpot-Pro and 95.5 on ScreenSpot-V2.Dynamic crop alone raises standard GRPO from 63.4 to 64.3, adaptive supervision alone raises it to 64.8, and supervised crop augmentation raises SFT from 59.8 to 60.5.

5 Conclusion

VISTA is a view-consistent self-verified training framework for GUI grounding that combines target-preserving cropped views, exact coordinate remapping, and a self-verified oracle anchor. Across benchmarks, scales, and backbones, it improves accuracy and crop robustness, indicating that view-consistent group construction provides an effective training signal.

  • Framework: VISTA combines target-preserving cropped views, exact coordinate remapping, and a self-verified oracle anchor to align reinforcement learning with GUI grounding.The framework is designed around view consistency and self-verification.
  • Results: Across benchmarks, scales, and backbones, VISTA improves GUI-grounding accuracy.The conclusion reports consistent gains across these evaluation dimensions.
  • Robustness: VISTA also improves crop robustness, supporting view-consistent group construction as an effective GUI-grounding training signal.The robustness improvement is presented as evidence for the framework’s training approach.

Limitations

VISTA is limited to actionable GUI grounding settings with coordinate-verifiable supervision, and its view-consistent cropping introduces optimization sensitivity that requires conservative augmentation.

  • Supervision scope: VISTA’s anchor mechanism should be selective for mixed actionable and refusal-style datasets, using oracle anchors only for actionable samples with valid target boxes.Refusal or non-actionable samples can use a separate objective or be excluded from anchor activation.
  • Optimization sensitivity: Aggressive cropping can increase reinforcement-learning update variance when crop probability or the number of cropped views is large.The method reduces p_crop, limits crop views per group, and retains original pass-through views to stabilize training and reduce train–test mismatch.

A Appendix · A.1 Training Details

Appendix A.1 summarizes the key optimization and training hyperparameters used in the VISTA experiments in Table 6.

  • A.1 Training Details: Table 6 summarizes the key optimization and training hyperparameters for the VISTA experiments.The provided passage does not enumerate the individual settings.
  • A.1 Training Details: The appendix identifies the hyperparameter material as part of the VISTA training experiments.This scope is stated in the surrounding appendix text and table caption.
  • A.1 Training Details: The reported settings concern both optimization and training hyperparameters.The passage names these two categories without listing their values.
  • A.1 Training Details: The hyperparameter information is presented as a summarized table rather than detailed in the supplied paragraph.The paragraph directs readers to Table 6.
  • A.1 Training Details: The supplied material establishes the location and purpose of Table 6 but does not provide individual hyperparameter values.Only the table reference and caption are available in the input.
  • A.1 Training Details: The table is explicitly titled “Hyperparameter settings used in the VISTA training experiments.”This wording comes directly from the table caption.

A.2 Implementation Details of View-Consistent Group Rollout

The implementation combines deterministic view augmentation with target-preserving crop geometry and exact coordinate remapping. It samples eight crop views per augmented example while retaining pass-through full-screen examples and evaluates consistency across the original and cropped views.

  • View augmentation: 80% of training examples use dynamic crops, while 20% remain pass-through examples with repeated original views.The pass-through branch preserves standard full-screen layouts in the training distribution.
  • Target-preserving crops: Each crop starts at 0.9W × 0.9H and enlarges undersized dimensions to the target box, preventing target truncation at image boundaries.Crop windows are sampled from feasible top-left locations that fully contain the target box.
  • View-consistent rollout: Eight independent crop windows are sampled from feasible ranges, matching num_generations=8.The target box is remapped exactly into each crop coordinate frame.
  • Coordinate remapping: Supervision coordinates use pre-resize crop geometry before conversion to the model’s 0–1000 coordinate system, preserving view consistency after smart_resize.Remapped boxes are clipped to [0, 1] before conversion.
  • Consistency metrics: VCR measures the fraction of instances whose original view and eight crop views share the same correctness label.The implementation also defines prediction flip rate across crop views relative to original-view correctness.

A.3 Additional Ablation and Analysis

The ablations identify one deterministic oracle anchor, K=8 target-preserving views, and partial crop augmentation as effective defaults. Training dynamics further show that coordinate-changing crops provide a more stable and useful cross-view signal than naive multi-image resizing, while VISTA’s gains remain complementary to inference-time aggregation.

  • Number of Oracle Anchors: 65.8 ScreenSpot-Pro accuracy with one oracle anchor exceeds 65.0 with two and 64.0 with four, favoring a single supervised completion.Additional anchors shift training toward fixed supervised imitation rather than model-generated sequences.
  • Oracle-anchor Point Choice: The deterministic center-point anchor gives the best ScreenSpot-Pro accuracy, while uniformly sampling inside the ground-truth box reaches 95.2 on ScreenSpot-V2 and 64.8 on ScreenSpot-Pro.The random-point result shows that valid interior clicks can stabilize training, but the center avoids additional coordinate noise.
  • Number of Views: K=8 target-preserving views achieve the best ScreenSpot-Pro accuracy at 65.8 while matching VISTA’s GRPO group size.More views increase geometric diversity, but performance is not strictly monotonic because crops may remove context or create harder layouts.
  • pcrop: 64.6 ScreenSpot-Pro accuracy with pcrop=1.0 shows that cropping every example is suboptimal because removing full-screen pass-through cases overly dominates the RL update.pcrop controls the mixture between cropped groups and full-screen groups.
  • Image Processing Strategy: The final training gap reaches 9.55 points: GRPO remains stable at 61.54, whereas multi-image resize falls from 60.34 at step 300 to 51.99 at step 2100.Resizing changes scale and tokenization without changing the target’s normalized location, producing redundant coordinate outputs.
  • Detailed ScreenSpot-Pro Results: VISTA improves GRPO by 1.3, 2.4, and 1.1 average points for the 4B, 8B, and 30A3B settings, while MVP raises the corresponding averages to 71.8, 72.0, and 74.1.The results indicate that view-consistent training remains complementary to inference-time multi-view aggregation.
Loading 2606.14579v1…