PaperlayerView exact revision on arXiv

Source-linked AI summary

Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding

Yan Zhang, Daiqing Wu, Huawen Shen, Can Ma, Yu Zhou

arXiv:2605.00642v3cs.AIcs.CV

TL;DR

GUI grounding lacks efficient, dense supervision because GRPO requires multiple rollouts and can produce sparse signals on hard samples, while OPSD had not been tailored to this setting. GUI-SD introduces visually enriched privileged teacher guidance and entropy-guided token weighting, and it outperforms GRPO-based methods and naive OPSD across six benchmarks in accuracy and training efficiency.

  • Problem

    GRPO-based GUI grounding depends on expensive multiple rollouts and sparse signals on hard samples, while OPSD's applicability to GUI grounding remained unexplored.

  • Method

    GUI-SD combines a bounding box and Gaussian soft mask for constrained visual teacher guidance with entropy-guided distillation based on digit significance and teacher confidence.

  • Results

    GUI-SD substantially outperforms GRPO-based methods and naive OPSD across six GUI grounding benchmarks in accuracy and training efficiency, with approximately 4× faster training.

  • Takeaways & Limitations

    GUI-SD establishes on-policy self-distillation as a promising paradigm for GUI grounding by delivering targeted token-level supervision from a single rollout.

Abstract

from arXiv · show

Graphical User Interface (GUI) grounding maps natural language instructions to the visual coordinates of target elements and serves as a core capability for autonomous GUI agents. Recent reinforcement learning methods (e.g., GRPO) have achieved strong performance, but they rely on expensive multiple rollouts and suffer from sparse signals on hard samples. These limitations make on-policy self-distillation (OPSD), which provides dense token-level supervision from a single rollout, a promising alternative. However, its applicability to GUI grounding remains unexplored. In this paper, we present GUI-SD, the first OPSD framework tailored for GUI grounding. First, it constructs a visually enriched privileged context for the teacher using a target bounding box and a Gaussian soft mask, providing informative guidance without leaking exact coordinates. Second, it employs entropy-guided distillation, which adaptively weights tokens based on digit significance and teacher confidence, concentrating optimization on the most impactful and reliable positions. Extensive experiments on six representative GUI grounding benchmarks show that GUI-SD consistently outperforms GRPO-based methods and naive OPSD in both accuracy and training efficiency. Code and training data are available at https://zhangyan-ucas.github.io/GUI-SD/.

1 Introduction

GUI grounding needs dense, reliable supervision because GRPO uses costly multiple rollouts and can provide zero reward on hard samples. GUI-SD addresses this gap with visually enriched teacher guidance and entropy-guided distillation, outperforming GRPO-based methods and naive OPSD across six benchmarks.

  • 1 Introduction: GRPO requires expensive multiple rollouts and suffers from sparse or zero reward signals on hard samples, motivating denser supervision from fewer interactions.OPSD provides token-level supervision from a single rollout by using asymmetric teacher and student contexts.
  • 1 Introduction: Naive OPSD can collapse toward supervised fine-tuning when coordinate-conditioned teacher distributions become nearly one-hot, while uniform optimization is indiscriminate.This collapse removes the soft-label information that motivates distillation.
  • 1 Introduction: GUI-SD constructs a teacher context with a target bounding box, Gaussian soft mask, and instructional hint that guides localization without exposing exact coordinates.The soft mask gradually fades surrounding regions, supplying informative but constrained visual guidance.
  • 1 Introduction: Entropy-guided distillation prioritizes higher-order coordinate digits and amplifies supervision from confident teacher predictions instead of weighting all tokens uniformly.This targets the coordinate tokens most consequential for grounding accuracy.
  • 1 Introduction: GUI-SD substantially outperforms GRPO-based methods and naive OPSD in accuracy and training efficiency across six representative GUI grounding benchmarks.The benchmarks are ScreenSpot-v2, ScreenSpot-Pro, UI-Vision, MMBench GUI L2, OSWorld-G, and OSWorld-G-Refine.
  • 1 Introduction: The paper presents GUI-SD as the first exploration of OPSD for GUI grounding and positions it as an alternative to rollout-heavy GRPO training.The contribution is framed around dense, reliable token-level supervision for precise coordinate generation.

2 Preliminary

On-policy distillation replaces sparse sequence-level reinforcement-learning feedback with token-level supervision along student-generated trajectories. OPSD removes the need for a separate teacher by using the same model under asymmetric contexts, granting privileged information only to the teacher.

  • 2 Preliminary: OPSD uses the same model as teacher and student, with the teacher receiving privileged information unavailable to the student.Examples of privileged information include ground-truth answers and verified reasoning traces.
  • 2 Preliminary: The student generates an on-policy trajectory from the input, while the teacher produces step-wise target distributions conditioned on the input and privileged context.Training aligns the teacher and student distributions at each decoding step.
  • 2 Preliminary: OPSD minimizes per-token divergence between student and teacher distributions, providing denser feedback than sparse sequence-level rewards.The trajectory prefix y<t and current token yt define each decoding step, while |y| is the total trajectory length.

3 Empirical Analysis of OPSD for GUI Grounding

The analysis identifies two failures in naive OPSD for GUI grounding: textual privileged information collapses teacher supervision toward hard labels, while token reliability varies by coordinate position. These findings motivate visually enriched and position-aware distillation.

  • 3.1 Distillation-to-SFT Collapse at Sample-level: Naive OPSD’s textual coordinate privilege produces near-zero-entropy teacher supervision, making distillation behave like hard-label SFT.The reported teacher signal has average entropy 0.17 and average top-1 probability 0.82.
  • Implications: GUI-SD addresses these failures with visually enriched teacher context and adaptive token weighting based on positional importance and teacher confidence.Figure 3 depicts the privileged context and weighted reverse-KL objective.
  • 3.2 Indiscriminate Optimization at Token-level: Teacher supervision is stronger at higher-order coordinate digits and becomes less certain toward units digits.Figure 2 compares entropy and ground-truth probability across hundreds, tens, and units positions on incorrectly predicted tokens.
  • 3.2 Indiscriminate Optimization at Token-level: Uniform reverse-KL distillation ignores position-dependent reliability and can amplify uncertain preferences on lower-order digits.The analysis names this failure mode Indiscriminate Optimization.

4 Method

GUI-SD replaces textual coordinate privilege with a visually enriched teacher context and replaces uniform reverse-KL with entropy-guided, position-aware weighting. Together, these components preserve informative supervision while emphasizing consequential and reliable coordinate tokens.

  • 4.1 Visual Privileged Guidance: GUI-SD gives the teacher a target bounding box, Gaussian soft mask, and hint prompt while leaving the student with the original context.The privileged context guides the teacher without directly exposing the exact coordinate.
  • 4.1 Visual Privileged Guidance: The Gaussian soft mask preserves the target region and smoothly attenuates surrounding pixels to improve localization in dense, high-resolution scenes.Its scale follows target size with a minimum floor to avoid over-masking small objects.
  • 4.2 Entropy-Guided Optimization: GUI-SD weights each reverse-KL term by positional significance and teacher reliability rather than treating all coordinate tokens uniformly.The combined weighting is designed to prioritize high-value digits while downweighting uncertain supervision.
  • 4.2 Entropy-Guided Optimization: Positional credit assignment gives exponentially decaying weights from more significant coordinate digits to less significant ones because higher-order errors cause larger spatial deviations.The scheme maps digit positions through k_t and sets non-numeric-token positional weights to 1.
  • 4.2 Entropy-Guided Optimization: Entropy-gated supervision strengthens low-entropy teacher predictions and automatically downweights uncertain tokens.The teacher’s per-token entropy controls the supervision factor.

5 Experiment

Experiments across six GUI grounding benchmarks show that GUI-SD improves accuracy over GRPO-based and naive OPSD methods while training substantially faster. Ablations attribute these gains to visually enriched teacher context and entropy-guided optimization.

  • Comparisons with Baselines: GUI-SD achieves the highest average accuracy across six benchmarks and trains approximately 4× faster per epoch than GRPO-based methods.The comparison uses training time measured per epoch and attributes the efficiency gain to single-rollout supervision.
  • Comparisons with SOTA Methods: GUI-SD reaches 60.7% on ScreenSpot-Pro and 70.9% on OSWorld-G-Refine, surpassing cited state-of-the-art methods without test-time scaling or external large models.It surpasses Propose-then-Critic [36] on ScreenSpot-Pro and ZwZ [38] on OSWorld-G-Refine.
  • Effectiveness of Teacher Visual Context: On ScreenSpot-Pro, visually enriched teacher context improves student accuracy from 53.0% with identical inputs to 55.6% with naive OPSD, while GUI-SD’s context provides stronger guidance.The ablation evaluates student accuracy alongside teacher signal quality under varying privileged-context designs.
  • Effectiveness of Entropy-guided Optimization: Combining positional credit assignment and entropy-gated supervision yields 60.7% ScreenSpot-Pro accuracy, 79.7% hundreds-digit accuracy, and 21.1% hard-subset accuracy.The hard subset contains samples that the base model fails on across all eight rollouts.
  • Training Dynamics: GUI-SD maintains higher hundreds-digit accuracy than GRPO-Gaussian and the ablated variant during within-epoch training dynamics.The paper attributes this pattern to positional credit assignment and entropy-gated supervision.

6 Related Work

Related work spans reinforcement-learning approaches for GUI grounding and on-policy distillation methods that provide token-level supervision. OPSD removes the need for a separate teacher by using asymmetric contexts for the same model.

  • On-Policy Distillation: OPD distills token-level teacher distributions along student trajectories, whereas OPSD uses the same model as teacher and student under asymmetric contexts.Privileged information may include reference solutions, verifier signals, or environment feedback.
  • Reinforcement Learning for GUI Grounding: GRPO-type reinforcement learning enables exploration under verifiable feedback and has been extended to GUI grounding, including binary and denser reward designs.GUI grounding methods include binary bounding-box rewards and distance- or Gaussian-based dense rewards.

7 Conclusion and Limitations

The paper concludes that GUI-SD is an effective OPSD framework for GUI grounding, combining visually grounded teacher guidance with entropy-guided optimization. Its current scope is limited to Qwen3-VL, with larger models, other model families, and long-horizon tasks left for future work.

  • Conclusion: GUI-SD is presented as the first OPSD exploration for GUI grounding, outperforming GRPO-based methods and naive OPSD while achieving approximately 4× faster training.The conclusion frames targeted token-level supervision as the central contribution.
  • Limitations: The study has not explored scaling GUI-SD to larger models or model families beyond Qwen3-VL.This is the stated model-scope limitation.
  • Future Work: Extending GUI-SD to long-horizon GUI-agent tasks remains future work because multi-step planning and sequential interaction introduce additional challenges.The paper identifies this as a prospective direction rather than an evaluated setting.

A Evaluation Benchmarks

The evaluation spans six GUI grounding benchmarks covering multiple platforms, application domains, instruction capabilities, and operating systems. These datasets collectively test generalization from element localization to diverse real-world GUI interactions.

  • ScreenSpot-v2: ScreenSpot-v2 evaluates general-purpose grounding across mobile, desktop, and web screenshots paired with instructions and target bounding boxes.It was designed to address annotation ambiguities in the original ScreenSpot benchmark.
  • ScreenSpot-Pro: ScreenSpot-Pro targets professional high-resolution software environments across real-world applications and industries.Its setting emphasizes precise grounding in development, creative, engineering, scientific, and office software.
  • UI-Vision: UI-Vision measures desktop element grounding across 83 open-source applications and more than 8,200 query-label pairs.Its cross-application diversity tests generalization across varied software.
  • OSWorld-G and OSWorld-G-Refine: OSWorld-G evaluates Linux GUI grounding across 564 samples, 32 element types, and four task categories including precise operation.OSWorld-G-Refine is included as a related benchmark variant.
  • MMBench GUI L2: MMBench GUI L2 evaluates element grounding as the second level of a hierarchical benchmark spanning Windows, macOS, Linux, iOS, Android, and Web.The benchmark separates content understanding, element grounding, task automation, and task collaboration levels.

B Training Details.

Training uses curated and diversified grounding data, while evaluation spans six benchmarks with detailed per-category analyses.

  • Training Data: Training data is sourced from ScaleCUA’s grounding subset and curated to remove noisy labels and diversify instructions.UI-Venus1.5-8B filters samples whose predictions disagree with original annotations, and Qwen3-VL-8B rewrites instructions into paraphrases.
  • Evaluation: GUI-SD consistently outperforms GRPO baselines across most reported sub-categories of the six evaluation benchmarks.The breakdown covers professional domains, platforms, grounding tasks, operating systems, difficulty levels, and OSWorld task types.

C.2 Ablation on Visual Privilege Design

Visual privilege designs substantially improve over naive OPSD, with adaptive zoom performing best among the compared variants.

  • C.2 Ablation on Visual Privilege Design: Table 12 compares Naive OPSD with standard zoom and adaptive zoom designs for visual privilege.The table reports ScreenSpot-Pro accuracy and performance differences relative to Naive OPSD.
  • C.2 Ablation on Visual Privilege Design: +4.5 and +4.7 performance gains over Naive OPSD are achieved by standard and adaptive visual privilege variants, respectively.Both variants deliver ground-truth information through the visual channel; adaptive zoom further adapts masking to varying target sizes.

C.3 The Self-teacher Improves during Training

GUI-SD improves the base model across Qwen3-VL scales, with larger gains reported at larger model sizes.

  • C.3 The Self-teacher Improves during Training: Table 13 evaluates current-policy, frozen-reference, and EMA teacher-update strategies.The current policy qθ used directly as teacher achieves 59.4%, while simultaneous updates weaken distillation by minimizing teacher–student divergence.
  • C.3 The Self-teacher Improves during Training: +3.7, +4.6, and +7.1 ScreenSpot-Pro gains are achieved at the 2B, 4B, and 8B Qwen3-VL scales, respectively.Similar improvements are observed across the other benchmarks, indicating effectiveness across model capacities.
  • C.3 The Self-teacher Improves during Training: The scale comparison covers six grounding benchmarks from Qwen3-VL 2B through 8B.Reported benchmarks include ScreenSpot-Pro, ScreenSpot-v2, UI-Vision, OSWorld-G, OSWorld-G-Refine, and MMBench GUI L2.
Loading 2605.00642v3…