PaperlayerView exact revision on arXiv

Source-linked AI summary

Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding

Jingyuan Huang, Zuming Huang, Yucheng Shi, Tianze Yang, Xiaoming Zhai, Wei Chu, Ninghao Liu

arXiv:2606.18101v2cs.AI

TL;DR

GUI grounding challenges VLMs to locate small interface elements precisely, while naive on-policy self-distillation can produce unreliable teacher signals from incorrect student prefixes. The paper combines correctness-aware gating with teacher-probability scaling, achieving 72.23 macro-average accuracy across six benchmarks and outperforming strong baselines.

  • Problem

    GUI grounding requires precise coordinate prediction for small, crowded targets, while existing post-training methods provide limited supervision for this coordinate-sensitive task.

  • Method

    Quality-aware self-distillation combines soft correctness-aware gating with teacher-probability scaling to calibrate coordinate-token supervision by signal reliability.

  • Results

    72.23 macro-average accuracy, 2.16 points above GUI-SD, with the method performing best across six GUI grounding benchmarks.

  • Takeaways & Limitations

    Spatial verifiability provides an effective basis for improving teacher-signal reliability in GUI grounding self-distillation.

  • Takeaways & Limitations

    The method relies on ground-truth bounding boxes during training and is designed specifically for coordinate-token prediction in GUI grounding.

Abstract

from arXiv · show

Graphical user interface (GUI) grounding requires vision-language models (VLMs) to identify small target elements in high-resolution screenshots and predict precise screen coordinates. On-policy self-distillation (OPSD) is a promising post-training approach for this coordinate-sensitive task, since it provides dense token-level teacher signals beyond hard coordinate labels. However, naive OPSD is not well suited to GUI grounding: OPSD evaluates the teacher on student-generated prefixes, the quality of coordinate-token teacher signals can degrade when the prefix has already deviated from the target coordinate, leading to unreliable teacher signal. To mitigate this, We propose quality-aware self-distillation for VLM-based GUI grounding, which improves coordinate-token teacher-signal quality through soft correctness-aware gating and teacher-probability scaling. The soft correctness-aware gate checks whether the teacher's current coordinate-token prediction can still be completed into the ground-truth box under the student-generated prefix. If not, the corresponding teacher signal is down-weighted. Teacher-probability scaling then uses the teacher's confidence as a lightweight factor to further calibrate the strength of the gated supervision. A key empirical finding is that neither component alone improves overall performance, whereas combining them consistently improves performance. This suggests that the two mechanisms play complementary roles: correctness-aware gating suppresses unreliable coordinate-token supervision, while teacher-probability scaling calibrates the strength of the remaining signals. Experiments across six GUI grounding benchmarks show that our method consistently improves the base model and outperforms strong baselines.

1 INTRODUCTION

GUI grounding requires precise coordinate prediction for small interface elements in complex, high-resolution screenshots, making teacher-signal quality critical for on-policy self-distillation. The proposed quality-aware self-distillation method calibrates coordinate-token supervision through soft correctness-aware gating and teacher-probability scaling.

  • Motivation: GUI grounding asks VLMs to identify an instructed interface element and output its screen coordinates from a screenshot, especially challenging in high-resolution, complex scenes.The task supports agents operating computers, mobile devices, and web applications.
  • Motivation: OPSD provides dense token-level teacher distributions along student-generated trajectories without requiring GRPO-style rollout volumes, but naive OPSD can produce unreliable signals for GUI grounding.Its effectiveness depends critically on teacher-signal quality because the teacher is queried on student-generated prefixes.
  • Method: The proposed method combines soft correctness-aware gating with teacher-probability scaling to calibrate coordinate-token supervision according to signal reliability.Correctness-aware gating leverages the spatial verifiability of coordinate predictions against ground-truth bounding boxes.
  • Evaluation: The work uses GUI grounding as a spatially verifiable setting to study teacher-signal reliability in OPSD and evaluates the method across six GUI grounding benchmarks.The experiments examine how unreliable teacher signals should be treated during training.

2 RELATED WORK

Prior GUI-grounding work uses supervised fine-tuning and on-policy self-distillation, but teacher signals can be unreliable on student-generated prefixes. Reliability-aware methods address this with weighting or privileged inputs, while this work exploits spatially verifiable coordinates and ground-truth boxes to calibrate teacher supervision.

  • GUI Grounding: GUI grounding localizes interface elements from screenshots and instructions, with supervised fine-tuning on annotated instruction-coordinate pairs used by several recent models.The passage names SeeClick, CogAgent, UGround, OS-Atlas, and RVLM as examples.
  • Self-Distillation and Teacher-Signal Reliability: OPSD trains students on self-generated trajectories while teachers provide token-level signals, but those signals are not uniformly reliable on student-generated prefixes.The reliability issue is specifically associated with conditioning the teacher on prefixes generated by the student.
  • Self-Distillation and Teacher-Signal Reliability: Privileged access to the full correct answer can make teacher signals overly sharp or near-deterministic, reducing the benefit of soft distillation.This limitation is reported for prior OPD/OPSD analyses.
  • Methods to Improve Teacher-Signal Reliability: Reliability-aware OPD/OPSD methods improve teacher signals through entropy- or perplexity-based weighting and privileged teacher inputs.Entropy-based methods adjust token-level update weights, while perplexity-based methods down-weight apparently unreliable guidance.
  • Methods to Improve Teacher-Signal Reliability: This work uses spatially verifiable coordinate predictions and the ground-truth box to softly down-weight and further scale unreliable coordinate-token teacher losses.The ground-truth box serves as a direct training-time reliability criterion before distillation.

3 METHODOLOGY

The method distills a privileged teacher on student-generated prefixes while selectively calibrating coordinate-token supervision. Prefix-aware compatibility gating down-weights potentially invalid predictions, and teacher-probability scaling further adjusts signal strength according to confidence.

  • On-policy teacher evaluation: The student samples an on-policy response, and student and privileged-teacher distributions are evaluated on the same generated prefix.The teacher distribution is stop-gradient and serves only as a teacher signal.
  • Soft correctness-aware gating: For each coordinate token, the compatibility check tests whether the teacher’s top prediction can still be completed into the ground-truth interval on its coordinate axis.The check uses the current student-generated axis-specific prefix and separately evaluates x- and y-coordinate intervals.
  • Soft correctness-aware gating: Compatible coordinate-token predictions receive gate value 1, while incompatible predictions receive α = 0.5 instead of being discarded.This soft gate preserves potentially useful teacher signals while reducing unreliable coordinate-token supervision.
  • Teacher-probability scaling: Teacher-probability scaling strengthens supervision for higher-confidence top-1 coordinate predictions and softens it for more uncertain predictions.The scaling applies to coordinate-token positions even when their predictions are spatially compatible with the target box.
  • Weighted distillation objective: Ordinary response tokens retain weight 1, compatible coordinate tokens receive λp_t, and incompatible coordinate tokens receive αλp_t.The main method sets α = 0.5 and λ = 3, and optimizes the resulting weights with a reverse-KL objective over response tokens while excluding prompt tokens.

4 EXPERIMENTS

Across six GUI grounding benchmarks, the method achieves the strongest overall performance, reaching 72.23 macro-average accuracy and outperforming GUI-SD by 2.16 points. Component analyses show that soft correctness-aware gating and teacher-probability scaling are complementary, with λ = 3 providing the best tradeoff.

  • Main results: 72.23 macro-average accuracy is the best result across six benchmarks, exceeding the GUI-SD baseline by 2.16 points.All experiments use Qwen3.5-9B as the backbone model and evaluate ScreenSpot-Pro, ScreenSpot-v2, UIEG, OSWorld-G, OSWorld-G-R, and MMG.
  • Main results: Relative to SFT and GRPO, the method improves macro-average accuracy by 4.14 and 6.37 points, respectively.The self-distillation objective uses teacher logits as soft supervision, providing richer training signals than hard target labels.
  • Component analysis: 70.43 macro-average accuracy from Vision-PV-Only falls to 69.97 with gating alone and 70.19 with teacher-probability scaling alone.Using either component independently can mismatch signal reliability: gating may suppress useful signals, while scaling may amplify unreliable ones.
  • Component analysis: 68.37 on SSP is achieved by the combined method, compared with 67.11 for gating alone and 67.24 for scaling alone.The combination first reduces erroneous teacher signals, then allows probability scaling to emphasize reliable signals with lower amplification risk.
  • Scaling coefficient: 72.23 macro-average accuracy is obtained at λ = 3, versus 71.20 at λ = 1, 71.32 at λ = 2, and 71.80 at λ = 4.The results indicate that λ = 3 best balances sufficient supervision on reliable coordinate tokens with robust grounding across benchmarks.

5 DISCUSSION AND LIMITATIONS

GUI grounding offers a spatially verifiable setting for assessing teacher-signal quality, motivating soft rather than binary treatment of unreliable signals. The approach is limited by its reliance on ground-truth boxes and coordinate-specific reliability criteria, suggesting broader extensions as future work.

  • Discussion: GUI grounding enables coordinate-token supervision to be assessed by whether a prediction can still complete into the ground-truth target region under a decoding prefix.This spatially checkable structure supports direct teacher-signal verification beyond indirect proxies.
  • Discussion: Incompatible teacher signals should be down-weighted rather than treated as entirely useless, because they may still encode local preferences or learned distributional structure.The discussion motivates a soft approach to teacher supervision instead of binary acceptance or rejection.
  • Limitations: The correctness-aware gate relies on ground-truth bounding boxes during training, limiting its direct applicability when spatial annotations are unavailable.This dependency is identified as a primary limitation of the method.
  • Limitations and Future Work: The current reliability criterion is designed for coordinate-token prediction in GUI grounding, so non-spatial tasks may require different verifiable teacher-signal assessments.Future work could examine transfer across model scales and other visually grounded agent tasks.

6 CONCLUSION

The paper presents quality-aware self-distillation for GUI grounding to improve the reliability of coordinate-token teacher signals in on-policy self-distillation. It combines soft correctness-aware gating with teacher-probability scaling to refine coordinate-token supervision.

  • 6 CONCLUSION: The method targets unreliable coordinate-token teacher signals in on-policy self-distillation for GUI grounding.Its goal is to improve teacher-signal reliability.
  • 6 CONCLUSION: Soft correctness-aware gating down-weights teacher predictions incompatible with the target region under the student-generated prefix.The gate assesses compatibility with the target region before weighting supervision.
  • 6 CONCLUSION: Teacher-probability scaling further refines the strength of coordinate-token supervision.It is applied after the gating mechanism to adjust supervision strength.
  • 6 CONCLUSION: The proposed method was evaluated on six GUI grounding benchmarks.The supplied conclusion passage reports experiments across six benchmarks but truncates the statement of their outcome.

A PROMPT TEMPLATES, PRIVILEGED VISUAL CUES, AND TRAINING TARGETS

The appendix specifies shared tool-use prompting, distinct teacher and student inputs, privileged visual cues for the teacher, and a structured coordinate-bearing training target. The student sees the original GUI task, while the teacher additionally receives a marked target region and textual hint.

  • A.1 SHARED SYSTEM PROMPT: The shared system prompt permits function use through a computer_use tool exposed inside <tools> tags.The tool description specifies computer interaction and supplies its function signature.
  • A.1 SHARED SYSTEM PROMPT: Only centered left-click actions are allowed, with coordinates specified as pixel offsets from the screen's left and top.The prompt says to center the cursor tip on targets, avoid edges unless asked, and disallow typing, key, scrolling, and dragging.
  • A.1 SHARED SYSTEM PROMPT: Function calls are returned as JSON containing a function name and arguments inside <tool_call> XML tags.This establishes the expected wrapper and argument structure for model outputs.
  • A.2 STUDENT AND TEACHER USER PROMPTS: The student user prompt contains the original GUI image and the original GUI instruction or query.The student template is shown as an image followed by the original task text.
  • A.2 STUDENT AND TEACHER USER PROMPTS: The teacher user prompt adds a privileged hint identifying where the answer is located.The example states that the answer is located within the green rectangle.
  • A.3 TRAINING TARGET FORMAT: The canonical training target is a structured computer_use left_click call containing the target coordinate [x, y].Main experiments omit additional natural-language reasoning or rationale from the target response.
  • A.3 TRAINING TARGET FORMAT: The teacher receives an augmented image with the target region marked by a green rectangle, whereas the student receives only the original GUI image.The teacher also receives a textual hint, while the student does not receive the visual privilege.

B CONSTRUCTION OF VISUAL PRIVILEGED INFORMATION · B.1 TEACHER VISUAL PRIVILEGED INFORMATION · B.2 VISUALIZATION OF PRIVILEGED AND NON-PRIVILEGED INPUTS

The method constructs target-aware visual privileged information for the teacher by masking the screenshot around the ground-truth box and adding visual and textual target cues. The student receives only the original screenshot and instruction, as illustrated by the teacher–student input comparison.

  • B.1 TEACHER VISUAL PRIVILEGED INFORMATION: The teacher receives target-aware visual privileged information constructed from the original GUI screenshot and ground-truth target bounding box.This follows GUI-SD (Zhang et al., 2026a).
  • B.1 TEACHER VISUAL PRIVILEGED INFORMATION: A Gaussian soft mask is built using each pixel’s Euclidean distance to the ground-truth bounding box.Pixels inside the box have distance 0.
  • B.1 TEACHER VISUAL PRIVILEGED INFORMATION: The mask preserves the target region fully while softly suppressing background regions farther from the target.The spatial decay is controlled by σ.
  • B.1 TEACHER VISUAL PRIVILEGED INFORMATION: The teacher input additionally includes a green rectangle around the ground-truth target region.This rectangle is one of the teacher-only visual cues used during training.
  • B.1 TEACHER VISUAL PRIVILEGED INFORMATION: A short textual hint tells the teacher that the answer is located inside the rectangle.The visual and textual cues are used only for the teacher during training.
  • B.2 VISUALIZATION OF PRIVILEGED AND NON-PRIVILEGED INPUTS: The student receives the original GUI screenshot and original instruction without any privileged visual cue.Thus, teacher and student inputs differ in whether target-specific visual information is provided.
  • B.2 VISUALIZATION OF PRIVILEGED AND NON-PRIVILEGED INPUTS: Figure 2 presents an example contrasting the teacher image containing visual privileged information with the student image remaining the original GUI screenshot.The figure visualizes the privileged and non-privileged inputs.

C TRAINING DETAILS

The self-distillation experiments use online self-distillation with an EMA teacher, whose training hyperparameters are provided in Table 6. The teacher starts from a legacy model and is updated after each successful optimizer step.

  • C TRAINING DETAILS: Online self-distillation uses an EMA teacher initialized from a legacy model and updated after each successful optimizer step.The training hyperparameters are reported in Table 6.

C.1 BASELINE-SPECIFIC TRAINING DETAILS

The experiments use a one-epoch final checkpoint by default, with GRPO evaluated at step 62 for a comparable budget; Naive-OPSD receives privileged ground-truth boxes, while GUI-SD follows its original hyperparameters.

  • Training budget: 62 optimization steps define one supervised-training epoch, while GRPO exceeds 400 steps per epoch.The main GRPO result therefore uses the checkpoint at step 62 for a similar training budget.
  • Method-specific details: Naive-OPSD receives the ground-truth text bounding box as privileged information during OPSD training.
  • Method-specific details: GUI-SD directly adopts the training hyperparameters reported in its original paper.
Loading 2606.18101v2…