PaperlayerView exact revision on arXiv

Source-linked AI summary

GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning

Qingsong Xie, Jiahao Tian, Jiacheng Tang, Zhifeng Li, Madalina Ciobanu, Navan Preet Singh, Qingqing Mao, Ritankar Das

arXiv:2610.19100v1cs.AIcs.CLcs.CV

TL;DR

High-quality GUI grounding data remain scarce across diverse environments. The paper introduces ScreenRef and trains GroundGUI-8B using SFT followed by D2RL. Across five benchmarks, it substantially outperforms models trained on existing open-source datasets and is competitive with or superior to leading models trained on closed-source data.

  • Problem

    High-quality grounding datasets with broad coverage are scarce, despite GUI agents needing to map language instructions to precise on-screen coordinates across diverse interfaces.

    Source: §1 · p. 1

  • Method

    The authors train GroundGUI-8B on ScreenRef using SFT followed by D2RL, which applies OPSD before RLOO.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    Across five benchmarks, GroundGUI substantially outperforms models trained on existing open-source datasets and is competitive with or superior to leading models trained on closed-source data.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The authors release ScreenRef, code, and the model to facilitate future research.

    Source: §1 · p. 2

    ScreenRef has not undergone a systematic human privacy audit, and its license restricts use to gated, non-commercial research.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Constructing robust GUI agents relies on multimodal visual grounding: the ability to precisely map natural language intents to actionable on-screen components. Despite abundant desktop, web, and mobile benchmarks, high-fidelity resources tailored for diverse environments are surprisingly scarce. To bridge this gap, we introduce the Screen Referring Grounding Dataset (ScreenRef), a comprehensive GUI grounding dataset consisting of GUI screenshots paired with natural language referring expressions and box or point annotations. Spanning desktop, web, and mobile domains, ScreenRef comprises 586K screenshots and over 633K referring expression instances, covering tasks such as text recognition, visual matching, spatial understanding, and refusal scenarios. To boost the model's grounding performance, we propose the Distillation-to-Reinforcement Learning (D2RL) method to address the limited generalization of supervised fine-tuning and the sparse reward problem in reinforcement learning. Utilizing ScreenRef, we train Qwen3-VL-8B-Instruct via sequential SFT and D2RL training, yielding GroundGUI-8B. GroundGUI-8B achieves 95.2% on ScreenSpot-V2, 63.9% on ScreenSpot-Pro, 65.4% on OSWorld-G, 86.8% on MMBench-GUI L2, and 38.3% on UI-Vision. Among 8B-scale models, it produces substantial gains over models trained on open-source datasets, with performance comparable or even superior to models trained on closed-source data. Code, models, and datasets are available at https://github.com/PrentisAI/GroundGUI.

1 INTRODUCTION

Robust GUI grounding is challenging across dense, diverse interfaces, and high-quality datasets with broad coverage are scarce. The paper introduces ScreenRef and D2RL, then reports strong results for GroundGUI-8B.

Motivation

  • Proprietary large-scale data is unavailable to the community, while released datasets have limited diversity or accuracy and their models are suboptimal.The authors identify high-resolution, dense layouts and specialized icon semantics as grounding challenges.Source: §1 · p. 1

Core contributions

  • ScreenRef comprises 586K screenshots and 633K referring expressions across desktop, web, and mobile, covering text recognition, visual matching, spatial understanding, and refusal tasks.Multi-stage filtering and verification produce data intended for robust cross-platform grounding.Source: §1 · p. 1
  • GroundGUI-8B scores 95.2% on ScreenSpot-V2, 63.9% on ScreenSpot-Pro, 65.4% on OSWorld-G, 86.8% on MMBench-GUI L2, and 38.3% on UI-Vision.The paper reports that it remarkably outperforms models trained on open-source datasets and surpasses Holo2-8B and Step-GUI-8B, both trained on closed-source data.Source: §1 · p. 2
  • D2RL places on-policy self-distillation before RLOO, using token-level supervision to address sparse rewards on hard samples and improve generalization and localization accuracy.The teacher sees the ground-truth bounding box during OPSD; the student sees only the original screenshot and instruction.Source: §1 · p. 2
  • GroundGUI-8B achieves state-of-the-art or competitive performance among models with 8B parameters or fewer, surpassing several 32B-scale models.The cited passage names ScaleCUA-32B, OpenCUA-32B, and GTA1-32B as examples.Source: §1 · p. 2

2 RELATED WORKS

Prior GUI agents use supervised fine-tuning and reinforcement learning, while existing grounding datasets leave gaps in coverage and annotation quality. The paper positions ScreenRef against these limitations.

GUI Agent

  • Prior SFT approaches are data-inefficient in complex desktop and web settings, while RL methods still face sparse rewards when all rollouts fail on difficult samples.The related-work discussion also notes that SFT can overfit to specific domains.Source: §2 · p. 2

GUI Grounding Datasets

  • Existing datasets have distinct coverage or quality limits: mobile layouts offer less visual variation, web collections favor text-bearing elements, and desktop resources face curation or annotation challenges.The passage cites inconsistent accessibility APIs, simplified synthetic screens, and annotation inaccuracies or sparse diversity among recent efforts.Source: §2 · p. 2
  • Table 1 compares existing GUI grounding datasets, but their instruction counts follow each dataset’s own reporting unit and are not directly comparable.Source: Table 1 · p. 3

3 SCREENREF DATASET

ScreenRef combines screenshot-based collection with tutorial-video-derived samples, using filtering and verification to build grounding data across several platforms and task types.

3.1 SCREENSHOT CRAWLING

  • Desktop accessibility trees, web DOM, and Android XML provide element metadata; boxes with IoU below 0.5 are removed before instructions are generated and verified.The pipeline uses OmniParser V2 to filter noisy boxes, then models to generate instructions and check answers against the image and box crop.Source: §3.1 · pp. 3–4

3.2 TUTORIAL VIDEO CRAWLING

  • Tutorial videos yield instruction-observation-action-coordinate samples through keyframe extraction, joint instruction and action annotation, and post-hoc visual grounding.Candidate frames are ranked using transcript action verbs and detected cursor coordinates, then filtered through confidence-based grounding and verification; infeasible queries are also verified as refusal samples.Source: §3.2 · p. 4

3.3 DATASET STATISTICS

  • ScreenRef comprises 585,981 screenshots and 633,369 instructions, including 284,531 grounding samples from pathway 1 and the remaining samples from pathway 2.It features multiple platforms and a wider resolution range than existing datasets. ScreenRef and JEDI are the only two datasets that include refusal samples.Source: §3.3 · p. 4

4 DISTILLATION-TO-REINFORCEMENT LEARNING

D2RL prepares an SFT grounding policy with on-policy self-distillation before optimizing click success with reinforcement learning. This pairing supplies dense guidance on the model’s own trajectories before task-level reward is used.

Stage I: On-Policy Self-Distillation

  • D2RL applies on-policy self-distillation to prepare the SFT policy before reinforcement learning optimizes grounding success.The preparation targets a stronger starting policy for RL, which can receive little signal when every rollout fails on a hard sample.Source: §4 · pp. 4–5
  • A frozen SFT teacher sees the target box and localization hint, while the student sees only the original screenshot and instruction.Both score the same student-generated prefix, providing dense token-level guidance along the student’s rollout; the teacher is not used at inference.Source: §4 · p. 5
  • OPSD weights coordinate-digit tokens by digit significance, emphasizing higher-order digits because their errors cause larger spatial deviations.The weighting is β multiplied by digit significance for coordinate digits and 1 otherwise.Source: §4 · p. 5

Stage II: REINFORCE Leave-One-Out

  • In Stage II, RLOO optimizes grounding under the original screenshot using a parsed click point’s in-box status as reward, while unparseable responses receive zero.The format check is a gate, not an additive reward term.Source: §4 · p. 5, Figure 2 · p. 5
  • RLOO compares each rollout’s reward with the average reward of the other responses in its group, providing a leave-one-out baseline without a separate critic.OPSD can guide token distributions even for failed rollouts with identical click rewards; RLOO then uses task-level success to reweight responses.Source: §4 · pp. 5–6

5 EXPERIMENTAL RESULTS

GroundGUI-8B gains over its Qwen3-VL-8B base across five benchmarks, and the ablation study favors D2RL over the tested alternatives. ScreenRef-only SFT also outperforms the compared grounding datasets on the evaluated benchmarks.

5.1 IMPLEMENTATION DETAILS

  • Post-training uses OPSD followed by RLOO on 12K GroundCUA samples, excluding screenshots identical to those in UI-Vision.Training runs for two epochs, one per stage, on eight B300 GPUs; both stages use AdamW with a 1 × 10−6 learning rate, cosine schedule, and batch size 42, without a KL penalty.Source: §5.1 · p. 6

5.2 EVALUATION BENCHMARKS

  • The five benchmarks assess grounding across general-purpose interfaces, complex instructions, professional software, realistic computer-use scenarios, and semantic or spatial instructions.ScreenSpot-V2 covers web, desktop, and mobile; ScreenSpot-Pro emphasizes small targets in dense, high-resolution software, while OSWorld-G tests layout, operation semantics, and fine-grained interaction reasoning.Source: §5.2 · pp. 6–7

5.3 COMPARISON WITH OTHER MODELS

  • GroundGUI-8B gains 3.5 points on ScreenSpot-V2, 5.5 on MMBench-GUI L2, 14.0 on ScreenSpot-Pro, 10.6 on OSWorld-G, and 20.8 on UI-Vision over Qwen3-VL-8B.It matches or outperforms all listed open-source 7B-scale baselines across the evaluated benchmarks.Source: Table 3 · p. 7, Table 4 · p. 7, §5.3 · pp. 7–8

5.5 ABLATION STUDY

  • D2RL beats SFT-only, SFT+OPSD, and SFT+RLOO across all benchmarks, and exceeds RLOO† by 0.8 points.The variants share an SFT checkpoint and post-training data; the authors attribute D2RL’s advantage to OPSD’s stronger initial policy, not extra data exposure.Source: Table 8 · p. 9, §5.5 · p. 9

5.4 COMPARISON WITH OTHER DATASETS

  • On four benchmarks, ScreenRef-only SFT gains 3.5 points over ScaleCUA and 2.7 points over GroundCUA.UI-Vision is omitted from this comparison because GroundCUA contains many images from that benchmark.Source: §5.4 · pp. 8–9, Table 7 · p. 9

6 CONCLUSION

The paper presents ScreenRef and GroundGUI-8B, trained with SFT followed by D2RL, and reports strong results across five benchmarks. It also describes the dataset and artifact releases, AI-assisted work, and reproducibility resources.

6 CONCLUSION

  • ScreenRef contains 586K screenshots and 633K referring expressions across desktop, web, and mobile, and GroundGUI-8B is trained with SFT followed by D2RL.The paper reports that GroundGUI performs substantially better than models trained on existing open-source datasets and is competitive with or superior to leading models trained on closed-source data across five benchmarks.Source: §6 · p. 9
  • Generative AI models helped generate and clean dataset content and polish the manuscript; the authors say they did not use them to design or evaluate methodology or interpret results.The authors report that AI-generated data passed automated verification and AI-assisted text was reviewed by the authors.Source: AI USE STATEMENT · pp. 9–10
  • The authors have not yet conducted a systematic human privacy audit of screenshots that may contain personal or account information.ScreenRef is released under a gated, non-commercial research-only license, while code and model checkpoints use the Apache 2.0 license.Source: ETHICS STATEMENT · p. 10
  • The paper describes dataset-construction and D2RL procedures, training settings, benchmarks, and dataset statistics as reproducibility resources.Prompts, filtering rules, per-stage filtering statistics, and the full training configuration are to be added to the code repository.Source: REPRODUCIBILITY STATEMENT · p. 10

A ADDITIONAL SCREENREF EXAMPLES

The appendix presents qualitative ScreenRef examples across platforms and describes the format of the annotated answers.

A ADDITIONAL SCREENREF EXAMPLES

B ADDITIONAL DETAILS FOR DATASET COMPARISON

The appendix explains how prior dataset statistics and refusal samples are treated in the comparison, and includes a platform-by-capability distribution table and web examples.

B ADDITIONAL DETAILS FOR DATASET COMPARISON

  • Image and resolution counts align with Feizi et al. (2026)’s GroundCUA comparison when available, and otherwise use figures reported in each dataset’s paper.The counts may differ from datasets’ stated totals, possibly because of different filtering; instruction counts are reported as stated, not derived from element counts.Source: B ADDITIONAL DETAILS FOR DATASET… · p. 15
  • Refusal counts grounding samples whose target is absent from the screenshot, not task-level infeasible actions; SCREENREF figures use the final dataset.Every prior dataset in the comparison lacks such samples except JEDI, which pairs instructions with unrelated screenshots.Source: B ADDITIONAL DETAILS FOR DATASET… · p. 15
  • Figure 4 provides web-environment examples of the four ScreenRef task types.Source: Figure 4 · p. 16
  • Table 9 presents the distribution of grounding capabilities across platforms.Source: Table 9 · p. 16

C ADDITIONAL SCREENREF STATISTICS

ScreenRef’s grounding instances are distributed unevenly across capabilities and platforms: text recognition is largest overall, while mobile is nearly balanced between text recognition and spatial understanding.

C ADDITIONAL SCREENREF STATISTICS

D ELEMENT METADATA FROM LIVE SCREENSHOT CRAWLING

Live screenshot crawling records native-interface metadata for elements, then filters unreliable screenshots and boxes before generating grounding instructions.

D ELEMENT METADATA FROM LIVE SCREENSHOT CRAWLING

  • Each element record contains a screenshot-coordinate bounding box, visible or accessibility text, and a functional category when available.Sources are the DOM on web, accessibility tree on desktop, and XML layout on Android; every record has non-empty text, but some Android categories may be absent.Source: D ELEMENT METADATA FROM LIVE… · p. 17
  • Metadata covers interface-exposed elements with text or an accessibility name, not every visual object on screen.Source: D ELEMENT METADATA FROM LIVE… · p. 17
  • Figure 6 shows filtered boxes by category on the left and three zoomed element records with bounding box, text, and category on the right.Source: Figure 6 · p. 18

Related papers

Loading 2610.19100v1…