PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Look Beyond Screens: Transferable Visual Observation from GUI Grounding

Ruilin Yao, Xinyu Qiu, Tianyu Zou, Yi Rong, Shengwu Xiong, Shuheng Shen, Changhua Meng

arXiv:2610.37093v1cs.CVcs.AI

TL;DR

The paper asks whether GUI grounding can provide spatial supervision for observation behavior that transfers beyond interfaces. It trains a crop-and-inspect policy with supervised initialization followed by reinforcement learning, and reports gains over SFT across non-GUI benchmarks. The findings support GUI grounding as a practical source of transferable supervision for active visual perception.

  • Problem

    A central challenge in active visual perception is connecting intermediate observation decisions to task success when correct answers can follow informative or redundant inspection.

    Source: §1 · pp. 1–2

  • Method

    The framework initializes crop-and-inspect behavior with SFT on teacher-generated GUI grounding trajectories, then uses multi-turn RL with spatial rewards and selective response updates.

    Source: §1 · p. 2

  • Results

    RL improves ScreenSpot-Pro accuracy over SFT by 1.83 points for Qwen3.5-35B-A3B and 6.13 points for Qwen3.8-27B, with PerceptionBench gains of approximately 3.0 and 7.10 points, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support GUI grounding as a practical source of supervision for visual observation whose benefits extend beyond the source domain.

    Source: §1 · p. 2, §5 · p. 11

    Crop overlap serves as a proxy for spatial relevance and does not directly evaluate interpretation of the returned image.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Visual perception is evolving from passive recognition toward active, task-oriented observation. We investigate GUI grounding as a scalable source of spatial supervision for learning observation behaviors that transfer beyond interfaces. We introduce LBS (Look Beyond Screens), a vision-language model trained through supervised initialization on teacher-generated GUI grounding trajectories, followed by reinforcement learning for multi-turn crop-and-inspect decisions. Under fixed inference settings, LBS models based on Qwen3.5-35B-A3B and Qwen3.8-27B consistently improve over their corresponding baselines. Gains beyond supervised fine-tuning transfer to PerceptionBench for both backbones and to HR-Bench and ChartQAPro for Qwen3.5-35B-A3B, supporting observation strategies that generalize beyond GUI-specific imitation. Further analysis suggests complementary roles for the two stages: SFT establishes visual inspection primitives, while RL improves their downstream effectiveness. These findings support GUI grounding as a practical supervision paradigm for transferable active visual perception and broader multimodal reasoning. Our code and model are publicly available at https://github.com/inclusionAI/UI-Venus/tree/LBS and https://huggingface.co/inclusionAI/LBS-27B, respectively.

1. Introduction

The paper asks whether GUI grounding’s spatial supervision can teach observation behavior that transfers to non-GUI perception. It introduces a two-stage training framework and reports gains beyond supervised initialization across multiple benchmarks.

Motivation

  • A correct final answer can follow informative or redundant inspection, so the central challenge is connecting intermediate observation decisions to task success.GUI targets make both localization and crop geometry measurable, while the authors ask whether this supervision transfers to natural images and charts.Source: §1 · pp. 1–2

Method

  • The framework initializes crop-and-inspect behavior with SFT on teacher-generated GUI grounding trajectories, then refines it with multi-turn RL using spatial rewards and selective response updates.The same crop interface is used across domains, with task-specific final answers.Source: §1 · p. 2

Results

  • RL improves ScreenSpot-Pro over SFT by 1.83 and 6.13 points, and PerceptionBench by approximately 3.0 and 7.10 points, across the Qwen3.5-35B-A3B and Qwen3.8-27B backbones.The Qwen3.5-based policy also improves on HR-Bench 4K/8K and ChartQAPro.Source: §1 · p. 2
  • Outcome-only RL improves source-domain GUI grounding but lowers PerceptionBench accuracy, whereas the spatially supervised configuration improves both, showing that higher GUI grounding does not ensure stronger non-GUI transfer.Source: §1 · p. 2

Contributions

  • The results support GUI grounding as supervision for observation policies that transfer beyond interfaces, while showing that better GUI grounding alone need not predict non-GUI gains.The contributions also include controlled edits to historical crop observations.Source: §1 · p. 2

2. Related Work

Related work studies agentic interaction and visual grounding through varied tools, training paradigms, and goals. This paper connects GUI grounding to these directions by using GUI target annotations to supervise observation actions and testing transfer to non-GUI perception.

Agentic visual reasoning

  • Agentic visual reasoning includes interaction-based approaches such as ReAct and Toolformer, as well as visual agents that manipulate images or use tools during reasoning.Prior work establishes the utility of additional observations and tool interaction; this study focuses on how inspection behavior is supervised and whether it transfers.Source: §2 · p. 3

Visually grounded reasoning

  • Visually grounded reasoning spans training-based and test-time methods, with approaches that suppress irrelevant background or enhance task-relevant regions through zooming.Examples include ViGoRL, which anchors reasoning to coordinates, and TreeVGR, which jointly supervises localization and reasoning.Source: §2 · p. 3
  • The study uses GUI target annotations to supervise intermediate observation actions, then evaluates the policy across different visual content and answer formats.Its distinction from the reviewed directions is the source of supervision and its transfer from one spatially annotated interaction domain.Source: §2 · p. 3

3. Method

The method learns a crop-and-inspect policy from GUI grounding, using supervised initialization followed by reinforcement learning with spatial feedback. Training-only target annotations guide crop rewards and response updates, while inference uses a shared crop interface across domains.

3. Method

  • The method uses SFT to teach an interaction protocol, then RL to score crop geometry and filter intermediate response updates using grounding outcomes.Target annotations remain training-only, and final answer formats adapt to each task.Source: §3 · p. 3

3.1. Grounding Supervision and Policy Initialization

  • Synthetic GUI examples pair instructions with rendered interfaces and filtered visible targets; successful teacher-generated trajectories are retained, although their intermediate inspection quality remains unconstrained.The resulting trajectories supply supervised initialization for an inspection-capable baseline.Source: §3.1 · pp. 3–5
  • The student is supervised on assistant outputs conditioned on prior images, responses, and tool feedback, producing the checkpoint that initializes RL.This permits evaluation of improvement beyond imitation of the tool-use protocol.Source: §3.1 · p. 5

3.2. Multi-Turn Visual Observation

  • At each turn, the policy can crop an available image, inspect returned crops recursively, and give its final grounding prediction in original-image coordinates.Crop results and their coordinate mappings enter the interaction history, while prior images and messages remain available.Source: §3.2 · p. 5

3.3. Outcome-Gated Spatial Rewards

  • The spatial process reward favors target overlap while penalizing repeated crops, declining overlap, and tool use; a separate bonus encourages coverage of small targets.The overlap proxy does not directly evaluate interpretation of the returned image, and the bonus can reward a broad crop containing the target center.Source: §3.3 · pp. 5–6
  • A text-based critic scores task focus, feedback use, reasoning–action consistency, progress, and conciseness, but receives neither images nor target annotations.Its score therefore measures textual consistency rather than visual correctness.Source: §3.3 · p. 6

3.4. Selective Response Updates and Optimization

  • Selective response updates mask spatially irrelevant intermediate outputs and filter eligible turns by token surprisal, while retaining the final response for updates.The multi-turn optimization adapts GRPO with groups of eight rollouts and a PPO-style clipped surrogate with a fixed-reference KL penalty.Source: §3.4 · p. 6

4. Experiments

Across the evaluated settings, RL improves GUI grounding and transfers beyond SFT to general perception, high-resolution imagery, and charts. Reward ablations and context comparisons show that transfer depends on reward design and the visual information supplied.

4.1. Experimental Setup

  • The study uses two backbones, freezes each vision encoder and projector, and evaluates SFT and RL under identical prompts, decoding settings, and inference limits.ScreenSpot-Pro measures GUI localization; PerceptionBench evaluates ten non-GUI capabilities, with additional tests on OSWorld-G-R, HR-Bench, and ChartQAPro.Source: §4.1 · p. 7

4.2. GUI Grounding Across Backbones

  • Qwen3.5-35B-A3B and Qwen3.8-27B improve ScreenSpot-Pro accuracy from 78.56% to 80.39% and from 75.02% to 81.15%, respectively.Qwen3.8-27B improves all six application groups over SFT; Qwen3.5-35B-A3B improves five and declines on office applications.Source: §4.2 · p. 7, Table 1 · p. 8

4.3. Transfer to General Visual Perception

  • On PerceptionBench, RL adds approximately 3.0 points for Qwen3.5-35B-A3B and 7.10 points for Qwen3.8-27B over SFT.Both final policies exceed their base models across all ten capabilities; the capability-level gains vary by backbone, and Qwen3.8-27B slightly declines in depth/3D perception.Source: §4.3 · pp. 8–9, Table 2 · p. 8

4.4. Transfer Across Visual Domains

  • RL exceeds SFT on all four Qwen3.5-35B-A3B benchmarks in Figure 2, including gains of 2.2, 3.1, and 3.2 points on HR-Bench 4K, HR-Bench 8K, and ChartQAPro.On OSWorld-G-R, accuracy rises from 66.9% after SFT to 72.7% after RL.Source: §4.4 · p. 9, Figure 2 · p. 9

4.5. Reward Design Ablation

  • Outcome-only RL raises ScreenSpot-Pro to 79.19% from SFT’s 78.56% but lowers PerceptionBench to 40.87% from 42.10%; the full configuration improves both.The full configuration exceeds outcome-only RL by 1.20 points on ScreenSpot-Pro and 4.20 points on PerceptionBench. Component effects depend on their combination, and the full configuration achieves the highest PerceptionBench accuracy (45.07%).Source: §4.5 · pp. 9–10, Table 3 · p. 10

4.6. Effects of Visual Context on Grounding and Reasoning

  • Metadata-only contexts outperform unupsampled crops on ScreenSpot-Pro (70.6% vs. 68.8%), while crops outperform metadata-only contexts on PerceptionBench (52.2% vs. 49.9%).Upsampled crops achieve the highest accuracy on both benchmarks (81.15%/59.67%); these reconstructed-context comparisons do not isolate observation behavior acquired during RL.Source: §4.6 · pp. 10–11, Table 4 · p. 10

5. Conclusion

GUI grounding offers a practical source of supervision for active visual perception, with improvements reported in both GUI grounding and non-GUI perception.

5. Conclusion

  • The results demonstrate improvements in both GUI grounding and non-GUI perception using open-weight VLMs and newly established baselines.The authors suggest that intermediate-observation supervision helps models inspect question-relevant regions, while crop upsampling makes fine-grained details more accessible.Source: §5 · p. 11
  • GUI grounding provides a practical source of transferable supervision for active visual perception.The authors hope this approach contributes to models that actively seek relevant evidence and develop a deeper understanding of the visual world.Source: §5 · p. 11
  • The authors suggest that supervising intermediate observation decisions helps models inspect question-relevant regions.They also suggest crop upsampling makes fine-grained details more accessible for visual understanding.Source: §5 · p. 11

6. Discussion

The discussion presents active visual perception as increasingly accessible with suitable supervision and a crop-and-inspect interface, while identifying video observation as a next direction. Extending observation to long videos and streaming inputs raises evidence, memory, context, latency, and computation challenges.

6. Discussion

  • With suitable supervision and a simple crop-and-inspect interface, modern VLMs can improve image-level perception without lots of additional general-domain VQA training.The authors describe agentic visual perception as increasingly accessible to modern VLMs.Source: §6 · p. 11
  • A promising direction is extending observation policies from individual images to video by selecting informative frames and maintaining useful visual memory.The discussion cites Video-Index as highlighting both the potential of agent tools and remaining challenges in video understanding, including evaluation requiring temporal evidence.Source: §6 · p. 11
  • Long-video and streaming observation must balance evidence acquisition against context, latency, and computation budgets.Source: §6 · p. 11

A. Additional Benchmark Protocols

The appendix describes three evaluation benchmarks: OSWorld-G for GUI grounding, HR-Bench for high-resolution perception, and ChartQAPro for chart question answering.

A. Additional Benchmark Protocols

  • OSWorld-G contains 564 finely annotated GUI grounding samples from OSWorld rollouts, covering five grounding categories.Unlike ScreenSpot-Pro's extreme resolutions, it uses standard 720p/1080p screenshots and emphasizes software commonsense and layout reasoning.Source: A. Additional Benchmark Protocols · p. 16
  • HR-Bench evaluates high-resolution perception at 4K and 8K, with 100 fine-grained single-instance and 100 cross-instance questions per resolution.Its cyclic-permutation protocol averages accuracy over cyclic rotations of answer choices.Source: A. Additional Benchmark Protocols · p. 16
  • ChartQAPro contains 1,341 charts from 157 sources and 1,948 questions across five question types, including unanswerable cases.Scoring is answer-type-specific: numeric answers allow 5% relative error, years and categories require exact matches, and textual answers use average normalized Levenshtein similarity.Source: A. Additional Benchmark Protocols · p. 16

B.1. Training Configuration

The appendix specifies the training configuration, selective response masking, group-relative optimization, and crop construction used in LBS.

B.1. Training Configuration

  • Both backbones use the same RL setup, including GRPO with eight sampled trajectories, AdamW, and a global batch of 512 trajectories across 64 NPUs.The training launcher configures one epoch, bfloat16 precision, and asymmetric token-level clipping bounds of 0.2 and 0.28.Source: §B.1 · p. 16

B.2. Selective Masking Details

  • Intermediate responses are masked sequentially by spatial relevance and a surprisal threshold, while the final assistant turn is always retained.The uncertainty filter removes turns below median uncertainty among spatially eligible turns; masked responses remain in the conditioning history but receive no direct response loss.Source: §B.2 · p. 16

B.3. Group-Relative Optimization

  • GRPO samples eight trajectories per task and subtracts their group-mean reward without variance normalization; critic-enabled runs add a within-group ranking signal through reward.Source: §B.3 · p. 17
  • The optimization averages retained tokens within each trajectory and then across trajectories, with KL regularization in the loss rather than the reward.The policy term uses clipped current-to-old policy ratios with separately configured lower and upper bounds.Source: §B.3 · p. 17

B.4. Crop Construction

  • Requested rectangles are rounded outward to pixel boundaries and clamped to the source image; degenerate crops and excessive aspect ratios are rejected.Nested requests read from the previously returned, resized crop rather than re-extracting pixels from the original image.Source: §B.4 · p. 18

B.5. Benchmark Evaluation Protocol

The protocol evaluates grounding and non-GUI perception on their benchmark-specific tasks and scores results separately. Grounding predictions are points in original-image coordinates, while non-GUI tasks use reference-based answers.

B.5. Benchmark Evaluation Protocol

  • Grounding accuracy uses inclusive point-in-box matching in original-image coordinates; invalid coordinate predictions count as incorrect.The evaluation covers ScreenSpot-Pro, OSWorld-G-R, HR-Bench 4K/8K, and ChartQAPro for Qwen3.5-35B-A3B, plus ScreenSpot-Pro and PerceptionBench for both backbones.Source: §B.5 · p. 18
  • PerceptionBench uses open-ended questions with short reference answers and reference-based judgment; its score is reported separately from other benchmarks.The paper does not combine heterogeneous benchmark scores.Source: §B.5 · p. 18
  • Evaluation uses benchmark-specific prompts and answer interfaces, with no ground-truth regions or training-time spatial rewards supplied at test time.Cold-start and RL checkpoints use the same task interface and inference configuration within each benchmark; Qwen3.8-27B uses xhigh reasoning, a 4,096-token thinking budget per turn, and at most three crop calls.Source: §B.5 · p. 18

C. Observation Feedback Protocols

The protocol reconstructs crop-observation histories from one reference interaction to compare the supplied visual context while retaining the original image. Upsampled crops score highest on both benchmarks, though the comparisons do not isolate crop-feedback effects.

C. Observation Feedback Protocols

  • Three reconstructed histories provide unupsampled crop images, metadata without crop images, or crop images at target scale α = 4; the original image remains available in each.The variants edit observations from a shared reference history rather than generating independent online trajectories.Source: C. Observation Feedback Protocols · pp. 18–19
  • The overall contrast includes differences in checkpoint and supplied context, so it does not isolate crop-feedback effects.Image and scale contrasts instead compare context variants reconstructed from the same reference history.Source: C. Observation Feedback Protocols · p. 20
  • Upsampled crops exceed metadata-only contexts by approximately 10.55/9.77 percentage points on SSPro/PerceptionBench and unupsampled crops by 12.35/7.47 points.They achieve the highest accuracy on both benchmarks; the results concern answer generation under reconstructed histories, not independently executed crop-selection policies.Source: C. Observation Feedback Protocols · p. 20
  • Metadata-only contexts score higher on SSPro than unupsampled crops (70.6% versus 68.8%), while unupsampled crops score higher on PerceptionBench (52.2% versus 49.9%).The authors suggest coordinate-level visual primitives may support grounding, while localized image content may aid non-GUI reasoning; both tasks benefit from suitably presented local evidence.Source: C. Observation Feedback Protocols · p. 20

D. Extended PerceptionBench Reference Results

This appendix section presents external PerceptionBench reference scores, example agent trajectories, and implementation details for agentic RL and trajectory scoring. The examples illustrate successive crop-and-inspect actions, while the critic evaluates text without access to images.

D. Extended PerceptionBench Reference Results

E. Example of a complete agentic grounding/reasoning trace

  • The GUI and non-GUI traces illustrate sequential crop-and-inspect behavior, including correcting misplaced crops and recursively inspecting returned observations.The PerceptionBench OCR and vehicle-recognition examples end with answers matching their references; GUI observations are reconstructed from logged actions, with colored annotations editorial.Source: Figure 4 · p. 22, Figure 5 · p. 23

F.1. System Prompt for Agentic RL

  • The RL prompt uses textual crop actions with local-image pixel coordinates, while final grounding coordinates must be normalized to the original image; crops may be nested.The protocol encourages at least one successful crop and asks the agent to answer once the target is localized, with a configured call budget capped at three.Source: §F.1 · pp. 24–26

F.2. Prompts for CoT Scoring

  • The blind critic sees task and transcript text but no images, and transcript limits can truncate the trajectory it scores.The default reward multiplies its parsed score by terminal correctness; trajectories without visible reasoning receive zero.Source: §F.2 · pp. 26, 28

Related papers

Loading 2610.37093v1…