IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints

1 Introduction

• IGG measures the reveal step that static grounding benchmarks leave unmeasured, separating visibility recovery from target recognition and final-click localization.It scores a single final click while requiring the agent to make the target visible beforehand.Source: §1 · p. 1
• IGG contains 601 tasks across 16 controllable GUI environments and seven visibility-constraint sub-types.Its compact action space includes scrolling, moving, dragging, waiting, and clicking.Source: §1 · pp. 1–2
• Human solvability exceeds model grounding accuracy across all three constraint types, with the largest gap varying by model.The gap is largest on viewport constraints for Qwen3-VL and on temporal and manipulation constraints for GPT-5.5.Source: §1 · p. 2
• The environments preserve interaction patterns such as hover reveals, popups, delayed activation, carousels, and movable occlusions.Deterministic initial states and stable target identities make evaluation replayable at click time.Source: §1 · p. 2
• Training on successful interaction trajectories followed by online GRPO improves visibility recovery consistently across three model families.The experiments identify interaction execution, rather than knowing which interaction to perform, as the main bottleneck.Source: §1 · p. 2

2 Related Work

• ScreenSpot, ScreenSpot-Pro, and OSWorld-G evaluate localization within screenshots where the target is visible.Universal grounding extends localization to agent settings but still predicts from the currently observed state.Source: §2 · p. 2
• Navigation benchmarks score end-to-end completion, collapsing planning, exploration, and grounding failures into one signal (Deng et al., 2023; Zhou et al., 2024; Xie et al., 2024).IGG complements rather than replaces them by scoring visibility recovery followed by one evaluated click.Source: §2 · pp. 2–3
• Table 1 presents the minimal action space for visibility recovery.IGG’s action set is restricted to reveal-and-click interactions.Source: Table 1 · p. 3

3 The IGG Benchmark

• IGG aims to measure visibility recovery in isolation and reproducibly, separating it from broader GUI navigation.The benchmark formalizes the task, categorizes required interactions, describes its environments and task construction, and specifies evaluation and human validation.Source: §3 · p. 3

3.1 Problem formulation

• An IGG task begins with a target that is not visible or actionable and a goal-only instruction; success requires clicking the target’s stable identifier.Agents use screenshot-only observations and a restricted set of five reveal-and-click actions; the episode ends on a correct click or action-budget exhaustion.Source: §3.1 · p. 3

3.2 Visibility constraint taxonomy

• The taxonomy covers viewport constraints, multi-state constraints, and temporal or manipulation constraints, each defined by how visibility or actionability is recovered.Examples include scrolling to an off-screen target, using hover or opening a hidden region, waiting for activation, and dragging an occluder.Source: §3.2 · p. 3

3.3 Environments and task construction

• IGG comprises 601 tasks across 16 controllable environments modeled on widely used applications.Fixed initial states, reproducible transitions, and stable element identifiers support click-time evaluation; tasks also include hover reveals, popups, delayed activation, carousels, and draggable occluders.Source: §3.3 · pp. 3–4
• Annotators revise instructions that reveal the procedure, add natural disambiguating cues when needed, and replay traces to verify that each target is unique and reachable.Figure 2 shows examples of the three constraint types, with additional examples in Appendix E.2.Source: §3.3 · p. 4, Figure 2 · p. 4

4.1 Evaluation setup

• Models receive screenshots, goal-only instructions, and action history, with DOM and target information hidden; the eight-action evaluation reports grounding accuracy and interaction-efficiency metrics.Figure 3 breaks down task counts by constraint type, sub-type, and platform; runtime is descriptive because local and API-backed models use different serving stacks.Source: §4.1 · pp. 4–5, Figure 3 · p. 5

4.2 Main results

• On validation, GPT-5.5 is more accurate but averages 66.5 seconds per task versus 14.2 seconds for trained Qwen3-VL, and requires paid API inference.Evaluating GPT-5.5 over the complete benchmark cost approximately $74.9.Source: §4.2 · pp. 5–6, Figure 5 · p. 6

4.3 Where does interactive grounding fail?

• Completing the reveal interaction raises Qwen3-VL accuracy by 51.1 points, while merely naming the interaction changes it by 3.1 points.Most remaining errors involve choosing the reveal action's location or parameters and executing it correctly.Source: §4.3 · p. 6, Table 4 · p. 6
• Fixed scrolling, hovering, and waiting reveal 68.8%, 61.4%, and 87.1% of matched tasks, but popup dismissal and occluder dragging reach only 17.6% and 16.9%.Failed runs omit required DRAG in 59/59 cases, SCROLL in 207/247, and MOVE in 71/96.Source: §4.3 · pp. 6–7

4.4 Learning missing interaction priors

• 59.2% GA versus 40.0% for Qwen3-VL after two-stage training, with gains also reported for InternVL3.5 and GUI-Owl-1.5.Frequency-balanced SFT adds rare reveal actions to policy support; GRPO then improves their selection and execution, but GRPO from the standard SFT seed does not improve over that seed.Source: §4.4 · p. 7, Figure 6 · p. 8, Figure 7 · p. 8

4.5 Human solvability and environment fidelity

• Humans solve 298/315 trials (94.6%); mock environments average 4.50, 4.64, and 4.75 for visual realism, interaction realism, and task executability, with model ordering preserved in preliminary paired mock–real tests.The paired mock–real evaluation covered 16 tasks, one per environment.Source: Table 5 · p. 7, Table 6 · p. 7, §4.5 · p. 8

• IGG makes visibility recovery a measurable component of GUI grounding.Source: §5 · p. 8
• Knowing which interaction is required is not enough; models most often fail to decide where and how to carry it out.Source: §5 · p. 8
• Training on successful interaction trajectories followed by online feedback improves visibility recovery consistently across model families.IGG offers a focused setting for evaluating and learning the interactions that precede a grounding click.Source: §5 · p. 8

6 Limitations

• IGG uses deterministic mockups for controlled, reproducible evaluation, but they do not replace evaluation on live interfaces.The authors release only controlled mockups, which are not connected to external services.Source: §6 · p. 9
• Because interactive grounding may support accessibility and UI testing but also be misused for automated interaction, the release is restricted to mockups without external-service connections.Source: §6 · p. 9

A Mockup Applications

• The benchmark spans 16 controllable applications combining deterministic state transitions and stable target identities with interaction patterns modeled on familiar interfaces.Table A1 lists the reference interfaces; Figures A1–A5 show representative states.Source: A Mockup Applications · p. 11, Table A1 · p. 11

B.1 Viewport robustness

• Responsive layouts can change whether a target is hidden and which interaction reveals it, making viewport size part of the task specification.The main benchmark fixes the viewport at 1440 × 900 to keep initial visibility and constraint type constant across models.Source: §B.1 · p. 11
• On 127 tasks valid across five viewport sizes, trained Qwen3-VL outperforms its base model by 11.1–17.3 points at every size, including mobile portrait.The subset has 103 training and 24 held-out tasks, measuring survival of behavior learned at 1440 × 900 across layout changes rather than held-out generalization.Source: §B.1 · p. 11, Table A2 · p. 11
• The robustness study regenerates 601 tasks at desktop, mobile portrait, tablet, split-screen, and zoomed sizes, retaining tasks valid in all five layouts.Validity requires an initially hidden target and successful replay of the reference trace.Source: §B.1 · p. 11, Table A2 · p. 11

C.1 Per-environment results

• Open-source models trail GPT-5.5 in every environment; Design Canvas is hardest for them, while Video Platform and Email Client are easiest.GPT-5.5 is weakest on Developer Hub, Community Chat, and Social Feed.Source: §C.1 · p. 11

C.2 Mockup fidelity

• Matched mock-real pairs have higher DINO and SigLIP similarity and lower DreamSim distance than non-matched pairs, indicating preserved application-level visual structure.DINO similarity is 0.7724 vs. 0.5442, SigLIP similarity 0.8094 vs. 0.6211, and DreamSim distance 0.2529 vs. 0.5229.Source: §C.2 · p. 11, Table A3 · p. 14
• The reported three mean ratings are 4.50, 4.64, and 4.75 out of 5 (Table 6), but the cited passage does not map them to individual measures.Source: §C.2 · p. 14
• Across 16 paired mock-real tasks, model ordering is preserved and mock-real accuracy is strongly consistent across four model families.Qwen3-VL SFT+GRPO leads in both settings with 56.3% GA; Pearson r = 0.98 and Spearman ρ = 1.00 across families.Source: §C.2 · p. 14, Table A4 · p. 14

C.3 Instruction design

• Instructions name the target without specifying the reveal procedure; annotators add natural cues when labels are ambiguous and replay tasks to verify targets are unique and reachable.Source: §C.3 · p. 14
• Procedural hints raise Qwen3-VL from 30.8% to 56.9% GA over 601 tasks, but it solves 0 of 8 held-out movable-occlusion tasks; SFT+GRPO solves 5 of 8 without test-time hints.The main benchmark retains goal-only instructions; the authors characterize training gains as an executable interaction prior rather than instruction following.Source: §C.3 · p. 14

D Evaluation Protocol

• Each model is evaluated once per task with greedy decoding from a reset environment, using the listed open-source and commercial baselines.The baselines include Qwen2.5-VL, Qwen3-VL, GUI-Owl-1.5, and GPT-5.5.Source: D Evaluation Protocol · p. 14
• A click succeeds only when the clicked element ID matches the annotated target ID; otherwise, the environment applies any defined transition and returns the next screenshot.Navigation outside the task environment is disabled, and episodes end on success or budget exhaustion.Source: D Evaluation Protocol · p. 14

E.1 Task specification format

• Each task record includes a goal-only instruction, target ID, initial state, and successful reference trace; the evaluator hides the target ID and uses the trace for efficiency metrics.Reference selectors resolve to the current bounding-box center, while models emit screenshot coordinates.Source: §E.1 · pp. 14, 17
• The example targets Kai in a direct-message list, with a scroll followed by a click on the revealed target.The target is identified by a stable ID in the task specification.Source: §E.1 · p. 17

E.2 Sub-type examples

• Figures A9–A15 pair an initial state where the target is not directly actionable with the final state at the successful target click.The figures provide one representative task for each IGG sub-type.Source: Figure A6 · p. 15, Figure A7 · p. 16, §E.2 · p. 17

F.1 Representative cases

• Failures across three models often omit the action that would reveal the target, as illustrated by representative traces.The table selects cases failed by Qwen2.5-VL-7B, Qwen3-VL-8B, and GUI-Owl-1.5-8B.Source: §F.1 · p. 17, Table A5 · p. 20, Figure A16 · p. 21

F.2 Trace-derived action-prior diagnostic

• The trace diagnostic compares each failed Qwen3-VL trajectory with its ground-truth trajectory and checks whether each required reveal action appears.Counts are non-exclusive because a task can require multiple setup actions.Source: §F.2 · p. 17, Table A6 · p. 20

F.3 Target-agnostic reveal heuristics

• Five target-agnostic heuristics test scrolling, hovering, dismissing popups, waiting, and dragging on their corresponding constraint sub-types.A heuristic succeeds only when the target becomes visible, unobstructed, and actionable.Source: §F.3 · pp. 17, 21, Table A7 · p. 22
• Fixed rules often resolve scrolling, hovering, and waiting, but rarely resolve removable or movable occlusions.This aligns with failed trajectories requiring DRAG, where that action is absent.Source: Table A6 · p. 20, §F.3 · p. 21, Table A7 · p. 22

G Training Details

• The two-stage training procedure uses SFT to introduce rare reveal actions and online GRPO to optimize their execution from browser feedback.The training setup excludes two unreliable tasks, then splits the remaining 599 into 479 training and 120 held-out validation tasks.Source: G Training Details · p. 21, §G.1 · p. 21

G.2 SFT

• SFT uses 1,141 action-level examples; frequency balancing expands the data to 1,460 examples and increases drag examples from 55 to 220.Actions are frequency-balanced by oversampling CLICK/SCROLL/MOVE/WAIT/DRAG by 1/1/2/2/4.Source: §G.2 · p. 21

G.3 Online GRPO

• GRPO collects online browser rollouts from 479 training tasks under the same eight-action budget used in evaluation.Its reward combines success, action count, wrong or empty clicks, invalid actions, repeated ineffective actions, repeated wrong-click locations, and a bounded reveal-progress bonus.Source: §G.3 · p. 21

G.4 Cross-architecture results

• 40.0% to 59.2% for Qwen3-VL, 28.3% to 47.5% for InternVL3.5, and 35.0% to 52.5% for GUI-Owl-1.5 after SFT+GRPO.The paper reports similar gains across architectures with different coordinate tokenization and pretraining.Source: §G.4 · p. 22, Table A8 · p. 22

H.1 Study setup

• Fifteen participants completed two browser-based protocols; the solvability study yielded 315 trials, and the fidelity study rated seven controlled environments against real-interface references.Fidelity ratings covered visual realism, task executability, and interaction realism on five-point scales.Source: §H.1 · p. 22

H.2 Task-solvability interface

• In the solvability interface, a trial ends at target click or after eight click-or-drag attempts; scroll sessions count after an 800-ms debounce, while hover events do not count toward the limit.The interface records scroll sessions for interaction counts.Source: §H.2 · p. 22, Figure A17 · p. 24

H.3 Fidelity-test interface

• In the fidelity interface, participants compare each mock interface with a real-application reference and rate visual realism, task executability, and interaction realism.The study covers seven controlled environments.Source: §H.3 · p. 22, §H.1 · p. 22, Figure A18 · p. 24