PaperlayerView exact revision on arXiv

Source-linked AI summary

One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding

Chen Liu, Ling Chen, Hanzhang Zhou, Liangyu Chen, Chenglin Cai, Xin Yu, Steven Hoi, Yue Wang

arXiv:2606.30084v1cs.CV

TL;DR

GUI grounding models can form target-region evidence before coordinate decoding but fail to preserve it precisely, while external zooming adds inference cost. InnerZoom bridges this evidence across decoder layers in one forward pass, achieving the best accuracy across six benchmarks, including a 4.1-point gain on OSW-G over the strongest prior result.

  • Problem

    Autoregressive coordinate generation can lose fine-grained target-region evidence between region identification and precise GUI click prediction.

  • Method

    InnerZoom extracts intermediate target-region evidence, preserves and refines it across decoder layers, and reinjects it to guide coordinates within one forward pass.

  • Results

    InnerZoom achieves the best accuracy on all six GUI grounding benchmarks, improving OSW-G over the strongest prior result by 4.1 points.

  • Takeaways & Limitations

    InnerZoom provides an effective single-forward alternative to two-pass zoom-in refinement while improving GUI grounding accuracy and maintaining efficient inference.

  • Takeaways & Limitations

    InnerZoom depends on intermediate target-region cues, so incomplete or biased internal responses may limit subsequent evidence refinement.

Abstract

from arXiv · show

MLLM-based GUI grounding methods commonly formulate target localization as autoregressive coordinate generation, enabling models to leverage the strong instruction-following and semantic understanding capabilities of MLLMs. However, this formulation requires the model to retain region-level target evidence while decoding coordinate tokens with the spatial precision demanded by GUI clicking. Our diagnostic analysis reveals that target-region awareness emerges in intermediate decoder layers but is neither retained nor translated into the final coordinate prediction. Existing ZoomIn-style methods address this issue through an external crop-and-rerun pass, which improves localization but increases end-to-end latency and computational cost. To retain the accuracy benefits of two-pass zooming without this extra cost, we propose InnerZoom, a single-forward framework for cross-layer evidence bridging. InnerZoom transforms target-related cues from the original forward pass into a compact cross-layer evidence state, then preserves, refines, and reinjects this state throughout later decoding layers to guide coordinate prediction. Extensive experimental results suggest that InnerZoom-4B achieves state-of-the-art performance on all six GUI grounding benchmarks, obtaining 64.7 on OSWorld-G, 40.2 on UI-Vision, 73.1 on OSWorld-GR, and 87.6 on MMBench-GUI, surpassing the previous best results by 4.1, 3.2, 2.9, and 2.3 points, respectively. Under a controlled 4B setting, InnerZoom improves the same SFT+RL baseline by 5.3 points on average and outperforms two-pass ZoomIn by 1.3 points on average, while reducing end-to-end latency by up to 31.8% and TFLOPs by about 29%. Code and models will be publicly available.

1. Introduction

MLLM-based GUI grounding commonly generates click coordinates autoregressively, but fine-grained target-region evidence can fade before becoming precise coordinate tokens. InnerZoom addresses this Region-to-Point Gap with single-forward cross-layer evidence preservation and reinjection during decoding.

  • Background: GUI grounding requires predicting an executable click coordinate from a user instruction and GUI screenshot.MLLM-based methods commonly formulate this as autoregressive coordinate generation.
  • Region-to-Point Gap: Autoregressive decoding must preserve fine-grained target-region cues while expressing them as discrete coordinate tokens.This creates a mismatch between region-level visual evidence and point-level coordinate prediction.
  • Diagnostic Analysis: Intermediate decoder layers often develop localized high-activation regions around the ground-truth target before final coordinate prediction.Controlled modulation suggests that amplifying these regions improves grounding accuracy, unlike an equal-strength random-token intervention.
  • InnerZoom: InnerZoom converts target-region evidence into a compact cross-layer state, progressively refines it, and reinjects it during later decoding layers.The framework uses the model’s own text-image responses and operates in a single forward pass.

2. Motivation for the Region-to-Point Gap in GUI Grounding

Intermediate decoder layers can identify target regions, but this evidence fades and is not reliably converted into final click coordinates. Controlled modulation shows that strengthening target-region evidence causally improves grounding, motivating methods that preserve and reuse it.

  • Intermediate evidence: 69.0% Top-1 ROI Recall peaks between decoder layers 19 and 23 across four benchmarks, then drops to 14.0% at the final layer.This shows that grounding failures are not solely caused by inability to find the target region.
  • Region-to-point gap: Increasing k rapidly raises ROI Recall toward saturation, yet final coordinate accuracy remains substantially lower.The main deficit is converting region-level evidence into a precise point-level action.
  • Region-to-point gap: Intermediate target-region evidence fades before final coordinate prediction and is not reliably converted into the final click.This finding identifies a preservation and conversion failure rather than an evidence-acquisition failure.
  • Causal evidence: Target-region token modulation improves grounding because it concentrates attention on target-region evidence rather than merely increasing attention magnitude.The controlled experiment establishes that intermediate target-region evidence causally influences final grounding.
  • Motivation: The bottleneck is preserving and using decision-relevant intermediate evidence for final coordinate prediction, not acquiring target-region evidence.Region-guided modulation confirms that strengthening target-region responses can improve grounding accuracy, but provides only a transient boost.

3. Method

InnerZoom is a single-forward GUI grounding framework that converts intermediate target-region awareness into refined evidence for precise coordinate prediction. It localizes target regions, recurrently refines evidence across decoder layers, and injects it through keys and values while preserving autoregressive decoding.

  • Target-region localization: InnerZoom derives a text-image relevance map from intermediate decoder responses to propose a coarse target region and retrieve fine-grained visual features.The retrieved features come from before visual-token merging, preserving richer local layout and appearance details.
  • Cross-layer evidence workspace: A shared Iterative Dual-Slot Evidence Adapter propagates two learnable evidence slots across selected decoder layers for recurrent refinement.The adapter maintains an evolving cross-layer workspace within the original forward pass.
  • Cross-layer evidence workspace: Each slot aggregates regional evidence using decoding context, slot state, and a cached visual log-prior that guides attention toward the estimated target region.A learnable gate controls how much candidate evidence is written while preserving evidence from previous layers.
  • KV evidence injection: Refined slot evidence is injected only into target-region key/value projections, leaving the token sequence, query projections, and non-region projections unchanged.This minimally invasive strategy preserves the original decoding flow while providing position-specific visual evidence for coordinate prediction.
  • Training: Training uses autoregressive cross-entropy during SFT, slot-separation regularization, and GRPO during RL with rewards based on whether parsed coordinates fall inside target bounding boxes.The SFT objective covers grounding reasoning and the final coordinate answer, while RL further optimizes point-level grounding accuracy.

4. Experiments

Experiments show that InnerZoom-4B achieves the best accuracy across six GUI grounding benchmarks and improves the accuracy–efficiency trade-off over two-pass zooming. Ablations indicate that compact cross-layer interaction and a two-slot evidence workspace are effective, while remaining errors involve difficult instruction and text-grounding cases.

  • Overall benchmark results: InnerZoom-4B achieves the best accuracy on all six benchmarks, improving over the strongest prior result by 2.9, 4.1, 0.3, 1.1, 3.2, and 2.3 points on OSW-GR, OSW-G, SS-V2, SS-Pro, UI-V, and MMB-GUI, respectively.It also surpasses substantially larger 7B/8B/32B/72B models.
  • Scale-matched comparison: InnerZoom-4B achieves the best results in the test-time scaling group on all four reported benchmarks, outperforming zoom/focus methods with larger backbones.The scale-matched comparison shows accuracy gains without repeated zooming or larger model scale.
  • Category-level results and limitations: InnerZoom-4B leads most category-level evaluations, including all three UI-Vision instruction types, key OSWorld-G-Refine categories, and most Windows, MacOS, Linux, and Android MMBench-GUI subcategories.On spatial UI-Vision grounding, it improves over MAI-UI-4B from 18.4 to 25.4; remaining errors mainly involve semantically challenging instructions, ambiguous text-to-image grounding, and on-screen text interference.
  • Cross-layer interaction ablation: 64.2 weighted accuracy is achieved by the four-layer design at layers 20, 23, 26, and 29 with 180.4M trainable parameters, while dense insertion raises parameters to 721.8M for only a 0.1-point gain.Two or three interaction layers reach 61.0 and 62.1 weighted accuracy, respectively.
  • Evidence-slot ablation: 64.2 weighted accuracy from the two-slot design exceeds the single-slot and three-slot variants at 62.5 and 62.8, respectively.The two-slot workspace balances capacity for target and contextual evidence against redundant aggregation.
  • Accuracy–efficiency trade-off: 1.18–1.27× latency and 1.16–1.23× TFLOPs keep InnerZoom close to the base model, versus 1.56–1.94× latency and 1.57–1.75× TFLOPs for two-pass Zoom-In.Compared with Zoom-In, InnerZoom reduces latency by 23.8–35.7% and TFLOPs by 26.0–32.0%, while outperforming it on OSW-GR, OSW-G, and UI-V by +0.6, +1.9, and +3.8 points.

5. Related Work

Prior GUI grounding work spans direct screenshot-based localization, alternative optimization objectives, zoom-based refinement, multi-step reasoning, and tool-augmented visual focus. Across these designs, methods supplement global GUI understanding with localized visual evidence for precise coordinate prediction.

  • GUI grounding: Early GUI grounding methods, including SeeClick, locate instruction-relevant interface elements directly from screenshots.GUI grounding maps natural-language instructions to executable actions such as clicking, typing, and navigation.
  • Optimization perspectives: Recent methods target instruction reasoning, position-to-coordinate mapping, or action-region prediction through distinct optimization formulations.UI-Ins improves instruction-level reasoning, RULER models position-to-coordinate mapping, and GUI-Actor predicts action regions instead of coordinates directly.
  • Zoom-based methods: Zoom-based approaches revisit local regions at higher effective resolution to improve fine-grained GUI grounding.RegionFocus formalizes dynamic zoom-in as visual test-time scaling, ZoomClick studies zooming as a training-free prior, and DiMo-GUI combines visual grounding with modality-aware reasoning.
  • Alternative refinement: Other approaches refine grounding through multiple attention-guided views, multi-step visual reasoning, or progressive cropping.MVP aggregates predictions from attention-guided views, while Chain-of-Ground and Iterative Narrowing use multi-step reasoning or progressive cropping.
  • Visual focus: Tool-augmented focus refinement and active visual perception similarly provide localized, high-resolution evidence for fine-grained coordinate prediction.Spotlight and GUI-Eyes exemplify these approaches, which supplement global GUI understanding with additional visual focus.

6. Conclusion

InnerZoom is a single-forward cross-layer evidence bridging framework for precise GUI grounding. It addresses the evidence-to-coordinate bottleneck by preserving and refining intermediate target evidence for coordinate decoding.

  • Conclusion: InnerZoom is a single-forward cross-layer evidence bridging framework for precise GUI grounding.It is presented as the paper’s central framework.
  • Conclusion: Intermediate decoder layers form useful target-region evidence but fail to reliably convert it into final click coordinates.This mismatch constitutes the identified evidence-to-coordinate bottleneck.
  • Conclusion: InnerZoom preserves and refines intermediate target evidence across decoder layers and makes it available for coordinate decoding.The framework is designed to address the gap between intermediate evidence and final coordinate prediction.

7. Limitations

InnerZoom’s evidence extraction can be limited by incomplete or biased intermediate-layer target cues, while explicit visual re-observation remains complementary for extreme ultra-wide or dual-screen interfaces. These limitations motivate more robust evidence extraction and selective zoom-based re-observation.

  • Evidence extraction: InnerZoom depends on target-region cues emerging from intermediate decoder layers, so incomplete or biased responses can constrain later evidence refinement.The selected region may limit subsequent evidence refinement when internal responses are incomplete or biased.
  • Evidence extraction: Because region selection is only indirectly optimized during training, future work could develop learnable or weakly supervised evidence-extraction mechanisms.These mechanisms are intended to make evidence extraction more robust.
  • Visual re-observation: For ultra-wide or dual-screen interfaces, two-pass zoom-in can provide higher effective target resolution and visual details difficult to recover from one forward pass.Explicit visual re-observation may therefore complement single-forward evidence bridging in extreme cases, despite InnerZoom’s strong results on high-resolution benchmarks such as ScreenSpot-Pro.

A. Appendix

The appendix details InnerZoom’s method, diagnostic intervention, experimental setup, and additional results. It also organizes fine-grained, qualitative, latency, TFLOPs, and failure analyses.

  • A.1 Method Details: Method details cover target-region proposal and feature retrieval, slot separation regularization, and the training objective.
  • A.2 Implementation Details of Diagnostic Attention Intervention: The appendix describes implementation details for the diagnostic attention intervention.
  • A.3–A.4 Experiment Details and More Experimental Results: It reports experiment details and more experimental results, including latency measurement, TFLOPs estimation, and fine-grained results.
  • A.4.3–A.4.4 Additional Analyses: Additional qualitative results and failure analysis are included.

A.1. Method Details

This section details InnerZoom’s implementation, covering token localization, heatmap construction, ROI selection, and mapping expanded regions to token indices.

  • Implementation Details: The method locates text and visual tokens, builds a heatmap from text-to-vision responses, selects ROIs via connected-component analysis, and maps expanded regions to token indices.These operations constitute the core implementation details described for the method.

A.1.1. Target-Region Proposal and Feature Retrieval

InnerZoom proposes target regions from a single intermediate-layer text-to-vision heatmap, preserving optimized attention backends through selective recomputation. It selects spatially coherent responsive components, expands their boxes, and maps them to visual-token positions for feature retrieval and evidence injection.

  • Text-to-Vision Heatmap Construction: A single intermediate decoder layer supplies the text-to-vision heatmap used to propose target regions.The method identifies user-instruction and visual-token spans before constructing the heatmap.
  • Text-to-Vision Heatmap Construction: Selective recomputation obtains the required attention response while preserving optimized attention backends.For the selected layer, the method reuses Q/K projections and computes restricted user-text-to-visual attention.
  • Connected Component Selection: q_thr = 0.90 retains the top 10% responsive positions, which 8-neighbor connected components group into candidate regions.Components are characterized by area, total and mean heat response, and minimum enclosing bounding boxes.
  • Bounding Box Expansion and Token Index Mapping: Expanded component boxes are mapped to flattened visual-token indices for fine-grained feature retrieval and KV-only evidence injection.Expansion compensates for heatmaps covering only highly responsive subregions and uses the same fixed ratio across experiments.

A.1.2. Slot Separation Regularization

Slot separation regularization prevents the focus and context slots from collapsing into redundant representations. It begins by breaking their symmetry through orthogonal initialization of both slot queries and initial cross-layer states.

  • Motivation: Similar initial query directions can make the focus and context slots attend to similar visual patches and converge to redundant representations.This collapse weakens separation between click-relevant local cues and contextual information.
  • Orthogonal Initialization: Orthogonal initialization gives the focus and context slot query embeddings approximately orthogonal directions.Distinct initial query biases induce different attention patterns from the first forward pass.
  • Orthogonal Initialization: Initial cross-layer slot states are initialized as small orthogonal vectors rather than zeros.This avoids identical early gated updates between the two slots.

A.1.3. Training Objective … A.4.4. Failure analysis.

InnerZoom is trained with supervised coordinate generation followed by GRPO grounding optimization, and evaluated through diagnostic interventions, controlled efficiency measurements, fine-grained benchmarks, qualitative examples, and failure analysis. Results show broad GUI-grounding gains, while remaining errors involve semantic difficulty, visual ambiguity, and confusion between interface text and user instructions.

  • A.1.3. Training Objective: InnerZoom uses supervised fine-tuning for coordinate generation and slot-separation stabilization, followed by GRPO to optimize point-level grounding accuracy.The SFT objective combines autoregressive cross-entropy with slot-separation regularization, using 𝜆sep = 0.02.
  • A.2. Implementation Details of Diagnostic Attention Intervention: Diagnostic attention intervention tests whether intermediate-layer target regions contain decision-relevant evidence by amplifying selected tokens and suppressing others.The hard-bias experiment uses 𝛽in > 0 for selected-region tokens and 𝛽out < 0 elsewhere, with a random-region control using the same amplification budget.
  • A.4.2. More Fine-grained Results.: 64.7 is InnerZoom-4B’s average OSWorld-G score, improving over MAI-UI-4B’s 60.3 and achieving the best average results across the reported fine-grained benchmark comparisons.InnerZoom-4B scores 66.2 on ScreenSpot-Pro and performs particularly well across dense icon-based categories and desktop/web splits.
  • A.4.3. Additional Qualitative Results.: Qualitative examples show accurate point-level predictions on dense professional interfaces and generalization across diverse UI layouts and interaction targets.Examples include small icons, tool panels, menu entries, formatting controls, debugging buttons, file operations, terminal regions, and layout elements.
  • A.4.4. Failure analysis.: The failure analysis identifies three remaining error types: semantically challenging instructions, ambiguous text-to-image grounding, and misinterpreting on-screen text as user instructions.These failures reflect missing domain knowledge, non-unique or visually similar targets, and distraction by instruction-like interface text.
Loading 2606.30084v1…