PaperlayerView exact revision on arXiv

Source-linked AI summary

GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding

Shijie Zhou, Viet Dac Lai, Hao Tan, Jihyung Kil, Wanrong Zhu, Changyou Chen, Ruiyi Zhang

arXiv:2511.00810v4cs.CVcs.AIcs.CLcs.HCcs.LG

TL;DR

Text-based coordinate generation is an indirect formulation of GUI grounding, despite useful query-visual signals already emerging in MLLM attention. GUI-AIMA directly supervises intrinsic attention with patch-wise grounding signals, and its 3B model performs strongly with one-stage fine-tuning on 101k screenshots. The paper presents this as data-efficient grounding without extra grounding modules.

  • Problem

    Existing MLLM GUI grounding commonly predicts click coordinates as text, an indirect way to express fine-grained visual localization despite available query-visual attention cues.

    Source: §1 · p. 2

  • Method

    GUI-AIMA supervises intrinsic MLLM multi-head attention with patch-wise grounding signals, using an anchor token for query-visual aggregation and query-aware head weighting.

    Source: §1 · p. 2, §3.2 · p. 7

  • Results

    With one-stage fine-tuning on 101k screenshots and no extra modules, GUI-AIMA-3B outperforms similarly sized models and remains competitive with substantially larger GUI grounding systems.

    Source: §1 · p. 3

  • Takeaways & Limitations

    GUI-AIMA shows that strong 3B-scale GUI grounding can be achieved through light one-stage fine-tuning without extra grounding modules.

    Source: §1 · p. 3

    Direct supervision on query-token attention may interfere with pretrained language-vision behavior and potentially impair pre-existing MLLM knowledge.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

Graphical user interface (GUI) grounding is a key capability for computer-use agents, mapping natural-language instructions to actionable regions on the screen. Existing Multimodal Large Language Model (MLLM) approaches typically formulate GUI grounding as a text-based coordinate generation task. However, directly generating precise coordinates from visual inputs is challenging and often data-intensive. A more intuitive strategy is to first identify instruction-relevant visual patches and then determine the exact click location within them. Motivated by recent observations that general MLLMs exhibit native grounding ability embedded in their attention maps, we propose GUI-AIMA, an attention-based and coordinate-free supervised fine-tuning framework for efficient GUI grounding. GUI-AIMA aligns the intrinsic multimodal attention of MLLMs with patch-wise grounding signals. These signals are calculated adaptively for diverse user instructions by multi-head aggregation on simplified query-visual attention matrices. Besides, its coordinate-free manner can easily integrate a plug-and-play zoom-in stage. GUI-AIMA-3B was trained with only 509k samples (around 101k screenshots), demonstrating exceptional data efficiency and verifying that light training can trigger the native grounding capability of MLLMs. It achieves state-of-the-art performance among 3B models, attaining an average accuracy of 61.5% on ScreenSpot-Pro, 92.1% on ScreenSpot-v2, 68.1% on OSWorld-G, 79.1% on MMBench-GUI-L2, and 60.0% on UI-Vision. Project page: https://github.com/sjz5202/GUI-AIMA .

1 Introduction

Existing GUI grounding methods often generate click coordinates as text, while useful query-visual signals already appear in MLLM attention. GUI-AIMA supervises that intrinsic attention with patch-wise grounding signals and reports data-efficient performance at the 3B scale.

1 Introduction

  • Coordinate-token generation is indirect for fine-grained localization, although MLLM attention can already provide query-visual grounding cues at patch level.TAG aggregates query-token attention in a vanilla manner, while GUI-Actor adds grounding modules and an adaptation stage.Source: §1 · p. 2
  • GUI-AIMA directly supervises MLLM multi-head self-attention with patch-wise signals, using an <ANCHOR> token to aggregate query-visual interactions.It also weights heads using visual-sink query tokens as proxies for grounding-relevant correspondences.Source: §1 · pp. 2–3
  • Overlap- and distance-aware patch labels better match click behavior, while two-step zoom-in refines a coarse region on a crop without additional training.The approach is intended to correct offset errors on dense, high-resolution interfaces.Source: §1 · p. 3
  • 5.88% higher ScreenSpot-Pro accuracy than vanilla attention grounding (43.39% versus 37.51%) under the ablation training setting.With one-stage fine-tuning on 101k screenshots and no extra modules, GUI-AIMA-3B outperforms similarly sized models and remains competitive with substantially larger GUI grounding systems.Source: §1 · p. 3
  • GUI-AIMA is presented as an attention-based, coordinate-free framework that uses intrinsic MLLM attention without extra grounding modules.Its contributions include simplified attention aggregation, grounding-relevant head weighting, click-aligned patch labels, and zoom-in inference.Source: §1 · p. 3

2 Related Work

GUI grounding has moved from structured interface metadata toward screenshot-based MLLMs, including approaches that generate coordinates or boxes as text. Related coordinate-free work instead uses patch-level attention or adds a grounding module.

2 Related Work

  • Structured inputs such as HTML, visual-parser extractions, and generated captions have supplemented screenshots, while screenshot-only MLLMs support end-to-end localization across GUI environments.Accessibility metadata can be inaccessible or verbose and may miss visual cues such as layout and icons.Source: §2 · p. 3
  • Many screenshot-based MLLM methods generate click centers or bounding boxes as text, an indirect alignment that has required scaled GUI corpora and OCR pretraining.Recent work also explores GUI-specific grounding rewards in RL training and autonomous GUI exploration.Source: §2 · p. 4
  • Coordinate-free approaches include TAG’s patch-level attention maps and GUI-Actor’s embedding-based grounding head, which adds a module and warm-up phase.The passage notes TAG’s generalization is restricted by its backbone and heuristics for selecting text tokens and attention heads.Source: §2 · p. 4

3 Methodology

GUI-AIMA supervises intrinsic MLLM attention with patch-wise grounding signals, using an anchor token and adaptive head weighting to produce coordinate-free GUI grounding. Its training and inference design adds overlap- and distance-aware patch labels, zoom-in refinement, and efficient extraction of grounding-relevant attention rows.

3 Methodology

  • GUI-AIMA uses intrinsic multi-head attention as patch-level grounding cues, adding a representative token and adaptive head-wise reweighting to aggregate them.The framework directly supervises attention maps rather than coordinate tokens or additional grounding modules.Source: §3 · p. 4

3.1 Intrinsic Attention for GUI Grounding with an Anchor Token

  • The <ANCHOR> token attends to preceding visual and query tokens, implicitly summarizing instruction context and avoiding explicit token-wise aggregation.Its patch-level attention vectors are aggregated across layers and heads with learned weights.Source: §3.1 · pp. 5–6

3.2 Attention Head Weighting using Visual-sink Query Tokens

  • Visual-sink query tokens are selected using hidden-state similarity to visual patches, then used to weight heads by query-to-visual attention mass.A shared token set across layers provides better stability and performance than layer-wise selection; the similarity signal is head-agnostic.Source: §3.2 · pp. 7–8

3.3 Training and Inference Design

  • Patch labels mark patches overlapping the ground-truth box and weight positives by overlap and distance to its center; KL divergence supervises the predicted patch distribution.For high-resolution screenshots, two-step zoom-in refines a coarse prediction on a crop without additional training, while selective attention extraction reduces overhead to roughly O((|Q| + 1) · |V|).Source: §3.3 · pp. 8–9

4 Experiments

Experiments evaluate GUI-AIMA’s benchmark performance, component ablations, and two-step zoom-in inference. Results report strong performance across GUI grounding benchmarks and improvements from selected design choices and zoom-in analysis.

4.1 Implementation Details.

  • GUI-AIMA-3B uses Qwen2.5-VL-3B-Instruct [2] and is trained for one epoch with all parameters unfrozen; its training set totals 509k instructions after adding 250k GroundCUA instructions.The paper evaluates five GUI grounding benchmarks and uses a 2× zoom-in ratio in the second inference pass.Source: §4.1 · p. 9

4.2 Main Results: Grounding Performance

  • 61.5% ScreenSpot-Pro accuracy and 68.1% OSWorld-G accuracy are reported for GUI-AIMA-3B with zoom-in, alongside strong results across other benchmarks.GUI-AIMA-3B reaches 53.8% on ScreenSpot-Pro before zoom-in; the paper also reports strong performance on ScreenSpot-v2 and MMBench-GUI-L2.Source: Table 1 · p. 10, §4.2 · p. 11, Table 2 · p. 11

4.3 Ablations

  • Simplified attention grounding with uniform weighting improves over GUI-Actor by 1.50% on ScreenSpot-v2 and 4.24% on ScreenSpot-Pro.The ablations compare coordinate-free methods trained on 45k examples; attention grounding converges faster than GUI-Actor on ScreenSpot-Pro.Source: §4.3 · p. 12, Table 4 · p. 13
  • Head-weighting variants without visual-sink query tokens perform worse on ScreenSpot-v2 than uniform weighting and show no ScreenSpot-Pro improvement.The passage attributes bias to visual-unrelated tokens in one variant and premature anchor-token grounding in the other.Source: §4.3 · p. 13
  • Global top-1 visual-sink query selection improves ScreenSpot-Pro by 1.90% over uniform weighting, while shared query tokens across layers give more balanced text and icon results.Layer-wise selection performs slightly better on icon tasks; global top-1 is best overall among the tested settings.Source: §4.3 · p. 13
  • Down-weighting partial-positive and distant patches adds 1.26% on ScreenSpot-Pro, and the Gaussian-labeling sensitivity analysis supports α = 0.8.Cross-model tests report consistent improvement over InternVL3.5 backbones; GUI-AIMA also converges fastest in the reported training curves and is more memory-consuming than coordinate-based autoregressive grounding.Source: Figure 5 · p. 13, Table 5 · p. 13, §4.3 · p. 14

4.4 Analysis of Two-step Inference with Zoom-in

  • A crop-only second-step setup corrects 57 offset errors, while 2× zoom-in gives the best performance and reduces Relax@3 and Relax@5 errors by up to 31% and 32%.Across zoom factors from 1.5× to 4×, slight deviations are especially reduced; two-step performance is reported as not very sensitive to the zoom factor.Source: Table 8 · p. 14, §4.4 · p. 15

5 Conclusion

The conclusion presents GUI-AIMA as a lightweight, scalable coordinate-free grounding framework. It reports strong 3B-scale performance and identifies broader visual grounding and complex interactions as future directions.

5 Conclusion

  • GUI-AIMA directly supervises intrinsic multi-head self-attention with patch-wise grounding signals for coordinate-free GUI grounding.The framework combines anchored attention, query-aware head weighting, click-aligned patch labeling, and plug-and-play two-step zoom-in inference without extra grounding modules.Source: §5 · p. 15

Supplementary Material

The supplementary material contains additional experiments, analyses, implementation details, visualizations, and failure cases.

Supplementary Material

  • Supplementary sections cover extra experiments and ablations, attention-head and visual-sink-query analyses, implementation details, visualizations, and failure cases.The additional experiments include running time and ablations on a larger training set.Source: Supplementary Material · p. 20

A Extra Results

The extra-results section points to further benchmark results and runtime comparisons in the supplementary material.

A Extra Results

  • Supplementary Tables 9 and 10 report ScreenSpot-v1 results and training-time and inference-time comparisons between GUI-AIMA-3B and GUI-Actor-3B.Source: A Extra Results · p. 20

B Extra Ablations

Additional ablations found that GUI-AIMA benefits from a multi-region verifier to the same extent as GUI-Actor, while variants on a larger ablation set yielded consistent results.

B Extra Ablations

  • GUI-AIMA and GUI-Actor benefit to the same extent from GUI-Actor’s multi-region verifier.Additional ablations on the 85k GUI-AIMA-lite training set were consistent with results from the larger ablation training set.Source: B Extra Ablations · p. 20

C Attention Head Contribution in GUI Grounding

Grounding-relevant attention heads receive higher learned weights, particularly in mid-to-late layers. Keeping the top-weighted heads largely preserves accuracy, whereas keeping the bottom-weighted heads substantially reduces it.

C Attention Head Contribution in GUI Grounding

  • Higher head weights concentrate in mid-to-late layers, with dominant heads in mid layers and broader moderately high weights in late layers.Examples include layers 22, 28, 33, and 34, with head 25-5 dominant and layer 34 showing a broader set of moderately high-weight heads.Source: C Attention Head Contribution in GUI… · p. 20
  • 91.0% on ScreenSpot-v2 and 53.5% on ScreenSpot-Pro with the top 30% of heads, versus 76.6% and 38.1% with the bottom 30%.The pruning results support learned head weights as indicators of grounding-relevant attention heads.Source: C Attention Head Contribution in GUI… · p. 20

D Analysis for Visual-sink Query Token

The visual-sink query token selected from hidden states performs better than a global-pattern alternative and has a head-weight vector similar to the anchor token’s. Supplementary figures and tables document head weights and ablation settings.

D Analysis for Visual-sink Query Token

  • The hidden-state visual-sink query achieves better performance and has a head-weight vector most similar to the anchor token’s.This supports using the top-1 visual-sink query to avoid bias from the randomly initialized anchor token.Source: D Analysis for Visual-sink Query Token · pp. 21–22

E Baselines Details

  • Most baseline results come from original papers; GUI-G2-3B is an exception, and UI-Vision and MMBench-GUI-L2 baseline results are sourced from UI-Venus-1.5 and GroundNext.Source: E Baselines Details · p. 22
  • Figure 6 presents layer-wise mean head weights and per-layer, per-head weights averaged across ScreenSpot-Pro, ScreenSpot-v2, and OSWorld-G.Source: Figure 6 · p. 23, E Baselines Details · p. 23
  • The top-1 visual-query token consistently has the largest cosine similarity with the anchor token’s multi-head attention vector.Figure 7 compares the anchor-derived vector with vectors computed between the anchor and each text token.Source: Figure 7 · p. 23

F Extra details of <ANCHOR> Implementations

Extra anchor tokens add no performance gain for single-region grounding, while the supplementary examples include both grounding visualizations and failure cases.

F Extra details of <ANCHOR> Implementations

  • Expanding one anchor to multiple anchor tokens adds redundancy without performance gains in single-region grounding; disentangled anchors for multi-region tasks are left for future work.Source: F Extra details of <ANCHOR>… · p. 24

G GUI Grounding Examples of GUI-AIMA

H Failure Cases

  • Failures include missing the true icon in a dense tool area, clicking a non-clickable same-function button, and selecting actions in the wrong order.In Word on macOS, the action should click “Picture” before “Select Picture.”Source: H Failure Cases · p. 24
Loading 2511.00810v4…