Source-linked AI summary
Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning
Hai-Ming Xu, Qi Chen, Lei Wang, Lingqiao Liu
TL;DR
GUI grounding—identifying interface components from an image and query—remains challenging even for MLLMs with strong GUI understanding. TAG uses selected attention patterns from a pretrained MLLM to ground elements without fine-tuning. The authors report accurate grounding compared with tuning-based methods and particular improvement in text localization.
Problem
Precise localization of GUI elements from images and text queries remains challenging despite MLLMs’ strong GUI-image understanding.
Source: §1 · p. 1Method
TAG selects and aggregates attention patterns from a pretrained MLLM, using relevant query or generated-response tokens to ground GUI elements without fine-tuning.
Source: §1 · p. 1Results
The authors report accurate GUI grounding compared with tuning-based methods and improved text localization using pretrained-model attention.
Source: §1 · p. 1Takeaways & Limitations
On MiniCPM-Llama3-V 2.5, TAG matches or exceeds traditional methods particularly in text localization, suggesting reduced need for resource-intensive fine-tuning within the paper’s demonstrated scope.
Source: §5 · p. 7TAG’s effectiveness may be limited by biases or insufficiently diverse pretraining data in the pretrained models it uses.
Source: §5 · p. 7
Abstract
from arXiv · showhide
Recent advancements in Multimodal Large Language Models (MLLMs) have generated significant interest in their ability to autonomously interact with and interpret Graphical User Interfaces (GUIs). A major challenge in these systems is grounding-accurately identifying critical GUI components such as text or icons based on a GUI image and a corresponding text query. Traditionally, this task has relied on fine-tuning MLLMs with specialized training data to predict component locations directly. However, in this paper, we propose a novel Tuning-free Attention-driven Grounding (TAG) method that leverages the inherent attention patterns in pretrained MLLMs to accomplish this task without the need for additional fine-tuning. Our method involves identifying and aggregating attention maps from specific tokens within a carefully constructed query prompt. Applied to MiniCPM-Llama3-V 2.5, a state-of-the-art MLLM, our tuning-free approach achieves performance comparable to tuning-based methods, with notable success in text localization. Additionally, we demonstrate that our attention map-based grounding technique significantly outperforms direct localization predictions from MiniCPM-Llama3-V 2.5, highlighting the potential of using attention maps from pretrained MLLMs and paving the way for future innovations in this domain.
1 Introduction
GUI grounding—locating text and icons from an image and query—is essential for automated GUI operation, yet precise localization remains challenging despite MLLMs’ strong GUI-image understanding. TAG uses pretrained-model attention rather than fine-tuning, and the paper reports accurate grounding with particular success in text localization.
1 Introduction
- GUI grounding locates key components such as text and icons from GUI images and queries, a task fundamental to automated GUI operation.The introduction identifies precise grounding as challenging even though MLLMs can interpret GUI images.Source: §1 · p. 1
- Many current solutions fine-tune MLLMs on specialized datasets to predict GUI-element locations directly (Hong et al. 2023; Cheng et al. 2024).TAG instead leverages attention patterns in a pretrained MLLM.Source: §1 · p. 1
- TAG selects and aggregates attention patterns from MiniCPM-Llama3-V 2.5 to ground GUI elements without additional fine-tuning.It identifies tokens in the user query or model-generated response and propagates their attention back to the image plane, filtering irrelevant heads.Source: §1 · p. 1
- The paper reports accurate grounding compared with tuning-based methods and improved text localization using pretrained-model attention.The authors describe this as evidence of the potential of inherent model capabilities for GUI automation.Source: §1 · p. 1
2 Related Works
Prior GUI-agent work uses MLLMs across desktop and mobile applications, but accurate grounding remains a challenge. Existing approaches often rely on specialized fine-tuning, whereas this paper proposes grounding through pretrained MLLMs’ inherent attention mechanisms.
2 Related Works
- GUI agents use MLLMs for tasks across desktop and mobile platforms, including routine desktop automation and interactive mobile assistance.The cited examples span multiple applications and platforms.Source: §2 · p. 1
- Figure 1 contrasts MiniCPMV2.5’s strong GUI-image understanding with poor element localization and shows attention-driven grounding enhancing localization without fine-tuning.The caption describes the proposed approach as leveraging the model’s inherent attention.Source: Figure 1 · p. 2
- GUI grounding identifies interface elements accurately, but traditional methods typically fine-tune on detailed, annotated datasets (Cheng et al. 2024; Chen et al. 2024; Hong et al. 2023; Fan et al. 2024).The discussion notes that these methods can face scalability issues and overfitting.Source: §2 · p. 2
- The paper proposes associating text queries with visual elements through pretrained MLLMs’ inherent attention mechanisms.The related-work discussion describes this as a tuning-free grounding approach.Source: §2 · p. 2
3 Our Method
The method addresses GUI localization by selecting and combining pretrained-model attention. It prompts for relevant descriptive tokens, propagates their attention through visual query tokens to image patches, and selects useful attention heads before producing a location.
3.1 Preliminary
- MiniCPMV2.5 compresses high-resolution image patches into visual query tokens, which Llama3 processes alongside text; GUI grounding still requires precise element localization.The model supports images up to 1344×1344 pixels, while prior state-of-the-art methods typically fine-tune MLLMs for direct localization (Hong et al. 2023; Cheng et al. 2024).Source: §3.1 · p. 2
3.2 Overview of Our Method
- TAG combines adaptive text-token selection, aggregation of self- and cross-attention for localization, and selection of high-quality heads from Llama3’s 1024 attention heads.These are the method’s three stated components.Source: §3.2 · p. 3
3.3 Adaptive Text Token Selection
- Because queries may only indirectly imply a target, TAG prompts the model to describe relevant content and uses attention from those descriptive tokens to visual tokens.The method is motivated by complex queries that may not align easily with dynamic UI changes.Source: §3.3 · p. 3
3.4 Attention-driven GUI Grounding
- TAG propagates selected text-token attention through visual query tokens to image patches, averages token maps, then thresholds regions and predicts the center of the highest-relevance region.The text-to-query-token attention is aggregated across heads; cross-attention maps visual query tokens to image patches.Source: §3.4 · pp. 3–4
3.5 Self-Attention Head Selection
- Naively averaging all attention heads can map text to the wrong region; TAG retains the top-K heads separately for each selected token.Higher average attention between a token and visual query tokens is more likely to map that token to its intended region.Source: §3.5 · p. 4
4 Experiments
The experiments compare TAG with foundation and fine-tuned models across optical-character grounding, GUI-element grounding, and GUI-agent evaluation, then examine components and parameter choices. Results show strong grounding performance, including improved text localization and comparable Mind2Web agent accuracy to the best tuning-based approach.
4.1 Task1: Optical Character Grounding
- The OCG benchmark uses OCR from homepage screenshots of 104 Mind2Web test-set websites and crops them into sub-images spanning 10 aspect ratios.Only OCR bounding boxes entirely within each sub-image are retained for evaluation.Source: §4.1 · p. 4, Figure 4 · p. 5
- 84.5% average accuracy across 10 aspect ratios: TAG outperforms MiniCPMV2.5 by 36.4% and SeeClick by 24.3% on the OCG benchmark.MiniCPMV2.5 ranges from 80.2% at 4:3 to 13.6% at 9:21.Source: Table 1 · p. 5, §4.1 · p. 6
4.3 Task3: GUI Agent Evaluation
- TAG improves MiniCPMV2.5 element accuracy across Mind2Web settings and achieves average accuracy comparable to the best tuning-based approach.The evaluation uses GUI images and counts a prediction as correct when its coordinate falls within the target element’s bounding box.Source: Table 3 · p. 6, §4.3 · p. 6
4.2 Task2: GUI Element Grounding
- 54.8% average accuracy: TAG achieves the highest ScreenSpot average, surpassing GUI-specific SFT models and reaching 88.3%, 82.5%, and 70.9% text accuracy on mobile, desktop, and web.The results suggest adaptive selection of text tokens from generated element descriptions can outperform using query text directly.Source: Table 2 · p. 5, §4.2 · p. 6
4.4 Ablation Study
- In component ablations, the full model performs best across metrics, reaching 88.3% Mobile Text and 28.6% Desktop Icon/Widget accuracy.Adding attention-driven grounding raises Mobile Text accuracy from 40.3% to 71.8%; adaptive text-token selection further improves Mobile Text and Icon/Widget results.Source: Table 4 · p. 6, §4.4 · p. 7
4.5 More Discussions
- Filtering self-attention weights works best at K = 10 for both aspect ratios; K = 1 and K = 1024 reduce accuracy.The paper uses K = 10 in all experiments.Source: §4.5 · p. 7, Figure 6 · p. 7
- The grounding threshold peaks at δ = 0.5; lower values include too many attended regions, and performance diminishes above the peak.The paper uses δ = 0.5 across all datasets.Source: §4.5 · p. 7, Figure 7 · p. 7
- On OCG at 4:3, TAG improves Qwen-VL-Chat accuracy from 2.7% to 10.2%, showing the mechanism also applies to another foundation MLLM.The paper presents this as evidence of broad applicability across foundation MLLMs.Source: §4.5 · p. 7
5 Conclusion
TAG uses pretrained MLLM attention for GUI grounding without additional fine-tuning; on MiniCPM-Llama3-V 2.5, it matches or exceeds traditional methods, particularly for text localization. Its effectiveness remains dependent on the capabilities and quality of the pretrained model.
5 Conclusion
- TAG grounds GUI elements by using pretrained MLLM attention without additional fine-tuning.The conclusion reports this approach applied to MiniCPM-Llama3-V 2.5.Source: §5 · p. 7
- TAG matches or exceeds traditional methods, particularly in text localization tasks.The authors describe this as evidence that built-in model capabilities can support grounding effectively.Source: §5 · p. 7
- TAG relies on pretrained model capabilities and quality, so biases or insufficiently diverse pretraining data could limit accuracy and generalization.Expanding pretraining datasets is described as future work beyond this paper’s scope.Source: §5 · p. 7
6 Supplementary Materials
The supplementary material provides additional method, dataset, and evaluation details, including qualitative grounding examples, inference-time analysis, and a VisualWebBench comparison.
6 Supplementary Materials
- The supplement collects full query outputs, MiniCPMV2.5 grounding details, OCG statistics, prompt and action-history details, efficiency analysis, and VisualWebBench results.These are the listed contents of the supplementary material.Source: §6 · p. 8
6.1 Full Output of Query in Figure 1 of Main Paper
- In six examples, MiniCPMV2.5 understood the text but failed to ground it accurately, while TAG grounded the text precisely.The examples compare optical-character grounding after MiniCPMV2.5 OCR generated the query texts.Source: §6.1 · p. 8
6.2 MiniCPMV2.5 for Grounding
- MiniCPMV2.5 tends to output locations in box format, so the prompt specifies that format for consistent parsing; coordinates rescaled to [0, 1000] are mapped back to image space, and final grounding uses the transformed bounding-box center.Source: §6.2 · p. 8
6.3 OCG Dataset
- OCG derives from the Mind2Web test set and contains 104 website screenshots with Azure Vision OCR text and bounding boxes.Screenshots are cropped to accommodate aspect ratios while retaining text elements and boxes fully within each crop; element counts vary by aspect ratio.Source: §6.3 · p. 8
6.4 Prompt Template for MiniCPMV2.5 on Mind2Web Dataset
- The MiniCPMV2.5 prompt template is almost the same as TAG’s, supporting a fair comparison.Source: §6.4 · p. 9
6.5 Action History for Mind2Web Dataset
- For Mind2Web’s multi-step GUI tasks, the prompt includes the latest four ground-truth action steps to show how the current state was reached.Source: §6.5 · p. 9
6.6 Inference Efficiency Analysis for TAG
- TAG adds attention-selection and product operations but avoids box-output tokens, potentially lowering inference time cost.The comparison uses the ScreenSpot desktop test split and an NVIDIA RTX 4090 GPU.Source: §6.6 · p. 9, Table 6 · p. 9
6.7 Evaluation on VisualWebBench
- On VisualWebBench Element Grounding, TAG outperforms all evaluated models.The task uses images marked with boxes and tags on seven potential elements for multi-choice prediction; MiniCPMV2.5 is evaluated with both multi-choice grounding and direct box prediction.Source: §6.7 · p. 9, Table 7 · p. 9