Source-linked AI summary
GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents
Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, Si Qin, Lars Liden, Qingwei Lin, Huan Zhang, Tong Zhang, Jianbing Zhang, Dongmei Zhang, Jianfeng Gao
TL;DR
GUI visual grounding requires localizing instruction-relevant screen regions, but coordinate-generation methods have weak spatial-semantic alignment, ambiguous supervision, and granularity mismatches. GUI-Actor replaces numeric coordinate generation with an <ACTOR>-based attention head, multi-patch supervision, and a verifier; it outperforms prior methods across benchmarks and generalizes better to unseen layouts and resolutions.
Problem
Visual GUI grounding is difficult because coordinate-generation methods weaken spatial-semantic alignment, mishandle ambiguous valid targets, and mismatch dense coordinates with patch-level visual features.
Method
GUI-Actor uses an attention-based action head to align a dedicated <ACTOR> token with relevant visual patches, trains with multi-patch supervision, and verifies candidate regions.
Results
GUI-Actor outperforms state-of-the-art methods across GUI grounding benchmarks and shows stronger generalization to unseen layouts and screen resolutions.
Takeaways & Limitations
Freezing the backbone while training only the newly introduced grounding components can produce substantial improvements and, with a verifier, rival fully fine-tuned coordinate-generation models.
Takeaways & Limitations
The fixed 28 × 28-pixel patch size of the backbone can insufficiently represent very small interface elements, especially in high-precision tasks such as CAD.
Abstract
from arXiv · showhide
One of the principal challenges in building VLM-powered GUI agents is visual grounding, i.e., localizing the appropriate screen region for action execution based on both the visual content and the textual plans. Most existing work formulates this as a text-based coordinate generation task. However, these approaches suffer from several limitations: weak spatial-semantic alignment, inability to handle ambiguous supervision targets, and a mismatch between the dense nature of screen coordinates and the coarse, patch-level granularity of visual features extracted by models like Vision Transformers. In this paper, we propose GUI-Actor, a VLM-based method for coordinate-free GUI grounding. At its core, GUI-Actor introduces an attention-based action head that learns to align a dedicated <ACTOR> token with all relevant visual patch tokens, enabling the model to propose one or more action regions in a single forward pass. In line with this, we further design a grounding verifier to evaluate and select the most plausible action region from the candidates proposed for action execution. Extensive experiments show that GUI-Actor outperforms prior state-of-the-art methods on multiple GUI action grounding benchmarks, with improved generalization to unseen screen resolutions and layouts. Notably, GUI-Actor-7B even surpasses UI-TARS-72B (38.1) on ScreenSpot-Pro, achieving scores of 40.7 with Qwen2-VL and 44.6 with Qwen2.5-VL as backbones. Furthermore, by incorporating the verifier, we find that fine-tuning only the newly introduced action head (~100M parameters for 7B model) while keeping the VLM backbone frozen is sufficient to achieve performance comparable to previous state-of-the-art models, highlighting that GUI-Actor can endow the underlying VLM with effective grounding capabilities without compromising its general-purpose strengths.
1 Introduction
GUI-Actor reframes GUI visual grounding as coordinate-free localization, using attention over visual patches rather than generating numeric coordinates. It addresses weak spatial-semantic alignment, ambiguous point supervision, and the mismatch between continuous action coordinates and patch-level visual features.
- Motivation: Visual GUI agents must map natural-language plans to screen regions while operating directly on rendered screenshots when structured metadata is unavailable or unreliable.This challenge arises across desktop, mobile, and web interfaces.
- Limitations of prior methods: Coordinate-generation methods output screen positions as text, weakening spatial-semantic alignment and requiring dense actions to be inferred from coarse visual patches.They lack explicit spatial inductive bias and direct supervision connecting visual features to action locations.
- Limitations of prior methods: Bounding-box methods still encode regions as raw coordinate strings detached from visual features unless they add mechanisms such as ROI pooling or spatial attention.Thus, they do not fully bridge linguistic intent and grounded visual action.
- GUI-Actor: GUI-Actor uses a dedicated <ACTOR> token and an attention-based action head to align instructions with relevant visual patch tokens and identify actionable regions.The attention map can produce one or more candidate regions in a single forward pass.
- GUI-Actor: Multi-patch supervision labels every patch overlapping the ground-truth box as positive, allowing valid spatial ambiguity while grounding actions at the backbone’s native resolution.The method therefore avoids over-penalizing reasonable clicks and supports robustness across screen sizes, resolutions, and layouts.
- GUI-Actor: GUI-Actor combines coordinate-free grounding with a grounding verifier that selects the most plausible action region from generated candidates.The contribution summary reports improved benchmark performance and robustness to unseen screen sizes and resolutions.
2 Related Work
GUI visual grounding localizes instruction-relevant regions in screenshots, a task made difficult by dense GUI semantics and regular layouts. GUI-Actor offers a coordinate-free alternative that attends directly to image patches instead of generating coordinate strings.
- GUI agents: GUI agents increasingly operate on raw screenshots because HTML, accessibility trees, and other structured representations are brittle or inconsistent across platforms.This vision-centric setting uses mouse and keyboard inputs to interact with interfaces.
- GUI Visual Grounding: GUI visual grounding locates the target interaction region given a GUI screenshot and a natural-language instruction.The task differs from natural-image grounding because GUI layouts have high semantic density and structural regularity.
- GUI Visual Grounding: Coordinate-based grounding methods generate point positions as output language tokens but exhibit weak spatial inductive bias, ambiguous point supervision, and resolution mismatches.These limitations motivate alternatives to text-based coordinate prediction.
- GUI-Actor: GUI-Actor introduces an <ACTOR> token that attends directly to relevant image patches through an attention-based action head.This provides a coordinate-free grounding framework for GUI agents.
3 The Design of GUI-Actor
GUI-Actor replaces coordinate generation with an <ACTOR>-anchored attention mechanism that grounds instructions directly onto visual patches. Multi-patch supervision provides spatially dense labels for actionable regions.
- <ACTOR> Token as a Contextual Anchor: GUI-Actor replaces coordinate spans with <ACTOR_START>, <ACTOR>, and <ACTOR_END> tokens for coordinate-free grounding.The final-layer <ACTOR> representation serves as a contextual anchor for action attention.
- Attention-Based Action Head: The action head computes attention from the <ACTOR> token over visual patch features to identify the target action region.A self-attention layer first contextualizes spatially related patches before attention scores are computed.
- Attention-Based Action Head: GUI-Actor projects the contextualized <ACTOR> and patch representations into a shared embedding space before calculating patch-level attention.Separate MLPs produce the shared representations used for attention scoring.
- Spatial-Aware Multi-Patch Supervision: Multi-patch supervision labels every patch overlapping the ground-truth bounding box positive and all other patches negative.This provides dense spatially structured learning signals across the full actionable element rather than relying on one click point.
- Training Objective: Training combines next-token prediction loss with action attention loss.The paper explicitly defines the joint objective as L = LNTP + LAction_Attn.
4 Grounding Verifier
GUI-Actor generates multiple candidate regions through attention and uses a lightweight grounding verifier to select the candidate that best matches the instruction. At inference, attention over visual patches supplies the spatial activation map used for action execution.
- Candidate Generation: Attention-based grounding produces multiple candidate action regions in one forward pass without additional inference cost.A verifier can then refine the decision instead of relying only on the top-scoring patch.
- Grounding Verifier: The grounding verifier evaluates marked candidate regions against the instruction and predicts whether each region fulfills the task intent.It is a lightweight VLM module that operates on the instruction and a screenshot containing a visual marker.
- Data & Training: Verifier training uses OS-Atlas triplets spanning desktop, mobile, and web domains, with positive and negative marked-point examples.Negatives come from incorrect bounding boxes or randomly sampled points outside the target region.
- Data & Training: The verifier is fine-tuned with supervised learning to generate a True or False token from the marked image and instruction.The training objective is cross-entropy loss.
- Inference: At inference, GUI-Actor extracts the <ACTOR> hidden state and computes attention over all visual patches to form a spatial activation map.The map identifies the screen region for executing the generated action.
- Candidate Selection: Candidates are checked in descending attention order, and the first candidate above the confidence threshold is returned.The selection procedure uses a score threshold such as s(I, x) > γ.
5 Experiments
GUI-Actor is evaluated against comparable GUI-grounding baselines on ScreenSpot, ScreenSpot-v2, and the higher-resolution, domain-shifted ScreenSpot-Pro benchmark. Across these experiments, it shows stronger benchmark performance, robust out-of-distribution behavior, improved sample efficiency, benefits from verifier-based refinement, and effective lightweight training with a frozen backbone.
- Main Results: GUI-Actor-2B and GUI-Actor-7B consistently outperform existing state-of-the-art methods across ScreenSpot-Pro, ScreenSpot, and ScreenSpot-v2.The 2B model also surpasses many competing 7B models; UI-TARS-7B is noted as an exception with substantially more training data and a more extensive training pipeline.
- Robust Out-of-Distribution Generalization: +9.0 and +5.0 points over UI-TARS on ScreenSpot-Pro are achieved by GUI-Actor-2B and GUI-Actor-7B, respectively, on the higher-resolution, domain-shifted benchmark.The authors attribute this robustness to explicit spatial-semantic alignment through the attention-based action head and direct grounding in visual regions.
- Improved Sample Efficiency: ∼60% of the training data is sufficient for GUI-Actor to reach final accuracy on ScreenSpot and ScreenSpot-v2, while Aguvis baselines plateau after 80–90%.The reported efficiency is associated with native patch-resolution grounding and multi-patch supervision.
- Enabling backbone VLM grounding on GUIs without sacrificing general-purpose strengths: GUI-Actor-LiteTrain improves over the unmodified backbone and, with the verifier, rivals fully fine-tuned coordinate-generation models while freezing the VLM backbone.Only newly introduced action-head and special-token parameters are updated in this setup.
- Boosting Performance via Grounding Verifier: The grounding verifier consistently improves grounding accuracy, boosting GUI-Actor-7B by nearly 4 points and GUI-Actor-7B-LiteTrain by 13 points on ScreenSpot-Pro.The verifier selects among diverse region proposals generated in a single forward pass.
- Multi-Region Prediction Without Extra Inference Cost: GUI-Actor can generate multiple candidate action regions in one forward pass without extra inference cost, with a substantial Hit@1-to-Hit@3 improvement.The evaluation uses Hit@k, where k is the number of top-ranked predictions considered.
6 Conclusion
GUI-Actor replaces text-based coordinate generation with coordinate-free grounding through a dedicated <ACTOR> token that attends to target visual patches. Its multi-patch supervision, candidate generation, and verifier support robust localization across layouts and resolutions.
- GUI-Actor introduces a dedicated <ACTOR> token that attends to target visual patches to directly localize GUI elements.
- The attention mechanism aligns spatial visual features with instruction semantics and supports bounding-box–based multi-patch supervision.
- GUI-Actor proposes multiple candidate regions in one pass and uses a lightweight verifier to select the most plausible target.
- Experiments show improved performance over state-of-the-art methods and stronger generalization to unseen layouts and screen resolutions.
A Limitations
GUI-Actor uses attention maps and patch-level supervision to represent actionable regions, while visualization code converts attention scores into image overlays. A key limitation is the backbone’s fixed patch size for very small interface elements.
- GUI-Actor supervises attention over entire target regions rather than relying on a single ambiguous click point.
- Ground-truth bounding boxes are converted into binary masks over the W × H image patch grid, labeling covered patches positive and others negative.
- The action-head loss uses KL divergence between predicted attention distributions and a normalized distribution derived from the binary mask.
- Figure 5 shows attention maps largely overlapping ground-truth areas in ScreenSpot and ScreenSpot-Pro examples.
- Attention visualizations normalize and reshape scores, apply a colormap, resize the map, and blend it with the original image.
D Training Datasets used for GUI-Actor
GUI-Actor’s training data comes from several publicly available GUI datasets, with dataset statistics summarized in Table 7. Wave-UI samples overlapping downstream test sets are excluded.
- Training data is compiled from several publicly available, high-quality GUI datasets.
- Table 7 provides summary statistics for the training datasets used by GUI-Actor.
- Samples from Wave-UI that overlap with downstream task test sets are excluded.
E GUI Visual grounding Benchmarks
GUI visual grounding benchmarks pair screenshots with natural-language instructions and require locating the corresponding interface element. ScreenSpot spans multiple platforms, while ScreenSpot-Pro targets high-resolution professional scenarios and estimates generalization.
- ScreenSpot contains 1,272 single-step instructions paired with target elements across mobile, desktop, and web environments.
- ScreenSpot-v2 corrects annotation errors and ambiguous instructions while keeping the total sample count unchanged.
- ScreenSpot-Pro contains 1,581 expert-annotated tasks across 23 professional applications and three operating systems.
- ScreenSpot-Pro uses higher-resolution screenshots and larger domain gaps, including industrial software and multi-window interfaces.
- ScreenSpot-Pro performance is viewed as a practical estimate of GUI visual-grounding generalization.
F.1 Data Construction
The grounding verifier is trained on balanced positive and negative examples, then scores candidate patches to select an action position. Neighbor clustering and weighted centers refine localization beyond individual patch centers.
- Verifier Data Construction: Positive examples mark the ground-truth box center, while negative examples mark incorrect points for the same image and query.The verifier labels each marked image-query pair as True or False.
- Verifier Data Construction: 730K examples form a balanced verifier dataset, equally split between positive and negative cases.The dataset is constructed from OS-Atlas triplets spanning desktop, mobile, and web domains.
- Patch Selection: The verifier scores candidate patch centers by normalizing the probabilities of the True and False tokens.For each candidate, the normalized True probability defines the selection score.
- Patch Selection: Clustering 4-connected neighboring patches and computing verifier-weighted centers produces candidate points between adjacent patches.This refinement improves localization without changing the base model’s patch size.
- Patch Selection: Up to M = 20 top-scoring patches are retained after filtering attention weights below 20% of the maximum, then clustered into candidate positions.The highest-scoring candidate is selected, and candidates are evaluated using cropped image regions centered on each position.
- Patch Selection: Candidate evaluation can stop early when a confidence score exceeds the task-specific threshold of 0.95 or 0.8.The threshold is 0.95 for ScreenSpot-Pro and 0.8 for ScreenSpot and ScreenSpot-v2.
G.1 Enhancing Generation with Verifier Self-Aggregation
Verifier Self-Aggregation averages verifier scores across crops at multiple scales to improve robustness on ScreenSpot-Pro. GUI-Actor also achieves higher grounding accuracy than AGUVIS with substantially lower inference computation.
- Verifier Self-Aggregation: Verifier Self-Aggregation averages verifier scores from 1200- and 1400-pixel crops to improve ScreenSpot-Pro performance.The method balances local detail from smaller crops with broader context from larger crops.
- Comparison with AGUVIS: AGUVIS uses one deterministic output plus 20 sampled candidate points, incurring substantial inference overhead.The additional candidates are sampled at temperature 0.7 after a temperature-0.0 deterministic output.
- Comparison with AGUVIS: GUI-Actor requires about 5% of AGUVIS’s computation while achieving higher accuracy on ScreenSpot and ScreenSpot-v2 and significantly outperforming AGUVIS on ScreenSpot-Pro.GUI-Actor proposes multiple candidate points from its attention map in a single pass.
H Online Benchmark Evaluation on OSWorld
GUI-Actor is evaluated on 49 live Windows tasks in OSWorld-W, where GPT-4o plans actions and the grounding model converts them into executable GUI actions. It achieves the highest task success rate among the compared grounding models and generalizes to unseen scenarios.
- Benchmark Setup: 49 Windows-specific OSWorld-W tasks cover multi-step office and multi-application scenarios with handcrafted verification scripts.OSWorld-W is a curated subset used for quick validation in realistic desktop environments.
- Evaluation Pipeline: GPT-4o generates natural-language plans, which the underlying grounding model converts into concrete coordinate-based or coordinate-free actions.The grounding model determines the executable action at each step.
- Results: 12.2% task success makes GUI-Actor the top performer, versus 10.2% for OmniAgent and NAVI and 4.0% for Aguvis-7B.The comparison is conducted on the OSWorld-W subset.
- Results: GUI-Actor generalizes to OSWorld-W despite no training exposure to its tasks.The authors report more accurate and reliable grounding than the compared alternatives in these unseen scenarios.