Source-linked AI summary
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, Zhiyong Wu
TL;DR
Existing GUI agents often rely on structured text that may be inaccessible, verbose, and visually incomplete. SeeClick uses screenshot-only visual GUI grounding, supported by curated grounding data and the ScreenSpot benchmark; it improves over LVLM baselines and grounding gains consistently correlate with better downstream task performance.
Problem
Structured GUI text can be inaccessible, verbose, visually incomplete, and varied across platforms, while existing LVLMs perform poorly on GUI grounding.
Method
SeeClick uses screenshot-only visual interaction, GUI grounding pre-training, automated web and mobile grounding-data curation, and the ScreenSpot benchmark.
Results
SeeClick significantly improves over LVLM baselines on ScreenSpot, and three GUI automation benchmarks consistently associate better GUI grounding with improved downstream performance.
Takeaways & Limitations
GUI grounding is a key capability for visual GUI agents and is directly correlated with performance on downstream agent tasks.
Takeaways & Limitations
SeeClick mainly supports clicking and typing, while multi-step mobile and computer tasks still require agent-specific training because of open-source LVLM limitations.
Abstract
from arXiv · showhide
Graphical User Interface (GUI) agents are designed to automate complex tasks on digital devices, such as smartphones and desktops. Most existing GUI agents interact with the environment through extracted structured data, which can be notably lengthy (e.g., HTML) and occasionally inaccessible (e.g., on desktops). To alleviate this issue, we propose a novel visual GUI agent -- SeeClick, which only relies on screenshots for task automation. In our preliminary study, we have discovered a key challenge in developing visual GUI agents: GUI grounding -- the capacity to accurately locate screen elements based on instructions. To tackle this challenge, we propose to enhance SeeClick with GUI grounding pre-training and devise a method to automate the curation of GUI grounding data. Along with the efforts above, we have also created ScreenSpot, the first realistic GUI grounding benchmark that encompasses mobile, desktop, and web environments. After pre-training, SeeClick demonstrates significant improvement in ScreenSpot over various baselines. Moreover, comprehensive evaluations on three widely used benchmarks consistently support our finding that advancements in GUI grounding directly correlate with enhanced performance in downstream GUI agent tasks. The model, data and code are available at https://github.com/njucckevin/SeeClick.
1 Introduction
SeeClick addresses limitations of structured-text GUI agents by using screenshots alone, with GUI grounding pre-training and the ScreenSpot benchmark supporting improved downstream task performance.
- Motivation: Structured text can be inaccessible, inefficiently verbose, visually incomplete, and difficult to standardize across GUI platforms.These limitations motivate an alternative to text-based GUI agents.
- GUI grounding: GUI grounding is the capacity to accurately locate screen elements from instructions and is identified as a foundational challenge for visual GUI agents.SeeClick addresses this challenge through GUI grounding pre-training and curated web and mobile grounding data.
- ScreenSpot: ScreenSpot contains more than 600 screenshots and 1200 instructions spanning iOS, Android, macOS, Windows, and webpages.The benchmark includes text elements as well as widgets and icons.
- Results: SeeClick surpasses the strong visual baseline Pix2Act while using merely 0.3% training data across mobile and web agent tasks.Evaluations on MiniWob, AITW, and Mind2Web consistently support a correlation between improved GUI grounding and downstream task performance.
- SeeClick: SeeClick performs clicking and typing across diverse GUI platforms using only interface screenshots.The visual approach avoids interaction with structured text and aims to adapt across platforms.
2 Related work
GUI navigation research has progressed from simplified web and mobile automation to LLM-centric and vision-based agents, with some visual methods still requiring GUI metadata.
- Autonomous GUI Navigation: Early GUI agents automated tasks in simplified web and mobile interfaces.These systems preceded the current LLM-centric agent paradigm.
- Autonomous GUI Navigation: LLM-centric agents became dominant as language models advanced, supporting web-task prompting and specialized agent training.Representative approaches use in-context learning and self-refinement.
- Vision-based GUI Navigation: Recent vision-based GUI navigation methods primarily use GPT-4V, while some also require GUI metadata as input.SeeClick instead customizes an open-source LVLM to operate across platforms without GUI metadata.
3 Approach
SeeClick is a visual GUI agent that performs interaction from screenshots by developing GUI grounding capabilities and pre-training on automatically curated web and mobile data. Its training combines GUI-specific grounding data with general vision-language instruction data to produce a 1M-example training set.
- 3 Approach: SeeClick performs clicking and typing directly from screenshots, targeting GUI grounding as the capacity to locate instructed screen elements.The approach avoids structured GUI text and frames grounding as predicting an element’s location from its screenshot and description.
- 3.1 GUI grounding for LVLMs: The grounding task trains an LVLM to model p(y|s, x), where s is an interface screenshot, x is an element description, and y is its bounding box or point.Coordinates are generated as natural-language numerical values rather than through an additional 1,000-token coordinate vocabulary.
- 3.2 Data Construction: Web data comes from approximately 300k Common Crawl webpages, using visible-text elements and elements with descriptive HTML title attributes.These two element types are automatically collected with their corresponding interactable regions for grounding training.
- 3.2 Data Construction: Mobile grounding data combines reversed widget-captioning examples with automatically collected RICO elements and instructions, totaling nearly 20k screenshots, 40k widgets, and 100k descriptions.The mobile data is intended to improve grounding generalization across diverse GUI contexts and also includes mobile UI summarization data.
- 3.3 Training Details: ScreenSpot evaluates grounding across diverse GUI environments and element types, while Table 1 reports comparisons among LVLMs after GUI grounding pre-training.The benchmark and results are introduced as evaluations of SeeClick’s grounding-oriented approach.
4 ScreenSpot: A Grounding Benchmark
ScreenSpot addresses limited and outdated GUI grounding evaluation by providing a realistic benchmark spanning mobile, desktop, and web interfaces. It emphasizes difficult localization of both text and icons or widgets using novel, annotated samples.
- Motivation: Earlier GUI grounding research was scant and largely confined to an Android dataset collected in 2017.The paper identifies this restriction as a research gap in evaluating grounding across modern GUI platforms.
- Benchmark Overview: ScreenSpot is an up-to-date, realistic benchmark for locating instructed screen elements across various GUI platforms.It is designed specifically to assess vision-language models’ GUI grounding ability.
- Various GUI platforms: The benchmark includes over 600 screenshots and 1200+ instructions from iOS, Android, macOS, Windows, and web platforms.Each instruction is paired with a corresponding actionable element.
- Icons/Widgets: ScreenSpot includes substantial numbers of icons and widgets in each GUI, making it more challenging than locating text alone.The benchmark’s element distribution is summarized in its platform and element-type statistics.
- Data Curation: Experienced annotators collected novel interfaces and labeled instructions with bounding boxes for actionable elements.Mobile and desktop samples use common apps and operations, while web samples cover development, shopping, forum, and tool sites from WebArena.
5 Experiments
Experiments evaluate SeeClick’s GUI grounding and screenshot-only operation across ScreenSpot, MiniWob, AITW, and Mind2Web. Results show stronger grounding is associated with better downstream task performance, although visual web navigation remains harder than HTML-based selection.
- GUI Grounding on ScreenSpot: ScreenSpot results show generalist LVLMs perform poorly on GUI grounding, while SeeClick achieves the best average performance across platforms and element types.All models struggle more with icons and widgets than with text, and SeeClick is slightly weaker than CogAgent on desktop and web text localization.
- MiniWob: SeeClick surpasses strong MiniWob baselines with substantially less training data and outperforms Qwen-VL by nearly 20 percentage points.Its task-level advantage is especially evident for dynamic layouts and changing element positions.
- AITW: SeeClick achieves the best average AITW performance among API-based LLMs and trained LVLMs, including a 9% click-accuracy increase over Qwen-VL.The result supports the relevance of precise GUI grounding to mobile-agent performance.
- Mind2Web: On Mind2Web, SeeClick nearly doubles element accuracy and step success rate compared with Qwen-VL.Despite operating without HTML, it trails state-of-the-art HTML-based methods because predicting coordinates is harder than selecting HTML candidates.
- Grounding and downstream performance: Across ScreenSpot and three downstream tasks, improvements in SeeClick’s grounding capacity consistently coincide with improved agent-task performance.The experiments use screenshot-only action selection for mobile and web tasks, with instructions and previous-action memory.
6 Conclusion
SeeClick is a screenshot-only visual GUI agent addressing GUI grounding through pre-training and automated web/mobile grounding-data curation. The authors introduce ScreenSpot and report improved grounding and consistently correlated gains across three GUI automation tasks.
- Contribution: SeeClick performs GUI task automation using screenshots alone, focusing on accurately locating screen elements from human instructions.Its intended actions include clicking and typing across GUI platforms.
- Contribution: The method enhances SeeClick with GUI grounding pre-training and automated curation of grounding data from web and mobile sources.This directly targets the grounding challenge identified for visual GUI agents.
- Benchmark: ScreenSpot is introduced as a realistic GUI grounding benchmark spanning mobile, desktop, and web platforms.It is designed to benchmark progress in GUI grounding.
- Results: Results show significant ScreenSpot improvement over LVLM baselines and consistently correlated improvements in downstream GUI-agent performance across three automation tasks.The conclusion links grounding advances with downstream performance without claiming that the benchmark result alone establishes causation.
Limitations
SeeClick currently supports a simplified GUI action space centered on clicking and typing. It excludes complex actions such as dragging and double-clicking.
- Action-space scope: SeeClick’s action space mainly covers clicking and typing, excluding complex actions such as dragging and double-clicking.This defines the current scope of supported GUI interactions.
Ethical considerations
GUI agents can improve digital-device efficiency, but their use raises privacy, safety, and bias concerns that require ongoing attention.
- GUI agents may access personal or sensitive interface information, making data protection and user consent important.
- Unintended harmful actions are a safety risk when GUI agents interact with the real world.
- Bias may produce unequal performance across user groups or interface designs, requiring mitigation for equitable access.
A.1 Pre-training Tasks
SeeClick is trained with GUI grounding and element-text tasks using coordinate representations, curated interface data, and continual pre-training.
- SeeClick’s grounding tasks predict either center-point coordinates or bounding boxes for instructed GUI elements.
- Its element-text tasks generate text from either a point or bounding box, supporting widget captioning alongside grounding.
- Coordinate targets use normalized point or bounding-box values, with each value recorded to two decimal places in [0,1].
- The pre-training data includes examples of the grounding tasks and is used for continual pre-training of SeeClick.
- Four experienced annotators captured interfaces, labeled clickable regions with bounding boxes, and supplied corresponding English text commands.
- ScreenSpot examples cover common GUI scenarios across mobile, desktop, and web platforms.
B.4 SeeClick Case Study & Error Analysis
SeeClick can locate instructed screen elements on ScreenSpot, while its remaining errors are generally near the target and reflect fine-grained localization difficulty.
- SeeClick comprehends human instructions and accurately locates screen elements on ScreenSpot.
- Figure 9 analyzes localization by measuring distances from predicted points to the ground-truth centers of target elements.
- Most incorrect predictions are close to the target bounding box, indicating target recognition but imperfect fine-grained localization.
C Downstream Agent Tasks
SeeClick formulates screenshot-based GUI interaction as next-action prediction and evaluates it across MiniWob, AITW, and Mind2Web with task-specific visual settings and metrics.
- Action space: The action space includes clicking, typing, selection, swipes, navigation commands, and task-completion states represented by action-type identifiers.
- Agent formulation: SeeClick predicts the next GUI action from an instruction, the current screenshot, and four previous actions.
- MiniWob: MiniWob provides roughly 100 Chrome-based web tasks involving low-level clicking and typing operations.
- AITW: AITW contains over 30K instructions and 700K Android automation episodes across five subsets.
- AITW: The original AITW split risks overfitting, so the paper introduces an instruction-wise split intended to better evaluate generalization.
- Mind2Web: Mind2Web is converted from HTML-based observations to visual evaluation using extracted screenshots and target-element bounding boxes.