PaperlayerView exact revision on arXiv

Source-linked AI summary

Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents

Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, Yu Su

arXiv:2410.05243v3cs.AIcs.CLcs.CV

TL;DR

GUI agents need robust grounding, but text-based interface representations can be noisy, incomplete, and costly. The paper develops SeeAct-V and UGround using synthetic web data and an adapted LLaVA architecture, then evaluates them across six benchmarks. UGround substantially outperforms prior grounding models, while vision-only agents achieve comparable or often better performance than agents using additional textual inputs.

  • Problem

    Current GUI agents rely on HTML or accessibility trees that can be noisy, incomplete, and computationally expensive, motivating visual grounding for human-like GUI interaction.

  • Method

    The paper combines screenshot-only SeeAct-V with UGround, trained from web-based synthetic data using a slight adaptation of the LLaVA architecture.

  • Results

    UGround substantially outperforms existing visual grounding models, while SeeAct-V agents with UGround are comparable or often better than state-of-the-art agents using additional textual inputs.

  • Takeaways & Limitations

    The results support the feasibility of GUI agents that navigate digital environments through visual perception and pixel-level operations alone.

  • Takeaways & Limitations

    UGround depends on very large-scale synthetic data, and long-tail GUI elements remain under-addressed despite demonstrated cross-platform generalization.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) are transforming the capabilities of graphical user interface (GUI) agents, facilitating their transition from controlled simulations to complex, real-world applications across various platforms. However, the effectiveness of these agents hinges on the robustness of their grounding capability. Current GUI agents predominantly utilize text-based representations such as HTML or accessibility trees, which, despite their utility, often introduce noise, incompleteness, and increased computational overhead. In this paper, we advocate a human-like embodiment for GUI agents that perceive the environment entirely visually and directly perform pixel-level operations on the GUI. The key is visual grounding models that can accurately map diverse referring expressions of GUI elements to their coordinates on the GUI across different platforms. We show that a simple recipe, which includes web-based synthetic data and slight adaptation of the LLaVA architecture, is surprisingly effective for training such visual grounding models. We collect the largest dataset for GUI visual grounding so far, containing 10M GUI elements and their referring expressions over 1.3M screenshots, and use it to train UGround, a strong universal visual grounding model for GUI agents. Empirical results on six benchmarks spanning three categories (grounding, offline agent, and online agent) show that 1) UGround substantially outperforms existing visual grounding models for GUI agents, by up to 20% absolute, and 2) agents with UGround outperform state-of-the-art agents, despite the fact that existing agents use additional text-based input while ours only uses visual perception. These results provide strong support for the feasibility and promises of GUI agents that navigate the digital world as humans do.

1 INTRODUCTION

GUI agents are moving toward complex real-world environments, but current text-based observations can be noisy, incomplete, and costly. The paper proposes visually embodied agents centered on accurate, general, and flexible visual grounding, and evaluates UGround across broad benchmarks.

  • Multimodal LLMs extend GUI agents from simulations to diverse web, desktop, and mobile environments.
  • Human-like GUI agents should visually perceive rendered interfaces and perform pixel-level operations through keyboard, mouse, or touchscreen effectors.The study excludes auditory perception from this embodiment.
  • Text-based HTML and accessibility-tree observations can be noisy, incomplete, and computationally expensive compared with visual renderings.HTML can consume up to 10 times more tokens than the corresponding visual, while accessibility annotations can contain errors.
  • Visual grounding is the major bottleneck because agents must map textual plans to precise GUI locations across platforms and referring-expression types.The desired grounding model must provide high accuracy, strong cross-platform generalization, and flexibility across MLLMs.
  • The paper proposes SeeAct-V, a generic framework for entirely visual observation and pixel-level GUI interaction.SeeAct-V adapts the SeeAct framework for human-like embodiment.
  • 10M GUI elements across 1.3M screenshots support UGround, while six benchmarks show up to 20% absolute improvement and competitive or better agent performance.The evaluation spans grounding, offline-agent, and online-agent benchmarks.

2 METHOD

The method combines a screenshot-only agent framework with a universal visual grounding model trained on synthetic web-derived data. It covers diverse referring expressions, cross-platform inputs, and high-resolution GUI screenshots.

  • SeeAct-V framework: SeeAct-V uses screenshots as its only environmental observation and a separate visual grounding model to produce action coordinates.An MLLM generates textual plans, which the grounding model maps directly to screen coordinates.
  • Universal grounding design: The grounding model is designed for accuracy across web, desktop, and mobile GUIs, diverse referring expressions, different MLLMs, and varied resolutions.The method emphasizes a plug-and-play universal grounding model.
  • Data construction: Training data consists of synthetic ⟨screenshot, referring expression, coordinates⟩ triplets derived from webpages with HTML-rendering correspondences.Element center-point coordinates serve as the expected output.
  • Data construction: GUI referring expressions are categorized as visual, positional, functional, or composite descriptions.Examples include text or color, absolute or relative location, and actions such as navigating home or opening a cart.
  • Data construction: The hybrid synthesis pipeline combines HTML attributes, LLM-generated descriptors, and rule-based positional expressions.Web-Hybrid data is collected from Common Crawl screenshots and web-element metadata, supplemented with Android and Web-Direct data.
  • Data construction: 10M UI elements comprise the compiled training dataset, with 90% originating from the hybrid synthesis pipeline.Elements from the same screenshot are batched to accelerate training.
  • Model design: UGround adapts the 7B LLaVA-NeXT architecture for GUI visual grounding and outputs natural-language numerical coordinates.The model uses autoregressive decoding and supports high-resolution screenshots through an expanded AnyRes configuration.

3 EXPERIMENTS

The experiments evaluate UGround and vision-only SeeAct-V across six benchmarks spanning grounding, offline agents, and online agents on web, desktop, and mobile platforms. UGround consistently outperforms existing grounding models, while SeeAct-V with UGround matches or exceeds agents using additional text-based input.

  • Evaluation scope: Six benchmarks cover grounding, offline agent evaluation, and online agent evaluation across web, desktop, and mobile platforms.The evaluation compares both UGround’s grounding performance and end-to-end agent performance.
  • GUI visual grounding: 20% average absolute improvement in ScreenSpot standard grounding and 29% in the agent setting distinguish UGround from existing models across platforms.UGround also performs especially well on icons and widgets and generalizes to desktop UIs despite no desktop training screenshots.
  • Offline agent evaluation: SeeAct-V with UGround outperforms all baselines across offline agent benchmarks while using only raw screenshots, and it consistently outperforms SeeClick.The offline evaluations include Multimodal-Mind2Web, AndroidControl, and OmniACT.
  • Online agent evaluation: SeeAct-V with UGround achieves comparable or higher performance in online agent evaluation and a much higher success rate than the SoM variant of M3A in Android environments.The comparison is notable because Android environments are generally more suitable for SoM due to less dense layouts and less label obstruction.
  • Error analysis: Planning errors dominate failures across ScreenSpot, AndroidControl, and Multimodal-Mind2Web, while long-tail icon semantics remain a grounding challenge on desktop and mobile UIs.UGround remains relatively robust on realistic agent tasks because those tasks mostly involve common elements rather than rare long-tail icons.
  • Training data analysis: Performance improves consistently as Web-Hybrid data scales, but returns diminish after 100K screenshots; 50K screenshots already surpass SeeClick by more than 10%.Additional data primarily improves recognition of less frequent elements, tiny text, and more accurate pointing to bounding-box centers.

4 CONCLUSIONS AND LIMITATIONS

The paper presents UGround and SeeAct-V as a vision-only approach to GUI agents, supported by broad benchmark results. It also identifies limitations involving synthetic-data efficiency, long-tail elements, desktop coverage, and reliance on an external planner.

  • SeeAct-V with UGround supports pixel-level GUI interaction using visual input alone and often outperforms agents that rely on additional textual inputs.
  • The released resources include a universal grounding model, large-scale grounding data, and a framework for vision-only agents.
  • UGround’s large synthetic training set leaves room for improved data efficiency through grouping and deduplication.
  • Desktop performance is limited because no desktop UI data is included in training, while long-tail mobile and desktop elements remain under-addressed.
  • UGround depends on an external planner and is not intended to function independently as a GUI agent.

ETHICS STATEMENT

The paper frames its web-based synthesis and data collection as research-only activities using Common Crawl material under non-commercial and legal-compliance constraints. It also describes moderation and regulated distribution practices.

  • The synthesis pipeline and data collection are intended solely for research on GUI grounding and GUI agents.
  • The webpages come from Common Crawl, a publicly available Internet archive for research and non-commercial use.
  • The authors state that data use and dissemination are exclusively academic, undergo content moderation, and follow applicable regulations and AI ethics guidelines.

B PHILOSOPHY BEHIND SEEACT-V AND UGROUND

SeeAct-V advocates modular GUI agents because digital environments are complex, dynamic, and idiosyncratic. A dedicated grounding module can specialize in domain semantics while enabling flexible and more consistent evaluation.

  • Motivation for modularity: Web environments combine over one billion websites, dynamic states, and idiosyncratic semantics such as uncommon icons and jargon.
  • Motivation for modularity: The authors argue that reliable generalization across diverse contexts requires a modular system coordinating a foundation model with specialized modules.
  • Role of grounding: A dedicated grounding module captures domain-specific semantics and can be adapted by fine-tuning the grounding model rather than the entire foundation model.
  • Practical advantages: Modularity allows UGround to be studied independently, while flexibility supports diverse multimodal LLMs and grounding models without downstream specialized fine-tuning.
  • Practical advantages: Standardizing planning reduces confounding variables when comparing grounding methods and their effects on agent performance.
  • Empirical implication: SeeAct-V with UGround outperforms end-to-end MLLMs, whose training requires extensive high-quality agent-trajectory data.

C FURTHER ABLATION STUDIES

The further ablation studies examine model design choices and the web-based synthetic dataset using grounding accuracy on ScreenSpot with GPT-4o as planner.

  • The ablations evaluate model design choices and synthetic-data effectiveness on ScreenSpot Agent Setting grounding accuracy with GPT-4o as planner.

C.1 CONTROLLED COMPARISON TO BASELINE MODELS

The controlled comparisons isolate contributions from model design, training data, image resolution, and referring-expression choices. UGround benefits from its architecture and hybrid positional/contextual grounding signals, while dynamic resolution improves efficiency across platforms.

  • Training Data: 10.1% average absolute improvement over SeeClick comes from UGround-Qwen trained solely on Web-Hybrid, despite SeeClick using additional mobile UI data.This comparison uses the same Qwen-VL-Chat backbone and highlights Web-Hybrid’s cross-platform generalization.
  • Model Design: 14.5% absolute improvement over UGround-Qwen demonstrates the contribution of UGround’s model design.UGround-Qwen uses the Qwen-VL-Chat backbone and only Web-Hybrid, providing the controlled design comparison.
  • Image Resolution: Larger image resolution generally improves performance on dense web and desktop interfaces containing small links and icons.Mobile interfaces benefit less because they are less dense.
  • Dynamic Image Resolution and Aspect Ratio: Dynamic resolution and aspect-ratio adaptation improve performance across platforms while using approximately 2/3 of the tokens of a fixed 1,344 x 1,344 model in 16:9 desktop and web scenarios.AnyRes adapts to varied resolutions and aspect ratios, including mobile and desktop interfaces.
  • RE Types: The referring-expression taxonomy is presented as a novel contribution, while comparisons with CogAgent are omitted because of its inferior performance relative to SeeClick.The taxonomy distinguishes diverse GUI references, and the omitted comparison concerns CogAgent’s reported performance and scale.
  • RE Types: Positional referring expressions generally enhance grounding performance when added to Web-Hybrid training data.The authors hypothesize that positional and contextual data help the model capture spatial relationships among UI elements.

D.4 TRAINING DATA

The appendix provides examples of training data from different sources and identifies representative evaluation-pipeline outputs for AndroidControl and related GUI benchmarks.

  • Training Data: Examples of training data from different sources are provided to illustrate the constructed dataset.The examples are collected in Figure D.4.

E DATA CONSTRUCTION

The dataset construction pipeline combines webpage rendering, structured HTML metadata, OCR, manually defined spatial rules, and language-model-generated descriptions to create diverse GUI referring expressions.

  • Webpage Rendering: Approximately 773K screenshots are generated by rendering webpages at varied resolutions and aspect ratios, with about one-third using mobile-friendly ratios.Long webpages are sampled across up to three viewport-sized content blocks to increase content diversity.
  • LLM-Based Descriptions: The pipeline uses LLaVA-NeXT-13B for element-level descriptions and Llama-3-8B to condense them into concise referring expressions.Descriptions are generated from cropped element images and salient HTML attributes.
  • Element Selection: Interactive elements are the primary grounding targets, while pure text elements can supply source material for generating referring expressions.Elements are grouped by HTML tags into interactive and pure-text categories.
  • Element Processing: OCR bypasses synthesis for textual elements when extracted and inner text have similarity above 0.7, preventing trivial referring expressions.Elements with identical text on a page receive additional disambiguation handling.
  • Positional and Contextual Expressions: Manually crafted rules generate absolute and relative positional expressions using neighboring elements, titles, rows, columns, and hierarchical sections.Attributes such as title, alt, and aria-label provide additional functional descriptors.
  • Web-Direct: Web-Direct uses GPT-4o with red boxes, arrows, and explicit visibility queries to reduce hallucinations and filter crawling or occlusion errors.Separate prompts generate free-form and functionally oriented referring expressions.

E.3 OPEN-SOURCE DATA

Open-source GUI datasets supplement the web-based data with annotated expressions and coordinate-grounded actions from Android and other GUI environments. The model architecture is built on LLaVA-NeXT with a frozen CLIP image encoder and Vicuna language backbone.

  • Supplementary Datasets: Android, GUIAct, Widget Caption, UIBert, and AITZ provide supplementary referring expressions or coordinate-grounded actions.These sources expand coverage of functional, instructional, and action-oriented grounding expressions.
  • Data Filtering: GUIAct and AndroidControl data are filtered to retain steps with coordinate-grounded targets, while GUIAct uses both instruction and action annotations.Widget Caption contributes two randomly selected functional captions per element.
  • Model Architecture: UGround’s architecture is based on LLaVA-NeXT, using a frozen CLIP-ViT-L-14 image encoder and Vicuna-1.5-7b-16k as the language backbone.The encoder supports flexible AnyRes splitting and the language model handles long visual contexts.
  • Model Architecture: AnyRes splits images into 224px grids while preserving aspect ratios through width-based resizing and bottom padding, with at most 36 grids.The maximum supported resolution is 1,344 x 1,344.

F.3 TRAINING

Training proceeds in two stages: adapting LLaVA-1.5 for coordinate-based grounding, then training UGround on GUI visual-grounding datasets. LoRA-based instruction finetuning is used to reduce the cost of high-resolution image processing.

  • Training has two stages: LLaVA-1.5 pretraining and finetuning, followed by GUI visual grounding training for UGround.
  • The finetuning stage changes grounding targets from normalized to absolute coordinates and uses a modified AnyRes setting.
  • LoRA is used for instruction finetuning in both stages because high-resolution image processing is computationally expensive.
  • The first stage takes about 50 hours on one 4x NVIDIA A100 machine, while large-scale GUI training takes about 6 hours on 112 NVIDIA H100 GPUs.

G EVALUATION DETAILS

Evaluation specifies planner endpoints, benchmark preprocessing, grounding criteria, and agent adaptations across desktop, mobile, and web settings. Vision-only agents remove textual element representations and use pixel-level actions, with task-specific prompt and execution procedures.

  • Evaluation setup: Different GPT endpoints are specified for the evaluated systems and baselines because endpoint choice can slightly affect GUI-task performance.
  • Evaluation setup: Full-page Multimodal-Mind2Web screenshots are divided into 1,280 × 1,000 pixel viewport blocks, with simulated scrolling between blocks.
  • Evaluation setup: Grounding is evaluated by element accuracy, counting predictions as correct when output coordinates fall inside the ground-truth element box.
  • Agent adaptations: SeeAct-V is integrated into M3A by removing SoM images and textual accessibility-tree element lists, then replacing element-based actions with pixel-level actions.
  • Agent adaptations: OmniACT evaluation generates PyAutoGUI scripts with element descriptions, grounds those descriptions to coordinates, and substitutes the coordinates into the scripts.
  • Agent adaptations: Unlike DetACT, the proposed method does not use a pre-generated element list; its planner produces precise element descriptions from screenshots alone.
  • Prompt specifications: The evaluation appendix provides task-specific planning prompts for Multimodal-Mind2Web, AndroidControl, OmniACT, and ScreenSpot, including continued multi-page tables.
Loading 2410.05243v3…