PaperlayerView exact revision on arXiv

Source-linked AI summary

Aria-UI: Visual Grounding for GUI Instructions

Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, Junnan Li

arXiv:2412.16256v2cs.HCcs.AI

TL;DR

Grounding GUI elements from language remains challenging amid diverse interfaces and dynamic tasks. Aria-UI combines pure-vision grounding with synthesized diverse instructions and action-history context, and reports state-of-the-art results across offline and online agent benchmarks. Its current operation still depends on a planner and does not correct planner-generated errors.

  • Problem

    Grounding language instructions across diverse GUI layouts, variable instructions, and dynamic task contexts remains a significant challenge.

    Source: §1 · p. 1

  • Method

    Aria-UI uses pure-vision GUI grounding, a scalable pipeline for diverse element instructions, and textual or text-image-interleaved action histories.

    Source: §1 · p. 2

  • Results

    Aria-UI reports state-of-the-art performance across evaluated benchmarks, outperforming existing methods on offline and online agent tasks.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The paper presents Aria-UI as a scalable, data-centric solution for universal GUI grounding with zero-shot generalization across platforms.

    Source: §5 · p. 8

    Aria-UI depends on a planner for instructions and lacks training for correcting planner-generated errors during dynamic tasks.

    Source: §6 · p. 9

Abstract

from arXiv · show

Digital agents for automating tasks across different platforms by directly manipulating the GUIs are increasingly important. For these agents, grounding from language instructions to target elements remains a significant challenge due to reliance on HTML or AXTree inputs. In this paper, we introduce Aria-UI, a large multimodal model specifically designed for GUI grounding. Aria-UI adopts a pure-vision approach, eschewing reliance on auxiliary inputs. To adapt to heterogeneous planning instructions, we propose a scalable data pipeline that synthesizes diverse and high-quality instruction samples for grounding. To handle dynamic contexts in task performing, Aria-UI incorporates textual and text-image interleaved action histories, enabling robust context-aware reasoning for grounding. Aria-UI sets new state-of-the-art results across offline and online agent benchmarks, outperforming both vision-only and AXTree-reliant baselines. We release all training data and model checkpoints to foster further research at https://ariaui.github.io.

1 Introduction

GUI agents must ground varied language instructions in visual interfaces, but existing approaches rely on inconsistent or incomplete trees and often omit task history. Aria-UI uses vision, diverse synthesized instructions, and action histories, and reports stronger results than vision-only and AXTree-reliant baselines.

1 Introduction

  • Grounding remains difficult across diverse layouts, variable planner instructions, and dynamic real-world task execution.Tree-based methods can be inefficient or biased when tree information is missing, and lack visual or positional cues; tree standards and quality also vary across platforms.Source: §1 · pp. 1–2
  • Aria-UI is a pure-vision multimodal model for GUI grounding that avoids reliance on AXTree-like inputs.The paper describes it as built on Aria (Li et al., 2024a).Source: §1 · p. 2
  • A scalable data pipeline generates diverse element-caption and instruction samples from Common Crawl and public data to address rigid instruction sources.The pipeline uses element captions to generate varied, human-like instructions aligned with potential interactions.Source: §1 · p. 2
  • Textual or text-image-interleaved action histories help Aria-UI ground elements in dynamic, multi-step scenarios, especially in zero-shot settings.The histories are incorporated from trajectory data during training.Source: §1 · p. 2
  • Aria-UI achieves state-of-the-art performance on offline and online agent benchmarks, with higher grounding accuracy and task success rates than vision-only and AXTree-reliant baselines.The introduction reports these comparisons across extensive benchmarks.Source: §1 · p. 2

2 Method

Aria-UI combines diverse grounding data, trajectory context, and model training designed for GUI screens. Its pipeline synthesizes varied instructions across platforms, extends samples with action histories, and trains the model in stages for static and dynamic grounding.

2 Method

  • Aria-UI is designed as a grounding model that integrates into general-purpose multimodal GUI agent frameworks.The method presents a scalable, data-centric approach to the grounding challenges.Source: §2 · p. 2

2.1 Large-scale Diverse GUI Data Synthesizing

  • Existing GUI grounding corpora are often closed, small, platform-limited, rigid in instruction format, or lacking task context; the method addresses these gaps with cross-platform data and trajectory context.The paper summarizes these dataset shortcomings in Table 1.Source: §2.1 · p. 3

2.1.1 Diverse Data Scaling from Multiple Platforms

  • A two-stage pipeline uses an LMM to caption GUI elements and an LLM to generate natural-language instructions from those captions.The element descriptions draw on screenshots and HTML text, while generated instructions correspond to potential element interactions.Source: §2.1.1 · p. 3
  • The pipeline scales data across web, desktop, and mobile using Common Crawl, desktop exploration, AMEX regeneration, and additional public datasets.Web pages are curated and rendered; a desktop agent explores screens; mobile samples receive regenerated captions and instructions.Source: §2.1.1 · pp. 3–4

2.1.2 Context-aware Data Extension from Trajectories

  • Trajectory data supplies textual action histories or text-image-interleaved histories, with stepwise instructions added for grounding actions and rules encoding other actions.The interleaved setup uses 1, 2, or 3 historical screen images; the text setup includes prior actions in text.Source: §2.1.2 · p. 4

2.2 Model Architecture

  • Aria-UI is built on Aria, a multimodal-native mixture-of-experts model with 3.9B activated parameters.The paper highlights Aria’s handling of complex interleaved contexts and inference speed relative to 7B dense models.Source: §2.2 · p. 4

2.2.1 Ultra Resolution Support

  • Aria-UI extends supported image resolution from Aria’s 980×980 to a maximum of 3920×2940 by splitting images into blocks and preserving screenshot ratio with padding before resizing.The authors cite NaViT (Dehghani et al., 2024) as inspiration for the ratio-preserving approach.Source: §2.2.1 · p. 5

2.3 Training and Inference Paradigm

  • Training first uses single-step grounding data, then adds context-aware trajectory data while retaining 20% single-step samples to preserve generic grounding and avoid over-fitting.At inference, the model can use historical agent actions and grounding actions as chat history.Source: §2.3 · p. 5

3 Experiments

The experiments evaluate Aria-UI on single-step grounding and offline and online agent tasks. Across these settings, results report strong grounding accuracy and task success, while ablations examine resolution, training data, instruction diversity, and action history.

3.1 GUI Grounding Evaluation

  • 82.4% average accuracy across ScreenSpot subsets is the highest among the evaluated models, with a particularly strong margin over UGround on textual-element tasks.The benchmark spans three platforms and two element types.Source: §3.1 · p. 5

3.2 Offline Agent Evaluation

  • On offline mobile trajectories, Aria-UITH reaches peak grounding accuracy and task success on AndroidControl, while Aria-UIIH performs best on GUI-Odyssey.Both text-only and text-image interleaved action histories provide useful context; text-only history balances efficiency and performance against the base and interleaved variants.Source: §3.2 · pp. 5–6, Table 3 · p. 6
  • 58.9% average accuracy for Aria-UITH and Aria-UIIH, versus 57.5% for base Aria-UI, accompanies significant gains over all baselines across zero-shot Multimodal-Mind2Web subsets.Aria-UIIH is strongest on cross-website and cross-domain subsets.Source: §3.2 · p. 6, Table 4 · p. 6

3.3 Online Agent Evaluation

  • On AndroidWorld, Aria-UITH achieves the best performance to date, with a 44.8% task success rate, surpassing UGround, SoM, and Choice; on MobileMiniWob++, Aria-UI outperforms UGround and choice-based methods.Source: §3.3 · p. 7
  • 15.15% average task success rate with GPT-4o planning and Aria-UITH is the highest reported on OSWorld’s 369 tasks, outperforming prior methods across computer-use scenarios.The evaluation uses a pure-vision agent framework.Source: §3.3 · p. 7, Table 6 · p. 7

3.4 Ablation Study

  • Ablations identify ultra-resolution support and Aria-UI training data as important, especially for icon/widget grounding; removing diversified instructions or caption supervision degrades performance across platforms.The reported ultra-resolution ablation has an average score of 61.1; CoT prompting helps mobile text tasks but struggles elsewhere due to visual-reasoning noise.Source: §3.4 · pp. 7–8, Table 7 · p. 8

4 Related Work

Related work spans vision-language grounding with specialized vision models and large multimodal models, alongside GUI agents for mobile, web, and desktop environments. Existing approaches face adaptation challenges on GUI screenshots and limitations in instruction diversity and dynamic context.

4 Related Work

  • Foundational vision-language grounding combines CLIP with specialized vision models; later work uses large multimodal models for complex reasoning and fine-grained visual-region grounding (Zou et al., 2023; Liu et al., 2023; Li et al., 2023).General-purpose models also incorporate grounding as a core training function (Bai et al., 2023; Li et al., 2024a).Source: §4 · p. 8
  • GUI agents have been developed for mobile, web, and desktop tasks, but screenshot-grounding methods still face insufficient instruction diversity and dynamic-context modeling.Earlier agents often relied on HTML or accessibility trees for grounding.Source: §4 · p. 8

5 Conclusion

Aria-UI combines a two-stage data pipeline for diverse GUI grounding data with dynamic action history. The paper reports that it outperforms existing methods across evaluated offline and online benchmarks, with zero-shot generalization across platforms.

5 Conclusion

  • Aria-UI is presented as a robust LMM for GUI grounding across diverse environments.Source: §5 · p. 8
  • Its two-stage pipeline produces high-quality, diverse grounding data from multiple platforms, and dynamic action history supplies cues for grounding in real-world environments.Source: §5 · p. 8
  • The paper reports that Aria-UI outperforms existing methods on all evaluated offline and online benchmarks and demonstrates zero-shot generalization across platforms.Source: §5 · p. 8

6 Limitations

Aria-UI grounds planner-provided instructions but cannot yet plan and ground tasks autonomously or correct planner errors during dynamic tasks.

6 Limitations

  • Aria-UI depends on a planner, whose quality can affect performance on complex tasks.It currently lacks the ability to autonomously perform both planning and grounding.Source: §6 · p. 9
  • Aria-UI training does not yet incorporate correction of planner-generated instructions, limiting its ability to correct planner mistakes during dynamic tasks.Source: §6 · p. 9

A.1 Comprehensive Dataset Statistics

The dataset collection scales across platforms, diversifies grounding instructions, and includes trajectories for context-aware training.

A.1 Comprehensive Dataset Statistics

  • Aria-UI’s collection has the largest sample size within each platform and scales to nearly 1M training samples.Source: §A.1 · p. 13
  • The collection uses high-quality diversified instructions, while other public datasets use plain tree tags or small-scale human annotations.Source: §A.1 · p. 13
  • Context-aware training uses diverse trajectories averaging 5.5 to 15.4 steps, with prior actions as text context or text-image histories using window sizes N = [1, 2, 3].Source: §A.1 · p. 13

A.2 Pseudo-code for Desktop Grounding Data Scaling Agent Implementation

The desktop data-scaling agent traverses operating-system interfaces using heuristic search, state hashing, and hotkey-based backtracking.

A.2 Pseudo-code for Desktop Grounding Data Scaling Agent Implementation

  • The agent uses heuristic depth-first search, with Gemini 1.5 prioritizing informative UI elements likely to reveal novel states while deferring exit elements.Source: §A.2 · p. 13
  • It hashes all intractable elements on the current screen to identify identical states and distinguish similar but distinct states.Source: §A.2 · p. 13
  • ESC and SPACE hotkeys force transitions to previous states for backtracking.Source: §A.2 · p. 13

A.3 Discussion on the Overlap between OSWorld Tasks and the Desktop Data

Desktop data collection avoids OSWorld task-specific content, but its screenshots cover simpler single-app interfaces than many benchmark tasks.

A.3 Discussion on the Overlap between OSWorld Tasks and the Desktop Data

  • The collection avoids OSWorld task-specific files, predefined states, webpages, slides, and videos to prevent data leakage.Source: §A.3 · p. 13
  • The collected screenshots cover general single-app grounding, whereas one-third of OSWorld test tasks involve multiple windows with specific contents.Source: §A.3 · p. 13
  • The data collection also includes Ubuntu system settings and app store interfaces outside the scope of OSWorld benchmark tasks.Source: §A.3 · p. 13

B Aria-UI Training Details

Aria-UI is trained in single-step and trajectory-based phases with distinct context lengths. The accompanying prompts specify element descriptions, coordinate grounding, instruction diversification, and context-aware histories.

B Aria-UI Training Details

  • Phase 1 uses a 4096-token context for 10K steps over 18 hours; phase 2 uses 8192 tokens for 2K steps over 6 hours.Training uses 64 NVIDIA H800 GPUs.Source: B Aria-UI Training Details · p. 13
  • Training uses expert parallelism factor 8, batch size 256, minimum learning rate 1e-8, and phase learning rates of 2e-5 and 1e-5.ViT training is enabled with its parameters’ learning rate set to one-tenth of the LLM rate.Source: B Aria-UI Training Details · p. 13

C Prompts for Data Augmentation and Model Inference

  • The OS Traverse Agent prompt uses an accessibility tree to extract entrances, has an LMM rank them, and explores simulated states while tracking visited states and memory.Traversal stops exploring a state when its entrance hash was visited or its depth exceeds the maximum.Source: C Prompts for Data Augmentation and… · p. 14
  • Captioning prompts pair a full-screen context image with an isolated element and request concise descriptions of appearance, function, screen position, and nearby elements.The instructions specify screen quadrant and surrounding-element relations, not position within the crop.Source: C Prompts for Data Augmentation and… · p. 15
  • Single-step prompts request relative coordinates for an element; a chain-of-thought format first produces a referring description, then grounded coordinates.The outputs are separated into referring and grounding fields.Source: C Prompts for Data Augmentation and… · p. 15
  • Context-aware prompts include the ultimate task, prior steps, and interleaved GUI images and instructions when requesting grounded coordinates.The prompt asks for relative 0–1000 pixel coordinates for the instructed element.Source: C Prompts for Data Augmentation and… · p. 16
  • Instruction-diversification prompts turn detailed GUI-element descriptions into several user-oriented instructions for interacting with the element.They ask for a short reflection and multiple possible instructions.Source: C Prompts for Data Augmentation and… · p. 16

Related papers

Loading 2412.16256v2…