PaperlayerView exact revision on arXiv

Source-linked AI summary

Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction

Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, Caiming Xiong

arXiv:2412.04454v2cs.CL

TL;DR

GUI automation is limited by textual, platform-specific representations and insufficient reasoning. AGUVIS addresses these gaps with a unified pure-vision framework, annotated data, and two-stage training, achieving state-of-the-art offline and online performance without closed-source models.

  • Problem

    GUI agents remain difficult to develop because existing approaches rely on platform-specific textual representations and have limited autonomous reasoning.

  • Method

    AGUVIS uses screen images, standardized cross-platform actions, structured inner monologue, a multimodal dataset, and separate grounding and planning-reasoning training stages.

  • Results

    AGUVIS achieves state-of-the-art performance across offline and real-world online benchmarks as a fully autonomous vision-based agent without closed-source models.

  • Takeaways & Limitations

    AGUVIS demonstrates the potential of pure vision-based agents for autonomous GUI interaction across diverse platforms and novel computing scenarios.

  • Takeaways & Limitations

    Real-world deployment remains constrained by the need to prevent harmful actions and handle ambiguous or potentially harmful situations safely.

Abstract

from arXiv · show

Automating GUI tasks remains challenging due to reliance on textual representations, platform-specific action spaces, and limited reasoning capabilities. We introduce Aguvis, a unified vision-based framework for autonomous GUI agents that directly operates on screen images, standardizes cross-platform interactions and incorporates structured reasoning via inner monologue. To enable this, we construct Aguvis Data Collection, a large-scale dataset with multimodal grounding and reasoning annotations, and develop a two-stage training pipeline that separates GUI grounding from planning and reasoning. Experiments show that Aguvis achieves state-of-the-art performance across offline and real-world online benchmarks, marking the first fully autonomous vision-based GUI agent that operates without closed-source models. We open-source all datasets, models, and training recipes at https://aguvis-project.github.io to advance future research.

1. Introduction

AGUVIS addresses GUI-agent limitations by replacing platform-specific textual representations with direct visual interaction and structured reasoning. It contributes a cross-platform framework, annotated dataset, and two-stage training pipeline.

  • GUI automation requires visual understanding, grounding, and planning and reasoning to map instructions into effective interface actions.
  • Existing GUI-agent approaches face challenges from textual representations, platform fragmentation, and limited autonomous reasoning.
  • AGUVIS operates directly on screen images while harmonizing visual observations and consistent action spaces across diverse GUI environments.
  • A plugin-based standardized action space preserves consistent interaction patterns while accommodating environment-specific requirements.
  • AGUVIS DATA COLLECTION provides multimodal grounding and reasoning annotations, while two-stage training separates GUI grounding from planning and reasoning.
  • AGUVIS reports state-of-the-art offline and online performance and operates as a fully autonomous vision-based agent without closed-source models.

2. AGUVIS

AGUVIS models GUI interaction as sequential visual decision-making and unifies visual observation, structured reasoning, and standardized actions across platforms. Its dataset and two-stage training pipeline separately support grounding and planning-reasoning capabilities.

  • 2.1. Problem Formulation: GUI interaction is formulated as a partially observable sequential process in which image observations lead to actions until a goal or terminal state is reached.
  • 2.2. Unified GUI Interaction Framework: The framework replaces platform-specific interface code with visual input, maintaining a constant cost of 1,196 tokens for 720p images.Traditional textual methods require 4k-6k tokens per interaction.
  • 2.2. Unified GUI Interaction Framework: A two-component inner monologue connects visual perception to adaptive planning and precise action generation for complex multi-step tasks.
  • 2.2. Unified GUI Interaction Framework: Pyautogui supplies a universal action interface, while plugins extend it with mobile gestures, platform shortcuts, and meta-actions.
  • 2.3. AGUVIS DATA COLLECTION: AGUVIS DATA COLLECTION combines standardized grounding data with VLM-augmented planning and reasoning trajectories.
  • 2.3. AGUVIS DATA COLLECTION: 86.7% of augmented data demonstrated intermediate reasoning aligned with both ground-truth actions and overall goal intention.
  • 2.5. Training Paradigm: Training is divided into Grounding Training and Planning & Reasoning Training, using distinct dataset splits to progressively build agentic abilities.

3. Experiments

Experiments evaluate AGUVIS on offline website, mobile, and grounding benchmarks and on real-world online GUI tasks. The reported results show strong cross-platform performance, including superior offline results and best-in-class performance on selected online benchmarks.

  • 3.1. GUI Grounding Evaluation: On ScreenSpot, AGUVIS-72B achieves a state-of-the-art average score of 89.2 across mobile, desktop, and website platforms.
  • 3.2. Offline GUI Agent Evaluation: On Multimodal-Mind2Web, AGUVIS operates solely on GUI screenshots and improves Step Success Rate by +51.9% on average over prior approaches.
  • 3.2. Offline GUI Agent Evaluation: On AndroidControl, AGUVIS shows superior performance in both high-level planning and low-level instruction-execution settings.
  • 3.3. Online GUI Agent Evaluation: Across Mind2Web-Live, AndroidWorld, and MobileMiniWob, AGUVIS-7B with GPT-4o planning performs better than existing methods, while AGUVIS-72B is best-in-class on Mind2Web-Live and MobileMiniWob.

4. Analysis

Aguvis’s unified training and pure-vision design improve cross-platform GUI performance, efficiency, and generalization, while error analysis identifies ambiguity and grounding as key failure sources.

  • Training Ablations: Omitting either training stage degrades performance, with removing Stage 2 causing significant drops across all metrics.The ablation compares AGUVIS-G, which lacks planning and reasoning training, with the specialized model and Qwen2-VL.
  • Training Ablations: 81.2%: LLaVA ScreenSpot performance after the complete AGUVIS training process, up from 3.8%.The result supports methodology-dependent improvement across different foundation models.
  • Training Ablations: ScreenSpot falls from 84.4% to 79.3% without inner monologue, while low-level AndroidControl drops from 80.5% to 69.1%.The reported declines indicate effects on both high-level planning and precise low-level execution.
  • Cross-Platform Benefits: Training on both web and mobile data outperforms platform-specific training on web-specific Multimodal-Mind2Web tasks.The unified action space supports knowledge transfer across device domains.
  • Generalization: 17.04%: OSWorld task success with GPT-4o planning, compared with 4.59% for SoM-based approaches and 14.9% for Claude Computer-Use.AGUVIS-72B independently achieves 10.26% despite training only on web and mobile trajectories.
  • Efficiency: 1,196 tokens: constant vision input cost per step for 720p images, versus about 4,000 tokens for typical HTML-based approaches.The vision-based method reduces costs by 93% in Mind2Web-Live.
  • Error Analysis: 40% of 50 ScreenSpot self-plan errors arise from ambiguous instructions, while 60% are grounding errors.The model also lacks the ability to indicate uncertainty or refuse ambiguous actions.
  • Error Analysis: Enforced inner monologue resolves 20% of grounding errors, but semantically complex queries still require deeper understanding and domain knowledge.The analysis motivates adversarial refusal examples and annotations of task complexity and reasoning depth.

5. Related Work

Related work evaluates autonomous GUI agents across web, desktop, and mobile environments, alongside cross-platform datasets and direct-interaction approaches.

  • Benchmarks: GUI-agent benchmarks span web, desktop, and mobile environments, assessing capabilities across diverse platforms.The cited benchmark landscape includes separate platform-focused evaluations.
  • Datasets: Cross-platform datasets such as ScreenSpot, OmniACT, GUICourse, and CRAB provide evaluation frameworks spanning multiple devices.These resources complement platform-specific benchmarks.
  • Existing Agents: Prior systems including WebGPT, Lemur, CogAgent, Auto-GUI, AppAgent, and ScreenAgent improve web navigation or direct GUI interaction.The cited approaches represent progress across web and GUI-agent settings.

6. Conclusion

Aguvis is presented as a unified pure-vision GUI-agent framework for diverse platforms, combining standardized actions, structured reasoning, open resources, and strong offline and online results.

  • Conclusion: Aguvis combines vision-only observations, a standardized action space, structured reasoning, a large-scale dataset, and two-stage training.The framework is designed for autonomous GUI agents operating across diverse platforms.
  • Conclusion: Extensive experiments demonstrate state-of-the-art performance in both offline and online GUI tasks.The authors open-source datasets, models, and training recipes to accelerate future research.

Impact Statement

Aguvis aims to democratize autonomous GUI interaction through open-source, pure-vision agents with standardized actions and native planning and reasoning capabilities.

  • Impact: Pure-vision GUI agents can automate tasks using existing human-centric tools, with potential benefits for human productivity.The work emphasizes open-source models rather than proprietary LLMs.
  • Impact: AGUVIS uses standard pyautogui actions with pluggable environment-specific actions to combine universality and flexibility.The action space is described as applicable across environments while accommodating specific requirements.
  • Data and Training: The dataset separates grounding examples from planning-and-reasoning examples, with the grounding split containing single-step trajectories.Planning data is augmented with goals, previous actions, current action instructions, and pyautogui commands.
  • Data Quality: 86.7% of sampled augmented data shows intermediate reasoning aligned with ground-truth actions and overall goal intention.The remaining cases are attributed to dataset noise or misinterpretation of action intention.

B.3.2. FAILURE CASES UNDER NOISY TRAINING DATA

The failure analysis identifies redundant actions in training data as a source of incorrect reasoning and planning, despite preserving trajectory completeness or correctness. The section also contrasts Stage 1 grounding data with Stage 2 planning data, which adds thoughts and low-level instructions.

  • Unnecessary training-data actions can prevent the VLM from connecting intermediate actions to the overall goal, causing incorrect reasoning and planning.
  • Redundant actions do not compromise trajectory completeness or correctness but make accurate planning more difficult for the VLM.
  • Training data schemas: Stage 1 grounding examples prompt the agent to generate the next pyautogui action from an image, instruction, and previous actions.
  • Training data schemas: Stage 2 planning examples add a thought and low-level instruction before the pyautogui action.
  • Planning and reasoning data: AGUVIS differs from existing trajectory datasets by explicitly modeling observations, thoughts, and natural-language low-level instructions rather than outputting action commands alone.

C.2. Training Details

This section describes implementation choices and evaluation benchmarks for AGUVIS training. It covers backbone-specific image processing, optimization settings, and benchmark protocols spanning GUI grounding, web planning, and Android actions.

  • Training configuration: AGUVIS uses 1280 × 720 maximum image pixels with Qwen2-VL and an anyres multi-patch strategy with LLaVA-OneVision.
  • Training configuration: Each training stage runs for 1 epoch with batch size 128, using Adam, BF16, gradient checkpointing, and DeepSpeed optimization.
  • Training configuration: Increasing the maximum image resolution to 1920 × 1080 did not significantly improve ScreenSpot performance in preliminary experiments.
  • Evaluation benchmarks: ScreenSpot contains 1.2K single-step grounding instructions across mobile, desktop, and website platforms, evaluated with original-instruction and self-plan settings.
  • Evaluation benchmarks: Multimodal-Mind2Web evaluates offline website planning with element accuracy, Operation F1, and step success rate, while AndroidControl reports OOD step accuracy for high- and low-level tasks.
  • Evaluation benchmarks: SeeClick and CogAgent comparisons report only element accuracy because their original evaluations used Mind2Web rather than Multimodal-Mind2Web.

D.3. Online GUI Agent Evaluation

The online evaluation uses unified vision-based agents across web and Android environments, with either GPT-4o planning plus AGUVIS-7B grounding or AGUVIS-72B performing both roles. Evaluation required adapting text-oriented web verification and was affected by website automation barriers.

  • Web evaluation: Mind2Web-Live contains 104 real-time web tasks and is evaluated in BrowserGym using task success rate.
  • Web evaluation: Mind2Web-Live evaluation verifies clicks through coordinate comparisons with CSS-selector bounding boxes and validates inputs by comparing entered values with expected values.
  • Evaluation limitations: 18 network errors and 6 incomplete reCAPTCHA tasks prevented scoring on 24 Mind2Web-Live tasks.
  • Android evaluation: AndroidWorld spans 20 applications and 116 tasks, but AGUVIS uses only raw screenshots with coordinate-level actions and basic mobile functions.
  • Android evaluation: MobileMiniWob instantiates 92 MiniWob++ tasks in AndroidWorld and uses the same observation and action space with real-time task-success evaluation.
  • Agent configurations: Online experiments use GPT-4o with AGUVIS-7B for planning and grounding, or AGUVIS-72B as both planner and grounder.
  • Agent configurations: The GPT-4o plus AGUVIS-7B setup modifies prompts so GPT-4o can delegate grounding actions and share planning outputs with AGUVIS-7B.
  • Action interfaces: The prompt instructions require reviewing the current page state and selecting a correctly formatted next action, including waiting, scrolling, filling, clicking, or messaging.

E.1.1. TRAINING STRATEGY ABLATION

The ablation compares staged, joint, and partial training strategies, showing that sequencing Stage 1 grounding before Stage 2 planning and reasoning balances grounding and planning performance across models.

  • Training-strategy comparison: Joint training performs better on GUI grounding but worse on planning datasets than staged training.The reported trade-off appears on MM-Mind2Web and AndroidControl High-level, while overall differences between staged and joint setups are relatively minor.
  • Training-strategy comparison: Stage 1 grounding data is more abundant, whereas Stage 2 data is higher-quality and better aligned with deployment scenarios.The authors use this data imbalance and alignment difference to justify placing Stage 2 later in training.
  • Stage contributions: Stage 2-only training preserves similar MM-Mind2Web and AndroidControl performance but declines on ScreenSpot GUI grounding.The results attribute this stability to Qwen2VL’s natural-image grounding pre-training and emphasize varied Stage 1 grounding data for ScreenSpot.
  • Cross-model validation: On LLaVA, performing both stages substantially improves performance and surpasses previous state-of-the-art results.This cross-model result supports the authors’ claim that the training method is model-agnostic across Qwen2-VL and LLaVA.

E.1.2. DATA STRATEGY ABLATION

The data ablations show that combining Web and Mobile data improves Web-domain generalization, while inner monologue training improves reasoning and action execution.

  • Cross-device data: Training with both Web and Mobile data consistently outperforms training exclusively on MM-Mind2Web.All compared settings include MM-Mind2Web fine-tuning, isolating the contribution of broader device-domain data.
  • Cross-device data: Mobile data improves cross-device domain generalization in the Web domain.The authors interpret the performance gain as evidence supporting their cross-platform data strategy.
  • Inner monologue: Inner monologue training produces clear performance gains across the evaluated GUI tasks.The authors attribute these gains to reasoning about the current step and providing context for planning subsequent steps.
  • Inner monologue: Low-level inner-monologue instructions improve action-execution accuracy on ScreenSpot and AndroidControl low-level tasks.These instructions function as atomic instruction-and-grounding-action pairs, also strengthening grounding ability.
  • Planning prompts: Enforced planning resolves approximately 20% of grounding errors by prompting consideration of task context, ambiguities, and available UI elements.The enforced-plan prompt requires high-level reasoning before actions, unlike self-planning, where the model can choose whether to reason first.
Loading 2412.04454v2…