Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction

• GUI automation requires visual understanding, grounding, and planning and reasoning to map instructions into effective interface actions.Source: §1 · p. 1
• Existing GUI-agent approaches face challenges from textual representations, platform fragmentation, and limited autonomous reasoning.Source: §1 · p. 1
• AGUVIS operates directly on screen images while harmonizing visual observations and consistent action spaces across diverse GUI environments.Source: §1 · p. 1, Figure 1 · p. 2
• A plugin-based standardized action space preserves consistent interaction patterns while accommodating environment-specific requirements.Source: §1 · p. 2
• AGUVIS DATA COLLECTION provides multimodal grounding and reasoning annotations, while two-stage training separates GUI grounding from planning and reasoning.Source: §1 · p. 2
• AGUVIS reports state-of-the-art offline and online performance and operates as a fully autonomous vision-based agent without closed-source models.Source: §1 · p. 2

2.1. Problem Formulation

• GUI interaction is formulated as a partially observable sequential process in which image observations lead to actions until a goal or terminal state is reached.Source: §2.1 · p. 2

2.2. Unified GUI Interaction Framework

• The framework replaces platform-specific interface code with visual input, maintaining a constant cost of 1,196 tokens for 720p images.Traditional textual methods require 4k-6k tokens per interaction.Source: §2.2 · p. 2
• A two-component inner monologue connects visual perception to adaptive planning and precise action generation for complex multi-step tasks.Source: §2.2 · pp. 2–3
• Pyautogui supplies a universal action interface, while plugins extend it with mobile gestures, platform shortcuts, and meta-actions.Source: §2.2 · p. 3

2.3. AGUVIS DATA COLLECTION

• AGUVIS DATA COLLECTION combines standardized grounding data with VLM-augmented planning and reasoning trajectories.Source: §2.3 · p. 3, Figure 2 · p. 3
• 86.7% of augmented data demonstrated intermediate reasoning aligned with both ground-truth actions and overall goal intention.Source: §2.3 · p. 4

2.5. Training Paradigm

• Training is divided into Grounding Training and Planning & Reasoning Training, using distinct dataset splits to progressively build agentic abilities.Source: §2.5 · p. 4

3.1. GUI Grounding Evaluation

• On ScreenSpot, AGUVIS-72B achieves a state-of-the-art average score of 89.2 across mobile, desktop, and website platforms.Source: §3.1 · p. 5

3.2. Offline GUI Agent Evaluation

• On Multimodal-Mind2Web, AGUVIS operates solely on GUI screenshots and improves Step Success Rate by +51.9% on average over prior approaches.Source: §3.2 · p. 5
• On AndroidControl, AGUVIS shows superior performance in both high-level planning and low-level instruction-execution settings.Source: §3.2 · p. 5, Table 3 · p. 5

3.3. Online GUI Agent Evaluation

• Across Mind2Web-Live, AndroidWorld, and MobileMiniWob, AGUVIS-7B with GPT-4o planning performs better than existing methods, while AGUVIS-72B is best-in-class on Mind2Web-Live and MobileMiniWob.Source: §3.3 · pp. 5–6, Table 5 · p. 6

Training Ablations

• Omitting either training stage degrades performance, with removing Stage 2 causing significant drops across all metrics.The ablation compares AGUVIS-G, which lacks planning and reasoning training, with the specialized model and Qwen2-VL.Source: §4.1 · p. 6
• 81.2%: LLaVA ScreenSpot performance after the complete AGUVIS training process, up from 3.8%.The result supports methodology-dependent improvement across different foundation models.Source: §4.1 · p. 6
• ScreenSpot falls from 84.4% to 79.3% without inner monologue, while low-level AndroidControl drops from 80.5% to 69.1%.The reported declines indicate effects on both high-level planning and precise low-level execution.Source: §4.2 · p. 6

Cross-Platform Benefits

• Training on both web and mobile data outperforms platform-specific training on web-specific Multimodal-Mind2Web tasks.The unified action space supports knowledge transfer across device domains.Source: §4.3 · p. 6

Generalization

• 17.04%: OSWorld task success with GPT-4o planning, compared with 4.59% for SoM-based approaches and 14.9% for Claude Computer-Use.AGUVIS-72B independently achieves 10.26% despite training only on web and mobile trajectories.Source: §4.3 · p. 7

Efficiency

• 1,196 tokens: constant vision input cost per step for 720p images, versus about 4,000 tokens for typical HTML-based approaches.The vision-based method reduces costs by 93% in Mind2Web-Live.Source: §4.4 · p. 7, Figure 3 · p. 7

Error Analysis

• 40% of 50 ScreenSpot self-plan errors arise from ambiguous instructions, while 60% are grounding errors.The model also lacks the ability to indicate uncertainty or refuse ambiguous actions.Source: §4.5 · pp. 7–8
• Enforced inner monologue resolves 20% of grounding errors, but semantically complex queries still require deeper understanding and domain knowledge.The analysis motivates adversarial refusal examples and annotations of task complexity and reasoning depth.Source: §4.5 · p. 8

Benchmarks

• GUI-agent benchmarks span web, desktop, and mobile environments, assessing capabilities across diverse platforms.The cited benchmark landscape includes separate platform-focused evaluations.Source: §5.1 · p. 8

Datasets

• Cross-platform datasets such as ScreenSpot, OmniACT, GUICourse, and CRAB provide evaluation frameworks spanning multiple devices.These resources complement platform-specific benchmarks.Source: §5.1 · p. 8

Existing Agents

• Prior systems including WebGPT, Lemur, CogAgent, Auto-GUI, AppAgent, and ScreenAgent improve web navigation or direct GUI interaction.The cited approaches represent progress across web and GUI-agent settings.Source: §5.2 · p. 8

Conclusion

• Aguvis combines vision-only observations, a standardized action space, structured reasoning, a large-scale dataset, and two-stage training.The framework is designed for autonomous GUI agents operating across diverse platforms.Source: §6 · p. 8
• Extensive experiments demonstrate state-of-the-art performance in both offline and online GUI tasks.The authors open-source datasets, models, and training recipes to accelerate future research.Source: §6 · p. 8

Impact

• Pure-vision GUI agents can automate tasks using existing human-centric tools, with potential benefits for human productivity.The work emphasizes open-source models rather than proprietary LLMs.Source: Impact Statement · p. 9
• AGUVIS uses standard pyautogui actions with pluggable environment-specific actions to combine universality and flexibility.The action space is described as applicable across environments while accommodating specific requirements.Source: §A.1 · p. 14, Table 9 · p. 14

Data and Training

• The dataset separates grounding examples from planning-and-reasoning examples, with the grounding split containing single-step trajectories.Planning data is augmented with goals, previous actions, current action instructions, and pyautogui commands.Source: Table 10 · p. 16, Table 11 · p. 16, §B.2 · p. 16

Data Quality

• 86.7% of sampled augmented data shows intermediate reasoning aligned with ground-truth actions and overall goal intention.The remaining cases are attributed to dataset noise or misinterpretation of action intention.Source: §B.3.1 · p. 17

• Unnecessary training-data actions can prevent the VLM from connecting intermediate actions to the overall goal, causing incorrect reasoning and planning.Source: §B.3.2 · p. 17
• Redundant actions do not compromise trajectory completeness or correctness but make accurate planning more difficult for the VLM.Source: §B.3.2 · p. 17

Training data schemas

• Stage 1 grounding examples prompt the agent to generate the next pyautogui action from an image, instruction, and previous actions.Source: §C.1 · p. 17
• Stage 2 planning examples add a thought and low-level instruction before the pyautogui action.Source: §C.1 · p. 19

Planning and reasoning data

• AGUVIS differs from existing trajectory datasets by explicitly modeling observations, thoughts, and natural-language low-level instructions rather than outputting action commands alone.Source: §C.1 · p. 19

Training configuration

• AGUVIS uses 1280 × 720 maximum image pixels with Qwen2-VL and an anyres multi-patch strategy with LLaVA-OneVision.Source: §C.2 · p. 19
• Each training stage runs for 1 epoch with batch size 128, using Adam, BF16, gradient checkpointing, and DeepSpeed optimization.Source: §C.2 · p. 19
• Increasing the maximum image resolution to 1920 × 1080 did not significantly improve ScreenSpot performance in preliminary experiments.Source: §C.2 · p. 19

Evaluation benchmarks

• ScreenSpot contains 1.2K single-step grounding instructions across mobile, desktop, and website platforms, evaluated with original-instruction and self-plan settings.Source: §D.1 · p. 21
• Multimodal-Mind2Web evaluates offline website planning with element accuracy, Operation F1, and step success rate, while AndroidControl reports OOD step accuracy for high- and low-level tasks.Source: §D.2 · p. 21
• SeeClick and CogAgent comparisons report only element accuracy because their original evaluations used Mind2Web rather than Multimodal-Mind2Web.Source: §D.2 · p. 21

Web evaluation

• Mind2Web-Live contains 104 real-time web tasks and is evaluated in BrowserGym using task success rate.Source: §D.3 · p. 21
• Mind2Web-Live evaluation verifies clicks through coordinate comparisons with CSS-selector bounding boxes and validates inputs by comparing entered values with expected values.Source: §D.3 · p. 21

Evaluation limitations

• 18 network errors and 6 incomplete reCAPTCHA tasks prevented scoring on 24 Mind2Web-Live tasks.Source: §D.3 · p. 22

Android evaluation

• AndroidWorld spans 20 applications and 116 tasks, but AGUVIS uses only raw screenshots with coordinate-level actions and basic mobile functions.Source: §D.3 · p. 22
• MobileMiniWob instantiates 92 MiniWob++ tasks in AndroidWorld and uses the same observation and action space with real-time task-success evaluation.Source: §D.3 · p. 22

Agent configurations

• Online experiments use GPT-4o with AGUVIS-7B for planning and grounding, or AGUVIS-72B as both planner and grounder.Source: §D.3.1 · p. 22
• The GPT-4o plus AGUVIS-7B setup modifies prompts so GPT-4o can delegate grounding actions and share planning outputs with AGUVIS-7B.Source: §D.3.1 · p. 22

Action interfaces

• The prompt instructions require reviewing the current page state and selecting a correctly formatted next action, including waiting, scrolling, filling, clicking, or messaging.Source: §D.3.1 · p. 23

Training-strategy comparison

• Joint training performs better on GUI grounding but worse on planning datasets than staged training.The reported trade-off appears on MM-Mind2Web and AndroidControl High-level, while overall differences between staged and joint setups are relatively minor.Source: §E.1.1 · p. 25
• Stage 1 grounding data is more abundant, whereas Stage 2 data is higher-quality and better aligned with deployment scenarios.The authors use this data imbalance and alignment difference to justify placing Stage 2 later in training.Source: §E.1.1 · p. 25

Stage contributions

• Stage 2-only training preserves similar MM-Mind2Web and AndroidControl performance but declines on ScreenSpot GUI grounding.The results attribute this stability to Qwen2VL’s natural-image grounding pre-training and emphasize varied Stage 1 grounding data for ScreenSpot.Source: §E.1.1 · pp. 25–26

Cross-model validation

• On LLaVA, performing both stages substantially improves performance and surpasses previous state-of-the-art results.This cross-model result supports the authors’ claim that the training method is model-agnostic across Qwen2-VL and LLaVA.Source: §E.1.1 · p. 26

Cross-device data

• Training with both Web and Mobile data consistently outperforms training exclusively on MM-Mind2Web.All compared settings include MM-Mind2Web fine-tuning, isolating the contribution of broader device-domain data.Source: §E.1.2 · pp. 26–27
• Mobile data improves cross-device domain generalization in the Web domain.The authors interpret the performance gain as evidence supporting their cross-platform data strategy.Source: §E.1.2 · p. 27

Inner monologue

• Inner monologue training produces clear performance gains across the evaluated GUI tasks.The authors attribute these gains to reasoning about the current step and providing context for planning subsequent steps.Source: §E.1.2 · p. 27
• Low-level inner-monologue instructions improve action-execution accuracy on ScreenSpot and AndroidControl low-level tasks.These instructions function as atomic instruction-and-grounding-action pairs, also strengthening grounding ability.Source: §E.1.2 · p. 27

Planning prompts

• Enforced planning resolves approximately 20% of grounding errors by prompting consideration of task context, ambiguities, and available UI elements.The enforced-plan prompt requires high-level reasoning before actions, unlike self-planning, where the model can choose whether to reason first.Source: §E.2.1 · p. 27