PaperlayerView exact revision on arXiv

Source-linked AI summary

Mobile-Agent-v3: Fundamental Agents for GUI Automation

Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, Junjie Cao, Zhengxi Lu, Jitong Liao, Qi Zheng, Fei Huang, Jingren Zhou, Ming Yan

arXiv:2508.15144v2cs.AI

TL;DR

GUI automation needs foundational agents that combine perception, reasoning, planning, grounding, and reliable action execution across diverse environments. This paper develops GUI-Owl with self-evolving data pipelines and scalable reinforcement learning, then integrates it into Mobile-Agent-v3. GUI-Owl and Mobile-Agent-v3 achieve strong cross-benchmark performance, including 37.7 on OSWorld and 73.3 on AndroidWorld for the combined framework.

  • Problem

    Existing GUI-agent approaches struggle with unfamiliar tasks, faithful instruction following, or compatibility with diverse agent frameworks, motivating a more capable foundational model.

  • Method

    GUI-Owl combines end-to-end multimodal GUI capabilities with self-evolving trajectory production, diverse capability datasets, scalable asynchronous reinforcement learning, and multi-agent coordination in Mobile-Agent-v3.

  • Results

    37.7 on OSWorld and 73.3 on AndroidWorld are achieved when GUI-Owl is combined with Mobile-Agent-v3, while GUI-Owl also shows state-of-the-art performance across broad GUI benchmarks.

  • Takeaways & Limitations

    GUI-Owl functions as both an autonomous GUI agent and a modular foundation for specialized standalone calls or collaborative multi-agent workflows.

Abstract

from arXiv · show

This paper introduces GUI-Owl, a foundational GUI agent model that achieves state-of-the-art performance among open-source end-to-end models on ten GUI benchmarks across desktop and mobile environments, covering grounding, question answering, planning, decision-making, and procedural knowledge. GUI-Owl-7B achieves 66.4 on AndroidWorld and 29.4 on OSWorld. Building on this, we propose Mobile-Agent-v3, a general-purpose GUI agent framework that further improves performance to 73.3 on AndroidWorld and 37.7 on OSWorld, setting a new state-of-the-art for open-source GUI agent frameworks. GUI-Owl incorporates three key innovations: (1) Large-scale Environment Infrastructure: a cloud-based virtual environment spanning Android, Ubuntu, macOS, and Windows, enabling our Self-Evolving GUI Trajectory Production framework. This generates high-quality interaction data via automated query generation and correctness validation, leveraging GUI-Owl to refine trajectories iteratively, forming a self-improving loop. It supports diverse data pipelines and reduces manual annotation. (2) Diverse Foundational Agent Capabilities: by integrating UI grounding, planning, action semantics, and reasoning patterns, GUI-Owl supports end-to-end decision-making and can act as a modular component in multi-agent systems. (3) Scalable Environment RL: we develop a scalable reinforcement learning framework with fully asynchronous training for real-world alignment. We also introduce Trajectory-aware Relative Policy Optimization (TRPO) for online RL, achieving 34.9 on OSWorld. GUI-Owl and Mobile-Agent-v3 are open-sourced at https://github.com/X-PLUG/MobileAgent.

1 Introduction

GUI-Owl is a foundational end-to-end multimodal agent for GUI automation, designed to unify core capabilities and support cross-platform and multi-agent operation. Mobile-Agent-v3 extends this foundation with scalable data production, diverse capability construction, and reinforcement learning.

  • Motivation: Existing GUI-agent approaches either rely on closed-source models that struggle with unfamiliar tasks or end-to-end models that lack faithful instruction following and framework compatibility.These limitations motivate a foundational model with strong UI perception, planning, reflection, and reasoning capabilities.
  • GUI-Owl: GUI-Owl unifies perception, grounding, reasoning, planning, and action execution within a single policy network.It supports multi-turn decision making, autonomous operation, and role-specific deployment in multi-agent systems.
  • Large-scale Environment Infrastructure: The training infrastructure spans mobile, PC, and web environments and supports a self-evolving GUI trajectory production pipeline.The infrastructure uses cloud-based phones and computers to create diverse, realistic interaction scenarios and collect training data.
  • Diverse Foundational Agent Capabilities: The capability pipelines target UI grounding, task planning, and action semantics to strengthen foundational GUI-agent abilities.These pipelines cover localization, fine-grained grounding, procedural knowledge, and long-horizon multi-application tasks.
  • Scalable Environment RL: The scalable reinforcement learning framework decouples experience generation from policy updates and supports fully asynchronous training.This design is intended to align GUI-Owl’s decision policy with real-world task success.
  • Results: 34.9 on OSWorld-Verified and 66.4 on AndroidWorld are achieved by GUI-Owl-7B, while Mobile-Agent-v3 reaches 37.7 on OSWorld and 73.3 on AndroidWorld.GUI-Owl-32B also surpasses models including GPT-4o and Claude 3.7 on MMBench-GUI and AndroidControl.

2 GUI-Owl

GUI-Owl performs end-to-end GUI interaction by mapping observations and histories to actions, while its surrounding data pipelines generate, validate, and organize training examples. These pipelines cover self-evolving trajectories, grounding, and task planning.

  • GUI-Owl: GUI-Owl unifies perception, planning, decision-making, and grounding for GUI interaction across mobile, PC, and web platforms.It can operate autonomously or assume specialized roles in multi-agent systems.
  • End-to-end GUI interactions: At each step, GUI-Owl selects an action from the environment’s action space using the current observation and history of past operations.The selected action is executed to produce the next environment observation.
  • End-to-end GUI interactions: The interaction prompt supplies the task, historical information, and observations, while the model response contains reasoning and action output.Actions are translated into device commands such as ADB commands for Android or pyautogui code for desktops.
  • Self-Evolving Trajectory Production: The self-evolving pipeline generates trajectories in virtual environments, evaluates them for correctness, and uses filtered data to improve the model.It is designed to reduce reliance on manual annotation through repeated rollout and validation.
  • Diverse GUI Data Synthesis: The grounding pipeline combines UI-element localization with fine-grained word and character localization.It draws on open-source datasets, accessibility-tree information, and OCR-derived spatial annotations.
  • Task Planning: Task-planning data is distilled from successful historical trajectories into task-execution manuals whose quality is evaluated by changes in task completion rate.The process converts page transitions and actions into procedural knowledge for complex tasks.

3 Training Paradigm

GUI-Owl is trained through progressive pre-training, iterative tuning, and reinforcement learning, with infrastructure designed for both single-turn and multi-turn interaction. Online RL uses trajectory-level rewards to improve learning for long-horizon GUI tasks.

  • Training Paradigm: GUI-Owl follows a three-stage training process covering pre-training, iterative tuning, and reinforcement learning.The stages progressively target GUI understanding, reasoning, and robust execution.
  • Pre-training Phase: Pre-training uses UI understanding, interaction trajectory, and general reasoning data to strengthen grounding, action prediction, and reasoning.The model is continually pre-trained from Qwen2.5-VL.
  • Iterative Tuning Phase: Iterative tuning deploys the model in desktop and mobile environments, then cleans, scores, and transforms resulting trajectories into reasoning datasets.The resulting offline training is intended to improve adaptability and decision-making across varied UI scenarios.
  • Reinforcement Learning Phase: The RL phase uses asynchronous interaction with real environments to reinforce successful behaviors and improve execution consistency.The framework is intended to increase both success rate and stability in practical deployments.
  • Scalable RL Infrastructure: The training infrastructure unifies single-turn reasoning and multi-turn agentic tasks through a modular task interface and decoupled rollout-update process.Its components can run in parallel, while rollout managers coordinate task IDs, trajectories, rewards, and data flow.
  • Trajectory-aware Relative Policy Optimization: TRPO uses whole-trajectory rewards to compute step-level advantages for online GUI reinforcement learning.The method addresses long, variable-length action sequences and sparse delayed rewards without assigning per-step rewards directly.
  • Trajectory-aware Relative Policy Optimization: A replay buffer replaces an entirely failed trajectory group with a successful trajectory from the same task when available.This injects positive examples into training batches to stabilize learning under sparse success.

4 Mobile-Agent-v3

Mobile-Agent-v3 coordinates specialized agents in a feedback-driven loop for adaptive, long-horizon GUI automation across phone and PC environments.

  • Architecture: The framework coordinates Manager, Worker, Reflector, and Notetaker agents for robust, adaptive, and long-horizon GUI automation.The architecture also includes RAG for external knowledge and a GUI device interface supporting phone and PC environments.
  • Manager Agent: The Manager decomposes high-level instructions into ordered subgoals and dynamically revises them using external knowledge, execution results, and feedback.It can re-prioritize, modify, or insert corrective subgoals during execution.
  • Worker Agent: The Worker selects and executes relevant subgoals from the current GUI state, prior feedback, and accumulated notes, producing reasoning, action, and intent.Its output is an action tuple recording the reasoning, action, and intent associated with each step.
  • Reflector Agent: The Reflector compares intended and actual state transitions, classifies outcomes as SUCCESS or FAILURE, and sends causal feedback to the Manager.This supplies the loop’s self-correction mechanism.
  • Notetaker Agent: The Notetaker stores critical screen elements after successful steps, preserving transient context for later planning and execution.Its cumulative memory can include codes and credentials and is updated only on SUCCESS.

5 Experiments

Experiments evaluate GUI-Owl across grounding, GUI understanding, end-to-end interaction, multi-agent use, and online reinforcement learning. The models achieve strong benchmark performance, while trajectory-level experience management improves training stability.

  • Evaluation Scope: GUI-Owl is evaluated in four dimensions: grounding, comprehensive GUI understanding, end-to-end agent capability, and multi-agent capability.The evaluation uses GUI benchmarks and realistic interactive environments, including AndroidWorld and OSWorld-Verified.
  • Grounding Capability: 54.9 on ScreenSpot-Pro demonstrates GUI-Owl-7B’s grounding performance on ultra-high-resolution images, exceeding UI-TARS-72B and Qwen2.5-VL 72B.GUI-Owl-7B achieves state-of-the-art performance among 7B models, while GUI-Owl-32B leads at its size.
  • Grounding Capability: 80.49 on MMBench-GUI-L2 substantially outperforms existing models on a broad and challenging grounding query set.GUI-Owl-32B reaches 82.97 and demonstrates leading grounding capabilities across domains.
  • Comprehensive GUI Understanding: 90.9 on MMBench-GUI-L1’s hard level accompanies scores of 84.5 and 86.9 on the easy and medium levels, respectively.On Android Control, GUI-Owl achieves 72.8 and the highest performance among 7B models.
  • Online Environment: 73.3 on AndroidWorld and 37.7 on OSWorld-Verified are achieved by GUI-Owl-32B integrated into Mobile-Agent-v3.The results show the model can operate independently and integrate into a multi-agent framework.
  • Ablation Study: 31.5% peak performance in the Online Filtering ablation is volatile and inferior because it removes successful-trajectory replay and leftover-rollout reuse.The full method retains and reuses generated rollouts for a richer training signal.
  • Filtering Strategies: 29.1 success is where Offline Filtering initially stagnates before degrading, whereas dynamic online filtering adapts the training distribution to the evolving policy.Long action sequences make final rewards sensitive to minor policy changes, contributing to offline filtering’s instability and overfitting.

5.3 Effect of Reasoning Data Synthesis

The paper studies offline and iterative online reasoning-data synthesis for GUI-Owl. Incremental data additions and repeated training with updated trajectories both improve AndroidWorld performance.

  • Data Synthesis: Reasoning-data synthesis combines offline hint-guided rejection sampling, multi-agent distillation, general-purpose reasoning SFT data, and iterative online sampling.The components are analyzed separately on AndroidWorld.
  • Offline Synthesis: Adding offline rejection-sampling data, multi-agent distillation, and general reasoning SFT data steadily improves AndroidWorld performance.General reasoning data provides a modest gain while helping maintain general reasoning capability.
  • Iterative Training: Iterative training with newly updated trajectory data produces sustained performance improvements from the same checkpoint.As reasoning improves, more training-query tasks are completed, increasing data diversity and supporting more robust reasoning capability.
  • Agentic Framework Adaptability: GUI-Owl-32B reaches 62.1 on AndroidWorld and 48.4 on OSWorld when used as the core vision model in established agentic frameworks.The evaluation tests adaptability across mobile and desktop platforms.

6 Details of Self-Evolving Trajectory Data Production

The self-evolving pipeline produces GUI interaction data across diverse environments by generating queries, executing rollouts, validating trajectories, and refining the resulting data. It targets costly manual annotation while supporting realistic mobile and computer tasks.

  • Manual annotation of GUI trajectories is tedious and costly, motivating automated self-evolving data production.
  • The pipeline constructs virtual environments, generates diverse queries, executes GUI-Owl actions, and produces rollout trajectories for validation.
  • Self-Evolution: Reinforcement fine-tuning feeds processed trajectories back into GUI-Owl, improving rollout generation and progressively reducing reliance on manual collection.
  • Mobile: Mobile query generation samples realistic paths from a human-annotated screenshot-action DAG, extracts metadata, and synthesizes constrained instructions with an LLM.
  • Computer: Computer query generation targets both atomic operations and multi-step software pathways, including clicking, input, dragging, scrolling, and configuration changes.
  • Trajectory Correctness Judgment: A trajectory correctness module evaluates rollouts and cleans erroneous steps, while step-level evaluation identifies actions requiring correction.

7.1 Core Components and Formalism

Mobile-Agent-v3 models GUI interaction as a state-driven process with explicit device, subgoal, action, feedback, and note variables. These variables formalize how agents represent progress, execute operations, assess outcomes, and retain information.

  • 7.1 Core Components and Formalism: The framework formalizes GUI interaction and task completion through state variables and specialized functions for constituent agents.
  • 7.1.1 State Variables and Definitions: Device state is represented as a screen tensor, while subsequent and completed subgoals track pending work and verified progress.
  • 7.1.1 State Variables and Definitions: An action contains a textual rationale, a concrete low-level command, and a concise summary of its intended effect.
  • 7.1.1 State Variables and Definitions: Reflection feedback records whether an action succeeded or failed and can include diagnostic text explaining failure.
  • 7.1.1 State Variables and Definitions: Notes preserve critical transient information from GUI states so later decisions can use information no longer visible on screen.

7.2 Agent Architecture in Detail

The architecture combines retrieval-augmented planning with specialized Manager, Worker, Reflector, and Notetaker agents. Together, they initialize plans, execute actions, evaluate transitions, preserve useful information, and update plans iteratively.

  • 7.2.1 External Knowledge Retrieval with RAG: RAG converts user instructions into search queries, retrieves documents, summarizes them, and supplies the resulting knowledge to the Manager.
  • 7.2.2 The Manager Agent: The Manager decomposes the instruction into subgoals and dynamically revises their order or content using execution outcomes and diagnostic feedback.
  • 7.2.3 The Worker Agent: The Worker selects a feasible high-priority subgoal, inspects the current screen, and translates guidance into a concrete action.
  • 7.2.4 The Reflector Agent: The Reflector compares pre-action and post-action states with the Worker’s intent to classify success or failure and provide causal feedback.
  • 7.2.5 The Notetaker Agent: The Notetaker records vital information after successful actions and makes cumulative notes available to the Manager and Worker.
  • 7.2.6 Execution Loop: Algorithm 1 initializes the agents and iterates through execution, reflection, note persistence, and Manager plan updates until termination.
  • 7.2.6 Execution Loop: The loop preserves notes after success and leaves them unchanged after failure before updating the pending and completed subgoals.

7.3 Integrated Workflow and Algorithm

Mobile-Agent-v3 uses a cyclical workflow that repeatedly plans, acts, reflects, records information, and updates the plan. Execution ends when all subgoals are complete or when no executable subgoal remains after retries and revisions.

  • The framework begins with a high-level instruction and cycles through action execution, state transitions, reflection, note-taking, and plan updates.
  • Termination Conditions: Task completion occurs when the Manager’s pending subgoal list becomes empty.
  • Termination Conditions: Execution reaches a stalemate when the Worker cannot execute any pending subgoal in the current state despite retries or plan revisions.
  • The structured reflective loop is designed to navigate complex interaction sequences and handle unexpected events while pursuing the user’s goal.

7.4 Case Study

A complete Mobile-Agent-v3 desktop operation shows coordinated manager, worker, and reflector behavior, including successful recovery after a failed click.

  • The manager continually updates subgoals while the worker produces actions guided by those subgoals.
  • A successful reflection identifies the worker’s failed click and provides corrective feedback.

8 Conclusion

The paper presents GUI-Owl as a scalable multimodal GUI agent unifying core automation capabilities, with Mobile-Agent-v3 supporting autonomous and collaborative decision-making.

  • GUI-Owl unifies perception, grounding, reasoning, planning, and action execution within a single scalable framework.
  • The framework supports versatile decision-making from autonomous single-agent execution to collaborative multi-agent role coordination.
  • Figure 11 presents the format of the end-to-end training data.
  • Figure 12 presents a complete Mobile-Agent-v3 operation flow involving manager, worker, and reflector outputs.
Loading 2508.15144v2…