Source-linked AI summary
Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents
Haiyang Xu, Xi Zhang, Haowei Liu, Junyang Wang, Zhaozai Zhu, Shengjie Zhou, Xuhao Hu, Feiyu Gao, Junjie Cao, Zihua Wang, Zhiyuan Chen, Jitong Liao, Qi Zheng, Jiahui Zeng, Ze Xu, Shuai Bai, Junyang Lin, Jingren Zhou, Ming Yan
TL;DR
GUI-Owl-1.5 addresses the challenge of building practical GUI agents that efficiently collect data, generalize across platforms, and support tools, memory, and multi-agent interaction. It combines hybrid data collection, unified capability enhancement, and MRPO-based multi-platform reinforcement learning. The open-source models achieve state-of-the-art performance among open-source models across more than 20 GUI benchmarks.
Problem
Practical GUI agents face costly real-world data collection, heterogeneous-platform adaptation, and requirements for tools, memory, multi-agent interaction, and real-time coordination.
Method
The paper develops GUI-Owl-1.5 using a Hybrid Data Flywheel, unified agent-capability enhancement, and MRPO for multi-platform reinforcement learning.
Results
GUI-Owl-1.5 achieves state-of-the-art performance among open-source models across more than 20 GUI benchmarks covering automation, grounding, tool calling, memory, and knowledge.
Takeaways & Limitations
The open-source GUI-Owl-1.5 family supports instruct/thinking variants from 2B to 235B across desktop, mobile, browser, and other devices.
Abstract
from arXiv · showhide
The paper introduces GUI-Owl-1.5, the latest native GUI agent model that features instruct/thinking variants in multiple sizes (2B/4B/8B/32B/235B) and supports a range of platforms (desktop, mobile, browser, and more) to enable cloud-edge collaboration and real-time interaction. GUI-Owl-1.5 achieves state-of-the-art results on more than 20+ GUI benchmarks on open-source models: (1) on GUI automation tasks, it obtains 56.5 on OSWorld, 71.6 on AndroidWorld, and 48.4 on WebArena; (2) on grounding tasks, it obtains 80.3 on ScreenSpotPro; (3) on tool-calling tasks, it obtains 47.6 on OSWorld-MCP, and 46.8 on MobileWorld; (4) on memory and knowledge tasks, it obtains 75.5 on GUI-Knowledge Bench. GUI-Owl-1.5 incorporates several key innovations: (1) Hybird Data Flywheel: we construct the data pipeline for UI understanding and trajectory generation based on a combination of simulated environments and cloud-based sandbox environments, in order to improve the efficiency and quality of data collection. (2) Unified Enhancement of Agent Capabilities: we use a unified thought-synthesis pipeline to enhance the model's reasoning capabilities, while placing particular emphasis on improving key agent abilities, including Tool/MCP use, memory and multi-agent adaptation; (3) Multi-platform Environment RL Scaling: We propose a new environment RL algorithm, MRPO, to address the challenges of multi-platform conflicts and the low training efficiency of long-horizon tasks. The GUI-Owl-1.5 models are open-sourced, and an online cloud-sandbox demo is available at https://github.com/X-PLUG/MobileAgent.
1 Introduction
GUI-Owl-1.5 addresses the efficiency, multi-platform, and capability challenges facing practical GUI agents through scalable data, unified capability enhancement, and multi-platform reinforcement learning. It achieves strong open-source results across automation, grounding, tool invocation, memory, and knowledge benchmarks.
- GUI agents must collect scalable real-world trajectories, operate across heterogeneous platforms, and support advanced capabilities such as tools, memory, and multi-agent interaction.
- GUI-Owl-1.5 provides instruct and thinking variants from 2B to 235B-A22B for edge deployment, complex planning, and edge–cloud collaboration.
- The Hybrid Data Flywheel combines simulated and cloud-based environments to improve UI-understanding and trajectory-data collection efficiency and quality.
- Unified capability enhancement adds GUI knowledge, world modeling, chain-of-thought reasoning, memory management, tool invocation, and multi-agent adaptation.
- 56.5% on OSWorld-Verified, 71.6% on AndroidWorld, and 46.6% on VisualWebArena demonstrate strong automation performance across platforms.
- 47.6% on OSWorld-MCP and 80.3% on ScreenSpot-Pro show strong tool-invocation and grounding performance, respectively.
2 Mobile-Agent-v3.5
GUI-Owl-1.5 extends the previous GUI-Owl with broader actions, improved context retention, and stronger synthetic-data, cross-platform, and agent-capability design. Built on Qwen3-VL, it supports multi-turn interaction across mobile, desktop, and browser interfaces and multi-agent collaboration.
- GUI-Owl-1.5 improves on GUI-Owl through a broader action space, better context retention, and enhanced data-generation, cross-platform, and agent-capability design.
- Built on Qwen3-VL, GUI-Owl-1.5 perceives, plans, decides, and locates interface elements across multi-turn mobile, desktop, and browser interactions.
- The model can also operate collaboratively in multi-agent systems.
2.1 Formulation
GUI-Owl-1.5 formulates GUI operation as closed-loop, multi-turn decision-making: it combines visual observations, user instructions, and hierarchical context to produce action conclusions and executable tool calls. Recent interactions remain detailed while earlier history is compressed for memory efficiency.
- 2.1 Formulation: The agent continuously perceives the environment, executes actions, and adapts its strategy using real-time feedback in a multi-turn interaction loop.
- Input Space: At each step, the input consists of a screenshot and natural-language user instruction.
- Interaction Flow: The system message specifies the action space, while the user message supplies the task, compressed histories, and current observation.
- Output Space: The output contains a natural-language action conclusion and a structured tool call that executes the action.
- Action Space: The expanded action space includes external tool and API calls in addition to primitive operations such as clicking, typing, and scrolling.
- Context Management: A sliding window retains recent multimodal turns while compressing earlier interactions into action-conclusion summaries, balancing task awareness with memory efficiency.
2.2 Data Preparation
The data-preparation pipeline combines trajectory supervision for long-horizon, tool-augmented execution with grounding supervision linking language intents to interface elements. It uses synthetic, mined, simulated, automated, and human-generated data to increase grounding diversity and trajectory fidelity.
- 2.2 Data Preparation: The unified pipeline curates trajectory data for long-horizon, tool-augmented decisions and grounding data aligned with natural-language intents and on-screen elements.
- Grounding Data: Existing grounding datasets lack sufficient complexity and diversity, motivating complementary hard-scenario synthesis and scalable data extension.
- Hard Grounding Data Generation: Hard grounding synthesis generates professional-app screenshots and validated multi-window, high-resolution scenes while preserving target-element visibility.
- High-Quality Grounding Data Extension: Scalable grounding extension mines validated trajectory annotations, tutorial-derived QA pairs, and filtered infeasible-query negatives.
- Trajectory Data Collection: Trajectory collection combines DAG-based task synthesis, validated real-device rollouts, expert demonstrations, and virtual-environment production.
- Task Production: DAGs represent atomic subtasks as nodes and feasible UI-state transitions as edges, enabling realistic multi-step task paths and controllable workflow coverage.
- Automated Trajectory Generation: Checkpointing accepts fully correct rollouts or truncates them at the last verified subtask, preserving clean partial supervision and repairing the remaining task.
- Virtual Environment-Based Production: Virtual environments provide precise subtask feedback and scalable generation while avoiding CAPTCHA, anti-bot, and inaccurate-feedback barriers in real-world exploration.
2.3 Agent Capability Enhancement
GUI-Owl-1.5 enhances GUI agents with knowledge injection, synthesized reasoning, memory, reflection, and multi-agent collaboration for long-horizon interaction. Its pipeline combines action-conditioned world modeling, trajectory-based thought synthesis, and role-specialized execution.
- GUI-Owl-1.5 combines GUI knowledge injection, unified CoT synthesis, and multi-agent collaboration to support planning, consequence reasoning, memory, and tool invocation.
- GUI Knowledge Injection: Action-conditioned descriptions of subsequent screenshots train the model to anticipate interface state changes before executing candidate actions.
- Unified CoT Synthesis: The CoT pipeline generates thoughts and conclusions from screen observations, memory, reflections, task progress, and action outcomes.
- Unified CoT Synthesis: The synthesized trajectories support long-horizon decision-making by reflecting on previous actions and improve memory by recording information needed in later steps.
- Multi-Agent Collaboration: Mobile-Agent-v3.5 uses Manager, Worker, Reflector, and Notetaker roles to separate planning, execution, verification, and persistent memory during iterative task completion.
2.4 Training Paradigm
GUI-Owl-1.5 is initialized from Qwen3-VL and trained through expanded pre-training, supervised fine-tuning, and MRPO reinforcement learning. The paradigm targets multi-device GUI control while addressing rollout collapse, tokenizer inconsistency, and cross-device gradient interference.
- GUI-Owl-1.5 uses a three-stage training process with expanded data diversity and task coverage for multi-platform automation, tool invocation, and complex agentic interactions.
- Pre-training: Pre-training adds QA/VQA knowledge, world-modeling data, and tool-invocation data alongside UI recognition and trajectory data.
- Supervised Fine-tuning: Supervised fine-tuning covers multi-device CoT trajectories, augmented grounding, structured tool and MCP invocation, and browser interaction data.
- Reinforcement Learning: MRPO unifies mobile, desktop, and web learning under a device-conditioned policy while addressing long-horizon training challenges.
- Reinforcement Learning: Online oversample-and-select buffers increase rollout diversity while keeping candidates on-policy, reducing collapsed GRPO groups without replay-induced off-policy bias.
- Reinforcement Learning: Token-id transport aligns environment-side inference with training-side optimization so log-probabilities are evaluated on the executed discrete event.
3 Experiments
GUI-Owl-1.5 is evaluated across multi-platform automation, grounding, knowledge, memory, and tool-use benchmarks, with strong results across these dimensions. Ablations further examine the contributions of virtual-environment trajectories, unified CoT synthesis, and reinforcement-learning strategies.
- Evaluation Setup: GUI-Owl-1.5 is evaluated across grounding, automation, tool invocation, memory, and knowledge dimensions using representative benchmark suites.The evaluation focuses on six model variants and spans multi-device GUI tasks.
- Browser Use: 46.7 on WebArena, 40.8 on VisualWebArena, 78.1 on WebVoyager, and 48.6 on Online-Mind2Web establish strong browser-use results for the 8B Thinking variant.Thinking variants consistently outperform Instruct counterparts on several long-horizon browser tasks, including WebVoyager and Online-Mind2Web.
- Grounding Capability: 80.3 accuracy on ScreenSpot-Pro with crop-based refinement exceeds the model’s 72.9 base score and prior methods on high-resolution grounding tasks.The 32B-Instruct model achieves state-of-the-art performance across the grounding benchmarks discussed.
- Knowledge and Memory: 75.45 overall accuracy on GUI Knowledge Benchmark and 27.1 on MemGUI-Bench show strong knowledge and long-horizon memory performance.The GUI Knowledge result is highest among evaluated models, while the 32B memory result surpasses prior native-agent baselines.
- Detailed Analyses: Removing virtual-environment trajectories reduces PC-Eval from 75.4% to 42.0% and Mobile-Eval from 86.7% to 50.0%.The ablation attributes the degradation to the loss of scalable, high-quality trajectory data for desktop and mobile scenarios.
- Detailed Analyses: Removing unified CoT synthesis lowers OSWorld from 52.9% to 47.4% and AndroidWorld from 71.6% to 65.0%.The two components are complementary: virtual environments improve trajectory coverage and quality, while CoT synthesis improves reasoning and decision-making supervision.
- Reinforcement Learning Analyses: Unstable-task-focused reinforcement learning converges faster and reaches higher final PC validation accuracy than full-dataset training.The comparison also examines simultaneous multi-platform optimization as a training strategy.
4 Conclusion
The paper concludes by presenting GUI-Owl-1.5 as an open-source native GUI agent family spanning multiple model sizes, variants, and device types. Its release is intended to support broader adoption of GUI agents for device automation across platforms.
- Conclusion: GUI-Owl-1.5 provides instruct and thinking variants from 2B to 235B-A22B for desktop, mobile, browser, and other devices.The conclusion frames the family as a native GUI agent model for multi-platform use.
- Conclusion: The paper reports state-of-the-art performance on more than 20 GUI benchmarks covering automation, grounding, tool calling, memory, and knowledge.The conclusion summarizes the evaluation scope rather than introducing a new benchmark result.
- Conclusion: The open-source release is intended to advance adoption of GUI agents for device automation across a wide range of platforms.The paper also provides an online cloud-sandbox demo and illustrates tasks involving stock-price retrieval, spreadsheets, and Python-script execution.