Source-linked AI summary
Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration
Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, Jitao Sang
TL;DR
Mobile device operation tasks challenge MLLMs because single-agent systems must navigate lengthy interleaved histories for both task progress and focus content. Mobile-Agent-v2 uses planning, decision, and reflection agents plus a progress-updated memory unit, and reports improved task completion over Mobile-Agent.
Problem
Single-agent MLLM architectures struggle to navigate task progress and focus content in lengthy interleaved image-text histories during mobile device operations.
Method
Mobile-Agent-v2 combines planning, decision, and reflection agents with a memory unit that stores task-related focus content and supports error correction.
Results
Mobile-Agent-v2 achieves a 55% success rate on advanced instructions versus 20% for Mobile-Agent, and averages a 27% success-rate improvement in English scenarios.
Takeaways & Limitations
The multi-agent architecture improves navigation and operation assistance compared with the single-agent Mobile-Agent architecture.
Takeaways & Limitations
The authors did not use Mobile-Eval because its difficulty was relatively low and Mobile-Agent-v2 achieved 99% accuracy, so they redesigned more challenging tasks.
Abstract
from arXiv · showhide
Mobile device operation tasks are increasingly becoming a popular multi-modal AI application scenario. Current Multi-modal Large Language Models (MLLMs), constrained by their training data, lack the capability to function effectively as operation assistants. Instead, MLLM-based agents, which enhance capabilities through tool invocation, are gradually being applied to this scenario. However, the two major navigation challenges in mobile device operation tasks, task progress navigation and focus content navigation, are significantly complicated under the single-agent architecture of existing work. This is due to the overly long token sequences and the interleaved text-image data format, which limit performance. To address these navigation challenges effectively, we propose Mobile-Agent-v2, a multi-agent architecture for mobile device operation assistance. The architecture comprises three agents: planning agent, decision agent, and reflection agent. The planning agent generates task progress, making the navigation of history operations more efficient. To retain focus content, we design a memory unit that updates with task progress. Additionally, to correct erroneous operations, the reflection agent observes the outcomes of each operation and handles any mistakes accordingly. Experimental results indicate that Mobile-Agent-v2 achieves over a 30% improvement in task completion compared to the single-agent architecture of Mobile-Agent. The code is open-sourced at https://github.com/X-PLUG/MobileAgent.
1 Introduction
Mobile device operation tasks require navigating both task progress and focus content across lengthy interleaved histories. Mobile-Agent-v2 addresses these challenges through specialized agents, memory, and reflection.
- Mobile device operation tasks require continuous multi-step processing from the initial screen until user instructions are fully executed.
- Long histories of interleaved image and text operations and screens make task-progress and focus-content navigation difficult for single-agent architectures.Focus content is task-relevant information retained in previous screens.
- Mobile-Agent-v2 uses planning, decision, and reflection agents to support progress navigation, operation generation, and error handling.
- A memory unit records task-related focus content from history screens and is updated as the task progresses.
- The reflection agent compares screen changes before and after operations and re-executes operations when outcomes fail to meet expectations.
- Dynamic evaluations across operating systems, language environments, and applications report significant improvements for Mobile-Agent-v2, with further gains from manual operation knowledge injection.
2 Related Work
Related work spans multi-agent collaboration and LLM-based agents for web and mobile UI operation. Mobile-Agent-v2 targets long-context navigation challenges in mobile device tasks.
- Prior multi-agent frameworks use role-based collaboration or multiple agents in sandbox environments to support autonomous task execution.
- Web-agent research improves real-world webpage task performance through datasets, self-experience learning, visual understanding, and webpage manipulation.
- Mobile-Agent-v2 introduces a multi-agent architecture for long-context navigation in mobile device operation tasks.
3 Mobile-Agent-v2
Mobile-Agent-v2 uses specialized agents, visual perception, and memory to navigate task progress and focus content during iterative mobile-device operation. A reflection stage evaluates operations and updates device state or history according to the outcome.
- Visual Perception Module: The visual perception module converts screenshots into screen text, icon information, and coordinates for downstream operation generation.It uses text recognition, icon recognition, and icon description tools.
- Planning Agent: The planning agent summarizes history operations into task progress so the decision agent can focus on unfinished subtasks.Its inputs include the instruction, focus content, previous operation, and previous task progress.
- Decision Agent: The decision agent generates and executes device operations while updating focus content, selecting actions from a restricted operation space.Available actions include opening apps, tapping, swiping, typing, returning home, and stopping.
- Memory Unit: The memory unit stores task-related focus content from screens and is updated by the decision agent for reference in subsequent decisions.This supports tasks requiring information observed earlier, including information gathered across multiple apps.
- Reflection Agent: The reflection agent compares screen states before and after an operation, classifying it as erroneous, ineffective, or correct.Erroneous operations revert the page, ineffective operations leave it unchanged, and only correct operations update operation history and page state.
4 Experiments
Experiments evaluate Mobile-Agent-v2 through dynamic mobile-device tasks, component ablations, model comparisons, and operation-sequence analysis. Results show improved task completion, benefits from reflection and memory, and stronger handling of long sequences.
- Evaluation Setup: Dynamic evaluation uses real Harmony OS and Android devices, with ADB simulating operations across non-English and English scenarios.The study evaluates five system apps and five popular apps in each scenario.
- Evaluation Setup: The evaluation measures success rate, completion rate, decision accuracy, and reflection accuracy.Success rate requires fulfilling all instruction requirements, while completion rate measures correct operations against ground truth.
- Task Completion: 55% success rate on challenging advanced instructions versus 20% for Mobile-Agent, while English-scenario success rate improves by an average of 27%.Mobile-Agent-v2 improves over the single-agent baseline in both basic and advanced instructions and across language scenarios.
- App Type: 37.5% SR and 44.2% CR improvements over Mobile-Agent occur in multiple-app tasks.The authors connect these gains with multi-agent collaboration and the memory unit, which support history-operation and focus-content retrieval.
- Ablation Study: Removing planning, reflection, or memory reduces performance, with planning having the largest overall impact.Reflection corrects erroneous operations, while memory supports multi-app execution and localization of critical UI elements.
- Operation Sequence Analysis: Errors in Mobile-Agent cluster in later task stages, whereas Mobile-Agent-v2 shows no obvious positional pattern.The analysis divides relative operation positions into three equal parts and associates the difference with long-sequence navigation.
5 Conclusion
Mobile-Agent-v2 addresses navigation difficulties in single-agent mobile operation assistants through multi-agent collaboration, with reported performance improvements over Mobile-Agent. The paper also identifies manual operation knowledge as a direction for further enhancement.
- Mobile-Agent-v2 uses multi-agent collaboration to address navigation difficulties in single-agent mobile device operation assistants.
- Mobile-Agent-v2 achieves significant performance improvements compared to the single-agent Mobile-Agent.
- Manual operation knowledge can further enhance performance, providing a direction for future work.
- The evaluated tasks cover commenting, searching, following, replying, and content-specific post retrieval across Weibo, Bilibili, WhatsApp, X, TikTok, and Little Red Book.
A.1 Evaluation Application and Instruction
The evaluation uses English and non-English mobile applications with basic and advanced instructions, while redacting privacy-sensitive application details.
- The dynamic evaluation covers English and non-English scenarios using applications and instructions listed in Tables 5 and 6.
- Basic instructions involve relatively simple operations with clear instructions within an app interface.
- Advanced instructions require a certain level of experience with app operations to complete.
- Privacy-sensitive application details are hidden, with “xxx” marking redacted information in the evaluation tables.
A.2 Agent Prompt
The agent prompts assign distinct inputs and roles across planning, decision, and reflection stages, with separate planning prompts for initial and subsequent operations.
- The system and user prompts define the planning, decision, and reflection agents across the multi-agent workflow.
- Initial and subsequent planning operations use separate prompts because the history of operations is empty at task initiation.
- The planning agent uses text-only history operations because planning and historical navigation are purely textual processes.
- The decision agent receives a screenshot of the current device state when generating an operation.
- The reflection agent receives the previous and current device-state screenshots in chronological order.