Source-linked AI summary
Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks
Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, Heng Ji
TL;DR
Mobile-Agent-E addresses mobile agents’ difficulty with complex, reasoning-intensive, long-horizon tasks and their lack of learning from prior experience. It uses hierarchical multi-agent planning and execution with persistent Tips and Shortcuts, alongside the Mobile-Eval-E benchmark. The approach achieves a 22.1% average absolute gain over previous state-of-the-art approaches across three foundation model backbones.
Problem
Current mobile agents struggle with complex real-world tasks involving reasoning, long-horizon planning, and learning from prior experiences.
Method
Mobile-Agent-E combines hierarchical planning and low-level execution with persistent Tips and Shortcuts updated from interaction histories.
Results
22.1% average absolute gain over previous state-of-the-art approaches was achieved across three foundation model backbones.
Takeaways & Limitations
Mobile-Agent-E provides a framework for improving planning, error recovery, and efficiency on complex real-world mobile tasks.
Takeaways & Limitations
The paper identifies risks from unauthorized or unintended actions involving sensitive information and calls for stronger privacy and safety mechanisms.
Abstract
from arXiv · showhide
Smartphones have become indispensable in modern life, yet navigating complex tasks on mobile devices often remains frustrating. Recent advancements in large multimodal model (LMM)-based mobile agents have demonstrated the ability to perceive and act in mobile environments. However, current approaches face significant limitations: they fall short in addressing real-world human needs, struggle with reasoning-intensive and long-horizon tasks, and lack mechanisms to learn and improve from prior experiences. To overcome these challenges, we introduce Mobile-Agent-E, a hierarchical multi-agent framework capable of self-evolution through past experience. By hierarchical, we mean an explicit separation of high-level planning and low-level action execution. The framework comprises a Manager, responsible for devising overall plans by breaking down complex tasks into subgoals, and four subordinate agents--Perceptor, Operator, Action Reflector, and Notetaker--which handle fine-grained visual perception, immediate action execution, error verification, and information aggregation, respectively. Mobile-Agent-E also features a novel self-evolution module which maintains a persistent long-term memory comprising Tips and Shortcuts. Tips are general guidance and lessons learned from prior tasks on how to effectively interact with the environment. Shortcuts are reusable, executable sequences of atomic operations tailored for specific subroutines. The inclusion of Tips and Shortcuts facilitates continuous refinement in performance and efficiency. Alongside this framework, we introduce Mobile-Eval-E, a new benchmark featuring complex mobile tasks requiring long-horizon, multi-app interactions. Empirical results show that Mobile-Agent-E achieves a 22% absolute improvement over previous state-of-the-art approaches across three foundation model backbones. Project page: https://x-plug.github.io/MobileAgent.
1. Introduction
Mobile-Agent-E addresses the mismatch between current mobile agents and complex real-world tasks by combining hierarchical planning, multi-agent execution, and experience-based self-evolution. It is paired with Mobile-Eval-E, a harder benchmark for long-horizon, multi-app interactions, and improves substantially over prior approaches.
- Motivation: Real-world mobile tasks require intensive reasoning, long-horizon planning across apps, and exploration under vague instructions.Existing tasks are typically short, straightforward, and goal-oriented.
- Motivation: Current mobile agents lack mechanisms to learn from prior interactions, repeating mistakes and allocating similar resources on each attempt.The paper contrasts this behavior with humans, who become faster and more accurate through repeated use.
- Framework: Mobile-Agent-E separates high-level subgoal planning from low-level action execution through a Manager and four subordinate agents.The subordinate agents are the Perceptor, Operator, Action Reflector, and Notetaker.
- Framework: Its self-evolution module persistently stores Tips and Shortcuts, which Experience Reflectors update from task histories for future planning and action decisions.Tips encode general guidance, while Shortcuts provide executable atomic-operation sequences for recurring subroutines.
- Benchmark: Mobile-Eval-E targets complex real-world tasks, with more than twice the expected operations per task and substantially more multi-app interactions than previous benchmarks.It also introduces Satisfaction Score and an SSS curve for evaluating human-aligned performance and efficiency.
- Results: 22.1% average absolute gain over previous state-of-the-art approaches was achieved across three foundation model backbones.The framework also showed self-evolution gains and reduced computational overhead through Shortcuts.
2. Mobile-Agent-E
Mobile-Agent-E separates high-level planning from low-level execution through specialized agents, while feedback and information aggregation support each action step. Its self-evolution module preserves task experience as Tips and Shortcuts for later use.
- Hierarchical Multi-Agent Framework: Mobile-Agent-E separates high-level planning by the Manager from low-level actions selected by the Operator.The Manager updates the overall plan and identifies the next subgoal, while the Operator selects concrete actions using current perception, history, errors, and memory.
- Hierarchical Multi-Agent Framework: After repeated failures, an Error Escalation Flag prompts the Manager to address the problem by refining the overall plan or adjusting the current subgoal.Before escalation, the Operator attempts to rectify errors using recent action outcomes and error history.
- Hierarchical Multi-Agent Framework: The Perceptor extracts fine-grained visual information from screenshots, while the Operator uses it and the original screenshot to generate action parameters.The action space includes atomic operations such as Open App, Tap, Swipe, Type, Enter, Switch App, Back, Home, and Wait, as well as Shortcuts.
- Hierarchical Multi-Agent Framework: The Action Reflector verifies action outcomes and supplies error feedback, while the Notetaker aggregates task-relevant information during navigation.The Action Reflector can report successful or partially successful outcomes, wrong-page failures, or failures with no change; some actions may require repetition.
- Self-Evolution Module: The self-evolution module updates persistent long-term memory through two Experience Reflectors using the task query, plan, histories, and existing Tips and Shortcuts.Tips provide general interaction guidance from trial-and-error experience, whereas Shortcuts are executable atomic-operation sequences for recurring subroutines and require state-specific preconditions.
3. Experiments
The experiments evaluate Mobile-Eval-E on real devices using human-assessed task and action metrics, comparing Mobile-Agent-E with established mobile-agent baselines and multiple LMM backbones. The benchmark emphasizes complex, long-horizon, multi-app tasks, while experiments also test self-evolution across sequential tasks.
- A More Challenging Benchmark: Mobile-Eval-E contains 25 manually crafted tasks across five real-world scenarios, emphasizing reasoning-intensive, long-horizon, multi-app interactions.The scenarios include restaurant recommendation, information searching, online shopping, trending topics, and travel planning.
- A More Challenging Benchmark: 76% of Mobile-Eval-E tasks require multiple-app interactions, compared with less than 10% in previous benchmarks.The benchmark also has more than twice the expected operations per task and a greater total number of operations than previous benchmarks.
- Evaluation Setup: The evaluation uses human review of recorded screenshots and action histories to measure Satisfaction Score, Action Accuracy, Reflection Accuracy, and Termination Error.Dynamic evaluation runs agents in real time on an Android phone controlled through Android Debug Bridge.
- Evaluating Self-Evolving Mobile Agents: Cross-task self-evolution is evaluated by sharing persistent memory across five sequential tasks within each scenario.Experience Reflectors update memory after each task using the current interaction history and the remaining task queries, allowing later tasks to accumulate Tips and Shortcuts.
- Models: Mobile-Agent-E is compared with AppAgent, Mobile-Agent-v1, and Mobile-Agent-v2 using aligned atomic operations, perception models, and initial Tips where applicable.The comparison uses GPT-4o as the backbone for the state-of-the-art evaluation, with additional experiments using GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-pro.
4. Results
Mobile-Agent-E outperforms prior state-of-the-art agents on complex mobile tasks, while self-evolution improves performance over time and Shortcuts reduce its computational overhead.
- Performance: 22.1% higher Satisfaction Score than the previous multi-agent state-of-the-art is achieved with an identical GPT-4o backbone.
- Performance: 33.9% improvement against the previous SOTA results when self-evolution is enabled, alongside lower termination error.
- Performance: 22.1% and 15.6% average absolute gains are observed with and without evolution across GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-pro.
- Efficiency: A steeper and higher Satisfaction Score vs Steps curve indicates that the agent reaches higher satisfaction with fewer steps.
- Self-evolution: Self-evolution gains generally become larger for later tasks as accumulated Tips and Shortcuts grow, though gains are not strictly monotonic because task difficulty varies.
- Efficiency: Shortcuts mitigate the hierarchical framework’s computational overhead by executing multiple atomic operations within one decision-making iteration.
- Self-evolution: Tips provide distinctive benefits even on instances where no newly generated Shortcuts are used.
5. Related Work
Mobile-Agent-E differs from earlier GUI-agent designs by explicitly separating high-level planning from low-level action decisions, supporting long-horizon mobile interaction and self-evolution through retrieved experience.
- GUI Agents: Earlier Mobile-Agent-v2 combines deciding what to do next with deciding where to tap, whereas Mobile-Agent-E assigns these functions across a hierarchy.
- GUI Agents: Mobile-Agent-E introduces an explicit hierarchy that separates high-level planning from low-level action decisions.
- Self-evolution: The framework’s self-evolution uses retrieved Tips and Shortcuts to complete an unseen challenging task and recover from an error.
6. Conclusion and Future Work
The paper concludes that Mobile-Agent-E improves long-term planning, error recovery, and efficiency on complex real-world mobile tasks through hierarchy and self-evolution, while identifying Shortcut reliability and safety as remaining concerns.
- Conclusion: Mobile-Agent-E is reported to enhance long-term planning, error recovery, and efficiency across complex real-world mobile tasks.
- Future Work: Remaining limitations include invalid Shortcut preconditions and erroneous agent-generated Shortcuts.
- Future Work: Future work targets improved Shortcut generation, invocation, and revision, along with personalization and stronger safety precautions.
Impact Statement
The framework may improve mobile task efficiency and accessibility, but autonomous actions create privacy and safety risks that require safeguards, fail-safe systems, and user-oriented protections.
- Potential Benefits: Mobile agents may assist users with physical impairments, cognitive challenges, or difficulty performing precise touchscreen interactions.
- Risks: Autonomous decision-making creates risks including unintended actions and misuse of sensitive information such as credit-card details or private data.
- Safeguards: Robust safeguards, error recovery, and fail-safe systems are needed to align agent actions with user intentions.
- Future Protections: Planned protections include explicit consent workflows, encryption during processing and storage, and automated flagging of harmful or unauthorized actions.
- Illustrative Outcome: A trajectory comparison reports that Mobile-Agent-E fulfills all rubrics while Mobile-Agent-v2 terminates early after interacting with two apps.
B. Error Recovery with Escalation to Manager
Error escalation strengthens recovery when the Operator makes an incorrect Shortcut call during task execution.
- Figure 9 illustrates error escalation as a mechanism for improving error recovery ability.The example concerns searching for multiple items on Walmart and an Operator error involving stale search text.
C.1. Misuse of Shortcuts due to Incorrect Perception of Phone State
Mobile-Agent-E can misuse Shortcuts when the Operator incorrectly perceives the phone state, despite explicit precondition verification.
- Incorrect phone-state perception can cause Shortcuts to be used in invalid states.The Operator is required to verify a Shortcut's precondition, but perception errors can still lead to misuse.
- A dedicated precondition-verification agent or stronger perception module could potentially mitigate this error.
C.2. Errors and Imperfections in Self-Evolved Shortcuts
Self-evolution can produce Shortcuts that contain errors or unnecessary actions, allowing mistakes to propagate when reused in later tasks.
- Agent-generated Shortcuts can remain erroneous or imperfect despite being effective in most cases.Examples include unnecessary actions and omitted actions required for successful execution.
- Erroneous Shortcuts can propagate errors when they are used in subsequent tasks.
- Future work should improve Shortcut generation and enable reflection on and revision of generated Shortcuts.
D. All Tasks in Mobile-Eval-E Benchmark
The supplementary materials document Mobile-Eval-E tasks, the atomic operation space, generated Tips and Shortcuts, and trajectory and error-recovery examples.
- All Tasks in Mobile-Eval-E Benchmark: Mobile-Eval-E task materials include input queries, involved app types, and scenarios for all tasks.Complete rubrics and human reference operation sequences are provided in supplementary material.
- All Tasks in Mobile-Eval-E Benchmark: The benchmark documentation includes the atomic operations considered by Mobile-Agent-E.
- All Tasks in Mobile-Eval-E Benchmark: Mobile-Agent-E generates Shortcuts and Tips after self-evolution across all 25 Mobile-Eval-E tasks.Figures 12 and 13 present the complete generated lists.
- All Tasks in Mobile-Eval-E Benchmark: The supplementary figures compare full trajectories and illustrate error recovery, Shortcut misuse, and generated Shortcut imperfections.The examples include Walmart search, invalid app state, and unnecessary or omitted actions.
- All Tasks in Mobile-Eval-E Benchmark: An example Shortcut specifies arguments, a description, a precondition, and an atomic action sequence for searching a location in Google Maps.
- All Tasks in Mobile-Eval-E Benchmark: The materials include initial user-provided Tips and agent-generated Tips for mobile-task execution.Examples address payment information, app state, search specificity, business hours, and filtering.