PaperlayerView exact revision on arXiv

Source-linked AI summary

Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks

Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, Heng Ji

arXiv:2501.11733v2cs.CLcs.CV

TL;DR

Current mobile agents and benchmarks focus largely on short, straightforward tasks, unlike real-world tasks that require intensive reasoning, long-horizon planning, and exploration. Mobile-Agent-E pairs a hierarchical multi-agent framework with persistent Tips and reusable Shortcuts updated from task histories. It achieves a 22.1% average absolute gain over previous state-of-the-art approaches across three foundation model backbones.

  • Problem

    Existing mobile agents and benchmarks largely address short, straightforward tasks, while real-world tasks demand intensive reasoning, long-horizon planning, and exploration across apps.

    Source: §1 · p. 2, §3.1 · p. 6

  • Method

    Mobile-Agent-E pairs a hierarchical multi-agent framework with persistent Tips and reusable Shortcuts updated from task histories.

    Source: §1 · p. 2, §6 · p. 11

  • Results

    22.1% average absolute gain over previous state-of-the-art approaches across three foundation model backbones.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Mobile-Agent-E is reported to enhance long-term planning, error recovery, and efficiency on complex real-world tasks.

    Source: §6 · p. 11

    Autonomous mobile agents can take unauthorized or unintended actions involving sensitive information, making robust safeguards and fail-safe systems important.

    Source: Impact Statement · p. 11

Abstract

from arXiv · show

Smartphones have become indispensable in modern life, yet navigating complex tasks on mobile devices often remains frustrating. Recent advancements in large multimodal model (LMM)-based mobile agents have demonstrated the ability to perceive and act in mobile environments. However, current approaches face significant limitations: they fall short in addressing real-world human needs, struggle with reasoning-intensive and long-horizon tasks, and lack mechanisms to learn and improve from prior experiences. To overcome these challenges, we introduce Mobile-Agent-E, a hierarchical multi-agent framework capable of self-evolution through past experience. By hierarchical, we mean an explicit separation of high-level planning and low-level action execution. The framework comprises a Manager, responsible for devising overall plans by breaking down complex tasks into subgoals, and four subordinate agents--Perceptor, Operator, Action Reflector, and Notetaker--which handle fine-grained visual perception, immediate action execution, error verification, and information aggregation, respectively. Mobile-Agent-E also features a novel self-evolution module which maintains a persistent long-term memory comprising Tips and Shortcuts. Tips are general guidance and lessons learned from prior tasks on how to effectively interact with the environment. Shortcuts are reusable, executable sequences of atomic operations tailored for specific subroutines. The inclusion of Tips and Shortcuts facilitates continuous refinement in performance and efficiency. Alongside this framework, we introduce Mobile-Eval-E, a new benchmark featuring complex mobile tasks requiring long-horizon, multi-app interactions. Empirical results show that Mobile-Agent-E achieves a 22% absolute improvement over previous state-of-the-art approaches across three foundation model backbones. Project page: https://x-plug.github.io/MobileAgent.

1. Introduction

The introduction identifies gaps in handling complex real-world mobile tasks and learning from prior interactions, then presents Mobile-Agent-E’s hierarchical agents and experience memory alongside the Mobile-Eval-E benchmark.

Introduction

  • Real-world mobile tasks require constraint-intensive reasoning, long-horizon multi-app planning, exploratory information gathering, and resilience to pop-up ads and changing layouts.Existing tasks are often short and straightforward, unlike tasks such as comparing prices across apps.Source: §1 · p. 2
  • Existing mobile agents treat each task as a first attempt, repeating mistakes and allocating the same computational resources despite prior interactions.This limits performance on complex, long-horizon tasks with reusable subroutines such as searching and creating notes.Source: §1 · p. 2
  • Mobile-Agent-E separates high-level subgoal planning from low-level action execution through a Manager and four agents for perception, action, verification, and information aggregation.The agents are the Perceptor, Operator, Action Reflector, and Notetaker.Source: §1 · p. 2
  • Its self-evolution module stores Tips for general interaction guidance and Shortcuts as executable atomic-operation sequences for recurring subroutines.After each task, Experience Reflectors update Tips and propose Shortcuts, which inform the Manager and Operator in future tasks; the paper relates them to episodic and procedural knowledge (Tulving, 2002; Squire & Zola, 1996; Anderson, 1982).Source: §1 · pp. 2–3
  • Mobile-Eval-E includes more than twice the expected operations per task and more multi-app interactions than prior benchmarks (Wang et al., 2024b; Wang et al., 2024a).Its human-rubric Satisfaction Score covers milestones and exploratory behavior, while the Satisfaction Score vs Steps curve evaluates efficiency; prior state-of-the-art methods achieve about 50–70% of human satisfaction.Source: §1 · p. 3
  • 22.1% average absolute gain over prior state-of-the-art approaches across three foundation-model backbones is reported for Mobile-Agent-E.Self-evolution improves performance by 6.5% absolutely versus no evolution; Shortcuts reduce computational overhead, reaching speeds comparable to prior models while delivering significantly better performance.Source: §1 · p. 3

2. Mobile-Agent-E

Mobile-Agent-E separates high-level planning from low-level action execution and uses persistent long-term memory to refine planning and actions across tasks.

2.1. Hierachical Multi-Agent Framework

  • The Manager updates the overall plan and selects the next subgoal using the request, screenshot, prior plan, Shortcuts, and notes, without fine-grained Perceptor results.The framework keeps detailed perception out of high-level planning because it may add noise.Source: §2.1 · p. 3
  • After consecutive failed actions, an Error Escalation Flag prompts the Manager to reconsider the overall plan or current subgoal; the Operator first handles initial errors.The Manager receives information about recent errors to support higher-level recovery.Source: §2.1 · p. 4
  • The Perceptor extracts screen text and icons with coordinates, the Action Reflector verifies action outcomes and reports errors, and the Notetaker aggregates task-relevant information.Reasoning agents receive the original screenshot as holistic visual context alongside fine-grained perception.Source: §2.1 · pp. 4–5
  • The Operator chooses actions using the plan, subgoal, recent action and error history, Tips, and Perceptor results; actions include atomic operations and reusable Shortcuts.The action history records outcomes, and the Operator is prompted to rectify unresolved failures.Source: §2.1 · p. 4

2.2. Self-Evolution Module

  • Long-term memory contains Tips, general guidance learned from prior trials, and Shortcuts, reusable sequences of atomic operations for recurring subroutines.Because a Shortcut may only apply in certain screen states, it has a precondition that the Operator must verify before use.Source: §2.2 · pp. 5–6
  • At each task’s end, two Experience Reflectors use the task query, plan, action and error histories, existing memory, and any provided future tasks to update Tips and generate Shortcuts for subsequent tasks.The updated memory informs both the Manager’s high-level planning and the Operator’s low-level action decisions.Source: §2.2 · p. 6

3. Experiments

The experiments evaluate mobile agents on real devices using human-reviewed interaction records and a benchmark designed for complex, long-horizon, multi-app tasks. They define human-aligned evaluation metrics and test cross-task self-evolution against existing agents and across model backbones.

3. Experiments

  • Evaluation runs dynamically on an Android phone controlled through ADB, with human evaluators reviewing recorded screenshots and action histories.The phone used was a Samsung Galaxy A15.Source: §3 · p. 6, §3.1 · p. 6

3.1. A More Challenging Benchmark: Mobile-Eval-E

  • Mobile-Eval-E contains 25 manually crafted tasks across five real-world scenarios, with over 2× the expected operations per task and 76% requiring multiple apps.Its tasks span restaurant recommendations, information searching, online shopping, trending topics, and travel planning.Source: §3.1 · pp. 6–7

3.2. Metrics with Better Human Alignment

  • Satisfaction Score measures fulfilled human-judged rubrics, while Action Accuracy, Reflection Accuracy, and Termination Error assess action-level performance and task exits.Rubrics combine milestone steps and satisfaction criteria; termination errors include exits from iteration, repeated-error or repeated-action limits, and other errors.Source: §3.2 · p. 7

3.3. Evaluating Self-Evolving Mobile Agents

  • Mobile-Agent-E + Evo runs tasks sequentially within each scenario, sharing persistent memory across five tasks and updating it after each task using interaction history and remaining queries.This setup lets later tasks benefit from a greater accumulation of Tips and Shortcuts.Source: §3.3 · p. 7

3.4. Models

  • The evaluation compares AppAgent, Mobile-Agent-v1, and Mobile-Agent-v2, aligning Mobile-Agent-v2’s operation space, perception model, and initial Tips with Mobile-Agent-E.AppAgent’s exploration phase did not fit the setting, so the initial Tips were added as additional knowledge.Source: §3.4 · p. 7
  • The reasoning agents use GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-pro backbones, with GPT-4o as the default.The Perceptor implementation uses DBNet and ConvNextViT-document for OCR, GroundingDINO for icon grounding, and Qwen-VL-Plus for cropped-icon captions.Source: §3.4 · pp. 7–8

4. Results

Mobile-Agent-E improves benchmark performance and efficiency, while its evolving memory supports progressive gains, reduces shortcut execution overhead, and helps retrieve relevant experience for an unseen task.

4.1. Evaluation on Performance

  • 22.1% higher Satisfaction Score than the previous multi-agent SOTA with GPT-4o; enabling self-evolution raises the improvement to 33.9%.The comparison uses an identical GPT-4o backbone for baselines and Mobile-Agent-E; the paper also reports lower termination error and stronger robustness and error recovery.Source: §4.1 · p. 8
  • 22.1% and 15.6% average absolute gains over the previous SOTA with and without evolution, respectively, across GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-pro.Self-evolution’s benefits are more pronounced with stronger backbones, including GPT-4o and Claude.Source: §4.1 · p. 8

4.2. Evaluation on Efficiency

  • Mobile-Agent-E reaches higher final satisfaction and fulfills rubrics faster, as shown by its Satisfaction Score vs Steps curve.The SSS curve tracks satisfaction after each step; a steeper, higher curve indicates better efficiency and completeness.Source: §4.2 · pp. 8–9, Figure 5 · p. 9

4.3. Further Analysis

  • Later tasks generally gain more from self-evolution as updated Tips and Shortcuts accumulate, although gains are not strictly monotonic because task difficulty varies.The analysis groups tasks by their order within each scenario and compares performance with and without evolution.Source: §4.3 · p. 9, Figure 6 · p. 9
  • Shortcuts mitigate the computational overhead of the hierarchical architecture and correlate with faster inference by combining multiple operations in one decision iteration.A Tap Type and Enter Shortcut saves two perception-and-reasoning iterations versus three atomic actions.Source: §4.3 · pp. 9–10, Table 5 · p. 9
  • Tips provide a distinctive benefit from self-evolution even when newly generated Shortcuts are excluded from the evaluated trajectories.The analysis filters out instances using the same set of unique Shortcuts or only atomic actions.Source: §4.3 · p. 10, Table 6 · p. 10

4.4. Case Study: A Closed-Loop Self-Evolving Agent

  • With GPT-4o, memory evolved across 25 tasks in five scenarios to 7 unique Shortcuts and 58 Tips; retrievers selected relevant experience to complete an unseen task.Mobile-Agent-E newly proposed 6 Shortcuts and 55 Tips during experience reflection.Source: §4.4 · p. 10

5. Related Work

Prior mobile GUI agents explored stronger perception, reasoning, and multi-agent decision-making, but faced planning and error-recovery challenges. Mobile-Agent-E distinguishes itself through hierarchical planning and mobile self-evolution with preconditioned Shortcuts.

5.1. GUI Agents

  • LMM-based GUI-agent research spans Web, PC, and mobile settings; mobile work has pursued stronger single-agent perception and reasoning through tools or exploration, alongside multi-agent approaches.The cited multi-agent studies include Rawles et al. (2024) and Wang et al. (2024a).Source: §5.1 · p. 10
  • Mobile-Agent-E separates high-level planning from low-level action decisions, addressing short-sighted planning and weak error recovery in prior multi-agent GUI agents.Mobile-Agent-v2’s planning module primarily tracks progress, while its decision-making module handles both planning and action choices.Source: §5.1 · p. 10

5.2. Self-Evolution in Foundation Models

  • Foundation-model research explores capability improvement and tool learning, but GUI-agent self-evolution remains understudied, with no systematic prior exploration in mobile environments.Mobile-Agent-E demonstrates self-evolution through Tips and Shortcuts.Source: §5.2 · p. 10
  • Unlike Cradle’s skills, which enter the atomic-operation space, Mobile-Agent-E Shortcuts specify preconditions to support decisions across apps and layouts.A case study shows automatically retrieved Tips and Shortcuts applied to an unseen challenging task, including recovery from an error.Source: §5.2 · pp. 10–11, Figure 7 · p. 11

6. Conclusion and Future Work

Mobile-Agent-E combines a hierarchical multi-agent framework with a self-evolution module to enhance long-term planning, error recovery, and efficiency on complex real-world tasks. The conclusion notes Shortcut-related limitations and points to improved Shortcut management, personalization, and safety as future work.

6. Conclusion and Future Work

  • Mobile-Agent-E’s hierarchical multi-agent framework and self-evolution module enhance long-term planning, error recovery, and efficiency across complex real-world tasks.The paper describes the assistant as excelling in a wide variety of such tasks.Source: §6 · p. 11
  • Remaining limitations include using Shortcuts with invalid preconditions and generating erroneous Shortcuts.The paper points to Appendix C for detailed examples.Source: §6 · p. 11
  • Future work will improve Shortcut generation, invocation, and revision, personalize assistance to users, and strengthen safety for human-agent collaboration.Source: §6 · p. 11

Impact Statement · A. Full Trajectory Comparison Example with Previous SOTA

The sections illustrate Mobile-Agent-E’s full-trajectory comparison with prior state of the art and discuss the potential benefits and risks of autonomous mobile agents.

Impact Statement

  • The framework aims to improve smartphone usability and efficiency for complex, multi-step tasks and enhance human-device interaction.Source: Impact Statement · p. 11
  • Automated mobile task management could improve accessibility for people with disabilities or limited mobility.The paper also identifies physical impairments, cognitive challenges, and difficulty interacting precisely with touchscreens as relevant conditions.Source: Impact Statement · p. 11
  • Autonomous mobile agents could take unauthorized or unintended actions, including misusing sensitive information such as credit card details or private data.Source: Impact Statement · p. 11
  • These risks underscore the need for safeguards, error recovery, and fail-safe systems that keep agent actions aligned with user intentions.Source: Impact Statement · p. 11
  • Future work will pursue privacy and safety mechanisms including explicit consent for sensitive operations, encryption, and automated flagging of potentially harmful or unauthorized actions.The paper describes these measures as important for minimizing risks and building user trust.Source: Impact Statement · p. 11

A. Full Trajectory Comparison Example with Previous SOTA

B. Error Recovery with Escalation to Manager

The section presents Mobile-Agent-E’s error escalation mechanism as improving error recovery, illustrated in Figure 9.

B. Error Recovery with Escalation to Manager

  • Figure 9 illustrates that Mobile-Agent-E’s error escalation mechanism enhances error recovery.The passage refers readers to the figure caption for a detailed description of the example.Source: B. Error Recovery with Escalation to… · p. 14

C. Remaining Limitations · C.1. Misuse of Shortcuts due to Incorrect Perception of Phone State

Operator verification does not prevent all Shortcut misuse: incorrect phone-state perception can cause execution in an invalid state, potentially addressed by dedicated precondition verification or improved perception.

C.1. Misuse of Shortcuts due to Incorrect Perception of Phone State

  • Despite requiring Operator state verification before Shortcut use, incorrect phone-state perception can trigger a Shortcut in an invalid state; a dedicated precondition verifier or improved perception may mitigate this.The proposed remedies are a dedicated agent to verify Shortcut preconditions or an enhanced perception module that better understands phone states.Source: §C.1 · p. 14

C.2. Errors and Imperfections in Self-Evolved Shortcuts

Although self-evolved Shortcuts are effective in most cases, generated errors and imperfections can propagate into later tasks. The paper calls for higher-quality Shortcut generation and agents that can reflect on and revise Shortcuts.

C.2. Errors and Imperfections in Self-Evolved Shortcuts

  • Errors and imperfections in generated Shortcuts can propagate when those Shortcuts are reused in subsequent tasks.The paper notes that Shortcuts are effective in most cases, but Figure 11 illustrates an erroneous and imperfect example.Source: §C.2 · p. 14
  • Future work should improve Shortcut quality and enable agents to reflect on and revise generated Shortcuts in later tasks.Source: §C.2 · p. 14

D. All Tasks in Mobile-Eval-E Benchmark … G. Full list of Self-Evolved Tips

The appendices document Mobile-Eval-E tasks and atomic operations, list self-evolved Shortcuts and Tips, and illustrate trajectory comparison and Shortcut-related failure cases.

D. All Tasks in Mobile-Eval-E Benchmark

E. Atomic Operation Space

F. Full list of Self-Evolved Shortcuts

G. Full list of Self-Evolved Tips

  • After self-evolving across all 25 Mobile-Eval-E tasks, Mobile-Agent-E generated a full list of Tips, shown for GPT-4o.Figure 13 presents the generated list.Source: G. Full list of Self-Evolved Tips · p. 14, Figure 13 · p. 21
  • Figure 8 compares full trajectories of Mobile-Agent-E and the previous state-of-the-art Mobile-Agent-v2 (Wang et al., 2024a).The supplied passage identifies the comparison but does not report its outcomes.Source: Figure 8 · p. 15
  • After two consecutive errors, the Manager revises the Walmart-search subgoal into detailed steps, enabling the Operator to clear the previous query before searching again.The Action Reflector raises an error, but the Operator initially fails to correct it on a second attempt.Source: Figure 9 · p. 16
  • A Shortcut can be misused in an invalid state when the agent switches apps but invokes a search Shortcut before reaching the app where its search bar is available.The example switches back to Walmart, then calls a search Shortcut before the search bar is available.Source: Figure 10 · p. 17
  • Generated Shortcuts can be imperfect or erroneous: one includes an unnecessary Tap action, while another omits a Tap needed to enter the desired app before searching.The examples are “Search Location in Maps” and “Switch App And Search,” respectively.Source: Figure 11 · p. 17

Related papers

Loading 2501.11733v2…