PaperlayerView exact revision on arXiv

Source-linked AI summary

Foundations and Recent Trends in Multimodal Mobile Agents: A Survey

Biao Wu, Yanda Li, Zhiwei Zhang, Yunchao Wei, Meng Fang, Ling Chen

arXiv:2411.02006v3cs.AI

TL;DR

Mobile-agent research must address real-time adaptation and multimodal interaction, while current benchmarks still leave gaps in capturing the diversity and dynamics of mobile tasks. This survey synthesizes agent components, benchmark approaches, and prompt-based and training-based methods, and identifies research directions including security, adaptability, and multi-agent collaboration.

  • Problem

    Mobile agents are expected to adapt in real time and process multimodal inputs, while existing GUI benchmarks can miss valid strategies, generalization, and reproducible evaluation.

    Source: §1 · p. 1, §3 · p. 4

  • Method

    The survey reviews mobile-agent components and benchmarks, categorizes agents into prompt-based and training-based approaches, and discusses challenges and future directions.

    Source: §1 · p. 2

  • Results

    The survey synthesizes benchmark advances and the strengths and challenges of prompt-based and training-based agents, and identifies security, adaptability, and multi-agent collaboration as future directions.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The survey points to security, adaptability, and multi-agent collaboration as directions for advancing mobile-agent capabilities.

    Source: §6 · p. 8

    The survey focuses on recent LLM-based mobile agents and gives limited coverage to traditional non-LLM systems, including older rule-based agents.

    Source: §7 · p. 9

Abstract

from arXiv · show

Mobile agents are essential for automating tasks in complex and dynamic mobile environments. As foundation models evolve, the demands for agents that can adapt in real-time and process multimodal data have grown. This survey provides a comprehensive review of mobile agent technologies, focusing on recent advancements that enhance real-time adaptability and multimodal interaction. Recent evaluation benchmarks have been developed better to capture the static and interactive environments of mobile tasks, offering more accurate assessments of agents' performance. We then categorize these advancements into two main approaches: prompt-based methods, which utilize large language models (LLMs) for instruction-based task execution, and training-based methods, which fine-tune multimodal models for mobile-specific applications. Additionally, we explore complementary technologies that augment agent performance. By discussing key challenges and outlining future research directions, this survey offers valuable insights for advancing mobile agent technologies. A comprehensive resource list is available at https://github.com/aialt/awesome-mobile-agents

1 Introduction

Mobile agents are increasingly expected to adapt in real time and process multimodal data, but traditional evaluations often miss the dynamic, interactive nature of mobile tasks. This survey reviews evaluation benchmarks, agent approaches, core components, and future directions.

1 Introduction

  • Mobile agents are expected to adapt continuously to changing inputs, process multimodal data, and interact with varied user interfaces.The introduction identifies mobile interfaces, autonomous navigation, and intelligent assistance as application areas.Source: §1 · p. 1
  • Traditional evaluations often miss the dynamic, interactive nature of real-world mobile tasks, while newer benchmarks assess task completion and responses to environmental changes.AndroidEnv and Mobile-Env are cited as examples of interactive benchmarks.Source: §1 · p. 1
  • Prompt-based methods use instruction prompting and chain-of-thought reasoning with LLMs, while training-based methods fine-tune multimodal models for mobile applications.The survey notes that scalability and robustness remain challenges for prompt-based systems.Source: §1 · pp. 1–2
  • The survey reviews mobile-agent components, benchmarks, prompt-based and training-based approaches, and future research directions.Its component overview covers perception, planning, action, and memory.Source: §1 · p. 2

2 The Components of Mobile Agents

Mobile agents combine perception, planning, action, and memory to operate in dynamic environments. The survey describes alternatives within each component, including visual or structured perception, static or feedback-driven planning, multiple action channels, and short- or long-term memory.

2 The Components of Mobile Agents

  • Mobile-agent components comprise perception, planning, action, and memory, which together support operation in dynamic environments.The survey frames them as enabling agents to perceive, reason, and execute.Source: §2 · p. 2

2.1 Perception

  • Perception extracts multimodal information for planning and execution, using either visual inputs alone or mobile-specific context such as UI structure and APIs.When structured UI information is available, DOM cues and functional APIs can provide interface structure and dynamic application information.Source: §2.1 · pp. 2–3

2.2 Planning

  • Static planning decomposes tasks into sub-goals but lacks error correction, whereas dynamic planning uses real-time feedback to backtrack and re-plan.OmniAct structures multimodal inputs and integrates external tools to support reasoning and efficient execution.Source: §2.2 · p. 3

2.3 Action

  • Agents act through GUI gestures, API calls, and interactions with other agents, supporting both native-app navigation and tasks beyond GUI inputs.Screen interactions can involve multi-step processes that require adaptation to changes or new inputs.Source: §2.3 · p. 3

2.4 Memory

  • Short-term memory retains recent context across text, images, and interactions, while long-term memory combines parametric memory with vector stores for episodic knowledge.Some approaches convert multimodal inputs into unified text to simplify retrieval and integration.Source: §2.4 · p. 3

3 Mobile Datasets and Benchmarks

Mobile-agent benchmarks range from static datasets to simulations and real-world environments, but current evaluation approaches face limits in capturing valid strategies, reproducibility, and cross-study comparability. The section reviews evaluation methods and compares reported performance across studies while noting the constraints on direct comparison.

3 Mobile Datasets and Benchmarks

  • Existing GUI benchmarks can miss diverse valid strategies, while interactive benchmarks may still rely on action similarity and uncontrolled evaluations can lack reproducibility and consistent criteria.The survey argues that improving metrics alone is insufficient and calls for a systematic categorization of environments.Source: §3 · pp. 3–4

3.1 Static Datasets

  • Static datasets provide predefined tasks and annotated solutions, progressing from low-level UI-element commands to task-oriented instructions with action-observation pairs and structured representations.AITW contains 715,142 episodes and 30,378 unique prompts.Source: §3.1 · pp. 4–5

3.2 Simulation Environments

  • Simulation environments support dynamic interaction, continuous adaptation, and feedback, shifting from earlier reinforcement-learning settings toward LLM agents handling natural-language tasks.Mobile-Env is cited as an effort toward autonomous exploration of multi-step tasks with less manual scripting.Source: §3.2 · p. 5

3.3 Real-world Environments

  • Open-world environments expose agents to changing content, unpredictable user behavior, and varied device configurations, but raise safety, reproducibility, and fairness risks.Fixing dynamic online content and using replay mechanisms are proposed to make evaluation more controlled.Source: §3.3 · p. 5

3.4 Evaluation Methods

  • Trajectory evaluation compares actions with reference paths, whereas outcome evaluation checks whether final goals are reached, potentially using multiple system signals.Multiple signals matter because some transitions, such as form submissions, may appear in the GUI but not in system logs or databases.Source: §3.4 · pp. 5–6

3.5 Performance Comparison

  • Unified comparison of methods is challenging because benchmarks, implementation methods, platforms, and evaluation metrics vary across studies.The survey compares AITW instruction accuracy and MobileAgentbench success rate, and notes the need for standardized benchmarks.Source: §3.5 · p. 6

4 The Taxonomy of Mobile Agents

The survey organizes mobile agents into prompt-based and training-based approaches. It reviews prompting and perception tools alongside model fine-tuning, mobile-specific pretraining, dataset improvements, and reinforcement learning.

4 The Taxonomy of Mobile Agents

  • The taxonomy distinguishes prompt-based agents that interpret and execute natural-language instructions from training-based agents that fine-tune models or use reinforcement learning.The approaches are presented in relation to GUI interaction, decision-making, and adaptability.Source: §4 · p. 6

4.1 Prompt-based Methods

  • Prompt-based methods use LLM prompting and reasoning, with perception tools translating GUI elements into text, though reliance on external tools and app-specific APIs can cause inefficiencies and errors.AppAgent's exploration phase stores prior interactions to support adaptation to new applications without additional training data.Source: §4.1 · pp. 6–7

4.2 Training-based Methods

  • Training-based methods optimize models using instruction-following data, and mobile-specific pretraining addresses general VLMs' limited sensitivity to interactive mobile elements.CogAgent uses a large-scale mobile dataset for pretraining representations; Spotlight relies on screenshots and regions, while VUT uses a dual-tower Transformer.Source: §4.2 · p. 7
  • Training-based improvements include synthetic or annotated UI datasets, training strategies such as separating planning from execution, and reinforcement learning from environment interactions.DigiRL combines offline and online reinforcement learning and scales online training with a VLM-based evaluator across 64 Android emulators.Source: §4.2 · pp. 7–8

5 Trade-off Analysis

The survey compares prompt-based and training-based mobile agents across deployment cost, inference speed, operating expense, and privacy. Prompt-based agents support rapid prototyping without local deployment, while training-based agents have lower per-task costs, faster inference, and full data control, at the cost of infrastructure and engineering effort.

5 Trade-off Analysis

  • Prompt-based agents require no local deployment and enable rapid prototyping, but cost approximately 0.70˘1.20 per task and take 5–25 seconds per step.They rely on commercial APIs such as GPT-4V and Gemini, and process user data externally, raising privacy concerns.Source: §5 · p. 8
  • Training-based agents cost $0.01–$0.05 per task and take 1–3 seconds per step, while providing full control over data privacy.The evaluated setup used a rented two-A100 server; purchase was estimated at approximately $30,000, while rental for the 56-day evaluation cost around $4,000.Source: §5 · p. 8
  • Prompt-based solutions suit lightweight or rapidly changing tasks, while training-based agents offer long-term efficiency, privacy, and scalability for high-frequency or latency-sensitive applications.Training-based deployment requires more engineering effort for deployment, fine-tuning, and ongoing maintenance.Source: §5 · p. 8

6 Conclusion

The survey reviews mobile-agent components, benchmarks, and prompt-based and training-based approaches. It concludes that benchmarks still need to capture real-world dynamics more comprehensively, and highlights security, adaptability, and multi-agent collaboration as future directions.

6 Conclusion

  • The survey identifies perception, planning, action, and memory as core components that enable mobile agents to adapt to their environments.It presents these components as the foundation of agent functionality.Source: §6 · p. 8
  • Reviewed benchmarks improve mobile-agent assessment, but more comprehensive methods are still needed to capture real-world dynamics.Source: §6 · p. 8
  • The survey distinguishes prompt-based and training-based methods, and highlights security, adaptability, and multi-agent collaboration as future research directions.Source: §6 · p. 8

7 Limitations

The survey focuses on recent LLM-based mobile agents and gives limited coverage to traditional, non-LLM-based systems; this may constrain the broader historical context it provides.

7 Limitations

  • The survey focuses on recent LLM-based mobile agents, with limited coverage of traditional, non-LLM-based systems.Source: §7 · p. 9
  • The discussion of older rule-based agents is limited.Source: §7 · p. 9
  • The limited discussion of older rule-based agents may constrain the broader context of mobile-agent technology development.Source: §7 · p. 9

A.1 Future Work

The survey proposes future work on model architecture, reinforcement learning, security and privacy, multi-agent collaboration, and lightweight deployment. These directions address grounding and decision-making, dynamic environments, safe interaction, coordination, and mobile-device resource constraints.

Model Architecture Optimization

  • Mixture-of-Experts architectures are proposed to improve grounding, action prediction, and adaptability across multi-domain tasks.The survey emphasizes balancing accurate element localization with action prediction and efficient decisions.Source: §A.1 · p. 15

Combine with Reinforcement Learning

  • Reinforcement learning is proposed for sequential decision-making in interactive environments where outcomes can shift the data distribution.The survey says this research remains early-stage and that current explorations have not achieved end-to-end alignment.Source: §A.1 · p. 15

Security and Privacy

  • Future work should strengthen security, privacy protection, and ethical mechanisms for mobile agents operating in open environments.Source: §A.1 · p. 15

Multi-agent Collaboration

  • Multi-agent research remains limited in scale and has little exploration of communication or organizational structures; the survey calls for efficient collaboration mechanisms.The proposed direction includes agents dynamically forming coalitions to complete tasks more effectively.Source: §A.1 · p. 15

Model Lightweighting

  • Mobile-device resource limits motivate smaller agents and faster inference; existing methods remain too large for on-device deployment, while LiMAC reduces fine-tuning costs without compressing parameters.The survey also points to inference-pipeline and hardware acceleration to improve response speed and reduce energy consumption.Source: §A.1 · pp. 15–16

A.2 Complementary Technologies

Complementary technologies augment mobile agents beyond their core components, supporting more capable and usable interaction with mobile environments.

A.2 Complementary Technologies

  • Complementary technologies support agents’ adaptation, learning, and efficient execution of complex tasks.The survey describes them as augmenting both performance and usability.Source: §A.2 · p. 16
  • UIED detects and classifies GUI elements using computer vision and deep learning to support interactive editing.Source: §A.2 · p. 16
  • WebGPT uses imitation learning and human feedback to fine-tune GPT-3 for web-based question answering, while WebVLN trains agents to navigate websites from question-based instructions using HTML.Source: §A.2 · p. 16

A.3 Available related technologies

The survey reviews related technologies and compares GUI agents across static action and grounding benchmarks. Its reported results vary by task, with UI-R1-3B leading ScreenSpot average accuracy.

A.3 Available related technologies

  • The related-technology review includes WebVoyager’s screenshot-based navigation evaluation, reported at 85.3% agreement with human judgments, and Widget Captioning’s 162,859 human-annotated UI phrases.It also notes desktop-agent benchmarks for software-tool use and PowerPoint tasks.Source: §A.3 · p. 16

A.4 Comparison of GUI Agents Performance

  • CogAgent’s 47.4% grounding performance on ScreenSpot is lower than its strong AITW action accuracy.The survey identifies a possible grounding gap under screenshot-based settings.Source: §A.4 · p. 16
  • 76.88% action accuracy on AITW is reported for CogAgent, compared with 76.20% for SeeClick.The table covers action accuracy on the static AITW dataset.Source: §A.4 · p. 16, Table 4 · p. 17
  • Table 5 reports grounding accuracy on ScreenSpot, distinguishing optimal results in bold and suboptimal results with underlining; ZS denotes zero-shot OOD inference, and RFT denotes rule-based reinforcement learning.Source: Table 5 · p. 18
  • Table 5 reports grounding accuracy on ScreenSpot; in the cited comparison, SeeClick reaches 53.4%.Source: §A.4 · p. 17, Table 5 · p. 18
  • The survey also lists an AndroidWorld leaderboard by model type, screen representation, and pass@1 success rate, alongside MobileAgentBench comparisons using success, error, latency, token, and false-negative/positive metrics.Source: Table 6 · p. 18, Table 7 · p. 18

Related papers

Loading 2411.02006v3…