PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects

Guangyi Liu, Pengxiang Zhao, Yaozhen Liang, Liang Liu, Yaxuan Guo, Han Xiao, Weifeng Lin, Yuxiang Chai, Yue Han, Shuai Ren, Hao Wang, Xiaoyu Liang, WenHao Wang, Tianze Wu, Zhengxi Lu, Siheng Chen, LiLinghao, Hao Wang, Guanjing Xiong, Yong Liu, Hongsheng Li

arXiv:2504.19838v3cs.HC

TL;DR

Phone automation research has grown, but comprehensive surveys focused on LLM-powered phone GUI agents are lacking. This paper surveys the field through a mobile-focused taxonomy and analysis of methods, datasets, and benchmarks. It concludes that progress is significant, while reliable real-world deployment remains challenging, and outlines directions for future work.

  • Problem

    Despite growing research, dedicated systematic surveys comprehensively examining phone automation from the perspective of large language models are lacking.

    Source: §1 · p. 2

  • Method

    The paper systematically reviews LLM-powered phone GUI agents, organizing frameworks, modeling approaches, datasets, and evaluation while focusing on mobile and drawing on cross-platform work where relevant.

    Source: §1 · p. 3

  • Results

    The survey concludes that foundational capabilities such as visual grounding have advanced, but reliable real-world deployment remains challenging and performance differs between offline and online evaluation.

    Source: §6.3 · p. 49

  • Takeaways & Limitations

    The survey offers a roadmap toward robust, user-friendly, and secure phone GUI agents that adapt to evolving mobile ecosystems.

    Source: §8 · p. 55

    Dataset tasks remain skewed toward simple, short-horizon, single-app activities, leaving real-world multi-app and compositional workflows underrepresented.

    Source: §6.4.1 · p. 49

Abstract

from arXiv · show

With the rapid rise of large language models (LLMs), phone automation has undergone transformative changes. This paper systematically reviews LLM-driven phone GUI agents, highlighting their evolution from script-based automation to intelligent, adaptive systems. We first contextualize key challenges, (i) limited generality, (ii) high maintenance overhead, and (iii) weak intent comprehension, and show how LLMs address these issues through advanced language understanding, multimodal perception, and robust decision-making. We then propose a taxonomy covering fundamental agent frameworks (single-agent, multi-agent, plan-then-act), modeling approaches (prompt engineering, training-based), and essential datasets and benchmarks. Furthermore, we detail task-specific architectures, supervised fine-tuning, and reinforcement learning strategies that bridge user intent and GUI operations. Finally, we discuss open challenges such as dataset diversity, on-device deployment efficiency, user-centric adaptation, and security concerns, offering forward-looking insights into this rapidly evolving field. By providing a structured overview and identifying pressing research gaps, this paper serves as a definitive reference for researchers and practitioners seeking to harness LLMs in designing scalable, user-friendly phone GUI agents. The collection of papers reviewed in this survey will be hosted and regularly updated on the GitHub repository: https://github.com/PhoneLLM/Awesome-LLM-Powered-Phone-GUI-Agents

1 Introduction

Phone GUI automation is moving beyond fixed scripts toward LLM-powered agents that interpret instructions and interact with dynamic mobile interfaces. This survey organizes the field’s methods, resources, and open challenges, with a focus on phone environments.

1 Introduction

  • Traditional phone automation relies on static scripts and has limited adaptability to dynamic mobile interfaces.Earlier automation also uses predefined scripts and templates that lack flexibility in variable interfaces and environments.Source: §1 · p. 1
  • LLM-powered phone GUI agents integrate language understanding, multimodal perception, planning, and action execution to handle complex mobile tasks dynamically.They can recognize interfaces, perceive changes in real time, and execute operations on the device.Source: §1 · p. 2
  • The survey addresses the lack of dedicated, comprehensive reviews of LLM-based phone automation by synthesizing current research and future directions.It reviews the literature and analyzes how large language models are being applied to phone automation.Source: §1 · pp. 2–3
  • Its taxonomy groups the field into agent frameworks, large language models for phone automation, and datasets and evaluation methods.The taxonomy includes representative works rather than an exhaustive categorization.Source: §1 · p. 3, Figure 2 · p. 4
  • The survey frames agent design across framework choices, model selection and training, data preparation, and evaluation protocols.The framework choices include single-agent, multi-agent, and plan-then-act designs; modeling approaches include prompting and training-based methods.Source: §1 · p. 3
  • It analyzes how language comprehension, multimodal grounding, reasoning, and decision-making connect user intent with GUI actions, while identifying research challenges and future directions.The discussion includes dataset coverage, on-device constraints, reliability, user-centric adaptation, security and privacy, long-horizon planning, and multi-agent coordination.Source: §1 · pp. 3, 5

2 Related Work

The survey situates its phone-focused analysis among GUI automation, RPA, and LLM-agent reviews, then distinguishes its mobile-specific framework and benchmark coverage.

2.1 Surveys on GUI Automation and Robotic Process Automation

  • Earlier GUI-testing and RPA surveys map automation methods, but identify semantic understanding and reasoning for complex, non-deterministic tasks as a gap.Mobile GUI reviews also address platform-specific issues such as screen-size diversity and touch interaction.Source: §2.1 · p. 5

2.2 Surveys on LLM-Powered Agents

  • Existing LLM-agent surveys cover agents and evaluation broadly, while newer GUI-agent surveys include some mobile work.The comparison identifies Wu et al. (2025a) as focused on multimodal agents on mobile platforms.Source: §2.2 · p. 5
  • This survey differentiates itself through a phone-centered taxonomy, analysis of mobile-specific interfaces and deployment constraints, and review of phone GUI datasets and benchmarks.Table 1 characterizes it as the first in-depth focused analysis of LLM-powered phone agents that includes quantitative comparison of mobile-specific benchmarks.Source: §2.2 · pp. 5–6, Table 1 · p. 6
  • Its historical account connects traditional mobile-automation challenges with LLM approaches and identifies phone-specific future concerns, including on-device deployment, privacy, security, and long-horizon planning.The survey presents this as a practical guide to the field.Source: §2.2 · p. 6

3 Development of Phone Automation

Phone automation developed from testing, shortcuts, and RPA toward LLM-powered agents. The section traces traditional methods’ limitations, LLM approaches, and emerging commercial applications.

3.1.1 Automation Testing

  • DinoDroid uses DQN to learn behavior models of existing Android apps, automatically explore and generate test cases, improving code coverage and bug detection; deep RL faces high training costs and poor generalization.Model-based testing required substantial manual model construction and maintenance. Humanoid relied on human-generated training datasets and predefined rules to guide testing.Source: §3.1.1 · p. 7

3.1.2 Shortcuts

  • Mobile shortcuts automate action sequences through predefined rules or triggers, but their manually designed IF-THEN logic limits scope and flexibility.Source: §3.1.2 · p. 7

3.1.3 Robotic Process Automation

  • Phone RPA records user action sequences to automate repetitive cross-app tasks, but struggles with dynamic interfaces and requires frequent script updates.Source: §3.1.3 · p. 8

3.2 Challenges of Traditional Methods

  • Traditional methods are app- and interface-specific, costly to update, dependent on manual scripting for complex instructions, and weak at adapting to varied, dynamic GUIs.Dynamic content and multi-layered tasks such as flight booking can require interpreting changing visual cues and interface controls.Source: §3.2.1 · p. 8, §3.2.2 · p. 8, §3.2.4 · p. 9

3.3 LLMs Boost Phone Automation

  • LLM phone-agent research uses prompt engineering with pretrained models or GUI-data training, including supervised fine-tuning and reinforcement learning.Prompting does not modify model parameters, whereas training-based methods adapt models on GUI-specific data.Source: §3.3 · p. 9, Figure 3 · p. 10
  • Larger and more diverse datasets are associated with consistent gains in phone-agent accuracy and with competencies for more abstract, multi-step instructions.The section also links growth in model size and data complexity to increasingly efficient and sophisticated GUI interaction.Source: §3.3 · pp. 10–11
  • LLM agents address intent and interface challenges through contextual language understanding, multimodal grounding of instructions to screen elements, and reasoning, planning, and adaptation.The section cites MobileAgent, visual-grounding systems, and MobileAgent-V2 as examples of these capabilities Wang et al. (2024b); Gou et al. (2024); You et al. (2024); Zhang et al. (2024d); Wang et al. (2024a).Source: §3.3 · p. 11

3.4 Emerging Commercial Applications

  • Commercial examples include on-device Apple Intelligence, vivo PhoneGPT, Honor YOYO Agent, and AutoGLM, which support varied phone interactions and multi-step tasks.The examples include privacy-focused on-device processing, preference-aware actions, and GUI navigation beyond API calls.Source: §3.4 · p. 12

4 Frameworks and Components of Phone GUI Agents

Phone GUI-agent frameworks combine perception, reasoning, and action to interpret user intent and operate in mobile environments. The section describes incremental single-agent decisions under partial observability and introduces multi-agent and plan-then-act alternatives.

Single-Agent Framework

  • A single agent repeatedly observes a changing phone environment and chooses its next action rather than precomputing an exhaustive sequence.The section models this incremental decision process as a POMDP because UI elements, device states, and context can change unpredictably.Source: §4 · pp. 12–13
  • In the POMDP formulation, actions alter environment states through potentially uncertain transitions, while observations provide only partial information about the underlying state.State includes UI, phone status, and task context; the agent uses observations to update its strategy toward the goal.Source: §4 · pp. 13–14

Frameworks and Components

  • The framework overview organizes agent operation around perceiving user intent and phone state, using knowledge and memory to plan and reason, then executing UI actions.The section also introduces collaborative multi-agent frameworks and plan-then-act designs that separate planning from execution.Source: §4 · pp. 12, 14, Figure 5 · p. 14

4.1 Perception in Phone GUI Agents

Perception captures and interprets the mobile environment as the initial input to reasoning and action. It draws on UI representations and phone-state information, which are converted into prompt tokens for the LLM.

4.1.1 UI Information Perception

  • UI trees are simplified into natural-language descriptions or HTML, while layout grouping and automatic scrolling manage complexity and improve information coverage.Source: §4.1.1 · p. 15
  • UI trees provide structured component hierarchies, while screenshots capture visual layouts without API access and can expose missing or overlapping controls.Source: §4.1.1 · p. 15
  • Set-of-Marks labels screenshot elements with identifiers, and OCR or icon data adds textual and visual cues for more precise action decisions.Source: §4.1.1 · p. 16

4.1.2 Phone State Perception

  • Phone-state cues such as keyboard status help determine when text input is needed; location data is described as a potential, not currently used, signal.Perception gathered from UI trees, screenshots, Set-of-Marks, OCR, and phone state is converted into prompt tokens.Source: §4.1.2 · p. 16

4.2 Brain in Phone GUI Agents

The LLM forms the agent’s cognitive core, using perception and stored information to generate actions. Storage supports context, while decision-making draws on planning, reasoning, and reflection.

4.2 Brain in Phone GUI Agents

  • The LLM serves as the agent’s reasoning and decision-making center, bringing language understanding, contextual awareness, and generalization across tasks.Source: §4.2 · p. 16

4.2.1 Storage

  • Memory retains past interaction information for continuity, while Self-MAP retrieves top-K relevant snippets by semantic match and similar trajectories to conserve context.Mobile-Agent-v2 records task-related focus content from historical screens for the decision module to reference and update.Source: §4.2.1 · pp. 16–17
  • Knowledge comes from pretrained capabilities, domain-specific training, or injected exploration and demonstration data, including stored UI transitions and dynamically updated knowledge bases.Source: §4.2.1 · p. 17

4.2.2 Decision Making

  • Planning decomposes goals into action sequences; environmental feedback can revise plans, while phased planning can reduce LLM calls by planning actions for a state at once.Source: §4.2.2 · pp. 17–18
  • Reasoning evaluates context and possible actions, while reflection checks outcomes against goals, supports remedial measures, and helps avoid repeating known errors.Source: §4.2.2 · p. 18

4.3 Action in Phone GUI Agents

The action component translates decisions from the LLM into operations on the phone, connecting high-level commands to device-level execution.

4.3 Action in Phone GUI Agents

  • The action component bridges LLM-generated commands to device operations, using automation tools such as UI Automator, XCTest, Appium, or Selenium.Source: §4.3 · p. 19
  • Actions range from taps to launching apps or changing settings, and execution mechanisms send commands intended to make decisions reliable operations on the device.Source: §4.3 · p. 19
  • Phone GUI actions include touch, gestures, typing and input, system operations, and media controls, which together support navigation and broader device functions.Source: §4.3 · p. 19

4.4 Multi-Agent Framework

Multi-agent frameworks address the limits of isolated agents by coordinating specialized capabilities. They either organize agents around general roles and fixed workflows or dynamically assign scenario-specific specialists.

4.4 Multi-Agent Framework

  • Collaboration among specialized agents can support parallel or coordinated work, improving task efficiency, adaptability, and scalability over isolated single-agent systems.Source: §4.4 · pp. 19–20

4.4.1 Role-Coordinated Multi-Agent

  • Role-coordinated systems assign general functions such as planning, decision-making, memory, reflection, or tool invocation within a predefined workflow.Figure 6 contrasts this fixed workflow with scenario-based assignment to specialized agents.Source: Figure 6 · p. 19, §4.4.1 · p. 20

4.4.2 Scenario-Based Task Execution

  • Scenario-based systems dynamically assign tasks to specialists matched to a task or application domain, adapting the team to varying requirements.Source: §4.4.2 · p. 21
  • Role coordination may add communication and synchronization overhead; scenario-based systems require resources to maintain specialists and mechanisms to recognize tasks and select agents.Source: §4.4.2 · pp. 21–22

4.5 Plan-Then-Act Framework

Plan-Then-Act separates high-level planning from UI grounding and execution, addressing the difficulty of translating textual plans into concrete actions. This modularity supports plan revision and independent refinement of planning and execution components.

4.5 Plan-Then-Act Framework

  • SeeAct separates coherent plan generation from UI execution, while UGround and related work use visual grounding to locate and manipulate planned UI components.SeeAct still faces difficulty bridging textual plans and underlying UI elements; UGround's grounding model can evolve independently of planning.Source: §4.5 · p. 22
  • LiMAC separates action-type selection from specialized language generation, invoking a VLM only when natural-language needs arise to reduce compute overhead and improve responsiveness.ClickAgent instead separates MLLM reasoning and action planning from a UI location model that identifies coordinates.Source: §4.5 · p. 23
  • Ponder & Press decomposes instructions with a general MLLM, then maps target elements to pixel coordinates with a GUI-specific MLLM using visual input alone.The framework is described as adaptable across software environments without HTML or UI Trees.Source: §4.5 · p. 23
  • SPlanner models application interaction logic as an EFSM, producing a stable start-to-goal path translated into a natural-language plan for a separate VLM executor.This separates high-level strategic planning from visual grounding and execution.Source: §4.5 · p. 23
  • Separating planning from execution lets agents revise ambiguous or infeasible plans before acting, while planning and UI modules can be improved independently.The agent first forms a conceptual plan, often as human-readable instructions, then grounds and executes it on the device UI.Source: §4.5 · pp. 22–23

4.6 Comparative Analysis of Frameworks

The framework comparison treats Perception, Brain, and Action as components to assemble for a task rather than mutually exclusive architectures. It highlights trade-offs in perception speed, cognitive efficiency and memory, and action-space dexterity.

4.6 Comparative Analysis of Frameworks

  • The best mobile-agent framework is a task-dependent assembly of Perception, Brain, and Action components, not a fixed choice among single-agent, multi-agent, and Plan-Then-Act paradigms.Table 3 summarizes this component-wise analysis.Source: §4.6 · p. 23
  • Rich perceptual inputs, including screenshots, UI trees, SoM, and OCR, improve UI understanding but make each step slow and computationally expensive.The discussion also identifies difficulty perceiving dynamic content such as animations and embedded videos.Source: §4.6 · p. 23
  • Single-step execution can take several seconds to tens of seconds, while multi-agent specialization may raise task success at the cost of latency and operational expense.The paper suggests acceleration methods and adaptive or hybrid frameworks that scale agent teams with task complexity.Source: §4.6 · pp. 23–24
  • Current agents may retain context within a task session but lack persistent memory for preferences and mistakes across sessions, limiting user-centric adaptation.The paper points to demonstration-based learning and adaptation from a few examples as directions for personalized behavior.Source: §4.6 · p. 24
  • Most current action spaces support basic taps, swipes, and typing but lack dexterity for drag-and-drop, pinch-to-zoom, and other multi-touch gestures.Proposed directions include learned complex action primitives and low-level continuous touch-event sequences.Source: §4.6 · pp. 24–25

5 LLMs for Phone Automation

This section surveys prompt engineering and training-based adaptation for LLM phone automation, spanning text-based, multimodal, and task-specific GUI approaches. It covers task-specific visual understanding, supervised fine-tuning, and reinforcement learning, including the trade-offs and hybrid development lifecycle of prompt-based and trained methods.

5.2 Training-Based Models

  • Training-based methods adapt LLMs to phone GUI tasks using GUI-specific data, including supervised fine-tuning and reinforcement learning (Cheng et al. (2024); Song et al. (2024a)).The survey frames task-specific architectures, supervised fine-tuning, and reinforcement learning as the next parts of this training-based discussion.Source: §5 · p. 25, §5.2 · p. 29

5.1 Prompt Engineering

  • Prompt engineering guides pretrained LLMs without fine-tuning by combining task descriptions with UI and device state, action history, action options, and chain-of-thought guidance.The prompt’s task description, interface information, real-time phone state, and action history provide context for selecting the next operation.Source: §5.1 · p. 26

5.1.1 Text-Based Prompting

  • Text-based agents select UI-tree elements by index, while work progresses from single-instruction mapping to sequential decisions, app memory, hierarchical reasoning, and accessibility testing.DroidBot-GPT enables sequential decision-making (Wen et al. (2023)); AutoDroid uses a UI Transition Graph to build app memory (Wen et al. (2024)); MobileGPT uses hierarchical decisions (Lee et al. (2023)); AXNav supports accessibility testing (Taeb et al. (2024)).Source: §5.1.1 · pp. 27–28

5.1.2 Multimodal Prompting

  • Multimodal prompting combines screenshots with textual UI information; agents use indexed visual elements or direct coordinates, but coordinate grounding remains a reliability challenge.SoM-based approaches annotate actionable widgets with IDs (Yan et al. (2023)), while coordinate-based agents locate elements directly; methods include VisionTasker (Song et al. (2024b)) and Mobile-Agent (Wang et al. (2024b;a)).Source: §5.1.2 · pp. 28–29

5.2.1 Task-Specific LLM-based Agents

  • General-purpose GUI architectures target phone-automation challenges by improving direct GUI interaction, high-resolution visual recognition, comprehensive environment perception, and conditional action prediction through multimodal inputs and architectural designs.CogAgent supports input resolutions up to 1120 × 1120 to recognize small page elements and text; ShowUI reduces redundant tokens by 33%, accelerating training speed by 1.4 times.Source: §5.2.1 · p. 31
  • UI-grounding models localize elements from descriptions, with approaches using layout-guided contrastive learning, screen history, action history, and shared action spaces.LVG addresses application and context sensitivity (Qian et al. (2024b)); UI-Hawk uses screen sequences (Zhang et al. (2024d)); TAG reports performance comparable to or better than tuned approaches (Xu et al. (2024a)).Source: §5.2.1 · pp. 32–33
  • UI-referring models describe specified screen elements, using resolution-aware image partitioning, adaptive grids, or predicted next-screen descriptions to support accuracy and data efficiency.Ferret-UI handles varying screen aspect ratios (You et al. (2024)); Ferret-UI 2 adds adaptive N-grid partitioning (Li et al. (2024e)); Textual Foresight predicts the next UI state’s global description from the current screen and a local action (Burns et al. (2024)).Source: §5.2.1 · p. 33
  • Screen question-answering models extract screen content and functionality, with ScreenAI, WebVLN, UI-Hawk, and MP-GUI addressing visual, structural, historical, or multimodal understanding.MP-GUI combines graphical, textual, and spatial perceivers with an adaptive fusion gate, and reports strong GUI-understanding performance despite limited training data (Wang et al. (2025e)).Source: §5.2.1 · pp. 33–34
  • Phone UI-specific models target UI grounding, UI referring, and screen question answering through techniques such as layout-guided contrastive learning, history-aware encoding, and flexible patching.These specialized architectures address screen understanding to improve agents’ GUI navigation and interaction.Source: §5.2.1 · p. 34

5.2.2 Supervised Fine-Tuning

  • Fine-tuning also supports accessibility, compact deployment, long-horizon memory, and error recovery through specialized models, data, and iterative training.IconDesc generates icon alt-text from partial UI data; ZonUI-3B uses staged specialization, Chain-of-Memory separates short- and long-term memory, and Agent-R trains on corrected trajectories.Source: §5.2.2 · pp. 35–36
  • Supervised fine-tuning improves GUI grounding, OCR, GUI knowledge, and cross-app navigation using targeted datasets and training strategies.Examples include screenshot-based grounding with SeeClick, GUIEnv’s OCR and grounding data, and GUI Odyssey’s cross-app trajectories for training OdysseyAgent.Source: §5.2.2 · pp. 35–36

5.2.3 Reinforcement Learning

  • DigiRL Bai et al. (2024) increased Android-in-the-Wild success rate by 49.5% absolute, while DistRL Wang et al. (2024h) improved success rate by 20% relative on general Android tasks.DigiRL combines offline initialization with offline-to-online learning; DistRL uses centralized training and decentralized data collection.Source: §5.2.3 · p. 36
  • Phone-agent RL spans offline-to-online and distributed learning, environment-free value estimation, rule-based optimization, and online self-reflection and adaptation.GUI-G1 identifies risks from excessive chain-of-thought, reward hacking, and GRPO bias, then proposes targeted changes to templates, rewards, and objectives.Source: §5.2.3 · pp. 36–38
  • Web RL work learns from exploration failures and successful or unsuccessful trajectories, while PC OS agents demonstrate screenshot-based control across operating systems.ETO and Agent Q develop web-navigation methods; ScreenAgent uses screenshots and natural-language instructions across Windows, Ubuntu, and macOS.Source: §5.2.3 · pp. 38–39

5.3 Comparative Analysis of Modeling Approaches

  • Prompt engineering offers flexible, low-development-cost generality, whereas training-based methods can improve in-domain accuracy but require costly data and risk overfitting.Prompting can incur substantial repeated API inference costs and has a performance ceiling; RL can support more adaptive policies but brings sample inefficiency and reward-design challenges.Source: §5.3 · pp. 39–40
  • A hybrid lifecycle combines a general foundation model, supervised fine-tuning on diverse GUI data, and reinforcement-learning refinement; training-based methods also enable smaller, responsive on-device agents.The paper frames prompting, SFT, and RL as complementary development tools rather than competing alternatives.Source: §5.3 · pp. 40–41

6 Datasets and Benchmarks

This section presents datasets and benchmarks as complementary foundations for developing and evaluating phone GUI agents. It surveys resources for training and assessment.

6 Datasets and Benchmarks

  • Phone GUI agent research needs diverse datasets, standardized benchmarks, and robust evaluation methodologies.Source: §6 · p. 41
  • Datasets provide rich annotations and task diversity, while benchmarks provide structured environments and metrics for consistent, reproducible assessment.Source: §6 · p. 41
  • The section reviews datasets for language grounding, UI navigation, and multimodal interaction, alongside benchmarks addressing reproducibility, generalization, and scalability.Source: §6 · p. 41
  • Table 9 summarizes dataset scale through distinct actions, demonstration sequences, applications, natural-language instructions, and average task steps.Source: Table 9 · p. 41

6.1 Datasets

Phone GUI datasets have expanded from foundational interface collections to resources for instruction grounding, feasibility, cross-app navigation, and agent training. Their diverse annotations and scenarios support training and evaluating more capable and generalizable agents.

6.1 Datasets

  • Rico provides screenshots, view hierarchies, and UI metadata as an early foundation, while MobileViews is currently the largest GUI dataset.RICO Semantics and GUI-WORLD also expanded dataset types and coverage.Source: §6.1 · p. 42
  • PixelHelp grounds natural-language instructions in mobile actions, UGIF extends grounding to multilingual and multimodal settings, and MoTIF annotates infeasible tasks and follow-up questions.These datasets target instruction mapping, cross-lingual challenges, and task feasibility.Source: §6.1 · p. 42
  • UIBert pre-trains jointly on UI images and text, while Meta-GUI pairs task-oriented dialogues with GUI operation traces.UIBert uses pre-training tasks based on correspondences between UI features.Source: §6.1 · p. 42
  • AndroidControl studies training-data scale and generalization, AITW supplies hundreds of thousands of interaction episodes, and AITZ adds Chain-of-Action-Thought annotations.GUI Odyssey targets navigation across apps, tasks, and devices.Source: §6.1 · p. 42
  • AMEX combines GUI-element grounding, functionality descriptions, and stepwise instruction-action chains to represent mobile interfaces at multiple levels.Its annotations aim to support deeper understanding of page layouts and UI-element functionality.Source: §6.1 · pp. 42–43
  • DreamUI, OS-Genesis, Learn-by-interact, Ferret-UI 2, FedMobileAgent, JEDI, and ScaleTrack illustrate generated, interaction-collected, and enriched data pipelines.DreamUI includes 9,774 labeled interfaces; JEDI constructs over 4 million examples, while ScaleTrack adds backtracking data for GUI planning.Source: §6.1 · p. 43

6.2 Benchmarks

Phone GUI benchmarks provide standardized settings for comparing agents, with evaluation pipelines spanning controlled environments, real-world tasks, and automated assessment. Their metrics cover task completion alongside action quality, efficiency, reliability, reasoning, and compliance.

6.2 Benchmarks

  • Benchmarks standardize environments, tasks, and metrics for reproducible comparisons, and Table 10 summarizes their characteristics.The survey notes that benchmarks address scalability, reproducibility, generalization, and integration of LLMs and LMMs.Source: §6.2 · p. 43, Table 10 · p. 44, §6.2.1 · p. 46

6.2.1 Evaluation Pipelines

  • Evaluation pipelines range from controlled environments such as MobileEnv and AndroidLab to AndroidWorld’s 116 programmatic tasks across 20 apps.AndroidWorld uses parameterized, natural-language tasks and ground-truth rewards.Source: §6.2.1 · pp. 44–45
  • SPA-BENCH and A3 broaden practical comparisons with cross-app and multilingual tasks, while Mobile-Bench-v2 adds noisy and ambiguous scenarios.SPA-BENCH includes 340 tasks and automated evaluation; Mobile-Bench-v2 uses offline multi-path evaluation.Source: §6.2.1 · pp. 45–46
  • Automated and capability-focused pipelines include AutoEval’s human-comparable judge, AgentStudio’s grounding and success-detection datasets, and LlamaTouch’s on-device evaluation.AutoEval reports over 94% accuracy compared with human verification.Source: §6.2.1 · pp. 44–45

6.2.2 Evaluation Metrics

  • Task completion and action-quality metrics measure success at task or sub-goal level, action correctness, trace alignment, and reasoning accuracy.End-to-end completion captures multi-step task completion, while Correct Trace measures alignment of the full trajectory with ground truth.Source: §6.2.2 · p. 46
  • Other metrics assess evaluation reliability, aggregate reward, resource use, reasoning, format compliance, and completion awareness or reflection.Efficiency measures include token consumption, step efficiency, and redundant action-path detours.Source: §6.2.2 · pp. 46–47

6.3 Performance Analysis and Current Limitations

The survey compares representative methods across visual grounding, offline execution, and online real-world benchmarks. Specialized approaches and reinforcement learning show strong results, but offline performance does not reliably carry over to complex online tasks.

6.3 Performance Analysis and Current Limitations

  • Specialized GUI models and targeted frameworks improve performance over basic foundation models across all reported benchmarks.The survey describes fine-tuning, reinforcement learning, and designed prompting as adaptation approaches.Source: §6.3 · p. 47
  • RL-based methods dominate offline evaluation, while prompt-based and ensemble methods lead on AndroidWorld, where Mobile-Agent-v3 reaches a 73.3% success rate.Table 11 compares methods across ScreenSpot, GUI-Odyssey, AndroidControl, and AndroidWorld.Source: §6.3 · pp. 47–48, Table 11 · p. 48
  • UI-TARS falls from over 90% on AndroidControl to 46.6% on AndroidWorld, illustrating the offline-to-online performance gap.The survey notes that current RL strategies may not fully capture long-horizon reasoning in dynamic online environments.Source: §6.3 · p. 48
  • Performance disparities across evaluation settings leave reliable real-world deployment as a substantial challenge despite progress in foundational capabilities.The survey identifies the offline-online gap as a research concern.Source: §6.3 · p. 49

6.4 Critical Analysis of Datasets and Benchmarks

Datasets and benchmarks have enabled reproducible progress, but their task coverage, annotations, and evaluation settings leave important gaps between controlled performance and robust real-world phone use. The section identifies dataset limitations and benchmark limitations, including narrow metrics, unrealistic scenarios, and insufficient robustness testing.

6.4.1 Limitations of Datasets

  • Datasets such as AITW emphasize simple, short-horizon, single-app tasks, leaving complex cross-app workflows, conditional logic, memory, and recovery underrepresented.The resulting imbalance is associated with agents that handle atomic subtasks but lack strategic planning for complex objectives.Source: §6.4.1 · p. 49
  • Many datasets record action trajectories without intent or reasoning, encouraging brittle behavioral imitation; richer Chain-of-Action-Thought annotations remain costly to create.A minor UI change can expose the lack of learned task semantics, while annotation expense limits wider adoption.Source: §6.4.1 · p. 50
  • Datasets collected in idealized settings rarely include interruptions or changing interfaces, leaving agents brittle when deployed amid real-world mobile unpredictability.Examples include notifications, pop-ups, permission requests, login requirements, and app updates.Source: §6.4.1 · p. 50

6.4.2 Limitations of Benchmarks

  • Most benchmarks prioritize binary task success, while operational efficiency measures such as step count and redundancy ratios remain secondary.This leaves important aspects of agent quality, including avoiding convoluted paths, less central in evaluation.Source: §6.4.2 · p. 50
  • Benchmark tasks are often atomic and unambiguous, unlike human goals that may be incomplete, context-dependent, conversational, or require clarification.Even programmatically generated AndroidWorld tasks vary structured templates rather than capturing the full range of underspecified goals described here.Source: §6.4.2 · p. 51
  • Robustness testing is uncommon: most benchmarks use static, predictable environments, while B-MoCA tests generalization through randomized UI layouts and language settings.The section characterizes agents as brittle under real-world unpredictability when this testing is absent.Source: §6.4.2 · p. 51
  • LLM-as-a-judge evaluation can incur high costs and hallucinate on long screenshot trajectories, while proposed hybrid judges combine rule checks with LLM assessments.The proposed directions also include lightweight open-source judges and less manual reward modeling.Source: §6.4.2 · pp. 51–52

7 Challenges and Future Directions

The paper frames future progress as resolving trade-offs in agent architecture, modeling, data and evaluation, deployment, and security. It argues for deeper datasets, more holistic evaluation, adaptive resource use, and security and ethics built into the ecosystem.

Frameworks and Components: The Performance Trilemma

  • Agent design faces a speed-memory-cost trilemma, with perception latency, slow stepwise reasoning, limited persistent memory, simple action spaces, and costly powerful models.The section points to adaptive resource scaling by task complexity, real-time perception, faster reasoning, persistent memory, and more expressive actions as research needs.Source: §7 · p. 52

Modeling Approaches: Bridging the Generalization-Specialization Gap

  • Prompting generalizes to new apps but faces base-model performance and operating-cost limits, whereas SFT and RL can specialize but require costly data and may not generalize to unseen apps.The survey describes SFT followed by RL as an emerging hybrid approach and reports RL-based methods consistently outperforming SFT-only approaches in its performance analysis.Source: §7 · p. 52

The Ecosystem of Data and Evaluation: Addressing the Capability Gap

  • Agents perform strongly on foundational offline tasks such as visual grounding but have low success on complex online tasks such as AndroidWorld, alongside gaps in task complexity, semantic depth, and dynamism in datasets.The proposed direction is compositional cross-app data and benchmarks that assess efficiency, robustness, and ambiguity handling.Source: §7 · p. 52

On-Device Deployment: The Practicality Trilemma

  • Lightweight on-device models have demonstrated low latency and enhanced privacy, but practical agents still need robustness to UI changes, interruptions, and variable networks under device size and inference-time constraints.The paper presents the gap between proofs of concept and robust real-world use as an outstanding challenge.Source: §7 · p. 53

Ensuring Reliability, Security, and Ethical Alignment

  • Security risks include environmental injection, poisoned visual triggers that persist after clean-data fine-tuning, and agent behavior shaped by biases in training data or base models.The survey calls for robust security protocols, safety evaluation, privacy-preserving methods, and continuous monitoring.Source: §7 · pp. 53–54
  • Federated learning trains locally and shares aggregated updates rather than raw data, while safety benchmarks and monitoring are proposed for evaluating and mitigating risks.FedMABench evaluates distributed mobile agents; MobileSafetyBench targets harmful instructions and indirect prompt injections.Source: §7 · p. 54

Challenges and Future Directions

  • The paper calls for a broader ecosystem of deeper datasets, holistic evaluations, adaptive frameworks, and security and ethics built in from the start.It presents these efforts as a route for next-generation agents to become efficient, trustworthy, and integrated into daily use.Source: §7 · p. 54

8 Conclusion

The survey synthesizes LLM-driven phone GUI agents as a shift from static scripts to systems that perceive, reason about, and operate mobile GUIs. It reviews agent frameworks and modeling techniques, identifies persistent obstacles, and outlines research directions toward robust, user-friendly, and secure solutions.

8 Conclusion

  • The survey presents LLM-driven phone automation as a shift from static scripts to dynamic systems that perceive, reason about, and operate mobile GUIs.It frames these systems as a paradigm shift in phone automation.Source: §8 · p. 54
  • Single-agent, multi-agent, and plan-then-act frameworks address different challenges in task complexity, adaptability, and scalability.Source: §8 · p. 55
  • Prompt engineering and training-based techniques, including supervised fine-tuning and reinforcement learning, bridge user intent and device action.Source: §8 · p. 55
  • Persistent challenges include dataset coverage, resource-constrained deployment, personalization, and security, while standardized metrics and protocols are proposed for fair comparison.Source: §8 · p. 55
  • The survey anticipates that refinements in architectures, on-device inference, and multimodal integration will expand agent capabilities, with embodied AI and AGI among future directions.Source: §8 · p. 55

Related papers

Loading 2504.19838v3…