PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on (M)LLM-Based GUI Agents

Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, Kaitao Song, Jian Shao, Weiming Lu, Jun Xiao, Yueting Zhuang

arXiv:2504.13865v2cs.HCcs.AIcs.CLcs.CV

TL;DR

GUI Agents address the inadequacy of traditional automation for complex, changing interfaces. This survey synthesizes their four-part architecture, evaluation methods, and remaining challenges, concluding that the field needs more reliable components and standardized assessment.

  • Problem

    Traditional rule-based automation is inadequate for complex and dynamic interfaces, motivating systematic understanding of GUI Agents and their evaluation.

  • Method

    The survey organizes GUI Agent research into perception, exploration, planning, and interaction, and reviews technologies, applications, benchmarks, challenges, and future directions.

  • Results

    The review identifies four integrated components and documents evaluation frameworks spanning platforms and scenarios while examining their methodological limitations.

  • Takeaways & Limitations

    GUI Agent development should address accurate localization, knowledge retrieval, long-horizon planning, safe execution, and standardized evaluation.

  • Takeaways & Limitations

    Trajectory-based evaluation can produce high false-negative rates because datasets often provide one reference path despite multiple effective solutions.

Abstract

from arXiv · show

Graphical User Interface (GUI) Agents have emerged as a transformative paradigm in human-computer interaction, evolving from rule-based automation scripts to sophisticated AI-driven systems capable of understanding and executing complex interface operations. This survey provides a comprehensive examination of the rapidly advancing field of LLM-based GUI Agents, systematically analyzing their architectural foundations, technical components, and evaluation methodologies. We identify and analyze four fundamental components that constitute modern GUI Agents: (1) perception systems that integrate text-based parsing with multimodal understanding for comprehensive interface comprehension; (2) exploration mechanisms that construct and maintain knowledge bases through internal modeling, historical experience, and external information retrieval; (3) planning frameworks that leverage advanced reasoning methodologies for task decomposition and execution; and (4) interaction systems that manage action generation with robust safety controls. Through rigorous analysis of these components, we reveal how recent advances in large language models and multimodal learning have revolutionized GUI automation across desktop, mobile, and web platforms. We critically examine current evaluation frameworks, highlighting methodological limitations in existing benchmarks while proposing directions for standardization. This survey also identifies key technical challenges, including accurate element localization, effective knowledge retrieval, long-horizon planning, and safety-aware execution control, while outlining promising research directions for enhancing GUI Agents' capabilities. Our systematic review provides researchers and practitioners with a thorough understanding of the field's current state and offers insights into future developments in intelligent interface automation.

1 INTRODUCTION

GUI Agents address the limits of brittle rule-based automation by combining LLM and multimodal advances with four coordinated components: perception, exploration, planning, and interaction. This survey organizes the field, reviews its evaluation methods, and identifies persistent challenges in reliable interface automation.

  • Motivation: GUI Agents emerged because rule-based scripts and screen playback struggled with modern interfaces’ complexity and dynamicity.They aim to connect natural-language instructions with complex interface operations.
  • Motivation: Recent LLM and multimodal advances support sophisticated GUI Agents that can handle complex tasks and varying interface states.Reasoning models have also begun to be applied to GUI automation.
  • Core components: Modern GUI Agents are organized around four components: perception, exploration, planning, and interaction.Together, these components support interface understanding, knowledge management, planning, and reliable execution.
  • Core components: Perception combines text-based parsing and multimodal understanding, while exploration builds knowledge from internal models, historical experience, and external information.Key challenges include element localization, dynamic-content tracking, resolution adaptation, and effective knowledge retrieval.
  • Core components: Planning uses thought-chain and reactive frameworks, whereas interaction generates and executes GUI or API actions with increasing attention to safety and error handling.Long-horizon planning, error recovery, and consistency across interaction paths remain challenges.
  • Survey scope: The survey analyzes technologies, benchmarks, applications, challenges, and future research directions across desktop, mobile, and web platforms.It emphasizes that standardized evaluation remains difficult because interfaces and interactions vary widely and evaluation must cover correctness and user experience.

2.1 Overview

The survey models GUI Agents as a four-module information-processing pipeline from perception through exploration and planning to interaction. It also classifies architectures by their use of multimodal models, training, multiple agents, or text-based interface representations.

  • Architectural framework: The architecture is decomposed into perception, exploration, planning, and interaction, reflecting progression from sensory input to physical action.This decomposition provides analytical clarity for examining contemporary GUI Agent systems.
  • Architectural categories: Recent GUI Agent architectures fall into four categories: general MLLM-based, training MLLM, multi-agent, and LLM-based approaches.The categories are identified through analysis of literature published from 2021 to 2025.
  • Perception: Perception parses DOM or XML structures, processes screenshots multimodally, and uses tools such as OCR and object detection.It supplies the interface understanding required by later pipeline stages.
  • Planning: Planning supports reasoning and decision-making through LLM, MLLM, and advanced reasoning models, alongside iterative or decomposition-based planning and verification.The taxonomy examines reasoning frameworks, task-planning methods, and plan-reliability mechanisms.
  • Interaction: Interaction translates internal representations and plans into concrete interface actions using classifications of action types, generation strategies, and safety controls.It is the final execution stage of the processing pipeline.

2.2 Perception

GUI Agent perception mechanisms are organized around text-based parsing and multimodal understanding, with specialized tools extending visual element localization. The survey also situates perception within broader taxonomies of knowledge, reasoning, interaction, and safety mechanisms.

  • Text-based perception: Text-based GUI Agents parse structured interface representations such as HTML, XML, and DOM trees to understand screen content.These agents rely on textual information rather than directly processing screenshots.
  • Multimodal perception: Multimodal GUI Agents process screenshots directly, often using models trained or fine-tuned for GUI grounding to localize and interpret interface elements.General multimodal models include GPT-4V, Claude-3.5 Sonnet, and GPT-4o.
  • Broader component taxonomy: The survey’s broader GUI Agent taxonomy includes internal knowledge, historical records, external sources, retrieval, reasoning, planning, interaction, API integration, and operation safety.These categories complement perception within the survey’s analysis of GUI Agent components.
  • Tool-enhanced perception: Specialized tools augment perception through OCR, icon recognition, object detection, and visual grounding for screen-element localization.Examples include IconNet, YOLO, Grounding DINO, OCR, DINO+SAM, and multi-tool pipelines such as OmniParser.
  • Perception limitations: Text-based perception is limited by redundant and noisy textual representations and by incomplete or inaccessible HTML or XML structures.These limitations can distract agents from relevant interactive elements and restrict interface understanding across platforms.

2.3 Exploration

GUI Agent exploration acquires and manages knowledge through internal representations, historical experience, and external information, then constructs, stores, and retrieves it for task decisions.

  • Motivation: Effective exploration addresses limitations caused by heterogeneous interfaces and evolving content by improving knowledge acquisition for planning and decision-making.GUI integration is presented as a response to agents’ internal-knowledge constraints in changing interface environments.
  • Exploration Types: GUI Agents acquire knowledge through internal representations, historical task experience, and external information retrieved from sources such as the internet.These sources support knowledge about interface functions, task trajectories, environmental feedback, skills, and external rules.
  • Historical Exploration: Historical exploration uses environmental feedback, successful task trajectories, and skill repositories to inform current decision-making.Feedback may be transformed into textual reflections or stored alongside execution histories and reusable skills.
  • Knowledge Processing: GUI Agents construct conceptual knowledge bases by converting explored concepts into logical repositories, organizing them structurally, and implementing retrieval mechanisms.The framework is described as a sequence of knowledge construction, storage, and application processes.
  • Knowledge Construction: Static construction gathers knowledge offline, whereas dynamic construction updates memory during interaction to refine contextual understanding.AutoDroid and AppAgent exemplify offline approaches, while MobA and WKM illustrate real-time updating based on interaction or expert feedback.

2.4 Reasoning and Planning

GUI Agent planning combines advanced reasoning frameworks with iterative or decomposition-based task planning and feedback mechanisms to manage complex interface tasks.

  • Planning Frameworks: GUI Agent planning draws on LLM, MLLM, and advanced reasoning frameworks to support systematic task reasoning and decision-making.The survey examines chain-, tree-, graph-, and action-interleaved reasoning approaches.
  • LLM Reasoning: Chain-of-Thought decomposes complex tasks into sequential intermediate reasoning steps, with zero-shot and few-shot prompting as two variants.The representation is an input followed by intermediate thoughts and a final output.
  • Advanced Reasoning: Tree-of-Thoughts generates multiple reasoning paths at each step and uses self-evaluation to prune options and plan a preferred path.Extensions combine tree-based reasoning with Monte Carlo Tree Search.
  • Reasoning and Acting: ReAct interleaves reasoning and actions by generating a thought, executing an action, observing the result, and repeating until completion.Its action space combines allowable actions with the reasoning trajectory.
  • Task Planning: Iterative planning adjusts the plan from current state and feedback, whereas task decomposition planning executes subtasks derived from a complex goal.Decomposition can reduce risks of catastrophic forgetting or hallucination by preserving links between subtasks and the overall task.
  • Feedback: Internal feedback lets agents refine plans through self-generated signals stored in prompts or memory modules, improving task-execution accuracy and reliability.External feedback is separately obtained from environmental responses to actions.

2.5 Interaction

GUI Agent interaction translates user goals into executable operations through action generation, GUI grounding, broader operation strategies, and constrained action spaces.

  • Interaction Components: Interaction is organized around action goals, action generation, and action spaces that determine how agents execute user-directed tasks.These components cover objectives, operation synthesis, and the available interface-constrained operations.
  • Action Generation: Action generation converts user intentions into executable operations by interpreting instructions and understanding the GUI environment.The survey divides action generation into GUI grounding and action strategy.
  • GUI Grounding: GUI grounding maps abstract instructions to specific interface elements and coordinates for precise screen-based clicks.The survey identifies MLLM-based, LLM-based, and domain-specific grounding approaches.
  • Action Strategies: Memory-based action generation uses prior interactions or external knowledge, while planning-based generation relies on structured plans created before execution.Both strategies support multi-step operations such as scrolling, navigation, typing, and other non-click actions.
  • Action Space: The action space comprises user-action simulation and API invocation, enabling either human-like interface operations or executable function calls.Simulated actions include tapping, pressing, swiping, typing, returning, and exiting; API invocation uses configured or generated tools.

3 APPLICATION

GUI Agents now support automation across mobile, desktop, web, and game environments, each imposing distinct interaction, perception, planning, and control requirements.

  • Application domains: GUI Agents span mobile devices, desktop computers, web browsers, and games, demonstrating broad deployment versatility alongside environment-specific technical challenges.These domains differ in touch interaction, application complexity, dynamic web content, and real-time visual control.
  • Mobile: Mobile agents automate system settings, application management, and social-media interactions while requiring precise touch localization, dynamic state tracking, and safety protocols.AppAgent-V2 requires confirmation for sensitive operations and maintains operation logs.
  • Desktop: Desktop agents combine GUI interaction, APIs, element detection, and hierarchical attention to automate operating-system functions, native applications, and productivity software.OS-Copilot translates high-level instructions into system-operation sequences and combines direct API calls with GUI interaction when necessary.
  • Web: Web agents perform navigation, e-commerce, form, and extraction tasks using DOM understanding and state tracking to handle dynamic content, page loads, and network variation.Web agents must maintain context across page loads and state transitions.
  • Games: Game agents combine visual understanding, strategic planning, tactical execution, and progressive skill acquisition under real-time constraints.VOYAGER learns and combines basic actions into complex operation sequences for adaptation to new scenarios.

4 DATASETS AND BENCHMARKS

GUI Agent benchmarks cover static replicas, simulated environments, and real-world applications across platforms and tasks. Static and simulated settings improve reproducibility, while real environments preserve operational uncertainty relevant to deployment.

  • Evaluation environments: Current GUI Agent evaluation environments comprise static replicas, simulated environments, and real-world environments across mobile, web, and desktop platforms.The survey organizes benchmark analysis around environments, tasks, and evaluation strategies.
  • Static replicas: Static replicas record interfaces as screenshots, HTML snapshots, or other fixed data, enabling reproducible evaluation without running the original application.AitW contains 715,142 episodes, 5,689,993 screenshots, and 30,378 unique task instructions.
  • Static replicas: Static datasets support large-scale benchmark construction but cannot exhaustively represent all possible execution paths.This limitation affects evaluation when agents can complete tasks through alternative trajectories.
  • Simulated environments: Simulated environments isolate and control interactions, eliminating variables from dynamic online content and improving evaluation repeatability.Examples include MiniWoB++, WebShop, WebArena, VisualWebArena, and MMInA-derived tasks.
  • Real-world environments: Real-world evaluation exposes agents to authentic websites and applications but introduces weather, network, security, popup, and maintenance uncertainties.Despite these challenges, real-environment task construction is described as a prevailing research trend important for advancing GUI Agents.

4.2 Task

GUI Agent benchmarks evaluate perception, operation, and dialogue capabilities. Their tasks range from interface understanding and single-step actions to complex multi-step execution and user-agent interaction.

  • Task taxonomy: GUI Agent datasets and benchmarks classify tasks into perception, operation, and dialogue categories targeting different capabilities.This taxonomy covers interface understanding, action execution, and interactive task completion.
  • Perception: Perception tasks evaluate recognition, extraction, and interpretation of interface elements as the basis for subsequent decisions and interactions.WebArena and VisualWebArena include web information-retrieval challenges testing visual comprehension.
  • Operation: Operation tasks assess planning and execution across software operations, web navigation, and other goal-directed interactions.Benchmarks range from single-step action prediction to multi-hop navigation across independent websites.
  • Operation: Broader operation benchmarks cover application operations, system settings, programming, multimedia tasks, and GUI defect discovery.Examples include Mobile-Env, OSWorld, AndroidWorld, WindowsAgentArena, and the GUI Testing Arena.
  • Dialogue: Dialogue-based tasks address mid-task requirement changes and agent requests for user confirmation that end-to-end instruction benchmarks generally omit.Multi-turn dialogue datasets span web, Android, and multi-platform GUI environments.

4.3 Evaluation Strategy

GUI Agent evaluation strategies balance process fidelity against outcome validity. Trajectory-based methods provide fine-grained action metrics but can penalize valid alternatives, whereas goal-oriented and graph-driven methods better accommodate solution flexibility.

  • Evaluation paradigms: Trajectory-based evaluation compares generated action or state sequences with references, while goal-oriented evaluation judges whether the final objective is achieved.Figure 9 contrasts these approaches with graph-driven evaluation, which represents operations as connected nodes and alternative paths.
  • Trajectory-based methods: Exact step-wise matching can unfairly reject agents that use exploratory actions, motivating LCS-based comparison and critical-node evaluation.LCS tolerates deviations, while critical-node methods require trajectories to contain necessary states rather than an exact path.
  • Trajectory-based methods: Trajectory-based methods can produce high false-negative rates because datasets often provide one reference path despite multiple valid solutions.Human verification scores have been higher than trajectory-based scores in prior studies.
  • Trajectory-based methods: Trajectory details enable fine-grained action-category metrics, but reliability issues limit the validity of trajectory-based outcomes.The trade-off is between process-sensitive measurement and robustness to alternative execution paths.
  • Goal-oriented approaches: Goal-oriented evaluation can rely on output or final-state signals, but surface-level UI evidence may fail to capture whether complex operations truly succeeded.WindowsAgentArena extracts critical final-state components and uses task-specific scripts for more accurate success determination.
  • Graph-driven evaluation: Graph-driven evaluation models tasks as directed graphs with sequential and parallel relationships, combining granular assessment with multiple valid paths.Crab uses graph structure to bridge trajectory-level detail and goal-oriented solution flexibility.

4.4 Challenges

GUI Agent evaluation is constrained by high cost, limited dataset diversity, and unresolved security and privacy risks. These limitations reduce benchmark coverage of realistic, globally varied, and safely deployable interface use.

  • Evaluation cost: Benchmarking GUI Agents can require substantial time, computation, financial resources, and human judgment, especially for complex multi-step tasks.OSWorld tasks typically require approximately 10–20 steps, while other benchmarks incur human-evaluation or commercial-API costs.
  • Dataset diversity: Existing datasets often lack diversity in modality, language, cultural context, and interface conditions, limiting assessment of generalizable agents.Mind2Web primarily covers websites frequently visited by U.S. users, while many datasets remain English-only or text-focused.
  • Interface coverage: Popular-application datasets can underrepresent less-optimized interfaces, while interface drift and environment randomization remain important evaluation concerns.AndroidWorld focuses on open-source applications with more than one million downloads, whose interfaces are often highly optimized.
  • Security and privacy: GUI Agent benchmarks raise privacy and security risks because collected data may contain sensitive information and agents may require extensive system privileges.Potentially exposed data includes personal details, financial records, browsing histories, and authentication credentials; system-level permissions can be exploited.
  • Security and privacy: Current benchmark security measures often isolate environments without comprehensive real-world threat modeling or standardized security protocols.Proposed safeguards include anonymization, permission management, adversarial testing, transparent data-collection documentation, and explicit consent.

5 FUTURE DIRECTIONS AND CHALLENGES

Future GUI Agent research should make datasets and evaluations more realistic while improving visual grounding, interruption recovery, decision-making, and adaptive learning. Reinforcement learning is promising for these challenges, but its effectiveness and reward design remain open questions.

  • Data collection and evaluation: Future datasets should represent interruptions, unexpected system prompts, changing interface states, and task resumption in realistic multi-step interactions.Such scenarios are currently rarely captured despite their relevance to real-world usage.
  • Data collection and evaluation: Standardized evaluations should assess GUI Agents across accuracy, robustness, efficiency, and contextual understanding using diverse interaction data.The proposed direction combines realistic usage patterns with holistic, cross-dimensional assessment.
  • Multimodal perception and visual grounding: GUI Agents remain unreliable at understanding hierarchical screens because dense, small interface elements cause hallucinated elements and localization errors.Specialized visual parsing tools and reasoning over interface hierarchies are identified as promising responses.
  • Strategic planning: Unexpected popups, network errors, and interface changes can cause premature termination or repeated failed actions during complex tasks.Agents also struggle to choose among ambiguous interaction paths and to handle transient elements such as tooltips and modal dialogs.
  • Reinforcement learning: Reinforcement learning is a promising direction for GUI Agents because practical tasks require multi-step reasoning and decision-making, but its performance benefits remain unconfirmed.High-dimensional screen states and reward functions balancing task completion with user experience create major challenges.

6 CONCLUSION

The survey organizes GUI Agent research around four modules—perception, exploration, planning, and interaction—and reviews their applications, benchmarks, limitations, and future directions. It presents this synthesis as a foundation for advancing intelligent interface automation across diverse platforms.

  • Survey synthesis: The survey structures GUI Agent architecture into perception, exploration, planning, and interaction modules.It examines how these modules collectively support interpreting, navigating, and manipulating graphical interfaces across platforms.
  • Survey synthesis: It reviews application scenarios and benchmarks while highlighting computational cost, linguistic and interface diversity, and security concerns.The survey also identifies remaining system challenges and outlines future research directions.
Loading 2504.13865v2…