Large Language Model-Brained GUI Agents: A Survey

1 INTRODUCTION

• GUIs improve accessibility through visual interaction but can be inefficient for repetitive or multi-step workflows, while traditional script- and rule-based automation remains narrow, rigid, and difficult to adapt across dynamic interfaces.These limitations motivate more flexible automation beyond predefined workflows and platform-specific scripts.Source: §1 · p. 1

1.1 Motivation for LLM-Brained GUI agents

• LLM-brained GUI agents use LLMs as cognitive engines to interpret natural-language requests, analyze GUI screens, and flexibly plan and execute actions across platforms without predefined workflows.By combining natural-language understanding, visual recognition, and decision-making, they can generalize across applications and control software through conversational commands.Source: §1.1 · pp. 2–3

1.2 Scope of the Survey

• The survey systematically examines GUI-agent evolution, foundational technologies, frameworks, datasets, Large Action Models, evaluation, applications, and future challenges to guide agent construction and identify research gaps.Its research questions span historical development, components, frameworks, data collection, purpose-built models, metrics and benchmarks, real-world applications, and future directions.Source: §1.2 · p. 3

1.3 Survey Structure

• The paper is organized into sections covering related work and background, agent evolution and technologies, frameworks, data, models, evaluation, applications, limitations, challenges, future directions, and conclusions.Figure 2 provides a structural illustration, while Table 1 lists abbreviations.Source: §1.3 · p. 3

2.1 Survey on GUI Automation

• Prior GUI-automation surveys cover GUI testing, computer-vision-based interface interpretation, and RPA, while identifying persistent challenges in intelligence and robustness.These works address mobile, Windows, and web GUI testing, and traditional RPA’s shift toward more intelligent, adaptive paradigms; LLM-powered GUI agents are presented as a potential response.Source: §2.1 · p. 5

2.2 Surveys on LLM Agents

• Surveys of LLM agents establish foundations for GUI agents through multimodal perception, multi-agent frameworks, reinforcement learning, planning, memory, tool use, and evaluation.Existing work spans mobile and cross-platform agents, while this survey extends prior narrower platform-focused treatments with a holistic perspective and practical reference tables.Source: §2.2 · pp. 5–6
• This survey synthesizes over 500 references across foundation models, data, frameworks, benchmarks, evaluation, and deployments, spanning the full development and deployment lifecycle.It also provides consolidated reference tables, foundational background, and taxonomies intended to support both newcomers and practitioners.Source: §2.2 · pp. 6–7

3 BACKGROUND

• LLM-brained GUI agents are grounded in LLMs [8], agent architectures and tools, and GUI automation, which together provide intelligence, environmental interaction, and automation capabilities.These three advances form the foundation for the survey’s discussion of GUI agents.Source: §3 · p. 7

3.1 Large Language Models: Foundations and Capabilities

• Transformers established the dominant LLM architecture, whose successors—including GPT-4 [90], LLaMA-3 [91], and Gemini [92]—support versatile natural-language, code-generation, and multimodal tasks.LLMs evolved from statistical, recurrent, and early neural models into transformer-based systems with larger scale, stronger generalization, and emergent capabilities –[78].Source: §3.1 · p. 7
• Few-shot learning [77], instruction following [98], long-term reasoning [99], code and tool use [102], and multimodal comprehension [10] enable GUI agents to generalize, plan, act, and interpret screenshots.Domain-specific fine-tuning on user requests, GUI screenshots, and action sequences further specializes LLMs for GUI interaction.Source: §3.1 · p. 7

3.2 LLM Agents: From Language to Action

• LLM agents turn language models into interactive systems whose “eyes and hands” perceive environments and translate textual outputs into actions [46].GUI agents commonly use screenshots and widget trees, perform clicks, keyboard inputs, and touch gestures, and require multimodal perception, planning, memory, and specialized toolkits [59], [51], [54],.Source: §3.2 · pp. 7–8

3.3 GUI Automation: Tools, Techniques, and Challenges

• Traditional GUI automation evolved from scripted testing into RPA and HCI, but scripting and rule-based methods, remain effective mainly for predictable workflows and struggle with dynamic interfaces.Computer vision, OCR, and object detection improve recognition and resilience by identifying GUI elements and extracting text as layouts change [35],,, –, [103].Source: §3.3 · p. 8
• Contemporary GUIs require contextual awareness, motivating LLM-integrated agents that generate adaptive scripts from the current interface state and respond flexibly to complex user requests,.Rigid automation cannot reliably handle dynamic, adaptive elements, on-screen text, user intent, and visual cues,.Source: §3.3 · p. 8

4.1 Early Automation Systems

• Early GUI automation progressed from random, rule-based, and script-based methods toward tools that enabled testing and structured workflow automation but remained inflexible and manually specified.Random testing explored unpredictable sequences but produced redundant trials, [128]; rule-based planning and RPA automated structured processes,,, while script-based tools recorded, replayed, and managed testing lifecycles,.Source: §4.1 · p. 8, §4.1.1 · p. 9, §4.1.2 · p. 9

4.2 The Shift Towards Intelligent Agents

• Machine learning, computer vision, and reinforcement learning made GUI agents more adaptive through screen filtering, widget detection, human-trace modeling, realistic environments, and workflow-guided exploration.These methods improved testing efficiency, interaction coverage, and human-like behavior, but generalization remained limited across unforeseen tasks and environments,.Source: §4.2 · p. 9, §4.2.1 · pp. 9–10, §4.2.3 · p. 10

4.2.2 Natural Language Processing

• Natural-language systems such as RUSS [142] and FLIN [143] mapped commands to GUI actions, but primarily handled simple instructions rather than long-term tasks.Instruction-to-action datasets broadened GUI control, while the approach remained constrained by limited long-horizon capability.Source: §4.2.2 · p. 10

4.3 The Advent of LLM-Brained GUI Agents

• Multimodal LLMs such as GPT-4o [93] transformed GUI automation in 2023 by combining language understanding, visual recognition, reasoning, planning, and generalization in end-to-end agents.Unlike earlier systems requiring separate modules, LLM-based agents support intuitive natural-language interaction across web, mobile, and desktop platforms.Source: §4.3 · p. 10, §4.3.1 · p. 10, §4.3.2 · p. 10

4.3.1 Web Domain

• LLM GUI agents expanded from WebAgent [152] and WebGUM [153] web navigation to smartphone automation through AutoDroid, mobile control systems, and desktop interaction with UFO [19].Web, mobile, and desktop research progressively introduced benchmarked navigation, domain knowledge, refined smartphone control, and interaction across browsers, terminals, multimedia tools, applications, and games.Source: §4.3.1 · p. 10, §4.3.2 · p. 10, §4.3.3 · p. 10

4.3.4 Industry Models

• Industry adoption accelerated with Claude 3.5 Sonnet’s computer-use feature, [164] and OpenAI Operator, signaling growing investment and competition in GUI agents.Operator is described as achieving state-of-the-art performance across various benchmarks, while commercialization is expected to transform interaction with GUI applications.Source: §4.3.4 · p. 10

4.4 GUI Agent vs. API-Based Agent

• GUI agents use universal clicks and keystrokes that generalize across applications and remain observable, whereas API agents are faster and more direct but depend on restricted interfaces and are less visible.GUI workflows can require costly sequential decisions; API actions can reduce execution complexity but may be harder for users to inspect and debug.Source: §4.4 · pp. 10–11
• Hybrid agents combine GUI compatibility with native-API efficiency, balancing broad generalization against task optimization [166].The survey identifies this combination as a critical focus for effective digital-agent design.Source: §4.4 · p. 11

5 LLM-BRAINED GUI AGENTS: FOUNDATIONS AND DESIGN

• LLM-brained GUI agents interpret natural-language requests, perceive GUI state through screenshots or UI element trees, and autonomously execute complex multi-step tasks across software interfaces [19].Their core capabilities derive from LLM-based language understanding, reasoning, and generation, adapted to GUI contexts.Source: §5 · p. 11

5.1 Architecture and Workflow In a Nutshell

• The foundational workflow cycles through environment perception, prompt engineering, model inference, action execution, and continuous memory use until the requested task is completed.Memory tracks prior actions, progress, and environment states, while external memory can support continual learning and adaptation.Source: §5.1 · pp. 11–12

5.2.1 Platform

• Agents operate across mobile, web, and desktop environments, whose distinct interface characteristics and constraints require tailored integration despite a shared automation framework.The platforms differ in interaction modalities, available UI structures, and the kinds of workflows they support.Source: §5.2 · p. 12, §5.2.1 · pp. 12–14
• Mobile agents must manage limited screen space, touch and gesture interaction, varied app architectures, navigation complexity, and security or privacy permissions,,,,.Accessibility frameworks such as Android Accessibility API and iOS VoiceOver Accessibility Inspector expose structured UI information.Source: §5.2.1 · pp. 12–13
• Web agents use HTML and DOM attributes but must continuously reassess dynamic content, responsive layouts, and asynchronous updates, whereas computer agents leverage system automation APIs for broad mouse, keyboard, and multi-window control,,,.These capabilities support intricate workflows but require adaptation across applications with differing levels of UI complexity.Source: §5.2.1 · pp. 13–14

5.2.2 Environment State Perception

• Accurate state perception combines screenshots, widget trees, UI element properties, and complementary computer vision to provide structured and visual representations for planning.Screenshots capture visual context, widget trees expose hierarchy and relationships, properties identify actionable elements, and OCR or object detection helps when structured APIs are incomplete,, [181],, [183], [184].Source: §5.2.2 · pp. 14, 16

5.2.3 Environment Feedback

• Agents evaluate action outcomes through screenshot differences, UI-structure changes, function returns, and exceptions, then retain or revise their plans based on the resulting feedback,, [187].This feedback enables real-time self-reflection and adaptation when interactions succeed, fail, or produce unexpected states.Source: §5.2.3 · p. 16

Prompt Construction

• Effective prompt construction is crucial because it equips the LLM-powered GUI agent to generate appropriate responses and execute tasks successfully.The agent gathers relevant environmental data before formulating the prompt.Source: §5.3 · p. 16

Prompt Components

• Prompts establish task context through the user request, agent instructions, multimodal environment states, and documented available actions.Instructions define roles, rules, inputs, and outputs; environmental states can include clean and annotated screenshots; action documents specify functions, arguments, and return values.Source: §5.3 · p. 16
• Demonstrated input/output examples activate in-context learning [97], helping the model understand and generalize GUI task requirements.These examples enhance the agent’s performance in carrying out its responsibilities.Source: §5.3 · p. 17
• Complementary information, including memory-derived history and retrieval-augmented external knowledge, [189], further refines planning and decision-making.This supplemental context supports inference by providing relevant historical data and retrieved documents.Source: §5.3 · p. 17

5.4 Model Inference

• Inference requires the LLM to generate both a plan and the specific actions needed to execute the user’s request in dynamic environments.The process comprises planning, action inference, and complementary outputs; Figure 13 illustrates an inference output.Source: §5.4 · p. 17

5.4.1 Planning

• Planning decomposes complex tasks into subtasks and timelines, using techniques such as CoT reasoning and, when needed, hierarchical global and local plans.Plans can be stored for reference during later inference steps, while long-term goals remain linked to immediate actions.Source: §5.4.1 · p. 17

5.4.2 Action Inference

• Action inference translates plans into function-call strings for clicks, keyboard inputs, gestures, or API calls, selected from predefined actions or generated as custom code.Custom code can improve adaptability to unforeseen circumstances but may reduce reliability because generated code can contain errors.Source: §5.4.2 · p. 18

5.4.3 Complementary Outputs

• Complementary outputs can provide reasoning, user or system communication, and task-status signals such as whether execution should continue or has finished.These outputs are tailored to specific needs and enrich the GUI agent’s overall performance.Source: §5.4.3 · p. 18

5.5 Actions Execution

• GUI agents execute inferred commands and gather feedback through three action categories: UI operations, native API calls, and AI tools [197].These categories expand the action space beyond direct interface manipulation.Source: §5.5 · pp. 18–19

5.5.1 UI Operations

• UI operations include mouse, keyboard, touch, and gesture inputs, offering broad compatibility but potentially higher latency because complex workflows require sequential steps.Their platform- and application-specific implementations require tailored handling.Source: §5.5.1 · p. 19

5.5.2 Native API Calls

• Native APIs can execute complex tasks efficiently with a single command, such as sending email through Outlook [198], but are limited in availability and generality [199].They can improve speed and reliability, although creating or exposing such interfaces may require substantial expertise.Source: §5.5.2 · pp. 19–20

5.5.3 AI Tools

• AI tools extend GUI agents to summarization, document enhancement, image or video generation, and collaboration with other agents or Copilot tools.Examples include invoking ChatGPT and DALL·E to address complex tasks and streamline workflows.Source: §5.5.3 · p. 20

5.5.4 Summary

• Advanced GUI agents should combine UI operations for compatibility, native APIs for efficiency, and AI tools for enhanced capabilities across applications.This multifaceted strategy supports more reliable, effective, and productive task completion.Source: §5.5.4 · p. 20

5.6 Memory

• Memory tracks prior actions, outcomes, and task status, transforming GUI agents from reactive systems into proactive, stateful systems capable of self-adjustment [200].Source: §5.6 · p. 20

5.6.1 Short-Term Memory

• Short-term memory stores and updates recent plans, actions, results, and environmental states, but its context-window limit requires selectively discarding or summarizing less relevant details [203].It supports coherent, context-aware decisions throughout multi-step workflows.Source: §5.6.1 · p. 21

5.6.2 Long-Term Memory

• Long-term memory stores completed tasks, successful action sequences, contextual tips, and learned insights externally, allowing retrieval of relevant history for efficient task completion [204].RAG-based retrieval supports adaptation from prior successes, lifelong learning [53], and personalization across sessions, as illustrated by [205].Source: §5.6.2 · pp. 21–22

5.7.1 Computer Vision-Based GUI Grounding

• Computer-vision grounding detects, localizes, segments, and semantically interprets GUI elements from screenshots when APIs or metadata are unavailable or incomplete, [244]–[246].OmniParser combines interactable-icon detection, OCR, and localized icon descriptions, improving GPT-4V’s screen comprehension and interaction accuracy.Source: §5.7.1 · p. 22
• ScreenSpot and ScreenSpot-Pro benchmark GUI grounding across diverse platforms and professional high-resolution environments using annotated screenshots and instructions [25],.ScreenSpot contains over 600 screenshots and 1,200 instructions, while ScreenSpot-Pro includes 1,581 expert-annotated tasks spanning software development, creative tools, CAD, science, and office productivity.Source: §5.7.1 · p. 22

5.7.2 Multi-Agent Framework

• Multi-agent frameworks improve complex GUI workflows through role specialization and coordinated information exchange among agents with complementary skills, [248].Specialized agents can extract documents, retrieve web information, design outputs, and evaluate results in an integrated workflow [249].Source: §5.7.2 · pp. 22–23

5.7.3 Self-Reflection

• Self-reflection lets GUI agents evaluate actions, respond to errors and feedback, revise plans, and complete tasks despite unexpected interface behavior [251].ReAct reassesses action outcomes step by step, while Reflexion incorporates linguistic environmental feedback to avoid repeated mistakes [252], [253].Source: §5.7.3 · pp. 25–26

5.7.4 Self-Evolution

• Self-evolution enables GUI agents to learn from trajectories, extract reusable guidance, and develop new toolkits that improve adaptation, efficiency, and system performance [254].Experience can support fine-tuning or in-context demonstrations, policy extraction through summarization or MCTS [256], [257], and reusable shortcuts or APIs [258].Source: §5.7.4 · p. 26

5.7.5 Reinforcement Learning

• Reinforcement learning models GUI automation as an MDP, mapping perceptions to states, UI operations to actions, and task outcomes to rewards for policy optimization,, [263].DigiRL uses offline initialization followed by online fine-tuning, whereas DistRL distributes data collection across heterogeneous devices for centralized training [264], [265].Source: §5.7.5 · pp. 26–28

5.7.6 Summary & Takeaways

• Combining advanced techniques enables GUI agents to automate complex multi-step tasks while improving adaptability, collaboration, error recovery, and continual performance.The techniques are complementary; for example, self-reflection can be integrated into multi-agent frameworks to improve strategies and recover from errors.Source: §5.7.6 · p. 28

• The roadmap progresses from architecture, design, environment interaction, and memory to frameworks, data and model optimization, evaluation, and practical applications.These stages span Sections 5–10 and target robust development, validation, and real-world deployment.Source: §5.8 · p. 28
• Together, the roadmap defines the pipeline for advancing an LLM-powered GUI agent from foundational concepts through implementation and real-world innovation.Source: §5.8 · p. 28
• Figure 19 presents a taxonomy of recent GUI-agent work across frameworks, data, models, evaluation, and applications, including overlaps between categories.The taxonomy is intended as a blueprint for navigating research and development across the field.Source: §5.8 · p. 28

6 LLM-BRAINED GUI AGENT FRAMEWORK

• Across the surveyed frameworks, the paper examines foundational designs, technical advances, specialized capabilities, and the challenges addressed in GUI automation.The frameworks span diverse platforms and applications, reflecting a rapidly evolving effort to enable agents to interpret requests, analyze GUI components, and autonomously perform actions.Source: §6 · p. 28

6.1 Web GUI Agents

• Web GUI frameworks combine multimodal grounding, code generation, search, predictive modeling, and hybrid API–browsing strategies to automate complex, dynamic websites.SeeAct [17] achieves a 51.1% success rate on real-time web tasks, WebAgent [267] reports a 50% improvement over traditional single-agent models on real websites, and Search-Agent [274] achieves up to a 39% increase across benchmarks such as WebArena.Source: §6.1 · pp. 28–31

6.2 Mobile GUI Agents

• Mobile GUI agents employ multi-agent systems, script-based automation, hierarchical app memory, and privacy-preserving federated learning to improve complex-task handling, efficiency, adaptability, and cost-effective performance.MobileGPT uses hierarchical memory for modular, reusable, adaptable tasks. FedMobileAgent [314] achieves performance comparable to human-annotated models at a fraction of the cost while preserving user privacy.Source: §6.2 · pp. 34, 36

6.3 Computer GUI Agents

• Computer GUI frameworks address cross-application automation through centralized multi-agent architectures that decompose global plans into application-specific subtasks.UFO [19] uses a HostAgent for global planning and AppAgents for execution, with safeguards and customizable actions; UFO2 [334] further integrates UIA metadata, visual grounding, GUI–API actions, and isolated execution on Windows.Source: §6.3 · p. 38
• More than 10 percentage points higher success and half as many LLM calls than Operator and other CUAs show UFO2’s effectiveness across 20+ Windows applications.UFO2 combines retrieval-augmented help documents, execution logs, and speculative multi-action planning, while remaining model-agnostic.Source: §6.3 · p. 39
• Computer GUI agents now support generalist, multi-application automation through visual interaction, self-reflection, cross-OS tool generation, and direct IDE control.Cradle [161], OS-Copilot [162], and Programming with Pixels [331] illustrate these capabilities, with PwP-Bench showing computer-use agents can match or surpass traditional tool-based approaches in some scenarios.Source: §6.3 · p. 40

6.4 Cross-Platform GUI Agents

• AutoGLM [336] improves cross-platform GUI automation by separating planning from grounding and using reinforcement learning for incremental adaptation and error recovery.Its Intermediate Interface Design supports web browsing and Android control, while its self-evolving online curriculum learns from real-world feedback.Source: §6.4 · p. 40
• 0.27 billion parameters yield 73% on Screenspot [25] and 58.3% on OmniAct [459] for TinyClick, which outperforms GPT-4V while targeting efficient single-turn GUI interaction.Other cross-platform systems use state-machine control, dynamic agent enrollment, multimodal collaboration, pure vision, or modular grounding and planning across desktop, mobile, web, and 3D environments.Source: §6.4 · pp. 40, 42

6.5 Takeaways

• Multi-agent synergy, multimodal inputs, expanded action sets, and world-model or search-based decision-making improve efficiency, adaptability, environmental understanding, and long-horizon task handling.The takeaways cite UFO [19], MMAC-Copilot [249], WebVoyager [269], SeeAct [17], Hybrid Agent [199], AutoWebGLM [270], WMA [266], and Search-Agent [274].Source: §6.5 · p. 42
• Cross-platform generalization and pure vision-based control are central directions, but screenshot-only agents require strong grounding, which fine-tuning or GUI parsing such as OmniParser [184] can improve.AutoGLM [336], OSCAR [340], and AGUVIS [236] exemplify movement toward unified assistance across ecosystems while avoiding dependence on widget metadata or system APIs.Source: §6.5 · pp. 42–43

7.1.1 Data Composition and Sources

• GUI-agent training data combines natural-language instructions, environment perceptions such as screenshots and widget trees, and multi-step task trajectories that specify executable actions.Instructions and perceptions serve as inputs, while the trajectory is the expected output grounded in the environment.Source: §7.1.1 · p. 43
• Instructions can come from designers, existing datasets, public materials, or LLM generation, whereas perception data comes from emulators or real environments and trajectories from scripts, annotators, or agent bootstrapping.Trajectory collection is especially challenging because sequences must be accurate, executable, and validated.Source: §7.1.1 · p. 43

7.1.2 Collection Pipeline

• The collection pipeline instantiates generic instructions in specific environments, filters and augments them, records perceptions with trajectories, and evaluates sequences to remove failures.Instantiation may be manual or LLM-based, while filtering, augmentation, and validation can use humans or LLMs.Source: §7.1.2 · p. 44

7.2 Web Agent Data

• MultiUI [220] provides 7.3 million multimodal samples from 1 million websites, improving VisualWebBench by 48% and Mind2Web element accuracy by 19.1%, while InSTA [351] scales to 150k websites and outperforms human-collected datasets in zero-shot and low-resource settings.These datasets support text-rich visual understanding, web navigation, scalable curation, and generalization across real-world interfaces.Source: §7.2 · p. 45

7.3 Mobile Agent Data

• Mobile datasets span foundational UI understanding and natural-language grounding, from Rico [355] and PIXELHELP to AITW, META-GUI [357], and MobileViews [364].MobileViews [364] contains over 600,000 screenshot–view hierarchy pairs from 20,000 Android apps, while mobile datasets collectively form the richest platform-specific collection.Source: §7.3 · p. 46

7.4 Computer Agent Data

• Desktop GUI datasets remain fewer than mobile and web resources, but ScreenAgent [366] and LAM [367] support multi-step planning, action, reflection, and GUI execution.LAM [367] includes 76,672 task-plan pairs and 2,688 task-action trajectories for Windows automation, validated through LLM instantiation, GUI testing, and manual review.Source: §7.4 · p. 46

7.5 Cross-Platform Agent Data

• ScreenAI [375], VisualAgentBench [374], GUI-World [371], and xLAM [372] support adaptable GUI agents across mobile, web, desktop, and XR environments through annotation, interaction evaluation, videos, and unified formats.ScreenAI offers hundreds of millions of annotated samples, while GUI-World spans desktop, mobile, and XR environments with over 12,000 annotated videos.Source: §7.5 · pp. 46–47

7.6 Takeaways

• The section identifies scale and diversity, cross-platform flexibility, automated collection, and unified formats as central requirements for robust, generalizable GUI agents.Automated annotation and standardized action spaces can reduce manual effort and improve compatibility across varied platforms, representations, and environments.Source: §7.6 · pp. 48–50

8 MODELS FOR OPTIMIZING LLM-BRAINED GUI AGENTS

• Foundation models provide GUI agents with intent understanding, screen comprehension, and environment-interacting actions, while fine-tuning transforms them into specialized LAMs for GUI demands.The progression targets improved adaptability, accuracy, and action orientation through GUI-specific datasets.Source: §8 · pp. 50–51

8.1 Foundation Models

• Foundation models underpin GUI agents, with recent close-source and open-source MLLMs improving efficiency, scalability, and multimodal reasoning.Table 25 overviews key models and characteristics.Source: §8.1 · p. 51

8.1.1 Close-Source Models

• Proprietary models provide powerful, directly usable GUI-agent capabilities; GPT-4V and GPT-4o are commonly used for multimodal reasoning, efficient processing, and safety, although GPT-4V deployment is limited by safety and ethical considerations.Claude 3.5 Sonnet uses real-time screenshots for vision-only desktop automation, without metadata or underlying GUI structure.Source: §8.1.1 · p. 51

8.1.2 Open-Source Models

• Open-source models provide flexibility for customization and optimization, enabling GUI agents to use contextual data and deploy on resource-limited devices while supporting diverse multimodal tasks and on-device applications.Supported examples include Qwen2-VL [231], InternVL-2, [380], and CogVLM [381], which are described as adaptable or valuable for GUI-agent development.Source: §8.1.2 · pp. 51–52

8.2 Large Action Models

• LAMs fine-tune foundation LLMs on GUI-specific datasets to improve action orientation, long-horizon planning, visual grounding, and efficient execution, often using compact models.Datasets can combine instructions, widget trees, UI properties, action sequences, and annotated screenshots; LAMs convert abstract plans into direct GUI actions.Source: §8.2 · p. 53

8.3 LAMs for Web GUI Agents

• Web GUI LAMs combine multimodal grounding, search, reinforcement learning, feedback, and adaptive curricula to improve navigation, multi-step reasoning, efficiency, and cross-domain generalization.WebGUM [153] outperforms prior models on MiniWoB++ [146] and WebShop [425], while GLAINTEL uses 780M parameters to achieve results comparable to GPT-4-based methods at lower computational cost; WebRL [388] outperforms proprietary APIs and other open-source models in WebArena.Source: §8.3 · pp. 53–54

8.4 LAMs for Mobile GUI Agents

• Mobile GUI LAMs address diverse screen sizes, touch interactions, and resource constraints through specialized pretraining, reinforcement learning, offline reinforcement learning, and lightweight architectures for navigation and UI automation.MobileVLM leverages Mobile3M, a corpus of 3 million UI pages and interaction traces, in a two-stage mobile-specific pretraining framework. UI-R1-3B [401] uses only 136 high-quality samples.Source: §8.4 · pp. 54–56

8.5 LAMs for Computer GUI Agents

• Computer GUI agents must support complex applications, multitasking, and varied interaction modalities, motivating specialized LAMs for more sophisticated desktop and laptop task execution.The section introduces an overview of computer GUI models in Table 29.Source: §8.5 · p. 58, Table 29 · p. 58
• ScreenAgent [366] integrates planning, acting, and reflection with screenshot inputs and JSON mouse-keyboard actions to support precise localization and continuous multi-step GUI tasks.It is fine-tuned on the ScreenAgent Dataset and based on CogAgent [15].Source: §8.5 · p. 58

8.6 Cross-Platform Large Action Models

• Cross-platform LAMs unify GUI understanding and action across web, desktop, and mobile environments, with UI-TARS outperforming GPT-4o and Claude on task-execution benchmarks.UI-TARS uses screenshot-only perception, a unified action space, System-2 reasoning, and reflection-based self-improvement across Web, Windows, macOS, and Android.Source: §8.6 · pp. 58, 60

8.7 Takeaways

• The section’s takeaways emphasize smaller on-device models, visual grounding, reinforcement learning, unified function calling, and inference-time reasoning for efficient and adaptable GUI agents.These approaches target privacy, latency, reduced reliance on structured metadata, dynamic error recovery, interoperability, and more robust long-horizon decisions.Source: §8.7 · p. 61

9 EVALUATION FOR LLM-BRAINED GUI AGENTS

• The evaluation pipeline uses action sequences, screenshots, UI structures, and environment states to assess agents across web, mobile, and desktop platforms with distinct interaction challenges.Web tasks involve dynamic content, mobile tasks involve constrained touch interfaces, and desktop tasks involve complex workflows, cross-application operation, and multiple operating systems.Source: §9 · p. 61, §9.3 · pp. 63–64

9.1 Evaluation Metrics

• GUI agents are evaluated with step, turn, and task success rates; efficiency costs; policy compliance; and risk ratios, covering accuracy, interaction quality, resource use, safety, and trustworthiness.Efficiency may measure time, memory/CPU/GPU, LLM calls, or total steps, while completion under policy assesses adherence to security, ethical, safety, privacy, or business rules.Source: §9.1 · pp. 61–62
• Benchmarks and metric implementations vary by platform and task formulation, so the survey maps differing original metric names into common evaluation categories for comparison.BrowserGym unifies several web benchmarks through standardized observation and action spaces, while WebOlympus supports live-website evaluation with safety monitoring.Source: §9.1 · p. 62, §9.4 · p. 65

9.2 Evaluation Measurements

• Evaluation measurements compare textual, visual, element, and action outputs with expected results while recording application state, logs, and environment variables for debugging and reliability analysis.Text matching can be exact, fuzzy, or semantic; image, element, and action matching assess visual outcomes, UI structures, interaction correctness, sequence alignment, and spatial targeting.Source: §9.2 · pp. 62–63

9.4 Web Agent Benchmarks

• Web benchmarks span realistic, multimodal, conversational, video, distraction-resistant, safety-oriented, and enterprise tasks, with WorkArena revealing a substantial gap between state-of-the-art agents and humans.WorkArena contains 19,912 instances across 33 tasks, while other benchmarks target visual grounding, video comprehension, policy compliance, or environmental distractions.Source: §9.4 · pp. 64–65

9.5 Mobile Agent Benchmarks

• Mobile GUI-agent benchmarks address task complexity, safety, efficiency, and scalability, while Android evaluation can use final UI state matching, event tracking, and metrics such as success rate, efficiency, latency, and token cost.MobileSafetyBench [443] evaluates helpfulness and safety in common Android tasks, including legal compliance, privacy, and indirect prompt injection. GTArena [447] evaluates test intention generation, task execution, and defect detection using GUI defects.Source: §9.5 · pp. 66–67

9.6 Computer Agent Benchmarks

• Desktop benchmarks evaluate multimodal agents in realistic computer workflows: OSWorld [419] supports arbitrary applications across Ubuntu, Windows, and macOS, with 369 Ubuntu tasks, 43 Windows tasks, and success-rate evaluation.WindowsArena tests planning, screen understanding, and tool usage in a real Windows environment, while OFFICEBENCH [455] evaluates cross-application automation across Word, Excel, and Email in Linux Docker.Source: §9.6 · pp. 68–69

9.7 Cross-Platform Agent Benchmarks

• Cross-platform benchmarks test GUI-agent versatility and robustness across web, mobile, desktop, and other interactive environments through visual grounding, multi-device execution, and high-level decision-making.VisualAgentBench [374] spans web, Android, and games; CRAB [461] evaluates tasks across Ubuntu Linux and Android with graph-based subgoals and multiple valid paths; ScreenSpot [25] measures visual click accuracy and localization across more than 600 screenshots and 1,200 instructions.Source: §9.7 · p. 72

9.8 Takeaways

• Benchmarking is shifting toward interactive, realistic, cross-platform, human-like evaluations while facing unresolved challenges in interface standardization, scalability, automation, safety, privacy, and compliance.Future directions include live environments, multi-turn interactions, unified access to platform-specific information, automated task generation and evaluation, and stronger legal, ethical, and data-security assessments.Source: §9.8 · pp. 72–73

10.1 GUI Testing

• LLM-powered GUI testing replaces traditional scripting with natural-language interactions for general testing, input generation, and bug reproduction.Agents can execute UI operations from natural-language descriptions and use screen understanding to verify outcomes, reducing human or script-based testing effort.Source: §10.1 · p. 73

10.1.1 General Testing

• General-testing agents improve coverage through conversational interaction, memory, intent-driven planning, multimodal analysis, and vision-based alignment,, [465], [466].One memory-based approach increased activity coverage by 32%, while DROIDAGENT pursued feature coverage through high-level intent-driven tasks and VisionDroid detected non-crash logical inconsistencies.Source: §10.1.1 · p. 74
• ProphetAgent uses a semantically annotated Clustered UI Transition Graph and dual-agent architecture to achieve 78.1% completion rate, 83.3% action accuracy, and over 70% tester-workload reduction.SemanticAgent performs annotation while GenerationAgent produces executable scripts, improving the mapping from natural-language test steps to GUI operations.Source: §10.1.1 · p. 74

10.1.2 Text Input generation

• LLMs generate context-aware Android text inputs and improve activity and page coverage by up to 52% through the QTypist fill-in-the-blank formulation [468],.These generated inputs were evaluated across multiple apps to improve coverage and detect unique bugs with minimal manual intervention.Source: §10.1.2 · p. 75

10.1.3 Bug Replay

• Bug-replay agents combine reinforcement learning, few-shot learning, chain-of-thought reasoning, visual understanding, and external knowledge to automate crash and bug reproduction,, [478].BugCraft translates unstructured Minecraft bug reports into executable in-game behaviors using GPT-4o, OmniParser [184], Minecraft Wiki knowledge, and a custom Macro API.Source: §10.1.3 · pp. 75–76

10.1.4 Verification

• MagicWand automates “How-to” verification by extracting, executing, and refining search-engine instructions, while accessibility agents support natural-language testing of VoiceOver and Dynamic Type [472],.AXNav generates annotated videos for interactive review, and the broader testing applications improve coverage, efficiency, accessibility, and automated crash reproduction.Source: §10.1.1 · p. 75, §10.1.4 · p. 76

10.2 Virtual Assistants

• Virtual assistants have evolved from basic utilities into intelligent GUI agents supporting complex, context-aware interactions through textual or voice commands, personalized responses, and intricate workflows across platforms.Applications include mobile task automation, business-process workflow automation, and accessibility support for blind and low-vision users through conversational chart exploration.Source: §10.2 · p. 76, §10.2.1 · p. 77

10.2.2 Open-Source Projects / 10.2.3 Production

• Open-source projects democratize customizable GUI assistants, while OpenAdapt, AgentSea, Open Interpreter, Power Automate, and MultiOn automate desktop and web workflows through demonstrations, modular tools, local code execution, or natural-language commands,,,.OpenAdapt learns from user demonstrations; AgentSea provides a modular toolkit; Open Interpreter executes code locally; Power Automate records actions from natural-language descriptions, and MultiOn interprets commands to perform web actions.Source: §10.2.2 · p. 79, §10.2.3 · pp. 79–80

10.2.3 Production

• In production, LLM-brained GUI agents enhance automation, productivity, and user engagement across platforms and industries.Source: §10.2.3 · p. 80

10.3 Takeaways

• The agents improve accessibility and user experience for people with disabilities or limited technical proficiency, while remaining early-stage and requiring continued research.VizAbility helps blind and low-vision users understand data visualizations, and EasyAsk assists older adults with smartphone navigation.Source: §10.3 · pp. 80–81
• Natural-language interfaces lower barriers for non-experts by automating GUI testing and enabling virtual assistants to execute complex commands.GPTDroid and AUITestAgent [465] support plain-language test specification, while LLMPA and ProAgent [485] interpret user commands across platforms.Source: §10.3 · p. 80
• Agents automate intricate multi-step workflows, improve testing coverage and efficiency, and reproduce bugs from textual descriptions without manual scripting.AutoTask [488] and GPTVoiceTasker [487] explore GUI environments from high-level goals or voice commands, while CrashTranslator and AdbGPT automate test-input generation and bug reproduction.Source: §10.3 · pp. 80–81
• Multimodal perception combines visual and textual inputs, screenshots, UI trees, and OCR to improve GUI-context understanding, decision-making, and interaction accuracy.VizAbility and OpenAdapt support precise GUI-element identification and manipulation in dynamic or visually complex interfaces.Source: §10.3 · p. 81

11.1 Privacy Concerns

• Privacy risks arise because GUI agents process screenshots, histories, credentials, and confidential documents, often through remote servers or third-party APIs, threatening adoption [546]–.Proposed safeguards include on-device inference, compression, redaction, secure communication, scoped data use, privacy controls, consent, federated learning, differential privacy, and homomorphic encryption [551]–.Source: §11.1 · p. 81

11.2 Latency, Performance, and Resource Constraints

• GUI agents face cumulative latency and resource constraints because interactive perception, planning, and control require repeated computation, especially on mobile and other limited devices [561]–[564].Model distillation, accelerators, parallelism, caching, compression, edge or distributed inference, and native high-level APIs can reduce latency and execution steps [565]–[570], [198], [199].Source: §11.2 · pp. 81–82

11.3 Safety and Reliability

• Direct actuation creates safety and reliability risks, including irreversible errors, hallucinated actions, inconsistent behavior, attacks, weak interpretability, and disruptions across applications or systems [548], [571]–.Validation, formal verification, exception handling, rollback, permission management, standardized protocols, and secure authentication or authorization are proposed safeguards [582]–.Source: §11.3 · p. 82

11.4 Human-Agent Interaction

• Human-agent collaboration is challenged by conflicting interventions, ambiguous instructions, unexpected interface states, and the need for clarification, permission, and confirmation [19], –[598].The email example requires secure login, recipient disambiguation, and confirmation before an irreversible action, motivating shared control, interruption handling, and proactive clarification [19],.Source: §11.4 · p. 82

11.5 Customization and Personalization

• GUI agents must personalize workflows and preferences across sessions, but user variability and the privacy of user-specific data make effective customization difficult [45], [605]–[607].User modeling, preference learning, feedback, context-aware methods, on-device learning, and anonymization are proposed ways to personalize without compromising privacy [608]–.Source: §11.5 · p. 83

11.6 Ethical and Regulatory Challenges

• GUI agents amplify ethical and regulatory risks involving accountability, fairness, consent, bias, traceability, data protection, and compliance during opaque multi-application actions [548], –.The roadmap calls for permission controls, runtime confirmations, activity logs, ethical guidelines, auditing, monitoring, bias mitigation, and transparent data-use controls –[624].Source: §11.6 · pp. 83–84

11.7 Scalability and Generalization

• GUI agents generalize poorly across interfaces and evolving layouts, so software changes can break behavior, increase maintenance, and require repeated adaptation or retraining [625]–[629].Diverse datasets, transfer learning, meta-learning, documentation, demonstrations, and runtime retrieval can improve adaptation to unfamiliar environments with less retraining [630]–[636].Source: §11.7 · pp. 84–85

11.8 Summary

• Overcoming these limitations requires sustained collaboration among researchers, industry, policymakers, and users, alongside continuous evaluation and adaptation to develop robust, safe, and user-friendly agents.Interdisciplinary collaboration is presented as essential for aligning technical progress with user needs and the ethical and societal implications of deployment.Source: §11.8 · p. 85

• LLMs act as the “brain” for language processing and GUI understanding, while automation tools act as the “hands” executing commands, enabling natural-language control of applications.This combination has generated applications across both research and industry.Source: §12 · p. 85
• The survey provides a comprehensive, systematic, and timely overview of LLM-powered GUI agents.Source: §12 · p. 85
• It covers agent components, advanced techniques, data collection, model development, frameworks, evaluation methodologies, real-world applications, limitations, and challenges.Source: §12 · p. 85
• The survey outlines future research directions and positions itself as a handbook for learners and a reference for researchers tracking developments in the field.Source: §12 · p. 85
• With continued research and development, LLM-brained GUI agents could enhance productivity and accessibility by transforming complex workflows into seamless, natural interactions.Source: §12 · p. 85