GUI Agents: A Survey

• GUI agents use Large Foundation Models to interact with digital systems and applications through visual interface elements, using actions such as mouse control and typing.The approach is relevant across desktops, mobile phones, and web browsers.Source: §1 · p. 1
• Dynamic layouts, diverse platform designs, and fine-grained recognition of small, scattered elements pose challenges for GUI interaction.Source: §1 · p. 1
• The survey proposes a unified, systematic framework to categorize contributions in GUI agent research.Source: §1 · p. 1
• The survey covers datasets and environments, architectural designs, training methods, and open problems and future prospects.Source: §1 · pp. 1–2

• A GUI agent is an autonomous agent that observes interactable screen elements and engages with them through actions such as clicking, typing, or tapping.Source: §2 · p. 2
• GUI agent tasks are modeled as sequential environment interactions, generally formalized as a POMDP with task, action, state, observation, and transition components.The state is not fully observable, and the transition function maps a state-action pair to a distribution over subsequent states.Source: §2 · p. 2
• A GUI agent is formalized as a policy mapping a probability distribution over states to an action, commonly implemented using the history of past actions and observations.At each step, the policy selects an action; the environment transitions, and may provide a reward.Source: §2 · p. 2

• A dataset is static, whereas an environment is interactive and dynamic, with actions able to change its state.Environments can be modeled as MDPs or POMDPs with action, state, observation, and transition components.Source: §3 · p. 2
• Closed-world benchmarks contain the knowledge needed for a task, while open-world benchmarks allow required information to exist outside the benchmark.Source: §3 · p. 2
• The survey presents a summary of existing GUI agent benchmarks in Table 1.Source: §3 · p. 2

3.1.1 Closed-World Datasets.

• Closed-world datasets cover varied web tasks, including instruction following, multimodal layouts, visual grounding, clicking and typing, and business-process workflows.Examples include RUSS, Mind2Web, TURKINGBENCH, VisualWebBench, ScreenSpot, and WONDERBREAD.Source: §3.1.1 · pp. 2–3
• Static closed-world datasets are lightweight to set up and support fine-grained, reproducible comparisons under identical conditions, but use fixed data rather than changing scenarios.Source: §3.1.1 · p. 3

3.1.2 Open-World Datasets.

• Open-world datasets include tasks requiring external web browsing or APIs and interactive navigation on real-world sites.Examples include GAIA and WebLINX.Source: §3.1.2 · p. 3
• Static open-world datasets balance evaluation ease with realism, but changing real-world websites make reproduction and comparison with prior methods more challenging.Source: §3.1.2 · p. 3

3.2.1 Closed-World Environments.

• Closed-world environments offer stable, reproducible evaluation while mimicking real-world dynamics, but often demand substantial storage and engineering effort to set up.Examples range from synthetic web tasks and e-shopping to enterprise workflows and realistic desktop tasks.Source: §3.2.1 · p. 3

3.2.2 Open-World Environments.

• Open-world environments test agents on dynamic, evolving websites, but frequent live-site changes make consistent evaluation and reproducibility difficult.Benchmarks include multimodal, multihop tasks and live-web evaluation frameworks.Source: §3.2.2 · pp. 3–4

3.3.1 Task Completion Metrics.

• Task completion is the primary benchmark metric, but success may mean reaching a goal state, matching ground-truth output, or completing required subtasks.Some evaluations also measure partial success, trajectory-based judgments, or aggregate turn-level success; task-specific metrics include ROUGE-L, F1, accuracy, precision, and recall.Source: §3.3.1 · p. 4

3.3.2 Intermediate Step Metrics.

• Step-wise metrics assess specific behaviors such as webpage navigation, element selection, intent matching, and correct actions, but reference labels are difficult to collect while allowing alternate task paths.WebCanvas, WebLINX, Mind2Web, and MT-Mind2Web use different intermediate-step measures.Source: §3.3.2 · p. 4

3.3.3 Efficiency, Generalization, Safety and Robustness Metrics.

• Beyond performance, benchmarks measure efficiency, generalization across diverse or compositional tasks, safety-policy adherence, and robustness to environmental distractions.The survey lists these as additional dimensions of GUI-agent evaluation.Source: §3.3.3 · pp. 4–5

4 GUI Agent Architectures

• The architecture taxonomy groups GUI-agent designs into perception, reasoning, planning, and acting modules.These respectively interpret observations, support cognition, decompose tasks into plans, and enable interaction through action representations such as templates, JSON, or code.Source: §4 · p. 5

4.1 Perception

• Perception interfaces are accessibility-based, HTML/DOM-based, screen-visual-based, or hybrid, offering different capabilities and privacy and implementation considerations.Unlike API-based agents using structured data, GUI agents must interpret interfaces designed for human consumption.Source: §4.1 · p. 5

4.1.1 Accessibility-Based Interfaces

• Accessibility APIs expose semantic cues and actionable elements, but depend on developer implementation and may miss dynamic or visual content; DOM interfaces provide structure but require preprocessing to handle noisy markup.Accessibility interfaces typically require minimal sensitive user data, while DOM approaches may use ranking, task-specific snippets, or HTML simplification.Source: §4.1.1 · p. 5, §4.1.2 · p. 5

4.1.3 Screen-visual-based Interfaces

• Screen-visual perception works when structured interfaces are unavailable or visual content matters, while hybrid systems combine visual and textual sources and can fall back when one is incomplete or misleading.Visual approaches can raise privacy and computational concerns; hybrid interfaces are used by OS-Atlas and UGround (Wu et al., 2024b; Gou et al., 2024).Source: §4.1.3 · pp. 5–6, §4.1.4 · p. 6

4.2 Reasoning

• The reasoning subsection surveys approaches including dual optimization, observation- and action-space refinement, code generation, and LLM-based reasoning, acting, and planning.Examples include WebPilot (Zhang et al., 2024d), WebOccam (Yang et al., 2024), OSCAR (Wang and Liu, 2024), and LAST (Zhou et al., 2023a).Source: §4.2 · p. 6

4.3 Planning

• Planning decomposes global tasks into subtasks; the survey distinguishes plans based on internal knowledge from those drawing on external knowledge and resources.It frames LLM-powered agents as an alternative to symbolic planning and reinforcement learning, which have stated expertise, error-tolerance, and training-data constraints.Source: §4.3 · p. 6

4.3.1 Planning with Internal Knowledge

• Internal-knowledge methods use simulated action outcomes, hierarchical management, or experience-augmented plans, while external-knowledge methods combine LLMs with search, retrieval, analogous problems, or world models.Examples include WebDreamer, MobA, and Agent S for internal planning, and Search-Agent, AgentQ, Toolchain, SGC, Thought Propagation Retrieval, and WMA for external planning.Source: §4.3.1 · p. 6, §4.3.2 · pp. 6–7

4.4 Acting

• GUI acting converts plans into executable actions, from semantic operations such as typing and scrolling to pixel-level coordinates; visual grounding can avoid noisy or costly metadata.OS-Atlas uses large-scale multi-platform training data for grounding across web, mobile, and desktop settings; WebAgent and Mind2Web use DOM or HTML representations, while AppAgent and MobileAgent use accessibility APIs.Source: §4.4 · p. 7

5 GUI Agent Training Methods

• GUI-agent training methods divide into prompt-based approaches and parameter-optimizing approaches such as pretraining, fine-tuning, and reinforcement learning.The survey presents a taxonomy of these methods in Figure 2.Source: §5 · p. 7

5.1 Prompt-based Methods

• Prompt-based methods adapt behavior through instructions and interaction context without changing model parameters, but are constrained by context size and the difficulty of designing effective prompts.Methods include self-reflection, intent discovery, state-space exploration, state machines, expert collaboration, and app memory.Source: §5.1 · p. 7

5.2.1 Pre-training

• Pretraining builds GUI-task foundations through screenshot parsing or continued training of vision-language models on large-scale and synthetic GUI data.Examples include PIX2STRUCT (Lee et al., 2023), SeeClick (Cheng et al., 2024a), and UGround (Gou et al., 2024); CogAgent and ShowUI also target computational efficiency.Source: §5.2.1 · pp. 7–8

5.2.2 Fine-tuning

• Fine-tuning adapts language and vision-language models to GUI interaction, with reported approaches targeting grounding, instruction reliability, hallucination reduction, and model size.UI-Pro reports reducing model size while maintaining state-of-the-art grounding accuracy; VGA reduces hallucinations through image-centric fine-tuning.Source: §5.2.2 · p. 8

5.2.3 Reinforcement Learning

• Reinforcement-learning methods address sparse rewards, real-world error recovery, and stochastic environments using techniques such as curriculum learning and modified advantage-weighted regression.WebRL generates tasks from unsuccessful attempts and uses an outcome reward model; DigiRL modifies AWR for offline learning in more stochastic environments.Source: §5.2.3 · pp. 8–9

User Intent Understanding

• 51.1% accuracy on unseen websites illustrates GUI agents’ difficulty inferring user goals across diverse applications.The survey points to generalization across varying tasks and contextual interactions as important research needs.Source: §6 · p. 9

Security and Privacy

• Sensitive data exposure, especially through cloud processing, raises privacy and security concerns; unauthorized access or incorrect actions could have severe consequences.Suggested directions include privacy-preserving protocols, local processing, and stronger authentication.Source: §6 · p. 9

Inference Latency

• Real-time responsiveness can conflict with complex GUI interactions, particularly in resource-constrained environments.The survey identifies lightweight architectures, hardware-aware optimization, and efficient decoding as potential research directions.Source: §6 · p. 9

Personalization

• Personalization aims to tailor interactions to individual preferences and behaviors, and future work could use feedback and richer user profiles.Berkovitch et al. (2024) introduced a method for identifying user goals from UI trajectories to infer intentions and proactively assist users.Source: §6 · p. 9

Conclusion

• The survey examines GUI-agent benchmarks, architectures, and training methods.It presents this synthesis as a resource offering researchers structure and practical guidance.Source: §7 · p. 9
• Intent understanding, security, latency, and personalization remain critical challenges despite considerable progress.Source: §7 · p. 9
• The survey states that GUI-agent progress has benefited people by enhancing daily productivity and transforming computer interactions.Source: §7 · p. 9

Limitations

• The survey excludes agent interactions through Command Line Interfaces and Application Programming Interfaces.Source: Limitations · p. 10
• CLI and API approaches are characterized as relatively limited in scope compared with GUI-based methods.Source: Limitations · p. 10
• The authors chose this exclusion to maintain a focused survey scope.Source: Limitations · p. 10