Source-linked AI summary
OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use
Xueyu Hu, Tao Xiong, Biao Yi, Zishu Wei, Ruixuan Xiao, Yurun Chen, Jiasheng Ye, Meiling Tao, Xiangxin Zhou, Ziyu Zhao, Yuhuai Li, Shengze Xu, Shenzhi Wang, Xinchen Xu, Shuofei Qiao, Zhaokai Wang, Kun Kuang, Tieyong Zeng, Liang Wang, Jiwei Li, Yuchen Eleanor Jiang, Wangchunshu Zhou, Guoyin Wang, Keting Yin, Zhou Zhao, Hongxia Yang, Fan Wu, Shengyu Zhang, Fei Wu
TL;DR
OS Agents promise autonomous computer and mobile-device task automation, but their rapidly growing literature requires consolidation. This survey organizes their fundamentals, construction, evaluation, and future directions, concluding that OS-specific defenses remain limited while safety, privacy, personalization, and self-evolution need continued research.
Problem
The rapidly advancing and growing body of OS Agent research lacks a consolidated account of its foundations, construction, evaluation, and open challenges.
Method
The survey synthesizes OS Agent components and capabilities, foundation models, agent frameworks, evaluation protocols, benchmarks, and future research directions.
Results
The survey provides an integrated view of OS Agent construction and evaluation across diverse tasks and identifies safety, privacy, personalization, and self-evolution as critical challenges.
Takeaways & Limitations
The survey offers insights for academic and industrial OS Agent development and maintains an open-source GitHub repository as a dynamic resource.
Takeaways & Limitations
Studies on defenses specific to OS Agents remain limited, including defenses against injection attacks, backdoor exploits, and other potential threats.
Abstract
from arXiv · showhide
The dream to create AI assistants as capable and versatile as the fictional J.A.R.V.I.S from Iron Man has long captivated imaginations. With the evolution of (multi-modal) large language models ((M)LLMs), this dream is closer to reality, as (M)LLM-based Agents using computing devices (e.g., computers and mobile phones) by operating within the environments and interfaces (e.g., Graphical User Interface (GUI)) provided by operating systems (OS) to automate tasks have significantly advanced. This paper presents a comprehensive survey of these advanced agents, designated as OS Agents. We begin by elucidating the fundamentals of OS Agents, exploring their key components including the environment, observation space, and action space, and outlining essential capabilities such as understanding, planning, and grounding. We then examine methodologies for constructing OS Agents, focusing on domain-specific foundation models and agent frameworks. A detailed review of evaluation protocols and benchmarks highlights how OS Agents are assessed across diverse tasks. Finally, we discuss current challenges and identify promising directions for future research, including safety and privacy, personalization and self-evolution. This survey aims to consolidate the state of OS Agents research, providing insights to guide both academic inquiry and industrial development. An open-source GitHub repository is maintained as a dynamic resource to foster further innovation in this field. We present a 9-page version of our work, accepted by ACL 2025, to provide a concise overview to the domain.
1 Introduction
OS Agents use operating-system environments and interfaces to automate tasks, bringing the goal of versatile AI assistants closer through advances in (M)LLMs. This survey consolidates their foundations, construction methods, evaluation, challenges, and future directions.
- OS Agents operate computers and mobile phones through operating-system environments and interfaces such as GUIs to automate user-defined tasks.
- Recent (M)LLM advances enable OS Agents to better understand complex tasks and execute actions on computing devices.
- The survey covers OS Agent fundamentals, domain-specific foundation models, agent frameworks, evaluation protocols, benchmarks, and future challenges.
- An open-source GitHub repository is maintained as a dynamic resource supporting continued development of OS Agents.
2 Fundamental of OS Agents
OS Agents engage with desktop, mobile, and web environments through observation and action spaces. Their core capabilities include understanding complex interfaces, planning adaptive action sequences, and grounding plans into executable operations.
- Environment: OS Agent environments include desktop, mobile, and web platforms supporting tasks from information retrieval to complex multi-step operations.
- Observation Space: The observation space contains system-state and user-activity information, including screen images and textual data, that guides decisions.
- Action Space: The action space comprises input operations, navigation operations, and extended operations using external tools or services.
- Understanding: Understanding requires interpreting complex HTML and GUI data, including high-resolution interfaces with small icons, text, and dense elements.
- Planning: Planning decomposes complex tasks into sub-tasks and action sequences, then dynamically adjusts them using environmental feedback and historical actions.
- Grounding: Action grounding translates textual instructions or plans into executable operations by identifying screen elements and supplying parameters such as coordinates or input values.
3 Construction of OS Agents
Constructing OS Agents involves developing domain-specific foundation models and building agent frameworks around them. These approaches incorporate specialized knowledge and capabilities for operating-system interaction.
- Domain-specific foundation models can enhance OS Agents by incorporating specialized knowledge and capabilities for interacting with operating systems.
- Construction also requires effective agent frameworks around foundation models to support OS Agent operation.
- Architecture design and targeted training strategies should align model capabilities with OS-specific tasks.
3.1 Foundation Model
Foundation models for OS Agents combine architecture choices with pre-training, supervised fine-tuning, and reinforcement learning. The surveyed approaches adapt language and multimodal models to process GUI information, ground actions, and plan multi-step tasks.
- Foundation-model construction centers on model architecture and training strategies, including pre-training, supervised fine-tuning, and reinforcement learning.
- Architecture: Architectures range from existing LLMs to concatenated or modified MLLMs that combine language and visual processing for OS tasks.
- Architecture: Modified MLLMs address GUI screenshot resolution limits because resizing can obscure text and small icons needed for OS tasks.
- Pre-training: Pre-training expands GUI understanding and visual-textual correlations, using continued pre-training from general models and data such as crawled web content.
- Pre-training: Pre-training supports screen-grounding tasks that extract target-element coordinates or bounding boxes from images using textual descriptions.
- Supervised Fine-tuning: Supervised fine-tuning enhances planning and grounding, but requires domain-specific GUI data to bridge the gap from natural-image tasks.
3.2 Agent Framework
OS Agent frameworks coordinate perception, planning, memory, and action to operate computing devices. The survey reviews textual and visual perception, GUI grounding, adaptive planning, and memory mechanisms for dynamic tasks.
- Framework components: OS Agent frameworks combine Perception, Planning, Memory, and Action to understand environments, generate plans, retain information, and execute operations.These components jointly support interaction with operating systems.
- Perception: Text-based perception converts OS states into structured descriptions such as HTML, DOM, or accessibility trees, but redundant information can cause incorrect actions.MobileGPT, for example, converts mobile screens into simplified HTML representations.
- Perception: GUI screenshot perception better aligns with human interaction, while GUI grounding identifies relevant buttons, menus, and text fields for multimodal agents.Grounding approaches include visual, semantic, and dual grounding, combining interface locations with semantic descriptions.
- Planning: Planning decomposes goals into action sequences, with global planning producing one fixed plan and iterative planning updating plans from actions or environmental changes.Iterative planning is designed for dynamic and unpredictable environments.
- Memory: Memory supports task execution through internal, external, and specific sources that store immediate information, external knowledge, and task-focused experience.Examples include action histories, screenshots, knowledge bases, APIs, and task-specific GUI information.
4 Evaluation of OS Agents
OS Agent evaluation combines protocols, metrics, benchmarks, and platform-specific tasks to assess capabilities and practical use. The survey distinguishes objective from subjective evaluation and step-level from task-level assessment.
- Evaluation framework: Evaluation frameworks assess OS Agents through principles and metrics tailored to different environments, applications, and benchmark settings.The survey organizes benchmarks by platforms, settings, and tasks.
- Evaluation principles: Objective evaluation uses standardized numerical measures, while subjective evaluation measures alignment with human expectations in difficult-to-quantify scenarios.Together, these approaches address controlled performance, reliability, and practical usability.
- Evaluation metrics: Step-level evaluation compares each action with reference actions or target elements using metrics such as operation accuracy, F1, element accuracy, and element F1.It provides fine-grained analysis of planning trajectories and action grounding.
- Evaluation metrics: Task-level evaluation measures whether agents reach desired final states through task completion and resource utilization.Metrics include success rate, accuracy, reward, step ratio, hardware expense, and time expenditure.
- Benchmarks: Benchmarks span mobile, desktop, and web platforms, each imposing distinct interaction and scalability challenges.Mobile tasks require precise gestures, desktops involve diverse systems and applications, and web environments support interactive browser-based evaluation.
5 Challenge & Future
The survey identifies security, personalization, and self-evolution as major challenges for OS Agents. It highlights attacks against agents, limited OS-specific defenses, and unresolved multimodal memory-management problems.
- Security: OS Agents face security risks from adversarial web instructions, images, pop-ups, and harmful browser interactions.Reported threats include Web Indirect Prompt Injection and attacks that interfere with agent decision-making.
- Security: Defenses specific to OS Agents remain limited, motivating robust and scalable protection against injection attacks, backdoors, and related threats.The survey identifies OS-specific defense development as a priority.
- Security: Security benchmarks such as ST-WebAgentBench evaluate web-agent safety and trustworthiness across reliability dimensions in enterprise environments.These benchmarks target high-risk contexts.
- Personalization and self-evolution: Personalized OS Agents are expected to adapt to user preferences, but many current multimodal models remain insufficiently personalized and self-evolving.Memory mechanisms can accumulate user data over time to improve experience and performance.
- Personalization and self-evolution: Expanding memory beyond text to images and voice, while managing and retrieving it effectively, remains challenging.The survey connects progress on these issues with more personalized, dynamic, and context-aware assistance.
6 Related Work
The survey situates OS Agents among broader work on LLMs, multimodal models, agent frameworks, personalized agents, GUI agents, and generalist virtual agents. It focuses specifically on their construction and evaluation through operating-system environments and interfaces.
- Prior surveys: Prior surveys cover foundational LLMs, multimodal LLMs, synthetic training data, general LLM-based agents, and agent-framework capabilities.These works provide background on models and general planning or memory mechanisms.
- Scope of this survey: Related work on personalized agents, GUI agents, and generalist virtual agents shares features with OS Agents but does not provide the same integrated focus.The survey unifies construction, evaluation, and open challenges for agents using operating-system environments and interfaces.
7 Conclusion
The survey synthesizes OS Agent fundamentals, construction methods, and evaluation, then identifies safety, privacy, personalization, and self-evolution as continuing research challenges.
- The survey reviews OS Agent fundamentals, including key components and capabilities.
- It examines construction approaches centered on domain-specific foundation models and agent frameworks.
- It discusses evaluation protocols and benchmarks for assessing OS Agents across diverse tasks.
- The survey identifies safety, privacy, personalization, and self-evolution as critical areas for continued research.
- Its synthesis is intended to support ongoing OS Agent development in academic and industrial settings.