UFO2: The Desktop AgentOS

• Existing CUAs rely heavily on screenshots and simulated input while underusing accessibility interfaces, application APIs, and process state, causing inefficiency, limited reliability, and disruptive desktop control.Source: §1 · p. 2
• UFO2 uses a centralized HostAgent to decompose instructions and dispatch subtasks to application-specialized AppAgents, supporting coordinated workflows across applications such as Excel and Outlook.Source: §1 · p. 2
• Its unified GUI–API layer combines traditional interaction with native APIs, while hybrid UIA–vision detection supports both standard and custom interface components.Source: §1 · p. 2
• Speculative multi-action execution predicts and validates action sequences with lightweight control-state checks, reducing LLM inference overhead without compromising correctness.Source: §1 · p. 2
• PiP provides an isolated nested desktop in which agents execute alongside users without disrupting the main desktop session.Source: §1 · p. 2
• UFO2 frames desktop workflows as programmable, composable, and robust OS-level entities rather than GUI scripts.Source: §1 · p. 2
• UFO2 evaluates across 20+ real-world Windows applications and shows consistent improvements in success rate, execution efficiency, and usability over state-of-the-art CUAs such as Operator.Source: §1 · p. 3
• Even with GPT-4o, UFO2 outperforms dedicated CUAs by over 10%, highlighting the reported benefit of system-level integration and architectural design.Source: §1 · p. 3

2.1 The Fragility of Traditional Desktop Automation

• Traditional RPA scripts break under minor interface changes and remain limited to narrow, repetitive workflows in stable environments.Source: §2.1 · p. 3

2.2 Rise of Computer-Using Agents

• CUAs replace fixed scripts with multimodal LLM-based interpretation and GUI action generation, but existing systems largely remain research prototypes.Source: §2.2 · p. 3

2.3 Systems Challenges in CUAs

• Most CUAs lack OS-level integration, forcing actions to be inferred from pixels instead of structured accessibility, process, or inter-process state.Source: §2.3 · p. 3
• Generalist CUAs lack application-specific introspection and cannot reliably leverage built-in APIs or vendor documentation, limiting generalization and increasing maintenance costs.Source: §2.3 · p. 3
• Direct automation on the user’s desktop prevents interaction during execution, introduces interference risk, and can monopolize the session for minutes.Source: §2.3 · p. 3

2.4 Missing Abstraction: OS Support for Automation

• Desktop operating systems provide no first-class abstraction for GUI application control, leaving RPA and CUAs as ad-hoc layers without unified execution, coordination, or introspection.Source: §2.4 · p. 3
• UFO2 addresses this gap by embedding automation as a Windows OS abstraction that exposes GUI controls, application APIs, and task orchestration as programmable system services.Source: §2.4 · p. 4

3.1 UFO2 as a System Substrate for Automation

• HostAgent coordinates AppAgents through a centralized, pluggable runtime that decomposes requests, manages application lifecycles, and supports cross-application workflows in an isolated PiP desktop.A global blackboard exchanges intermediate results and execution metadata, while each AppAgent remains an isolated application-specific execution context.Source: §3.1 · p. 4

3.2 HostAgent: System-Level Orchestration and Execution Control

• HostAgent serves as a system-level control plane using dependency-ordered subtasks, a finite-state controller, dual visual-semantic introspection, and private/shared state for coordinated fault-isolated execution.Its services include task decomposition, process inspection and launching, AppAgent instantiation, scheduling, synchronization, and structured outputs for downstream execution.Source: §3.2 · pp. 5–6

3.3 AppAgent: Application-Specialized Execution Runtime

• AppAgents are isolated, application-specialized workers that combine multimodal perception, ReAct-style reasoning, GUI/API actions, local FSMs, and shared state for robust, extensible execution.Their SDK supports declarative API registration and searchable application knowledge, allowing third parties to add capabilities without retraining models or disrupting the core system.Source: §3.3 · pp. 6–7

3.4 Hybrid Control Detection

• UFO2 fuses Windows UIA metadata with OmniParser-v2 vision grounding to construct a unified control graph covering both standard and UI-invisible or custom-rendered controls.Bounding-box deduplication removes visual detections overlapping UIA controls by more than 10% IoU, while remaining detections become pseudo-UIA objects for downstream annotation.Source: §3.4 · p. 7

3.5 Unified GUI–API Action Orchestrator

• Puppeteer dynamically selects application APIs or GUI automation per action, preferring semantically equivalent APIs for reliability and atomicity while falling back to simulated clicks or keystrokes when needed.This unified execution layer can collapse long GUI sequences into single API calls, reducing execution time and failure surface [17] [34].Source: §3.5 · pp. 8–9

3.6 Continuous Knowledge Integration Substrate

• UFO2’s persistent knowledge substrate combines indexed documentation with execution traces in a versioned RAG pipeline, enabling runtime application-specific guidance without fine-tuning [39].Semantic retrieval makes knowledge fast, interpretable, and cache-friendly, while versioned indexing lets artifacts evolve with software updates and reduces model obsolescence [41] [42] [43].Source: §3.6 · p. 9

3.7 Speculative Multi-Action Execution

• The design targets a fundamental CUA bottleneck in which isolated action execution requires full LLM inference per GUI step, increasing latency, resource use, and cumulative errors in complex workflows.Dynamic GUI changes can invalidate plans, motivating runtime validation alongside batching.Source: §3.7 · pp. 9–10
• Speculative multi-action execution uses one LLM inference to predict multiple plausible actions, amortizing planning overhead across several GUI steps.Each predicted action includes a target control, operation, and rationale.Source: §3.7 · pp. 9–10
• Windows UIA validates each action’s preconditions, such as control visibility and enabled status, before execution to ensure targets remain valid and interactive.OS-level validation avoids relying on vision models for this safety check.Source: §3.7 · p. 10
• Execution proceeds sequentially with immediate early exit on validation failure; the system reports completed actions and replans using the refreshed interface context.This preserves correctness when prior actions alter the GUI and invalidate later speculative steps.Source: §3.7 · p. 10

4.1 Virtualized User Environment with Minimal Disruption

• PiP creates a separate virtual desktop session so automation runs concurrently without blocking the user’s primary desktop.The resizable window uses Windows RDP loopback and preserves the user’s identity, credentials, settings, and network context.Source: §4 · p. 10, §4.1 · p. 11

4.2 Robust Input and State Isolation

• Distinct input queues and device contexts prevent agent actions from interfering with foreground activity or propagating failures to the primary desktop.GUI changes and focus transitions remain scoped to the virtual environment, supporting controlled recovery from unexpected states.Source: §4.2 · p. 11

4.3 Secure Cross-Session Coordination

• Authenticated, encrypted Named Pipes provide two-way coordination between the PiP runtime and host coordinator for assignments, updates, cancellation, and exceptions.A host-side panel provides real-time visibility and partial control without requiring direct access to the PiP window.Source: §4.3 · pp. 11–12

4.4 System-Level Implications

• PiP functions as a system-level isolation primitive that reconciles concurrency, usability, and safety while simplifying recovery from automation failures.The design uses existing RDP capabilities with minimal system overhead and remains backwards-compatible.Source: §4.4 · p. 12

5.1 Multi-Round Task Execution

• Persistent sessions retain context across execution rounds, enabling iterative refinement, follow-up tasks, and human-in-the-loop intervention.This model preserves transparency while supporting progressive convergence on complex workflows.Source: §5.1 · p. 12

5.2 Safeguard Mechanism

• Risky AppAgent actions enter a pending state and require explicit user confirmation before execution, with customizable risk criteria.Unapproved actions are aborted to reduce harmful operations and support organization-specific safety policies.Source: §5.2 · pp. 12–13

5.3 Everything-as-an-AppAgent

• An agent registry wraps third-party components as interchangeable AppAgents, allowing the HostAgent to dispatch work across diverse execution backends.Even lightweight wrappers, including one for OpenAI Operator, produced tangible performance gains.Source: §5.3 · p. 13

5.4 AgentOS-as-a-Service

• Client-server deployment separates local GUI operations from centralized orchestration and LLM execution, supporting security, maintainability, and multiple concurrent clients.Server-side updates can propagate without modifying the client.Source: §5.4 · p. 13

5.5 Comprehensive Logging and Debugging Infrastructure

• Structured logs capture prompts, outputs, control metadata, UI snapshots, and errors, enabling replay, prompt editing, visualization, and targeted debugging.The observability layer acts as lightweight provenance for agent behavior and supports rapid iteration.Source: §5.5 · p. 13

5.6 Automated Task Evaluator

• An LLM-as-a-judge evaluator parses session traces, assigns partial scores, and classifies outcomes as success, partial, or failure.These results feed dashboards and support offline failure analysis and self-monitoring.Source: §5.6 · p. 13

6 Evaluation

• UFO2 is tested across more than 20 Windows applications using the WAA and OSWorld-W benchmarks with automated task verification.WAA contains 154 tasks across 15 applications, while OSWorld-W contains 49 Windows-focused tasks.Source: §6 · p. 14, §6.1 · pp. 14–15

6.1 Experimental Setup

• Benchmark environments use isolated virtual machines, Azure OpenAI services for GPT-family models, and an NVIDIA A100 for visual grounding.Reported OSWorld baseline scores differ slightly from prior Ubuntu-focused results because of verification corrections and Windows-task alignment.Source: §6.1 · pp. 14–15
• The comparison includes five representative GPT-4o-based CUAs and both UFO2-base and the full UFO2 implementation.The full system adds hybrid detection, GUI–API interactions, and continuous knowledge augmentation; API integrations were limited to three office applications in OSWorld-W.Source: §6.1 · p. 15
• Success Rate measures completed tasks, while Average Completion Steps measures LLM-involved inference steps and therefore practical efficiency.Fewer completion steps correspond to lower inference latency and computational overhead.Source: §6.1 · p. 15

6.2 Success Rate Comparison

• 27.9% SR on WAA and 28.6% SR on OSWorld-W with GPT-4o exceed Operator by 7.1 and 14.3 percentage points, respectively.With o1, UFO2 reaches 30.5% SR on WAA and 32.7% on OSWorld-W.Source: §6.2 · p. 15
• The evaluation supports deeper OS and application integration as a basis for higher reliability and broader automation coverage than primarily visual CUA designs.The authors note remaining potential in niche or legacy applications with limited API availability.Source: §6.2 · pp. 15–16
• 40.0% SR on web-browser tasks with UFO2-o1 and 58.3% on coding tasks with UFO2 GPT-4o lead their respective application categories.The results are attributed to hybrid GUI–API interaction and continuous knowledge integration.Source: §6.2 · p. 16, Table 2 · p. 17
• LibreOffice tasks perform poorly across agents because of limited accessibility support, while OSWorld-W Microsoft 365 tasks reach 51.9% SR with UFO2-o1.The contrast links stronger results to richer OS-native APIs and structured accessibility data.Source: §6.2 · p. 16
• UFO2 reaches 9.1% SR on cross-application tasks, indicating promise for HostAgent-led decomposition and AppAgent coordination despite modest performance.These tasks require coordination across application boundaries and remain challenging.Source: §6.2 · p. 16
• More than 62% of UFO2-base failures on WAA are control-detection failures, whereas OSWorld-W has more plan errors associated with complex workflows.These findings motivate hybrid visual grounding, richer documentation, domain knowledge, and application APIs.Source: §6.2 · pp. 16–17

6.3 Evaluation on Hybrid Control Detection

• Up to 9.86% of previously irrecoverable cases became completions with hybrid control detection, which consistently outperformed UIA-only and OmniParser-only settings.The approach merges complementary coverage while reducing redundancy from overlapping detections.Source: §6.3 · p. 17

6.4 Effectiveness of GUI + API Integration

• GUI + API actions increased Success Rate by 6.1% for GPT-4o and 8.2% for o1 over GUI-only execution.GPT-4o mainly recovered control-detection failures, whereas o1 more often used APIs to address planning errors.Source: §6.4 · p. 18
• GUI + API actions reduced completion steps by 6.5% for GPT-4o and 58.5% for o1 on identical tasks.API calls can replace multiple GUI interactions with concise application-level operations.Source: §6.4 · p. 19
• In an Excel export case, GUI-only execution required five steps, whereas the save_as API completed the same task in one call.The one-step API path also avoids accumulating errors across multiple GUI interactions.Source: §6.4 · pp. 18–19

6.5 Continuous Knowledge Integration Evaluation

• Knowledge integration reduced planning failures by up to 17.7% through help-document retrieval and summarized execution experience.Self-experience improvements with o1 were consistent across both benchmarks, while help-document benefits depended on task complexity and specificity.Source: §6.5 · p. 19

6.6 Effectiveness of Speculative Multi-Action Execution

• Speculative multi-action execution retained comparable Success Rate while reducing average steps by up to 10% on WAA and 51.5% on OSWorld-W.Fewer execution steps reduce LLM calls, latency, and cost.Source: §6.6 · p. 19, Table 7 · p. 19
• Speculative planning completed dependent, non-interfering Word actions in one LLM call instead of separate inferences.The example selected heading text and then centered it as a single predicted sequence.Source: §6.6 · pp. 19–20

6.7 Operator as an AppAgent

• UFO2 + Operator achieved a 26.0% WAA success rate versus 20.8% for Operator alone.HostAgent decomposition, additional tips, and a single-application view were identified as contributing factors.Source: §6.7 · p. 20, Figure 23 · p. 20

6.8 Efficiency Analysis

• Fully integrated UFO2 reduced average execution steps versus UFO2-base by up to 50%.The reduction reflects hybrid GUI–API orchestration and advanced control-detection strategies.Source: §6.8 · p. 20, Table 8 · p. 21
• LLM inference dominated latency at around 10 seconds per inference, while UFO2’s hybrid control detection added approximately 1 second per step beyond the baseline.The additional detection cost was attributed largely to OmniParser-v2 and treated as a robustness-efficiency trade-off.Source: §6.8 · p. 21
• Overall task completion times remained approximately 1 minute because fewer required steps offset per-step latency.Profiling was conducted on successfully completed tasks to support fair comparisons.Source: §6.8 · pp. 20–21

6.9 Model Ablation

• Models with built-in reasoning typically achieved higher Success Rates across UFO2 and UFO2-base configurations.The ablation compared GPT-4V, GPT-4o, Gemini 2.0 Flash Thinking, and o1.Source: §6.9 · p. 21, Figure 25 · p. 21

7 Discussion & Future Work

• Multi-action tasks can take 1–2 minutes because UFO2 invokes LLM inference at each decision step.The PiP interface reduces user inconvenience during these longer-running automations, while lightweight task-specific models remain future work.Source: §7 · p. 21
• UFO2 does not yet consistently achieve human-level performance across all Windows applications.The paper identifies visual-language-model improvements and tighter OS/API integration as required directions.Source: §7 · pp. 21–22
• UFO2 targets Windows, but its modular layered design is described as adaptable to Linux and macOS through analogous accessibility APIs.Cross-platform deployment remains future work rather than an evaluated capability.Source: §7 · p. 22

8.1 Computer-Using Agents (CUAs)

• Existing CUAs automate complex workflows through multimodal models, but largely remain prototypes lacking deep OS and native application integration.UFO2 addresses this gap with modular AgentOS architecture, hybrid GUI detection, and non-disruptive execution.Source: §8.1 · p. 22
• UFO2 extends prior CUA research by combining system-level integration, application capabilities, hybrid GUI detection, and non-disruptive execution.Source: §8.1 · p. 22

8.2 LLMs for Operating Systems

• AIOS research positions an LLM at the center of operating-system design, with agents exposing specialized capabilities through natural language.Source: §8.2 · p. 22
• Concrete AIOS prototypes integrate LLM interactions and tool APIs with privileged OS functions, while semantic file management extends system-level agent capabilities.Source: §8.2 · p. 22
• AutoOS applies LLMs to autonomous Linux kernel-parameter tuning, illustrating system-level optimization as another form of operating-system integration.Source: §8.2 · p. 22
• UFO2 applies the emerging LLM-operating-system paradigm specifically to scalable and practical multimodal desktop automation.Source: §8.2 · p. 22

9 Conclusion

• UFO2 combines a centralized HostAgent with application-specialized AppAgents that integrate GUI actions, native APIs, and application-specific knowledge.A PiP virtual desktop enables concurrent user-agent workflows without interference.Source: §9 · p. 23
• Over 20 real-world Windows applications show significant improvements in UFO2’s robustness, accuracy, and scalability over state-of-the-art CUAs.The evaluation also reports that GPT-4o with UFO2 surpasses specialized CUAs such as Operator.Source: §9 · p. 23