Source-linked AI summary
UFO: A UI-Focused Agent for Windows OS Interaction
Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang
TL;DR
Windows-focused UI agents remain limited despite the platform’s broad use and complex cross-application workflows. UFO addresses this gap with a GPT-Vision dual-agent architecture and grounded control interaction, and reports strong benchmark and case-study performance while acknowledging backend and unfamiliar-UI limitations.
Problem
Windows offers widely used, diverse applications and complex cross-application tasks, but VLM agents tailored to autonomously fulfill requests on Windows remain largely unexplored.
Method
UFO uses GPT-Vision, HostAgent-AppAgent coordination, GUI and control observation, grounded control interaction, and customizable Windows actions to automate requests.
Results
UFO achieves an 86% success rate across the WindowsBench benchmark, more than doubling the best baseline, while also attaining the highest completion rate and fewest steps.
Takeaways & Limitations
UFO supports automated, extensible task completion across Windows applications, including longer requests that span multiple applications.
Takeaways & Limitations
UFO currently supports only controls and actions available through pywinauto and Windows UI Automation, excluding applications that use other backends.
Abstract
from arXiv · showhide
We introduce UFO, an innovative UI-Focused agent to fulfill user requests tailored to applications on Windows OS, harnessing the capabilities of GPT-Vision. UFO employs a dual-agent framework to meticulously observe and analyze the graphical user interface (GUI) and control information of Windows applications. This enables the agent to seamlessly navigate and operate within individual applications and across them to fulfill user requests, even when spanning multiple applications. The framework incorporates a control interaction module, facilitating action grounding without human intervention and enabling fully automated execution. Consequently, UFO transforms arduous and time-consuming processes into simple tasks achievable solely through natural language commands. We conducted testing of UFO across 9 popular Windows applications, encompassing a variety of scenarios reflective of users' daily usage. The results, derived from both quantitative metrics and real-case studies, underscore the superior effectiveness of UFO in fulfilling user requests. To the best of our knowledge, UFO stands as the first UI agent specifically tailored for task completion within the Windows OS environment. The open-source code for UFO is available on https://github.com/microsoft/UFO.
1 Introduction
Windows is a promising but largely unexplored platform for natural-language agents that visually understand and autonomously operate application interfaces. UFO addresses this gap with a GPT-Vision-based dual-agent system that grounds actions directly on Windows controls.
- Windows combines broad use, diverse applications, and cross-application tasks requiring long-term planning, yet Windows-focused VLM agents remain largely unexplored.
- UFO introduces a UI-focused Windows agent that uses GPT-Vision to analyze screenshots and GUI information, select applications and controls, and execute natural-language requests.
- Its dual-agent framework separates application selection and GUI analysis from control navigation and action execution.
- A control interaction component grounds GPT-Vision actions on application controls, enabling fully automated execution without human intervention.
2 Related Work
Prior work established LLM agents, multi-agent systems, and multimodal GUI interaction, but primarily targeted general tasks or mobile environments. UFO extends this direction to cross-application UI manipulation on Windows.
- LLM agents augment language models with planning, observation, memory, and responsive actions for more intricate tasks.
- Multi-agent frameworks allocate tasks among customizable agents whose complementary strengths support complex task completion.
- Multimodal GUI agents use screenshots and visual-language models to navigate mobile applications, while specialized models target GUI understanding and navigation.
- UFO distinguishes itself as a multimodal agent tailored to Windows application UI manipulation and tasks spanning different applications.
3 The Design of UFO
UFO coordinates a HostAgent and an AppAgent to plan, select applications, observe annotated interfaces, and execute grounded controls across multi-application tasks. Windows-specific features improve control selection, extensibility, feedback, and safety.
- 3.1 UFO in a Nutshell: The HostAgent selects or switches applications and formulates a global plan, while the AppAgent executes actions iteratively within the selected application.
- 3.1 UFO in a Nutshell: HostAgent decisions use desktop screenshots, application information, memory, and task status to select applications and coordinate progress.
- 3.2 HostAgent: AppAgent observes current and prior screenshots, annotated controls, control information, memory, and examples before choosing controls and actions.
- 3.5.1 Interactive Mode: The framework supports interactive requests and user feedback, enabling longer and more intricate tasks through an interactive mode.
- 3.5.2 Action Customization: Action customization extends operations beyond current UI controls to keyboard shortcuts, macros, plugins, and multimodal screenshot summaries.
- 3.5.3 Control Filtering: Control filtering reduces screenshot clutter and irrelevant choices because Windows UI Automation can expose hundreds of controls.
4 Experiment
UFO is evaluated on WindowsBench through quantitative comparisons and case studies spanning diverse tasks and applications. It achieves strong benchmark performance, handles cross-application workflows, and uses visual interaction and safeguards to complete complex requests.
- Benchmark & Baselines & Metrics: WindowsBench contains 50 user requests across 9 popular Windows applications representing work, communication, coding, reading, and web browsing.
- Benchmark & Baselines & Metrics: The evaluation measures success, action steps, completion rate, and safeguard rate, with three tests per request and the highest completion-rate run selected.
- Performance Evaluation: 86% success rate across WindowsBench, more than double GPT-4’s rate, while UFO also achieves the highest completion rate, fewest steps, and 85.7% safeguard rate.
- Performance Evaluation: UFO’s advantage over baselines is attributed to direct application interaction and visual capabilities, whereas GPT-3.5 and GPT-4 rely on human surrogates and textual instructions.
- Performance Evaluation: Across applications, Adobe Acrobat is an exception because unsupported Windows UI Automation control types reduce performance to 60% success and 78.7% completion.
- Performance Evaluation: For multi-application requests, UFO achieves 80% success, 83% completion, and 100% safeguard rates while averaging 9.8 steps.
- Case Studies: In a PowerPoint case study, UFO finds the hidden ‘Remove All Presentation Notes’ function and requests confirmation before deleting notes.
- Case Studies: In a cross-application case, UFO extracts meeting-note text, interprets an image, and composes and dispatches a confirmed Outlook email.
5 Limitations & Lessons Learned
UFO’s current scope is constrained by supported UI Automation backends and difficulty exploring unfamiliar application interfaces. The authors propose broader backends and external search knowledge to improve coverage and adaptability.
- UFO currently supports only UI controls and actions provided by pywinauto and Windows UI Automation, excluding applications or controls using other standards or backends.
- The authors propose alternative backends, dedicated GUI models, and online search knowledge to broaden supported applications and improve planning on unfamiliar UIs.
- UFO may require substantial time to identify actions in niche or unfamiliar application interfaces.
6 Conclusion
UFO is a GPT-Vision UI agent for Windows that combines application selection, control interaction, and automation to complete natural-language requests. Its demonstrated workflows include cross-application content extraction and email composition, with safeguard confirmation before sending.
- System and workflow: UFO uses GPT-Vision to analyze application screenshots and control information, dynamically select applications and controls, and ground actions through a control interaction module.Its dual-agent design supports application switching and fully automated execution while retaining safeguards for sensitive actions.
- Case study: UFO composes an email by combining meeting action items with a workflow description extracted from an image.The completed draft includes the recipient, subject, action items, and workflow content; sending remains subject to user confirmation.
- Case study: The meeting-note workflow identifies four assigned action items with owners, deadlines, and resource or communication responsibilities.The listed owners are Zac Yang, Kevin Cheung, Tom James, and Sophia Chow.
A Requests in WindoesBench and Detailed Evaluations
WindowsBench contains complete requests across nine popular Windows applications, with tables reporting UFO’s task success and safeguard behavior. The evaluation records whether sensitive requests trigger confirmation and whether requests are completed successfully.
- Benchmark scope: WindowsBench evaluates UFO on requests spanning nine popular Windows applications and common functions.Requests with follow-up tasks are sequentially numbered in the reported tables.
- Evaluation criteria: The benchmark tables report both safeguard activation for sensitive requests and success or failure in completing each request.A check mark denotes successful safeguard activation or request completion, while a cross denotes safeguard failure or task failure; a dash indicates no confirmation is required.
B Performance Breakdown of GPT-3.5 and GPT-4
GPT-4 outperforms GPT-3.5 on WindowsBench, but both human-surrogate baselines remain substantially less effective than UFO overall. Their weaknesses vary by application, including application selection and missed interaction steps.
- Overall comparison: GPT-4 significantly outperforms GPT-3.5, while both baselines perform inconsistently across applications and fall substantially short of UFO overall.The comparison is reported in the detailed performance breakdowns for GPT-3.5 and GPT-4.
- Application-specific weaknesses: GPT-3.5 frequently selects the wrong application initially in Visual Studio Code tasks, contributing to lower performance.The reported comparison identifies Visual Studio Code as a particularly pronounced weakness for GPT-3.5.
- Application-specific weaknesses: GPT-3.5 often overlooks crucial steps in Edge Browser tasks, resulting in request failures.This is presented as another application-specific example of baseline inconsistency.
- Case studies: The evaluation includes six additional case studies covering both single-application tasks and transitions across multiple applications.These case studies are presented to illustrate UFO’s effectiveness across diverse Windows tasks.
D.1 Reading a PDF
UFO reads and explains PDF content by using visual input to summarize text and locate figures. In the reported example, it describes ImDiffusion’s anomaly-detection approach and compares forecasting, reconstruction, and imputation.
- PDF reading: UFO summarizes a PDF abstract with OCR in visual input and answers a follow-up request about its first figure.The example asks for both the paper’s main contribution and an explanation of the first figure.
- Paper comprehension: ImDiffusion combines time-series imputation with diffusion models to support accurate and robust anomaly detection in multivariate time series.The reported summary describes diffusion models as a preprocessing step for reconstructing anomaly-detection time series.
- Figure interpretation: Figure 1 presents examples of forecasting, reconstruction, and imputation modeling for time-series anomaly detection.The figure description connects these modeling approaches with the paper’s anomaly-detection framework.
- Additional examples: UFO also applies a PowerPoint Designer format, revises the choice based on aesthetic judgment, and downloads a Docker extension in Visual Studio Code.The examples cover presentation formatting and operation on a dark-mode GUI.
D.4 Post a Twitter
UFO completes a Twitter-posting request by navigating Edge to Twitter, locating the posting control, entering the requested text, and proceeding through its safeguard.
- UFO navigates to Twitter through the browser and identifies the page’s Post button.The agent enters the Twitter address in Edge’s address bar, opens the perceived main page, and clicks Post.
- UFO enters the requested text, “It’s a good day.”, into the Twitter posting interface.
- The safeguard activates before the post is completed, adding a confirmation step to the interaction.
D.5 Sending the News
UFO searches for Microsoft news in Edge, summarizes the retrieved content, and sends it to File Transfer in WeChat after user confirmation.
- UFO searches Edge for the latest Microsoft news, opens a search result, and reads a page containing multiple news panes.
- UFO retains the summarized news in memory, triggers its safeguard for user confirmation, and sends the news to File Transfer on WeChat.
- The retrieved Microsoft News page displays several articles, including announcements about Copilot, earnings, and generative-AI solutions.
- The example demonstrates UFO’s ability to transition between applications while completing a long-term, complex task.
D.6 Search Paper in a PowerPoint Slide and Summarize
UFO reads a paper title from a PowerPoint slide, searches for and opens the paper online, then downloads its PDF and generates a summary.
- UFO reads the slide title “Xpert: Empowering Incident Management with Query Recommendations via Large Language Models.”
- UFO searches the title in Edge, skips the first advertisement, and opens the paper from the second search result on arXiv.
- UFO downloads the paper using “Download as PDF” and applies OCR to generate a summary.
- The generated summary describes Xpert as a framework that automates Kusto Query Language recommendations using historical incident data and large language models.
- The example highlights UFO’s cross-application operation and OCR capabilities while completing the multi-part request.