Source-linked AI summary
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, Tao Yu
TL;DR
Existing benchmarks do not adequately represent diverse, interactive computer use across operating systems, interfaces, and applications. OSWORLD introduces a scalable real-computer environment and a 369-task benchmark with reproducible setup and execution-based evaluation. Evaluations reveal substantial deficiencies in current agents, particularly in GUI grounding and operational knowledge.
Problem
Existing benchmarks lack interactive environments or are limited to specific applications and domains, failing to cover diverse real-world computer use across operating systems and interfaces.
Method
OSWORLD provides a scalable real-computer environment and a 369-task benchmark with configurable initial states and custom execution-based evaluation across real applications and operating systems.
Results
12.24% is the highest reported baseline success on OSWORLD, while evaluated agents range from 0.99% to 12.24% and show substantial weaknesses in computer assistance.
Takeaways & Limitations
OSWORLD provides a unified basis for studying multimodal agents on open-ended computer tasks and identifies GUI understanding and operational knowledge as essential areas for development.
Takeaways & Limitations
Agents lack prior software-use knowledge and can perform poorly on specialized tasks such as brightness adjustment in GIMP.
Abstract
from arXiv · showhide
Autonomous agents that accomplish complex computer tasks with minimal human interventions have the potential to transform human-computer interaction, significantly enhancing accessibility and productivity. However, existing benchmarks either lack an interactive environment or are limited to environments specific to certain applications or domains, failing to reflect the diverse and complex nature of real-world computer use, thereby limiting the scope of tasks and agent scalability. To address this issue, we introduce OSWorld, the first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across various operating systems such as Ubuntu, Windows, and macOS. OSWorld can serve as a unified, integrated computer environment for assessing open-ended computer tasks that involve arbitrary applications. Building upon OSWorld, we create a benchmark of 369 computer tasks involving real web and desktop apps in open domains, OS file I/O, and workflows spanning multiple applications. Each task example is derived from real-world computer use cases and includes a detailed initial state setup configuration and a custom execution-based evaluation script for reliable, reproducible evaluation. Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWorld reveals significant deficiencies in their ability to serve as computer assistants. While humans can accomplish over 72.36% of the tasks, the best model achieves only 12.24% success, primarily struggling with GUI grounding and operational knowledge. Comprehensive analysis using OSWorld provides valuable insights for developing multimodal generalist agents that were not possible with previous benchmarks. Our code, environment, baseline models, and data are publicly available at https://os-world.github.io.
1 Introduction
Existing benchmarks do not provide a sufficiently broad, executable environment for evaluating multimodal agents on diverse real-world computer workflows. OSWORLD addresses this gap with a scalable real-computer environment and a 369-task benchmark, while evaluations reveal substantial limitations in current agents.
- Motivation: Existing benchmarks lack interactive execution or restrict tasks to particular applications and domains, limiting coverage of complex workflows across interfaces.Some prior benchmarks use demonstrations without executable environments, while executable environments simplify observation and action spaces or narrow tasks to web navigation and coding.
- Contribution: OSWORLD provides raw keyboard and mouse control, configurable task states, execution-based evaluation, and interactive learning across Ubuntu, Windows, and macOS.It supports open-ended tasks involving arbitrary applications without requiring application- or domain-specific simulated environments.
- Benchmark: 369 real-world tasks span web and desktop applications, open domains, OS file I/O, GUI and CLI interaction, and multi-application workflows.Nine authors annotated initial-state configurations and custom execution-based evaluation scripts; the benchmark contains 134 unique evaluation functions.
- Results: 12.24% is the highest reported baseline performance on OSWORLD, with evaluated LLM- and VLM-based agents ranging from 0.99% to 12.24%.For workflows requiring cooperation across multiple applications, the highest baseline performance is 6.57%.
- Analysis: Current agents struggle with screenshot grounding, repetitive actions, unexpected windows, and basic GUI or application-specific operational knowledge.Additional accessibility information can help but may also misguide models; higher resolution and more trajectory history can improve performance while increasing context demands.
2 OSWORLD Environment
OSWORLD models autonomous computer use as interaction with real operating systems through configurable, executable environments. Its infrastructure combines realistic task initialization, broad observations and actions, and example-specific evaluation for complex workflows.
- Task formulation: OSWORLD formalizes autonomous-agent tasks as a partially observable Markov decision process with state, observation, action, transition, and reward components.Observations include natural-language instructions and computer screenshots, accessibility trees, or their combination.
- Environment infrastructure: Virtual-machine infrastructure supports task initialization, execution-based evaluation, and interactive learning across Ubuntu, Windows, and macOS.Snapshots provide isolated environments that can be reset efficiently, while configuration files specify setup, post-processing, retrieved artifacts, and evaluation parameters.
- Task setup: Configuration files initialize intermediate computer states, reflecting scenarios where applications are already open or the computer has encountered a crash.The environment prepares required files, opens and resizes software windows, and recreates unavailable inputs as realistically as possible.
- Execution-based evaluation: Example-specific evaluation combines setup, post-processing, retrieval, and dedicated functions tailored to each software task’s requirements.These scripts assess functional correctness by interpreting internal files, retrieving data, and using software-specific scaffolding where necessary.
- Observation space: The observation space includes complete desktop screenshots and XML accessibility trees, preserving multiple windows, files, folders, sizes, and orders.These rich observations support multi-application interaction but create long-horizon challenges from high-resolution images and structured text.
- Action space: The action space covers human-computer mouse and keyboard inputs, including movement, clicks, dragging, keystrokes, and hotkeys.OSWORLD uses pyautogui to reproduce and replay these inputs through code.
3 OSWORLD Benchmark
OSWORLD is a real-computer benchmark built around 369 Ubuntu tasks and 43 additional Windows tasks, with annotated initial states, execution-based evaluation, and broad application coverage. Its construction combines diverse real-world scenarios with author cross-checking and comparison against other digital-agent environments, while human results indicate substantial task complexity.
- Benchmark scope: OSWORLD includes 369 Ubuntu tasks and 43 additional Windows tasks for analyzing autonomous computer agents.The Windows tasks require user activation because of copyright restrictions.
- Task construction and evaluation: Each example specifies a natural-language instruction, an initial-state setup with files and actions, and a manually crafted execution-based evaluation script.Getter functions extract relevant final-state components, which evaluator functions use to assess success.
- Tasks and scenarios: Tasks are collected from diverse real-world resources and include web, desktop, operating-system, and multi-application workflows.Sources include tutorials, videos, how-to sites, Q&A forums, courses, blogs, and guidelines.
- Quality control: Two non-annotating authors attempt every example, and four additional checking rounds address unclear instructions, crashes, and false positives or negatives.The authors report more than 400 man-hours for these later checks, while noting that further red teaming could improve quality.
- Benchmark comparison: OSWORLD compares with existing environments using universal mouse-and-keyboard actions, multimodal observations, cross-app support, intermediate states, and execution-based evaluation functions.Its comparison framework also considers task instances, environment control, and scalability to arbitrary applications in open domains.
- Human performance: 72.36% human accuracy on OSWORLD is below the 88% reported for WebArena, while many OSWORLD examples require at least 900 seconds.The reported results characterize OSWORLD tasks as more time-consuming and challenging than the pure web task dataset.
4 Benchmarking LLM and VLM Agent Baselines
The benchmark evaluates LLM- and VLM-based agents across OSWORLD’s diverse task categories and input settings. Results show large, uneven performance gaps from humans, with GUI grounding and multi-application workflows remaining difficult.
- Benchmark and settings: OSWORLD evaluates baseline agents across OS, Office, Daily, Professional, and multi-application Workflow task categories.The benchmark groups success rates by software type and workflow composition.
- Benchmark and settings: The experiments compare screenshot, accessibility-tree, combined screenshot-plus-tree, and Set-of-Marks inputs for agent grounding.Screenshots are sent directly to VLMs, while accessibility trees are filtered into compact tabular representations and combined settings provide both modalities.
- Results: 5.26% to 5.80% success rates occur for the strongest screenshot-only VLMs, while a11y-tree settings for advanced language models range from 2.37% to 12.24%.These figures remain well below the 72.36% human performance reported for participants unfamiliar with the software.
- Results: Agent performance varies substantially by task type: CLI-oriented tasks generally outperform GUI tasks, while multi-application workflows are generally below 5%.Human performance remains around 70% across task types with less than 5% variance, contrasting with model behavior.
- Results: A11y-tree and Set-of-Marks benefits depend on the model, with GPT-4V declining under Set-of-Marks and Gemini-Pro showing reversed conclusions.The authors associate Set-of-Marks degradation with high-resolution operating-system interfaces, dense elements, noise, and coordinate-level operations.
- Results: Screenshot-only input achieves the lowest performance at 5.26% but avoids reliance on a11y trees and their potentially large inference burden.The paper presents purely vision-based agents as a longer-term direction for generalization and physical-world integration.
5 Analysis
OSWorld analysis shows that agent performance deteriorates on longer, multi-application, visually perturbed, and specialized tasks. Qualitative failures chiefly reflect weak GUI grounding, software knowledge, and reliable execution, despite occasional success on complex workflows.
- Task difficulty: Model success drops as human completion time increases, with tasks longer than 180 seconds becoming almost impossible for agents.Human performance remains 84.91%, 81.08%, and 49.57% across easy, medium, and hard groups.
- Number of apps involved: 13.74% average performance on single-app tasks is more than double the 6.57% observed for multi-app workflows.GUI-intensive Office tasks, including some LibreOffice Calc subsets, often perform worst.
- Observation variance: Window position, window size, and irrelevant software clutter reduce performance by over 60% to 80% on tasks initially solved at 50.79%.Agents can switch windows to some degree but often fail to maximize them or manage window states comprehensively.
- Success and failure cases: Agents can solve complex workflows such as extracting subtitles with ffmpeg, but they also fail simple GUI requirements through irrelevant actions.A document title-centering task produced repeated selections and irrelevant menu actions.
- Operational knowledge: Specialized tasks expose insufficient software knowledge: in GIMP, the agent randomly searches menus for brightness adjustment until exhausting its step budget.The analysis reports poor performance across GIMP, LibreOffice Calc, and Chrome.
- Qualitative analysis: More than 75% of 550 failed examples contain mouse-click inaccuracies, indicating weak execution despite detailed planning.Misclicks trigger repetitive clicks, environmental noise, and mismatches between actions and application state.
- Efficiency: Agents are as inefficient as humans on repetitive copying, pasting, and batch-editing tasks.The paper attributes agent inefficiency to absent or insufficiently trained APIs and sluggish responses that can cause timeouts.
- Model comparison: Claude’s average accuracy is lower than GPT-4V by 2.84% to 7.76%, despite providing satisfactory high-level solutions.The reported difference is associated with hallucinations in detailed grounding.
6 Related Work
Prior multimodal-agent benchmarks primarily target coding, web browsing, or mobile applications. Related GUI-interaction work often uses structured sources such as HTML, accessibility trees, or view hierarchies for grounding, although these sources can be noisy or unavailable.
- Benchmarks for multimodal agents: Benchmarking digital interaction agents spans coding environments, web scenarios, and mobile applications.The cited prior work evaluates programming and software-engineering activities, web interfaces, and open-ended web tasks.
- Vision-language models for multimodal agents: GUI-interaction systems commonly use HTML, accessibility trees, or view hierarchies as grounding sources.These sources may be verbose, non-intuitive, noisy, or inaccessible, making screenshot-based perception important.
7 Conclusion and Future Work
OSWORLD provides a realistic, cross-platform environment for developing and evaluating autonomous computer agents, while exposing persistent challenges in GUI understanding, operational knowledge, safety, and agent methodology.
- Conclusion: OSWORLD spans multiple operating systems, interfaces, and applications to support realistic computer-task execution and interactive learning.The environment supports Ubuntu, Windows, and macOS, along with GUI and CLI interaction.
- Future directions: Future work should improve VLM efficiency, long-context processing, GUI grounding, precise action generation, and image-based history encoding.These improvements are identified as needs for robust GUI interactions and memory or reflection.
- Future directions: Future agent architectures should support autonomous exploration, efficient memory, reflection, personalization, and protocols for GUI and CLI operation.The proposed directions address lengthy interaction histories, user-specific behavior, and efficient action execution.
- Safety: Realistic deployment requires safety evaluation beyond task correctness because agents may perform unnecessary damaging actions that current evaluations do not reliably detect.The environment uses virtual machines to reduce irreversible host damage, but lacks a reliable isolated-environment safety metric.
- Environment: Virtual machines provide cross-platform compatibility across operating systems and CPU architectures while supporting multiprocess training and evaluation.This distinguishes them from commonly used Docker-based setups.
- Environment: OSWORLD exposes screenshots, accessibility trees, terminal output, and raw mouse-and-keyboard control, with extensible observations and pyautogui-based actions.The action space includes clicks, dragging, keystrokes, hotkeys, and other human-computer inputs.
A.3.2 COMPUTER_13
The computer_13 variant structures pyautogui control into a finite parameterized action space to support reinforcement-learning research, while the benchmark selects representative cross-platform software and task resources.
- COMPUTER_13: computer_13 wraps pyautogui in a finite action class with 13 parameterized action types, excluding three special task-control actions.The structured enumeration is intended to provide distinct actions that reinforcement-learning systems can learn and optimize.
- Platform and software selection: Ubuntu is selected as the benchmark foundation because it offers broad open-source software and community support for everyday and advanced tasks.The software-selection criteria include availability, support resources, and category diversity.
- Platform and software selection: The selected software covers media playback, web browsing, email, coding, office productivity, and image editing across eight application types.The set includes VLC, Chrome, Thunderbird, VS Code, LibreOffice components, and GIMP.
- Task resources: Task examples draw on resources including application support pages, forums, tutorials, question-and-answer sites, and videos.The listed resources span applications such as Thunderbird, Chrome, and GIMP.
B.4 Task Examples Collection
The task-example collection reports benchmark statistics and uses reproducible initial-state preparation alongside application-specific evaluation methods across Ubuntu and Windows environments.
- Benchmark statistics: The benchmark statistics cover 369 Ubuntu examples and a 43-example Windows analytic set, with intent distributions visualized against other digital-agent benchmarks.The visualization samples 300 examples per dataset and applies t-SNE with randomly selected hyperparameters.
- Environment preparation: Initial-state setup starts a virtual-machine snapshot, optionally prepares files or software, and configures the system for task execution.The setup is designed to simulate a human work in progress.
- Application evaluation: Application evaluations combine final-file comparison, configuration or profile inspection, APIs, accessibility-tree checks, browser automation, and custom extensions.The exact signals vary across LibreOffice, Thunderbird, VLC, Chrome, VS Code, and GIMP tasks.
- Cross-platform evaluation: Windows tasks reuse Linux evaluations where possible, modifying only platform-specific account profiles or port-transfer tools.Thunderbird profiles are revised for Windows, while Chrome replaces socat with Ncat.
- Task diversity: Examples are selected from varied application sets to represent distinct operational logic and capabilities required for computer interaction.The collection includes diverse use cases such as presentation display, file copying, privacy cleanup, media control, and video editing.
C.5 Full Results of Baseline Methods
Baseline analysis compares methods across LLMs and VLMs and examines how GPT-4V changes action-step distributions and termination behavior under different observation settings.
- Baseline results: Baseline breakdowns report performance across different LLM- and VLM-based methods for reference.The section presents comparative results rather than a single model configuration.
- Observation settings: GPT-4V exhibits different numbers of executed steps and termination points under Accessibility Tree, Screenshot, Screenshot+A11y Tree, and Set-of-Mark settings.These behavioral differences are presented as clues for future work.
D Examples of Qualitative Analysis
The qualitative analysis provides code-execution traces for examples, including GPT-4V agents’ step distributions across four settings.
- The analysis supplements qualitative examples with detailed code-execution processes and additional error-cause analysis.These traces are intended to provide insights for further improvement.
- Figure 15 presents the distribution of steps taken by GPT-4V-based agents across four settings.
D.1 Success and Failure Cases
The examples cover successful and failed tasks across video processing, desktop applications, spreadsheets, and browsers, illustrating both task completion and operational errors.
- Success and Failure Cases: A successful video task removes subtitles, exports them as subtitles.srt, and stores the file beside the video.
- Success and Failure Cases: The examples include extracting subtitles with ffmpeg and creating a subtitle-free video, alongside a table of success rates divided by application domains.
- Success and Failure Cases: A failed LibreOffice Writer task attempts to center-align a heading through mouse dragging and toolbar clicking.
- Success and Failure Cases: GIMP brightness adjustment fails after repeated menu-navigation attempts and duplicate coordinate actions cannot locate the correct control.
- Success and Failure Cases: LibreOffice Calc examples fill conference-city cells through repeated click, type, and Enter operations, including NeurIPS 2017 at Long Beach.
- Success and Failure Cases: Chrome examples try to change the default search engine to Bing by navigating approximate coordinates through menus and settings.
D.2 Common Errors by GPT-4V Agents
The qualitative analysis identifies mouse-click inaccuracies and inadequate handling of environmental noise as two common GPT-4V error types in web-page tasks.
- Common Errors: Mouse-click inaccuracies can select the wrong interface element, producing a discrepancy between the current and target states.In the example, a coordinate error clicks a favorite button instead of product-category images.
- Common Errors: Environmental noise from cookies or pop-up ads can make subsequent actions ineffective when the agent does not close those elements.
- Common Errors: The analyzed web-page errors are illustrated using Figure 18 and Figure 17.
D.3 Discrepancies in Task Difficulty between Agent and Human
The examples compare human- and agent-oriented tasks and show that agents can execute many interface actions while still failing to satisfy task requirements.
- Task Difficulty Discrepancies: The supplementary examples include tasks labeled as easier for humans and tasks labeled as easier for agents.
- Human-Favored Tasks: Formatting examples remove character highlighting and add or bold slide text through toolbar and text-box interactions.
- Agent-Favored Tasks: An agent trims a video from 2 seconds to 4 seconds with ffmpeg but does not follow the task’s full requirements.
- Agent-Favored Tasks: System-monitoring examples use sar 1 30 and redirect output to System Resources Report.txt on the Desktop.
- Agent-Favored Tasks: Other command-line examples attempt to open the report or force-quit LibreOffice with killall or pkill commands.
- Claude-3 Failure Examples: Claude-3 failure examples include finding secret.docx, copying spreadsheet data into Chrome search, and replacing text with test.
D.5 Analysis on SoM Effectiveness
The SoM-equipped agent struggled with a VS Code settings task, while agents without SoM succeeded by editing settings.json. SoM’s fixed visual labeling and longer interaction path reduced flexibility in this diverse task.
- Interaction path: The non-SoM sequence used direct mouse actions to open settings, search for Python, edit the configuration, enter reportMissingImports: 'none', and save.The sequence also included repeated search attempts and a reported inability to find the right-click area.
- Interaction path: The SoM interaction path included opening VS Code settings, searching for Python-related settings, and selecting the settings.json editing option.Recorded actions show menu navigation, search, and selection of the “Edit in settings.json” control.
- Task comparison: Agents without SoM succeeded by editing settings.json to disable Python code checking in VS Code.The task involved turning off the Python plugin’s code-checking function.
- Task comparison: SoM-equipped agents instead searched for the interactive setting and struggled to find and deselect the checkbox.The SoM path required locating the setting through the interface, whereas the alternative directly modified the configuration file.
- Interpretation: SoM shortens the action space for labeled-box web tasks but can hinder exploration and adaptability in tasks requiring expertise and flexibility.The authors attribute the difficulty partly to longer action paths and incorrect mouse clicks in the VS Code example.