OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

1 Introduction

• Earlier benchmarks lack executable environments or restrict interaction and task scope, limiting evaluation of alternative correct solutions and complex workflows across apps and interfaces.Those limitations also constrain interactive learning and real-world exploration.Source: §1 · pp. 1–2
• OSWORLD provides a scalable real-computer environment with raw keyboard and mouse control, task-state setup, execution-based evaluation, and interactive learning across Ubuntu, Windows, and macOS.It supports open-ended tasks involving arbitrary applications without requiring application- or domain-specific simulated environments.Source: §1 · p. 2, Figure 1 · p. 2
• The benchmark contains 369 real-world tasks spanning open-domain web and desktop apps, OS file I/O, and multi-app GUI and CLI workflows.Nine authors annotated tasks with initial-state configurations and custom execution-based evaluation scripts; the benchmark uses 134 unique evaluation functions.Source: §1 · p. 2, Figure 1 · p. 2
• OSWORLD task examples are more time-consuming and challenging than examples in prior work, according to the human performance study.Source: benchmark · p. 2

benchmark.

• Baseline performance ranges from 0.99% to 12.24%, and the best baseline achieves only 6.57% on workflows involving multiple apps.The results indicate that current LLMs and VLMs are far from capable of serving as computer assistants.Source: benchmark · pp. 2–3
• VLM-based agents struggle with screenshot grounding, repetitive actions, unexpected windows, and basic GUI or app-specific interactions.Higher resolution and more trajectory history can improve performance by as much as doubling it, but require longer context; accessibility-tree and Set-of-Mark information can help or misguide agents depending on the model.Source: benchmark · p. 3

2.1 Task Definition

• Tasks are modeled as POMDPs in which agents act on partial observations until DONE, FAIL, or a step limit, receiving execution-based rewards for success, partial success, or correctly predicted infeasibility.Other outcomes receive zero reward.Source: §2.1 · p. 3

2.2 Real Computer Environment Infrastructure

• OSWORLD uses virtual machines to initialize, execute, and evaluate tasks across operating systems, with isolation and snapshots enabling safe interaction and efficient resets.Configuration files specify initialization, post-processing, retrieved evaluation data, and evaluation functions; environments can run in parallel and headless mode.Source: §2.2 · p. 3, Figure 2 · p. 4

2.2.1 Overview

• A host-side Coordinator initializes virtual machines from task configurations, while a Simulator executes agent code and a Task Manager performs post-processing, retrieves data, and runs evaluation scripts.Multiple virtual machines can run simultaneously on one host to parallelize training and evaluation.Source: §2.2.1 · p. 4

2.2.2 Initial Task Environment Setup

• Initial states target realistic intermediate computer-use scenarios and use a hybrid setup—starting a VM, optionally preparing files, and optionally running reprocessing commands—instead of storing large snapshots for every example.Convenient APIs standardize initial conditions and world settings for extensibility.Source: §2.2.2 · p. 4

2.2.3 Execution-Based Evaluation

• Example-specific evaluation combines tailored setup and post-processing with functions that retrieve software, VM, cloud, and accessibility data to assess task-specific outcomes.Dynamic getters can obtain real-time values at evaluation time for comparison with agent results.Source: §2.2.3 · pp. 4–5, Table 1 · p. 5

2.3 Observation Space

• Observations include full desktop screenshots and XML accessibility trees, preserving human-like perception and cross-application context while challenging agents with high-resolution images and long structured text.Accessibility trees are obtained through platform-specific tools such as ATSPI 2 on Ubuntu and PyWinAuto on Windows.Source: §2.3 · p. 5

2.4 Action Space

• Actions use syntax-correct pyautogui Python code for mouse and keyboard inputs, with WAIT, FAIL, and DONE supporting timing, infeasibility, and task completion.The action space aims to cover human-computer inputs, including right-clicking and modified clicks that narrower domain-specific spaces may omit.Source: §2.4 · p. 6

3.2 Tasks

• The benchmark includes 369 Ubuntu tasks and 43 Windows tasks, assembled from real-world resources and annotated with instructions, setup configurations, and evaluation scripts.Task sources include tutorials, forums, courses, and blogs; authors also devised cross-application scenarios, retained 30 infeasible examples, and integrated 84 tasks from other benchmarks. Task creation and checking took about 1,800 man-hours across nine students over three months.Source: §3 · p. 6, §3.2 · p. 7

3.1 Operating System and Software Environments

• OSWORLD supports Ubuntu, Windows, and macOS, combining real applications and OS tools with GUI, CLI, and cross-app tasks.Its representative applications include Chrome, VLC, Thunderbird, VS Code, LibreOffice, and GIMP, as well as terminal, file manager, image viewer, and PDF viewer.Source: §3.1 · p. 6

3.2 Tasks

• Task configurations prepare files and initial states, while execution-based evaluators extract final-state components and check task success.Examples were tested and reviewed iteratively, resulting in 134 unique evaluation functions; evaluation development and examination averaged about two student man-hours per example.Source: §3.2 · p. 7, environment · p. 7

environment.

• Two non-annotating authors attempted each example for quality control, followed by four rounds of checks that took over 400 man-hours.The authors note that further investment and red-teaming could reduce false positives and negatives.Source: environment · pp. 7–8

3.3 Data Statistics

• Of 369 tasks, 268 (72.6%) target single applications, 101 (27.4%) involve workflows, and 30 (8.1%) are infeasible.The dataset also includes 84 tasks (22.8%) from related datasets, 302 distinct initial states, and 134 evaluation scripts.Source: §3.3 · p. 8, Table 3 · p. 8
• OSWORLD combines raw mouse-and-keyboard actions, screenshot observations, executable exploration, cross-app tasks, intermediate initial states, and example-wise execution-based evaluation.The paper describes these features as enabling universal actions, open-ended task creation, and evaluation beyond static next-step prediction or single-app interactions.Source: §3.3 · p. 8, Table 4 · p. 9

4.1 LLM and VLM Agent Baselines

• The baselines compare open- and closed-source models using compact accessibility trees, screenshots, combined screenshot/tree input, and Set-of-Marks annotations.The tree is filtered and compacted; SoM marks elements with numbered boxes and metadata so the VLM can select an element index rather than predict coordinates.Source: §4.1 · pp. 9–11

4.2 Results

• OSWORLD’s category-grouped results show a11y-tree LLMs at 2.37%–12.24% and screenshot-only GPT-4V/Gemini-Pro-vision at 5.26%–5.80%, versus humans’ 72.36% overall.The baseline comparison covers OS, Office, Daily, Professional, and Workflow tasks.Source: Table 5 · p. 10, §4.2 · p. 11
• Agent performance varies more than human performance across task types: CLI-oriented OS tasks tend to outperform GUI-oriented Office tasks, while multi-app workflows are generally below 5%.Model and setting differences exceed 20% in some cases, whereas human performance stays around 70% with less than 5% variance.Source: Table 5 · p. 10, §4.2 · p. 11

4.1 LLM and VLM Agent Baselines

• Experiments use the three most recent observation-action turns in chat mode, after few-shot prompting yielded only 2.79% success in the pure-screenshot setting.Agents are prompted to finish within 15 steps; inputs are truncated from the beginning if they exceed model token limits.Source: §4.1 · p. 10

4.2 Results

• The effect of accessibility-tree input varies by model: GPT-4V and Claude-3 differ from their screenshot-only results, while Gemini-Pro shows the reverse pattern.The reported pattern suggests remaining difficulty in perceiving and reasoning about GUI elements accurately.Source: §4.2 · p. 11
• For GPT-4V, Set-of-Marks underperforms direct screenshot-plus-tree input, possibly because dense, high-resolution interfaces add box noise and some tasks require precise coordinates.The authors contrast this result with SoM’s effectiveness in classic image understanding and web-agent settings.Source: §4.2 · pp. 11–12
• Screenshot-only VLMs score lowest at 5.26%, yet this configuration needs no accessibility tree and performs decently on multi-app workflows.The paper notes that trees may be unavailable or noisy across software and can impose a large token burden, motivating future work on vision-only agents.Source: §4.2 · p. 12

5.1 Performance by Task Difficulty, Feasibility and App Involved

• Human success fell from 84.91% on easy tasks to 49.57% on hard tasks; GPT-4V did slightly better on infeasible than feasible tasks, 16.67% versus 13.34%.Tasks were grouped by human completion time; the authors caution that agents may falsely label infeasible tasks as failures and emphasize improving overall performance.Source: Table 6 · p. 12, §5.1 · p. 12
• Single-app tasks scored 13.74%, more than double the 6.57% for multi-app workflows; GUI-intensive Office tasks, including LibreOffice Calc, often performed worst.The authors highlight the need for better collaboration across software and greater proficiency in specialized scenarios.Source: Table 6 · p. 12, §5.1 · p. 13

5.2 Performance by Multimodal Observation Variances

• Higher screenshot resolution improved screenshot-only performance, while SoM performed best at 768×432; longer a11y history helped SoM but not screenshot-only inputs.About 90% of single-observation a11y trees required a context length of 6000; adding text history also raises efficiency challenges and calls for better long-context support and memory architectures.Source: Figure 5 · p. 13, §5.2 · pp. 13–14, Figure 7 · p. 13
• Window-position, window-size, and screen-clutter perturbations reduced performance by over 60% to even 80%, exposing weak strategies for managing window states.Agents could switch windows to some degree but often failed to maximize them as an intermediate step.Source: Figure 8 · p. 14, §5.2 · p. 14

5.3 Performance across Different Operating Systems

• GPT-4V scored 4.88% on Ubuntu and 2.55% on Windows, with a 0.7 correlation coefficient across operating systems.The authors interpret this correlation as evidence that OSWorld insights and methods can transfer to Windows despite differences in observation spaces.Source: Table 7 · p. 14, §5.3 · p. 14

5.4 Qualitative Analysis

• Agents sometimes solved complex tasks but failed simple GUI grounding and specialized-software tasks; their perceived task difficulty diverged from humans, and repetitive batch work also exposed inefficiency.Text/design work often challenged agents because of weak grounding and GUI-logic understanding, while code-solvable tasks could be easier for agents but lead them to ignore tool constraints; repetitive tasks suffered from missing APIs, limited API training, or slow responses.Source: §5.4 · pp. 14–16
• More than 75% of 550 failed examples involved mouse-click inaccuracies, which drove repetitive clicks and environmental-noise dilemmas alongside software-knowledge, instruction-interpretation, and visual-oversight errors.Agents could plan detailed steps yet fail to execute accurate clicks, and could struggle to recover after unintended pop-ups or applications appeared.Source: §5.4 · p. 15
• Claude’s average accuracy was 2.84% to 7.76% lower than GPT-4V’s, despite satisfactory high-level solutions, because its detailed grounding could hallucinate actions or interface elements.Examples included treating a double-click as file selection rather than opening, misidentifying a spreadsheet column, and failing to click global replace.Source: §5.4 · p. 16

Benchmarks for multimodal agents

• OSWORLD offers interactive operating-system setup, diverse tasks, precise evaluation scripts, and a controllable setting for learning and evaluating general-purpose digital agents.The paper contrasts this with prior efforts focused on singular environments or lacking executability.Source: §6 · p. 17
• Evaluation of digital interaction agents spans coding, web, and mobile environments, with additional work on GUI grounding and comprehensive tasks; prior efforts often focus on single environments or lack executability.Source: §6 · p. 17

Vision-language models for multimodal agents

• GUI agents often use HTML, accessibility trees, or view hierarchies [9] [15] [27] [37] [64] [46] [62] [66], but noisy or unavailable source code makes multimodal or vision-only perception necessary.Specialized screenshot models target web [4] [12] [18] [23] [43] and mobile [17] [63], while general-purpose foundation models [5] [26] [31] [67] also show potential.Source: §6 · p. 17
• Prompt-based methods [13] [16] [55] [65] and visual reasoning have facilitated digital-agent performance across web pages, mobile apps, and desktop.Source: §6 · p. 17
• Evaluations across text-only, vision-only, and multimodal inputs and multiple methods find current multimodal models far from capable computer agents, with gaps in planning, screenshot perception, pixel locating, and world knowledge.Source: §6 · p. 17

7 Conclusion and Future Work

• OSWORLD broadens computer-agent evaluation across operating systems, interfaces, and applications, while exposing persistent challenges in GUI understanding and operational knowledge.The authors identify these weaknesses as key areas for future research and development.Source: §7 · p. 17
• Future VLMs should handle longer contexts efficiently, ground precise actions robustly despite window changes, and encode image histories for memory and reflection.The authors point to improvements in pretraining, fine-tuning, and model structure, citing previous work [9] [17] [33]; they also invoke the robotics community [6] [7] on efficient inference.Source: §7 · p. 17
• Future agent methods should improve autonomous exploration, history encoding, memory, reflection, knowledge grounding, personalization, and GUI/CLI action protocols.The authors identify long raw observation and action records as a challenge, and propose user profiling and long-term interaction memories to support customization.Source: §7 · p. 18
• Future safety work should assess and constrain harmful side effects, since current task-focused evaluation misses unnecessary damage and lacks a reliable safety metric.Virtual machines make irreversible damage to host machines more difficult, but the authors note that they have not found an efficient way to detect latent side effects.Source: §7 · p. 18
• Future datasets and environments should cover specialized domains, deploy across hardware and software settings, address accessibility-tree quality, and ease data collection.The authors specifically call for filtering redundant tree elements, handling missing ones, and transforming collected computer-operation data into agent capabilities.Source: §7 · p. 18

A.1 Environment Infrastructure

• Virtual machines support diverse operating systems and CPU architectures, with training and evaluation on headless servers or personal computers.Unlike Docker, they can operate their own kernel and system.Source: §A.1 · p. 23

A.2 Observation Space

• OSWORLD implements complete screenshots, accessibility trees, and terminal output as observations, and supports refactoring or extending observations; its accessibility trees represent UI elements and may encode properties such as object type and state.Source: §A.2 · p. 23, §A.2.2 · p. 23

A.3 Action Space

• The main action space uses pyautogui for mouse movements, clicks, dragging, scrolling, text entry, and key sequences; valid actions also include WAIT, FAIL, and DONE.The environment also implements a second, self-designed action space; pyautogui is preferred because its syntax uses fewer prompt tokens. Actions may be represented as discrete labels or numbers.Source: Mouse Controlling Functions · pp. 24–25, Keyboard Controlling Functions · p. 25

A.3.2 COMPUTER_13

• computer_13 defines 13 parameterized action types, excluding three task-process-control actions, as a finite action space for reinforcement learning.Table 8 summarizes each action type’s parameters, including their type, range, and whether they are optional.Source: §A.3.2 · p. 25, Table 8 · p. 26

B.1 Operating System Selection

• The authors exclude ChromeOS because its reliance on Google accounts makes it less suitable for a public benchmark, and avoid macOS over installation copyright concerns.Windows and macOS were described as having the richest software ecosystems, but their closed-source nature raised potential copyright concerns for direct use.Source: §B.1 · p. 26
• Ubuntu is the public benchmark’s foundation because it offers open-source software for common tasks and avoids the copyright concerns associated with closed-source platforms.The authors also developed Windows interaction components and a relatively small Microsoft Office example set; they did not build on macOS because installing it on non-Apple devices was considered potentially illegal.Source: §B.1 · p. 26

B.2 Software Selection

• Ubuntu software selection considers availability on Ubuntu 22.04, open-source licensing, popularity, community and support resources, and diversity of categories.These criteria were chosen in light of the high cost of obtaining task and evaluation-script annotation data.Source: §B.2 · p. 26
• The selected software spans eight types: VLC, Chrome, Thunderbird, VS Code, LibreOffice Calc, Writer and Impress, and GIMP.The authors group VLC, Chrome, and Thunderbird under general usage, and VS Code and LibreOffice apps plus GIMP under professional usage.Source: §B.2 · p. 27

B.3 Task Example Sources

• Task-example resource listings include app support pages, online forums and Q&A sites, tutorials, YouTube, and GitHub resources.The supplied listings name sources including Thunderbird Support, Google Chrome Help, Super User, Stack Overflow, WikiHow, and LibreOffice Forum.Source: Table 9 · p. 27, §B.3 · pp. 27–28

B.4 Task Examples Collection

• 369 Ubuntu tasks and 43 Windows analytic tasks make up the reported sets; the Windows set has no infeasible or integrated tasks.The instruction-embedding visualization shows a broad semantic distribution distinct from the other compared benchmarks, while the authors describe OSWorld’s environment as controllable and executable for reliable evaluation.Source: §B.4 · pp. 28–29, Table 11 · p. 28, Figure 13 · p. 29

B.5 Initial State Setup Details

• Initial-state setup starts a VM from a snapshot, optionally prepares files or software, then optionally runs task-specific preprocessing.Convenience APIs configure initial conditions and world settings to standardize and extend task setup.Source: §B.5 · p. 29

B.6.1 Ubuntu

• Ubuntu evaluation checks office-suite result files or attributes, Thunderbird profile data, VLC interfaces or files, and Chrome via Playwright and website information.LibreOffice tasks primarily compare output files with golden references or inspect file properties; Thunderbird email composition is checked through the accessibility tree before sending.Source: §B.6.1 · pp. 29–30
• VS Code and GIMP evaluations distinguish file-manipulation tasks, checked against reference outputs, from software-feature tasks, checked through internal settings or configuration.VS Code uses a custom extension and settings JSON checks; GIMP file outputs are analyzed with image-processing libraries such as Pillow.Source: §B.6.1 · p. 30

B.7 Windows

• Windows reuses LibreOffice evaluation for Microsoft Office tasks, adapts Thunderbird’s account profile, and substitutes Ncat for Chrome’s port transfer.The Office applications edit the same xlsx, docx, and pptx formats as their LibreOffice counterparts; other Chrome configurations and evaluations are shared with Linux tasks.Source: §B.7 · p. 30

B.8 More Task Examples

• The example collection spans app sets with distinct operational logic and capabilities, illustrating varied challenges in computer interaction.The described examples include app-specific operations and tasks that span multiple applications.Source: §B.8 · p. 31

C.2.2 Prompt for SoM Setting

• The SoM prompt requests concise Python action code using tags or carefully checked coordinates, with brief sleeps between multiple actions and no shared variables or functions from history.It also permits WAIT, FAIL, and DONE responses, asks for a short reflection on the current screenshot and prior actions, and requires returning only the requested code or special code.Source: §C.2.2 · p. 36

C.3 Accessibility Tree Filtering

• Because the original accessibility tree can exceed 1 million XML tokens, nodes are filtered by criteria including tags, visibility, and availability.Table 13 presents the keeping criteria for Ubuntu and Windows; its specific rules are not detailed in the supplied text.Source: §C.3 · p. 36, Table 13 · p. 36

C.4 Set-of-Mark Implementation Details

• SoM maps clickable buttons found via the accessibility tree to numbered bounding boxes, supplying VLMs with the marked image and a metadata table; low-quality or misleading boxes can occur.The authors identify software-ecosystem support as a factor in box quality and suggest it may contribute to SoM’s poor performance.Source: §C.4 · p. 37, Figure 14 · p. 37

C.5 Full Results of Baseline Methods

• The section compiles baseline-method breakdown results across different LLMs and VLMs for reference.The passage does not provide the breakdown values.Source: §C.5 · p. 37
• GPT-4V’s step counts and termination behavior differ across A11y Tree, Screenshot, Screenshot+A11y Tree, and Set-of-Mark settings.The step distributions are presented in Fig. 15; the passage describes them as potential clues for future work.Source: §C.5 · p. 37

• The appendix details code execution for §5.4 examples and adds error-cause analysis for improvement; Figure 15 depicts GPT-4V agents’ step-count distributions across four settings.The passage does not provide the distribution values.Source: D Examples of Qualitative Analysis · p. 37, Figure 15 · p. 38

D.1 Success and Failure Cases

• The subtitle-removal example uses a terminal command to extract the video’s first subtitle stream as subtitles.srt.The task asks to save the file in the same directory as the video; the shown command is ffmpeg -i video.mp4 -map 0:s:0 subtitles.srt.Source: §D.1 · p. 38
• The LibreOffice Writer failure example asks for a centered heading, but the shown action only moves to the heading’s beginning to select it.The excerpt does not show the subsequent alignment action or its outcome.Source: §D.1 · p. 39
• In the GIMP brightness example, repeated incorrect menu selections and duplicate actions leave the agent unable to find the right coordinates.The shown sequence cancels several dialogs, revises the assumed menu position, and repeats the attempted selection.Source: §D.1 · pp. 41–42
• The LibreOffice Calc example has the agent fill conference-city cells, including Scottsdale, Atlanta, Lake Tahoe, and Long Beach.The task requests meeting cities for three machine-learning conferences from 2013 through 2019; the excerpt shows cell-by-cell entry actions.Source: §D.1 · p. 42
• The Chrome example attempts to make Bing the default search engine through approximate menu clicks, a settings URL, and tab navigation.The excerpt includes repeated attempts to reach Settings and then navigate the search-engine options; it does not show the final outcome.Source: §D.1 · pp. 42–43

D.2 Common Errors by GPT-4V Agents

• In the Nike-store task, a coordinate error makes the agent click the favorite button instead of product categories, leaving it unable to backtrack and restart.The intended task was to find women’s jerseys priced over 60 dollars.Source: Figure 17 · p. 44
• When cookie requests or pop-up ads appear, the agent attempts to interact with the page without dismissing them, so its actions are ineffective.The example attributes the failure to unfamiliarity with the structure of pages containing overlays.Source: Figure 18 · p. 44

D.3 Discrepancies in Task Difficulty between Agent and Human

• The examples contrast human-favored document and slide edits with agent-favored command-line tasks.The human-favored tasks are removing highlighted marks and editing a slide; agent-favored examples include video trimming, resource monitoring, and force-quitting an application.Source: §D.3 · pp. 45–47
• The video-trimming example is marked as agent-favored, but its result notes that the agent did not follow the demanding instruction.The attempted command trims from 2 to 4 seconds with ffmpeg.Source: §D.3 · p. 46
• The resource-monitoring example presents an agent-favored task requiring sar to run once per second for 30 seconds and save a report on the Desktop.The shown command is `sar 1 30 > ~/Desktop/System Resources Report.txt`.Source: §D.3 · pp. 46–47
• The force-quit example presents an agent-favored task of closing frozen LibreOffice Writer from Ubuntu’s command line without restarting the computer.The shown approach types `killall soffice.bin`.Source: §D.3 · p. 47

D.4 Comparative Analysis: Claude-3 vs. GPT-4V

• Claude-3 is labeled as failing a request to find `secret.docx` and copy its path, despite searching in Files and copying the selected file.The demonstrated steps open the file manager, search for the filename, then use Ctrl+C on the selected file.Source: §D.4 · p. 48
• Claude-3 is labeled as failing a spreadsheet-to-browser task: it responds with a channel answer after describing Cell B36, while the request specifies Cell B6.The action passage says the task is complete, but its response does not match the requested cell-copy-and-search task.Source: §D.4 · p. 49
• Claude-3 is labeled as failing a document request to replace every instance of `text` with `test`.The shown actions open the replace dialog, enter the two terms, press Enter, and save.Source: §D.4 · p. 49

D.5 Analysis on SoM Effectiveness

• Without SoM, the script opens VS Code Settings through Manage, searches for Python, and edits settings.json to disable missing-import reporting.It enters `reportMissingImports: none` and saves the change.Source: §D.5 · p. 50
• With SoM, the script opens Settings through the command palette and searches for Python and Python missing imports.The sequence includes clicking the settings search fields and typing the queries.Source: §D.5 · pp. 50–51
• The SoM run ends with the note, “Can not find the right click area.”The preceding steps use the command palette and search for Python-related settings.Source: §D.5 · pp. 50–51

D.6 Analysis on SoM Effectiveness

• SoM did not excel on diverse tasks requiring expertise and flexibility, despite being effective for web tasks focused on labeled bounding boxes.Its shortened action space can hinder agent exploration and adaptability.Source: §D.6 · p. 51
• In professional software, SoM labeling errors can make element blocks difficult to change, while agents without SoM can use the a11y tree to calculate and adjust coordinates.Source: §D.6 · p. 51
• In the VS Code settings task, agents without SoM succeeded by editing settings.json, while SoM-equipped agents struggled to find and deselect the checkbox.The SoM route had a longer action path and failed because of incorrect mouse clicks.Source: Figure 21 · p. 51, §D.6 · p. 51