PaperlayerView exact revision on arXiv

Source-linked AI summary

Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale

Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, Zack Hui

arXiv:2409.08264v2cs.AI

TL;DR

Computer-agent benchmarks often provide limited modality or domain coverage and can take days to evaluate sequentially. This paper introduces Windows Agent Arena, a reproducible real-Windows benchmark with scalable Azure parallelization, and evaluates Navi within it. Navi achieves 19.5% success in Windows tasks versus 74.5% for an unassisted human.

  • Problem

    Existing benchmarks often cover restricted modalities or domains, while sequential multi-step evaluations can take on the order of days.

  • Method

    The paper builds a reproducible Windows benchmark with 154 diverse tasks, adapting about two thirds from OSWorld and adding Windows-specific tasks and infrastructure.

  • Results

    19.5% success rate is achieved by Navi's best Windows configuration, compared with 74.5% human performance.

  • Takeaways & Limitations

    Windows Agent Arena provides a realistic framework for testing multimodal agents and supports faster experimentation and data generation through scalable evaluation.

  • Takeaways & Limitations

    Human-in-the-loop evaluation could benefit agent performance but would require human involvement and be harder to benchmark.

Abstract

from arXiv · show

Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena. Webpage: https://microsoft.github.io/WindowsAgentArena Code: https://github.com/microsoft/WindowsAgentArena

1 INTRODUCTION

Windows Agent Arena addresses limitations in realistic computer-agent evaluation by providing a reproducible Windows environment with diverse tasks and scalable parallel execution. The paper also introduces Navi as a multimodal agent benchmarked in this environment.

  • Benchmark motivation: Windows Agent Arena targets realistic workflows by embedding evaluation in a real Windows OS spanning applications and web domains.It builds on OSWorld while focusing specifically on Windows.
  • Benchmark construction: 154 diverse, realistic multi-step tasks cover multiple Windows applications and web domains.The benchmark is designed to test planning, screen understanding, and tool usage.
  • Evaluation scalability: Azure task parallelization reduces full benchmark evaluation to as little as 20 minutes, compared with hours or more for serial computation.The infrastructure uses secure Docker containers running in Azure virtual machines.
  • Agent baseline: Navi is a new multimodal Windows agent evaluated across configurations and on the web-based Mind2Web benchmark.Its best configuration combines Set-of-Marks prompting, the system accessibility tree, and pixel-based element detectors.
  • Research contributions: The code, benchmark, and models are released open source to support agent development, faster experimentation, and data generation at scale.The benchmark is intended to be continuously updated with new tasks and capabilities.

2 PRIOR WORK

Prior agent benchmarks span static datasets, interactive environments, and domain-specific settings, but realistic cross-domain workflows and scalable evaluation remain limited. Windows Agent Arena addresses the latter through native cloud parallelization.

  • Benchmark comparison: Benchmark comparisons distinguish whether an environment exists and whether it is natively scalable or parallelizable.The table notes that OSWorld can be parallelized within one machine but has limited scalability.
  • Static benchmarks: Static benchmarks evaluate human-collected trajectories but cannot capture the full range of agent trajectories for solving tasks.Examples include Mind2Web, WebLinx, PixelHelp, MoTiF, AiTW, and OmniAct.
  • Interactive benchmarks: Interactive benchmarks provide task-completion reward signals and more dynamic evaluation, but environments differ in how openly agents can act.GAIA permits web and tool use without specifying a closed operating environment.
  • Evaluation scalability: Sequential evaluation of hundreds of multi-step tasks can take one or more days on a local machine, while OSWorld's single-machine parallelization remains resource-limited.Windows Agent Arena instead provides native cloud parallelization that can scale to the number of benchmark tasks.

3 THE WINDOWSAGENTARENA ENVIRONMENT

WINDOWSAGENTARENA formalizes Windows computer interaction as a partially observable task environment with multimodal observations, executable actions, and execution-based rewards. Its 154 tasks span representative Windows applications, while containerized Azure deployment supports local testing and parallelized evaluation.

  • Environment formulation: WINDOWSAGENTARENA models agent behavior as a POMDP with OS state, partial observations, executable actions, transitions, and task rewards.Observations may include instructions, clipboard content, window metadata, screenshots, DOM or UI Automation trees, and pixel-based annotations; actions support Python or Computer-class functions.
  • Task curation: Task completion is evaluated from the final system or application state using scripts that inspect settings, files, or dynamically assessed web results.Rewards can be binary, graded by similarity, or non-zero for correctly predicting failure on infeasible tasks.
  • Task curation: Approximately two thirds of tasks are adapted from OSWorld’s Linux configurations, while the remaining third were created for Windows-specific applications and activities.Adaptation includes Windows paths, PowerShell commands, browser communication, task conditions, evaluators, and instructions.
  • Task curation: 74.5% human success provides a reference point, with Windows Utilities highest at 91.7% and VLC Player lowest at 42.8%.These results are reported alongside analyses of task steps and perceived difficulty.
  • Deployment infrastructure: Dockerized Windows 11 virtual machines support local execution and secure Azure parallelization, reducing evaluation time by distributing tasks across workers.The design uses a client scheduler, VM bridge, Azure compute instances, snapshot storage, log aggregation, and isolated communication ports.

4 BASELINE RESULTS AND ANALYSIS

Navi combines multimodal screen representations with reasoning and action tools to operate in Windows Agent Arena. Its strongest configuration remains well below human performance, while performance varies by interface type, visual grounding quality, and base model.

  • Navi implementation: Navi receives window metadata, screen representations, task context, action history, clipboard content, and memory before selecting executable actions.Screen processing variants include UIA and DOM trees, OCR, icon and image detectors, and OmniParser.
  • Baseline results: 19.5% success rate is achieved by UIA combined with OmniParser and GPT-4V-1106, versus 74.5% for humans.Performance is stronger on clear text-dominant interfaces and weaker on tasks relying heavily on keyboard shortcuts and icons.
  • Visual prompting: 57% performance boost is obtained by adding high-quality UIA markers to OmniParser, compared with 52% for open-source models and 15% for proprietary pixel models.UIA querying can take from a few seconds to several minutes depending on screen complexity.
  • Failure analysis: Visual-language misalignment causes failures when textual actions do not match selected visual IDs, while icon captioning may help connect IDs with textual descriptions.The paper illustrates an action describing the red color while selecting the yellow palette.
  • Base-model comparison: GPT-4V-1106 outperforms Phi3-V, GPT-4o-mini, and GPT-4o in most categories, exceeding GPT-4o by more than double with UIA+OmniParser.Phi3-V tends to hallucinate during multi-step planning or with long input contexts.
  • Mind2Web results: Navi achieves state-of-the-art performance on Mind2Web relative to the original SeeAct agent, with image and DOM Set-of-Marks prompting performing best.This evaluation tests Navi in a separate web-based setting.

5 DISCUSSION

The discussion considers human assistance, specialization, reinforcement learning, action-space design, and responsible deployment as directions for generalist OS agents. It also identifies evaluation and data challenges associated with these directions.

  • Full autonomy vs. human-in-the-loop: Human-in-the-loop agents could seek clarification, additional information, or intervention, but evaluating them would likely require human involvement.The proposed assistance comes at a cost and would make benchmarking harder.
  • Generalist vs. specialized agents: Specialized fine-tuned subsystems could be organized as an agent of agents for specific domains.This is proposed as a research direction alongside generalist OS agents.
  • Imitation vs. reinforcement learning: Reinforcement learning could leverage imperfect actions when reward objectives are well-defined, including task completion in Windows Agent Arena.The benchmark is proposed as a source of training data at scale.
  • Action-space design: Stricter predefined action functions or skill libraries could improve execution precision, shifting remaining errors toward planning and decision-making.The paper contrasts this with fully autonomous agents using free-form actions.
  • Safety and security: Responsible OS agents should let users understand, direct, diagnose, and override actions while respecting privacy and fairness.The discussion frames these capabilities as part of ethical and responsible AI use.

6 CONCLUSION

WINDOWSAGENTARENA is an open-source, reproducible environment for evaluating multimodal agents in a real Windows OS. It provides diverse tasks, scalable Azure evaluation, and Navi baselines to support future agent research.

  • WINDOWSAGENTARENA provides a reproducible environment for evaluating multimodal agents within a real Windows OS.
  • 154 diverse tasks test abilities including planning, screen understanding, and tool usage in a realistic agent-evaluation framework.
  • Azure parallelization significantly reduces the evaluation time for full benchmark runs.
  • Navi is evaluated across different configurations, while the benchmark’s quantitative and qualitative analyses provide insights for future agent development.
  • The released code, benchmark, and baseline models are intended to facilitate research on more efficient and capable computer-control agents.

A.1 OBSERVATION SPACE

The observation space combines window and clipboard information, accessibility data, and current and previous screenshots. Agent configurations can annotate screenshots with Set-of-Marks, while actions use pyautogui or a custom computer wrapper.

  • The observation space includes foreground and background window titles extracted with pygetwindow.
  • Clipboard observations contain copied text or a VLM-generated description when the clipboard contains an image.
  • The Windows UI Automation tree supplies element names, types, and positions for some configurations to create screenshot Set-of-Marks annotations.
  • Current and previous screenshots are captured as 1440 × 900 × 3 RGB arrays to provide visual context and completion cues.
  • Set-of-Marks annotations may use proprietary pixel-based models, open-sourced pixel-based models, UIA parsing, or Omniparser with text and icons.
  • The action space combines pyautogui with a custom Computer wrapper covering GUI, keyboard, and OS functions.

A.3 REWARD EVALUATION BASED ON AGENT EXECUTION

Task evaluation queries the virtual machine’s post-execution device state against its initial state. Evaluators return either binary or continuous non-negative rewards according to task-specific outcome rules.

  • Evaluators compare the post-execution virtual-machine device state with the initial state to determine task accomplishment.
  • Binary rewards are R ∈ {0, 1} and indicate whether a strict task-outcome rule is satisfied.
  • Continuous rewards provide non-binary non-negative scores when task outcomes are evaluated on a graded basis.

A.4 TASK & PROGRAM SELECTION

The benchmark contains 154 realistic Windows tasks selected for diversity across contexts, difficulty levels, and commonly used applications. Tasks span document editing, spreadsheets, browsers, media players, file management, and development tools.

  • 154 tasks represent diverse contexts and difficulty levels while reflecting common Windows user activities and applications.
  • Writer: Writer tasks include document formatting, PDF export, sharing, and mixed text alignment operations.
  • Calc: Calc tasks cover spreadsheet renaming, age calculation, formula filling, concatenation, and revenue summarization.
  • Browsers: Chrome and Edge tasks test privacy settings, browsing-data management, navigation, search configuration, shortcuts, and bookmarks.
  • Applications: VLC, File Explorer, and VSCode tasks cover player customization, file organization and sharing, shortcuts, extensions, and editor settings.

A.5 WINDOWSAGENTARENA HUMAN EVALUATION

The human evaluation measures feasibility recognition, task completion, steps, and perceived difficulty on Windows Agent Arena. Success is strongly inversely associated with perceived difficulty, while step counts vary across task domains.

  • Evaluation protocol: Human evaluation covered feasible and infeasible tasks, counting recognition of infeasibility as success and testing completion without external assistance.The participant was a casual Windows user who spent approximately 1.5 hours across five sittings.
  • Evaluation protocol: Participants recorded completion steps for successful tasks and rated each task’s difficulty from 1 to 3.Difficulty ratings were subjective, and open-ended tasks allowed multiple valid procedures.
  • Human results: 1.9 was the human participant’s overall average perceived task difficulty.VLC Player was most difficult on average, followed by VS Code and LibreOffice Calc.
  • Human results: LibreOffice Calc and Windows Utilities required the most human steps on average while also achieving the highest average human success rates.This demonstrates that more steps did not correspond uniformly to lower success across domains.
  • Metric relationships: Average success rate was strongly and inversely correlated with perceived difficulty, whereas step-count correlations were weaker and domain-dependent.The correlations use Spearman’s ρ because the data are discrete and relationships are nonlinear across metrics.

A.6 TASK DEFINITION & CONFIGURATION

Windows Agent Arena tasks are specified by natural-language instructions, initial-state configurations, and automatic evaluators. The configuration prepares the environment, while evaluation compares the resulting virtual-machine state with ground-truth rules.

  • Task structure: Each task JSON contains an identifier, natural-language instruction, initial configuration, evaluator, and result specification.These five components are represented as colored JSON keys in Figure 12.
  • Task structure: The “id” key uniquely identifies each task, while “instruction” states the task in natural language.The example asks the agent to modify the folder used to store recordings to the Desktop.
  • Initial configuration: The “config” key establishes the starting context through setup actions such as launching applications, invoking PowerShell commands, or downloading files.The example launches VLC and executes a pyautogui click after launch.
  • Evaluation: The evaluator checks the post-trajectory virtual-machine state against expected ground-truth values or rules.The example evaluator calls a VLC recordings-folder function to determine whether the task outcome matches the target.

A.7 WINDOWSAGENTARENA: AZURE PARALLELIZATION DETAILS

Windows Agent Arena accelerates evaluation by running Windows environments in parallel Azure infrastructure rather than trying to speed up asynchronous OS simulation. Most configurations complete a full evaluation in under 20 minutes.

  • Infrastructure requirements: Azure deployment requires a virtual-machine type supporting nested virtualization with KVM.The experiments rely primarily on the Standard D8 v3 machine, and listed VM costs were reported as of August 2024.
  • Evaluation speed: Most Navi configurations complete a full Windows Agent Arena evaluation in less than 20 minutes.Run times vary with job count, compute instances, and tasks per job; the Omniparser configuration is an exception.

B WINDOWSAGENTARENA RESULTS DETAILS

The benchmark includes examples of successful and unsuccessful Windows Agent Arena trajectories. The unsuccessful example shows that correct task understanding can still fail when a required graphical control is not recognized as manipulable.

  • Successful outcomes: Figures 13–15 illustrate successful Windows Agent Arena outcomes by showing the agent’s action trajectory at each step.The examples are edited for clarity.
  • Unsuccessful outcome: The unsuccessful example fails because the font-size slider is not recognized as a manipulable object.The agent repeatedly identifies the required increase but cannot move the slider to the right.

C MIND2WEB RESULTS DETAILS

For Mind2Web, Navi receives multimodal screen information and must reason about the current objective, anticipate future steps, and select executable actions. Its prompt combines screenshots, annotated elements, textual descriptions, action history, memory, and code-based interaction instructions.

  • Mind2Web evaluation gives the agent a task description, action sequence, screenshot, and screen-parsing entities.
  • The prompt combines raw and annotated screens with OCR text and candidate interactive elements identified by unique IDs, types, content, and locations.Image and icon elements are annotated with colored bounding boxes, while button text is not annotated.
  • The agent executes one action per step through Python or computer-module functions, while using previous actions and textual memory to maintain context.Available operations include keyboard input, clipboard manipulation, program launching, and switching application windows.
  • Navi must relate screen content to the current objective, predict future steps, and formulate the next action with a rationale.
Loading 2409.08264v2…