Source-linked AI summary
WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments
Jinchao Li, Yunxin Li, Chenrui Zhao, Zhenran Xu, Baotian Hu, Min Zhang
TL;DR
Existing GUI-agent benchmarks underrepresent the cross-application coordination required in professional workflows. WindowsWorld introduces a process-aware benchmark for such tasks and finds that the best agents achieve only a 20% success rate.
Problem
Existing GUI-agent benchmarks underrepresent cross-application coordination, despite its importance for professional workflows.
Method
WindowsWorld uses human-reviewed multi-agent generation to create 181 tasks across 17 desktop applications, organized by professional personas and task complexity.
Results
The best agents achieve only a 20% success rate on WindowsWorld.
Takeaways & Limitations
Current GUI agents remain challenged by process-aware, cross-application task completion.
Takeaways & Limitations
Intermediate scores rely on full-trajectory execution and manually reviewed checkpoints, limiting scalability for large-scale or online reinforcement learning.
Abstract
from arXiv · showhide
While GUI agents have shown impressive capabilities in common computer-use tasks such as OSWorld, current benchmarks mainly focus on isolated and single-application tasks. This overlooks a critical real-world requirement of coordinating across multiple applications to accomplish complex profession-specific workflows. To bridge this gap, we present a computer-use benchmark in cross-application workflows, named WindowsWorld, designed to systematically assess GUI Agents on complex multi-step tasks that mirror real-world professional activities. Our methodology uses a multi-agent framework steered by 16 occupations to generate four difficulty-level tasks with intermediate inspection, which are then refined by human review and executed in a simulated environment. The resulting benchmark contains 181 tasks with an average of 5.0 sub-goals across 17 common desktop applications, of which 78% are inherently multi-application. Experimental results of leading large models and agents show that: 1) All computer-use agents perform poorly on multi-application tasks (< 21% success rate), far below the performance of simple single-app tasks; 2) They largely fail at tasks requiring conditional judgment and reasoning across $\geq$ 3 applications, stalling at early sub-goals; 3) Low execution efficiency, where tasks often fail despite far exceeding human step limits. Code, benchmark data, and evaluation resources are available at github.com/HITsz-TMG/WindowsWorld.
1 Introduction
Existing GUI benchmarks underrepresent professional cross-application coordination, motivating WindowsWorld, a human-reviewed benchmark of realistic multi-application workflows with intermediate process checks. It contains 181 tasks across 17 desktop applications and evaluates performance beyond final task success.
- Motivation: Multi-application tasks remain sparsely represented in existing benchmarks, accounting for less than 28% of instances.Most benchmarks instead emphasize single-app navigation or short-horizon workflows such as OSWorld and AndroidWorld.
- Construction: WindowsWorld is constructed through human-centric multi-agent task generation followed by human review, targeting real-world professional workflows.The framework uses persona-based scenario generation, refinement to remove redundancy, and environment generation for mission files.
- Benchmark: WindowsWorld comprises 181 professional-grade tasks spanning 17 desktop applications, generated from 16 personas across 5 categories and organized into four difficulty levels.The difficulty levels progressively increase planning horizon and cross-application coordination, including infeasible tasks for evaluating goal recognition and abstention.
- Benchmark: About 78% of WindowsWorld workflows are realistic multi-application tasks involving structured information transfer and common office productivity scenarios.These workflows are intended to capture professional cross-application coordination rather than isolated application use.
- Evaluation: Intermediate checking points validate essential sub-goals and assign partial-progress scores, increasing discrimination for long-horizon and high-difficulty workflows.This approach departs from evaluation based only on final success.
- Evaluation: Extensive evaluations reveal a significant performance gap for leading GUI models and agents on non-linear workflows and process-dependent constraints compared with simple tasks.The evaluated systems include Gemini-3-Pro, GPT-5.2, and Agent S3.
2 Related Work
Prior desktop benchmarks largely target single-step accuracy or simple tasks, while newer environments expand OS coverage, scalability, and task complexity. GUI-agent evaluation has also moved beyond binary success rates toward process-level validation, motivating WindowsWorld’s flexible checkpoint-based scoring.
- Desktop OS Benchmarks: Early benchmarks primarily evaluate computer-use agents on single-step accuracy or simple tasks.Examples include MiniWoB, MiniWoB++, WebShop, Mind2Web, WebArena, and VisualWebArena.
- Desktop OS Benchmarks: OSWorld introduced scalable execution-based evaluation across Ubuntu, Windows, and macOS, while Windows Agent Arena added Windows focus and cloud-based parallelization.OSUniverse further introduced increasing task-complexity levels, from Paper to Gold, to benchmark agent dexterity and precision.
- Evaluation of GUI Agent: Most early GUI-agent benchmarks use final-state matching or success rate, but success rate can collapse outcomes into binaries that obscure specific failure modes.ProBench addressed this limitation with a Process Provider for capturing intermediate information in process-related tasks.
- Evaluation of GUI Agent: SPA-Bench and A3 validate states at the step level but may depend on rigid trajectories or unreliable LLM-based decomposition.WindowsWorld instead uses flexible checkpoint-based scoring, permitting alternative valid paths while requiring functional correctness at key process nodes.
3 WindowsWorld Benchmark
WindowsWorld is a process-centric benchmark for evaluating computer-use agents on realistic, professional workflows in controlled Windows environments. It organizes tasks by complexity and persona, constructs them through human-in-the-loop validation, and emphasizes multi-application, long-horizon execution with intermediate checkpoints.
- Benchmark Setup: WindowsWorld evaluates computer-use agents in controlled Windows virtual machines using standard GUI operations across a fixed set of productivity applications.Agents cannot access internal APIs or privileged system calls, supporting reproducibility and deployment.
- Task Design: Tasks are organized by four complexity levels—L1 single-app atomic, L2 multi-app linear, L3 dynamic reasoning, and L4 infeasible—and 16 professional personas.L3 tasks require conditional reasoning across applications, while L4 tasks test whether agents reject unachievable goals.
- Construction Pipeline: A human-in-the-loop multi-agent pipeline generates persona-grounded tasks, performs automated deduplication and validation, and applies final human quality control.The pipeline includes a 4-node refinement stage and rejects ambiguous, subjective, or infeasible tasks requiring unavailable software or services.
- Benchmark Composition: 181 tasks span 17 Windows applications, with 71.8% non-trivial L2/L3 tasks and 77.9% involving two or more applications.Tasks also include an average of 4.97 intermediate-state checkpoints, enabling fine-grained diagnosis of long-horizon failures.
- Benchmark Distinctiveness: 77.9% of WindowsWorld tasks are multi-application, compared with 9.50% for AndroidWorld and 27.4% for OSWorld; minimum action steps average 9.67 for L1, 18.13 for L2, and 27.81 for L3.These figures characterize WindowsWorld as focused on cross-application coordination and substantially longer-horizon workflows than prior Windows benchmarks.
4 Experiment
WindowsWorld evaluation shows that current GUI agents lose effectiveness as workflows become longer, more complex, and cross-application. Structured observation helps intermediate progress, but agents still struggle to convert local execution into reliable completion.
- Efficiency-Completion Gap: Gemini-3-flash (Hybrid) reaches Sint = 50.32% but only Sfinal = 20.44%, exposing a large efficiency-completion gap.Agents often complete sub-goals without synthesizing them into a successful terminal state.
- Complexity Bottlenecks: Gemini-3-flash’s Sfinal falls from 35.90% on L1 tasks to 16.67% on L3 tasks, revealing bottlenecks in cross-application coordination and state maintenance.Agents handle isolated operations better than the conditional reasoning required by professional workflows.
- Negative Constraint Handling: Best GPT-5.2 (SoM) achieves only 25% success on L4 tasks, while UiPath’s high L4 score often reflects declaring tasks incomplete.Reliable detection and reporting of infeasible instructions remains difficult for current models.
- Observation Modalities: Hybrid observation produces Gemini-3-pro’s +7.9% Sint gain, whereas raw screenshots yield Gemini-3-flash Sint = 43.5% but Claude and GPT-5.2 below 8%.Structured metadata is described as a more stable foundation, while Set-of-Marks inputs can add cognitive noise.
- Efficiency and Drift: Gemini-3-flash’s failure-success step gap widens from 4.64 to 14.14 between L1 and L3 in Hybrid, indicating increasingly inefficient, late-stage failures.Failed trajectories remain locally plausible for longer as task difficulty increases.
5 Conclusion
WindowsWorld introduces a process-aware computer-use benchmark for professional cross-application workflows, combining multi-application tasks, intermediate process checking, and automatic task construction. Experimental results show that the best agents achieve only a 20% success rate.
- Benchmark contribution: WindowsWorld is presented as the first process-aware computer-use benchmark in a cross-application environment.It targets professional-grade workflows rather than isolated application tasks.
- Benchmark contribution: 78% of the challenging benchmark tasks are professional-grade multi-application tasks with fine-grained intermediate process checking.The benchmark also incorporates an automatic task construction method.
- Experimental results: 20% is the best-agent success rate reported in the experimental results.This result underscores the difficulty of the benchmark’s cross-application workflows.
Limitations
WindowsWorld’s intermediate scoring depends on full-trajectory execution and manually reviewed checkpoints, limiting scalability for large-scale or online reinforcement learning in long-horizon tasks. The benchmark also lacks automated generalizable rewards and evaluation for Model Context Protocol tools.
- Limitations: Full-trajectory execution and manually reviewed checkpoints restrict scalability for large-scale or online reinforcement learning in long-horizon tasks.The authors identify this scoring design as a limitation for scaling evaluation or training.
- Limitations: Automated, generalizable reward formulations remain an open challenge.This limitation directly affects the use of intermediate scores for reinforcement learning.
- Limitations: WindowsWorld does not yet incorporate evaluation for Model Context Protocol tools.The authors leave MCP-tool evaluation for future expansion.
Ethical Considerations and Reproducibility … B.2 Infeasible Task
WindowsWorld documents ethical annotation practices, reproducibility commitments, vision-centric observation, multiple action spaces, and illustrative feasible and infeasible task cases. Its L4 examples omit intermediate checkpoints, unlike feasible L1–L3 cases.
- Ethical Considerations and Reproducibility: Four postgraduate researchers filtered instructions and validated intermediate checkpoints, receiving 1.5 USD per task.The instructions were LLM-assisted and manually reviewed, with no personally identifiable information or harmful content.
- Ethical Considerations and Reproducibility: The complete research framework, task instructions, evaluation environment, and process-aware metrics will be open-sourced on GitHub upon publication.The release is intended to support community growth, reproducibility, and further GUI-agent research.
- A.1 Observation Modalities: WindowsWorld focuses on vision-centric modalities rather than text-only A11y settings because structural metadata can be incomplete or insufficient for visually dense professional interfaces.The benchmark evaluates agents under distinct visual-input settings, including raw screenshots requiring semantic reasoning and coordinate grounding.
- A.2 Action Space: WindowsWorld provides free-form action spaces for agent interaction.The supplied passage introduces the action-space options but truncates their detailed description.
- A.2 Action Space: The benchmark supports pyautogui and computer_13 from OSWorld, with pyautogui used mainly in experiments.Detailed computer_13 actions are displayed in Table 5.
- B.1 Feasible Task: Feasible L1–L3 task cases show natural-language instructions, categories, involved applications, final criteria, and intermediate checkpoints.The benchmark also provides a visual cross-application interaction sample in Figure 10.
- B.1 Feasible Task: Figure 6 presents an L1 task in JSON, including its instruction, involved applications, and intermediate and final evaluation criteria.Figure 7 presents an example WindowsWorld L2 task.
- B.2 Infeasible Task: Infeasible L4 cases retain the feasible-task content but omit intermediate checkpoints.Figure 9 provides an example of this infeasible-task format.
B.3 Error Case
The error cases expose failures caused by incompatible input handling and unstable application-state transitions. These failures motivate explicit verification of GUI state during cross-application workflows.
- B.3 Error Case: Chinese IME interception misinterprets pyautogui keystrokes as Pinyin, triggering unintended shortcuts, file-access errors, and corrupted input strings.pyautogui.write simulates discrete keystrokes rather than directly injecting strings, making non-ASCII input especially vulnerable.
- B.3 Error Case: Unstable window focus causes intended Excel-to-Word transfers to paste content back into Excel when the agent fails to verify the active GUI context.The failure occurs during application switching and clipboard execution.
- B.3 Error Case: These application-workflow failures underscore the need for explicit state-verification mechanisms during task transitions.
C Additional Experimental Results · C.1 Reliability and Error Analysis of the VLM Judge
The VLM-as-judge protocol was validated against human annotations on stratified execution tasks covering 518 intermediate checkpoints. Agreement was strong, with Pearson correlations of 0.9108 for Sint and 0.8316 for Sfinal, and Cohen’s κ reaching 0.8.
- C.1 Reliability and Error Analysis of the VLM Judge: 100 stratified tasks were used to validate the automated VLM-as-judge protocol.The evaluation included L1, L2, and L3 tasks.
- C.1 Reliability and Error Analysis of the VLM Judge: 24 L1, 50 L2, and 26 L3 tasks formed the stratified validation set.The task distribution totaled 100 tasks.
- C.1 Reliability and Error Analysis of the VLM Judge: 518 intermediate checkpoints were evaluated across the stratified tasks.The checkpoints came from execution trajectories.
- C.1 Reliability and Error Analysis of the VLM Judge: Two human annotators manually assessed whether each intermediate checkpoint was satisfied from the execution trajectory.Their consensus was compared with the automated judge.
- C.1 Reliability and Error Analysis of the VLM Judge: Qwen3-VL-Plus served as the VLM judge whose decisions were compared with human consensus.The comparison evaluated the reliability of the automated judging protocol.
- C.1 Reliability and Error Analysis of the VLM Judge: 0.9108 for Sint and 0.8316 for Sfinal were the Pearson correlations between the VLM judge and human consensus.These correlations indicate strong agreement under the reported validation protocol.
C.2 Persona-Wise Results … D.3 Prompt for Environment Generator
WindowsWorld reveals substantial persona-dependent variation in agent difficulty, with failures often emerging during late-stage coordination rather than initial execution. Its human-in-the-loop framework uses dedicated prompts for task generation, refinement, evaluation assertion design, and environment-file creation.
- C.2 Persona-Wise Results: Persona-stratified results expose substantial heterogeneity in professional-workflow difficulty beyond aggregate scores.Table 8 evaluates intermediate checkpoint score (S_int) and final completion score (S_final) across five persona categories, input modalities, and agent frameworks.
- C.2 Persona-Wise Results: Large progress–completion gaps show that agents often make intermediate progress but fail to reach correct terminal states.These failures frequently occur after partial task execution rather than during initial interaction.
- C.2 Persona-Wise Results: Productivity-oriented personas are especially difficult because document-centric and visually driven tasks require global consistency across multiple steps.Agents often complete local operations but struggle with late-stage coordination and constraint satisfaction, producing low final completion rates.
- C.2 Persona-Wise Results: Technical/IT personas show tighter coupling between intermediate progress and final success in workflows dominated by code editing and command-line interactions.Explicit goals and deterministic state transitions make these tasks comparatively easier for current models.
- C.2 Persona-Wise Results: Intermediate checkpoint scores provide essential diagnostic signal because WindowsWorld difficulty varies by persona and failures frequently arise during late-stage coordination.Checkpoint evaluation complements final-state evaluation when analyzing heterogeneous professional workflows.
- D.1 Prompt for Generator: The Generator agent uses a system prompt combined with persona settings and is input into the DeepSeek v3.2 model.The prompt is presented in Figure 13 within the human-in-the-loop multi-agent framework.
- D.2 Prompt for Refiner: The Refiner agent is specified through system prompts, including a Dependency Reasoner that converts action-based pre-conditions into verifiable state descriptions.A separate Metric Refiner generates programmable assertions for automated evaluation.
- D.3 Prompt for Environment Generator: The environment-generation prompts define success criteria, intermediate checkpoints, and task-aligned files whose content must strictly satisfy instructions.The unified generator produces executable, verifiable, and consistent text files, spreadsheets, and documents.