PaperlayerView exact revision on arXiv

Source-linked AI summary

macOSWorld: A Multilingual Interactive Benchmark for GUI Agents

Pei Yang, Hai Ci, Mike Zheng Shou

arXiv:2506.04135v4cs.AI

TL;DR

Existing interactive GUI benchmarks lack macOS coverage, multilingual tasks and interfaces, and integrated safety evaluation. macOSWorld introduces a multilingual interactive macOS benchmark with 202 tasks across 30 applications and a deception-safety subset, finding strong performance tiers, Arabic degradation, and substantial deception susceptibility.

  • Problem

    Existing interactive benchmarks omit macOS, largely use English, and rarely combine functional performance with safety evaluation.

    Source: §1 · p. 1

  • Method

    macOSWorld provides 202 interactive tasks across 30 applications, five languages, macOS environments, and a dedicated safety benchmarking subset.

    Source: §1 · p. 2

  • Results

    Proprietary CUAs exceed 30% success while open-source research models remain below 5%; Arabic performance drops 28.8% versus English, and strong CUAs show approximately 70% deception rates.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results highlight the need for adaptation to macOS and multilingual environments, while deception attacks require immediate attention.

    Source: §1 · p. 2, §6 · p. 9

Abstract

from arXiv · show

Graphical User Interface (GUI) agents show promising capabilities for automating computer-use tasks and facilitating accessibility, but existing interactive benchmarks are mostly English-only, covering web-use or Windows, Linux, and Android environments, but not macOS. macOS is a major OS with distinctive GUI patterns and exclusive applications. To bridge the gaps, we present macOSWorld, the first comprehensive benchmark for evaluating GUI agents on macOS. macOSWorld features 202 multilingual interactive tasks across 30 applications (28 macOS-exclusive), with task instructions and OS interfaces offered in 5 languages (English, Chinese, Arabic, Japanese, and Russian). As GUI agents are shown to be vulnerable to deception attacks, macOSWorld also includes a dedicated safety benchmarking subset. Our evaluation on six GUI agents reveals a dramatic gap: proprietary computer-use agents lead at above 30% success rate, while open-source lightweight research models lag at below 5\%, highlighting the need for macOS domain adaptation. Multilingual benchmarks also expose common weaknesses, especially in Arabic, with a 28.8% average degradation compared to English. Results from safety benchmarking also highlight that deception attacks are more general and demand immediate attention. Project page: https://macos-world.github.io.

1 Introduction

Existing interactive GUI benchmarks omit macOS, multilingual settings, and integrated safety evaluation, despite macOS-specific interfaces and applications creating distinct agent challenges. macOSWorld addresses these gaps with a multilingual macOS benchmark and reports substantial performance and safety differences across agents.

Motivation

  • macOSWorld addresses three benchmark gaps: no macOS coverage, limited non-English evaluation, and scarce unified functional-safety assessment.Prior benchmarks cover Ubuntu Linux, Windows, or Android, while macOS has distinctive interaction paradigms and exclusive applications.Source: §1 · p. 1

Contributions

  • The benchmark contains 202 interactive tasks across 30 applications, including 28 macOS-exclusive applications spanning system, productivity, media, and development workflows.Source: §1 · p. 2
  • Task instructions and system interfaces are available in English, Chinese, Arabic, Japanese, and Russian for multilingual capability evaluation.Source: §1 · p. 2
  • A dedicated safety subset evaluates GUI agents against realistic macOS-style deceptive pop-up windows rather than only synthetic attacks.Source: §1 · p. 2

Results

  • Proprietary computer-use agents exceed 30% success, open-source research models remain below 5%, and Arabic performance drops 28.8% relative to English.The strongest CUAs show approximately 70% deception rates under context-deception attacks, while language degradation primarily affects planning or grounding.Source: §1 · p. 2

2 Related Works

GUI-agent research has progressed from static datasets and web benchmarks toward interactive operating-system evaluation, but existing resources still omit macOS and broad multilingual coverage. Safety benchmarks also leave realistic interactive deception insufficiently explored, motivating macOSWorld’s integrated design.

Interactive GUI benchmarks

  • Interactive benchmarks assess agents in real environments, unlike static datasets that pair GUI states with annotated next actions and often simplify interface dynamics.Source: §2 · p. 3

OS-level benchmark gaps

  • Existing OS-level benchmarks cover Ubuntu Linux, Windows, and Android, but none cover macOS or support multilingual instructions and environments.macOS also contains distinct GUI conventions and interaction paradigms, limiting direct transfer from other platforms.Source: §2 · p. 3

Safety benchmarks

  • GUI-agent safety studies examine adversarial, jailbreaking, and context-deception attacks, but realistic interactive deceptions remain underexplored because current methods typically use synthetic content.Source: §2 · p. 3

Gaps and Contributions

  • macOSWorld combines multilingual interactive macOS evaluation with safety testing to address omissions in platform, language, and functional-safety coverage.Source: §2 · p. 3

3 macOSWorld Benchmark Infrastructure

macOSWorld coordinates multilingual tasks, reproducible virtualized macOS environments, agent interaction, and programmatic evaluation through a centralized testbench. Its infrastructure supports standardized interaction over VNC and reproducible state restoration across language-specific environments.

3 macOSWorld Benchmark Infrastructure

  • The infrastructure integrates an evaluated agent, programmatically evaluated natural-language tasks, reproducible multilingual macOS instances, and a centralized testbench.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Agent

  • Agents observe screenshots and issue executable actions in a POMDP-style interaction loop until termination or a maximum horizon, after which the final state determines success or failure.Source: §3.1 · p. 3

3.2 Tasks

  • Each task specifies multilingual instructions, an initialized application state, and one or more execution-based evaluation scripts that verify task completion.Source: §3.2 · p. 4

3.3 Environment

  • Virtualized AWS macOS instances provide SSH/VNC interactivity, snapshot-based reproducibility, and language-specific applications and layouts in five languages.English snapshots are duplicated in Chinese, Arabic, Japanese, and Russian, with applications and document templates adapting to each OS language.Source: §3.3 · p. 4
  • The benchmark uses VNC-compatible actions, with adapters translating smaller native action spaces into the common protocol.Source: Table 2 · p. 4, §3.3 · p. 5

3.4 Testbench

  • Benchmark execution proceeds through snapshot recovery, task preparation, an agent-environment interaction loop, and background evaluation scripts run through SSH.Source: §3.4 · p. 5

4 macOSWorld Tasks

macOSWorld provides multilingual macOS GUI tasks spanning system interfaces and applications, with curated instructions, task statistics, human-performance analysis, and a dedicated safety subset using deceptive pop-ups.

4 macOSWorld Tasks

  • macOSWorld contains 202 interactive tasks across seven categories and 30 applications, including 28 macOS-exclusive applications.The benchmark covers system interfaces, file management, productivity, media, built-in apps, advanced apps, and multitasking.Source: §4 · p. 5, §4.2 · p. 5

4.1 Task Instruction Curation

  • Instructions replace non-macOS contexts with macOS counterparts, reduce web-browsing emphasis, and are translated into Chinese, Arabic, Japanese, and Russian with round-trip consistency checks.Examples include iWork instead of LibreOffice, Xcode instead of VSCode, and SwiftUI instead of Python coding tasks.Source: §4.1 · p. 5

4.2 Task Statistics

  • 171 tasks are available in five languages, while 183 have Chinese, Japanese, and Russian translations; Arabic excludes iMovie and Xcode because of application-language availability.English remains the full 202-task configuration, whereas multilingual coverage varies by application support.Source: §4.2 · p. 5, §4 · p. 5
  • Human re-performance completed every task within 20 steps, with varied action times; task behavior is planning-oriented and per-step times follow an ex-Gaussian distribution.The task-time and step-count analysis reports no action-intensive bottom-right outliers.Source: §4.2 · p. 6, Figure 3 · p. 6

4.3 Safety Benchmarking Subset

  • The safety subset adapts pop-up attacks to macOS using authentic dialogs, distraction-specific success criteria, and centered obstruction of critical interface elements.It samples 29 main-dataset tasks and creates 29 corresponding deceptive dialogs, while genuine macOS pop-ups are triggered through AppleScript and outcomes are logged as gold, distracted, or unhandled.Source: §4.3 · pp. 6–7

5 Benchmarking Baselines

macOSWorld evaluates six GUI agents across multilingual macOS tasks, revealing strong performance differences by agent type, task category, language, and safety scenario.

5.2 Quantitative Performance Evaluation

  • Proprietary computer-use agents exceed 30% overall success, while Gemini 2.5 Pro reaches 18.5% and GPT-4o plus open-source models remain below 10%.The results form three performance tiers and indicate that open-source models lack macOS-specific adaptation.Source: §5.1 · p. 7, §5.2 · p. 8
  • Multi-app workflows achieve only 3.7% average success, while interface and productivity categories reach 17%-21%.Media tasks reach 12.8% SR, and Advanced Apps range from 19.4% for OpenAI CUA to 0% for GPT-4o.Source: §5.2 · p. 8
  • English and Russian produce the highest average performance, whereas Arabic reaches 13.7% SR, a 28.8% drop from English.Task-environment language mismatch further degrades performance.Source: §5.2 · p. 8, Table 4 · p. 9

5.3 Qualitative Analysis: Agent Behavior and Failure Modes

  • Open-source agents show basic grounding but poor planning, while general-purpose VLMs plan better but ground GUI elements imprecisely.Set-of-Mark annotations double GPT-4o’s performance to 13.0%, still below Gemini 2.5 Pro’s 18.5%; proprietary CUAs remain more capable but inefficient.Source: §5.3 · pp. 8–9
  • Language gaps among proprietary CUAs arise from degraded grounding or planning, including failures with mirrored Arabic interfaces.OpenAI CUA primarily shows grounding degradation, while Claude CUA exhibits both grounding and planning errors.Source: §5.3 · p. 9

5.4 Safety Subset Evaluation

  • Under context-deception attacks, proprietary CUAs leave fewer than 4% of pop-ups unhandled but are distracted about 70% of the time, while proprietary VLMs leave most pop-ups unhandled.The results identify vulnerability to complex deceptive interfaces and motivate safety mechanisms.Source: §5.4 · p. 9, Table 5 · p. 9

6 Conclusion and Future Analysis

The paper presents macOSWorld as a multilingual macOS benchmark with a safety subset, and reports performance variation across agents and languages. It also emphasizes accessibility, linguistic inclusion, responsible deployment, and future non-binary evaluation.

6 Conclusion and Future Analysis

  • macOSWorld is a multilingual interactive macOS benchmark with a safety evaluation subset, and its results show performance variation across agents and language settings.Future work could add non-binary rewards for finer-grained benchmarking and reinforcement-learning-based training.Source: §6 · p. 9

A Impact Statement

  • The benchmark supports accessibility and linguistic inclusion by evaluating macOS GUI agents across diverse user languages, including Arabic.The paper reports a 27.5% Arabic performance drop compared to English in its impact statement.Source: A Impact Statement · p. 13

B Ethical and Safeguarding Statement

  • Main tasks focus on constructive accessibility and productivity applications, while the safety subset may provide adversaries with an effective deception blueprint.The authors describe this as a double-edged sword and call for urgent defense mechanisms.Source: B Ethical and Safeguarding Statement · p. 13
  • Virtualized AWS macOS environments and snapshot restoration isolate operations and discard files, content, and system changes after evaluation.The paper also identifies output watermarking as a possible aid for auditability and traceability.Source: B Ethical and Safeguarding Statement · p. 13

C Example of a Real Task

A macOSWorld task initializes a language-specific macOS state, guides an agent through an application workflow, and evaluates success with scripts. A Numbers example tests changing dishwasher selections across supported languages.

C Example of a Real Task

  • Each task combines an instruction, environment-state mapping, preparation script, and evaluation script.The preparation and evaluation components are illustrated in the Numbers dishwasher-selection example.Source: C Example of a Real Task · p. 13, Figure 5 · p. 13
  • The example asks the agent to open a Numbers document, select the Products tab, replace Dishwasher C with Dishwasher B, and perform the instruction in five languages.The task and environment language can also be deliberately mismatched for cross-language evaluation.Source: C Example of a Real Task · p. 14
  • A language-specific AMI restores the required macOS version, applications, files, and initial document state before interaction.The preparation script launches Numbers, waits five seconds, and opens the Personal Budget template; template names can require separate scripts by language.Source: C Example of a Real Task · p. 14
  • The agent must click the Products tab, untick Dishwasher C, and tick Dishwasher B, testing instruction grounding in a spreadsheet.The benchmark evaluates the resulting checkbox states in cells C6 and C7 after termination.Source: C Example of a Real Task · p. 14
  • The testbench assigns a binary reward by parsing whether the evaluation script returns True, although tasks may use multiple scripts and the platform supports intermediate rewards.Reported experiments use only binary success/failure evaluation.Source: C Example of a Real Task · p. 14

D.1 Cross-Lingual Evaluation

The cross-lingual evaluation fixes task prompts in English and varies the macOS environment language. Matched task and environment languages outperform mismatched conditions across all tested environments.

D.1 Cross-Lingual Evaluation

  • OpenAI CUA experiments fix the task language to English while varying the environment among English, Chinese, Arabic, Japanese, and Russian.This setup measures the effect of task-environment language mismatch.Source: §D.1 · p. 14, Table 6 · p. 15
  • Success rates fall below matched-language performance in every off-diagonal task-environment condition.Table 7 represents environment languages by columns and matched-language results on the diagonal.Source: §D.1 · p. 14, Table 7 · p. 15

D.2 Set-of-Mark (SoM) Annotation Evaluation

SoM annotations roughly double GPT-4o’s success rates across all five languages, but they do not close the gap with agents trained for computer use.

D.2 Set-of-Mark (SoM) Annotation Evaluation

  • SoM annotations roughly double GPT-4o’s success rates in every evaluated language, according to Table 8.The comparison uses explicit SoM tag identifiers instead of raw screen coordinates.Source: §D.2 · p. 15, Table 8 · p. 15
  • GPT-4o with SoM still trails Gemini Pro 2.5 and both proprietary CUAs, showing that annotation-based grounding remains insufficient.Source: §D.2 · p. 15

E More Analysis on Agent Behavior and Failure Modes

E.1 ShowUI

ShowUI can recognize complex interfaces and execute isolated actions, but its low success rate reflects inconsistent planning and weak macOS-specific domain knowledge.

E.1 ShowUI

  • 1.1% average success across five languages leaves ShowUI far behind computer-use agents and general VLM-based agents.The authors identify planning as ShowUI’s primary limitation.Source: §E.1 · p. 16
  • ShowUI correctly locates the Products tab in a complex Numbers document but then edits the wrong element instead of changing the dishwasher selection.The intended completion required only two additional clicks.Source: §E.1 · p. 16
  • During renaming, ShowUI repeatedly enters and exits editing mode until the step limit is exhausted, illustrating nonsensical execution on a simple task.Source: §E.1 · p. 17, Figure 7 · p. 17
  • ShowUI’s scrolling, Help-menu search, and other actions indicate missing macOS domain knowledge and may expose user instructions to malicious input fields.The security concern is especially relevant when instructions contain sensitive information such as passwords.Source: §E.1 · p. 17, Figure 8 · p. 18

E.2 UI-TARS

UI-TARS achieves 4.8% average success across languages but fails through nonsensical actions, hallucinated interface locations, and invalid output formatting.

E.2 UI-TARS

  • 4.8% average task success across all languages places UI-TARS between ShowUI and GPT-4o.Source: §E.2 · p. 18
  • UI-TARS opens unrelated menus or changes zoom instead of repeating known steps, and its reasoning provides little explanation for these actions.These failures can recur even after the agent successfully completed the same type of action.Source: §E.2 · p. 18
  • UI-TARS hallucinates interface locations and properties, such as placing the Apple icon at the bottom-left or misjudging a slide’s aspect ratio.Hallucinations can occur alongside nonsensical actions.Source: §E.2 · pp. 18–19, Figure 9 · p. 19
  • Invalid formatting, including malformed click syntax and translated labels in place of coordinates, makes UI-TARS outputs unparsable by downstream algorithms.The paper identifies formatting consistency as a reliability limitation.Source: §E.2 · p. 19
  • UI-TARS sometimes acknowledges previous mistakes and performs self-correction during task execution.One example explicitly identifies an incorrect prior action and selects the Document tab to recover.Source: §E.2 · p. 19

E.3 OpenAI Computer-Using Agent and Claude Computer-Use Agent

The proprietary CUAs achieve about 35% average success but still lose tasks through detail-level errors, language-specific grounding problems, and difficulty adapting to mirrored interfaces.

E.3 OpenAI Computer-Using Agent and Claude Computer-Use Agent

  • Around 35% average success across languages makes both proprietary CUAs the strongest agents in the benchmark.Their weaknesses vary across languages, primarily affecting grounding and planning in less proficient languages.Source: §E.3 · p. 19
  • UI-TARS examples show that nonsensical actions can arise even during previously successful procedures, including menu navigation and repeated table insertion.Source: Figure 10 · p. 20, Figure 11 · p. 21
  • Minor selection and renaming mistakes cause cascading corrections that consume steps and prevent task completion within the limit.OpenAI CUA selects unintended files, while Claude CUA appends text to an already named folder.Source: §E.3 · pp. 19–20
  • OpenAI CUA repeatedly misclicks when opening Settings in Arabic despite completing the same task in English, Russian, and Chinese.This degradation primarily manifests as reduced grounding capability.Source: §E.3 · pp. 20–21
  • Claude CUA struggles with Arabic’s mirrored Dock layout, requiring a slower Apple-menu route instead of directly clicking Settings.The comparison indicates reduced grounding and some diminished planning in the mirrored interface.Source: §E.3 · p. 21
  • UI-TARS may simultaneously hallucinate interface locations and misinterpret task completion, such as claiming a 16:9 presentation is 4:3.Source: Figure 12 · p. 22, Figure 13 · p. 22, Figure 14 · p. 23

F Environment Implementation Details

The environment uses macOS Sequoia 15.2 with applications spanning the benchmark, while agent implementations differ in prompting, action formats, screenshot history, and annotation support. GPT-4o with Set-of-Mark annotations adds labeled visual transcripts and tag-based grounding, and Gemini Pro 2.5 uses a concatenated prompt because it does not accept system prompts.

F Environment Implementation Details

G.2 GPT-4o with Set-of-Mark Annotations

  • GPT-4o with Set-of-Mark annotations adds labeled bounding boxes, parsed transcripts, and tag-based coordinates to aid inaccurate grounding.The parser substitutes each tag with the center coordinate of its corresponding bounding box.Source: §G.2 · pp. 31, 33

G.3 Gemini Pro 2.5

  • Gemini Pro 2.5 uses the same prompting settings as GPT-4o but concatenates system and user prompts because it does not accept system prompts.Source: §G.3 · p. 33

G.4 OpenAI Computer-Using Agent

This section documents the prompting and interaction adaptations, configurations, and reproducibility considerations for GUI agents, together with limitations, societal impacts, safeguards, and supplementary checklist information. It also covers benchmark asset licensing and documentation, research ethics, infrastructure, and the stated use of LLMs.

G.4 OpenAI Computer-Using Agent

  • OpenAI CUA uses T=1 and top_p=0.9 with an added system prompt, post-action screenshots, full conversation history, and n = 3 screenshot filtering via black-image substitution.The prompt adds FAIL/DONE control signals and specifies available keys; earlier screenshots are replaced with a 5×5 black image.Source: §G.4 · pp. 33–34

G.5 Claude Computer Use Agent

  • Claude CUA follows its format with T=1, an augmented prompt, default n = 3 image filtering, and screenshot requests that alternate with actions, requiring approximately 30 dialogue rounds for 15 steps.The prompt requires screenshot-based step verification and explicit reasoning before proceeding.Source: §G.5 · p. 34

G.6 UI-TARS

  • UI-TARS was configured at T=1 and top_p=0.9; the current query contained only that round’s screenshot, the system prompt was retained, and n=3 recent screenshots were filtered.Its output format uses Thought and Action, with Chinese required in the Thought part; documented actions include click, left_double, right_single, and drag, with available hotkeys listed.Source: §G.6 · pp. 34–35

G.7 ShowUI

  • ShowUI uses no system prompt; each user query combines fixed instructions, task details, the current screenshot, and action history, with normalized coordinates and dictionary-formatted actions.Its action space includes CLICK, INPUT, HOVER, ENTER, SCROLL, ESC, and PRESS, while histories concatenate previous outputs.Source: §G.7 · pp. 35–36

NeurIPS Paper Checklist

  • The checklist states that the paper’s abstract reflects its macOS benchmark, multilingual design, and safety-subset contributions, while acknowledging binary rewarding as the main limitation.It also reports no theoretical results and no reported error bars or statistical significance information.Source: NeurIPS Paper Checklist · pp. 37–39
  • The checklist reports that experimental details and implementation configurations are sufficient for reproduction, with open data and code available through the project page.It cites Section 5.1, Appendix G, the main-paper testbench algorithms, and AWS-hosted environments as reproduction resources.Source: NeurIPS Paper Checklist · pp. 38–39

10. Broader impacts

  • The paper discusses both potential positive and negative societal impacts, with safeguards for responsible release described in Appendix B.The broader-impact discussion considers harms from intended use, incorrect results, and misuse, plus possible mitigation strategies.Source: §10 · pp. 40–41, §11 · p. 41

11. Safeguards

  • The paper notes that high-risk released models should include safeguards, while scraped datasets require attention to potential safety risks.The checklist recommends controlled access, usage restrictions, or safety filters where appropriate.Source: §11 · p. 41

12. Licenses for existing assets

  • The benchmark runs macOS environments on AWS-hosted dedicated Mac Mini machines under a licensed arrangement complying with Apple’s Software EULA.The justification states that both AWS and Apple could benefit from this arrangement.Source: §12 · p. 41

13. New assets

  • The paper answers yes that new assets are well documented, while marking the accompanying justification as not applicable.The checklist guidelines request structured details covering training, licensing, limitations, consent, and anonymization when relevant.Source: §13 · p. 42

14. Crowdsourcing and research with human subjects

  • The paper marks crowdsourcing, human-subject research, and related IRB requirements as not applicable.The checklist explains that the paper does not involve crowdsourcing or research with human subjects.Source: §14 · p. 42

16. Declaration of LLM usage

  • The paper marks LLM declaration as not applicable because LLMs were used only for writing, editing, or formatting, not as core methodological components.The stated use does not affect the research’s core methodology, scientific rigor, or originality.Source: §16 · p. 43

Related papers

Loading 2506.04135v4…