PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use

Hanzhang Zhou, Long Li, Quyu Kong, Lei Wang, Panrong Tong, Long Chen, Tianyu Xia, Gongjie Zhang, Xu Zhang, Chen Liu, Jianan Zhang, Chenglin Cai, Zhengwen Zeng, Steven Hoi, Yue Wang

arXiv:2610.22200v1cs.CV

TL;DR

GUI-only studies and largely closed frontier systems leave broader computer-use design difficult to study systematically. OpenGCUA releases an open GCUA stack with code execution and harness-to-model decision transfer; on OSWorld-v2, its full setting improves scores across all eight evaluated models while reducing reported steps.

  • Problem

    Computer-use research has largely studied GUI-only agents, while frontier systems combining visual and programmatic access remain closed, limiting systematic study of their design.

    Source: §1 · p. 3

  • Method

    OpenGCUA develops an open GCUA stack in which the model chooses how to observe, act, and organize work, using code execution and transferring selected harness decisions to the model.

    Source: §1 · pp. 3–4

  • Results

    Across eight frontier models, the full GCUA setting raises mean partial score by 19.6 points and cuts steps per task by 53.0% versus GUI-only CUA.

    Source: §1 · p. 4, §7 · p. 25

  • Takeaways & Limitations

    The first release provides the implementation, prompts, evaluation trajectories, and preliminary real-device harness and cross-device SDK for community study.

    Source: §1 · p. 3

    Reported wall-clock measurements are exploratory because API bandwidth, rate limits, concurrent environments, provider queuing, and tool latency were not controlled across models.

    Source: §3.3.6 · p. 16

Abstract

from arXiv · show

Computer use has reached a turning point: the focus is expanding from GUI-only interaction to systems that complete tasks on users' computers through visual and programmatic interfaces. We define a generalist computer-use agent (GCUA) as a system in which the model chooses how to observe, act, and organize its work across visual and programmatic interfaces. Yet frontier implementations, such as GPT-6 Astra's computer-use system, remain largely closed, limiting systematic study of their design. We introduce Open Generalist CUA (OpenGCUA), an ongoing project to develop and open the full stack for GCUAs with the community. This report presents the first release of OpenGCUA. Code execution is a key feature of our agent design: it lets intermediate results determine subsequent operations and enables models to apply their existing programming capabilities directly to computer interaction. We also propose harness-to-model decision transfer, moving selected observations and task-state decisions from fixed harness rules to the model. On OSWorld-v2, our full configuration improves task scores across all eight evaluated models, including GPT-6 Astra and Claude Opus 5, raising mean partial score by +19.6 points and reducing reported steps per task by an average of 53.0% relative to batched GUI-only execution. We release our implementation and prompts, evaluation trajectories from all eight models, and preliminary versions of a real-device harness and a cross-device SDK. The harness addresses real-device challenges not fully captured by benchmarks, while the SDK provides a common programming interface across computers, mobiles and browsers. Across four real macOS tasks under our real-device harness, selected Qwen3.8-Max runs use 19.4 times as many reported reasoning tokens as GPT-6 Astra, highlighting substantial differences between models in real-device execution efficiency.

1 Introduction

Computer-use research is shifting from GUI-only interaction toward systems that complete work across visual and programmatic interfaces, but frontier systems remain largely closed. OpenGCUA presents a first release of an open GCUA stack, including an agent design, evaluation results, and preliminary real-device tools.

1 Introduction

  • A GCUA is an agent system whose model chooses what to observe, how to act, and how to organize work across visual and programmatic interfaces.GUI, CLI, multi-agent hybrid, and GUI+CLI agents are special cases that fix some of these decisions in advance.Source: §1 · pp. 3–4
  • OpenGCUA is an ongoing project to open the full GCUA stack, and this report's first release covers agent design, evaluations, trajectories, and preliminary real-device tools.The release does not yet include a trained model.Source: §1 · p. 3
  • Across eight frontier models, the full GCUA setting raises mean partial score by 19.6 points and cuts steps per task by 53.0% versus GUI-only CUA.The evaluation uses the offline OSWorld-v2 set.Source: §1 · p. 4
  • With the same GUI and shell access, code execution adds 6.1 points to mean partial score and cuts steps per task by 14.8% against the GUI+CLI baseline.Intermediate results can determine subsequent operations without another model call, while code lets models apply existing programming capabilities to computer interaction.Source: §1 · p. 4
  • Harness-to-model decision transfer delegates selected workflow choices, including when to request screenshots and how to track task state, to the model.On-demand observation alone increases cache hit rates by 10.5 and 14.4 percentage points for two models without lowering task scores.Source: §1 · p. 4
  • The preliminary real-device harness addresses shared-machine and background-window challenges, while the cross-device SDK gives programs one interface across computers, browsers, and phones.On four real macOS tasks, Qwen3.8-Max uses 19.4 times as many reported reasoning tokens as GPT-6 Astra.Source: §1 · pp. 4–5

2 Generalist CUA: Definition and Positioning

OpenGCUA defines a GCUA by model-controlled choices about observation, action, and work organization, rather than simply by having multiple interfaces. It positions existing agent families as fixing some of those choices and motivates code execution as a way to make actions conditional within a model call.

2.1 Definition

  • A GCUA lets the model choose views, write programs across GUI, accessibility, shell, and API operations, and maintain its working state, unlike batched GUI+CLI agents with fixed screenshots and harness rules.A GCUA can use loops, branches, and variables in its programs.Source: §2.1 · p. 6

2.2 Relation to Existing Agents

  • Existing families fix choices differently: GUI agents lock all three, CLI agents restrict observation and action, and multi-agent hybrids choose interfaces per subtask rather than per step.Table 1 compares agent families by the three decisions and marks the progression from GUI+CLI agents to GCUAs.Source: §2.2 · pp. 6–7, Table 1 · p. 7

2.3 Why Now: The Route Flip to Code Execution

  • Code execution allows branches, loops, and verification to depend on runtime results within one model call, whereas batched actions commit to a list before seeing the first result.A task that takes a dozen round trips can become one call.Source: §2.3 · p. 8

2.4 Efficiency as a First-Class Metric

  • Success remains the primary measure, while efficiency is tracked through steps per task, output tokens, reasoning tokens, and cost.Fewer steps reduce model-call overhead; fewer output tokens can lower generation time and cost.Source: §2.4 · p. 8

3 Agent Design: Code Execution and Harness-to-Model Decision Transfer

OpenGCUA studies code execution and model-directed workflow decisions as ways to let existing frontier models act, observe, and organize work across computer interfaces. Its evaluation compares the full design with GUI-only and hybrid baselines, then examines resource-aware performance.

3.1 Code Execution over GUI and CLI

  • Code execution lets a model combine GUI actions, shell commands, and Python control flow in one program, using intermediate results to choose subsequent operations.The harness syntax-checks and runs the program, then returns execution feedback; the real-device harness also supports Playwright and requested accessibility-tree observations.Source: §3.1 · p. 8, §3.1.1 · p. 9

3.1.2 Representative Code-Execution Patterns

  • Programs coordinate GUI actions with checks, repeat procedures across targets, and use conditional logic to decide whether to continue or act.Examples include checking seven webpage sections in a loop and clicking Send only when a draft matches a saved version; that step does not verify delivery.Source: §3.1.2 · pp. 9–10
  • Programs can prepare observations and verify both intended changes and preserved content, leaving the model to interpret evidence that still requires judgment.Examples include assembling 25 views into a contact sheet and checking that 64 package entries and non-background slide content remain unchanged.Source: §3.1.2 · p. 10

3.2 Harness-to-Model Decision Transfer

  • Harness-to-model decision transfer delegates selected workflow choices, such as whether to request a screenshot and how to track task state, while the harness retains execution and context management.Screenshot-only interfaces make result-dependent checks harder when properties such as button state are not program-readable; accessibility trees can expose supported properties as structured data.Source: §3.2 · p. 11, §3.1.3 · p. 11

3.3.2 CUA versus GCUA

  • Mean partial score rises from 45.3 to 64.9 (+19.6 points) and steps per task fall by 53.0% across eight models versus batched GUI-only execution.Full-task success also increases for every model.Source: Table 2 · p. 13, §3.3.2 · pp. 13–14

3.3.3 GUI+CLI Hybrid CUA versus GCUA

  • In selected runs, OpenGCUA raises task scores by 4.4 to 9.1 points across all four models with fewer reported steps, while code execution beats each alternative across three models.Against batched GUI actions, code execution improves scores by 15.5 to 33.6 points and reduces steps per task by 55.3% to 63.1%.Source: §3.3.3 · p. 14, §3.3.4 · p. 15

3.3.5 Harness-to-Model Decision Transfer Ablation

  • On-demand observation raises cache hit rates in two selected models, while todo tracking raises task scores; intent annotations target user understanding, and micro-compaction lowers cache hit rates.The rolling screenshot buffer improves cache hit rates without lowering task scores in the selected comparisons.Source: §3.3.5 · p. 16

3.3.6 How Much Task Performance Can Money and Time Buy?

  • Resource-constrained comparisons show that the highest-scoring model can depend on the spending cap, while recorded-time curves favor GPT-6 Astra at most time limits.The time measurements are exploratory because bandwidth, rate limits, concurrency, queuing, and tool latency were not controlled across models.Source: §3.3.6 · p. 16

4 Real-Device Generalist CUA

Real-device operation adds challenges absent from controlled benchmarks, including shared control, changing interfaces, evolving requests, and existing user state. OpenGCUA presents a cross-device SDK and real-device harness, then reports preliminary results and case studies across four macOS tasks.

4.1 The Gap Between Benchmarks and Real Machines

  • Real-device tasks involve shared user control, changing windows, evolving requests, and pre-existing files or sessions, so benchmark scores alone do not establish reliable device operation.Relevant materials may be mixed with unrelated content, and expired sessions or missing permissions may require user involvement.Source: §4.1 · pp. 17–18

4.2 Cross-Device SDK

  • The Cross-Device SDK provides a common interface for coordinating devices, applications, and browser tabs through explicit targets, shared observations, and platform-specific capabilities.Programs can reuse host-side variables across turns and transfer files between resources; structured and visual observations complement one another.Source: §4.2 · pp. 18–19

4.3.1 Five-Layer Architecture

  • The five-layer harness runs model-generated programs through a persistent runtime, SDK, and drivers, while separating component responsibilities and reporting detectable failures.Successful action delivery does not establish that the intended task-level effect occurred; available operations remain constrained by permissions, application state, and accessibility support.Source: §4.3.1 · pp. 19–20

4.3.2 Responding to Real-Device Challenges

  • The harness prioritizes less intrusive actions, selective target observation, filtered accessibility information, and user interaction to preserve task continuity.Foreground input may require coordination; observations may need refreshing as interfaces change, and clarification or authorization can involve the user.Source: §4.3.2 · pp. 20–21

4.3.3 Preliminary Real-Device Results

  • Qwen3.8-Max averages 73.4K output tokens per task versus 12.9K for GPT-6 Astra, while costing $1.49 versus $18.29 under archived pricing assumptions.Across four macOS tasks, the study includes up to three runs per task and model; the reported aggregates include incomplete tasks, and one run per model limits generalization.Source: §4.3.3 · p. 21

4.3.4 Case Studies

  • 2,030 of Qwen3.8-Max’s 2,105 output tokens are reasoning tokens for a four-line screenshot program, versus 11 of GPT-6 Astra’s 124 tokens.The passage notes that such generation overhead can introduce substantial execution latency.Source: §4.3.4 · p. 22
  • Qwen3.8-Max’s desktop screenshot shows the foreground Taobao login page instead of its Amazon target; GPT-6 Astra captures the selected browser page directly.The contrast concerns observation scope, not screenshot use: both models use screenshots.Source: §4.3.4 · p. 22, Figure 10 · p. 22
  • GPT-6 Astra fills all 125 required Level answer cells, while Qwen3.8-Max leaves 40 empty after processing only 85 questions.GPT’s program checks seven supplied examples and identifies question rows; Qwen’s fixed row ranges omit questions without checking for omissions.Source: §4.3.4 · p. 23, Figure 11 · p. 23

5 Roadmap

The roadmap treats OpenGCUA’s first release as a basis for community research, with future work planned on models, efficiency, and real-device evaluation. The project invites contributions while leaving release timing and priorities open.

5.1 From Design Validation to Model Release

  • This report validates agent-design choices with existing models rather than introducing a trained GCUA model, and releases implementations, prompts, and evaluation trajectories.Preliminary real-device harness and cross-device SDK versions accompany the release.Source: §5.1 · p. 23

5.2 Research Priorities

  • Planned research priorities include a unified computer-and-browser model, more efficient reasoning and execution, and real-device benchmarks covering broader applications and changing states.The planned model is intended to coordinate visual and programmatic operations, request observations, and maintain task state.Source: §5.1 · p. 23, §5.2 · p. 24

5.3 How to Contribute

  • The project invites contributions to its harness, SDK, real-device evaluation, and research agenda, but sets no fixed delivery timeline and expects priorities to evolve.The agenda may change with experimental evidence and community proposals.Source: §5.3 · p. 24

6 Related Work

Related work spans frontier computer-use systems, open GUI-centric and hybrid agents, and open computer-use infrastructure. OpenGCUA positions its work as an effort to open a broader research stack, including a trained model, real-device harness, cross-device SDK, and execution trajectories.

Frontier Computer Use

  • Frontier systems combine graphical control with precise tools and cross-application workflows, but public releases leave implementation details and trajectories unavailable for systematic reproduction.Examples discussed include GPT-6 Astra and Anthropic’s Claude Cowork and Claude Code.Source: §6 · p. 24

Computer-Use Agents

  • Open CUA work includes GUI-centric models and newer hybrid systems that combine graphical control with code, APIs, tool calls, or program-state operations.RecreationWorld and WeaveBench provide environments and benchmarks for interleaved trajectories; Qwen-UI-Agent is described as prior unified GUI–CLI work.Source: §6 · pp. 24–25

Open Computer-Use Infrastructure

  • Existing open harnesses provide complementary infrastructure, while OpenGCUA aims to open a full GCUA stack coupling a trained model, real-device harness, cross-device SDK, and trajectories.The cited infrastructure includes cross-platform drivers, orchestration frameworks, and desktop-control runtimes.Source: §6 · p. 25

7 Conclusion

The conclusion presents OpenGCUA’s first release as an open effort around generalist computer use. Its benchmark results support the full configuration across eight models, while real-device findings remain preliminary and no newly trained model is introduced.

7 Conclusion

  • On OSWorld-v2’s offline subset, the full OpenGCUA configuration improves scores across all eight models, raising mean partial score by 19.6 points and reducing steps per task by 53.0% versus batched GUI-only execution.Ablations also report benefits from code execution when GUI and CLI access are held constant.Source: §7 · p. 25
  • OpenGCUA releases its implementation, prompts, and evaluation trajectories alongside preliminary real-device harness and cross-device SDK versions.The report studies design using existing models rather than introducing a newly trained model.Source: §7 · p. 25

A Evaluation Details

Evaluation uses the 82-task offline subset of OSWorld-v2 v2.1 and excludes tasks requiring external Internet access. Scores and resource metrics are aggregated using specified per-task and cross-model rules.

A Evaluation Details

  • Cross-model summaries weight each model equally, averaging score gains in points and calculating relative resource reductions within each model before averaging.Steps and token counts include all selected tasks, including failures; each task contributes its latest attempt with an evaluator result, not its highest-scoring attempt.Source: A Evaluation Details · p. 28
  • The offline evaluation uses 82 OSWorld-v2 v2.1 tasks, excluding 26 tasks requiring external Internet access while retaining local web applications and desktop tasks.Source: A Evaluation Details · p. 28
  • Task score is the mean evaluator partial score on a 0–100 scale, while full-task success requires a normalized evaluator score greater than 0.999999.Steps and token counts are averaged over all selected tasks, including failures.Source: A Evaluation Details · p. 28

B.1 Task Descriptions

The task descriptions specify four work products: a completed workbook, a filled PDF, a product-research document, and a hotel-comparison workbook. Each task sets constraints on the required inputs, verification, or permitted actions.

B.1 Task Descriptions

  • The workbook task requires computing 125 Level answers and requested bonus answers from race rules, a track layout, and worked examples, then saving a completed copy without altering the original.Source: §B.1 · p. 28
  • The PDF task requires filling applicable fields in a four-page Schengen form using synthetic identity, employment, travel, and accommodation records, then saving a new, correct PDF.Source: §B.1 · p. 28
  • The shopping task requires verifying 22 snack candidates, documenting recommendations with evidence and uncertainty about unverified ingredients or allergens, and making no purchase.When Amazon verification is unavailable, the task directs the agent to official brand or original product pages.Source: §B.1 · p. 28
  • The hotel task requires researching four prescribed plans for two Paris hotels and documenting tax-inclusive prices, refundability, breakfast, and source limitations in a four-sheet workbook; reservations are prohibited.The plans cover a December 20–30, 2026 stay with one hotel change and at least three nights at each hotel.Source: §B.1 · p. 28

B.2 Task Instructions and Materials

This appendix presents task instructions and materials for workbook, visa-form, snack-research, and hotel-comparison tasks, alongside recorded prompts for code execution, decision transfer, and OSWorld-v2. Together, these materials specify task inputs and outputs and guide agents in combining visual inspection with code-based actions and task-state checks.

B.2.1 Workbook reasoning and editing

  • The workbook task asks the agent to solve the challenge’s questions using the supplied workbook, submit website answers, and report actual and maximum scores.The source workbook contains 125 Level 1–7 questions, seven worked examples, and five bonus questions.Source: §B.2.1 · p. 28

B.2.2 Document-grounded PDF form filling

  • The visa task uses synthetic identity records and asks for a completed, saved four-page Schengen form populated from the supplied application materials.Requested information includes personal and passport details, itinerary, accommodation, transportation, occupation, and other clearly identifiable data.Source: §B.2.2 · p. 29

B.2.3 Evidence-grounded online shopping research

  • The snack task asks for all 22 candidates to be classified by checking Amazon ingredient and allergen information against the supplied dietary restrictions.The Word document must give each item’s name, assessment, rationale, any needed adjustment, and an “Ingredient information unverified” label when details cannot be verified.Source: §B.2.3 · pp. 30–31

B.2.4 Constrained live hotel comparison

  • The hotel task requires a verified, ranked Excel comparison of valid two-hotel Paris stays across official sites, Expedia, and Booking.com, without booking.The stay is Dec 20–30, 2026, for two adults; requirements include refundable rates, breakfast coverage on specified nights, rate details, and the three lowest-cost alternatives.Source: §B.2.4 · p. 32

C.1 Code Execution

  • Code-execution instructions define a fresh Python process per step, with printed output returned in the next observation and execution stopped after 300 seconds.Variables and imports do not persist, while external state such as files and application state does.Source: §C.1 · p. 33

C.2 Harness-to-Model Decision Transfer

  • The full setting adds model-requested screenshots and task-state updates, while the harness manages image retention, output compaction, and ledger storage.The prompt also specifies a short intent annotation and a single code block for each response; claimed verification records the model’s assessment, not independent verification.Source: §C.2 · p. 35

C.3 OpenGCUA Harness Prompt for OSWorld-v2

  • The all-configuration OSWorld-v2 prompt is reproduced with task-specific inputs supplied separately and only the environment credential redacted.The listing preserves its instructions and examples.Source: §C.3 · p. 35
  • The prompt provides pyautogui operations for mouse, keyboard, scrolling, typing, and position checks, while reserving image-location helpers as brittle last resorts.It notes that non-ASCII characters are typed through xdotool when available.Source: §C.3 · p. 36
  • Use normalized coordinates from 0 to 1000 on both axes, convert raw pixels before acting, and treat scrolling as wheel notches rather than pixels.Values outside the coordinate contract may be silently reinterpreted; the prompt also recommends waiting after actions that trigger loading or open windows and menus.Source: §C.3 · p. 36
  • Inspect when the next action depends on visible results, but batch confident GUI actions and include a screenshot at the end of the same script when needed.A script may contain at most one screenshot, as its final statement; split steps only when output or the screen must be read first.Source: §C.3 · p. 36
  • Keep unverified GUI sequences short, checking after about 5–10 actions, and avoid branching on unverified pixels because pixel colors and image matching are brittle.The prompt encourages loops and conditions over values actually read from file contents or bash output.Source: §C.3 · p. 36
  • Use GUI and bash actions in the same script when useful, and verify the requested final state before reporting success.Commit GUI edits when needed; if an action fails or state does not change, try a different approach, allowing for slow loading before treating it as failure.Source: §C.3 · pp. 36–37
  • Maintain a persistent todo ledger for multi-step work, updating task status in the same script as real actions where possible.Ledger calls do not affect the computer but still consume a step; the ledger is shown with observations and can be printed in full with todo.list().Source: §C.3 · p. 37
  • Submit each step as one concise intent followed by exactly one plain-Python code block, combining that step’s actions in the block.The intent is only a post-decision label for the code’s immediate purpose, not task instructions, reasoning, memory, or a completion claim.Source: §C.3 · p. 37
  • Prefer bash for task completion and reliable checks, while accounting for unsaved GUI state and reading exact values from the screen or command output.The prompt says bash sees the same machine but not unsaved application state, so save or close before reading files and reload after editing an open file.Source: §C.3 · pp. 36–37

D Trajectory Examples

The section presents selected successful GPT-6 Astra and Claude Opus 5 trajectories under the full GCUA setting. Their complete trajectories and visualizations are publicly available on GitHub.

D Trajectory Examples

  • Selected stages from successful GPT-6 Astra and Claude Opus 5 runs illustrate the full GCUA setting, with complete trajectories and visualizations available on GitHub.Source: D Trajectory Examples · p. 38

D.1 GPT-6 Astra: Selecting and Verifying Slide Backgrounds

The run created rainy and snowy slide composites, checked a slide’s transparency, and verified the saved presentation and output files. The saved evaluator result was 1.0.

D.1 GPT-6 Astra: Selecting and Verifying Slide Backgrounds

  • The agent inspected labeled contact sheets, selected transparent rain and black-backed snow overlays, and used Normal and Screen blending, respectively.The selected rain image had alpha extrema (0, 190), distinguishing it from opaque alternatives.Source: §D.1 · p. 38
  • The snowy slide’s saved 40% background transparency was checked visually in the Slide Properties panel.This property check was not the basis for the full-task score.Source: §D.1 · p. 38
  • The program compared the saved presentation with the original package and checked expected changes, background transparency, embedded image bytes, PNG dimensions, and opacity.The final checks confirmed both overlays at 1920 × 1080 and 40% transparency, with original slide content intact.Source: §D.1 · p. 38
  • The saved evaluator result is 1.0, and the presentation and both requested PNGs were saved successfully.The run also passed checks for the two slide backgrounds and found all 64 other original package entries unchanged.Source: §D.1 · p. 38

D.2 Claude Opus 5: Recording and Checking a Dashboard Walkthrough

The run recorded a fullscreen walkthrough of seven dashboard sections, then tested GIF encodings and checked the selected artifact. The saved evaluator result was 1.0.

D.2 Claude Opus 5: Recording and Checking a Dashboard Walkthrough

  • The screen capture recorded all seven sidebar sections at 1920 × 1080 for 14 seconds.A CDP viewport query failed with 403 Forbidden during setup.Source: §D.2 · p. 39
  • A contact-sheet screenshot was requested to check that all seven pages were visited and rendered fully.The recorded control result confirms screenshot=true.Source: §D.2 · p. 39
  • The first tested GIF variant optimized to 1,060,035 bytes and exceeded 1 MiB; the next two optimized to 929,833 and 829,415 bytes and met the limit.The run selected the 3-fps variant from the tested options.Source: §D.2 · p. 39
  • The saved evaluator result is 1.0; the chosen GIF was copied to Desktop/promo.gif, checked, and opened fullscreen for visual inspection.The encoding was selected from tested variants, without a claim of optimal compression.Source: §D.2 · p. 39
  • The accompanying manifest records source hashes, run identifiers, evaluator results, and source-line ranges; exported records preserve executed actions and tool outputs except marked credential replacements.These records are evidence records, not standalone replay programs.Source: §D.2 · p. 40

Related papers

Loading 2610.22200v1…