PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenComputer: Verifiable Software Worlds for Computer-Use Agents

Jinbiao Wei, Qianran Ma, Yilun Zhao, Xiao Zhou, Kangqi Ni, Guo Gan, Arman Cohan

arXiv:2605.19769v1cs.AIcs.SE

TL;DR

Realistic, reproducible desktop environments and reliable state verification remain costly bottlenecks for evaluating computer-use agents. OpenComputer addresses these gaps with verifier-grounded software worlds and machine-checkable tasks, finding that agents still struggle with end-to-end completion while hard-coded verifiers better match human adjudication than LLM judges.

  • Problem

    Evaluating computer-use agents is limited by the costly construction of realistic, reproducible desktop environments and tasks, together with reliable programmatic verification of outcomes.

  • Method

    OpenComputer automatically synthesizes realistic software worlds and desktop tasks around app-specific verifiers, executable success criteria, and auditable partial-credit rewards.

  • Results

    Agents struggle with reliable end-to-end completion; GPT-5.4 achieves a full task success rate of 68.3%, while hard-coded verifiers align more closely with human adjudication than LLM judges.

  • Takeaways & Limitations

    OpenComputer provides a reproducible, machine-checkable basis for studying agent reliability on realistic software workflows and fine-grained application state.

  • Takeaways & Limitations

    Some realistic tasks depend on visual or geometric properties that are difficult to express through application state and reliable hard-coded checks.

Abstract

from arXiv · show

We present OpenComputer, a verifier-grounded framework for constructing verifiable software worlds for computer-use agents. OpenComputer integrates four components: (1) app-specific state verifiers that expose structured inspection endpoints over real applications, (2) a self-evolving verification layer that improves verifier reliability using execution-grounded feedback, (3) a task-generation pipeline that synthesizes realistic and machine-checkable desktop tasks, and (4) an evaluation harness that records full trajectories and computes auditable partial-credit rewards. In its current form, OpenComputer covers 33 desktop applications and 1,000 finalized tasks spanning browsers, office tools, creative software, development environments, file managers, and communication applications. Experiments show that OpenComputer's hard-coded verifiers align more closely with human adjudication than LLM-as-judge evaluation, especially when success depends on fine-grained application state. Frontier agents struggle with end-to-end completion despite partial progress, and open-source models exhibit sharp drops from their OSWorld-Verified scores, exposing a persistent gap in robust computer automation.

1 Introduction

OpenComputer addresses the costly construction and verification of realistic desktop environments for computer-use agents by making verifier-grounded synthesis the organizing principle. Its benchmark and evaluations show that current agents remain unreliable on realistic end-to-end software workflows.

  • Motivation: Computer-use agents promise general-purpose interaction with everyday software, but training and evaluation are constrained by the cost of realistic, reproducible desktop environments and tasks.The challenge extends beyond model capability to constructing the software worlds in which agents are trained and evaluated.
  • Construction bottleneck: Realistic desktop task construction requires designing plausible goals, preparing coherent application states, and ensuring those states are reproducible.Preparation may include files, folders, spreadsheets, documents, browser history, bookmarks, emails, or calendars.
  • Verification bottleneck: Desktop evaluation must verify application state, file contents, metadata, and persistent side effects beyond what screenshots reveal.Each task therefore often needs custom inspection logic to determine whether the intended state was achieved.
  • OpenComputer: OpenComputer makes verification the organizing principle of environment and task construction through a verifier-grounded framework for synthesizing verifiable software worlds.The framework couples app-specific state verification, self-evolving verification, task generation, and trajectory-based evaluation.
  • Empirical findings: 68.3% was GPT-5.4’s full task success rate, compared with 64.4% for Claude-Sonnet-4.6 and 58.8% for Kimi-K2.6, while open-source agents lagged substantially behind.Verifier-grounded evaluation aligned more closely with human adjudication than LLM-as-judge evaluation; the benchmark spans 33 desktop applications and 1,000 finalized tasks.

2 Related Work

Prior computer-use benchmarks divide into static trajectory datasets, which mainly assess offline action prediction, and interactive environments, which evaluate agents through feedback. Recent work likewise identifies environment construction as a bottleneck and develops scalable simulated environments for tool use, function calling, and reinforcement learning.

  • Benchmarks for Computer-Use Agents: Computer-use benchmarks fall into static trajectory datasets and interactive task environments.Static datasets include Mind2Web and Android in the Wild, while interactive benchmarks evaluate agents through environment feedback, including OSWorld.
  • Benchmarks for Computer-Use Agents: Static datasets provide broad web or mobile-interface coverage through human demonstrations but primarily evaluate offline action prediction.The passage names Mind2Web and Android in the Wild as examples.
  • Synthetic Environments for Agents: Recent work treats environment construction as a key bottleneck for training interactive agents.This trend spans tool-use, function-calling, and reinforcement-learning settings.
  • Synthetic Environments for Agents: AgentScaler, Agent World Model, and Simia develop scalable simulated environments using database-backed APIs, code-driven multi-turn interactions, and simulated feedback.These systems target tool use, function calling, reinforcement learning, and reasoning-model-based environment simulation.

3 OpenComputer

OpenComputer constructs verifiable computer-use tasks in real desktop software by coupling executable environments with app-specific, machine-checkable verification. Its pipeline evolves verifiers through execution-grounded feedback, synthesizes verifier-grounded tasks, and evaluates trajectories with partial-credit rewards based on passed checks.

  • Problem setup: OpenComputer formulates each task as an application, user goal, executable initialization procedure, agent-facing description, and machine-checkable success criteria.Agents act from an initial sandbox state through screenshots and GUI actions, producing a final application state.
  • Problem setup: The framework addresses both coherent software-world construction and reliable determination of whether the user goal was achieved from application state.This casts environment construction as a constrained synthesis problem over applications and natural-language goals.
  • System architecture: OpenComputer couples verifier generation, verifier evolution from calibration executions, and verifier-aware task and environment synthesis.The resulting instances pair executable environments with user instructions and machine-checkable criteria grounded in inspectable software state.
  • App-specific verifiers: App-specific Python verifiers expose JSON CLI inspection and checking endpoints over reliably inspectable state surfaces, using channels such as debugging protocols, D-Bus, UNO, SQLite, accessibility state, and saved-file parsing.Verifier development includes endpoint references, written test plans, live integration tests, positive and negative cases, JSON-validity checks, and failure-mode coverage.
  • Self-evolving verification: Approximately 15 calibration tasks per application provide execution-grounded probes for exposing brittle schemas, incomplete endpoint coverage, and mismatches with real software behavior.An LLM evaluator and programmatic verifier compare criterion-level verdicts on the same executions, while genuine agent failures are discarded during diagnosis.
  • Evaluation harness: R = Npass/Ntotal assigns each task the fraction of checks passed, preserving exact machine-checkable conditions while supporting partial credit after final verification in the sandbox.For applications where persistence matters, the harness attempts a final save action before executing checker commands.

4 Experiment

OpenComputer evaluates computer-use agents on a 33-application benchmark with machine-checkable criteria and partial-credit rewards. Results show that even frontier agents struggle with complete task execution, while open-source models transfer poorly from OSWorld-Verified to OpenComputer.

  • Benchmark: 33 desktop applications comprise the finalized benchmark, with each task initialized in a fresh sandbox and judged against machine-checkable success criteria.Tasks include natural-language instructions, executable initialization, and required files, profiles, configurations, and application artifacts.
  • Overall performance: 88.4% average reward and 68.3% task success make GPT-5.4 the strongest evaluated agent, yet it fails nearly one third of benchmark tasks.Average reward measures the fraction of passed verifier checks, while task success requires satisfying all required criteria.
  • Efficiency: 19.0 steps per task make GPT-5.4 the most efficient agent, partly because it combines multiple low-level operations into single computer-control steps.The evaluation setting also limits GPT-5.4 outputs to executable actions rather than long reasoning traces.
  • Cross-benchmark generalization: 52.3% OSWorld score versus 5.7% OpenComputer success for GUI-OWL-1.5-8B, and 46.1% versus 10.9% for EvoCUA-8B, reveal substantial cross-benchmark drops.These results indicate limited generalization from reported OSWorld performance to OpenComputer.

5 Analysis

Hard-coded verification aligns more closely with human judgments than LLM judging, especially when critical state is hidden from screenshots. GUI agents achieve higher pass rates on the shared CLI-compatible subset, while self-evolving verification repairs most checker-side errors and improves human-checker agreement.

  • Verifier–Human Alignment: 113 of 120 tasks match human verdicts with the hard-coded verifier, versus 95 of 120 for the LLM judge; checklist agreement is 97.3% versus 92.2%.Fine-grained desktop mistakes can be visually tiny, limiting screenshot-based evaluation.
  • Verifier–Human Alignment: Hard-coded verification has a larger advantage in Blender and developer tools, where scrollback logs and intermediate artifacts may not be simultaneously visible on screen.Programmatic verifiers directly inspect post-execution application state, whereas LLM judges infer from a narrow terminal window.
  • GUI–CLI Comparison: 14 applications and 343 tasks form the shared CLI-compatible subset; GUI agents achieve higher pass rates, while Claude Code averages 141 seconds per task versus 288 and 622 seconds for GUI controls.The GUI baselines are GPT-5.4 at 288 seconds and Claude Sonnet 4.6 at 622 seconds.
  • Self-Evolving Verification: 450 calibration tasks include 159 checker–reference disagreements, of which 76 are checker-side errors; self-evolution repairs 68, an 89.4% repair rate.The calibration tasks probe verifier reliability rather than agent capability, with a maximum repair budget of three iterations per task.
  • Self-Evolving Verification: 47 checker-side errors are repaired after one iteration, 15 after two, and 6 after three; human-checker agreement rises from 85.2% to 94.1%.Eight checker-side cases remain unresolved within the three-iteration budget.

6 Conclusion

OpenComputer introduces verifier-grounded software worlds that make application state inspectable across task construction and benchmark execution. The framework exposes persistent reliability challenges in computer-use agents while providing infrastructure for grounded evaluation, reproducible tasks, and agent improvement.

  • Framework contribution: OpenComputer makes inspectable application state a core design constraint across verifier construction, task synthesis, and benchmark execution.This supports executable desktop tasks with machine-checkable success criteria while preserving realistic software workflows.
  • Agent limitations: Frontier agents often make meaningful partial progress, but reliable end-to-end completion remains difficult when success depends on fine-grained application state, persistent files, metadata, or hidden side effects.These environments expose failure modes that are difficult to capture through surface-level task progress alone.
  • Research infrastructure: OpenComputer provides infrastructure for grounded rewards, reproducible task construction, trustworthy environments, and verifiers supporting evaluation and training.The framework is positioned as a foundation for studying reliability, collecting grounded trajectories, analyzing failures, and improving agents.

Limitations and Future Work

OpenComputer’s hard-coded verification cannot reliably capture every realistic desktop task, particularly those requiring visual or geometric judgments. Such criteria are delegated to LLM-based visual judgment, while affected tasks are excluded from the official benchmark and reported evaluations to preserve auditability and reproducibility.

  • Verifier limitations: Some realistic desktop tasks require visual or geometric judgments that application state alone cannot reliably express.In Draw.io, verifiers can inspect shapes, labels, and connectors but may struggle to determine whether an arrow is visually and semantically correct.
  • Verifier limitations: Criteria that hard-coded verifiers cannot reliably check are marked for LLM-based visual judgment rather than treated as fully programmatic rewards.This handling distinguishes non-programmatic criteria from executable verification.
  • Benchmark exclusions: 17 generated tasks contained at least one success criterion that could not be fully checked by a hard-coded verifier.The passage reports this count for the current task-generation process.
  • Benchmark exclusions: Tasks containing unreliable hard-coded criteria are excluded from the main benchmark and all reported evaluation results to maintain auditability and reproducibility.These exclusions prevent partially non-programmatic tasks from entering official evaluations.

A Case Study: Self-Evolving Verification in a Programmatic Verifier

A darktable case study shows how execution-grounded feedback corrected a schema-drift bug that caused a programmatic verifier to reject a successfully completed task. Updating database queries and joins made the verifier agree with the reference judgment on all ten criteria.

  • Task: The kimi-k2.6 run completed the darktable task in 53 interaction steps, importing three images, tagging them, and assigning one-, three-, and five-star ratings.The frozen trajectory was reused throughout self-evolution.
  • Reference judgment: The LLM judge marked all ten criteria satisfied, including tag creation, attachment to all three images, and the requested ratings.It verified the final image information and rating flags.
  • Detected disagreement: Four tag-related criteria produced verifier-side failures because the checker queried tag metadata using an outdated database assumption.The LLM returned TRUE while the verifier returned FALSE for tag existence and the three image-tag associations.
  • Root cause: The verifier assumed tags were stored in library.db, but current darktable state stores tag definitions in data.db while associations remain in library.db.The resulting missing-table SQL errors were counted as negative verifier results.
  • Verifier evolution: Self-evolution rerouted tag queries to data.db, rewrote image-tag checks as a cross-database join, and updated the verifier documentation without changing the trajectory or task.The public checker interface remained unchanged; only internal SQL sources and join paths were modified.
  • Outcome: After re-execution on the same cached final state, the updated verifier passed all ten criteria with zero remaining divergences from the LLM reference.The case demonstrates feedback-driven repair of brittle verifier assumptions such as application schema drift.

B Case Study: Comparison between LLM as Judge and Hard-Coded Verifier

The case study shows that screenshot-based LLM judges can miss decisive application state, motivating LLM-as-judge for verifier debugging but hard-coded verifiers for final scoring.

  • Failure modes: Dense interfaces can hide failures in a single cell boundary, hidden formula, or formatting change that screenshots render as broadly plausible.One example places “alpha beta” in one cell instead of “alpha” and “beta” in adjacent cells.
  • Failure modes: Terminal-heavy tasks exceed screenshot context because decisive evidence may reside in logs, exit codes, filesystem state, and intermediate outputs.A screenshot shows only one scroll position and pane layout, forcing inference about earlier logs and missing artifacts.
  • Implication for OpenComputer: LLM-as-judge serves as a flexible reference for detecting verifier bugs, underspecified criteria, and pipeline issues during task construction.It is used as a reference signal for verifier debugging rather than as the final benchmark reward.
  • Implication for OpenComputer: Hard-coded verifiers perform final scoring by inspecting application-grounded state, making rewards reproducible, auditable, and sensitive to exact success conditions.This role separation is motivated by the limitations of screenshot-based judgment in dense interfaces and terminal-heavy workflows.

C Examples of Generated Verifiable Tasks

OpenComputer generates realistic desktop tasks whose outcomes are checked by executable criteria over application state, files, metadata, and persistent side effects. Representative tasks span hierarchical organization, music editing, spreadsheet formulas, note-taking, and 3D object relationships.

  • Task Design: Generated tasks pair realistic desktop actions with executable checks of application state, files, metadata, and persistent side effects.The examples cover multiple desktop applications and machine-checkable outcome conditions.
  • Zotero: Three-Level Collection Hierarchy: Zotero tasks require a three-level collection hierarchy in which Vision is a child of Papers and Object Detection is a child of Vision.The verifier checks that all three collections exist and that each parent-child relationship is direct.
  • MuseScore: Piano Solo to Piano Quartet: MuseScore tasks transform a solo piano score into four parts while checking measures, notes, dynamics, articulations, and exported MusicXML.The saved score must contain Piano, Violin, Viola, and Violoncello parts, with at least 52 notes and four staccato articulations.
  • LibreOffice Calc: Sales Commission Spreadsheet: Calc tasks use nested IF formulas to assign 10%, 8%, or 5% commission rates, compute amounts for 20 records, and create a summary sheet.Verification checks headers, formulas, tier examples, and the Commission Summary sheet.
  • Obsidian and Blender: Obsidian and Blender examples verify structured note vaults and parent-child object hierarchies, including folders, links, tags, object types, and saved files.The Obsidian vault contains four recipe notes plus Index.md, while Blender checks Rig, Torso, Head, and Arm relationships.
Loading 2605.19769v1…