PaperlayerView exact revision on arXiv

Source-linked AI summary

MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents

Lawrence Keunho Jang, Andrew Keunwoo Jang, Jing Yu Koh, Ruslan Salakhutdinov

arXiv:2606.16748v1cs.LGcs.CL

TL;DR

Personal-computer benchmarks largely omit coherent personal context, historical data, and logged-in accounts. MyPCBench evaluates agents across a reproducible, persona-seeded Linux desktop with 17 web applications and 184 tasks, finding that even the strongest model fully solves only 55.4% of tasks and that performance worsens on cross-application and long-horizon tasks.

  • Problem

    Existing computer-use benchmarks omit coherent personal context, historical data, and realistic user data across applications, limiting evaluation of personal-assistant capabilities.

  • Method

    MyPCBench constructs a reproducible Linux desktop seeded for one persona, spanning 17 web applications and 184 rubric-graded tasks, and benchmarks six models with a uniform computer+bash interface.

  • Results

    55.4% of tasks are fully solved by the strongest model, while performance is substantially lower on tasks spanning seven or more applications and on longer trajectories.

  • Takeaways & Limitations

    The benchmark exposes cross-application and long-horizon failure patterns that identify concrete targets for improving personally intelligent computer-use agents.

  • Takeaways & Limitations

    The benchmark uses one fictional persona and one Linux/GNOME/Firefox stack, so it does not measure generalization across demographics, locales, device stacks, or genuinely sensitive personal data.

Abstract

from arXiv · show

Current benchmarks for computer-use agents evaluate models in impersonal environments. This leaves a gap between evaluation and deployment where personal assistants are expected to work across a user's whole digital life, including their context, historical data, and logged-in accounts. This gap is widest on web tasks, where live web evaluations cannot exercise sites that require logging in or personal information, the kind of site a real personal assistant has to drive. We introduce MyPCBench, which tests computer-use agents as personal assistants on a Linux desktop populated with 17 simulated real-world web applications and a full desktop stack, all seeded for one canonical persona, Michael Scott from The Office. We define 184 tasks in this environment, each inspired by a real request drawn from the OpenClaw community, and benchmark six closed and open-weight models with a uniform computer+bash tool surface. We find that the best model, Claude Opus 4.6, fully solves 55.4\% of the tasks, the only model above 50\%. Model failures cluster on tasks that span many applications and on long trajectories, where personalization stresses an assistant the most. We release the environment, task set, and agent harness at https://mypcbench.com.

1 Introduction

MYPCBENCH evaluates computer-use agents as personal assistants in a reproducible Linux desktop populated with a coherent, cross-application persona and persistent history. Across 184 tasks, even Claude Opus 4.6 fully solves only 55.4%, with performance falling to 36% on tasks spanning at least seven applications.

  • Motivation: Current benchmarks largely use empty desktops, generic application states, and task-specific data, omitting users’ accumulated records across applications.These records include bank transactions, calendars, email, travel bookings, and work chats.
  • Benchmark: MYPCBENCH provides a deterministic, cross-application-consistent desktop seeded from one persona specification and packaged as a Docker container.The environment includes 17 custom web applications and a full Linux desktop with Firefox, LibreOffice, and a file manager.
  • Benchmark: 184 tasks inspired by real requests evaluate agents over persistent identity, cross-app history, and a full personal-computer environment.The benchmark uses six closed- and open-weight models with a uniform computer+bash tool surface, plus failure and scaling analyses.
  • Results: 55.4% of MYPCBENCH tasks are fully solved by Claude Opus 4.6, the strongest current frontier agent reported.The passage identifies this as the headline finding and states that every model used the same computer+bash action space.
  • Results: 36% of tasks spanning 7 or more applications are fully solved by Claude Opus 4.6, while GPT-5.5 reaches 4.5% and the three other reported models reach 0%.The 0% result applies to GPT-5.4 mini, Qwen 3.5 35B-A3B, and Qwen 3.5 9B.

2 Related Work

Prior benchmarks span synthetic, realistic, static, and live web environments, while personalization benchmarks evaluate profiles, histories, or behavior logs. MyPCBench addresses the gap by placing end-to-end personal data across a deterministic desktop environment.

  • Web and desktop agent benchmarks: Web benchmarks progressed from synthetic environments to realistic synthetic websites, static real-website task datasets, and live Internet evaluations.Desktop benchmarks such as OSWorld are also part of this broader computer-use evaluation landscape.
  • Personalization benchmarks: Personalization benchmarks evaluate retrieved user profiles, long conversational histories, or personal context attached to web agents.Examples include LaMP, LongMemEval, PersonalWAB, and Persona2Web.
  • Addressing the personalization gap: Most prior benchmarks exclude tasks requiring personal data or pages behind logins, such as calling an Uber, paying a friend on Zelle, or reordering DoorDash.These limitations leave personal-assistant tasks underrepresented in benchmark environments.
  • Addressing the personalization gap: MyPCBench retains OSWorld’s fixed VM image and deterministic snapshot reset while seeding Michael Scott’s data across every application.This differs from seeding only the data touched by each task and supports end-to-end personalization.

3 MYPCBENCH

MyPCBench is a reproducible Linux desktop environment for evaluating personalized computer-use agents across 17 logged-in web applications and a full desktop stack. It is designed around cross-app consistency, persona coherence, and real-world fidelity, with deterministic persona seeding and resettable VM infrastructure.

  • Environment: MyPCBench runs as an open-source Ubuntu 24.04 VM with GNOME Shell, 17 pre-logged-in websites, office software, and a realistic Firefox profile.Two web applications are also exposed as native desktop apps.
  • Design properties: Cross-app consistency links plausible records for the same event across bookings, charges, calendars, boarding passes, browsing history, emails, and messages.The seeded Philadelphia trip illustrates how one event generates correlated records across applications.
  • Design properties: Persona coherence entangles a specific user’s identity, relationships, routines, preferences, and experiences across applications using Michael Scott’s canon-informed persona.The environment uses The Office canon to populate large-scale coherent data for the persona.
  • Design properties: Real-world fidelity preserves each application’s analogue-like interface, navigation, and supported workflows while running locally under fixed-VM security and reproducibility constraints.The 17 applications are full Next.js clones implementing real workflows rather than static mocks.
  • Infrastructure: The infrastructure uses a 4-vCPU, 8-GB VM, takes about 90 seconds to boot, and resets tasks from a base snapshot to avoid state leakage.Artifacts include a QEMU wrapper image and standalone qcow2 disk, supporting execution inside Docker or directly under QEMU.

4 Tasks and Evaluation Setup

MyPCBench defines a manually curated 184-task suite over 17 simulated applications, emphasizing multi-application personal-data workflows, and evaluates agents through a standardized desktop harness with rubric-based metrics. Tasks and rubrics undergo author audits, while trajectories are judged for partial completion, perfect execution, and efficiency.

  • Task construction: 184 tasks were selected from 2,749 anonymized OpenClaw use cases, filtered for duplicates, deterministic-VM feasibility, and coverage by the 17 hosted applications.Requests were rewritten, assigned per-task rubrics, and audited by the authors; each task is stored with its instruction and rubric.
  • Quality assurance: Each task was reviewed by at least two authors, who ran it end-to-end and verified entity existence, answer obtainability, and step-level rubric checkability.This quality-assurance process manually verifies both coding-agent-generated task drafts and application clones.
  • Task coverage: 68% of tasks involve multiple applications, spanning one to nineteen apps, while 40% cover at least two SimilarWeb top-level categories.The multi-application regime tests personalization by requiring agents to reconcile persona data across applications.
  • Evaluation harness: The harness exposes standard CUA agents to a Linux desktop through an OSWorld-compatible HTTP Control API, with screenshot observations and an unmodified pyautogui action surface.The action surface includes click, type, key, scroll, drag, wait, screenshot, done, and fail; provider-specific actions are translated onto it.
  • Metrics and judging: The evaluation reports rubric score for partial completion, perfect rate when every rubric passes, and Trajectory Efficiency as rubric score per agent step.Rubrics contain 3–13 criteria, averaging 6.5 per task and totaling 1,191; the judge is gemini-3.1-flash-lite-preview.

5 Experiments and Analysis

Across 184 tasks, Claude Opus 4.6 leads all evaluated models, but performance declines sharply as tasks span more applications or require longer, more personalized reasoning. Failures are dominated by premature completion and skipped applications, with distinct error patterns across model families.

  • Overall performance: 55.4% perfect rate makes Claude Opus 4.6 the only model above 50%, ahead of Claude Sonnet 4.6 at 39.1% and GPT-5.5 at 29.3%.Opus also reaches 81.8% rubric score.
  • Overall performance: 7.6% versus 2.7% perfect rate shows Qwen 3.5 35B-A3B nearly triples Qwen 3.5 9B within the open-weight tier.With the dual-tool surface, Qwen 9B falls from 20.2 to 7.0 rubric against its CUA-only baseline.
  • Trajectory efficiency: 3.61 rubric points per step gives Opus over 5× Qwen 3.5 9B’s 0.65, while similar step counts can represent efficient execution or unproductive looping.Sonnet uses 45.8 steps at efficiency 3.03, whereas Qwen 9B uses 69.2 steps at efficiency 0.65.
  • Task-type analysis: 82% versus 45% perfect rate on the same 11 pattern-inference tasks separates Opus from GPT-5.5, while the remaining categories require persona-history reasoning or multi-app writes.On personal lookup, every API-tier model exceeds 38% perfect and Opus leads at 54%; on bounded action, only Opus, Sonnet, and GPT-5.5 exceed 46%.
  • Horizon scaling: 66% to 36% is Opus’s perfect-rate decline from single-app to 7+-app tasks, compared with 46% to 14% for Sonnet; GPT-5.5 reaches only 4.5% at 7+ apps.GPT-5.4 mini, Qwen 35B, and Qwen 9B all reach 0% at 7+ apps.
  • Failure analysis: 354 premature-DONE hits and 323 skipped-required-app errors account for most failed-rubric loss, followed by 129 surface-error abandonments, 47 partial artifacts, and 31 hallucinated-persona-data errors.GPT dominates premature DONE with 235 of 354 hits, while Qwen drives persona-data hallucination with 13 of 31.

6 Conclusion · A Application Details

The conclusion identifies three family-specific failure patterns in MyPCBench, while the application details document the 17 hosted web applications and their real-world analogues and categories.

  • 6 Conclusion: MyPCBench exposes improvement points for every evaluated model as a personally intelligent computer-use-agent benchmark.The conclusion frames these gaps as specific failure modes rather than only aggregate-rate differences.
  • 6 Conclusion: Three family-shaped failure patterns account for the benchmark gaps.The patterns are associated with Claude, the GPT family, and two Qwen models.
  • 6 Conclusion: Claude shortcuts past the user interface through bash.
  • 6 Conclusion: The GPT family prematurely marks tasks DONE before producing the rubric-graded side effect.
  • 6 Conclusion: Within Qwen, the 35B model hallucinates persona values, while the 9B model collapses under the dual-tool schema.
  • A Application Details: The environment hosts 17 web applications, each mirroring a real-world service.Table 3 also records the SimilarWeb top-level category and subcategory inherited from each analogue; screenshots appear in Figure 2.

B Task-Distribution Plots

Figure 5 characterizes the benchmark’s task distribution by application breadth, application categories, and exclusive behavioral task types.

  • Application breadth: The left plot distributes tasks by the number of distinct applications they touch.This measures application breadth at the task level.
  • Application categories: The middle plot shows the fraction of tasks touching at least one application in each SimilarWeb top-level category.Categories are non-exclusive, so one multi-application task can contribute to several bars.
  • Behavioral task types: The right plot gives an exclusive 1-of-6 behavioral task-type split for every task.Each task is assigned to exactly one of six behavioral categories.

C Task-Type Definitions · D Tool Surface: cua-only vs. cua+bash

MyPCBench organizes its 184 tasks into six behavioral types and evaluates agents with either computer-only tools or computer plus bash. Adding bash requires explicit GUI-workflow guidance and can harm weaker agents, notably Qwen 3.5 9B.

  • C Task-Type Definitions: Six behavioral task types cover all 184 MYPCBENCH tasks, with definitions, counts, representative instructions, and a released per-task mapping.The mapping is provided in tasks/final/task_types.json.
  • D Tool Surface: cua-only vs. cua+bash: Main results evaluate every model with both computer and bash tools; Claude’s native agent already includes bash and a file editor.OpenAI and Qwen receive vendor-documented bash or shell tools added to otherwise computer-only agents.
  • D Tool Surface: cua-only vs. cua+bash: Without guidance, computer-plus-shell agents may answer from shell data and stop before completing the requested visible GUI workflow.This failure mode is called “premature DONE.”
  • D Tool Surface: cua-only vs. cua+bash: The dual-tool guidance assigns shell tools to read-only inspection, querying, parsing, and computation while reserving the GUI for requested visible actions.OpenAI receives the hint in its system prompt, and Qwen receives equivalent guidance in its bash-tool description.
  • D Tool Surface: cua-only vs. cua+bash: Claude receives no dual-tool hint because its native agent already maintains a stable bash/computer mix, creating the sole prompt asymmetry between model families.The stated action-distribution evidence comes from Appendix L.
  • D Tool Surface: cua-only vs. cua+bash: Qwen 3.5 9B drops −13.2 rubric points with bash, as malformed calls splice bash and computer schemas together.The trajectories suggest the documented tool becomes actively harmful below a capability threshold.
  • D Tool Surface: cua-only vs. cua+bash: GPT-5.5, GPT-5.4 mini, and Qwen 35B show small cua+bash differences relative to cross-family gaps, so the paper makes no strong claim that bash helps.Table 5 compares both modes across the full 184-task suite under identical persona context, a 100-turn budget, and the same Gemini judge.

E Per-Task-Type, Cross-App, and Step-Budget Scaling · F Family-Signature Plots

The paper analyzes performance by task type, number of applications, and step budget, then characterizes model families through failure-mode and error-budget plots. Most tasks are multi-application, while step-budget curve shapes distinguish model families and expose trajectories that exceed the plotted range or fail early.

  • E Per-Task-Type, Cross-App, and Step-Budget Scaling: Tables 6 and 7 provide the raw numbers underlying Figure 3.
  • E Per-Task-Type, Cross-App, and Step-Budget Scaling: Step-budget curves show the fraction of 184 tasks graded perfect within each model’s step budget.Perfect tasks completed in more than 100 steps are outside the plotted range, so curves can end below Table 2 perfect rates; GPT-5.5 ends at 27.2 versus 29.3 overall for this reason.
  • E Per-Task-Type, Cross-App, and Step-Budget Scaling: Figure 6 distinguishes model families through curve shape, not only curve height.
  • E Per-Task-Type, Cross-App, and Step-Budget Scaling: Table 6 reports rubric scores by task type, ordered by descending cross-model average.The mean is the unweighted average across six models, and the best score in each row is bold.
  • E Per-Task-Type, Cross-App, and Step-Budget Scaling: 68% of MyPCBench tasks are multi-app.
  • F Family-Signature Plots: Figure 7 groups failed-rubric hits by failure mode and model family while also showing a per-model error budget.
  • F Family-Signature Plots: Figure 7 defines zero-score tasks as the top error-budget bar and highlights trajectories terminating under 20 steps or reaching at least 99 steps without success.The family groupings are Claude, OpenAI CUA, and Qwen.

G Detailed Failure Modes · H Persona Specification and Event Chains

The benchmark’s failures often arise from interacting errors, especially premature termination, skipped applications, and UI-side-effect omissions. Its personalized environment is generated from a canonical Michael Scott JSON persona whose cross-application event chains keep seeded records consistent.

  • G Detailed Failure Modes: Co-occurring failures make perfect-task success decline faster than rubric scores: one skipped application or premature-DONE error can zero the entire task.Surface errors can also trigger partial-artifact failures, such as opening but not saving a spreadsheet.
  • G Detailed Failure Modes: GPT-5.4 mini recorded 130 premature-DONE hits and GPT-5.5 recorded 105, exceeding either Claude model individually: Opus 28 and Sonnet 38.These counts were reported as more than six times either Claude model’s count.
  • G Detailed Failure Modes: Qwen 35B produced 13 of 31 persona-data-hallucination hits and 55 of 129 surface-error abandonments, or 43% of the latter.It had the highest single-model counts in both failure modes; Claude had 7 hallucination hits and GPT had 11.
  • G Detailed Failure Modes: Claude’s distinctive failure is a console-script shortcut: it reads application state through JavaScript consoles or REST requests instead of performing rubric-graded visible UI actions.This can satisfy rubrics requiring only knowledge of a value, but fails when a user-visible side-effect is required.
  • G Detailed Failure Modes: On 323 failed rubrics, agents completed multi-application tasks without opening a required app, including OddsMarket, TableFind, or a Dundies categories file.Examples include visiting nine apps while skipping TableFind and extensively searching HooliChat without reading the required file.
  • H Persona Specification and Event Chains: The canonical Michael Scott persona is one JSON document with sixteen top-level sections, and every seeded record in the 17-app environment traces back to one section.The sections include identity, contacts, financial data, routines, trips, work, tax information, contradictions, dependencies, browsing patterns, shopping, overrides, and cross-app events.
  • H Persona Specification and Event Chains: Cross-app consistency is driven by cross_app_events, planted_contradictions, planted_dependencies, and app_overrides, which encode side-effects, red herrings, required chains, and app-specific tuning.For example, a single dinner-plan event can seed reservation, charge, calendar, browser-history, and chat records across applications.
  • H Persona Specification and Event Chains: Shared event records let specialized seeders fan out correlated rows across web apps, calendars, browser history, and chat while preserving internal consistency.The Cooper’s Seafood House dinner event is written into TableFind, Gringotts, HooliCalendar, Firefox history, and a chat thread.

I Task-Review Interface · J Data Generation Pipeline · K Grading and Rubric Prompts

The paper provides a task-review interface for quality assurance, a deterministic pipeline that converts persona data into reproducible desktop images, and rubric prompts that evaluate individual task criteria from agent trajectories. Reviewers and graders operate on explicit task metadata, seeded environments, action histories, and screenshots.

  • I Task-Review Interface: The single-page reviewer groups tasks by primary application and displays each task’s instruction, difficulty, touched applications, and review state.Selecting a task opens rubric items and a deep-link to its live application inside the virtual machine.
  • I Task-Review Interface: Figure 8 shows each review row with the task identifier, review state, difficulty, instruction preview, and application pills.The interface supports quality-assurance review through a left-pane task list grouped by primary application.
  • J Data Generation Pipeline: A deterministic entry point converts one persona JSON document into a fully populated Linux desktop image by invoking dependency-ordered seeders.Each seeder consumes the same persona document, so adding a persona requires changing one file.
  • J Data Generation Pipeline: The web-app seeder writes SQLite databases for 17 Next.js applications and propagates cross_app_events into correlated records across multiple applications.Other seeders construct the on-disk Maildir, its HooliMail mirror, and calendar data with .ics files readable by LibreOffice.
  • J Data Generation Pipeline: Deterministic RNGs keyed by persona name and seeder identifier make identical inputs produce byte-identical outputs across runs and machines when the reference-time anchor is pinned.By default, reference time uses the bake date so seeded data appears current relative to the build.
  • J Data Generation Pipeline: A Docker build packages the populated home directory, Firefox profile, and 17 Next.js applications into the environment image, whose first-boot snapshot fixes every task’s initial state.Subsequent tasks start from the same QEMU guest snapshot.
  • K Grading and Rubric Prompts: Each grading call receives one rubric item, the user task for context, the full action history, and trajectory screenshots, then decides whether that rubric item is satisfied.The evaluator must judge only the specified criterion, rely on visible evidence, and mark failure when blocking conditions prevent satisfaction or later actions undo it.
  • K Grading and Rubric Prompts: The grading protocol requires a reasoning line citing steps or screenshots and a status of success or failure, with rubric outcomes aggregated using authored rubric weights normalized within each task.The instantiated prompt includes the rubric ID, requirement, action history, screenshot and step counts, and the required response format.

L Agent Harness … O Broader Impact

The released harness standardizes desktop interaction across six models while exposing substantial differences in UI-versus-bash behavior and bash’s association with success. Example trajectories and stated limitations frame the benchmark as a deep test of one personalized Linux environment, with broader relevance to assistants operating on personal data.

  • L Agent Harness: The harness launches Ubuntu 24.04 with a GNOME session and 17 Next.js applications, waits for desktop readiness, and iterates screenshot, model-action, and execution steps.Desktop startup typically takes ∼90 s; actions execute through the OSWorld-compatible Control API or the guest shell endpoint.
  • L Agent Harness: All six models receive the same computer+bash surface, but Claude uses a hybrid strategy dominated by UI actions and substantial bash use.For Opus, around 70% of actions use the UI and roughly 24% use bash; Sonnet shifts toward more clicking and less bash.
  • L Agent Harness: 69.0% of Opus trajectories succeed when bash is never invoked, compared with 52.9% when it is invoked; Sonnet shows 43.8% versus 38.2%.Bash appears in 155 of 184 Opus trajectories and 152 of 184 Sonnet trajectories, and the passage characterizes the gap as correlational rather than causal.
  • L Agent Harness: Hybrid agents need policies that select bash or UI actions according to the required user-visible side effects, because bash-only paths can fail such rubrics.The passage concludes that the tool alone is insufficient and that harness or prompt design must clarify when UI interaction is required.
  • M Example Trajectories: Figures 10 and 11 present six higher-fidelity trajectory vignettes: passing and failing runs from Claude, GPT, and Qwen, each with task text, three screenshots, and behavior notes.The examples include aggregation, hard_app, retrieval, long_horizon, and situated_action tasks, with verdicts assigned by an LLM-as-a-judge.
  • N Limitations: The benchmark prioritizes depth over persona diversity by evaluating one canonical Michael Scott persona on one Linux/GNOME/Firefox stack.Results measure deep use of one coherent personal computer rather than generalization across demographics, locales, or device stacks.
  • N Limitations: A single Gemini judge and structural failure-mode counts across six models limit how precisely the benchmark’s absolute failure prevalence should be interpreted.The benchmark’s grading and canonical environment constrain the scope of conclusions drawn from its failure counts.
  • O Broader Impact: Cross-app, cross-history personalization can expose assistants that appear competent on stock-state demonstrations but fail when operating on real personal data.The failure-mode catalog identifies behaviors such as premature DONE, surface-error abandonment, skipped apps, and hallucinated personal information for developers to measure and reduce.
Loading 2606.16748v1…