OSWorld-Pro: Process-based Evaluation for Computer Use Agents

• Outcome-based evaluation can give zero to trajectories that fail at the first or ninth subgoal and offers little step-level information about progress.Process-based evaluation assesses how agents reach outcomes, including partial reward for early success.Source: Figure 1 · p. 1, §1 · pp. 2–3
• OSWorld-Pro evaluates open-ended CUA tasks by breaking overarching goals into individually assessable, sequentially dependent subgoals.Earlier subgoals must be completed before later ones are attempted.Source: §1 · p. 3
• The benchmark contains over 67,000 human annotations across more than 2,800 progressive subgoals in over 300 long-horizon tasks.Source: §1 · p. 3
• Its tasks average 9.2 dependent subgoals, and Claude Opus 5 sometimes performs subgoal-irrelevant actions for over 50 steps despite completing all subgoals.Source: §1 · p. 3
• 75.7% vs. 83.4%: Claude Opus 5 scores lower on OSWorld-Pro than on OSWorld.Source: §1 · p. 3

• OSWorld-Pro contains 67,264 human-annotated labels across 305 tasks and 2,814 progressive subgoals.Source: §2 · p. 3
• Its 305 tasks comprise 117 Diversity tasks, 109 Coordination tasks requiring at least four applications, and 79 Robustness tasks across Linux distributions and graphical interfaces.Source: §2 · p. 3
• Each task includes agent trajectories and step-level labels for subgoal targeting, feasibility, progress, and completion; annotations took over 5,000 person-hours.Source: §2 · p. 3

Annotator Recruitment

• Annotators are recruited with relevant experience and qualifications, screened for English, Linux, and workflow skills, and supported by experienced reviewers.Across 305 tasks, 25 annotators and reviewers from five countries participated.Source: §3 · pp. 3–4

Task Curation

• Tasks cover underrepresented apps, coordination across at least four apps, and environments beyond Ubuntu/GNOME, including Fedora, Alpine, AlmaLinux, bspwm, Xfce, and LXQt.The distribution comparison reports 31 applications versus 13 in OSWorld, a mean of 3.45 versus 1.34 apps per task, and 19 Linux distributions plus 13 graphical interfaces versus Ubuntu Jammy with GNOME.Source: §3 · p. 4, Figure 3 · p. 4

Subgoal Decomposition

• Task goals are decomposed into atomic, independently assessable subgoals, each paired with a specific required application.DeepSeek-V4-Pro is prompted to generate the decomposition.Source: §3 · p. 4

Human Annotation

• Validation removes unclear or unsafe tasks and tasks with unrelated subgoals; annotators label targeted subgoals and feasibility, then mark progress and completion when feasible.A reviewer independently annotates before giving feedback, and annotators are prohibited from using LLMs.Source: §3 · pp. 4–5

4 CAN LLMS EFFECTIVELY JUDGE LIKE HUMANS DO?

• The judge identifies targeted subgoals, feasibility, progress, and completion for each trajectory step; evaluation compares these labels with human annotations.The study also measures agreement at step, subgoal, and task levels, with conditional scoring for dependent fields.Source: §4 · p. 5, §4.1 · p. 5

4.1 EVALUATION

• Independent annotators show strong agreement, with Cohen’s κ of 0.987 for targeting, 0.847 for feasibility, 0.869 for progress, and 0.940 for completion.Source: §4.1 · p. 6

4.2 RESULTS

• 61.9 vs. 91.6% on feasibility and 74.7 vs. 94.1% on progress: GPT-5.6-Sol Max trails humans most on these judgments.It approaches human performance on identifying targeted subgoals and comes within 3.3% absolute difference on completion measures.Source: §4.2 · p. 6
• 75.7 to 80.9 to 82.0: at maximum reasoning, step-level performance rises from GPT-5.6-Luna to Terra to Sol, while cost rises about 10x then 1.6x.The reported step-level metric is the mean of four Macro-F1 scores.Source: §4.2 · p. 6
• Feasibility=No predictions fall from 9.7% at GPT-5.6-Sol None to 5.1% at GPT-5.6-Sol Max, versus 15.3% in human labels; the authors link this to lower feasibility Macro-F1.Models generally perform better at higher reasoning effort across most metrics, with exceptions.Source: §4.2 · pp. 6–7

• 77.7% Overall is the best OSWorld-Pro score (Claude Opus 4.8 Max), below the top Opus model's 83.4% on OSWorld.The comparison indicates that OSWorld-Pro is more challenging than OSWorld for the top Opus model.Source: Table 2 · p. 7, §5 · p. 7
• Agents receive a goal in a Linux GUI environment they see through screenshots, generate a trajectory, and have feasible subgoals evaluated by GPT-5.6-Sol Max.The reported task-level measure is the percentage of tasks completing all subgoals deemed feasible by human annotators.Source: §5 · p. 7
• Open-weight models score highest on Diversity, then Coordination, and lowest on Robustness.Robustness tasks test generalization across Linux distributions and graphical interfaces.Source: §5 · p. 7
• No obvious evidence shows parameter count alone predicts OSWorld-Pro performance; smaller models typically take more steps, which may partly compensate for capacity.The authors also note that training recipes could influence performance more than model size alone.Source: §5 · p. 8
• 20x: similar-performing GPT-5.6 Luna costs $0.51 per task versus $9.56 for GPT-5.6-Sol, while low-step models can use many output tokens.The paper distinguishes step count and output tokens as latency considerations from token cost for cost-sensitive users.Source: §5 · p. 8

• Process-based evaluation reveals trajectory-level efficiency and failure information that outcome-based evaluation like OSWorld cannot.Source: §6 · p. 8
• 39.0-41.9% vs. 72.4-92.5%: Minimax M3 and Kimi K3 have lower click-progression likelihoods than stronger models, with Minimax M3's coordinate errors illustrated in Figure 4.The likelihood counts progress steps targeting at least one feasible subgoal; action categories observed fewer than five times are excluded.Source: §6 · p. 8, Figure 4 · p. 9
• Opus 4.8 improves Click (86.3 to 92.5%) and Drag+Move (41.0 to 74.3%); Claude models score ≥89.2% on Keyboard and ≥84.1% on Scroll.Source: §6 · p. 9
• 4.9 and 5.1 steps/subgoal: GPT-5.6-Sol and Kimi K3 show lower Successful Effort than Sonnet 5 (9.9) and Minimax M3 (21.2).For Infeasible Persistence, Opus 5 (9.5) and GPT-5.6-Terra (12.1) spend fewer steps than Sonnet 5 (39.8).Source: §6 · p. 9

7 CONCLUSION

• The benchmark is presented as the first process-based evaluation for Computer-Use Agents, complementing outcome-based evaluations such as OSWorld.It is based on over 67,000 human annotations.Source: §7 · p. 9
• OSWorld-Pro reveals failure modes missed by outcome-based evaluation, including subgoal-irrelevant behavior and click-based operations.The authors suggest future applications to improving CUA performance and efficiency through harness optimization or process reward signals in reinforcement learning.Source: §7 · pp. 9–10

ETHICS STATEMENT

• The vendor reviewed ethical and legal standards before data collection and provided transparent pay, timely payment, reasonable working hours, and break guidance.Annotators also had confidential escalation paths and the option to remove work content that could pose undue risk without additional safeguards; compensation followed applicable local labor laws and internal worker-protection standards.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

• Data-collection procedures are documented in §3, B and C, while evaluation details are in §4.1, 5 and E.Source: REPRODUCIBILITY STATEMENT · p. 11

A EXAMPLE DATA

• Figure 7 is identified as an OSWorld-Pro robustness example.Source: Figure 7 · p. 19

B FURTHER DESCRIPTIVE STATISTICS

• The 305 tasks average 9.2 subgoals each, and trajectories average 55.1 steps.Task goals average 553.2 characters; trajectories range from 5 to 149 steps.Source: B FURTHER DESCRIPTIVE STATISTICS · p. 19
• Across annotated trajectories, 23.1% of subgoals were unattempted and 65.3% were completed.Completed subgoals averaged 6.2 steps, compared with 18.7 steps for progressed but incomplete subgoals.Source: B FURTHER DESCRIPTIVE STATISTICS · p. 20

C ANNOTATOR RECRUITMENT

• Recruitment distinguished technical and non-technical annotator pools, with education backgrounds reported for each.The technical pool included 55% with a Bachelor of Engineering; the non-technical pool included 40% with that degree.Source: C ANNOTATOR RECRUITMENT · p. 20

D ANNOTATION GUIDELINES

• The guidelines separate goal validity, subgoal-decomposition quality, and step-level targeting, feasibility, progression, and completion.They define procedures for evaluating trajectories at goal, subgoal, safety, and per-step levels and for comparing reviewer and annotator judgments.Source: D ANNOTATION GUIDELINES · p. 20

D.1 TERMINOLOGY

• The terminology defines a subgoal as a separately checkable milestone contributing to the overall goal, and a step as one trajectory action unit.Source: §D.1 · p. 21

D.2 REQUIRED EVALUATION ORDER

• Annotators score goal clarity, specificity, and safety first, proceeding only at 5/5; otherwise they record the reason and skip the task.They then review the subgoal list, its one-to-one App Used mapping, and sequential dependency before labeling each step.Source: §D.2 · p. 21

D.3 TASK FILTERING

• Tasks proceed only when the goal is clear, precise, and safe; a score below 5 leads annotators to give a reason and skip the task.The highest clarity label requires an explicit goal that fully specifies the intended outcome.Source: §D.3 · p. 21, §1 · p. 21

D.4 SUBGOAL LIST AND APP USED CORRECTNESS (BOOLEAN)

• Subgoals must align with the goal, avoid overlap, cover it collectively, map one-to-one to apps, and appear in matching list order.The task sequence must also follow the logical execution order, with later subgoals depending on earlier ones.Source: §D.4 · p. 21, §D.5 · p. 22

D.6.1 SUBGOAL TARGETED (MULTI-SELECTION OR NONE)

• Each step is labeled for the subgoal it targets and whether that target is feasible in the current environment.Feasibility can be blocked by application, system, resource, or sandbox limitations.Source: §D.6.1 · p. 22, §D.6.2 · p. 22

D.6.3 SUBGOAL PROGRESSION (BOOLEAN)

• Progression requires observable, meaningful advancement; verification generally counts only when it materially changes evaluation or the next action.A partial advance can have Progression = Yes while Completion remains No; completion is marked only on the first step that fully achieves the target.Source: §D.6.3 · p. 23, §D.6.4 · p. 23

D.6.5 DEPENDENCY QUICK REFERENCE

• The dependency reference distinguishes no target, infeasibility, partial advance, first full achievement, redundant verification, and post-completion cleanup by their required labels.For example, partial advancement is Progress = Yes and Completion = No, while first full achievement is Yes for both.Source: §D.6.5 · p. 24

D.7 REVIEWER WORKFLOW

• Reviewers first annotate blindly, then compare against annotator labels; disagreement above 5% sends the task back for rework.Disagreements are consolidated by step across the labeling parameters.Source: §D.7 · p. 24

D.9 WORKED EXAMPLE

• The worked example decomposes a presentation-editing goal into app-specific milestones, while the prompt template requires each subgoal to have one binary-assessable objective and one app.A click can advance slide creation without completing that subgoal unless it first fully creates the required slide.Source: §D.9 · p. 25, E PROMPT TEMPLATES · p. 25

F LLM JUDGE

• The judge design moves from O(k*m) parallel calls to O(k) sequential calls with memory, then to one O(1) request containing the trajectory.The single-request approach was limited to a small set of models able to process the large screenshot payload; benchmarking used one run per model and up to 250 turns.Source: F LLM JUDGE · pp. 25–26, G INFERENCE SETUP · p. 27