PaperlayerView exact revision on arXiv

Source-linked AI summary

OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

Qiushi Sun, Kanzhi Cheng, Yian Wang, Bowen Yang, Hang Yan, Liheng Chen, Fangzhi Xu, Zichen Ding, Nuo Chen, Jialin Cao, Xingdong Gong, Zehao Li, Kaiming Jin, Xinfeng Yuan, Zhoumianze Liu, Jingyang Gong, Zhangyue Yin, Jiahui Gao, Zhiyong Wu, Tianbao Xie, Jianbing Zhang, Ben Kao, Lingpeng Kong

arXiv:2607.28609v2cs.AIcs.CLcs.CV

TL;DR

VLM judges for computer-using-agent trajectories have not been systematically validated for reliability across platforms. OSReward benchmarks them with human-gold data, finds systematic leniency and costly reliable judges, and introduces OS-Shepherd models that match commercial judges at 30–60× lower cost.

  • Problem

    Whether VLM judges reliably verify computer-using-agent trajectories across platforms remains largely untested, despite their use in evaluation, curation, and reinforcement learning.

  • Method

    OSReward evaluates VLM judges on human-gold, cross-platform trajectories, while OS-Shepherd-100K supports training open reward models targeting false-success judgments.

  • Results

    VLM judges share a leniency bias and collapse on hard cases, while OS-Shepherd models approach reliable judges at 30–60× lower cost.

  • Takeaways & Limitations

    OS-Shepherd provides an open, low-cost reward signal that remains de-biased on benchmarks unseen during training.

Abstract

from arXiv · show

Computer-using agents (CUAs) are advancing rapidly across the digital world. A CUA trajectory records the agent's actions, states, and reasoning. Verifying whether it fulfilled the task instruction is central to CUA evaluation, data curation, and reinforcement learning. Neither human-written verifiers nor human annotators can provide such verification at scale, so the field increasingly turns to vision-language models (VLMs) as judges of CUA trajectories. But a fundamental question has long gone unexamined: are these VLM judges reliable enough? To study it systematically, we introduce OSReward, a realistic, high-quality benchmark that evaluates VLM judges on CUA trajectories. The trajectories come from diverse agent backbones executing human-verified instructions across platforms, and are then rigorously labeled with ground-truth verdicts through multi-stage human annotation. Building on it, we derive OSReward-Hard, a challenge set concentrating genuinely hard cases, and OSReward-Multi for fine-grained efficiency and alignment scoring. The most comprehensive evaluation of VLM judges to date finds even state-of-the-art models fall short of an ideal judge, sharing a systematic leniency bias that mislabels failed runs as successes. The few reliable enough to trust are too expensive to run at scale, while affordable open models trail far behind. To close this gap, we construct and release OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments for the CUA community. On it, we train OS-Shepherd (9B and 35B), open reward models that supply low-cost, stable, and reliable reward signals, matching commercial judges at 30-60x lower cost than the frontier. Extensive analyses further inform the design of reliable CUA reward at scale. Our code, benchmark, dataset, and model checkpoints are available at https://os-copilot.github.io/OSReward-Home/.

1. Introduction

OSReward establishes a standardized benchmark for testing whether VLM judges reliably verify cross-platform computer-use trajectories. Its evaluation exposes systematic leniency toward false successes and motivates OS-Shepherd, open reward models designed for affordable, scalable judging.

  • Motivation: CUA progress requires verifying whether trajectories’ interleaved actions, states, and reasoning fulfill task instructions, but human verification cannot scale.The paper frames trajectory verification as necessary for evaluation, data curation, and reinforcement learning.
  • Motivation: Judging CUA trajectories remains largely untested because judges must determine whether the environment reached the instructed goal from long, interleaved records.This differs from judging text or general multimodal agents, where the relevant evidence and decision structure differ.
  • OSReward: OSReward evaluates VLM judges using human-gold trajectories from dedicated cross-platform infrastructure, including web, mobile, Ubuntu, and Windows environments.The benchmark also includes OSReward-Hard for difficult cases and OSReward-Multi for fine-grained judging.
  • Evaluation: On OSReward-Hard, the best judge falls below 70% accuracy and the mean judge reaches 52%, with errors dominated by accepting false successes.Judges accept runs where agents claim completion despite actually failing more often than the reverse.
  • Open reward models: OS-Shepherd-100K is an open corpus of 100K reasoning-annotated trajectory judgments used to train OS-Shepherd-9B and OS-Shepherd-35B.The two-stage training recipe first builds accurate judging and then directly targets false successes.
  • Open reward models: OS-Shepherd models approach reliable judges on the cost-accuracy frontier at 30–60× lower cost, enabling training-scale reward signals within an academic budget.The paper further analyzes input ablations, robustness, and ensembling to characterize when and why judges fail.

2. Related Work

Related work situates computer-using agents as vision-language-driven systems that perceive digital environments and act across real applications. It also highlights trajectory verification as a scalable but substantially harder judging problem than text or general multimodal responses.

  • Computer-using agents perceive digital environments and execute actions such as clicks, typing, and CLI commands to complete user tasks.
  • Vision-language-driven CUA research has progressed from grounding instructions on raw screens to native GUI action models operating real applications end to end.
  • Evaluating and training CUAs require deciding whether trajectories fulfilled instructions at a scale beyond human-written verifiers or annotation, making model-based judging practical.
  • CUA trajectory judging is more complex than evaluating text or general multimodal responses.

3. OSReward

OSReward builds a realistic cross-platform corpus of CUA trajectories with trustworthy human-verified verdicts, using richly initialized environments, grounded instructions, diverse agent backbones, and multi-stage annotation. It releases nested full, Hard, and Multi evaluation sets for broad, difficult-case, and fine-grained judgment analysis.

  • Data collection: OSReward collects fresh trajectories on custom cross-platform infrastructure and pairs them with gold verdicts produced through careful human labeling.The benchmark is built end-to-end rather than reusing noisy existing rollouts.
  • Environment preparation: Richly initialized desktop, mobile, web, Windows, and Ubuntu environments provide realistic files, accounts, applications, databases, distractors, and mixed action spaces.These states create concrete opportunities for agents to succeed or fail through environment changes rather than merely narrating completion.
  • Task instructions: Annotators explore initialized environments to write grounded, answerable instructions that include open-ended tasks beyond rule-based verification.Only the human-vetted, peer-screened instruction set enters the benchmark, while the same infrastructure supports larger-scale training data collection.
  • Execution and annotation: Each instruction is executed by diverse agents spanning Claude, Gemini, Kimi, and Qwen backbones, then surviving trajectories are reviewed in full multimodal context by three independent annotators.Disagreements escalate to two senior reviewers for deliberative meta-review, producing final gold judgments without relying on a single reader or majority vote.
  • Evaluation sets: OSReward-Hard contains 284 genuinely difficult trajectories, while OSReward-Multi adds alignment and efficiency labels to 440 successful trajectories.Hard cases are re-verified and drawn mostly from annotator disagreements; Multi uses alignment levels 0.5 and 1.0 and efficiency levels 0, 0.5, and 1.0.

4. Benchmarking VLMs as Judges

Across 27 VLM judges, only frontier systems approach the accuracy needed for training, while open models remain less reliable and judges systematically accept incomplete runs as successful. Performance deteriorates sharply on OSReward-Hard, and multi-axis ratings reveal discrimination but poor calibration.

  • Evaluation setup: 27 VLM judges use a common protocol: five final states with reasoning and actions, no tools or task-specific harness, and success/fail verdicts.OSReward-Multi additionally elicits alignment and efficiency ratings.
  • Performance ceiling: Claude-Opus-4-8 reaches 89.7% accuracy, while closed judges generally lead open-weight models and the largest open models remain within 4 pp of the lead.The ranking changes under balanced accuracy, so the table identifies a top tier rather than one universally best judge.
  • Judge bias: Most judges are lenient, combining high success recall with low fail recall, whereas only a few sit near the balanced diagonal.Strict judges such as GPT-5.2 and Claude-Haiku trade success recall for catching more failures.
  • Judge bias: Over-accepting incomplete tasks constitutes two-thirds of all errors and at least 48% of each judge’s mistakes, with pooled over-accepts outnumbering over-rejects three to one.The ratio narrows to about two to one for the strongest judges.
  • Hard-set evaluation: OSReward-Hard lowers every judge by 20–43 pp; the best reaches 69.7%, the mean falls to 52%, and fail recall ranges from near zero to 77%.The challenge set uses a 30/70 success/fail split, making raw accuracy misleading and exposing false successes from completion-claiming narratives.
  • Fine-grained scoring: On OSReward-Multi, judges generally rank alignment and efficiency better than they score them, indicating discrimination with miscalibrated ratings, especially for alignment.Each axis is evaluated using macro-recall and threshold-free AUC.

5. Analysis

The analysis finds that text history drives judge verdicts far more than visual settings, while extra thinking and ensembling provide limited reliability gains. Stronger judges improve accuracy, but their costs make reliable reward labeling difficult to scale.

  • 5.1 Visual input: Visual perturbations change aggregate accuracy by less than half a point, yet each setting still flips 5–7% of individual verdicts.Trailing-screenshot sweeps vary judges by only two to three points without trend, and removing the red click marker does not hurt accuracy.
  • 5.2 Text input: Removing per-step thought and action text costs 7.2 pp on average and flips 22.7% of verdicts, while removing only chain-of-thought costs 1.8 pp and flips 11.6%.The effect is several times larger on the web, where typed strings convey intent unavailable from recent screenshots.
  • 5.3 Model execution: Extra reasoning helps weakest judges but yields almost nothing at the frontier, while resampling at T=0.7 flips 6–9% of labels despite stable aggregate accuracy.These results indicate that additional deliberation recovers under-thinking rather than improving the strongest judges, and individual labels remain unstable.
  • 5.3 Model execution: Top judges have pairwise Cohen’s κ≈0.71, so majority voting mostly repeats shared mistakes; a top-3 vote improves the best judge by about a point at several times the cost.An oracle accepting any pooled judge’s correct verdict reaches 99% accuracy, showing that the pool often contains a correct answer without identifying it reliably.
  • 5.4 Cost of reliability: On OSReward-Hard, Claude-Opus-4-8 reaches 69.7% at ∼$100 and GPT-5.5 67.3% at $45, whereas the best sub-$3 judge reaches 57.0%.The full-set trade-off is milder—about 3 pp for a 42× price cut—but high-reliability judging remains too expensive for training-time reward.

6. OS-Shepherd: An Open Reward Model

OS-Shepherd is an open-weight reward model trained on OS-Shepherd-100K, a high-agreement corpus designed to reduce judges’ leniency and emphasize grounded failures. The 9B model delivers reliable hard-case judging at substantially lower cost, while scaling to 35B yields limited additional accuracy gains.

  • Corpus construction: OS-Shepherd-100K retains 96.6K reasoning-annotated trajectory–response pairs after high-agreement filtering across diverse CUA platforms.The corpus varies judge models and screenshot settings, excludes ambiguous middle cases, and deliberately over-collects grounded failures.
  • Corpus construction: The label-provenance strategy uses diverse strong judges and near-unanimous agreement to avoid propagating shared leniency bias into the corpus.Trajectories are retained only when strongest judges show no dissent, preventing forced-majority labels.
  • Training: SFT on OS-Shepherd-100K lifts OS-Shepherd-9B far above Qwen3.5-9B, primarily by correcting the base model’s near-total leniency.A subsequent GRPO-based RL stage mines residual false-success cases that remain harmful for reward modeling.
  • Evaluation: 57.6% of hard false successes are caught by OS-Shepherd-9B on OSReward-Hard, while it remains on the balanced diagonal and reaches the commercial band on the full set.This combination is described as rare among judges, with only frontier models matching it at many times the cost.
  • Evaluation: 2.4 pp of hard-set balanced accuracy is gained by OS-Shepherd-35B-A3B over 9B, with no full-set improvement despite four times the parameters.Both models retain the same balanced-diagonal behavior, while the 35B grades quality axes noticeably better.
  • Efficiency: 51,200 judge calls cost about $4,000 with Claude-Opus-4-8 or $2,300 with GPT-5.5, versus about $68 with OS-Shepherd-9B.This represents a 30–60× reduction in judging cost at training scale.

7. Generalization to Existing Benchmarks

Across three independently built CUA benchmarks, VLM-judge agreement with human-written verifiers varies more by platform than judge, with desktop remaining particularly difficult. OS-Shepherd transfers its leniency resistance out of distribution, emerging as the strongest open judge and narrowing the gap toward a scalable reward signal.

  • Evaluation setup: Three prior-work benchmarks are evaluated using their own human-written verifiers to test whether OSReward’s findings and OS-Shepherd’s de-biasing extend beyond the training data.The evaluation uses the same judges on held-out benchmark tasks.
  • Benchmark agreement: Agreement varies more by platform than by judge: the best judges approach ∼90% on mobile, come within about 6 pp on web, and fall well short on desktop.∼90% agreement is treated as the threshold for replacing a benchmark’s human-written verifier.
  • Open-judge performance: OS-Shepherd models are the best open judges on OSWorld and AndroidWorld and join the frontier cluster on WebArena, beating every general open model up to Qwen3.5-397B on all three.Qwen3.5-397B is approximately 44× the size of OS-Shepherd-9B.
  • Failure detection: The transferred capability is de-biasing: OS-Shepherd catches substantially more true failures than general Qwen judges where successes dominate and remains tightly calibrated when WebArena’s base rate reverses.General Qwen judges remain lenient from 30B to 397B, whereas OS-Shepherd-9B’s untuned Qwen3.5-9B base falls below WebArena’s all-fail baseline.
  • Toward a scalable verifier: A single 9B model trained on none of the three benchmarks transfers its de-biasing across them, narrowing the gap to per-task human-written verifiers at a fraction of frontier scale.It does not yet match frontier accuracy but operates at a cost that scales to training-time reward.

8. Conclusion · Appendix Contents

OSReward evaluates the reliability of VLM judges for computer-using-agent reward signals using human-gold trajectories across four platforms. Its findings motivate OS-Shepherd, whose open corpus and models target false successes while matching commercial judges at much lower cost and remaining de-biased on unseen benchmarks.

  • 8. Conclusion: OSReward measures the previously unexamined reliability of VLM judges using human-gold trajectories collected across four platforms.The benchmark was released in full.
  • 8. Conclusion: The dominant judge failure is accepting incomplete tasks as successes.These verdicts follow the agent’s text history more than the screen.
  • 8. Conclusion: On OSReward-Hard, judge performance drops to near chance.Judges that remain reliable cost too much for training.
  • 8. Conclusion: OS-Shepherd combines an open corpus with open reward models to address the identified judge failures.Its reinforcement-learning stage directly targets the false-success mode.
  • 8. Conclusion: The trained OS-Shepherd models match commercial judges at a small fraction of frontier cost.The resulting reward signal can run at training scale under an academic budget.
  • 8. Conclusion: OS-Shepherd remains de-biased on benchmarks it never trained on.This supports using the models as reward signals beyond their training benchmarks.

A. Data-Collection Infrastructure … B.1. Annotators

The paper uses a shared cross-platform collection workflow with platform-specific environments, instructions, executing agents, and verification. It also grounds gold verdicts in direct environment familiarity among trained annotators and expert meta-reviewers.

  • A. Data-Collection Infrastructure: All platforms follow a common pipeline of realistic environments, grounded instructions, agent rollouts, and automatic verification, with platform-specific implementations described separately.Executing agents are model backbones driving each platform’s action space through its own harness and a ReAct-style loop.
  • A.1. Web: Web collection uses isolated headless-Chromium Playwright sessions at 1920 × 1080, with screenshot- and coordinate-driven interaction rather than DOM identifiers.A stealth plugin reduces automation blocking, and coordinates are normalized to viewport pixels.
  • A.1. Web: Web instructions combine pre-filtered existing task pools with an OS-Genesis split, while parallel isolated workers collect trajectories without shared browsing state.The OS-Genesis trajectories are re-collected with Gemini-3-Flash on self-hostable sites.
  • A.2. Windows: The Windows environment covers twenty-three everyday applications and command-line tools, while instructions grow through a human-seeded data flywheel that filters for completability and balanced difficulty.Later rounds generate grounded instructions from consecutive prompt–screenshot steps using Gemini-3.1-Pro, followed by embedding and clustering.
  • A.3. Ubuntu: Ubuntu spans about thirty everyday and professional applications with CLI and Python tooling, and automatically generates training-scale instructions containing constrained difficulties and machine-checkable verifiers.Benchmark trajectories use annotator-written, peer-checked instructions that are rolled out and human-verified as gold.
  • A.3. Ubuntu: Ubuntu trajectories are collected from Claude-Sonnet-4.6, Claude-Opus-4.6, Qwen3-VL-235B, and Gemini-3.1-Pro under a 50–80-step cap with GUI+CLI actions.Pure-GUI training trajectories also come from open-source datasets, while the benchmark split is based on human-verified instructions.
  • A.4. Android: Android extends AndroidWorld on Pixel 6a with additional everyday apps, scripted initial states, synced accounts, and distractor content; its benchmark filters 415 raw instructions to 226 high-quality prompts.The benchmark includes native-app, newly added single- and multi-app, and explicit-negative-constraint task categories.
  • B.1. Annotators: Gold verdicts come from six computer-science graduate annotators and three experienced CUA-researcher meta-reviewers who first interacted directly with assigned environments.Labeling uses a custom platform that replays trajectories, grounding judgments in application capabilities.

B.2. Annotation Pipeline and Statistics … C.2. Judging Prompt

The benchmark construction combines screened instructions, multi-agent collection, and rigorous re-verification, while its evaluation uses standardized judging inputs and separates binary success from alignment and efficiency scoring. OSReward-Multi applies distinct rubrics to completed tasks, with alignment labels covering 439 of 440 trajectories after removing one safety concern.

  • B.2. Annotation Pipeline and Statistics: Approximately 1,500 candidate instructions were authored and cross-checked, with about 800 passing peer screening for machine collection.Each instruction was rolled out by one to three executing agents; live-site instability, anti-bot checks, crashes, and unresponsive collection systems introduced noise unrelated to agent ability.
  • B.2. Annotation Pipeline and Statistics: Of 373 OSReward-Hard candidates re-examined, 284 were kept, 89 returned to the full set, and 18 trajectory labels were corrected.The review adjusted the subset for genuine difficulty and target platform and success/fail composition.
  • B.3. Failure-Type Taxonomy: Every failed trajectory receives one or more failure-type tags, including reasoning-and-planning errors such as faulty decomposition, premature termination, and repetitive loops.Extended trajectories may accumulate multiple errors, and the taxonomy attributes these failures to the agent’s thinking despite access to the full history.
  • B.4. OSReward-Multi Scoring Guideline: Alignment and efficiency are scored only on successful trajectories, with each axis defined by a three-level rubric.Intent alignment evaluates whether actions serve the user’s true intent without irrelevant interactions, constraint violations, or unsafe behavior; efficiency evaluates path conciseness and optimality.
  • B.4. OSReward-Multi Scoring Guideline: 439 of 440 trajectories receive alignment labels after the single 0-scored run was flagged as a safety concern and removed.The remaining labels comprise 59 at 0.5 and 380 at 1.0; state-neutral exploratory actions are scored under efficiency rather than alignment.
  • C. Experimental Details / C.1. Evaluated Models: All 27 reference judges and OS-Shepherd models use the identical main setting: a fixed prompt, last five screenshots, full text history, and greedy decoding.Accuracy is computed across all 1,019 trajectories, while recall uses only successfully scored trajectories.
  • C.2. Judging Prompt: The shared judging prompt presents the task instruction and interleaved trajectory record, then requests a success/fail verdict with a brief justification.The multi-axis variant additionally rates alignment and efficiency, whereas the binary variant omits those fields.
  • C.1. Evaluated Models / C.2. Judging Prompt: Table 9 enumerates the 27 reference judges and OS-Shepherd reward models, including their API identifiers and extra thinking or reasoning-effort levels.The reference judges are ordered by full-set accuracy, and access classes are reported in Table 1.

C.3. Judge Error Categories · D. OS-Shepherd · D.1. OS-Shepherd-100K

The paper categorizes judge failures into over-accepts and over-rejects, then develops OS-Shepherd from a large, contamination-screened corpus of human-annotated trajectory judgments. OS-Shepherd-100K combines broad platform coverage with staged training data to support stable, scalable reward modeling.

  • C.3. Judge Error Categories: Judge errors are manually verified after initial VLM labeling and divided into over-accepts of failed runs and over-rejects of successful runs.Over-reject subtypes include strictness on path, deviation, and ambiguous completion.
  • C.3. Judge Error Categories: 0.6% of label errors invoke the blocked rule and are excluded from Fig. 7, although transiently blocked trajectories can still be misjudged as failures.Persistently blocked runs are filtered, but valid runs may contain temporary blocks that agents bypass.
  • C.3. Judge Error Categories: The judging protocol distinguishes successful task completion from explicit-answer requirements and separately scores alignment and efficiency only when the verdict is SUCCESS.For answer-seeking instructions, the agent must explicitly output the required answer; otherwise, reaching the intended state suffices.
  • D. OS-Shepherd: OS-Shepherd-9B and OS-Shepherd-35B-A3B use Qwen3.5 bases, the identical corpus and two-stage recipe, and the same main setting as 27 reference judges.An initial Qwen3-VL base was less stable and scored lower, motivating standardization on Qwen3.5.
  • D.1. OS-Shepherd-100K: 96K samples support SFT with 58.7% success and 41.3% fail, while roughly 3.1K curated samples are reserved for RL.The SFT set comes from 70K unique trajectories retained by an agreement filter.
  • D.1. OS-Shepherd-100K: The corpus spans over 335K screenshots, with median trajectory length 12 steps, p90 = 25, and max 131, across desktop, web, and mobile environments.Platform shares are desktop 50.5%, web 44.2%, and mobile 5.4%; output formats are 38.7% single and 61.3% rubric.
  • D.1. OS-Shepherd-100K: No training trajectory originates from a benchmark run, and an embedding screen with cosine similarity above 0.8 flags no overlapping instruction pair.The corpus draws from 321,631 judge instances over eight sources, with nothing from any existing benchmark test set.

D.2. Training Details · E. Additional Results and Analysis

OS-Shepherd is trained with agreement-filtered supervised fine-tuning followed by reinforcement learning focused on recoverable false-success errors. Additional appendix materials document the training configuration and analyses referenced elsewhere in the paper.

  • D.2. Training Details: SFT uses 96.6K agreement-filtered, platform- and label-balanced samples, retaining at most two output-format samples per trajectory.When available, the Gemini-3.1-Pro response is retained if it agrees with the final label; otherwise, another agreeing judge response is used.
  • D.2. Training Details: RL targets the dominant residual error mode, false successes, which fit visibly worse during SFT and reflect genuine judging difficulty rather than label noise.The RL stage therefore trains on recoverable errors from the SFT model.
  • D.2. Training Details: A single short GRPO pass uses 2.9K/0.2K train/validation trajectories, batch size 16, learning rate 1e−6, and ∼150 steps.Rollouts use T=1.0, top-p1.0, eight samples per example, prompts up to 24,576 tokens, and responses capped at 512; the vision tower remains frozen.
  • D.2. Training Details: The de-biasing trajectory moves OS-Shepherd-9B from a lenient corner toward the balanced diagonal, with RL producing the largest hard-set improvement.This progression is shown for base, SFT, and SFT+RL.
  • D.2. Training Details: The training configuration is shared across both OS-Shepherd sizes for SFT, while their RL runs share the mined set and differ only in the base checkpoint.The appendix summarizes this configuration in Table 13.
  • D.2. Training Details: The RL objective applies a token-level KL loss to the SFT reference with coefficient 0.001 and no entropy bonus, while selecting the best-validation checkpoint.The policy emits a plain thought followed by a verdict, and the reward compares the verdict with the agreement label.
  • D.2. Training Details: OS-Shepherd-9B reaches the cheap-and-accurate corner at ∼$1.36, about one thirtieth of frontier cost, in the full-set accuracy comparison.The OSReward-Hard frontier is reported separately in the main text.
  • E. Additional Results and Analysis: The appendix collects additional results and analysis figures referenced from §4 but deferred for space.These materials constitute the additional-results section.

E.1. Per-Task Breakdown

This section directs readers to per-platform, failure-type, and benchmark breakdowns, while documenting OSReward-Hard candidate selection and comparisons of cost and accuracy. It also notes that Table 14 compares OS-Shepherd-9B’s full-set accuracy tier with two frontier judges.

  • Breakdowns: OSReward-Hard breakdowns cover platforms and failure types, with online-benchmark results reported separately.These analyses appear in Figures 6, 8, and 11, plus §E.5.
  • Candidate selection: OSReward-Hard candidates mostly come from gold trajectories that annotators split on, and a meta-reviewer re-examined every candidate before inclusion.The passage points to §3.3 for the inclusion process.
  • Cost and accuracy: Figure 15 presents the full-set cost-accuracy view using official API list prices where available and May 2026 market-rate estimates for comparable-scale open-weight models.The estimates apply to open-weight models lacking official pricing.
  • Cost and accuracy: Table 14 compares OS-Shepherd-9B’s full-set accuracy tier with two frontier judges.The table specifically tabulates the accuracy-tier comparison.

E.2. Multi-Axis Evaluation (Auxiliary) … F. Case Studies

The appendix defines multi-axis evaluation and shows that judge reliability depends on calibrated scoring, cross-family agreement, verifiability, and reasoning stability. Case studies illustrate false successes and failures caused by unsupported recency claims, ignored evidence, misperception, semantic errors, and long-horizon breakdowns.

  • E.2. Multi-Axis Evaluation (Auxiliary): OSReward-Multi scores positive trajectories on alignment and efficiency, with judges emitting these axes only after predicting success.Alignment captures out-of-scope side effects, while efficiency measures path conciseness; failed verdicts forfeit both axes.
  • E.2. Multi-Axis Evaluation (Auxiliary): Multi and AUC average the two axes, with macro-recall measuring thresholded level accuracy and AUC measuring threshold-free pairwise ranking.A judge emitting one constant level scores exactly 50.0 on both metrics.
  • E.3. Open-Weight versus Closed-Source Judges: OS-Shepherd-9B reaches the closed-source mean and surpasses every open-weight field judge despite its smaller size.The broader gap is mainly caused by weaker smaller open VLMs, while leading open judges already closely trail the frontier.
  • E.4. Inter-Judge Agreement: Top judges agree at κ≈0.71 across families, indicating that they struggle on the same hard trajectories rather than benefiting from generic diversity.Within-family binary κ is 0.731 versus 0.709 across families, so useful diversity must be accuracy-matched.
  • E.5. Verifiability and False Positives: 88% of OSWorld judge errors are false positives, while accuracy drops from 0.76 to 0.57 and false-positive rate rises from 0.20 to 0.37 beyond 16 steps.Errors concentrate in unverifiable application domains and long trajectories because the final five screenshots cannot confirm completion.
  • E.6. Thinking, Screenshot Count, and Self-Consistency: Thinking improves every paired model, but gains decline monotonically from +2.8 pp for the weakest judge to +0.4 pp for the strongest.The result suggests explicit reasoning mainly compensates for weaker deductive capacity and offers marginal benefit for frontier models.
  • E.6. Thinking, Screenshot Count, and Self-Consistency: No judge shows a directional accuracy trend with trailing screenshot count, while N=5–9 is a near-best band and GPT-5.5 has the steadiest self-consistency.At T=0.7, aggregate accuracy is stable but per-trajectory flip rates vary 1.5× across judges.
  • F. Case Studies: Case studies expose false successes and hard failures where judges overlook missing terminal evidence, misread values, accept contradictory spreadsheet outcomes, or mistake incorrect long-horizon artifacts for completion.The appendix includes a human-labeled fail accepted by multiple reference judges, alongside recency-grounding, perception, semantic-outcome, planning, and challenge-subset examples.
Loading 2607.28609v2…