AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use Tasks

1 INTRODUCTION

• Long workflows can scatter relevant evidence across hundreds of screenshots and action logs, while reviewing trajectories manually is expensive and scripts are difficult to specify for tasks with multiple valid outcomes.Direct judges receive a fixed representation; agentic judges can use tools to filter and review observations, but their reliability and the value of that flexibility remain unclear.Source: §1 · pp. 1–2
• A plausible-looking trajectory may violate a constraint, such as drawing a pentagon instead of the requested triangle, despite later recoloring and export.Judges must distinguish proper task completion from a trajectory that merely terminates successfully.Source: §1 · p. 2
• AgentHorizon pairs human-recorded trajectories with closely related instructions, creating negatives by swapping instructions to test whether judges detect mismatches.The benchmark contains 523 positive and 850 negative trajectories reviewed independently.Source: §1 · p. 2
• The benchmark provides AH for challenging tasks, AH-S for simpler tasks, and AH-D for developing judges while avoiding repeated evaluation on the main splits.Source: §1 · p. 3
• Across eleven models and five harnesses, the strongest agentic judge achieves 80.9% balanced accuracy on AH, while tool access does not consistently improve performance.GPT-5.5 benefits from Codex over direct prompting, whereas Qwen 3.6 27B performs better with direct prompting than with OpenCode.Source: §1 · p. 3

Evaluating computer-use agents

• Computer-use benchmarks cover web, enterprise, and operating-system tasks, while other work identifies evaluator errors and studies automated safety judging.AgentHorizon instead evaluates judges against reviewed trajectory labels.Source: §2 · p. 3

LLM judges

• LLM-judge research studies bias and preference evaluation; AgentRewardBench reports below-70% precision for tested judges on web-agent traces, whose failures arise in agent executions, unlike AgentHorizon’s paired human demonstrations.Other work surveys long-horizon assessment and proposes tools for workspace inspection.Source: §2 · pp. 3–4

Trajectory and process evaluation

• Trajectory and process-evaluation work grades research replication, repositories, web tasks, intermediate steps, and repeated-trial reliability, whereas this paper evaluates complete single-agent computer-use trajectories.Related work also studies multi-agent failure attribution and collaboration.Source: §2 · p. 4

Judgments in learning pipelines; Adversarial evaluation and paired data

• Judges used to filter training examples can discard valid trajectories or admit incorrect ones, while AgentHorizon’s paired intervention changes the instruction associated with a valid recording.Its construction relates to contrast sets, counterfactually augmented data, and dynamic benchmarking.Source: §2 · p. 4

3 AGENTHORIZON BENCHMARK

• Each item pairs an instruction with screenshots, desktop actions, and a binary success label; negative items also receive a failure-type label, and agents judge recordings rather than performing tasks.Source: §3 · p. 4

3.1 TASK COVERAGE

• The benchmark records human demonstrations across Windows, Linux, and macOS, six task domains, and one or more applications; the median trajectory is 121 steps.It contains 166 hours of recordings, and 83 items have at least 200 steps.Source: §3.1 · p. 4

3.2 PAIRED CONSTRUCTION AND LABEL REVIEW

• Closely related instructions differ in a success-relevant requirement; matching instruction–trajectory combinations are positive candidates, while crossed combinations are negative candidates reviewed for validity.The three mismatch types are Critical Mistake, Bad Side Effect, and Misunderstanding.Source: §3.2 · pp. 4–5, Figure 3 · p. 5
• The reviewed set contains 1,373 items, including 523 positives and 850 negatives; vendor review of negatives from every pair covered 850 items in 212.5 contributor-hours, and all were retained.Removing a positive candidate does not necessarily require removing its paired negative; the negative remains valid if the recording does not satisfy the swapped instruction.Source: §3.2 · p. 5

3.3 DEVELOPMENT HOLDOUT AND DIFFICULTY PARTITION

• The 162-item AH-D pair-aware development set is withheld from evaluation pools, and the remaining items are split into AH when pooled model verdicts agree with reference labels at most 18 times, otherwise AH-S.The resulting subsets contain 528 AH items and 683 AH-S items; this difficulty definition depends on the selected models and threshold.Source: §3.3 · p. 6

3.4 METRICS

• Balanced accuracy averages positive and negative class accuracies, while mistake-type recall requires predicting both failure and the reference category for a negative.Missing, malformed, null, and non-Boolean predictions count as errors; per-trajectory cost and runtime measure computational efficiency.Source: §3.4 · p. 6

4.1 JUDGE PERFORMANCE

• GPT-5.5 with Codex achieves 80.9% balanced accuracy on AH, followed by Gemini 3.1 Pro with Gemini CLI at 77.7% and Claude Opus 4.7 with Claude Code at 76.0%.Qwen 3.6 27B with OpenCode is the strongest open-weight agentic judge at 70.4%; systems generally perform better on the easier AH-S subset.Source: §4.1 · p. 7, Table 2 · p. 7
• Tool-use comparisons differ by model: Qwen 3.6 27B scores 77.6% with direct judgment, while GPT-5.5 rises from 68.6% with direct prompting to 80.9% with Codex.Because the overall harness changes, these comparisons do not isolate screenshot selection, context management, or another individual component.Source: §4.1 · p. 7

4.2 EXAMPLE FAILURE CASES

• Claude Opus 4.7 accepts a trajectory that posts an unrequested message to an organization-wide channel, but also rejects a valid GitHub task over its repository location.GPT-5.5 rejects the side-effect case and correctly accepts the GitHub trajectory, according to the ground-truth labels.Source: §4.2 · pp. 7–8

• Harness choice changes balanced accuracy even with identical prompts and items, so tool access alone cannot explain judge performance.On AH, switching OpenCode for Codex changes GPT-5.5 by 3.7 points and Qwen 3.6 27B by 1.2 points, while preserving their order.Source: §5 · p. 8
• Qwen 3.6 27B labels 98.9% of both matched and swapped full trajectories successful without the instruction, showing that apparent completion cannot establish instruction adherence.Bad Side Effect has the lowest exact recall among primary judges on the full reviewed corpus, so requested output may exist despite an unacceptable action.Source: §5 · p. 8
• Rejecting valid trajectories can remove useful training examples, while accepting invalid ones has the opposite impact; balanced accuracy does not encode application-specific error costs.Positive and negative accuracies in Table 2 remain relevant when judges have similar overall scores.Source: §5 · p. 8
• Rankings across three model-defined AH subsets correlate highly (ρ = 0.963–0.975), and the ensemble partition preserves principal ranking trends.This supports the difficulty partition's robustness to which model defines the subset.Source: §5 · p. 8
• The benchmark targets subtle near-miss errors, not the full distribution of agent failures; instruction-free controls should not model general failures.Its corpus is English-only, covers a finite application set across three operating systems, and provides screenshots and action logs rather than comprehensive application states.Source: §5 · p. 8
• Future work should test common agent failures and compare evidence selection and long-context strategies within one harness.Recordings also allow direct evaluation from video, while instruction pairs can provide training examples where changing a subtle requirement changes success.Source: §5 · p. 8

6 CONCLUSION

• AgentHorizon evaluates judges across long computer-use trajectories, using closely related fine-grained instructions to test detection of subtle failures in otherwise complete-looking executions.The conclusion reports evaluation of eleven judge models and says tool use does not consistently boost performance.Source: §6 · p. 9

AI USE STATEMENT

• The authors used coding agents for experimental code, drafting, figures, tables, literature search, scripts, experiment design, and analysis, but not for recording demonstrations.They state that authors wrote the final manuscript and verified its results and citations.Source: AI USE STATEMENT · p. 9

ETHICS STATEMENT

• The benchmark evaluates instruction adherence, not whether a task is inherently dangerous, so a high score is insufficient evidence for production deployment.Demonstration accounts and profiles were created for annotation, with protocols prohibiting personal communications and authentication secrets.Source: ETHICS STATEMENT · p. 9

REPRODUCIBILITY STATEMENT

• The paper documents trajectory construction and review, split partitioning, scoring denominators, invalid-output handling, and experimental preprocessing to support replication.It also points to open-source evaluation code.Source: REPRODUCIBILITY STATEMENT · p. 9

A.1 ANNOTATION PROCESS DETAILS

• Professional annotators recorded long trajectories on Windows, macOS, or Ubuntu, storing structured actions alongside screenshots and producing 166 hours of OBS video.Task pairs were minimally different: completing the counterpart instruction instead was a failure, and annotators independently recorded both tasks using the same software subset.Source: §A.1 · p. 13
• Table 4 tracks benchmark accounting: the 327 removals affect candidate positives only, while AH-D, AH, and AH-S are disjoint and exhaust the reviewed set.Table 3 reports vendor effort over 850 unique trajectories; calibration added review passes beyond the original 1,960-hour estimate.Source: Table 4 · p. 14, Table 3 · p. 14

A.2 MISTAKE TAXONOMY AND CALIBRATION RUBRIC

• The three failure labels are supported by an annotator calibration document with a decision flow, pair-level examples, and a recovery-cost matrix; Table 5 gives the taxonomy's operational boundaries.Source: §A.2 · p. 14, Table 5 · p. 14

A.3 PARTITIONING MODEL AND SUBSET DEFINITION

• The benchmark groups instruction pairs across splits, then uses three open models' repeated judgments to assign remaining items to challenging AH or simpler AH-S.AH-D contains 162 trajectories; for other items, at most 18 of 24 judgments matching the reference assigns an item to AH.Source: §A.3 · p. 14

A.4 PROMPT DEVELOPMENT

• Prompts were iteratively developed on AH-D by examining issues such as missing tool use or screenshot use; Figure 4 presents the final evaluation prompt.Table 6 reports resource use for primary agentic-judge configurations, with means over 1,373 trajectories unless telemetry is omitted and no marginal API charge for self-hosted rows.Source: §A.4 · p. 14, Figure 4 · p. 15, Table 6 · p. 16

• Trajectories have a median of 121 steps, a 90th percentile of 186, and a maximum of 358; AH and AH-S medians are 122 and 118 steps, respectively.The AH and AH-S 90th percentiles are 195 and 179 steps; 83 overall data points have at least 200 steps.Source: §A.5 · p. 16, Figure 5 · p. 16
• The benchmark covers 112 desktop applications, with a mean of 1.98 applications per task and a median of 2; 83.2% of tasks use exactly two applications.Source: §A.5 · p. 16

B.1 COMPUTE AND PREPROCESSING

• Proprietary models use official APIs, while open-weight judges run through vLLM on H100 GPUs.Source: §B.1 · p. 16
• Direct-judge screenshots are downscaled to 512×332 pixels, and four consecutive frames are mosaicked for open-weight models with shorter context limits.The release pins environment versions and provides preprocessing and evaluation scripts for reproducibility.Source: §B.1 · p. 16

B.2 MISTAKE-TYPE RECALL

• Bad Side Effect has the lowest recall, indicating models are poorly calibrated to distinguish side effects from other mistake types.Table 7 reports exact mistake-type recall for the primary agentic-judge configurations.Source: §B.2 · p. 16, Table 7 · p. 17

B.3 EVALUATION WITH ADDITIONAL HARNESSES

• GPT-5.5 performs 3.7 percentage points lower on OpenCode than Codex, while Qwen 3.6 27B is 1.2 points lower on OpenCode and 20.4 points lower on OpenHands.The authors conclude harness choice can substantially affect judge performance and recommend native interfaces or comparisons across open harnesses.Source: §B.3 · p. 17

B.4 EFFECT OF REMOVING REQUIRED INFORMATION

• Removing the instruction and action log yields 47.8%–49.8% balanced accuracy, close to random, while removing screenshots yields 57.7%.Source: §B.4 · p. 17, Table 9 · p. 18
• The authors conclude that trajectories alone do not suffice for predicting success and that screenshots provide important information for judgment.Source: §B.4 · p. 17