PaperlayerView exact revision on arXiv

Source-linked AI summary

Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution

Zhiyong Cao, Hengjie Zhu, Yongxin Wang, Xuchen Li, Jintao Li, Shuxia Liu, Dunqiang Liu, Bo Wang, Kairan Zhang, Cong Lin, JiangDi Yu, Siyuan Liu, Sijia Yao, Yi-Liang Wei, Lingxu Ma, Aijia Wang, Anxin Chen, Zixian Wei, Xiang Ruan, Ruizhi Xu, Ruqian Shi, Baihua Xiao, Sicheng Zhou

arXiv:2610.32285v1cs.AI

TL;DR

Existing agent benchmarks focus largely on task outcomes, offering limited insight into whether harness mechanisms function reliably. TraceTrial-Bench evaluates native harnesses separately from agent performance using mechanism-oriented tasks and evidence-based attribution. Across 3,420 selected runs, task success and harness-criterion satisfaction diverged, including successful runs with failed harness criteria.

  • Problem

    Existing benchmarks mainly measure task completion and provide limited insight into the reliability of specific harness mechanisms.

    Source: §1 · p. 2

  • Method

    TraceTrial-Bench uses 95 mechanism-oriented tasks, execution traces, multi-source evidence, separate Agent and Harness assessments, and explicit mechanism chains for failure attribution.

    Source: §1 · p. 2

  • Results

    Across 3,420 selected runs, Harness Macro differed by 12.2 percentage points, task-success and harness-satisfaction leaders differed, and 31.1% achieved Agent success despite a failed Harness criterion.

    Source: §8 · p. 21

  • Takeaways & Limitations

    The results show that outcome-only evaluation cannot characterize harness reliability, while separate assessments support evidence-based comparison and diagnosis of harness behavior.

    Source: §8 · p. 21

    The targeted suite, one selected execution per task-model-harness combination, and variation in revisions, runtime settings, and scoring budgets limit generalization and causal conclusions about individual design choices.

    Source: §6.1 · p. 20

Abstract

from arXiv · show

LLM agents are increasingly used to solve real-world problems, and their effectiveness depends on both the underlying models and the harnesses that support their operation. However, most existing benchmarks focus on model capabilities and overall agent performance, with limited attention to the evaluation of agent harnesses. To address this gap, we introduce TraceTrial-Bench, which organizes harness reliability into seven components covering context, execution, governance, lifecycle, observability, tools, and verification. We construct 95 expert-level, mechanism-oriented tasks with executable or state-based ground truth to support semantically equivalent comparisons while preserving the native behavior of each harness. We design a Trajectory Tracing (TT) module to capture agent execution traces and introduce a two-layer evaluation framework that separately assesses agent performance and harness reliability. In addition, we design a Multi-source Evidence Acquisition (MEA) module to provide evidence from the agent context, harness, and execution environment for LLM-based evaluation, and develop a Failure Attribution (FA) module that uses explicit mechanism chains to support attribution at the run level. Across 4 harnesses, 9 models, and 95 tasks, our evaluation of 3,420 runs yields macro-averaged harness scores ranging from 59.8% to 72.0%, while 31.1% of runs achieve task success despite failing at least one harness criterion. TraceTrial-Bench provides an auditable and reproducible foundation for comparing agent harnesses, diagnosing their failures, and studying how harness design interacts with model behavior.

1 Introduction

Outcome-only benchmarks cannot reliably distinguish model errors from harness shortfalls, motivating a benchmark that evaluates native harness mechanisms separately from agent performance. TraceTrial-Bench combines mechanism-oriented tasks, execution evidence, and explicit failure-attribution chains; its evaluation finds that task success and harness-criterion satisfaction can diverge.

1 Introduction

  • Existing outcome-focused benchmarks provide limited insight into whether harness mechanisms function reliably, because task outcomes combine model and harness effects.Task success can conceal harness shortfalls, while task failure can occur despite faithful harness execution.Source: §1 · p. 2
  • TraceTrial-Bench evaluates native harness reliability across seven components using 95 tasks with explicit requirements and executable or state-based ground truth.Cross-harness adapters preserve common objectives and criteria while retaining each harness’s native interfaces and execution behavior.Source: §1 · p. 2
  • 31.1% of runs achieved task success despite failing at least one harness criterion, and task-success and harness-satisfaction assessments yielded different harness rankings.The evaluation covered 3,420 selected executions across four harnesses, nine models, and 95 tasks.Source: §1 · p. 2
  • Trajectory Tracing, Multi-source Evidence Acquisition, and a two-layer framework separately assess agent performance and harness reliability using execution records, artifacts, and environment state.The Failure Attribution module connects failed criteria through predefined mechanism chains and requires further evidence review to validate propagation.Source: §1 · p. 2
  • The paper introduces failure attribution through explicit mechanism chains, distinguishing candidate links from propagation validated through evidence review.The benchmark’s tasks, configurations, traces, and verdicts are provided for reproducible analysis.Source: §1 · p. 2

2 Background

The background frames an agent as a model operating through a harness that manages its interaction with tools and the environment. It distinguishes task outcomes from harness reliability and sets boundaries for evidence and attribution claims.

2.1 Agent, Model, and Harness

  • The harness boundary includes orchestration and control functions across scaffold and runtime, while external tools, environments, artifacts, evaluators, and measurement apparatus remain outside it.This boundary helps distinguish harness obligations from agent decisions and measurement failures.Source: §2.1 · p. 3

2.2 Harness Reliability and Task Outcomes

  • Harness reliability concerns faithful information delivery and execution semantics, but does not guarantee task success or follow from a correct final artifact.TraceTrial-Bench therefore evaluates Agent outcomes and Harness obligations with separate criteria.Source: §2.2 · pp. 3–4

2.3 Evidence Boundaries and Levels of Claim

  • Evidence sources support different claims: environment state establishes what occurred, artifacts identify persisted deliverables, and native model-facing records show what the model could observe.A criterion verdict, confirmed Harness defect, activated candidate link, and validated propagation are distinct claim levels; validation requires observable intermediate distortion, supported event order, and consideration of competing explanations.Source: §2.3 · p. 4

3 TraceTrial-Bench

This section presents TraceTrial-Bench’s framework for evaluating native harness behavior separately from agent task performance, using admitted runs, trace and probe evidence, and criterion-level scoring to support auditable failure analysis. It defines conservative runtime labels, distinguishes candidate failure links from manually validated propagation claims, and specifies the archived evidence and operational metrics used to summarize agent performance, harness criteria, and attribution.

3 TraceTrial-Bench

  • The benchmark asks whether agents meet task requirements, whether native harnesses preserve required behavior, and whether agent failures match predeclared harness-to-agent mechanisms.Candidate links activate automatically only when both endpoint criteria fail and the declared activation conditions are satisfied.Source: §3 · p. 4

3.2.1 Harness Reliability Taxonomy

  • Harness reliability is organized into seven dimensions—Context, Execution, Governance, Lifecycle, Observability, Tools, and Verification—across 40 mechanism-level properties.The dimensions follow the harness obligation under test; measurement-apparatus integrity is handled separately through admission.Source: §3.2.1 · pp. 5–6, Figure 4 · p. 6

3.2.2 Task Suite

  • The 95-task suite pairs harness obligations with agent deliverables and deterministic checks where possible, using practical scenarios with independent truth anchors.It includes 46 single-turn, 45 fixed multi-turn, and 4 simulator-driven tasks; task packages fix prompts, workspaces, tools, budgets, outputs, and verifiers.Source: §3.2.2 · pp. 6–7

3.2.3 Harnesses and Cross-Harness Execution

  • Four harnesses are run through source-specific adapters that preserve native execution strategies, while task semantics and evaluation conditions remain fixed.Task-scoped containers, frozen baselines, bounded resources, and provenance-linked manifests support controlled execution and measurement.Source: §3.2.3 · p. 7, §3.2.4 · p. 7

3.3.1 Trace Collection and Normalization

  • Source-specific adapters normalize native observations into an auditable event sequence while retaining source records and leaving ambiguous call–result links unpaired.The tracer also captures provider-boundary tool declarations and preserves distinctions between external execution state and model-facing results.Source: §3.3.1 · pp. 7–8

3.3.2 Evidence Packets and Forensic Reconstruction

  • Criterion-specific evidence packets use deterministic selectors and claim-specific authorities, while the full archived trace remains available for audit and replay.Packets are bounded and provenance-tagged; integrity checks, digest bindings, and secret minimization protect the released evidence.Source: §3.3.2 · pp. 8–10

3.4 Two-Layer Evaluation

  • Separate Agent and Harness score sheets combine deterministic rules with criterion-specific semantic judgments, with Agent verdicts finalized before Harness judging.An empty required evidence selection scores zero without independently proving a harness defect; the layers are distinct but not statistically independent.Source: §3.4 · pp. 10–11

3.5 Failure Attribution

  • Failure attribution distinguishes supported harness-side failures from candidate propagation links, excluding bypass-affected harness zeros from the adjusted attribution set.The reported empirical results use raw harness zero scores; an activated link is operational evidence, not a confirmed defect or validated propagation claim.Source: §3 · p. 4, §3.5 · pp. 11–12
  • Activated links establish endpoint presence, but only manually reviewed links with verified endpoints and an observable, correctly ordered intermediate distortion support validated propagation claims.Review also considers provenance and plausible competing explanations.Source: §3.5 · p. 12
  • Seven ordered decision rules yield six stored labels, but `harness_caused` denotes candidate-link activation, not causation, and `agent_caused` does not imply a defect-free Harness.Confirmed defect absorption requires the relevant Harness endpoint to belong to DH; validated propagation requires at least one validated link.Source: §3.5 · p. 12, Table 5 · p. 13

3.6 Benchmark Outputs and Metrics

  • Each scored run’s archive preserves configuration and provenance, execution evidence, evaluation artifacts, and serialized verdicts, candidate links, and bypass annotations.Release manifests associate selected runs with task configurations and artifact paths and checksums.Source: §3.6 · pp. 12–13
  • Harness pass rates are reported as micro averages weighted by finalized criteria and macro averages weighted by runs; both measure operational criterion satisfaction, not confirmed-defect rates.Agent task success requires no zero-scored Agent criterion, and Harness rates incorporate outcome-conditioned resolution of raw state 2.Source: §3.6 · p. 13
  • Attribution metrics separately track activated candidate links, Harness zeros without Agent zeros, and unlinked Agent zeros; measurement-quality records remain outside performance aggregates.The analysis retains finalized cells and does not estimate selector recall, Judge agreement, or candidate-link precision.Source: §3.6 · p. 13

4 Experiments

The experiments compare four native harnesses across nine model groups and 95 tasks using a finalized matrix of 3,420 selected executions. The analysis specifies execution and scoring procedures, failure-intersection measures, and reproducibility inputs.

4.1 Evaluated Systems and Experimental Population

  • The population comprises Claude Code, Codex, DeepSeek Harness, and OpenClaw; nine model groups; and 95 tasks spanning 40 mechanism-level properties.The tasks cover Context, Execution, Governance, Lifecycle, Observability, Tools, and Verification, with one selected execution per model–harness–task cell.Source: §4.1 · p. 14

4.2 Execution and Evaluation Configuration

  • The archive contains selected outcomes, including replacement executions, rather than independent repetitions, so the study does not estimate within-cell repeatability.Runs use Docker; task-specific effective wall-time limits span 120–5,400 seconds, and scheduling and recovery differences constrain causal interpretations of harness differences.Source: §4.1 · p. 14, §4.2 · p. 14

4.3 Inclusion and Aggregation Policy

  • Failure intersections use raw failed Harness and Agent criteria; co-occurring failures or stored candidate links alone establish neither a confirmed harness defect nor causal propagation.Candidate-link incidence is reported among joint failures, counting each run once; the pooled rate weights dimensions by their joint-failure counts.Source: §4.3 · p. 15

4.4 Reproducibility

  • Outcome aggregates can be recomputed from archived final score files, task and model identities, and task dimensions; individual verdicts can be inspected using traces and scoring evidence.Table 6 reports aggregate percentages, with detailed model-level results in Table 7; resource estimates additionally require recorded usage and the frozen price catalog.Source: §4.4 · p. 15, Table 6 · p. 16

4 Experiments

  • The experiment compares four harnesses on the same 855 model–task cells each, with Agent Score and Harness Macro averaged equally in the overall results.Agent Pass requires all Agent criteria to pass; Clean requires all criteria in both layers to pass, while Micro pools Harness criteria.Source: §4.3 · p. 15, Table 6 · p. 16

5 Results

Across the evaluated harnesses, task success and harness-criterion satisfaction have different leaders, and reliability also varies by model and taxonomy dimension. Harness shortfalls often coexist with successful Agent outcomes, while candidate-link activation captures only part of joint failures.

5.1 Evaluation Population and Score Coverage

  • Across 3,420 runs, mean per-run scores are 72.9% for the Agent layer and 67.9% for the Harness layer, but only 8.4% pass every criterion in both.All runs had final combined scores and nonempty binary verdict vectors for both layers.Source: §5.1 · p. 16

5.2 Overall Layered Outcomes

  • Claude Code has the highest Agent task-success rate (44.0%) and combined score (73.5), while OpenClaw has the highest Harness macro rate (72.0%).The Harness macro range is 12.2 percentage points, from Codex at 59.8% to OpenClaw at 72.0%; thus the task-success and Harness-satisfaction leaders differ.Source: §5.2 · p. 16, Table 7 · p. 17

5.3 Harness–Model Interaction

  • Within-model Harness macro ranges span 10.0–16.2 percentage points, and Harness leaders vary across models; Agent success also varies by Harness configuration.Grok-4.6 ranges from 33.7% with OpenClaw to 55.8% with Claude Code; Opus 5 ranges from 30.5% with Codex to 53.7% with Claude Code.Source: §5.3 · pp. 17–18, Table 8 · p. 18

5.4 Reliability by Taxonomy Dimension

  • Execution has the lowest pooled Harness micro rate (64.6%), while Verification has the highest (90.1%); within Execution, Codex records 51.5% and OpenClaw 77.1%.Agent and Harness layer gaps change direction across dimensions; Tools has 69.7% Harness micro satisfaction but 21.4% Agent success over 612 runs.Source: §5.4 · p. 18, Table 9 · p. 18

5.5 Failure States and Candidate Attribution

  • Of 1,700 joint-shortfall runs, 842 (49.5%) contain a candidate link, while 1,065 of 3,420 runs achieve Agent success despite at least one Harness failure.Among Agent-failure runs, 1,773 of 2,069 (85.7%) retain at least one failed Agent criterion not reached by a link.Source: §5.5 · p. 19, Table 10 · p. 19

5.6 Recorded Resource Use

  • Mean estimated Agent costs per selected run range from $0.436 for Claude Code to $0.633 for OpenClaw.Mean total recorded tokens range from 372.7k for DeepSeek Harness to 680.0k for OpenClaw; judge costs are excluded from the reported resource columns.Source: Table 7 · p. 17, §5.6 · p. 20

5.7 Findings and Interpretation Limits

  • The full-grid results show that outcome quality and Harness-criterion satisfaction are distinct, and that model and taxonomy dimension both shape observed Harness reliability.Within-model Harness macro ranges reach 16.2 points, and failure co-occurrence exceeds candidate-link activation. Selected trajectory studies support inspection of specific behaviors, not a population-level estimate of attribution accuracy.Source: §5.7 · p. 20

6 Discussion

The evaluation characterizes a targeted task suite and selected configurations, with limited evidence about deployment generalization or repeated-run stability. Future work proposes controlled, repeated evaluations and held-out testing to examine Harness changes and transfer.

6.1 Limitations

  • The targeted suite has uneven mechanism coverage and one selected execution per task-model-harness combination, limiting evidence about deployment generalization and repeated-run stability.Task revisions, runtime settings, and scoring budgets also constrain causal conclusions about individual harness design choices.Source: §6.1 · p. 20

6.2 Future Directions

  • Future work proposes trace-guided Harness changes evaluated on held-out tasks with fixed models and budgets, using controlled interventions and repeated executions to check resolution and regressions.Crossed Agent and Harness version evaluations and testing on unfamiliar Harnesses are proposed to assess contributions, interactions, and transfer.Source: §6.2 · p. 20

7 Related Works

Related Harness research addresses context, interfaces, continuity, safeguards, configuration optimization, and implementation repair. Agent evaluation and failure-analysis work covers verifiable outcomes, execution fidelity, risks, localization, diagnosis, and controlled replay.

7.1 Agent Harnesses

  • Harness research spans context and interfaces, continuity and recovery, policy enforcement, configuration or skill optimization, and implementation repair.TraceTrial-Bench adds mechanism-level criteria and obligation-specific evidence while preserving native harness behavior.Source: §7.1 · pp. 20–21

7.2 Agent Evaluation and Failure Analysis

  • Related evaluation work assesses verifiable outcomes, model–harness configurations, execution fidelity, adversarial risks, error localization, constraint violations, and controlled replay.TraceTrial-Bench complements these approaches with separate Agent and Harness criteria and evidence-based failure analysis.Source: §7.2 · p. 21

8 Conclusion

TraceTrial-Bench evaluates native harnesses with mechanism-oriented tasks, tracing, multi-source evidence, separate Agent and Harness scoring, and mechanism-chain failure analysis. Its taxonomy assigns tasks to a primary component and mechanism-level property, with scope tied to observable harness responsibilities.

8 Conclusion

  • TraceTrial-Bench covers 95 mechanism-oriented tasks across seven reliability components and supports auditable failure analysis through tracing, evidence acquisition, and explicit mechanism chains.Across 3,420 selected runs, Harness Macro differs by 12.2 percentage points, and 31.1% achieve Agent success despite a failed Harness criterion.Source: §8 · p. 21

A.1 Harness Reliability Taxonomy

  • Each task maps to one primary reliability component and mechanism-level property, while secondary tags capture interactions without duplicating component observations.The taxonomy includes a property only when the Harness mediates it, comparable task conditions can exercise it, and execution evidence can check its outcome.Source: §A.1 · p. 24, Table 11 · p. 24
  • Responsibility boundaries assign tool declarations and argument transport to Tools, process outcomes to Execution, child context to Context, and child completion to Lifecycle.Verification covers Harness support for checking progress; the independent oracle remains part of the measurement apparatus.Source: §A.1 · p. 24

A.2 Task Design and Benchmark Populations

Tasks pair a harness mechanism with a practical deliverable, using controlled interaction conditions to test whether the intended behavior supports correct delivery. The release contains 95 unique task paths.

A.2 Task Design and Benchmark Populations

  • Each task couples a harness mechanism with a practical deliverable, with private or one-time values testing whether correct delivery depends on the intended interaction.Recovery tasks provide a bounded recovery route.Source: §A.2 · p. 25
  • The task package fixes prompts, initial workspace, tools, budgets, required outputs, verifier, and score sheets while adapters preserve native execution behavior.Changing task semantics through an adapter creates a different comparison.Source: §A.2 · p. 25
  • The release population is the 95 unique paths in lean95.manifest; construction groups describe task origins and receive no separate scoring weights.The protocol distinguishes task construction from execution and evaluation.Source: §A.2 · p. 25

A.3 Task Cards, Oracles, and Hard Gates

Task cards summarize contracts, execution conditions, and verification sources, while task packages provide executable verifiers and separate Agent and Harness score sheets. Oracles and gates distinguish task outcomes from interaction violations and unusable evidence.

A.3 Task Cards, Oracles, and Hard Gates

  • Released task packages include prompts, turn specifications, runtime profiles, workspaces, executable verifiers, and separate Harness and Agent score sheets.The archived task package is authoritative; task cards summarize its contract, execution conditions, and verification sources.Source: §A.3 · p. 25, Table 13 · p. 25
  • An oracle may use a truth file, protected fixture, simulator ledger, process outcome, or executable condition.For C1-escalation-export-001, the verifier compares privately recorded codes and requests with delivered tool results and the final JSON mapping.Source: §A.3 · p. 26, Table 14 · p. 26
  • Correct final artifacts can still violate interaction contracts, so required actions, ordering constraints, and prohibited shortcuts are encoded in prompts, schedules, score sheets, and verifiers.These checks distinguish Agent actions, harness-mediated behavior, and collection failures.Source: §A.3 · p. 26
  • Ordinary criteria measure individual requirements, runner hard gates can override execution-time verifier results, and apparatus checks assess evidence usability.Validation checks structural consistency, oracle and verifier behavior, trigger reachability, budgeted solvability, and shortcuts that could bypass the target mechanism.Source: §A.3 · p. 27, Table 15 · p. 27

A.4 Benchmark Coverage

TraceTrial-Bench describes task coverage, harness configurations, native capability profiles, and comparison eligibility. Its protocol controls common task conditions while retaining harness-controlled behavior and distinguishes unsupported or non-comparable pairs from measured capability absence.

A.4 Benchmark Coverage

  • The 95-task benchmark reports primary-component, scenario, execution, and overlapping oracle and evidence coverage; its score sheets contain 783 Harness and 556 Agent criteria.The score sheets comprise 909 programmatic rules and 430 semantic rubrics.Source: §A.4 · pp. 27–28, Table 17 · p. 28, Table 18 · p. 28

B.1 Harness Selection and Configuration

  • Four harnesses support repeated model–tool interaction, configurable model access, isolated execution, and observable native activity, with resolved configurations retained for each run.Codex and Claude Code use a local LiteLLM gateway, OpenClaw its native Gateway, and DeepSeek Harness its SDK and JSON-RPC runtime.Source: §B.1 · pp. 28–29, Table 19 · p. 28

B.2 Native Capability Discovery and Harness Profiles

  • Capability profiles distinguish documented support, model exposure, exercise in a run, and behavior verified against external state.Trace normalization preserves native content and identities, retains unmatched or malformed records, and does not reconstruct missing information.Source: §B.2 · p. 29, §B.3 · pp. 30–31

B.4 Semantic Equivalence and Eligibility

  • A pair is unsupported when a task cannot be administered or observed, and non-comparable when an adapted path changes its semantics.A target capability's absence can itself be the behavior under test; this study retains all finalized cells, with 855 model–task cells per harness.Source: §B.4 · pp. 31–32

C.1 Evidence Sources and Acquisition Architecture

Evidence acquisition preserves distinct sources and their visibility, then reconstructs trace events using native identities and claim-specific source precedence. Trace IR retains source-supported fields and evidence-quality distinctions; archive integrity checks do not establish the truth of every source record.

C.1 Evidence Sources and Acquisition Architecture

  • The collection pipeline preserves native records, provider metadata, task-tool ledgers, environment state, and apparatus records, archiving initial and final workspaces for offline evaluation.A service ledger establishes what a service received or committed, while a correlated model-facing record establishes what the model received through the harness.Source: §C.1 · pp. 32–33, Table 24 · p. 32

C.2 Trace Reconstruction and Source Precedence

  • Trace reconstruction binds records to runs and sessions, correlates exchanges, merges resumed turns, and preserves native identities and supported ordering.Persisted history does not necessarily equal active model context, and timestamps alone cannot establish cross-session causality.Source: §C.2 · p. 33

C.3 Trace IR Schema and Event Dictionary

  • Trace IR separates a common event envelope from event-specific payloads, populating optional identities and values only when native evidence supports them.Tool-call results preserve source-indicated error status, exit code, duration, and known truncation; a missing result differs from a successful empty string.Source: §C.3 · pp. 34, 35, 37
  • Evidence quality is assessed through completeness, fidelity, traceability, integrity, privacy, replayability, and bounded overhead.Archive manifests detect changes and incorrect bindings but do not establish every source record's truth; offline scoring does not replay the agent or guarantee reproduction of its decisions.Source: §C.4 · pp. 37–38

D.1 Evaluation Layers and Criterion Catalog

TraceTrial-Bench separates Agent performance from Harness execution behavior, with apparatus checks serving as an evidence-integrity gate rather than a third score. Its verdict procedure keeps layer scores distinct and uses explicit mechanism chains to distinguish candidate failures from validated attribution.

D.1 Evaluation Layers and Criterion Catalog

  • The Agent layer scores task actions and deliverables, the Harness layer scores mediated execution behavior, and apparatus checks gate evidence integrity without adding a third score.The two verdict vectors remain separate, although Harness state 2 depends on the completed Agent outcome.Source: §D.1 · pp. 38–39
  • Rules check task state, artifacts, or exact trace properties deterministically, while rubrics apply semantic judgment to bounded evidence packets.Criteria specify identifiers, types, statements, and required evidence.Source: §D.1 · p. 39
  • Each layer contributes up to 50 points to the combined score, while separate layer percentages use their own passed-criterion counts and denominators.A combined pass requires Stotal = 100; Agent task success requires only PA = NA.Source: §D.1 · p. 39

D.2 Verdict Rules and Evaluation Procedure

  • Harness state 2 passes only when all Agent criteria pass, whereas state 3 resolves to zero.Agent verdicts can resolve state 2 but do not independently establish a Harness defect.Source: §D.2 · p. 39
  • Deterministic results are fixed before Agent and Harness rubrics, and each semantic judgment retains its evidence packet, prompt, response, verdict, and rationale.Missing required evidence receives zero, and finalized criteria remain in their layer denominator, including bypass-affected zeros.Source: §D.2 · p. 39

D.3 Failure Attribution and Evidence Requirements

  • Attribution separates raw Harness failures from confirmed defects: bypassed observations are excluded from the confirmed-defect set, which additionally requires criterion-specific source reconciliation.Bypass annotations leave scores unchanged; the optional runtime summary uses non-bypassed failures, while the reported contingency uses raw failures.Source: §D.3 · pp. 39–40
  • A mechanism links Harness and Agent criteria through an intermediate distortion; activation flags a candidate, while validation requires confirmed endpoints, observable distortion, supported event order, and consideration of alternatives.Candidate descriptors are not calibrated probabilities.Source: §D.3 · p. 40, Table 34 · p. 40
  • The stored harness_caused label records candidate activation rather than validated propagation, and agent_caused does not imply that no Harness criteria failed.Linked and unlinked Agent failures can coexist; recovery claims additionally require an observed recovery action and verified final state.Source: §D.3 · p. 40

E Task-Level Score Heatmaps

The task-level heatmaps present shared-scale comparisons across Harnesses, tasks, and models. Selected case-study trajectories focus on output recovery, failure reporting, delegation, terminal delivery, and session handoffs.

E Task-Level Score Heatmaps

F Case Studies

  • Figure 14 shows selected Opus 5 output-recovery trajectories under truncation, with fractions denoting passed criteria over total criteria for each run.The case-study heading classifies the example under execution and observability.Source: Figure 14 · p. 46, F Case Studies · p. 47
  • Figure 15 compares selected Opus 5 trajectories with textual versus structured failure reporting.Source: Figure 15 · p. 47
  • Figure 16 distinguishes dispatch acceptance from delivered completion in selected asynchronous delegation trajectories with Opus 5.Source: Figure 16 · p. 48
  • Figure 17 compares end-of-file delivery with subsequent task state in selected native terminal trajectories with Opus 5.Source: Figure 17 · p. 49
  • Figure 18 distinguishes correct final values from exactly-once interaction compliance in selected session-handoff trajectories with Qwen3.8-Max.Source: Figure 18 · p. 50

Related papers

Loading 2610.32285v1…