PaperlayerView exact revision on arXiv

Source-linked AI summary

Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents

Minki Kang, Ryo Hachiuma, Shaokun Zhang, Subhashree Radhakrishnan, Yonggan Fu, Jindong Jiang, Mingjie Liu, Ehsan Hosseini-Asl, Yi Dong, Yu-Chiang Frank Wang, Byung-Kwan Lee

arXiv:2609.39982v1cs.CLcs.AIcs.MA

TL;DR

Terminal agents can generate useful actions yet fail because executed commands change the environment that subsequent decisions depend on. Mid-Harness samples and verifies actions before execution without changing the generator or harness; reported results show benefits from verification and combined action- and trajectory-level scaling.

  • Problem

    Terminal agents’ stochastic action sequences can fail despite useful actions, and the effect of jointly scaling candidate sampling and verification on trajectory success remains incompletely understood.

    Source: §1 · p. 2

  • Method

    Mid-Harness samples candidate actions from the same interaction history, verifies them, and forwards one for execution while keeping the generator and harness unchanged.

    Source: §1 · p. 2

  • Results

    With TMAX-9B on TerminalBench-Lite, Mid-Harness plus one Sequential Refine round surpasses Best-of-T at T=7 using less than half its estimated token cost, with gains also reported across models, tasks, and harnesses.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings establish action scaling as a complementary axis of test-time compute scaling and show it can compose with parallel and sequential trajectory scaling.

    Source: §1 · p. 3, §8 · p. 11

    The evaluation lacks gold action labels, limiting direct measurement of verification correctness and candidate coverage.

    Source: §8 · p. 11

Abstract

from arXiv · show

Terminal agents act through stochastic model generations, yet the ability to generate a useful action does not ensure its reliable execution. A poor command (e.g., wrong package install) can change the environment in ways that hinder subsequent progress, even when the model could generate a better alternative. We investigate whether allocating test-time compute at the model-harness boundary can improve action reliability and trajectory success, and what makes this allocation effective. To study these questions, we introduce Mid-Harness, which samples and verifies candidate actions before forwarding one for execution, while keeping the generator and harness unchanged. With a TMAX-9B generator, more action sampling yields little benefit under weak verification, whereas a capable verifier can exploit useful alternatives from the same generator. On TerminalBench-Lite, a GPT-5.6 Sol verifier raises Pass@1 from 50.00% for the base agent to 68.03% with 8 sampled actions. When the same TMAX-9B model serves as the verifier, pairwise verification performs best among the evaluated verification mechanisms. Distilling responses from the stronger verifier into TMAX-9B further improves Pass@1, while leaving the action generator unchanged. With TMAX-9B on TerminalBench-Lite, combining action and trajectory scaling reaches higher success at lower estimated token cost than generating more trajectories alone. Mid-Harness also improves performance across additional models, benchmarks, and harnesses. These findings identify action scaling as a promising target for test-time compute scaling in terminal agents.

1. Introduction

Terminal agents can generate useful actions yet fail because stochastic executed commands change the environment subsequent decisions depend on. The paper studies how action sampling and verification before execution affect trajectory success, introducing Mid-Harness and reporting gains from stronger verification and combined action- and trajectory-level scaling.

1. Introduction

  • The study asks when additional computation before action execution improves trajectory success and what makes that scaling effective.Prior work allocates test-time compute to sampling, verification, and refinement, but the benefit of more candidates depends on verification ability.Source: §1 · p. 2
  • Mid-Harness samples several actions from the same interaction history, verifies them, and forwards one for execution while keeping the generator and harness unchanged.The study varies candidate width, verification mechanism, and verifier capability to examine whether useful alternatives exist and can be identified before execution.Source: §1 · p. 2
  • GPT-5.6 Sol verification raises TMAX-9B Pass@1 from 50.00% for the base agent to 68.03% without changing or post-training the generator.This provides evidence that the generator produces useful alternatives that verification can exploit.Source: §1 · p. 2
  • With self-verification, wider sampling yields marginal improvement under weak verification, while pairwise verification performs best among evaluated mechanisms.Distilling GPT-5.6 Sol pairwise responses raises Pass@1 from 54.76% to 57.14% without changing the action generator; disagreement over command semantics and execution feasibility persists.Source: §1 · p. 2
  • With TMAX-9B, Mid-Harness plus one Sequential Refine round surpasses Best-of-T at T=7 using less than half its estimated token cost.The paper also reports gains across additional models, tasks, and harnesses.Source: §1 · p. 3
  • The contributions identify verification as governing the benefit of action sampling and establish action scaling as compatible with parallel and sequential trajectory scaling.The study varies sampling and verification at a fixed model-harness boundary.Source: §1 · p. 3

2. Mid-Harness

Mid-Harness adds action sampling and verification at the model-call boundary, returning one selected action to the unchanged harness. Its action-level comparisons use candidates from the same history and evaluate different ways to select among them.

2. Mid-Harness

  • Mid-Harness adds candidate sampling and verification inside a model-call wrapper, returning one action without changing the harness, model weights, or serving architecture.The integration pseudocode locates the additions in the wrapper rather than the execution harness.Source: §2 · p. 3

2.1. Scaling Actions Within a Trajectory

  • Candidates from the same history can serve different purposes, so comparing them against the unresolved task requirement may improve the next action and later states while using one environment instance.This contrasts action-level comparison with trajectory-level verification of completed runs.Source: §2.1 · p. 3
  • At each step, Mid-Harness samples N actions conditioned on the same history, selects one proposed action with a verifier, and executes only that choice before updating the history.The remaining candidates are discarded, and the next set is generated from the updated history.Source: §2.1 · pp. 3–4

2.2. Verification Mechanisms

  • Listwise verification chooses from the full candidate set in one call, while pointwise verification scores each candidate independently and selects the highest-scoring action.Pointwise comparisons can run in parallel but require scores for different-purpose actions to be comparable.Source: §2.2 · p. 4
  • Pairwise verification compares candidate pairs and aggregates their preferences, giving alternatives a shared reference but requiring more calls than listwise or pointwise verification.With eight candidates, comparing every unordered pair takes 28 calls, versus one listwise call or eight pointwise calls.Source: §2.2 · p. 4

3. Experimental Setup

The experiments evaluate action and trajectory compute scaling, verifier distillation, and inference costs, primarily with TMAX models on TerminalBench Lite and the Vanillux2 harness. They report run-level success measures and estimated decoding or token costs.

3. Experimental Setup

  • The main evaluation uses TerminalBench Lite with a TMAX-9B generator and Vanillux2, with TMAX-4B and TMAX-27B added for model-scale experiments.The frontier verifier in Section 4.1 is GPT-5.6 Sol; otherwise the verifier generally uses the generator model.Source: §3 · p. 5
  • Pass@1 averages exact-success rates across three runs per task, while Pass@3 measures tasks solved by at least one of those runs.The base agent executes one sampled action per step; additional compute is organized before action execution or across complete runs.Source: §3 · p. 5
  • Parallelized output tokens estimate idealized decoding latency, verifier output tokens sum verifier-call outputs, and Figure 5 uses reference token prices as a cost proxy.The trajectory-level baselines include Best-of-T and Sequential Refine; action-level Mid-Harness can use zero-shot or distilled verification.Source: §3 · p. 5

4. When Does Action Scaling Work?

The section tests candidate coverage, verifier mechanisms, distillation, and costs with a fixed TMAX-9B generator. A stronger verifier exploits useful sampled alternatives, while self-verification benefits depend on the comparison mechanism and verifier training.

4.1. Candidate Coverage

  • With GPT-5.6 Sol verification, TMAX-9B reaches 64.63% Pass@1 at N=4 and 68.03% at N=8, indicating useful alternatives in its sampled actions.Candidate coverage is probed indirectly through trajectory success because there are no ground-truth action labels.Source: §4.1 · p. 5, Figure 2 · p. 6

4.2. Candidate Width and Verification

  • Doubling zero-shot listwise width from N=4 to N=8 changes Pass@1 from 49.32% to 51.02% and Pass@3 from 66.33% to 67.35%.The passage contrasts this small gain with substantially higher success from the frontier verifier using the same listwise mechanism.Source: §4.2 · p. 6, Figure 2 · p. 6
  • With N=8 fixed, pairwise verification reaches 54.76% Pass@1 and 71.43% Pass@3, performing best among the evaluated mechanisms.Pointwise improves Pass@1 only slightly over listwise and leaves Pass@3 unchanged.Source: §4.2 · p. 6, Figure 2 · p. 6

4.3. Verifier Distillation

  • At N=8, pairwise verifier distillation raises Pass@1 from 54.76% to 57.14% and Pass@3 from 71.43% to 75.51%.At N=4, Pass@1 rises from 54.42% to 55.44% and Pass@3 from 68.37% to 70.41%; the verifier is fine-tuned on frontier-verifier responses while the action generator remains unchanged.Source: §4.3 · p. 6

4.4. Cost for Action Scaling and Verification

  • At N=8, zero-shot pairwise responses add an estimated 26.6k POT, versus at most 1.3k for listwise and pointwise; total verifier output is 125.7k, 0.6k, and 7.0k, respectively.Figure 3 plots Pass@1 against parallelized output tokens and total verifier output tokens per run.Source: §4.4 · pp. 6–7, Figure 3 · p. 7
  • At N=8, TMAX-9B decision-only pairwise verifiers improve Pass@1 over their reasoning counterparts while lowering reference-priced token cost by 20.9% for zero-shot and 24.1% for distilled verification.At 4B and 27B, evaluated decision-only variants have lower estimated token cost but also lower Pass@1 than their reasoning counterparts.Source: §4.4 · p. 7

5. Analysis: What Still Limits Verification?

The analysis finds that distillation improves agreement with the frontier verifier, while later-trajectory decisions and command semantics remain difficult. Its offline diagnostics measure agreement with the teacher, not action correctness or task success.

5. Analysis: What Still Limits Verification?

  • Offline diagnostics compare pairwise outputs on stored TMAX-9B trajectories; agreement measures teacher alignment, not action correctness or trajectory success.Pairwise agreement compares A/B/TIE preferences, while verification agreement compares the top-ranked candidate across models.Source: §5 · p. 7

5.2. Remaining Verification Challenges

  • Verification agreement rises in every turn bin, from 68.13% at turns 1–4 to 54.07% at turns 17–32, leaving substantial later-trajectory disagreement.The offline comparison uses 1,355 states with valid comparisons for both models, or 83.4% of states.Source: Figure 4 · p. 8, §5.2 · p. 8

5.1. What Distillation Improves

  • Distillation reduces pairwise-comparison score MAE from 2.59 to 1.05 and raises pairwise agreement from 59.01% to 74.58% and verification agreement from 38.52% to 57.79%.Verification agreement measures whether the offline-selected candidate matches the frontier verifier’s candidate.Source: Table 1 · p. 8, §5.1 · p. 8

5.2. Remaining Verification Challenges

  • Command semantics and execution feasibility account for 67.4% of reviewed distilled-verifier failure cases, although reviewed cases fall from 3,328 to 1,810 after distillation.GPT-5.6 Terra reviewed teacher disagreements and classified those it judged clear verifier failures.Source: §5.2 · p. 8, Figure 4 · p. 8

6. Composition and Transfer of Action Scaling

Action verification composes with trajectory-level scaling and transfers across generator sizes, benchmarks, models, and harnesses. The section reports improvements without requiring more environment runs for the parallel-scaling comparison.

6.1. Composition with Trajectory Scaling

  • 60.20% with SR and 66.33% with Best-of-T=3 both exceed Best-of-T=7’s 59.18% at lower reference-priced token cost.The figure compares Pass@1 against per-run output tokens, input tokens, and reference-priced token cost; decision-only verifiers reduce composition cost by 22–24%.Source: Figure 5 · p. 9, §6.1 · pp. 9–10
  • 66.33% Pass@1 with distilled Mid-Harness plus Best-of-T=3, versus 55.10% with Best-of-T=3 alone, uses the same three environment executions.Zero-shot Mid-Harness reaches 61.22% Pass@1 with those three trajectories.Source: §6.1 · p. 9, Table 2 · p. 9
  • Distilled Mid-Harness with sequential refinement raises Pass@1 from 55.10% to 60.20% and Pass@3 from 71.43% to 75.51%.Action verification remains useful when a refined trajectory is conditioned on experience from an earlier run.Source: §6.1 · p. 9

6.2. Transfer Across Models, Benchmarks, Harnesses

  • Across 4B and 27B TMAX generators, distilled Pass@1 reaches 43.88% and 76.19%; gains also extend to challenging benchmarks, Nemotron models, and Terminus-2.For TMAX-9B, distillation lowers Pass@3 relative to zero-shot verification on FeatureBench-Mini and does not improve over zero-shot verification on Terminal-Bench 2.1.Source: §6.2 · p. 10, Table 3 · p. 10

7. Related Work

Related work studies test-time compute for reasoning and agent performance, including action verification and trajectory-level scaling. This paper distinguishes its focus on action scaling in terminal agents with the generator and harness fixed.

Related Work

  • Test-time scaling work includes aggregation, verification [11], and search, while process reward models evaluate intermediate steps.V1 uses pairwise comparisons and tournaments for mathematical and code solutions.Source: §7 · p. 11
  • Trajectory methods scale complete runs, while prior action-ranking work combines a trained critic with trajectory selection for SWE tasks [13].The paper instead studies verification mechanisms using the generator as verifier across broader terminal tasks.Source: §7 · p. 11

8. Conclusion

The paper presents action scaling as complementary test-time compute for long-horizon terminal agents, while noting limits in verification measurement and command safety. It also documents AI use, safeguards, and reproducibility materials.

8. Conclusion

  • Useful alternatives can improve trajectory success, but wider sampling yields little benefit under weak verification; pairwise comparison and verifier distillation improve success without changing the generator.The paper reports that composing action and trajectory scaling can reach higher success at lower estimated token cost than generating more trajectories alone.Source: §8 · p. 11
  • The evaluation lacks gold action labels, limiting direct measurement of verification correctness and candidate coverage, and distillation leaves a gap to frontier verification.The authors identify better verifier training, such as reinforcement learning, as a motivation for future work.Source: §8 · p. 11

AI use statement

  • Generative AI supported synthetic-data generation, implementation, methodology feedback, qualitative analysis, translation, figure editing, and paper readability; authors reviewed the work.The statement says AI tools did not propose or refine hypotheses, clean or reformat datasets, interpret results, or write proofs.Source: AI use statement · p. 11

Ethics statement

  • Benchmark task-completion reliability does not establish command safety or security; deployment should retain restricted permissions, isolation, and human approval for sensitive actions.The authors state that action verification should complement, not replace, these safeguards.Source: Ethics statement · pp. 11–12

Reproducibility statement

  • Appendices document verification settings, distillation training, evaluation procedures, inference-cost calculations, and offline agreement analyses.Source: Reproducibility statement · p. 12

A. Limitations and Future Work

The limitations identify evaluation constraints and scope boundaries, while outlining ways to improve verifier training and extend the study.

Improving verifier training

  • Distillation transfers part of the frontier verifier’s capability but does not establish how to close the performance gap.Reviewed failures center on command semantics and execution feasibility, while teacher agreement is lower in later states.Source: A. Limitations and Future Work · p. 24

Evaluating action verification

  • Without gold action labels, the evaluation cannot directly measure verification correctness, candidate coverage, or success attainable with perfect verification.Branching from intermediate states could support richer evaluation, but requires environment cloning or restoration.Source: A. Limitations and Future Work · p. 24

Allocating inference compute

  • The generator’s reasoning effort stays fixed, and sampling and verification at every action step leaves open whether gains can be retained with less compute.Decision-only verification lowers reference-priced cost but can trade off Pass@1; the comparison is limited by missing usage and separate live runs.Source: A. Limitations and Future Work · p. 24

Extending beyond this study

  • The study focuses on terminal agents and supervised verifier distillation with TMAX models, leaving other settings and training approaches for future work.Suggested extensions include other distillation and training methods, and tests in computer use and robotics.Source: A. Limitations and Future Work · p. 24

B.1. Evaluation and Baseline Procedures

This section describes the evaluation datasets and procedures, verifier mechanisms and training, inference-cost accounting, and prompt criteria used in the experiments. It also characterizes trial execution effort and distilled-verifier input lengths, and explains how verification scores candidate actions by expected progress using pointwise ratings or structured pairwise comparison.

B.1. Evaluation and Baseline Procedures

  • Evaluation covered 98 TerminalBench-Lite tasks, all 89 Terminal-Bench 2.1 tasks, 50 SWE-bench Verified Mini tasks, and 23 CPU-only FeatureBench-Mini tasks.TerminalBench-Lite excludes two tasks because of sandbox service errors; Terminal-Bench 2.1 is a task-corrected revision. FeatureBench-Mini evaluates TMAX-9B and TMAX-27B with Vanillux2 and pairwise verification at N=8, K=4. Each task has three runs; success requires exact reward, with failures retained in the denominator and fractional rewards not counted as success.Source: §B.1 · p. 25
  • Best-of-T compares completed trajectories, while Sequential Refine summarizes a source trajectory and starts a fresh run from that summary.Best-of-T uses T=3 in Table 2 and T=5,7 for TMAX-9B in Figure 5; Sequential Refine uses one round in Table 2 and R=1,2,3 in Figure 5. Transfer tests include pairwise N=8, K=4 for TMAX-9B on Terminal-Bench 2.1 and eight-candidate zero-shot Mid-Harness across specified Terminus-2 model and benchmark combinations.Source: §B.1 · pp. 25–26

B.2. Verification Mechanisms and Hyperparameters

  • Verification compares candidates independently (pointwise), as a set (listwise), or pairwise, with pairwise preferences aggregated by margin-weighted win rates.The N=8 pairwise tournament uses a ring followed by a pivot stage with K=4; the first-runnable diagnostic instead executes the first candidate that parses, without verification.Source: §B.2 · pp. 26–27

B.3. Verifier Training Data and Settings

  • GPT-5.6 Sol supervision for pairwise verifier distillation yielded 117,631 training inputs from 244 tasks and 10,543 held-out analysis inputs from 21 disjoint tasks.Training examples were balanced across A and B winners while retaining ties; the primary comparison keeps the action generator at Base TMAX-9B. A separate decision-only corpus has 113,728 training and 10,188 development examples, excludes ties, and removes teacher reasoning.Source: §B.3 · p. 28

B.4. Inference Cost Accounting

  • The distilled TMAX-9B verifier’s input length at N=8, K=4 had a 6,495-token median and 10,380-token 90th percentile, below its 61,440-token limit.The distribution covers verifier calls from 294 runs on 98 TerminalBench-Lite tasks; long inputs generally came from long action candidates. The accounting also distinguishes parallelized output tokens, summing the maximum output per sequential verifier stage, from total verifier output and reference-priced pipeline token costs.Source: §B.4 · pp. 28–29, Figure 6 · p. 29

B.5. Evaluation Effort

  • Across 98 TerminalBench-Lite tasks, median agent execution time was 7.2 minutes; across 89 Terminal-Bench 2.1 tasks, it was 10.5 minutes.The corresponding median step counts were 21.0 and 28.3, and 90th-percentile execution times were 13.3 and 20.0 minutes; the longest Terminal-Bench 2.1 task mean reached 60 minutes. These measurements reflect the reported agent and serving setup, not intrinsic benchmark durations, and exclude environment setup and final evaluation.Source: §B.5 · p. 30, Figure 7 · p. 30

B.6. Verifier Prompt Templates

  • Verifier prompt templates combine instructions and task input, filling in the task, executed history, terminal state, and candidate actions at inference time.Source: §B.6 · p. 30

B.6.1. Listwise Verification

  • The listwise prompt selects one candidate for progress toward the original task, weighing command correctness, constraints, history, and whether the task is actually complete.It discourages repeated no-progress actions and broad verification when evidence supports finalization.Source: §B.6.1 · pp. 30–31

B.6.2. Pointwise Verification

  • Pointwise verification rates one candidate’s likely task progress and next-state effect on a 0–10 scale, reserving top scores for evidence-backed, decisive actions.It penalizes repetition and unsupported completion, and rewards targeted fixes or observations only when they address a known blocker or required unknown.Source: §B.6.2 · p. 32

B.6.3. Pairwise Verification

  • Pairwise verification compares exactly two candidate next-actions from the same current state, scores each with 1–10 integers, and explains why one outranks the other through predicted effects and relevant history or terminal-state evidence.When scores differ, the higher score must win; equal integer scores require TIE.Source: §B.6.3 · pp. 32–33

C.1. Numerical Performance Results

The numerical results compare action-scaling configurations, decision-only verification costs, additional verifier inference, and uncertainty in gains over the base agent.

C.1. Numerical Performance Results

  • At N=4, both decision-only variants have lower Pass@1 than their reasoning-based counterparts, so the higher success observed at N=8 does not extend to both widths.Table 6 reports action-scaling results for zero-shot TMAX-9B verifiers and the GPT-5.6 Sol frontier verifier.Source: §C.1 · p. 34, Table 6 · p. 34

C.2. End-to-End Costs with Decision-Only Verification

  • At N=8, decision-only verification cuts total output tokens by 62.8% for zero-shot and 44.7% for distilled Mid-Harness, while reference-priced costs fall by 20.9% and 24.1%, respectively, and Pass@1 increases.At N=4, reference-priced costs also fall by 29.5% and 20.8%, respectively, although Pass@1 is lower than with pairwise verification using explicit reasoning.Source: §C.2 · p. 35, Figure 8 · p. 35, Table 9 · p. 36
  • With distilled decision-only verification, Best-of-T composition costs 23.7% less with a 1.02-point lower Pass@1, while SR composition costs 22.4% less with a 5.78-point higher Pass@1.The reported costs include the source trajectories and trajectory verification for Best-of-T, and the source trajectory, summary, and refinement run for SR.Source: §C.2 · pp. 35–36, Figure 8 · p. 35

C.3. Additional Verifier Inference Compute

  • Sampling five responses per comparison lowers both success measures despite nearly five times the verifier output; rubric scaling yields only small gains with more than twice the output.The thinking-enabled configuration reaches higher Pass@1 with nearly five times the verifier output.Source: §C.3 · p. 36, Table 10 · p. 36

C.4. Uncertainty in Comparisons against Base Agent

  • All six Pass@1 point estimates favor Mid-Harness; bootstrap intervals lie above zero for distilled 9B and 27B, while the other four include zero.The zero-shot 9B estimate improves by 4.76 percentage points, with an interval of [−0.68, 10.20].Source: §C.4 · pp. 36–37, Table 11 · p. 37

C.5. Mid-Harness Across Task Groups

Task-group analyses find gains across several domains and difficulty groups, with mixed domain-specific effects and no universal trend by execution length.

C.5. Mid-Harness Across Task Groups

  • Distilled Mid-Harness improves Pass@1 in software engineering, machine learning, debugging, and system setup at all three model sizes, but scientific-computing results are mixed.Software-engineering gains are 2.6, 5.1, and 20.5 percentage points for 4B, 9B, and 27B; scientific-computing Pass@1 falls 10.0 points for 27B.Source: Figure 9 · p. 37, §C.5 · p. 38
  • Distilled verification improves Pass@1 in every difficulty group at all three model sizes; Hard-task gains are 2.3, 6.9, and 10.3 points for 4B, 9B, and 27B.Extreme and build/dependency groups contain only three and four tasks, respectively, so their large percentage changes reflect small task populations.Source: Figure 9 · p. 37, §C.5 · p. 38
  • Distilled verification improves Pass@1 in the Long group by 11.1, 5.1, and 14.1 points for 4B, 9B, and 27B, but gains do not rise monotonically with execution length across models.The groups are defined from each model’s observed base-agent shell-call behavior, not a shared intrinsic task horizon.Source: Figure 9 · p. 37, §C.5 · p. 38, Table 12 · p. 38

D. Verifier Analysis

The appendix defines offline verifier-agreement diagnostics, examines teacher-relative disagreements, and traces qualitative cases where action selection supports recovery or fails.

D.1. Offline Verifier Agreement

  • The diagnostics retain 1,355 of 1,624 states with complete valid comparisons and plot comparative score distributions, not task-success probabilities.The score plots contain 20,394 candidate-score observations and 10,197 gaps; joint A/B panels preserve candidate orientation.Source: §D.1.1 · pp. 38–39, Figure 10 · p. 39, Figure 11 · p. 39

D.1.1. Score and Turn Diagnostics

  • Offline pairwise agreement compares A/B/TIE preferences; verification agreement compares each model’s top candidate by win counts, not online margin-weighted scores.Verification agreement is conditional on states with complete valid responses for both models.Source: §D.1.1 · p. 39

D.2. Remaining Verifier Disagreements

  • Candidate semantics and execution feasibility accounted for 1,926 zero-shot cases and 1,220 distilled cases, the latter 67.4% of reviewed distilled failures.These are model-judged disagreements against a teacher reference, not independently established action errors or observed trajectory failures.Source: §D.2 · p. 40

D.2.1. Examples of the Two Largest Failure Categories

  • Examples show the verifier rejecting equivalent C integer additions and favoring a launch command that can terminate itself before starting the server.The reference preferred the OCR-grounded search in the first example and the alternative launch action in the second.Source: §D.2.1 · pp. 40–41

D.3. Qualitative Examples of Action Verification

  • Across three React runs, the base agent succeeds 0/3 and zero-shot Mid-Harness 3/3; an alias-task verifier choice fails its corresponding test.The alias run finishes with 11/12 tests passed and reward 0.Source: §D.3 · p. 41

D.3.1. Detailed Action-Verification Traces

  • Detailed traces retain original candidate indices, local A/B labels, verifier responses, and subsequent outcomes.The examples cover one recovery and one verifier error using eight sampled candidates and pairwise verification in a four-pivot tournament.Source: §D.3.1 · p. 41, §D.3 · p. 41

Recovery Example: React and TypeScript Production Build

Verifier Error: Build-System Alias Canonicalization

  • In the alias task, the verifier favored a speculative header-excluding canonicalization despite the prompt conflict; the run passed 11/12 tests but received reward 0.The evaluator failure matched the alias rule introduced by the chosen candidate.Source: Verifier Error: Build-System Alias… · pp. 43–44

Related papers

Loading 2609.39982v1…