PaperlayerView exact revision on arXiv

Source-linked AI summary

Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents

Yuyao Ge, Shenghua Liu, Peizhong Gao, Chendi Ge, Chenxu Liu, Yiwei Wang, Lingrui Mei, Baolong Bi, Jiayu Yao, Jiafeng Guo, Xueqi Cheng

arXiv:2610.20344v1cs.SEcs.AI

TL;DR

In complex LLM-agent settings, the harness shapes how model capability reaches the environment, while supervising every step has become difficult. Doc2Harness evaluates harness construction by building candidate harnesses and testing them with scripted mock models and positive controls. GPT-6 Astra scored 53.05% pass rate versus 27.44% for the weakest model, and failures cluster around functionality split across modules.

  • Problem

    As LLM agents enter complex settings, the harness governs tool use, context, permissions, session state, failure recovery, and verification, while supervising every step has become difficult.

    Source: §1 · p. 2

  • Method

    Doc2Harness has build models construct candidate harnesses from an interface-only skeleton and development document, then tests them end-to-end with scripted mock models and positive controls.

    Source: §1 · p. 2

  • Results

    53.05% pass rate for GPT-6 Astra versus 27.44% for the weakest model; even the strongest model fails nearly half of cases, with failures clustering around functionality split across modules.

    Source: §1 · p. 2, §7 · p. 14

  • Takeaways & Limitations

    Doc2Harness provides a deterministic benchmark for assessing how well models develop harnesses.

    Source: §7 · p. 14

    The protocol measures single builds, not improvement through iterative build–evaluate–repair with behavioral feedback.

    Source: §6 · p. 13

Abstract

from arXiv · show

Agents driven by large language models are increasingly deployed on long-horizon tasks, where they use tools and modify real workspaces. How well they perform depends as much on the harness around the model as on the model itself. Existing benchmarks measure downstream task success under a fixed harness, leaving harness engineering itself unexplored. We propose Doc2Harness, a two-stage benchmark that evaluates long-horizon harness engineering directly. In Stage 1, a build model operating inside a fixed builder harness receives an interface-only contract skeleton and a development document that consolidates behavior across twelve frontier harnesses, and then constructs a complete candidate harness in an isolated container. In Stage 2, the frozen candidate harness runs in containers separate from the construction environment, one per test case, and is driven end to end by a scripted mock model. We score each test case using four observable channels, namely the requests sent to the model, the files on disk, the process exit code, and the command output; positive controls rule out vacuous passes. Across nineteen frontier models, each evaluated with three independent builds on 328 test cases, GPT-6 Astra achieves the highest pass rate at 53.05%, while the weakest model reaches 27.44%, a gap of roughly twenty-six percentage points. Even the strongest model fails nearly half of the test cases, and failures cluster around how cross-cutting functionality is split across modules. To our knowledge, Doc2Harness provides the first deterministic benchmark for measuring long-horizon harness engineering.

1 Introduction

Doc2Harness targets the gap between evaluating downstream coding tasks under a fixed harness and evaluating the engineering of the harness itself. It builds candidate harnesses from documentation and interface-only contracts, then evaluates them behaviorally; results show substantial differences across models and persistent implementation challenges.

1 Introduction

  • Existing software-engineering benchmarks use mature repositories and fixed-interface tests, which do not directly assess harness correctness through a driven model.Harnesses manage interaction, tools, permissions, state, recovery, and verification, shaping how much model capability reaches the environment.Source: §1 · p. 2
  • Doc2Harness builds a candidate harness from a development document and interface-only contract, then drives it end to end with a scripted mock model.Evaluation observes model requests, files, exit code, and command output; positive controls help rule out vacuous passes.Source: §1 · pp. 2–3, Figure 2 · p. 3
  • 53.05% for GPT-6 Astra versus 27.44% for the weakest model across nineteen models; category ranks vary by up to sixteen positions.Each model was evaluated with three candidate harnesses.Source: §1 · pp. 2–3
  • Higher-scoring candidate harnesses tend to place cross-cutting functionality in fewer modules.Source: §1 · p. 3

2 Doc2Harness

Doc2Harness separates candidate construction from behavioral evaluation. The build is frozen after a form-focused contract check, then tested in isolated containers with a scripted mock model.

2.1 Task Description

  • Stage 1 combines the evaluated build model with a fixed builder harness to construct a candidate from a development document and contract skeleton.The contract check verifies compilation, installation, required commands, absence of stubs, and three interfaces, but runs no test cases.Source: §2.1 · p. 3
  • A successful form check freezes the candidate; a failed check returns it for revision, with up to three submissions allowed.Each scored candidate reflects the model's submitted state, not simply the state at the time limit.Source: §2.1 · pp. 3–4
  • Stage 2 runs the frozen candidate in a separate container per test case, where a scripted mock model drives it and positive controls gate PASS status.Model ability is reflected indirectly in the pass rate, so models rather than candidate harnesses are ranked.Source: §2.1 · p. 4
  • OpenCode is fixed as the builder harness, while scripted mock responses avoid instability from a real LLM during evaluation.Source: §2.1 · p. 4

2.2 Task Inputs

The task inputs aim to make candidate behavior observable while leaving implementation choices open. The development document consolidates and disambiguates features, while the small contract skeleton fixes only external interfaces.

2.2.1 Development Document

  • The development document is built by collecting a feature inventory, turning it into clauses, and disambiguating those clauses for consistency.The inventory draws on user documentation from twelve frontier harnesses and is merged, deduplicated, and pruned.Source: §2.2.1 · p. 4
  • Each clause specifies syntax, meaning, and expected behavior, with feature effects observable through requests, files, exit code, or command output.Apart from the contract interfaces, implementation manner is unconstrained; overlapping commands are merged while distinct concepts sharing a name are separated.Source: §2.2.1 · pp. 4–5

2.2.2 Contract Skeleton

  • The contract skeleton is a stripped-down mini-swe-agent framework chosen to maximize model-written code while keeping the candidate observable.Source: §2.2.2 · p. 5
  • The skeleton contains about seventy lines across five Python files and fixes three interfaces: the CLI, headless entry point, and evaluation-driving API.A representative candidate has about 30 Python files and 16,000 lines, roughly 99.6% written by the build model.Source: §2.2.2 · p. 5
  • Each build starts in a newly created empty container without external network access apart from the designated model gateway.Preflight checks verify the container contents and that OpenCode starts normally.Source: §2.2.2 · p. 5

2.3 Test Case Synthesis

Test cases are designed to be clause-anchored and discriminative, using three methods in a yield-driven cycle. The methods draw on public issues, mature harness implementations, and controlled variations of existing tests.

2.3 Test Case Synthesis

  • The synthesis cycle uses public records to broaden coverage, source comparison to broaden it further, and single-variable expansion to deepen inspection.The next method begins when the current method's marginal yield declines; each method uses filtering suited to its source.Source: §2.3 · p. 5

2.3.1 Public Issue Mining

  • Public issue mining ranks more than 20,000 records, clusters repeated defects, and retains only candidates that yield deterministic assertions not already covered.Records are reviewed from the highest-ranked downward until marginal yield falls below review cost; redundant cases are filtered when they would pass and fail together.Source: §2.3.1 · p. 6

2.3.2 Cross-Harness Comparison

  • Cross-harness comparison partitions candidate and mature-harness source by feature, then tests document-grounded behavioral differences rather than implementation differences alone.Its coverage is bounded by the comparison target's functional surface.Source: §2.3.2 · pp. 6–7

2.3.3 Single-Variable Expansion

  • Single-variable expansion changes one scenario variable while keeping its assertion fixed, varying tools, input forms, erroneous parameters, or result location.Candidates too close to their seed are discarded; retained cases seed later rounds, and the method alternates with the first two to fill unseeded gaps.Source: §2.3.3 · p. 7

2.4 Positive Control

Positive controls check whether the observation precondition holds, so an apparent test pass reflects the required behavior rather than a failure of observation. A failed control gates its associated cases as FAIL, regardless of their own status.

2.4 Positive Control

  • A positive control reuses the test scenario and fixture but changes the input or access route to check an outcome that should occur outside the tested constraint.For path escape, the control reads the same external symlink through bash and confirms the file is readable.Source: §2.4 · p. 8
  • If any gating control fails, the test case is recorded as FAIL regardless of its own status; controls do not count toward the pass rate.Cases sharing an observation precondition can share a control, and cases with several preconditions can be gated by several controls.Source: §2.4 · p. 8
  • The three control classes check that results reach the observed channel, that an unconstrained access works, and that valid inputs are not categorically refused.The reverse control uses an in-workspace symlink to test whether the implementation accepts a valid symlink read.Source: §2.4 · p. 8

2.5 Statistics

The suite contains 328 cases grouped into nine mutually disjoint categories. Positive controls gate a subset of cases to verify their observation preconditions.

2.5 Statistics

  • The 328 test cases are partitioned into nine mutually disjoint categories based on the development document and the contract skeleton’s three external contracts.The category breakdown is shown in Figure 4.Source: §2.5 · p. 8
  • 130 of the 328 test cases are gated by at least one positive control: 114 by one, 15 by two, and one by three.Positive controls check whether the observation preconditions shared by test cases hold.Source: §2.5 · p. 8
  • The widest-scope control gates six test cases and checks that an external symlink is readable; two others each gate five cases to check range reading and search hits.These controls check preconditions shared by the most test cases.Source: §2.5 · p. 8

3 Experimental Setup

The evaluation uses nineteen models from eleven vendors and runs both benchmark stages on the same server. The builder and build limits are specified, and each Stage 2 case runs in its own container.

3 Experimental Setup

  • The evaluation covers nineteen models from eleven vendors, including four Claude Opus models and three GPT models.The listed GPT models are GPT-5.5, GPT-5.6-sol, and GPT-6 Astra.Source: §3 · pp. 8–9
  • Both stages run on an AMD EPYC 7K83 server with 48 logical processors and 128 GB of memory.Source: §3 · p. 9
  • Stage 1 uses OpenCode 1.17.0, with up to 24 turns, a two-hour wall-clock limit, and at most three submissions; each Stage 2 case has a 900-second limit.Stage 2 runs each test case once in its own container and fixture; Figure 4 indicates positive-control coverage varies by category, with artifact checks marked NA.Source: §3 · p. 9, Figure 4 · p. 9

4 Results

Across the 328-case suite, GPT-6 Astra scores highest, but even its pass rate is just over half. Results vary across categories, and architecture analyses associate higher scores with less fragmented harnesses.

4.1 Main Results

  • 53.05% is GPT-6 Astra’s highest overall pass rate, versus 27.44% for Hy3; even the strongest model fails nearly half the cases.The rightmost column of Table 2 reports pass rate over all 328 cases as the mean ± standard deviation across three builds; positive controls are excluded from denominators.Source: §4.1 · p. 9, Table 2 · p. 10
  • 80.68% is Opus 5’s ordinary-scenario pass rate, versus 72.46% for GPT-6 Astra; on boundary scenarios, Astra scores 48.97% versus Opus 5’s 37.02%.The reported overall gap between these two models is concentrated in their handling of boundary clauses.Source: §4.1 · p. 9

4.2 Results by Category

  • 50.8 points is the permissions category’s strong-weak gap, with a 43.7% average pass rate and 51 of 61 cases discriminating among models.File operations averages 39.6% with a 20.9-point gap, while interactive terminal averages 79.9% and is approaching saturation.Source: §4.2 · p. 10, Table 3 · p. 10
  • 56 cases are failed by every candidate harness, concentrated in direct drive, session state, permissions, and headless categories.These floor cases provide no discrimination within the evaluated cohort.Source: §4.2 · p. 11
  • Ranks vary by as many as 16 positions across the six main categories, although all 15 category-pair correlations are positive, averaging +0.521.The paper describes this pattern as consistent with largely one-dimensional capability differences rather than clear category trade-offs.Source: §4.2 · p. 11
  • A KR-20 coefficient of 0.913 across 57 harnesses and all test cases indicates high internal consistency.The authors describe this as expected if the cases measure one underlying capability.Source: §4.2 · p. 11

4.3 Architecture of the Candidate Harnesses

  • 95.9% of code-line variance is between models; mean code size ranges from 1,271 to 16,681 lines, and code lines correlate with pass rate at 0.629 between model means.Figure 5 plots code lines against pass rate and shows the three scored builds per model; Table 4 summarizes architecture over each model’s three builds.Source: §4.3 · pp. 11–12, Table 4 · p. 11, Figure 5 · p. 12
  • Controlling for code lines, module count has a −0.423 partial correlation with pass rate, while lines per module has a +0.504 partial correlation.Leave-one-model-out checks keep these correlations between −0.484 and −0.309 for module count and between 0.312 and 0.649 for lines per module.Source: §4.3 · p. 12

5 Related Work

Prior benchmarks cover code completion and repository generation, or evaluate existing harnesses and individual harness capabilities. Doc2Harness instead tests deterministic, clause-by-clause behavioral conformance of a newly developed harness.

5.1 Code and Software Engineering Benchmarks

  • Code benchmarks range from function-level completion through repository-level issue resolution and full repository generation, with evaluation criteria varying by available references.Examples include unit tests, upstream tests, output equivalence, fuzz-generated tests, and task-specific verifiers.Source: §5.1 · pp. 12–13

5.2 Harness Benchmarks

  • Harness benchmarks evaluate existing harness choice, configuration effects, or individual capabilities, rather than directly measuring harness construction.Downstream-task scores can confound harness quality with test-time search and the model driving the tasks; output-only scoring can also miss unauthorized-resource trajectories.Source: §5.2 · p. 13

5.3 Position of This Work

  • Doc2Harness consolidates features from twelve frontier harnesses into a new specification, then tests a candidate end to end with a scripted mock model and four observable channels.Positive controls establish that the feature under test ran, enabling clause-by-clause checks, including whether a rejected operation occurred.Source: §5.3 · p. 13

6 Limitations

The protocol captures a single build without behavioral-feedback-based improvement after freezing. Test variants can also be correlated, so one defect may cause multiple failures.

6 Limitations

  • A build allows up to three submissions, but the protocol does not measure iterative repair from behavioral feedback after the candidate harness is frozen.Because the contract check tests form rather than behavior, an early submitter’s pass rate is a lower bound, not an upper bound.Source: §6 · p. 13
  • Related test variants remain correlated despite a distance check, so one defect can fail several cases at once.The variants are derived from a shared seed through single-variable expansion.Source: §6 · pp. 13–14

7 Conclusion

Doc2Harness deterministically evaluates harness development by testing a completed candidate through scripted sessions and explicit behavioral assertions. Experiments find that even the strongest model fails nearly half the cases, with failures concentrated around module-splitting choices.

7 Conclusion

  • Doc2Harness uses a fixed builder harness and an interface-only contract skeleton to construct a candidate, then tests it with a scripted mock model and explicit behavioral assertions.The paper presents it as the first benchmark to assess harness development deterministically.Source: §7 · p. 14
  • Even the strongest model fails nearly half the test cases, with failures concentrated on whether functionality is split across several modules.The paper proposes Doc2Harness as a benchmark for measuring this construction ability.Source: §7 · p. 14

Related papers

Loading 2610.20344v1…