PaperlayerView exact revision on arXiv

Source-linked AI summary

More Programs or More Rolls? Separating Coverage from Specialization in LLM Harnesses

Ziyang Xu, Haitian Zhong, Hao Zhou, Hao Qin, Chenhan Jin, Te Qi, Shengze Xu, Tieyong Zeng

arXiv:2609.35873v1cs.AIcs.LGcs.SE

TL;DR

The paper asks whether generated LLM harnesses provide useful task specialization beyond the coverage available from repeated executions of one program. It compares generated harnesses with byte-identical controls and evaluates repeatable advantages, selection, and execution-matched coverage. Generated differences chiefly identify persistent weaknesses, while the tested selector gains nothing and fixed-program executions match full-budget oracle coverage.

  • Problem

    Additional answer coverage from a harness population can arise from repeated attempts, leaving unclear how much different programs contribute beyond further executions of one program.

    Source: §6 · p. 9

  • Method

    The study compares generated harnesses with byte-identical baseline copies, tests held-out repeat-based choices and a frozen selector, and replays oracle coverage at matched harness-execution budgets.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    The frozen selector gains 0.00 pp, and both arms reach 98.70% oracle coverage at 27 harness executions, while persistent generated-program differences are dominated by losses.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Harness diversity needs reproducible task advantages that a usable decision rule can exploit and that add value over further executions of a fixed program.

    Source: §7 · p. 10

    The MATH evidence covers one solver, a selected low-call panel, and three repeats, leaving stable complementarity unresolved without establishing equivalence.

    Source: §6 · p. 9

Abstract

from arXiv · show

Automated generation of LLM harnesses promises to improve inference through task specialization. Yet additional answer coverage can arise from repeated execution of the same program, making specialization difficult to identify. We introduce a controlled evaluation that separates answer coverage, repeatable task advantages, and gains from pre-execution selection. On 386 MATH-500 tasks, we compare eight generated harnesses plus a baseline with nine byte-identical baseline copies, using three executions per member. Identical programs yield 2.16 percentage points of repeat-averaged oracle headroom. Generated programs exhibit substantially more repeatable score patterns, but these chiefly reveal persistent weaknesses: losses relative to the baseline persist across all three repeats on 100 tasks, while persistent wins occur on only one task and are sensitive to answer extraction. The frozen selector gains 0.00 percentage points, and both populations reach 98.70% oracle coverage at 27 harness executions. Stable complementarity remains unresolved at three repeats. Supporting BIRD traces locate failures in mechanism implementation, activation, and output validity. Together, these findings establish why coverage and repeatability alone cannot justify claims of useful specialization. They motivate an evaluation standard for harness diversity: task advantages should persist across executions, guide usable decisions, and improve on additional fixed-program executions under matched inference budgets.

1 INTRODUCTION

The paper asks whether generated harness populations provide useful task specialization or merely more attempts, since repeated executions of identical programs can also expand answer coverage. It evaluates coverage, repeat-stable advantages, and pre-execution selection separately, finding persistent generated-program differences dominated by losses and no gain from the tested selector.

1 INTRODUCTION

  • A population can contain more correct answers because it supplies more attempts, so oracle coverage alone cannot establish that different programs provide the gain.Repeated sampling and aggregation methods can exploit complementary successes even when every member has identical code.Source: §1 · p. 2
  • The evaluation separates answer coverage, repeat-stable task advantages, and usable selection because repeatable score patterns can reflect persistent weaknesses rather than exploitable specialization.Useful pre-execution specialization requires advantages that recur and can be recognized from the task.Source: §1 · p. 2
  • The study compares generated harnesses with byte-identical baseline copies, tests choices learned from earlier executions on held-out repeats, freezes a selector on development data, and replays equal execution budgets.Separate BIRD traces examine whether proposed mechanisms are implemented, activated, and able to return valid answers.Source: §1 · p. 2
  • −0.26 pp is the clone-adjusted held-out gain; persistent losses span 100 tasks, persistent wins occur on one extraction-sensitive task, and the frozen selector gains 0.00 pp.Both populations reach 98.70% oracle coverage at 27 harness executions, while stable complementarity remains unresolved at three repeats.Source: §1 · p. 2
  • The study’s target is task advantages that persist, guide a usable decision, and add value beyond further executions of a fixed program.The analyses combine a same-code sampling reference, directional repeated-win/loss analysis, and separate pre-execution choice and post-execution coverage evaluations.Source: §1 · p. 2

2 RELATED WORK

Related work covers automated harness design, repeated sampling and ensembles, and task-dependent selection or resource allocation. This paper connects these areas by examining program populations through execution traces and repeated outcomes, while comparing their gains against same-code sampling.

Automated agent and harness design

  • Automated harness work searches or edits executable programs, adapts harnesses from traces, and studies adaptation, validation, diagnosis, and policy evolution.The paper also connects program equivalence and mutation testing with performance attribution in generated harness populations.Source: §2 · p. 2

Repeated sampling and ensembles

  • Repeated sampling can improve answer coverage with a fixed program, while post-execution selection and pre-execution routing represent distinct decision timings.The paper uses same-code repeats as a sampling reference and distinguishes its frozen program selector from oracle replay across recorded outputs.Source: §2 · pp. 2–3

Selection and resource allocation

  • Algorithm selection and portfolio research motivates task-dependent choices, while the paper measures oracle coverage, repeat-stable advantages, and realized selector gain within one fixed-model population.The distinction between quantitative and qualitative interaction explains how effect sizes can vary without changing the best action.Source: §2 · p. 3

3 PROBLEM FORMULATION

The formulation distinguishes available correct answers, persistent task-wise advantages, and decisions that can use those advantages. It defines oracle headroom against the best fixed member and separates population-level complementarity from finite-repeat selection.

Coverage

  • The score Y(h, x, r) is binary correctness for member h on task x in repeat r, and proportions convert to accuracies or gains by multiplying by 100.The population includes the single-call baseline H0.Source: §3 · p. 3
  • A one-run oracle selects a member after observing correctness, and its headroom measures available correct answers relative to the best fixed member.The repeat-averaged plugin replaces single-repeat scores with each member’s observed mean, but few repeats can make that estimate favor noise.Source: §3 · pp. 3–4

Stable complementarity

  • Stable complementarity requires task-wise crossover: the better member changes with the task, rather than merely having a varying advantage size.Identical programs under the same condition have equal success probabilities and therefore zero stable complementarity.Source: §3 · p. 4
  • A task-dependent performance gap can leave one member preferable on both task types, producing zero stable headroom despite variation in advantage size.In the constructed example, H0 has probabilities 0.90 and 0.60 while H1 has 0.80 and 0.20; both the task-wise oracle and best fixed member achieve 0.75.Source: §3 · p. 4

Usable selection

  • A usable selector maps pre-execution task features to a program, while oracle replay compares portfolio coverage with further fixed-program executions.Replay counts harness executions, although each execution may contain multiple model calls and tokens.Source: §3 · p. 4

4 EXPERIMENTAL DESIGN

The experimental design holds the solver fixed while comparing generated harnesses with same-code controls, then tests repeat-based choices and execution-budget allocation. It also calibrates the repeat diagnostic and evaluates a frozen question-based selector separately from post-execution oracle replay.

4.1 HARNESS POPULATIONS AND EVALUATION SETUP

  • The study holds GLM-5.3-Flash fixed while varying Python harnesses, and its primary MATH comparisons use 386 complete evaluation problems after excluding 14 incomplete cases.The selected low-call panel contains eight generated programs plus the single-call baseline, with three fresh repeats per member; nine byte-identical baseline slots form the control.Source: §4.1 · pp. 4–5
  • The generated MATH programs use distinct answer-resolution strategies, but their presence does not establish that a mechanism activates or improves answers.The BIRD admission study varies free versus forced strategies and ungated versus conformance-gated admission; evaluation uses a shared 400-question core across arms.Source: §4.1 · p. 5

4.2 SAME-CODE CONTROLS AND REPEAT VALIDATION

  • Held-out selection gain compares choices learned from two repeats against the best fixed member on the third, with generated-panel gain differenced against the same-code arm.A positive difference indicates better transfer for distinct programs, but it is not an interval for population-level stable headroom; positive support also requires effect-size, uncertainty, and completeness criteria.Source: §4.2 · pp. 5–6
  • Residual correlation measures repeatability after removing additive member and task effects, while held-out selection gain measures choice performance; known-truth simulations assess diagnostic sensitivity.The repeat diagnostic and common-complete-set analysis were developed after collection, while recovery retained the programs, solver, tasks, and repeat design.Source: §4.2 · p. 6
  • Known-truth calibration includes zero-headroom settings and a strong two-specialist case with 17.5 pp of true headroom.The specialist case uses 400 tasks split evenly, with each specialist at 0.95 on its own half and 0.60 on the other; the best fixed accuracy is 0.775.Source: §4.2 · p. 6

4.3 FROZEN SELECTION AND EXECUTION-MATCHED REPLAY

  • The frozen selector predicts a task’s best member from development data and task features, then falls back to the single-call program when the top-two probability gap is below 0.15.Its policy is fixed before evaluation, and its accuracy is compared with the development-selected fixed member.Source: §4.3 · p. 6
  • Offline oracle replay compares a development-ranked portfolio with same-code executions at budgets of 3, 9, 15, or 27 harness executions.Both arms use post-execution correctness, and model-call and token use can vary by program despite the matched harness-execution budget.Source: §4.3 · p. 6

5 RESULTS

The generated population covers more tasks than any single member, but repeatable differences chiefly expose persistent losses rather than usable specialization. The tested selector adds no gain, full-budget oracle coverage ties with same-code controls, and BIRD traces locate failures between code and valid execution.

5.1 THE POPULATION COVERS MORE TASKS THAN ANY SINGLE MEMBER

  • 99.22% oracle coverage from the full generated population plus single-call exceeds 95.08% for the best fixed member, while every generated member is individually worse.The union adds 4.15 pp of coverage available to an oracle after execution.Source: §5.1 · p. 7

5.2 IDENTICAL PROGRAMS ALSO PRODUCE POSITIVE ORACLE ESTIMATES

  • 2.16 pp repeat-averaged oracle headroom appears among identical programs, compared with 2.33 pp for the generated panel, so positive oracle estimates alone do not identify specialization.The nine clone slots implement the same program, making their population-level stable complementarity zero by construction.Source: §5.2 · p. 7, Table 2 · p. 7

5.3 REPEATABLE SCORE PATTERNS ARE DOMINATED BY LOSSES

  • Generated programs have more repeatable scored-outcome patterns than clones, but the held-out clone-adjusted mean is D = −0.26 pp and provides insufficient evidence of positive selection gain.The interval includes zero; the result is not an equivalence conclusion.Source: §5.3 · p. 7
  • 100 tasks show all-three-repeat losses, while all-three-repeat wins occur on one task and are sensitive to answer-format extraction.The sole persistent winning task is scored differently because the baseline's “#### 12th grade” payload is marked incorrect while a generated answer reaches numeric fallback and is marked correct.Source: §5.3 · p. 7
  • At three repeats, the tested diagnostic has 0% power in the simulated strong crossover case, so the empirical nonpositive result leaves stable complementarity unresolved.The diagnostic reaches 100% power at R = 10 and R = 20 in that setting, while smaller cyclic crossovers remain largely undetected even at R = 20.Source: §5.3 · pp. 7–8

5.4 THE TESTED SELECTOR ADDS NO GAIN AND FULL-BUDGET COVERAGE TIES

  • The tested frozen selector gains 0.00 pp, and at 27 executions the development-ranked portfolio and same-code controls both reach 98.70% oracle coverage.The frozen classifier selects single-call on every evaluation task; its accuracy over three repeats matches the development-selected fixed member at 95.85%.Source: §5.4 · p. 8, Table 3 · p. 8

5.5 BIRD TRACES LOCATE BREAKS BETWEEN CODE AND EXECUTION

  • BIRD traces distinguish absent mechanisms, inactive repair branches, and broken outputs; a separate admission comparison reports −0.16 pp with no increase in observed mean headroom.Six of twelve candidates describe retrieval or self-checking without implementing either; the audited repair baseline never triggers its branch, and one candidate returns prose as final SQL.Source: §5.5 · pp. 8–9

6 DISCUSSION AND LIMITATIONS

The discussion distinguishes repeatable score differences from task-wise complementarity and usable population value. The study's scope and diagnostic power leave stable complementarity unresolved, while the full-budget replay tie does not establish equal-cost utility.

6 DISCUSSION AND LIMITATIONS

  • Repeatable disadvantages can vary in size without changing which member is preferable; complementarity requires different members to retain advantages on different tasks.The paper frames the key evaluation comparison as what changing programs adds beyond further executions of a fixed program.Source: §6 · p. 9
  • A useful mechanism must be implemented, activated for the relevant input, and produce a valid answer; execution-success triggers can miss semantically wrong SQL.The proposed follow-up compares enabled and disabled versions on tasks exercising the intended trigger, then tests any advantage on fresh executions.Source: §6 · p. 9
  • Population value depends on a usable decision: routing needs recognizable pre-answer advantages, while multi-program execution needs an answer-selection or aggregation procedure.The discussion calls for repeat counts calibrated to detect crossovers and an implementable decision rule under a matched resource budget.Source: §6 · p. 9
  • The MATH evidence covers one solver, a selected low-call panel, and three repeats; the tested diagnostic can miss crossovers, and the result does not establish equivalence.Replay matches harness executions rather than model-call or token costs, so the observed tie does not establish equal-cost utility; BIRD uses separate populations and scoring conditions.Source: §6 · pp. 9–10

7 CONCLUSION

On the tested MATH panel, repeated baseline executions match the portfolio’s oracle coverage at 27 harness executions, while the frozen selector gains 0.00 pp. The conclusion sets a higher bar for specialization: reproducible task advantages must guide usable decisions and outperform further executions of a fixed program.

7 CONCLUSION

  • The frozen selector gains 0.00 pp, and repeated baseline executions match the portfolio’s 98.70% oracle coverage at 27 harness executions; stable complementarity remains unresolved at three repeats.Source: §7 · p. 10
  • Harness design should target reproducible task advantages that a usable decision rule can exploit, with gains measured against further fixed-program executions.Source: §7 · p. 10
  • The experiments use public BIRD and MATH-500 data and collect no human-participant data, private records, or personally identifiable information.Source: ETHICS STATEMENT · p. 10

A NOTATION SUMMARY

The notation distinguishes score and gain units, repeat-based estimates, selection diagnostics, and execution budgets. It also specifies the harness interface and points to the appendices documenting the MATH population and BIRD protocols.

A NOTATION SUMMARY

  • Scores and gains use proportions, while accuracies use % and gains use percentage points; a gain of 0.01 equals 1 pp.Source: A NOTATION SUMMARY · p. 13
  • Repeat-averaged plug-in headroom replaces single-run outcomes with repeat estimates, while stable population headroom is computed from success probabilities.Source: A NOTATION SUMMARY · p. 13
  • The notation separates clone-adjusted gain, residual repeatability, pre-execution selection, and harness-execution budgets from model-call and token counts.Source: A NOTATION SUMMARY · p. 14

B HARNESS GENERATION, PROGRAM INVENTORY, AND DATA SPLITS

B.1 MATH POPULATION, COMMON ANALYSIS SET, AND COVERAGE

The MATH population’s generated union expands single-run answer coverage, but the measured results describe one seed, one target, and a completeness-selected task set. The repeat-study panel and model inventory further delimit the evaluation setup.

B.1 MATH POPULATION, COMMON ANALYSIS SET, AND COVERAGE

  • The analysis uses 386 fully observed tasks after excluding 14 problems with incomplete repeat observations, and describes one seed and one target.Source: §B.1 · p. 15
  • 99.22% oracle coverage exceeds the best fixed member single-call (95.08%) by 4.15 pp on the common complete set of 386 tasks.The generated union repairs 84.21% of baseline errors.Source: §B.1 · p. 15

B.2 SELECTED MATH PROGRAM INVENTORY

  • The repeat-study table describes nine frozen members; its listed operations indicate what programs implement, not whether branches fire or improve correctness.Source: §B.2 · p. 15, Table 5 · p. 16

B.3 MODELS AND VERSIONS

  • The model inventory holds the target frozen within each phase, and Phase-II builders come from six independent labs.Source: Table 6 · p. 16

C SAME-CODE CONTROLS AND REPEATED-EXECUTION PROTOCOL

The repeated-execution protocol compares a generated panel with byte-identical baseline slots across three repeats on shared tasks. The diagnostic finds substantially more repeatable score patterns among generated members, but all-repeat losses greatly outnumber wins, which are sensitive to answer extraction.

C SAME-CODE CONTROLS AND REPEATED-EXECUTION PROTOCOL

  • The panel uses fixed programs, solver, judge, development split, and three repeat identities; nine identical baseline slots form the control.Source: C SAME-CODE CONTROLS AND… · p. 16
  • The clone-adjusted held-out gain is −0.26 pp (95% interval [−1.45, 0.95] pp; INSUFFICIENT EVIDENCE), leaving stable complementarity unresolved.Removing three conflict-affected tasks gives D = −0.61 pp (interval [−1.96, 0.64] pp), with the same decision.Source: C SAME-CODE CONTROLS AND… · p. 17

C.1 EXPLORATORY REPEATABILITY AND SCORE SENSITIVITY

  • Generated members’ residual score patterns correlate 0.789 across repeats, versus 0.006 for clones, after removing additive effects.Source: §C.1 · p. 17
  • All-repeat losses span 570 member–task pairs on 100 tasks, while all-repeat wins comprise eight pairs on one format-sensitive task.The extractor scores the baseline’s “#### 12th grade” as incorrect but the generated “12th grade” as correct.Source: §C.1 · p. 17

D STATISTICAL ANALYSIS, CALIBRATION, AND SENSITIVITY

The section separates finite-repeat task selection from population-level stable complementarity and calibrates how reliably the former detects the latter. It also specifies a frozen selector and execution-matched replay, then uses BIRD traces to distinguish harness failures and observed outcome patterns.

D.1 HELD-OUT-REPEAT STATISTIC AND DECISION RULE

  • The held-out statistic compares gains learned from discovery repeats in generated and clone arms, requiring positive support across completeness, task-count, randomization, bootstrap, and effect-size checks.The decision rule requires D > 1.0 pp and classifies the current evidence as insufficient, not equivalent.Source: §D.1 · pp. 17–18

D.2 POPULATION TARGET AND LIMITS OF THE DIAGNOSTIC

  • A near-zero clone-adjusted D does not bound population stable complementarity near zero, because D measures finite-data learning rather than access to true success probabilities.The randomization reference is exact under equal-ability member-label exchangeability; error control for the broader global-dominance null is empirical over the calibration grid.Source: §D.2 · p. 18
  • 8.69 pp in the equal-ability simulation and 2.16 pp in the empirical same-code arm show that positive plug-in headroom alone is insufficient evidence of specialization.The simulation uses M = 9, T = 400, and R = 3; its true stable complementarity is zero.Source: §D.2 · p. 18

D.3 KNOWN-TRUTH CALIBRATION AND STATISTICAL POWER

  • 0% power at R = 3 for the tested two-specialist case with Hstable = 17.5 pp rises to 100% at R = 10 and 20, while smaller cyclic crossovers remain largely undetected at R = 20.Table 8 summarizes the calibration at M = 9 and T = 400.Source: Table 8 · p. 19, §D.3 · p. 19

E.1 FROZEN FEATURE SELECTOR

  • The frozen selector predicts each task’s best member from question features and training-fold member accuracies, then falls back to single-call when the top-two probability gap is below 0.15.It uses five task folds and development data; evaluation reports repeat-mean accuracy with harness executions as the unit.Source: §E.1 · pp. 19–20

E.3 EXECUTION-MATCHED REPLAY AND COLLECTION ACCOUNTING

  • Execution-matched replay evaluates ranked panels at 3, 9, 15, and 27 harness executions against the exact expected coverage of uniform clone subsets.Each arm has nine members or slots with three repeats; coverage uses a post-execution oracle.Source: §E.3 · p. 20, §E.2 · p. 20

F.2 DISCOVERY MEASUREMENTS AND HAND-WRITTEN CONTROLS

  • Across 91 BIRD pairs, source similarity has ρ = −0.010 association with correctness disagreement, while 36/91 pairs have identical correctness vectors.The comparison covers observed outcomes, not program equivalence; generated pairs also show 21/66 identical correctness vectors.Source: Figure 2 · p. 21, §F.2 · p. 21

F.3 TRACES DISTINGUISH ABSENCE, NON-ACTIVATION, AND FAILURE

  • BIRD traces distinguish absent mechanisms, untriggered repair, and broken output: six of twelve candidates omit claimed steps, while one multi-turn repair path returns prose as SQL.Four hand-written controls yield 6.67 pp headroom on 60 tasks and 7.28 pp on 151 items, but lack matched repeats and clones and overlap development items.Source: §F.3 · p. 21

F.4 WORKED EXAMPLES FROM THE TRACE AUDIT

The worked traces show that nominal repair and retry mechanisms can be absent, untriggered, or unable to produce valid SQL. The admission study reports no observed headroom improvement from its implemented forced-and-gated bundle and audits sensitivity, repeat variability, and recorded cost.

F.4 WORKED EXAMPLES FROM THE TRACE AUDIT

  • The audited single-call candidate makes no SQL executions, the timeout watchdog never triggers, and the multi-turn repair candidate returns prose that fails the SQL output contract.For react, all 10 audited final queries are byte-identical to single-call’s; the repair failure flips a baseline-correct item to harness-wrong.Source: §F.4 · pp. 22–23

G BIRD ADMISSION-POLICY STUDY

  • The 2×2 admission study crosses free versus forced strategy with ungated versus gated admission, using six builders and three seeds (18 builder–seed cells per arm) and eight generation slots per cell.II-E is development-only; evaluation uses 1,169 questions from nine databases and a pre-drawn 400-item core.Source: G BIRD ADMISSION-POLICY STUDY · p. 23, §G.1 · p. 23

G.1 DESIGN AND PRIMARY CONTRASTS

  • −0.16 pp is the forced/gated minus free/ungated contrast in baseline-inclusive oracle headroom; on core-400, the gate contrast is −0.69 pp and the candidate-count-matched contrast is −0.31 pp.The strategy and interaction effects on core-400 are inconclusive.Source: Table 10 · p. 24, §G.1 · p. 24, Table 11 · p. 24

G.2 SAME-CORE HEADROOM DECOMPOSITION

  • Table 12 separates coverage, best-fixed accuracy, and headroom on the same core tasks and single-call outcomes; headroom is oracle minus best-fixed accuracy, not realized routing gain.Source: Table 12 · p. 25, §G.2 · p. 25

G.3 SENSITIVITY, REPEAT AUDITS, AND RECORDED COST

  • R3 changes correctness on 58/400 baseline tasks (14.5%) and produces a 6.50 pp same-code oracle gap; R2 records 8.54% flips, with acquisition time and caching confounded.Recorded costs count logical solver calls, including cache-served requests, rather than billed requests or tokens.Source: §G.3 · p. 25, Table 13 · p. 25, Table 14 · p. 26

G.4 ADMISSION SCOPE AND IMPLEMENTATION

The admission study’s free-arm strategy choice is limited to four declarations, and the evaluated gate is a specific implemented screening-and-conformance bundle. Its benchmark results therefore describe that bundle, not later revisions.

G.4 ADMISSION SCOPE AND IMPLEMENTATION

  • Free builders choose among repair, vote, twostage, and plain declarations, so strategy choice is confined to that four-item vocabulary.Source: §G.4 · p. 26
  • The implemented structural screen can reject valid prompt-only transformations, and later gate revisions have no replacement benchmark outcomes.Development checks also found false rejections, attribution and interface defects, and unresolved judgments.Source: §G.4 · p. 26
  • All 108 format-guard attempts fail neutral validity, with logs reporting unterminated strings, missing Python fences, or a provider timeout.Unavailable builder messages and historical extractor versions limit attribution of these upstream validity failures.Source: §G.4 · p. 26

Related papers

Loading 2609.35873v1…