PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated Discovery Has No Universally Superior Harness

Akshat Gupta, Jermaine Lei, Alexander Lu, Gopala Anumanchipalli, Leshem Choshen

arXiv:2607.18235v1cs.CLcs.AI

TL;DR

Because discovery runs are stochastic and harnesses combine multiple design choices, it is difficult to tell whether their ingredients transfer across models and problems. The paper decomposes and evaluates budget-matched harnesses with repeated trials, then tests online allocation using partial-run feedback. No fixed harness is universally superior, while adaptive allocation improves average final performance over committing to one harness or running an unpruned ensemble.

  • Problem

    Because harness evaluations often use few independent trials despite stochastic discovery runs and composite design choices, it remains unclear whether harness benefits transfer across models and problems.

    Source: §1 · p. 2

  • Method

    The paper decomposes two harness families and evaluates budget-matched configurations with repeated trials, then tests an adaptive policy that prunes weaker partial runs and reallocates their compute.

    Source: §1 · p. 3, §5.2 · p. 9

  • Results

    No fixed harness is universally superior across evaluated model–problem pairs, and online adaptation improves average final performance over single-harness commitment and an unpruned ensemble.

    Source: §1 · p. 3, §7 · p. 13

  • Takeaways & Limitations

    Harness choice is better treated as an online, model- and problem-dependent decision guided by partial-run performance than as a fixed choice before search.

    Source: §1 · p. 3

    Pair-level p-values are unadjusted and interpreted as nominal, exploratory evidence.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Autonomous discovery systems such as OpenEvolve and TTT-Discover are often used as general-purpose harnesses. However, in practice these are composite systems combining several design choices about archives, parent selection, exploration, and budget allocation into a single recipe. Because discovery runs are expensive and inherently stochastic, existing harnesses are often compared using too few independent trials to distinguish key methodological improvements from run-to-run variance. We systematically decompose OpenEvolve-style evolutionary search and the TTT-Discover search harness into its constituent components and systematically evaluate 30 budget-matched harnesses across 12 model-problem pairs using more than 3.1 million LLM rollouts and repeated-trial statistical analysis. Our results show that discovery harnesses have a generalization problem: No fixed harness is reliably superior across the evaluated model-problem pairs, and variants of OpenEvolve generally underperform simpler alternatives. Thus, harness choice is better viewed as a hyperparameter rather than as a universal recipe, and should be tailored to the specific problem and underlying model. We also find that early discovery progress predicts final performance, and use this property to present a budget-matched adaptive-allocation experiment that starts multiple harnesses, prunes weak partial runs, and reallocates compute to stronger survivors, outperforming both commitment to a randomly sampled fixed harness and a non-adaptive harness ensemble. Together, these results motivate shifting from fixed harness selection to online adaptation guided by early performance. We release all run pools including baseline null distributions for every model-problem pair as reusable statistical infrastructure against for future harness proposals.

1 Introduction

Discovery harnesses combine design choices whose benefits may not transfer across models and problems, while stochastic runs make component effects difficult to distinguish with few trials. The paper decomposes and evaluates harnesses, finds no universally superior fixed choice, and tests using early progress to allocate compute online.

Online adaptation

  • Early discovery progress correlates with final performance, supporting an adaptive policy that prunes weaker partial runs and reallocates compute to stronger survivors.The authors report that online harness choice alleviates the generalization problem.Source: §1 · p. 3

Evaluation

  • The study evaluates 30 budget-matched harnesses across 12 model–problem pairs using more than 3.1 million rollouts and repeated-trial statistical tests.The evaluation decomposes OpenEvolve-style evolutionary search and TTT-Discover-style PUCT search from a Sequential Best-of-N baseline.Source: §1 · pp. 2–3

Main finding

  • No fixed harness is universally superior across the evaluated model–problem pairs, and OpenEvolve-style harnesses generally rank below simpler lightweight-exploration and PUCT-style variants.The authors frame harness choice as a model- and problem-dependent hyperparameter rather than a transferable recipe.Source: §1 · pp. 2–3, Figure 1 · p. 2

Reusable infrastructure

  • The released run pools contain over 3.1 million rollout records, per-step evaluator scores, and baseline null distributions for future harness comparisons.Source: §1 · p. 3

2 A Unified View of Harnesses for Discovery Systems

The paper treats discovery harnesses as rules for selecting previously evaluated solutions to expand, and builds two search families by changing a greedy Sequential Best-of-N baseline one design choice at a time.

Unified formulation

  • The shared formulation tracks evaluated programs and scores, selects parents from an archive, generates candidate children, and holds total rollout budget constant across harnesses.The top-K archive contains the K highest-scoring programs in the full history.Source: §2 · p. 3

The Baseline: Greedy Sequential Best-of-N

  • Sequential Best-of-N repeatedly mutates the best program found so far, providing a simple baseline for isolating the contributions of more complex harnesses.The harness evaluates candidate modifications and appends them to the search history.Source: §2 · pp. 3–4

From Sequential BoN to OpenEvolve

  • OpenEvolve-style search progressively adds a top-K archive, epsilon-greedy exploration, deeper budget allocation, inspiration sampling, MAP-Elites, and multiple islands.These changes relax greedy parent selection and alter how programs and prior solutions are used during search.Source: §2 · p. 4

From Sequential BoN to TTT-Discover

  • TTT-Discover-style search replaces greedy selection with subtree value estimates, UCT exploration, prior-weighted PUCT, and expansion of multiple parents per iteration.The paper studies the search harness rather than the test-time-training component.Source: §2 · pp. 3–4

3 Experimental Setup and Statistical Evaluation

The evaluation matches rollout budgets within each model–problem pair and uses repeated Sequential BoN runs as a reference. It combines pair-level bootstrap tests with a cross-pair majority-win analysis to assess outcomes and transfer.

3.1 Models, Problems and Budgets

  • The evaluation covers four models from 3B to 120B parameters and three mathematical discovery tasks, with five independent runs per harness and budgets matched within each pair.Per-run budgets are 1,600 rollouts for the Qwen models, 320 for GPT-OSS-20B, and 160 for GPT-OSS-120B.Source: §3.1 · p. 4

3.2 Best-of-Five Significance Test Against Sequential BoN

  • Pair-level tests compare a candidate harness’s best-of-five score with bootstrapped best-of-five maxima from the repeated Sequential BoN reference pool.The bootstrap uses 100,000 trials; non-rejection indicates insufficient evidence of improvement in the five-sample experiment, not inability to improve in future runs.Source: §3.2 · p. 5

3.3 Evaluating Harness Transfer Across Model–Problem Pairs

  • Larger archives alone yield no significant gains, while exploration helps some model–problem pairs but hurts others; added OpenEvolve- and TTT-Discover-style machinery does not consistently help.Figures 2–4 show archive and exploration ablations and progressive additions to both search families.Source: Figure 2 · p. 6, Figure 3 · p. 7, Figure 4 · p. 7
  • The cross-pair statistic estimates the probability that a harness beats matched Sequential BoN outcomes on at least seven of 12 pairs, with permutation tests and Holm correction for harness comparisons.The study uses 100,000 Monte Carlo trials for the bootstrap and permutation procedures.Source: §3.3 · pp. 5–6

4 Fixed Harnesses Do Not Generalize Uniformly Across Model–Problem Pairs

Component ablations show that harness effects vary across model–problem pairs, and cross-pair comparisons do not support replacing OpenEvolve with another universal recipe.

4.1 Ablating Harness Components

  • Larger Top-K archives produced no pair-level significant gain over Sequential BoN, while full-history exploration helped selected pairs but not uniformly.The reported gains include Qwen2.5-3B on Circle Packing and both GPT-OSS models on Heilbronn Triangle.Source: §4.1 · p. 7
  • Adding OpenEvolve components did not yield monotonic gains; complete configurations were often worse than simpler variants and reduced performance on some tasks.The full recipes produced nominally significant gains only for GPT-OSS-20B on Heilbronn Triangle.Source: §4.1 · p. 8
  • UCT/PUCT parent selection produced the largest observed pair-level gains, but later increases in search depth and multi-parent expansion were inconsistent and sometimes reversed them.Source: §4.1 · p. 8

4.2 Choosing One Fixed Harness

  • No tested fixed harness is significantly better than a simple baseline across the suite, and the strongest observed choice varies by setting.Lightweight exploration and simpler tree-search methods have stronger observed cross-pair performance than complete OpenEvolve-style recipes, but no universal replacement is supported.Source: §4.2 · p. 8

5 Early Feedback Enables Online Harness Allocation

Partial-run scores become informative about final performance as discovery proceeds; under a matched budget, adaptive pruning reallocates compute toward stronger runs and improves average outcomes.

5.1 Early Discovery Progress Predicts Final Performance

  • At the 50% checkpoint, correlations with final performance exceeded 0.70 for 11 of 12 pairs and were 0.651 for GPT-OSS-120B on the Second Autocorrelation Inequality.At 10%, correlations ranged from 0.000 to 0.474.Source: §5.1 · p. 9

5.2 Budget-Matched Online Harness Allocation

  • The adaptive harness ensemble ranks partial runs by their best evaluator score, prunes weaker configurations, and spends the remaining budget completing survivors.Its policies use five full-run equivalents, trading broader partial evaluation for completing fewer runs.Source: §5.2 · pp. 9–10
  • The budget-matched comparison includes five Sequential BoN runs, five repeats of one sampled harness, and five distinct fully run configurations without intermediate feedback.The unpruned portfolio controls for harness diversity when assessing the value of adaptive reallocation.Source: §5.2 · p. 10
  • 85.75% for the three-stage 12→5→2→1 schedule exceeded 84.54% for the unpruned portfolio, 82.49% for single-harness commitment, and 84.35% for Sequential BoN.It also outperformed the unpruned portfolio on 11 of 12 model–problem pairs.Source: Table 1 · p. 11, §5.2 · p. 11

6 Related Work

Prior work studies discovery systems, evaluation rigor, search and resource allocation, and harness adaptation; this paper focuses on decomposing harness choices and testing their transfer across settings.

LLM-guided discovery systems and evolutionary coding agents

  • Discovery systems use iterative LLM proposal, evaluation, and improvement, while follow-up systems add ingredients such as novelty filtering, adaptive optimization, and island models.Examples include FunSearch, AlphaEvolve (Novikov et al., 2025), and OpenEvolve.Source: §6 · pp. 11–12

Evaluation rigor and simple baselines

  • Evaluation studies motivate repeated trials and simple baselines; this paper instead ablates harness ingredients under matched budgets and multirun empirical null distributions.The authors report that ingredients can significantly help some model–problem pairs while degrading others.Source: §6 · p. 12

Search algorithms and budgeted allocation

  • The work connects harness decomposition to UCT/PUCT, archive-based diversity, and fixed-budget allocators, while distinguishing allocation across harnesses from methods operating within one run.It proposes no new bandit or early-stopping algorithm.Source: §6 · p. 12

Harness adaptation and meta-harness search

  • Harness-search work includes modular agent design and meta-harness methods, but this study offers an allocation framing rather than a learned or meta-evolved harness.Its framing treats harness selection as online resource allocation when fixed choices do not transfer and early feedback is informative.Source: §6 · pp. 12–13

7 Conclusion

The conclusion frames harness choice as setting-dependent and favors online adaptation when early feedback can guide resource allocation; the appendix provides allocation procedures.

7 Conclusion

  • Fixed harnesses do not reliably generalize across models and problems, while online adaptation improves average final performance over single-harness commitment and an unpruned portfolio.The paper releases rollout-level data and statistical artifacts for future research.Source: §7 · p. 13

A Online Budget Allocation Algorithms

  • The adaptive procedure advances active configurations to checkpoints, retains top-scoring runs, then completes the survivors.The algorithm specifies a harness pool, checkpoints, and survivor counts, and returns the best final score.Source: A Online Budget Allocation Algorithms · p. 18

B.1 Search-Harness Ablation Details

The appendix expands the main-text harness comparisons into task-level ablations and implementation details. It traces how archive, exploration, budget-allocation, context, and parent-scoring choices build from Sequential BoN toward OpenEvolve-style and PUCT search.

B.2 From Sequential BoN to OpenEvolve

  • OpenEvolve-style search progresses from Sequential BoN through archive sampling, exploration, and depth-oriented allocation.The appendix presents these ablations as a progression from the Sequential BoN baseline.Source: §B.1 · p. 18, §B.2 · pp. 18–19, Figure 7 · p. 19

B.1 Search-Harness Ablation Details

  • The appendix ablations diagnose individual mechanisms, while the cross-pair battle analysis tests whether their gains persist across settings.Table 2 links compressed main-text comparisons to complete task-level evidence.Source: §B.1 · p. 18, Table 2 · p. 19

B.2.1 Adding a TopK Elite Archive

  • Sampling from the top-K elite archive lets search revisit several high-scoring programs instead of committing all mutations to one lineage.K = 1 recovers Sequential BoN; K > 1 preserves some lineage diversity while focusing on high-scoring programs.Source: §B.2.1 · p. 19

B.2.2 Adding Epsilon-Greedy Exploration

  • Epsilon-greedy selection mixes elite-archive sampling with occasional sampling from the full history, allowing lineage changes.With K = 1 and ϵ = 0, the rule reduces to Sequential BoN.Source: §B.2.2 · p. 20

B.2.3 Trading Breadth for Depth

  • At fixed rollout budget, larger batches explore alternatives around one parent, while more sequential updates enable longer refinement chains.Decreasing N requires increasing T; N = 1 is the most depth-oriented allocation and most closely matches OpenEvolve.Source: §B.2.3 · p. 20

B.2.4 OpenEvolve-Specific Context and Parallelism

  • MAP-Elites inspiration programs augment the LLM context, while multi-island evolution maintains separate histories and periodically migrates high-scoring programs.These components modify context and parallelization rather than the basic history update rule.Source: §B.2.4 · pp. 20–21

B.3 From Sequential BoN to TTT-Discover

  • UCT selects parents using the best score in a program's descendant subtree plus a visitation-count exploration bonus.PUCT adds a rank-based prior to its exploration bonus, with higher-ranked programs receiving larger prior mass.Source: §B.3 · p. 21

C.1 Task Definitions

  • The task suite includes two geometric optimization problems and a non-geometric autocorrelation objective.The autocorrelation task searches for a non-negative vector whose autoconvolution balances L1, L2, and L∞ norms.Source: §C.1 · pp. 21–22, Figure 9 · p. 22

C.2 Qualitative Strategies on Circle Packing

Across these model–problem pairs, strategy and search-harness effects vary: larger models use different optimization strategies, and no archive or breadth–depth setting is consistently best. Deeper OpenEvolve-style refinement is one tradeoff point, not a uniformly superior endpoint.

C.2 Qualitative Strategies on Circle Packing

  • GPT-OSS-20B advances circle packing by choosing centers, solving radii globally, and searching centers, while Qwen2.5-3B and Qwen3-4B mainly refine geometric templates.GPT-OSS-20B’s later gains include joint center-and-radius optimization; the smaller models generally stop short of that abstraction.Source: §C.2 · pp. 22–23

C.3 Qualitative Strategies on Heilbronn Triangle

  • On Heilbronn triangle, Qwen2.5-3B relies on local perturbations, Qwen3-4B uses structured layouts, and GPT-OSS-20B shifts from objective optimization to annealing and restarts.The reported strategy transitions accompany distinct score thresholds, including roughly 0.5 for Qwen3-4B’s lattice constructions and around 0.865–0.925 for GPT-OSS-20B’s annealing-style updates.Source: §C.3 · p. 23

C.4 Qualitative Strategies on the Second Autocorrelation Inequality

  • On the second autocorrelation inequality, Qwen2.5-3B scales the baseline optimizer, Qwen3-4B structures the function, and GPT-OSS-20B varies optimizer conditioning between low- and high-score regimes.For GPT-OSS-20B, all nine final programs with learning rate at least 0.1 enter the high mode, while eleven of sixteen with learning rate at most 0.02 remain in the low mode.Source: §C.4 · p. 24

D.3 Sequential BoN Baseline

  • Sequential BoN baseline distributions differ substantially by model and task, so harness comparisons must be read against both each model’s baseline score and run-to-run spread.GPT-OSS-20B is near the top with low variance on circle packing, while the Qwen models are lower or more variable; the models are closer on the second autocorrelation task.Source: §D.3 · pp. 27–28, Figure 12 · p. 28

D.4 Adding an Elite Archive; D.5 Adding Exploration

  • Larger elite archives and general-history exploration show model–problem-dependent effects, not monotonic gains; exploration can spend budget on less relevant programs.Archive-size changes shift some Qwen3-4B circle-packing distributions, but increasing K does not consistently improve results; exploration likewise has no dominant rate.Source: §D.4 · p. 28, §D.5 · pp. 29–30, Figure 13 · p. 29

D.6 Full Epsilon-Greedy Grid

  • The full epsilon-greedy grid shows that archive size and exploration interact, with no globally dominant (K, ϵ) setting and mostly small, non-monotonic shifts on the second autocorrelation task.The available grid is incomplete for GPT-OSS-20B because of compute constraints.Source: Figure 11 · p. 27, §D.6 · p. 30

D.7 Progression toward OpenEvolve

  • Deeper OpenEvolve-style refinement is competitive but does not uniformly dominate simpler variants; its one- and four-island endpoints reduce both Qwen models’ performance on the second autocorrelation task.The progression figures present model-specific breadth–depth ladders, so the comparisons are within-model rather than a perfectly matched grid across models.Source: Figure 10 · pp. 25–26, §D.7 · p. 31

D.8 From Sequential BoN to UCT and PUCT; D.8.2 Breadth–Depth Ablations

  • UCT/PUCT breadth–depth effects depend on proposal quality: deeper refinement does not reliably help weaker models, and the preferred exploration strength varies across model–problem pairs.Non-baseline configurations have five runs each, so the authors emphasize distributional trends and consistency rather than a definitive optimum for an individual setting.Source: §D.8.2 · pp. 31–32, §D.8 · p. 31, Figure 15 · p. 32

D.8.3 PUCT: P ablations

Increasing the number of parent programs does not uniformly improve performance across the three tasks. Its benefit depends on whether the selected roots seed useful optimization trajectories.

D.8.3 PUCT: P ablations

  • Across the three tasks, increasing P does not uniformly improve performance.Figure 16 varies child batch size and adjusts update steps to keep the total rollout budget fixed; the P ablation compares parallel-parent search structure, not identical batch sizes across model families.Source: §D.8.3 · p. 32, Figure 16 · p. 33
  • On the second autocorrelation inequality, changing the number of root programs produces only modest, model-dependent shifts.The benefit depends on whether each root can seed a useful optimization trajectory.Source: §D.8.3 · pp. 32–33
  • PUCT and UCT are not replacements for model capability; their structured search allocates effort most usefully when the model can already generate promising candidates.GPT-OSS-20B remains robust under structured search, while weaker models show greater sensitivity and less reliable improvement.Source: §D.8.3 · p. 33

E Additional Config Vector Battle Leaderboards

The progression plot centers majority-win probability against Sequential BoN and rescales it, while the complete leaderboards show lightweight tree-search and archive variants near the top and full OpenEvolve-style configurations near the bottom.

Progression plot

  • The progression score subtracts the baseline's 0.5 self-win probability and rescales the centered value by a factor of two.This maps baseline performance to 0, always beating the baseline to +1, and never beating it to −1.Source: E Additional Config Vector Battle… · pp. 33–34

Parallel-parent ablations

  • Figure 17 groups UCT/PUCT parallel-parent ablations by selected parent count P because model budgets differ.The grouping compares parallel-parent structure rather than identical child batch sizes.Source: Figure 17 · p. 34

Progression plot

  • Simpler variants, especially UCT/PUCT and small archives, remain above baseline, whereas adding the full OpenEvolve machinery drives performance substantially below baseline.The progression plot orders configurations from Sequential BoN through the OpenEvolve and TTT-Discover design choices.Source: E Additional Config Vector Battle… · pp. 33–34

Full cross-pair leaderboards

F Complete Per-Configuration Results

The appendix provides per-configuration results and matched-baseline bootstrap comparisons across harness families, with run budgets and pair/model abbreviations specified.

Per-configuration results

  • Tables 3–12 report the Sequential BoN baseline pool and five-run configuration statistics alongside matched-baseline one-sided bootstrap p-values.Each null distribution uses 100,000 bootstrap samples of five runs; < 10−5 denotes outperforming all bootstrap samples for that statistic.Source: F Complete Per-Configuration Results · p. 38, Table 3 · p. 38

Experimental settings

  • Unless noted otherwise, Qwen configurations use N=16 and T=100, while GPT-OSS-20B configurations use N=8 and T=40.Pair codes are CP for circle packing, HT for Heilbronn triangle, and SAI for second autocorrelation inequality; model codes are Q2.5, Q3, and GPT.Source: F Complete Per-Configuration Results · p. 38

OpenEvolve-style configurations

  • The configuration set progresses from top-K archives and full-history exploration to epsilon-greedy variants and basic or multi-island OpenEvolve.The tables distinguish top-K-only, full-history, epsilon-greedy, single-child, one-island, and multiple-island configurations.Source: Table 4 · p. 38, Table 5 · p. 39, Table 6 · p. 40

UCT and PUCT configurations

  • UCT results include breadth–depth and parent-count ablations, while PUCT results cover single-parent and multiple-parent settings.The multiple-parent PUCT endpoint corresponds to TTT-Discover.Source: Table 10 · p. 42, Table 11 · p. 43, Table 12 · p. 44

Per-configuration results

  • 0.9316 mean, 0.9490 median, and 0.9730 maximum are reported for HT/GPT20 P=1, C=1, N=4; pmean=0.00004 and pmedian and pmax are < 10−5.The table columns identify the three summary statistics and their matched-baseline p-values.Source: F Complete Per-Configuration Results · pp. 44–45

G Full Budget-Matched Allocation Results

The budget-matched allocation results compare policies by simulated vector battles against matched Sequential BoN baselines, with results summarized as mean scores across twelve pairs.

Statistical comparison

  • The comparison simulates policy and matched Sequential BoN outcomes pair by pair, then decides each vector battle by majority vote across available task–model pairs.Pruning and ensembling scores are obtained by replaying policies against the run pool; the reference samples five baseline runs with replacement and takes their maximum.Source: G Full Budget-Matched Allocation Results · p. 46
  • Win, loss, and tie rates are estimated across resampling iterations, and decisive vector battles are tested with a one-sided binomial tail test against equal win probability.The test discards vector-level ties; the baseline-vs-baseline reference provides a check for nearly symmetric rates and a non-significant p-value.Source: G Full Budget-Matched Allocation Results · p. 46

Budget-matched evaluation

  • Table 13 reports mean final scores over 100,000 empirical resampling trials under a five-full-run-equivalent budget, with averages unweighted across twelve model–problem pairs.Scores are percentages; higher is better, and SE is reported.Source: G Full Budget-Matched Allocation Results · p. 46, Table 13 · p. 47
  • Tables 14–16 provide corresponding budget-matched evaluations for PUCT, UCT, and TopK archive configurations.Their captions specify the same 100,000-trial, five-full-run-equivalent evaluation and unweighted twelve-pair average.Source: Table 14 · p. 48, Table 15 · p. 49, Table 16 · p. 50

Related papers

Loading 2607.18235v1…