PaperlayerView exact revision on arXiv

Source-linked AI summary

Building the Harness Automatically: Self-Play in Code Distills a Text Harness for Black-Box Optimization

Yi Wu, Zheng Ren, Zhiyu Hu, Haochen Wang, Daryl Chang, Li Wei, Ting Wang, Zhen Li, Pooja Gupta, Nitin Jindal, Lukasz Heldt

arXiv:2609.09468v1cs.LG

TL;DR

The paper addresses whether executable practice can produce dependable numerical search behavior for low-budget black-box optimization, where unaided language models are unreliable. It repeatedly evaluates optimizer programs on development objectives, distills a passing program and record once into frozen text, and tests that artifact independently. Harness A reduces Gemini Flash regret by 48% in an independent N=30 study, while the framework transfers across executors, landscapes, and an independently replicated harness.

  • Problem

    Low-budget black-box optimization requires expensive, gradient-free search under small query budgets, while unaided language models remain unreliable numerical optimizers.

  • Method

    The method repeatedly practices with optimizer programs, gates the retained result on fresh development data, distills it once into text, and freezes the artifact and execution configuration before evaluation.

  • Results

    Harness A reduces Gemini Flash regret from 69.5 to 36.0, a 48% reduction in an independent N=30 study (p < .001), and lowers mean regret across all three held-out BBOB landscapes.

  • Takeaways & Limitations

    Executable practice discovers a testable search policy, while one-shot distillation makes it portable across executors, model providers, and objective geometries.

  • Takeaways & Limitations

    Most public experiments use one budget, one dimension, and N = 10 paired trials; the central effect therefore relies on the independent N = 30 study for power.

Abstract

from arXiv · show

Can an agent learn a numerical search strategy through executable practice and then transfer that strategy as text? We study low-budget black-box optimization, where unaided language models remain well below strong classical optimizers. During development, an agent repeatedly writes and evaluates optimizer programs. It then distills the resulting program and practice record once into a 197-word primary Harness A, which is frozen before evaluation. Harness A reduces Gemini Flash regret by 48\% in an independent $N=30$ study ($p<.001$), enters the GP-BO performance range on the practice family, and lowers mean regret on all three held-out BBOB landscapes. The same text improves every tested Gemini executor and transfers to Claude Sonnet, reducing regret by 43\% and 49\% ($p\leq.005$). An independent end-to-end replication produces Harness B, a different program and text at the same performance tier. The same framework also attains the lowest regret on a sealed YouTube reward-tuning production benchmark. Executable practice is thus a viable way to discover a search policy, and language a portable medium for deploying it.

1. Introduction

The paper asks how to learn dependable search behavior without letting evaluation outcomes revise the deployed artifact. It proposes executable practice followed by one-time distillation, gating, and freezing before evaluation.

  • Low-budget black-box optimization requires expensive, gradient-free objective optimization under small query budgets, while unaided LLMs remain unreliable numerical optimizers.
  • The central question is how to learn search behavior that remains dependable after development without leaking evaluation information.
  • The framework distills practice on external development objectives once, gates the result, and freezes it before public or internal evaluation.
  • The contributions include a practice–gate–distill–freeze framework and an evaluation that seals before testing, reproduces an independently accepted run, and exposes a rejected run.

2. Related Work

Prior work uses language models to propose solutions, combine language with established optimization procedures, or evolve optimizer code and prompts. This paper instead distills executable development practice into a portable, frozen text harness.

  • LLM optimization methods use textual trajectories or couple language models with Bayesian-optimization components and exploration–exploitation control.
  • Code-evolution systems write or evolve optimizer programs, and this paper adapts that paradigm by distilling executable practice back into portable language.
  • Feedback-based systems use critique, search, reflection, prompt evolution, or end-to-end harness search, whereas this method distills once and freezes before evaluation.
  • One-shot sealing produces a portable frozen text harness that operates zero-shot without further revision, prioritizing auditability and test-time stability.

3. Method

The method learns optimizer behavior through executable practice on development objectives, validates the retained program, and distills the program and record into a frozen text artifact. Evaluation changes only the observed history, not the artifact or execution settings.

  • The pipeline distinguishes the retained program, distilled text, and deployed harness, learning on external development objectives before freezing the result.
  • The optimizer observes evaluated points and objective values, then proposes the next point using the remaining budget; performance is reported as final simple regret.
  • At each practice round, a challenger program and incumbent receive equal budgets on the same fresh development batch, enabling controlled selection.
  • A challenger replaces the incumbent only when it is valid and achieves lower mean regret; the record preserves code, trajectories, scores, failures, and decisions.
  • 3.3. Gate, Distill, and Freeze: After practice, a fixed validation gate evaluates the retained program on fresh development seeds; failed runs remain audit records and are not promoted.
  • 3.3. Gate, Distill, and Freeze: The execution interface, parser, bounds, and inference settings are fixed, so neither the distilled text nor configuration can respond to test outcomes.
  • Because practice is stochastic, reproducibility is defined behaviorally: independently accepted runs should recover the same performance tier on fresh objectives.

4. Experimental Setup

The experiments use low-budget optimization tasks with a primary final-simple-regret metric and compare the harness against classical, retained-program, and LLM controls. Development and evaluation are separated, with artifacts and configurations sealed before testing.

  • The public study minimizes unknown objectives on [−5, 5]^8 with a budget of 20 sequential evaluations, using quadratics for practice and three transformed BBOB landscapes for held-out evaluation.
  • Each primary-study cell contains N = 10 paired instances, with means, normal-approximation 95% confidence intervals, and two-sided exact paired Wilcoxon tests.
  • Comparators include random search, CMA-ES, GP-BO variants, the retained practice program, unaided models, handwritten hints, and inference-time-compute controls.
  • For each executor, Base and Harness share the history formatter, parser, bounds, and inference configuration; only the distilled text differs.
  • Practice splits, budgets, synthesis instructions, and text-length limits are fixed in advance, and the manifest records the accepted artifact and execution configuration before held-out or production evaluation.

5. Results

Harness A improves low-budget black-box optimization across executors, held-out landscapes, and a production benchmark, while an independent Harness B reaches the same performance tier. The strongest evidence supports transfer into the GP-BO range rather than universal BO dominance.

  • Practice-family results: 48% lower Flash regret in the independent N = 30 study moves Harness A into the GP-BO performance range.Regret decreases from 69.5 to 36.0 with p < .001; the paired mean improvement is 33.5 and 27/30 instances improve.
  • Transfer across executors: Mean regret decreases in all six executor–task cells under the unchanged frozen text, with five paired effects satisfying p ≤ .005.Transfer includes Claude Sonnet; the harness also composes with additional reasoning resources on Flash but remains neutral on equipped Flash-Lite.
  • Transfer across landscapes: Harness A lowers Base-LLM mean regret on all three held-out BBOB landscapes.Only Gallagher survives Holm correction (pHolm = .018), while BO-baseline comparisons do not survive family-wise correction.
  • Independent replication: Both independently accepted harnesses improve over their Base LLM on every one of four fresh landscapes.Harness B uses a different program and shares no sentence with Harness A; both also achieve lower mean regret than plain GP-BO throughout, with all four comparisons surviving Holm correction.
  • Production benchmark: The Replication harness has the lowest point estimate at every budget checkpoint on the sealed YouTube reward-tuning benchmark.At 20 evaluations, its point estimate is 0.0495 versus 0.0894 for Google Vizier and 0.1410 for standard GP-BO.

6. Discussion

The discussion argues that the frozen text artifact transfers a policy for allocating evaluations across executors and objective geometries, while remaining dependent on the executor interpreting it. The main scope limits concern executor interaction, limited benchmark coverage, and the exploratory status of many comparisons.

  • What transfers: The evidence supports a text-only policy for spending evaluations that changes behavior across executors and transfers across model providers and objective geometries.The practice record may contribute beyond the retained procedure, but the strongest comparison lacks released per-instance trajectories and should be replicated.
  • Executor dependence: A text harness is not executor-free: the same instruction can help one model, harm another, or compose differently with inference-time tools.The study controls this interaction by comparing Base and Harness under a shared interface and freezing the execution configuration.
  • Limitations: The central effect relies on an independent N = 30 study because most public experiments use B = 20, D = 8, and N = 10 paired trials.The classical comparison omits HEBO and TuRBO, the cross-provider test covers one additional model family, and the production case uses one historical dataset.
  • Limitations: Secondary comparisons are exploratory unless correction is stated, and one-shot distillation prioritizes auditability over adaptation.Multi-champion redistillation may be stronger but requires a new separation between development and evaluation.

7. Conclusion

The conclusion presents executable practice as a way to discover a testable search policy, distill it into portable language, and evaluate it under frozen conditions. The artifact’s benefit is not reducible to generic optimization wording, and practice information may matter beyond the retained procedure.

  • Conclusion: Executable practice discovers a testable search policy, while one-shot distillation makes it portable and freezing keeps evaluation interpretable.The reported outcome persists across models, objectives, and independent runs at the level of competent search under scarce evaluations.
  • Artifact: The deployed Harness A is a 197-word text distilled once from the champion program and practice log, then frozen and used verbatim.The harness text itself contains guidance on spending evaluations and probing independent directions.
  • Evaluation: Table 4 reports primary-study quadratic regret at 20 evaluations as mean ± 95% CI over N = 10, with lower regret better.The table is a primary-study comparison of model and artifact configurations.
  • Portability: Harness A is copied without model-specific tuning, with regret reduction reported from Lite through Pro and on Claude Sonnet.These cells test portability rather than provider ranking.

C.3. Test-Time Compute

The test-time-compute analysis separates the value of the text harness from raw inference resources and shows that resource effects depend on the executor. Artifact ablations likewise indicate executor-dependent use of content beyond the procedural skeleton.

  • Compute interaction: The text-only harness is stronger than every tested Gemini Flash Base configuration, while adding thinking or sandbox access is not monotone.The harness can compose positively with those resources, so its value is separate from raw inference-time compute.
  • Compute interaction: Flash-Lite with sandbox and unbounded thinking is neutralized by the harness: 29.0±15.2 unprompted versus 31.5±11.6 with the harness, p = .70.Both arms interleave probing with exploitation, while the same resources compose positively with the harness on Flash.
  • Artifact ablations: The code-only Gemini Pro ablation is worse on the practice family: 32.7 versus 21.4, paired p = .013.On textbook Rastrigin it is directionally worse, 65.2 versus 54.3, a +10.9 difference with p = .17.
  • Artifact ablations: Ablation effects are executor-dependent: transformations degrade GPT-5 progressively but remain within noise of the full text on both text-only Gemini executors.Claims about a single essential component should therefore be indexed by the executor running the text.
  • Artifact ablations: Table 5 reports quadratic artifact ablations as mean ± 95% CI with lower regret better, separating GPT-5 agent-runtime cells from text-only Gemini cells.The Pro code-only summary is not independently regenerable because its trajectories are not released.
  • Exact-solve audit: Across three text-only held-out cells, the exact-solve rate is 0/30, and the sandbox-equipped textbook-Rastrigin arm also solves 0/10.The study does not use an oracle score to support its identifiability claim.

Appendix E. Consolidated Primary-Study Results

The primary-study appendix documents the evaluation setup, trajectory signatures, and recorded artifacts for the consolidated results. Harness A changes proposal behavior toward shorter, more local steps, while the appendix preserves execution traces and study configuration details.

  • Evaluation setup: Primary-study rows report regret at 20 evaluations as mean ± 95% CI with N = 10.Arm names encode model, thinking, code-execution, and role configurations; ablations include codeonly, min, half, and v2.
  • Held-out landscapes: The consolidated primary-study results include text-only LLM arms evaluated on the same paired instances across three held-out BBOB landscapes.Figure 3 reports best-so-far regret for these landscape comparisons.
  • Trajectory signatures: Harness A changes the quadratic query sequence toward shorter, less variable, more local proposals and more improvement events.Proposals within radius 2 of the incumbent rise from 69% to 79%, and late proposals move closer to the incumbent.
  • Execution traces: Per-turn traces record thought summaries, executed code, execution outputs, and committed answers.The committed JSON answer is treated as the ground truth of the proposed point, while code and result blocks are executed artifacts.

Appendix G. Independent 30-Instance Reproduction

The independent 30-instance reproduction tests the frozen text on fresh paired seeds, across model families and landscape constructions. It reproduces the central improvement, while also documenting replication-specific scope and diagnostic limitations.

  • Independent synthetic comparison: 33.5 mean improvement with 95% interval [19.0, 49.7] occurs across 30 paired Base–Harness differences, with 27 of 30 instances improving.On the quadratic family, Harness A moves regret from 69.5 to 36.0 (p < .001), with no detected difference from GP-BO (p = .64).
  • Cross-model transfer: The same frozen text improves every recorded model family on both evaluation families.Flash-Lite and Sonnet improve significantly on both families at p ≤.005.
  • Scope of comparisons: Additional BBOB cells use code execution and therefore do not support a words-only claim.On Bent Cigar, Harness A+code obtains 18.8M ± 7.3M regret and the retained program obtains 11.8M ± 3.7M.
  • Replication: Both accepted harnesses improve over their Base LLM on every landscape in the independent replication.Harness B is a distinct 205-word text produced by an independently generated end-to-end run.
  • Interpretive limits: With only two accepted runs, the gate–outcome association is diagnostic rather than prospective validation of the threshold.Harness A has three landscape comparisons surviving Holm correction, while its quadratic comparison does not (pHolm = .28).
  • Practice diagnostics: The replication’s reconstructed artifacts trail the reference harness on both families, and neither achieves an exact solve on the diagnostic.The reference harness+code remains the strongest all-round arm.

J.1. Sealed Historical Benchmark

The sealed historical benchmark evaluates candidate proposals under a fixed online reward protocol using stratified bootstrap replicates. Its reporting emphasizes replayability and auditability, while the replicates remain resamples rather than independent production trials.

  • Resampling design: Thirty paired stratified-bootstrap replicates are drawn from one historical dataset, not independent production trials.Each replicate resamples within pre-specified production strata and evaluates every method on the same resampled dataset.
  • Protocol: The benchmark uses a sealed candidate list, a projection rule, a seven-dimensional reward vector, and 20 evaluations per method.The confidential manifest stores reward factors, watch time, content rating, traffic information, timestamps, and a deployed-baseline indicator.
  • Candidate execution: A raw proposal is projected onto the sealed candidate list after coordinate normalization by legal ranges, unless a direct evaluator permits bounded execution without projection.The protocol logs projection distance and adds only the selected candidate, reward, and score to the next prompt.
  • Outcome definition: For each seed, feasible best-so-far utility is defined as the maximum observed watch-time value whose content rating meets threshold τ.When no proposal is feasible, the deployed feasible vector supplies the utility; reported metrics are lift, violation fraction, and feasible AUC.
  • Reporting boundary: The online protocol is documented for replay and audit, but released materials contain no shareable per-run online outcomes or numeric online table.Public reporting may rename factors, normalize to the deployed vector, and round counts if the transformation is stated.
Loading 2609.09468v1…