PaperlayerView exact revision on arXiv

Source-linked AI summary

Operator Packages, Proposer Strength, and Construction-Family Plateaus in Office-Scale Verified Search

Roberto I. Ono Filho

arXiv:2609.29636v1cs.CLcs.AIcs.LGcs.SE

TL;DR

The paper addresses limited controlled evidence about proposer-side components in verified search and how much outcomes depend on the proposer. It tests three operator packages in a fixed office-scale loop and compares proposer strength. The composition has a reported +0.196 gain in the nominal two-stage analysis, but the factorial finds no positive memory-by-repulsion interaction and the family-hint test does not cross the final gap.

  • Problem

    Controlled ablations of proposer-side components are rare, and published systems change the proposer and scaffold together, leaving their separate contributions unmeasured.

    Source: §1 · p. 1

  • Method

    The study tests schematic memory, a named obstacle, and behavioural repulsion in a fixed verified-search loop, including a factorial and a proposer comparison.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    +0.196 is the composition’s reported gain in the nominal two-stage analysis; memory+repulsion is the only configuration with zero collapses and is within 0.025 of the full composition’s mean.

    Source: §1 · p. 2, §10 · p. 10

  • Takeaways & Limitations

    In this office-scale study, memory+repulsion is the parsimonious recipe supported by the data, while the composition has the highest mean.

    Source: §10 · p. 10

    The nine problems cover a small universe of construction families, and no per-term effect survives correction at n=9.

    Source: §9 · p. 9

Abstract

from arXiv · show

Verified search, in which a language model proposes programs, a hard evaluator scores them, and selection keeps the best, has recently moved mathematical records; controlled ablations of the proposer-side components remain rare. We instrument a minimal FunSearch-style loop at office scale (a 30B local model on a laptop, 120-600 verified samples per run) with three operator packages: a schematic notebook the model writes and carries instead of verbatim elites, a named obstacle, and behavioural repulsion from constructions already found. On nine construction problems from a public repository, the complete 2^3 factorial with two replicates favours the primary contrast in a nominal two-stage analysis: the composition closes more of the seed-to-record gap (+0.196; nominal pooled p=0.023, stage-combination p~0.08; median per-problem effect +0.045). Repulsion raises construction-hash diversity everywhere (p=0.0039; partly a manipulation check). The factorial finds no positive memory-by-repulsion interaction (bounded to about +/-0.04); the gain decomposes additively, and memory+repulsion is the only arm that never collapses (0 of 18 runs), within 0.025 of the full composition. A frontier proposer under the identical loop reaches in tens of samples what the local model does not in hundreds; in single scoping runs its gains arrive without the operators. The search stalls after closing ~92% of the gap on the flagship problem, and the registered family-hint test gives the stall its first reading: named in words, the reference family is adopted and loses; handed as code, it is optimized, but our best finite-grid implementation remains below the plateau reached unaided. The loop transported and optimized the idea it was handed; no unaided run produced it. We release the harness, every candidate, and the dated pre-registrations.

1 Introduction

The paper measures proposer-side operators and proposer strength within a fixed, office-scale verified-search loop. Its factorial favors the composed package, while the results also qualify how its gains and collapse resistance should be read.

1 Introduction

  • The study pre-registers its batteries and reports exact paired tests, dated amendments, and negative results alongside positive ones.The tested packages are schematic memory, a standing question, and behavioural repulsion.Source: §1 · pp. 1–2
  • +0.196 is the composition’s reported gain in the nominal two-stage analysis, with nominal pooled p = 0.023.The contribution passage reports the complete factorial and notes robustness to the metric’s seed floor.Source: §1 · p. 2
  • The factorial finds no positive memory×repulsion interaction, bounding any such interaction to about ±0.04 of frac.The remaining anchoring pattern is descriptive collapse prevention rather than superadditive gain.Source: §1 · p. 2
  • The composition closes a median 0.71 of the seed-to-reference gap, with per-problem values from 0.10 to 0.99.Two problems share plateaus across all arms.Source: §1 · p. 2
  • The operator packages are evaluated separately from proposer strength, which is compared under the same loop and budget.In a frontier-strength scoping run, single-run operator differences are at most 0.001–0.003.Source: §1 · p. 2

2 Related work

Prior verified-search systems add archives, novelty rejection, or verbal reflections, while recent work also explores persistent memory. This paper distinguishes its schematic notebook and tests the operator packages factorially under a fixed local proposer.

2 Related work

  • The paper identifies a gap in complete factorial tests of a schematic notebook, obstacle line, and construction-hash duplicate control under one fixed local proposer.It distinguishes its persistent, compressed lineage memory from ReEvo’s reflections, which comment on pairs.Source: §2 · p. 2
  • Related systems combine planning, hybrid memory, or search niches, but the paper frames its factorial as a way to isolate components in such architectures.LoongFlow is identified as the closest architecture to the composed arm.Source: §2 · p. 2
  • Behavioural repulsion transplants novelty search into the proposer prompt, using the construction itself rather than an embedding as the behaviour descriptor.The cited motivation includes collapse into attractor regions and reduced pass@k coverage after RL tuning.Source: §2 · pp. 2–3

3 The harness, the operators, the problems

The harness uses a fixed verified-search loop and tests bundled prompt and duplicate-control changes across nine construction problems. Runs use cheap exact verification and a fixed 120-sample budget.

3 The harness, the operators, the problems

  • Each loop generation builds island-specific prompts, samples candidates, scores them in a sandbox with the problem verifier, retains bounded elites, and migrates best programs every third generation.The local proposer is Qwen3-Coder-30B-A3B-Instruct; the paper also describes an API proposer under the identical loop.Source: §3 · p. 3
  • The schematic notebook replaces three verbatim elites with one best program and a model-written recap; the agenda names an obstacle, while repulsion rejects duplicate constructions and lists found constructions as forbidden.The factorial tests packages, not individual mechanisms, because packages bundle multiple prompt and filtering changes.Source: §3 · p. 3
  • Nine repository construction problems are selected for cheap exact verification and known headroom, with 120 verified samples per run.The primary metric frac is the fraction of the seed-to-record gap closed; auc averages it over the budget, and div measures distinct behaviours among valid candidates.Source: §3 · p. 3

4 The factorial

In the completed factorial, the composed arm’s pooled seed-to-record-gap gain is positive but nominal across a two-stage sequence, and effects vary by problem. The cube finds no positive memory-by-repulsion interaction, while memory+repulsion uniquely avoids collapse.

4 The factorial

  • +0.045 was the median per-problem E−A effect; leave-one-problem-out means ranged +0.153 to +0.224 despite large within-problem replicate spread.Circle packing’s two replicate effects were +0.01 and +0.56.Source: §4 · p. 4, Figure 1 · p. 5
  • +0.307 on construction-hash diversity (D vs A; [+0.149, +0.477], p = 0.0039) was positive in all nine problems in both replicates.In the factorial, repulsion-on cells had 0.68–0.80 construction-hash diversity versus 0.41–0.59 without repulsion.Source: §4 · p. 4
  • The factorial found no positive memory×repulsion interaction (β = +0.003 [−0.039, +0.035], p = 0.47); the nominal composite gain decomposes additively.The descriptive decomposition assigns roughly 58% to notebook, 16% to agenda, 12% to repulsion, and 14% to the three-way term; per-term intervals are wide and the parts remain unresolved.Source: §4 · p. 4
  • Across the combined nine problems, the pooled E vs A contrast on frac was +0.196 [+0.059, +0.343] (p = 0.023), nominal accumulated evidence given stage-combination p ≈ 0.08.With the seed floor removed, the contrast is unchanged at +0.196 (p = 0.023).Source: §4 · pp. 3–4, Figure 1 · p. 5
  • E vs BD differed by +0.025 (p = 0.55); BD (memory+repulsion) alone had zero collapses across 18 runs, versus 3 for D and 4 for CD.The collapse asymmetry persists across the reported frac thresholds; these were performance collapses, not validity collapses.Source: §4 · pp. 4–5
  • Repulsion alone was best on wide valid manifolds but associated with collapses under fragile validity; E had the best mean and was best or tied-best in five of nine problems.The table caption identifies isosceles-free and 180! as family plateaus where every arm meets the same wall.Source: Table 1 · p. 4, §4 · p. 5

5 Scale, proposers, and the wall

On beat-the-average, the local proposer’s plateau persisted across the tested grid and verifier-time changes, while Claude Opus 5 reached a higher value quickly. The section also connects the remaining gap to construction families and shows that operators do not supply a construction the proposer lacks.

5 Scale, proposers, and the wall

  • At 600 samples, the composed arm reaches 0.3816, exceeding repulsion alone, which stalls at 0.3514; refining the pmf grid from L=5000 to L=20000 at equal budget makes the local proposer worse (0.3561).Source: §5 · p. 5
  • Memory+repulsion is the only factorial arm without a collapse; memory-on cells hold the four best gap fractions, while repulsion-on cells have the highest construction-hash diversity.The figure reports cell means over nine problems and two replicates, with the memory-by-repulsion axes barely interacting.Source: Figure 2 · p. 6
  • 92–98% of the seed-to-reference gap is closed, yet the proposers’ numerically found finite measures differ from the reference’s recursively structured family, which cannot be optimal if finitely supported.The local best is a bimodal sparse measure of 16 atoms; Claude’s best uses an exact sparse-atom objective, analytic gradient, and multiplicative-weights refinement.Source: §5 · pp. 5–6
  • On max–min distance, the frontier proposer emits a minimax program in generation zero and every valid sample converges to the same value; the loop verifies rather than discovers it.The scoping runs therefore locate a boundary: operators structure search but do not inject construction knowledge the proposer lacks.Source: §5 · p. 6

6 The family-hint test: verbal specification versus executable seed

In the registered family-hint test, verbal instruction led the model into the named family but lost in search, while executable seeds were optimized and unaided runs did not produce the family signature.

6 The family-hint test: verbal specification versus executable seed

  • The best tested finite-grid family seed scored 0.3742, below the 0.3816 sparse-atom plateau, and was an author-implemented finite approximation rather than the theoretical multiscale construction.The seed’s initial score differed from other conditions, so final values mix family and starting-point advantages.Source: §6 · pp. 7–8
  • About 0.025 was lost when the structural hint led the model into the target family, which is inferior at this grid.The family signature in the final constructions indicates that the instruction was followed.Source: §6 · p. 7
  • +0.085 from the crude in-family seed, the loop optimized the supplied code and remained loyal to the family, while no unaided registered run matched its signature.No strong-seed run deserted to the better sparse family.Source: §6 · pp. 7–8
  • The registered hypothesis was not supported: neither verbal instruction nor the tested finite encoding crossed the last mile under this budget.The historical 0.3816 comparison comes from a different battery under strong seed variance; richer encodings, the theoretical family, or a larger budget remain open possibilities.Source: §6 · p. 8

7 Discussion

At office scale, the composed arm closes a median 0.71 of the seed-to-reference gap across nine problems, while the operator results support memory paired with repulsion as a practical low-collapse recipe. The discussion also cautions that proposer-strength and plateau interpretations remain bounded by the tested settings.

7 Discussion

  • A median 0.71 of the seed-to-reference gap was closed across nine problems, with per-problem results from 0.10 to 0.99, using 120–600 verified samples and a local model on one laptop.A frontier proposer at about US$50 of API reached parity with values large systems report on the flagship problem; the measured local-model, hundreds-of-samples band was unoccupied.Source: §7 · p. 8
  • Memory+repulsion is the only zero-collapse arm and has a mean within 0.025 of the full composition, supporting it as the parsimonious recipe despite the full composition’s highest mean.The full composition’s additional mean is not distinguishable from zero and comes with the agenda’s extra call; notebook lineage and duplicate control offer the proposed operational explanation.Source: §7 · pp. 8–9
  • +0.008 absolute over the local model’s 600-sample best, the frontier proposer reaches the value with roughly 20× fewer verified samples on this one problem; grid, time, and more samples move the plateau by only +0.0002.This is not yet a proposer-strength curve: further problems, reasoning effort, ensembles, alternative representations, and thousand-sample budgets remain untested.Source: §7 · p. 8
  • The supported design rules are to avoid repulsion alone on fragile manifolds, use a construction-derived rounded hash rather than an embedding, and test plateaus as candidate family boundaries.The data cannot distinguish the full composition’s +0.025 mean over memory+repulsion from zero; the authors recommend spending on the proposer before spending on samples.Source: §7 · p. 9

8 Experimental provenance

The experimental provenance records preregistration before runs and makes explicit that the nine-problem primary analysis combines sequential looks rather than independent confirmation.

8 Experimental provenance

  • Every battery was registered before running, and the combined analyses are sequential.The table makes the order of looks explicit.Source: §8 · p. 9
  • The nine-problem primary analysis is a two-look sequence: discovery followed by a prospective extension.The sequence is not an independent confirmation.Source: Table 3 · p. 9
  • The discovery-plus-extension analysis should be read as sequential evidence, not as an independent replication.Table 3 records the order of registrations and looks.Source: Table 3 · p. 9, §8 · p. 9

9 Limitations

The limitations constrain inference to a small, selected suite and to verifier-evaluation efficiency, while several measures and comparisons have specific scope boundaries. The proposer comparisons do not rank systems or establish de novo discovery.

9 Limitations

  • At n=9, no per-term effect survives correction, so the operator decomposition is directional rather than statistically resolved, and collapse asymmetry is descriptive.The problems were selected rather than randomly sampled; the exact tests quantify sign consistency within this suite, not generalization to the repository or construction problems generally.Source: §9 · p. 9
  • Budgets match proposals and verifier evaluations, not demonstrably tokens; sample efficiency therefore means verifier-evaluation efficiency, with arms differing in candidates surviving deduplication.Operator calls add 1–3% of proposer-output characters, while input-token accounting is incomplete.Source: §9 · p. 9
  • The diversity metric shares the repulsion filter’s rounded construction hash, lacks symmetry canonicalization, and has untested rounding sensitivity.A retrospective AST-structure metric is directionally consistent on logged-code runs, but its paired D−A result is at the n=3 resolution floor.Source: §9 · p. 9
  • Public problems and literature leave proposer performance unable to distinguish retrieval of pretrained construction knowledge from de novo discovery.The generation-zero minimax program is identified as an extreme case.Source: §9 · p. 9
  • The scaling ladder covers one domain, the API proposer is not bitwise reproducible, and no system-level head-to-head with ShinkaEvolve or AlphaEvolve is attempted.Repository comparisons place the study’s wall but do not rank systems; the operators are framed as structuring search, not replacing construction knowledge the proposer lacks.Source: §9 · p. 10

10 Conclusion

In the fixed office-scale loop, the composed operator arm has the highest mean, while memory plus repulsion is the only configuration with no collapses and is within 0.025 of that mean. Frontier-proposer and family-hint tests expose limits the operators do not resolve.

10 Conclusion

  • The composed arm has the highest mean E, with one collapse in eighteen runs; memory+repulsion is the only zero-collapse configuration and is within 0.025 of E’s mean.The authors identify memory+repulsion as the parsimonious recipe these data support.Source: §10 · p. 10
  • A frontier proposer reaches a higher value with roughly 20× fewer verifier evaluations on the one problem where the proposers were compared.On beat-the-average, both proposers stalled inside one small finite-support construction family, below a multiscale reference; other plateaus are only consistent with analogous boundaries.Source: §10 · p. 10
  • In the registered flagship test, the reference family was adopted and lost when named in words; supplied as code, it was optimized, but the best tested finite-grid version stayed below the unaided plateau.Across three registered runs, no unaided candidate matched the idea’s signature.Source: §10 · p. 10
  • The authors conclude that the loop transported and optimized the supplied idea, while their finite encoding did not close the gap and whether the last mile requires an idea remains open.Source: §10 · p. 10

A Reproducibility

The release includes the search harness, verifiers, run data, analyses, generated appendices, and dated pre-registrations and amendments. It logs every candidate from every reported run.

A Reproducibility

  • The release provides the search harness, problem verifiers, run batteries, analysis scripts, generated appendices, and every candidate from every reported run.Source: A Reproducibility · p. 10
  • The dated laboratory notebook records every pre-registration and amendment, while docs/PLANO.md dates every hypothesis, amendment, and result.Source: A Reproducibility · p. 10
  • The local proposer was Qwen3-Coder-30B-A3B-Instruct in 8-bit MLX, and the frontier comparison used anthropic.claude-opus-5 on Amazon Bedrock.Source: A Reproducibility · p. 10
Loading 2609.29636v1…