PaperlayerView exact revision on arXiv

Source-linked AI summary

A Hybrid Nested Harness for Decoupling Structure and Parameters in LLM-Driven Optimization

Víctor Gallego

arXiv:2608.08156v1cs.LGcs.NE

TL;DR

LLM-driven optimization conflates structural design with numerical tuning, even though these decisions favor different optimizers. The paper introduces a hybrid nested search in which an LLM proposes structural sketches and an inner solver tunes their numeric gaps. Across three scientific problem families, the hybrid method outperforms both vanilla LLM search and pure numerical optimization, while remaining bounded by solver, dimensionality, and novelty constraints.

  • Problem

    LLM-driven search jointly updates structure and continuous parameters, although language models are less efficient than numerical solvers at parameter optimization.

  • Method

    A frozen LLM proposes structural sketches with numeric holes, and a pluggable inner optimizer tunes those holes before the outer loop scores and revises each structure.

  • Results

    Across three problem families, hybrid search outperforms vanilla LLM-driven search and pure numerical optimization; on inter-cloud Cloudcast, it achieves $169 versus $213 and $317.

  • Takeaways & Limitations

    Separating structure from parameters removes parametric aliasing and makes the advantage grow with the tuning gap across inner solvers, outer optimizers, and scientific domains.

  • Takeaways & Limitations

    The method requires parseable manifests, specialized solvers for mixed-integer or high-dimensional holes, and remains best suited to human-in-the-loop use because of a novelty ceiling.

Abstract

from arXiv · show

In evolutionary algorithms powered by language models, the LLM acts as a single operator that simultaneously updates structural components (like control flow) and continuous parameters. While LLMs can be good at the first, they are not efficient at the second, wasting tokens taking discrete jumps inside a trial and error loop. We resolve this by formalizing a hybrid nested search, in which an outer loop has the LLM propose a structural sketch, with numeric gaps, and an inner numerical optimizer tunes the sketch. Both the outer and inner solvers are pluggable: any text-based optimizer can be combined with a zero-order optimizer (CMA-ES), gradient-based routines, or MCMC samplers. We validate our framework across three scientific domains: (i) meta-optimizers on closed-form test functions, (ii) code-based policies for systems research and social dilemmas; and (iii) approximate Bayesian inference tasks. Across all three, the hybrid optimizer is superior to both vanilla LLM-driven search and pure numerical optimization baselines. Code at: https://github.com/vicgalle/hybrid-nested-search

1 Introduction

The paper separates LLM-suited structural decisions from numerically tunable parameters, formalizing a nested search that combines both capabilities. Its framework lets an outer text-based optimizer propose sketches while an inner solver tunes their numeric gaps.

  • Motivation: Artifacts decompose into text-based structure, such as control flow, and continuous or mixed-integer parameters, such as thresholds and learning rates.The paper argues that language models benefit from structural priors, whereas traditional numerical solvers are more efficient for parameter optimization.
  • Framework: Hybrid nested search formalizes this decomposition as an outer LLM structural search paired with an inner numerical parameter optimizer.The framework supports any text-based outer optimizer and numerical inner optimizer through a sketching interface.
  • Validation: The framework validates this factorization across meta-optimizers, code-based policies, and approximate Bayesian inference tasks, with multiple solver choices.The reported contributions include CMA-ES, VI, NUTS, simple LLM search, and reflective prompt evolution.
  • Framework: The inner loop tunes each proposed sketch before its fitness is used for scoring, promotion, and feedback to the next outer iteration.This replaces evaluation at the LLM’s suggested parameters with evaluation at the tuned result.

2 Hybrid Nested Search

Hybrid nested search treats structure and parameters as separate optimization levels: the LLM proposes and revises sketches, while a pluggable inner solver tunes their numeric holes. This exposes structural quality more faithfully by ranking candidates after tuning, though its benefit depends on tuning gaps, evaluation costs, and solver limitations.

  • 2.1 Decomposition and the parameter manifest: A candidate program is a structure τ with h(τ) numeric holes and a parameter vector θ whose dimension can vary with the structure.An instantiation map fills the holes to produce executable code.
  • 2.1 Decomposition and the parameter manifest: Each numeric gap is specified through a manifest containing bounds, type, and the LLM’s suggested value, which initializes the inner optimizer.Supported types include continuous, integer, logarithmic, and power-of-2 coordinates.
  • 2.2 Bilevel objective and algorithm: The resulting bilevel objective selects the structure with the highest fitness after optimizing its parameters.The inner solver estimates the optimized fitness within an evaluation budget, while the LLM searches over structures.
  • 2.2 Bilevel objective and algorithm: The inner Tune routine is solver-agnostic, supporting CMA-ES, gradient-based methods, or MCMC depending on the objective and inference target.CMA-ES handles nondifferentiable black-box objectives, while gradient methods suit differentiable magnitudes and MCMC supports distributional optimization.
  • 2.2 Bilevel objective and algorithm: The outer LLM receives optimized fitness, parameters, and diagnostics, then proposes a revised structure using this feedback.The feedback exposes a structure’s potential at its parametric best rather than only the original guessed-parameter score.
  • 2.3 Parametric de-aliasing: Parametric aliasing occurs when differences in parameter guesses distort structural rankings, causing vanilla search to reject structurally superior candidates.The proposition bounds hybrid ranking error by 2ε for an ε-accurate inner solver, while vanilla search can discard the structural optimum.
  • 2.3 Parametric de-aliasing: The hybrid advantage grows with the tuning gap and vanishes when the LLM already guesses parameters near optimally.The paper identifies selection and tuning contributions to the advantage and measures the gap empirically.
  • 2.4 On the cost of hybrid search: Hybrid cost is K(c_llm + B_in c_ev), adding 10–100% overhead when evaluations are much cheaper than LLM proposals and B_in is about 100.When evaluations are expensive, sample-efficient or amortized inner solvers are preferred instead of general-purpose CMA-ES.

3 Experiments

Across meta-optimization, executable-policy, and Bayesian-inference tasks, hybrid nested search combines LLM structural proposals with inner numerical tuning and generally outperforms vanilla and fixed-structure numerical baselines. Its gains are largest when structures contain parameters that the LLM cannot guess reliably, while solver and outer-optimizer choices remain flexible.

  • Experimental setup: The evaluation compares vanilla LLM joint search, numerical-only fixed-structure optimization, and hybrid search across three problem families.Unless noted, the inner budget is B_in = 100 and the objective is withheld from the LLM.
  • Meta-optimizers: Hybrid search improves vanilla by up to many orders of magnitude on Ackley, Rastrigin, and Rosenbrock, and beats oracle CMA-ES on multimodal cases.The methods tie on ellipsoid; Schwefel remains hardest, and no method reaches the global optimum.
  • Executable policies for systems and social dilemmas: Hybrid search wins every evaluated cloud-systems regime; on inter-cloud Cloudcast, hybrid costs $169 versus $213 for joint LLM search and $317 for pure CMA-ES.This is reported as a 1.9× improvement over pure CMA-ES using a hand-designed topology.
  • Executable policies for systems and social dilemmas: Hybrid search achieves the highest welfare in 9/9 Cleanup model×seed runs, with its advantage over pure CMA-ES increasing monotonically with difficulty.All three models converge on the same deadzone-plus-ramp-plus-saturation-cap family, but hybrid tuning reaches heavy-task welfare 0.582.
  • Outer-optimizer robustness: Replacing the evolutionary outer loop with GEPA preserves the qualitative advantage in 6/8 non-negligible-gap regimes, while reducing the marginal inner-loop gain from approximately 10–20% to 5–10%.The two lowest-gap regimes tie, and the hybrid parameterization advantage does not reverse where tuning matters.
  • Approximate Bayesian inference: In Bayesian inference, affine Gaussian reparameterizations improve VI by +37.7/+9.3 nats but tie under NUTS, whereas nonlinear funnel and eight-schools transforms improve NUTS by +3.1/+2.2 decades.NUTS already adapts a linear preconditioner, while nonlinear curvature is not fixable by a mass matrix.
  • Approximate Bayesian inference: Simulation-based calibration certifies proposed transforms: every proposer found a reparameterization yielding at least 10× ESS/grad on sticky horseshoe regression, while rejecting a smooth but wrong posterior transform.The best improvement reached 24×, and 0/25 live proposals were SBC-rejected.
  • Approximate Bayesian inference: On the banana target, tuning one hidden curvature parameter changes ELBO from approximately −74 to −13, a +60-nat gain on identical structure.After VI finds the curvature, NUTS samples the de-curved space at +1.57 decades, approximately 37×.

4 Related work

Related work spans LLM-driven program discovery, using language models as optimizers, and bilevel optimization or inference reparameterization. This paper distinguishes itself by assigning structural search and numeric tuning to operators matched to their variable types.

  • LLM-driven program and algorithm discovery: LLM-driven discovery systems place frozen language models in evolutionary loops to synthesize programs, algorithms, heuristics, agents, and other artifacts.Recent work emphasizes sample efficiency, recursive self-improvement, automated agent design, and end-to-end autoresearch.
  • LLMs as optimizers, and prompt optimization: Another research line uses LLMs to optimize numeric or structured spaces, including evolution strategies, Bayesian optimization, reward code, and prompt-related objectives.The paper instead delegates continuous search to a matched numerical solver through de-aliasing.
  • Bilevel optimization, reparameterization, and automatic inference: The framework formulates its target as a bilevel program and assigns the outer and inner levels to operators matched by variable type, connecting to hyperparameter optimization, meta-learning, and automatic inference.Its inference examples relate to non-centering and normalizing-flow transport methods.

5 Conclusion

Factoring structure from parameters removes parametric aliasing in LLM-driven search, with benefits that grow with the tuning gap and disappear when initial guesses are already good. The method remains bounded by parsing, solver dimensionality, and novelty limitations.

  • Separating structure from parameters removes parametric aliasing that can cause joint search to discard good structures with default parameters.
  • The hybrid advantage scales with the mean tuning gap and vanishes when the model already guesses parameters well.
  • The manifest must be parseable, and mixed-integer parameter gaps require more specialized solvers.
  • Generalist CMA-ES degrades beyond a few dozen dimensions, requiring capped inner problems or future neural inner operators.
  • A novelty ceiling remains, so the method is best used as a human-in-the-loop copilot rather than an autonomous frontier system.

A Proof of Proposition 1

The proof formalizes how parameter gaps distort vanilla structural comparisons and shows that inner tuning removes these inversions up to solver error. Expected advantage grows with both the mean and variance of the tuning gap, under a simplifying independence assumption.

  • Vanilla and hybrid agree on a structural preference when parameter-gap differences do not reverse the sign of the true structural-value difference.
  • Vanilla can reject a structural optimum when its tuning-gap difference exceeds the optimum’s true-value advantage, whereas hybrid avoids this error when the advantage exceeds 2ε.
  • The deployed vanilla-versus-hybrid advantage decomposes into non-negative tuning and selection terms, up to ε.
  • The expected tuning term increases with the mean gap μ and vanishes exactly when Δ is identically zero.
  • Under independent structural values and tuning gaps, expected advantage increases with both E[Δ] and Var(Δ), and equals zero when Δ ≡0.

B Additional results

Additional closed-form results show that practical gains depend on the objective landscape: both methods can reach precision on an easy ellipsoid, while deceptive Schwefel landscapes expose differences and shared local-minimum failures.

  • The evaluation uses three proposals per arm, an inner CMA-ES budget of 100, and a direct-CMA-ES reference budget of 2000.
  • On the convex ellipsoid, both arms reach machine-precision basins, making their practical difference moot.
  • On Schwefel, GLM and Gemini hybrid improve over vanilla and beat direct CMA-ES, while Opus ties because both arms remain in the same local minimum.
  • No arm reaches the global optimum on Schwefel, where the global basin is distant and the gradient is deceptive.

B.2 Cloud-systems policies

The cloud-policy evaluation compares delivered costs across regimes and proposer models, using pure CMA-ES as a model-independent reference. Hybrid search achieves the lowest cost in every reported cell, while the near-zero-gap intra-cloud case provides a tie control.

  • The configuration uses three proposals per arm, inner CMA-ES budget 100, and delivered cost defined as the minimum tuned loss across proposals.
  • Figure 3 compares Can’t Be Late and Cloudcast delivered costs by arm and regime for two proposer models, with lower costs preferred.
  • Intra-cloud Cloudcast routing is the E[Δ] ≈0 control, where cost-only shortest paths are already near-optimal and the methods tie.
  • Hybrid delivers the lowest cost in every evaluated cloud-policy cell.
  • The measured E[Δ] ordering for Can’t Be Late is not monotone in engineered regime difficulty, because untuned guesses are poorest in the cheap-restart regime.

B.3 Cleanup

Cleanup results show that hybrid search gains more over vanilla when parameter guesses are weaker, while remaining robust across selection rules and solver choices.

  • Model and selection effects: Opus has the highest heavy-regime vanilla welfare at 0.45, while Gemini shows the larger hybrid advantage at 1.70× versus Opus’s 1.29×.The comparison follows model strength and the measured tuning gap.
  • Model and selection effects: Hybrid search wins 9/9 under last-iteration selection and is less sensitive to selection rules than vanilla.The inner tuner repairs the current structure’s parameterization.
  • Outer-optimizer robustness: GEPA-hybrid wins every non-negligible-gap regime, while the two ties occur in the lowest-Δ regimes.GEPA replaces the basic outer loop while CMA-ES remains the inner tuner.
  • Outer-optimizer robustness: Under GEPA, the hybrid edge shrinks because reflective optimization recovers more parameter gain, but it does not reverse when Δ is materially nonzero.The low-Δ headline ties are attributed to structure-selection noise after post-hoc tuning.
  • Bayesian suite: The Bayesian suite compares matched nonlinear geometries, where hybrid wins, with affine controls, where NUTS predicts ties.VI gains are measured in nats, while NUTS gains use decades of divergence-aware ESS/grad.

B.6 The hardest case studies

The hardest case studies test whether hybrid gains survive on practitioner-difficult Bayesian targets, using SBC-certified reparameterizations and specialized inner solvers.

  • Horseshoe logistic regression: On horseshoe logistic regression, all five proposer models found SBC-certified reparameterizations achieving at least 10× ESS/grad over identity plus NUTS adaptation.The identity baseline has 520 divergences and ESS/grad 6.2 × 10^-4; 0/25 live proposals were rejected by SBC.
  • Curved banana: On the banana target, all three proposers constructed the nonlinear quadratic shear, whose single curvature parameter hybrid search tuned.The shear lies outside the tested non-centering, affine, and sinh idioms.
  • Curved banana: Hybrid tuning improves the banana ELBO by approximately +60 nats on the same structure with one hidden parameter.This is presented as the clearest de-aliasing demonstration in the project.
  • Curved banana: After VI finds the curvature, NUTS samples the de-curved space at +1.57 decades of ESS/grad, approximately 37×.The result illustrates complementary solver strengths rather than interchangeable tuning ability.

C A gallery of discovered artifacts

The artifact gallery makes the factorization concrete: LLMs supply recognizable algorithmic structures, while inner optimization determines whether their numeric constants work.

  • Discovered optimizers: CMA-ES tunes a Nesterov optimizer’s learning rate from 0.05 to 6.0 and clip threshold from 1.0 to 0.011, reaching loss 2.0 × 10^-6.The tuned configuration uses a large step rate stabilized by tight gradient clipping.
  • Discovered optimizers: Periodic random restarts augment Nesterov heavy-ball optimization on the multimodal rastrigin lattice.The structure adds globally aware exploration absent from plain CMA-ES on the objective.
  • Discovered policies: The Cleanup artifact uses hysteresis: cleaner commitment changes at upper and lower pollution thresholds to avoid chattering.The policy is exposed through a stateless pure-function API.
  • Discovered policies: The Can’t Be Late scheduler maps slack, restart overhead, and remaining work to wait, spot, or on-demand actions through tuned thresholds.The selected tuned values include spot_offset −3480, finish_spot_buffer 11337, and hysteresis 6714.
  • Discovered reparameterizations: On banana, VI tunes a nonlinear quadratic shear’s curvature to 0.937 and ridge width to 0.300, recovering generative constants 1.0 and 0.3.The same gallery describes this as a single hidden hole determining the transform’s curvature.
  • Discovered reparameterizations: On horseshoe regression, partial non-centering reaches ESS/grad 1.48 × 10^-2 versus identity plus adaptation at 6.2 × 10^-4.The transform combines an affine map with one tunable partial-non-centering knob.
  • Discovered reparameterizations: On gauss_rot, VI tunes 38 affine holes from untuned ELBO −33.4 to +4.30, a +37.7-nat gain over vanilla.The same transform barely moves NUTS because its warmup mass matrix already supplies linear preconditioning.
  • Gallery-wide pattern: Across eight artifacts, LLMs provide structural ideas while inner solvers provide the constants that determine whether those ideas work.The examples span optimizers, policies, and reparameterizations.

D Consolidated results: every cell

Table 2 consolidates every experimental cell across executable policies and approximate Bayesian inference, reporting vanilla, numerical-only, and hybrid results with tuning gaps and hybrid advantages. The results show hybrid gains across nontrivial regimes, while ties occur in low-gap or already-adapted cases.

  • Table schema: Table 2 reports vanilla, numerical-only, hybrid, mean tuning gap E[Δ], and hybrid advantage for every experimental cell.Bold marks the better vanilla/hybrid result; numerical-only is an oracle only in block A and a deployable no-structure baseline elsewhere.
  • Executable policies: With GEPA outer search and CMA-ES inner tuning, hybrid wins every non-negligible-gap regime; the two ties are the lowest-Δ regimes.Across Can’t Be Late, Cleanup, and Cloudcast, reported advantages range from +3.3% to +14.4% in winning regimes.
  • Executable policies: Cloudcast inter-cloud routing reaches cost $201.8 with hybrid, versus $213.3 vanilla and $317.4 numerical-only, a +5.4% hybrid advantage.The consolidated table reports cost as the metric for Cloudcast and distinguishes intra- from inter-cloud regimes.
  • Approximate Bayesian inference: Hybrid improves VI ELBO by +5.8 nats on eight_schools, +9.3 on gauss_ill, and +37.7 on gauss_rot, while funnel ties.The listed values compare untuned vanilla with tuned hybrid in the VI reparameterization suite.
  • Approximate Bayesian inference: Under NUTS, hybrid gains +3.12 and +2.17 decades on funnel and eight_schools, but gauss_ill and gauss_rot tie.The affine controls show near-zero advantages (+0.01 and −0.02), consistent with NUTS warmup adapting a mass matrix.
  • Approximate Bayesian inference: On horseshoe logistic regression, Gemini 3.1 Pro achieves ESS/grad 1.48×10−2 versus 6.2×10−4 numerically, a +1.37-decade advantage.The numerical baseline has 520 divergences, whereas the partial non-centering transform has 73 and passes SBC.
Loading 2608.08156v1…