PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMs

Richard Cornelius Suwandi, Feng Yin, Juntao Wang, Renjie Li, Tsung-Hui Chang, Sergios Theodoridis

arXiv:2509.17998v2cs.LGcs.AI

TL;DR

BO’s reliance on GP kernels makes kernel choice important, while fixed or heuristic choices may fit an objective poorly. CAKE uses LLMs to evolve kernels from observations and BAKER ranks them using fit and expected improvement; the paper reports consistent gains across real-world optimization tasks.

  • Problem

    Fixed or heuristic kernel choices may be poorly suited to an objective, while limited BO observations make adaptive kernel selection difficult.

    Source: §1 · p. 2

  • Method

    CAKE uses LLMs as crossover and mutation operators to evolve kernels from observations, while BAKER balances BIC-based fit with expected improvement for kernel selection.

    Source: §1 · pp. 2–3, §3 · p. 4

  • Results

    CAKE-based BO consistently outperformed established baselines across hyperparameter optimization, controller tuning, and photonic chip design tasks.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    The experiments report CAKE’s accuracy advantage in hyperparameter optimization, strong average rewards in controller tuning, and effective balancing of conflicting objectives in photonic chip design.

    Source: §6 · p. 10

    The initial kernel grammar focuses on addition and multiplication, with other closure-preserving operators left for future work.

    Source: D Limitations and Future Work · p. 27

Abstract

from arXiv · show

The efficiency of Bayesian optimization (BO) relies heavily on the choice of the Gaussian process (GP) kernel, which plays a central role in balancing exploration and exploitation under limited evaluation budgets. Traditional BO methods often rely on fixed or heuristic kernel selection strategies, which can result in slow convergence or suboptimal solutions when the chosen kernel is poorly suited to the underlying objective function. To address this limitation, we propose a freshly-baked Context-Aware Kernel Evolution (CAKE) to enhance BO with large language models (LLMs). Concretely, CAKE leverages LLMs as the crossover and mutation operators to adaptively generate and refine GP kernels based on the observed data throughout the optimization process. To maximize the power of CAKE, we further propose BIC-Acquisition Kernel Ranking (BAKER) to select the most effective kernel through balancing the model fit measured by the Bayesian information criterion (BIC) with the expected improvement at each iteration of BO. Extensive experiments demonstrate that our fresh CAKE-based BO method consistently outperforms established baselines across a range of real-world tasks, including hyperparameter optimization, controller tuning, and photonic chip design. Our code is publicly available at https://github.com/richardcsuwandi/cake.

1 Introduction

Bayesian optimization can be slowed by kernels poorly matched to the objective, yet BO’s expensive evaluations leave little data for kernel selection. CAKE uses LLMs to adapt kernels during optimization and pairs this evolution with BAKER, reporting gains across several real-world task types.

1 Introduction

  • Poorly matched or overly general GP kernels can bias BO’s sampling and slow convergence, motivating more sophisticated kernel design.The concern is especially noted for complex functions in moderate-to-high-dimensional spaces.Source: §1 · p. 2
  • With few observations and unavailable gradients, automatic kernel selection is difficult; the paper asks whether LLMs can evolve kernels from observed data to improve BO.The authors connect this setting to few-shot learning and LLM in-context learning.Source: §1 · p. 2
  • CAKE uses LLMs to iteratively generate and refine kernels, while BAKER balances BIC-based model fit with expected improvement to select a kernel.The authors report consistent gains over established baselines in hyperparameter optimization, controller tuning, and photonic chip design.Source: §1 · pp. 2–3

2 Preliminaries

The preliminaries describe BO as sequentially updating a probabilistic surrogate and using its posterior to choose new evaluations. They also introduce a compositional kernel grammar for building expressive kernels from base kernels.

2 Preliminaries

  • BO addresses noisy, expensive, gradient-free black-box optimization by using a probabilistic surrogate, typically a GP, to approximate the objective.The surrogate is updated as observations are collected.Source: §2 · p. 3
  • At each iteration, the posterior conditioned on prior observations informs an acquisition function that selects the next point to evaluate.The acquisition function defines a policy over candidate points in the search space.Source: §2 · p. 3
  • The kernel grammar builds expressive kernels by adding or multiplying valid base kernels, with combinations such as LIN + PER representing a periodic pattern with a linear trend.The grammar also permits replacing a base kernel with another base kernel.Source: §2 · p. 3

3 Context-Aware Kernel Evolution

CAKE evolves a population of GP kernels using LLM proposals conditioned on observations, then uses BAKER to select a kernel for the next BO query. Its operators, fitness evaluation, and query selection form an iterative process that continues until the evaluation budget is exhausted.

3 Context-Aware Kernel Evolution

  • CAKE initializes observations from sampled inputs and uses them as few-shot examples, prompting the LLM to identify patterns and propose kernels.The system prompt frames the LLM as a GP expert and asks it to reason about the observed data.Source: §3 · p. 3, Figure 2 · p. 4
  • A population of kernels is maintained and scored by BIC, which measures the trade-off between model fit and model complexity.Fitness scores are normalized to [0, 1] across tasks.Source: §3 · pp. 3–4
  • LLM crossover combines two fitness-weighted parent kernels, while mutation can replace a base kernel in the fittest candidate with probability pm.The proposed kernels join the population, and the top np candidates form the next generation.Source: §3 · p. 4
  • BAKER weights each kernel’s acquisition value by its BIC-derived weight and selects the kernel maximizing that weighted value for the next query.The selected kernel proposes the next point, whose evaluation updates the observations; this repeats until budget T is exhausted.Source: §3 · pp. 4–5

4 Related Work

Related work spans expressive-kernel construction, adaptive GP surrogates, and uses of LLMs in optimization. The paper positions CAKE as an in-context approach to generating and refining kernels during BO.

4 Related Work

  • Prior expressive-kernel approaches include multiple-kernel learning [4], kernel-structure search [12], spectral approximations [54], and GP-neural-network combinations [55].The passage notes restrictions, modeling expertise, stationarity assumptions, or complex inference for these approaches.Source: §4 · p. 5
  • Adaptive BO surrogates include deep GPs, discrete GP mixtures, parallel GPs, and GP ensembles [31].The passage notes increased computational complexity for deep GPs.Source: §4 · p. 5
  • CAKE extends LLM-based surrogate work [29] [11] [59] by generating and refining kernels during BO, using LLMs in-context without fine-tuning [41].The authors describe this as the first use of LLMs as genetic operators for adaptive, expressive GP-kernel design in BO.Source: §4 · p. 6

5 Experiments

Across hyperparameter optimization, controller tuning, and photonic chip design, CAKE performs strongly against established baselines. The experiments also evaluate its components and compare LLM-driven kernel evolution with random recombination and genetic algorithms.

5.1 Hyperparameter Optimization

  • CAKE achieved the highest test accuracy across all hyperparameter-optimization tasks and excelled early, when fewer observations were available.The experiments cover 60 tasks across 12 OpenML datasets and five ML models, with 100 trials over 20 random seeds per task.Source: §5.1 · pp. 6–7, Figure 4 · p. 7

5.2 Controller Tuning

  • CAKE achieved the highest average rewards in both controller tasks; in robot pushing it converged faster, while only CAKE and ABO reached 200 points in lunar landing.ABO showed greater performance fluctuations than CAKE, especially in more challenging environments.Source: §5.2 · pp. 7–8, Figure 5 · p. 8

5.3 Photonic Chip Design

  • In less than 40 trials, CAKE found a significantly higher-scoring chip solution, equivalent to a tenfold speedup, and achieved the highest score and hypervolume.The study evaluates five competing objectives and reports that CAKE can use different kernels tailored to individual objectives.Source: §5.3 · p. 8, Figure 6 · p. 9

5.4 Ablation Study

  • CAKE + BAKER performed best overall on HPOBench; removing either component degraded performance, and random kernel sampling performed worst.CAKE + BIC and CAKE + Utility each underperformed the combined method; CKS + BAKER outperformed Adaptive + BAKER but still fell short of CAKE + BAKER.Source: §5.4 · p. 9, Table 1 · p. 9

5.5 Evolution of Population Fitness

  • After one LLM edit, kernel fitness shifted upward; successive edits raised mean fitness and narrowed its variance, while random recombination and GA converged more slowly.An LLM edit performs both crossover and mutation.Source: §5.5 · pp. 9–10, Figure 7 · p. 10

6 Conclusion

The paper concludes that CAKE uses LLM crossover and mutation for adaptive surrogate modeling in BO, with BAKER complementing kernel design. Its appendix describes GP modeling and BIC-based kernel selection.

6 Conclusion

  • CAKE uses LLMs as crossover and mutation operators for adaptive surrogate modeling in BO, while BAKER addresses cases where good model fit may not improve optimization progress.Experiments report superior accuracy on hyperparameter tasks, highest average rewards in controller tuning, and a significant speedup in photonic chip design.Source: §6 · p. 10

A.1 Bayesian Optimization with Gaussian Processes

  • A GP models the objective with a mean function and a kernel; finite collections of outputs are jointly Gaussian, with evaluations assumed to have σ-sub-Gaussian noise.The mean function is often assumed to be zero in practice.Source: §A.1 · p. 15

A.2 Model Selection

  • Kernel selection searches a discrete, infinite space of kernel families, and the paper uses BIC as an approximation to Laplace-based model selection that rewards fit while penalizing complexity.Each kernel family includes its hyperparameters.Source: §A.2 · p. 15

B Experimental Details

B.1 Implementation

The implementation evolves candidate kernels using crossover, mutation, and a population, with parameter settings balancing diversity, exploration, and computational cost. Baselines span fixed, adaptive-selection, and compositional kernel methods.

B.1 Implementation

  • CAKE uses 5 crossovers, mutation probability 0.7, and population size 10 to balance exploration, diversity, and computational cost.More crossovers can broaden kernel combinations but increase LLM calls; larger populations require repeated GP fitting, while excessive mutation risks disrupting promising structures.Source: §B.1 · p. 16

B.2.1 Fixed

  • Fixed-kernel baselines use widely used BO kernels with hyperpriors; the Matérn variants use smoothness ν = 1/2, 3/2, or 5/2.The Matérn settings correspond to Matérn-1/2, Matérn-3/2, and Matérn-5/2; kernels can be applied to individual input dimensions.Source: §B.2.1 · pp. 16–17

B.2.2 Adaptive

  • Adaptive baselines select kernels randomly, by acquisition utility, or by the lowest BIC.The adaptive implementation follows Roman et al..Source: §B.2.2 · p. 18

B.2.3 Compositional

  • Compositional baselines include Deep GP, Ensemble GP, Compositional Kernel Search, and Automated BO.Ensemble GP uses CAKE’s six base kernels; Compositional Kernel Search applies greedy search to those same base kernels.Source: §B.2.3 · p. 18

B.3 Benchmarks

The benchmarks cover HPO, controller tuning, and photonic chip design, with prompt ablations evaluating which CAKE components matter. The full prompt performs best among the tested variants.

B.3 Benchmarks

  • Benchmark experiments ran on a consumer-grade laptop except photonic chip design, which used an HPC cluster for physics-based simulation.The laptop was a MacBook Air M2 with 8 GB unified memory and 256 GB SSD storage.Source: §B.3 · p. 18, §B.3.1 · p. 18

B.3.1 Hyperparameter Optimization

  • HPOBench experiments use 12 OpenML datasets and discretized search spaces for tuning machine-learning model hyperparameters.The search spaces specify parameter types, bounds, and whether values are sampled on linear or logarithmic scales.Source: §B.3.1 · pp. 18–19

B.3.2 Controller Tuning

  • Controller tuning covers robot pushing with 14 parameters and a lunar-lander task with a reward system for landing, contact, and engine use.For robot pushing, the reward favors reducing objects’ distances from their goals; the lunar-lander implementation uses Gymnasium.Source: §B.3.2 · p. 19

B.3.3 Photonic Chip Design

  • Photonic chip design optimizes five indicators: Q-factor, wavelength, lasing area, power, and divergence angle.The study identifies target values for the design indicators; the wavelength target is motivated by telecommunications and satellite applications.Source: §B.3.3 · pp. 19–20, Figure 9 · p. 20

B.4 Prompt Design

  • The prompts combine optimization history, kernel-generation instructions, and a request for natural-language reasoning.The ablations remove one component at a time while retaining the others.Source: §B.4 · pp. 20–22
  • Full CAKE outperforms all prompt ablations on HPOBench; without instructions, only 68% of generated kernels are valid.No Context performs significantly worse, while No Reasoning degrades slightly; the authors attribute the validity drop to removing kernel-grammar generation guidelines.Source: Figure 11 · p. 21, §B.4 · p. 22

C Additional Results

The additional-results material identifies a table comparing methods’ average computational time per iteration.

C Additional Results

  • Table 5 reports average computational time in seconds per iteration, with method and main bottleneck as its listed fields.The supplied section text introduces additional empirical analysis but does not state the table’s values.Source: Table 5 · p. 23, C Additional Results · p. 23

C.1 Benchmark Function Optimization

Across benchmark functions, CAKE improves normalized regret relative to the second-best method, and the additional analyses examine runtime, kernel proposals, interpretability, and LLM choice.

C.1 Benchmark Function Optimization

  • CAKE improves mean regret 22.2% and median regret 21.9% over the second-best method, ranks top two on all 15 functions, and wins 12.Table 4 reports average normalized regret over 20 random seeds.Source: §C.1 · p. 23

C.2 Computational Time

  • CAKE has higher average iteration runtime than baselines; LLM inference takes about 1.72 seconds per call and fitting population GPs adds about 3–5 seconds.The runtime comparison excludes black-box evaluations; the authors say API latency and connectivity contribute to CAKE’s overhead, while black-box evaluations are often more time-consuming than BO itself.Source: §C.2 · pp. 23–24

C.3 Case Study

  • In sample proposals, the LLM combines LIN and SE by addition, then proposes replacing SE with RQ to model varying smoothness.The paper’s examples describe these as proposals to improve fitness, with LIN capturing linear trends and SE smoothness.Source: §C.3 · p. 24, Figure 12 · p. 24, Figure 13 · p. 24

C.4 Interpretability of CAKE

  • CAKE’s LLM directly explains kernel expressions, unlike ABCD’s approach of simplifying expressions into canonical sum-of-products form before template-based descriptions.The authors present this as generating natural-language interpretations without explicit simplification or handcrafted template rules.Source: §C.4 · p. 25, Figure 14 · p. 25

C.5 Choice of LLM

  • Across HPOBench functions, CAKE tends to perform better with more recent and capable LLMs; reasoning-based DeepSeek improves performance but takes longer to infer.The study compares three open-source and three closed-source models.Source: §C.5 · p. 25

C.6 Choice of Acquisition Function

CAKE remains comparably effective with EI, UCB, and Thompson sampling, and on most functions their differences are not statistically significant. The tested acquisition functions retain distinct exploration–exploitation trade-offs.

C.6 Choice of Acquisition Function

  • Thompson sampling has slightly higher standard errors from its stochasticity, while CAKE-UCB sometimes performs above or below CAKE-EI.The acquisition functions represent different exploration–exploitation trade-offs.Source: §C.6 · pp. 25–26
  • CAKE-UCB and CAKE-TS perform close to CAKE-EI across benchmark functions, with differences not significant on most functions.The comparison uses normalized regret over 20 random seeds.Source: §C.6 · p. 26, Table 7 · p. 26
  • A one-sided paired Wilcoxon signed-rank test at the 5% level found no statistically significant differences on most functions.The test compares CAKE-EI with CAKE-UCB and CAKE-TS across multiple random seeds.Source: §C.6 · p. 26

C.7 Quantitative Analysis

CAKE accumulates much of its total improvement early in the optimization budget. The analysis measures progress relative to the final performance at the full budget.

C.7 Quantitative Analysis

  • 67.5% of CAKE’s total improvement arrives within 25% of the budget; it exceeds 83% by halfway and is nearly converged by 75%.These averages are reported for HPOBench at different budget levels.Source: §C.7 · p. 26, Table 8 · p. 27
  • Normalized improvement measures the share of total progress achieved by trial t, using initial, best-so-far, and final performance.The final performance is measured at 100% of the budget.Source: §C.7 · p. 26
  • The authors interpret the early gains as rapid identification of effective kernels, especially useful in data-scarce regimes.The paper links early progress to effectiveness when observations are limited.Source: §C.7 · p. 27

D Limitations and Future Work

The authors identify LLM inference cost, possible pretraining exposure, and the initial kernel grammar as boundaries or open directions for CAKE. The checklist also records the paper’s experimental disclosures and its rationale for marking societal-impact discussion not applicable.

D Limitations and Future Work

  • LLM inference can increase computational footprint over traditional BO, which the authors frame as a trade-off for improved sample efficiency.They suggest using CAKE in an earlier optimization stage as one possible efficiency-oriented integration.Source: D Limitations and Future Work · p. 27
  • Pretraining exposure to benchmark literature is possible, but the authors say there is no evidence that specific kernel expressions or optimization trajectories appeared in public data.They cite prompt ablation results as support for in-context adaptation rather than memorization.Source: D Limitations and Future Work · p. 27
  • The initial grammar uses addition and multiplication; the authors say it can be extended with convolution, composition, or affine transformations.The two current operators can already form polynomial and multidimensional kernels.Source: D Limitations and Future Work · p. 27
  • The authors’ longer-term goal is adapting LLM-guided kernel evolution to other kernel-based methods, including SVMs, kernel PCA, and metric learning.These are proposed extensions, not evaluations reported here.Source: D Limitations and Future Work · p. 27

NeurIPS Paper Checklist

  • The checklist reports no theoretical results and says figures use standard errors across independent runs with different random seeds.The paper marks statistical-significance reporting as satisfied.Source: NeurIPS Paper Checklist · pp. 29–30
  • The checklist says experimental details are in the text and Appendix B, and that code is available online.It also marks experimental reproducibility as satisfied.Source: NeurIPS Paper Checklist · pp. 29–30
  • The authors mark societal-impact discussion not applicable, stating that the method does not directly create impacts they feel require highlighting beyond algorithmic improvements.Source: NeurIPS Paper Checklist · p. 31

Related papers

Loading 2509.17998v2…