PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models to Enhance Bayesian Optimization

Tennison Liu, Nicolás Astorga, Nabeel Seedat, Mihaela van der Schaar

arXiv:2402.03921v2cs.LGcs.AI

TL;DR

BO must search expensive black-box functions efficiently from limited observations, where sparse data challenges surrogate modeling and candidate selection. LLAMBO uses natural-language prompting and in-context learning to enhance warmstarting, surrogate modeling, and candidate sampling, and reports strong hyperparameter-tuning performance across diverse benchmarks. Its main cost is greater inference computation, and results can depend substantially on the chosen LLM.

  • Problem

    BO seeks efficient optimization of expensive black-box functions, but sparse observations make surrogate modeling and candidate sampling challenging.

  • Method

    LLAMBO uses structured natural-language prompts and in-context learning to enhance warmstarting, surrogate modeling, and conditional candidate sampling.

  • Results

    LLAMBO improves zero-shot warmstarting, surrogate modeling, and candidate sampling, especially when observations are sparse, and performs strongly as an end-to-end HPT method.

  • Takeaways & Limitations

    LLAMBO’s modular components can complement existing BO frameworks or operate as a stand-alone BO method.

  • Takeaways & Limitations

    LLAMBO requires substantially more computation than traditional BO because inference through LLMs has a larger computational footprint.

Abstract

from arXiv · show

Bayesian optimization (BO) is a powerful approach for optimizing complex and expensive-to-evaluate black-box functions. Its importance is underscored in many applications, notably including hyperparameter tuning, but its efficacy depends on efficiently balancing exploration and exploitation. While there has been substantial progress in BO methods, striking this balance remains a delicate process. In this light, we present LLAMBO, a novel approach that integrates the capabilities of Large Language Models (LLM) within BO. At a high level, we frame the BO problem in natural language, enabling LLMs to iteratively propose and evaluate promising solutions conditioned on historical evaluations. More specifically, we explore how combining contextual understanding, few-shot learning proficiency, and domain knowledge of LLMs can improve model-based BO. Our findings illustrate that LLAMBO is effective at zero-shot warmstarting, and enhances surrogate modeling and candidate sampling, especially in the early stages of search when observations are sparse. Our approach is performed in context and does not require LLM finetuning. Additionally, it is modular by design, allowing individual components to be integrated into existing BO frameworks, or function cohesively as an end-to-end method. We empirically validate LLAMBO's efficacy on the problem of hyperparameter tuning, highlighting strong empirical performance across a range of diverse benchmarks, proprietary, and synthetic tasks.

1 INTRODUCTION

Bayesian optimization efficiently searches expensive black-box functions from limited observations, but sparse data makes surrogate modeling and candidate sampling difficult. LLAMBO investigates whether LLM priors and few-shot abilities can improve these components and end-to-end BO.

  • Bayesian optimization uses observed data to learn a surrogate and iteratively propose promising points without direct access to the objective or its gradients.
  • Sparse observations make accurate surrogate modeling difficult, while misspecification can bias the sampling of potential solutions.
  • The few-shot BO setting demands rapid learning and generalization from very few examples.
  • LLMs may address this setting through few-shot prediction, contextual understanding, and encoded priors learned from large-scale pretraining.
  • LLAMBO frames BO components in natural language to test whether LLMs improve surrogate modeling, candidate sampling, and cohesive end-to-end operation.

2 LLAMBO: LLMS TO ENHANCE BO

LLAMBO translates BO components and search history into natural-language prompts so LLMs can contribute priors and in-context learning to warmstarting, surrogate modeling, and candidate sampling. The study evaluates these integrations systematically across BO tasks.

  • LLAMBO represents BO components in natural language, allowing an LLM to suggest and evaluate solutions using the problem description and search history.
  • The BO objective is to find h* in H minimizing the costly black-box function f, whose gradients are unavailable.
  • Surrogate modeling predicts outputs from inputs and observations, while candidate sampling generates points from a posterior informed by observed data and priors.
  • LLMs can contribute encoded prior knowledge and few-shot learning through in-context learning, particularly when observations are sparse.
  • LLAMBO applies in-context learning to zero-shot warmstarting, conditional candidate sampling, and discriminative or generative surrogate modeling.
  • The investigation analyzes components individually before assessing LLAMBO end to end, using 74 Bayesmark and HPOBench tasks with GPT-3.5.
  • Structured prompts incorporate the problem description, optimization history, and additional prompt elements specified for each BO component.

3 RELATED WORKS

Related work spans probabilistic, neural, and tree-based BO surrogates, transfer learning across related optimization tasks, and emerging uses of LLMs for optimization. LLAMBO differs by enhancing multiple BO components rather than focusing primarily on one.

  • BO research includes Gaussian-process, neural-network, and tree-based surrogates for flexible modeling across optimization settings.
  • Transfer-learning approaches reuse common structures across related black-box functions or previously optimized functions, often within a fixed search space.
  • Prior LLM optimization work includes prompt optimization, evolutionary search operators, and molecular BO focused primarily on surrogate modeling.
  • LLAMBO extends LLM use to multiple model-based BO components and systematically investigates the gains from that integration.

4 WARMSTARTING THE BO PROCESS

LLAMBO uses zero-shot prompting to warmstart BO with points informed by varying levels of problem context. Across experiments, contextual warmstarts improved search, especially early, while also reflecting task-specific hyperparameter correlations.

  • Method: LLAMBO samples warmstart points with zero-shot prompts under no-context, partial-context, and full-context settings.
  • Experimental setup: The warmstarting evaluation compares GP and TPE against Random, Sobol, and Latin Hypercube initialization over 5 initial points and 25 trials.
  • Empirical insights: Zero-shot warmstarting without context surpasses random initialization, indicating generalizable knowledge about hyperparameter correlations.
  • Empirical insights: Additional dataset information improves warmstart search performance for partial and full context, with the strongest gains before trial 5.
  • Empirical insights: LLM-recommended initial points show stronger and task-dependent hyperparameter correlations than independently sampled random points.
  • Empirical insights: Zero-shot warmstarting transfers information about optimization-landscape correlations and enhances search performance.

5 SURROGATE MODELING

LLAMBO uses in-context learning to build surrogate models that predict objective values and uncertainty from sparse observations. Its discriminative approach improves prediction performance, while uncertainty calibration remains weaker than probabilistic methods and benefits from shuffled few-shot examples and informative priors.

  • Method: The discriminative surrogate predicts the objective’s mean and uncertainty, while a generative alternative estimates the probability of exceeding a performance threshold.The two approaches correspond to p(s|h; Dn) and p(s ≤τ|h; Dn).
  • Method: LLAMBO uses natural-language serializations of observed trajectories and query points as few-shot inputs to an LLM.The model outputs predicted scores and probabilities, with repeated predictions used to estimate mean and standard deviation.
  • Robustness: Permuting few-shot examples improves both uncertainty quantification and prediction performance.The approach addresses sensitivity to example ordering in left-to-right LLM processing.
  • Prediction performance: LLAMBO consistently outperforms GP and SMAC in prediction performance across sampled observations.Prediction performance is evaluated using NRMSE and R2, alongside regret.
  • Uncertainty calibration: GPs provide the best uncertainty estimates, while LLAMBO approaches similar calibration as observations increase and achieves lower sharpness as coverage improves.Calibration is assessed using LPD, empirical coverage, and sharpness.
  • Role of prior knowledge: Informative problem descriptions and hyperparameter names improve prediction performance and calibration relative to LLAMBO (UnInf).The ablation omits the problem description and replaces hyperparameter names with “Xi”.

6 SAMPLING OF CANDIDATE POINTS

LLAMBO samples candidate points by conditioning an LLM on a desired objective value and optimization history. The exploration parameter α controls the trade-off between candidate quality and diversity, including limited extrapolation beyond observed regions.

  • Sampling mechanism: LLAMBO generates candidate points by directly conditioning on a desired objective value and the observed optimization history.This extends TPE-style sampling from good regions to targeting specific objective values.
  • Target selection: The target objective s′ is defined from the best and worst observed values, with α controlling extrapolation or conservatism.Positive α targets values better than the current best, whereas negative α keeps the target within the observed range.
  • Effect of α: As α increases from −0.5 to 0, average regret and best regret improve, but average regret worsens beyond 0.Sampling increasingly outside the observed distribution compromises candidate reliability.
  • Effect of α: α=0.01 produces the optimal best regret while sampling points outside regions occupied by good points.At α=−0.2, sampled points cover a region similar to good points.
  • Quality–diversity trade-off: Increasing α decreases generalized variance and log-likelihood as candidates concentrate in smaller regions and move farther from observed points.These metrics capture diversity through spread and proximity-related likelihood.
  • Quality–diversity trade-off: Direct conditioning can generate high-quality candidates, but α-dependent sampling may sacrifice diversity.The exploration hyperparameter provides a mechanism for balancing these competing objectives.

7 END-TO-END DEMONSTRATION OF LLAMBO

LLAMBO is evaluated as an end-to-end Bayesian optimization method on diverse hyperparameter-tuning tasks, using its discriminative surrogate model. It achieves the best tuning performance across both public and private/synthetic datasets, particularly early in search.

  • LLAMBO uses its discriminative surrogate model for end-to-end Bayesian optimization.
  • The evaluation covers 50 hyperparameter-tuning tasks from public, proprietary, and synthetic datasets, with five seeded searches per task and 25 trials per search.
  • LLAMBO achieves the best tuning performance on both public datasets and private and synthetic datasets.
  • LLAMBO is especially effective early in the search, when fewer observations are available.
  • The end-to-end pipeline exhibits sample-efficient search and allows individual components to be integrated into existing frameworks.

8 DISCUSSIONS

LLAMBO extends Bayesian optimization with LLM-based components and reports improvements across warmstarting, surrogate modeling, and candidate sampling. The framework trades greater inference cost for sample efficiency, while its demonstrated scope remains focused on hyperparameter-tuning tasks.

  • LLAMBO integrates LLMs into Bayesian optimization through zero-shot warmstarting, generative and discriminative surrogate models, and conditional candidate sampling.
  • The study reports improvements across all three integrations, especially when fewer samples are available.
  • LLAMBO is also reported as an effective stand-alone Bayesian optimization method on diverse benchmarks.
  • Bayesian optimization addresses expensive black-box evaluations by learning a surrogate model and iteratively proposing potential points.
  • The Bayesian optimization cycle updates the surrogate, samples candidates, scores them with an acquisition function, and evaluates the selected point.

B GENERATIVE SURROGATE MODEL [† MOVED DOWN FROM MAIN PAPER]

The generative surrogate models the probability that a candidate belongs to the good region by reframing density-ratio estimation as probabilistic classification with in-context learning. It produces stronger score correlations and better identifies high-potential points than the evaluated TPE variants.

  • Method: The generative surrogate models inputs conditioned on objective values, separating good points below a threshold from bad points above it.
  • Method: Bayes’ rule converts the density ratio into the probability that a candidate produces a good objective value.
  • Method: The LLM estimates candidate scores through in-context probabilistic classification of whether each observed score meets the threshold.
  • Experimental setup: The evaluation compares LLAMBO’s generative surrogate with independent and multivariate TPE variants using score correlation and regret.
  • Results: LLAMBO achieves higher score correlations, especially at n=5, while baseline correlations improve as n increases.
  • Results: Generative surrogate modeling via in-context learning predicts scores more accurately and improves identification of high-potential points.
  • Prompt design: The prompt appendix includes zero-shot warmstarting prompts, in-context surrogate prompts, and target-conditioned candidate-sampling prompts.

C.2 ABLATION STUDY

The ablation study tests the effects of optimization-problem context and task instructions in LLAMBO prompts. The complete configuration performs best, while removing candidate-generation instructions substantially lowers candidate acceptance.

  • Ablation design: The ablation study removes optimization-problem metadata or additional task instructions from the standard LLAMBO prompts.
  • Performance: The standard LLAMBO configuration outperforms the ablated variants on Bayesmark RandomForest tasks.
  • Performance: LLAMBO without context retains competitive optimization performance despite omitting metadata about the underlying task.
  • Candidate sampling: Removing candidate-generation instructions reduces acceptance to 69.26% ± 0.79%, versus 91.60% ± 0.45% for LLAMBO.
  • Candidate sampling: The reduced acceptance rate limits the effective candidate pool available for surrogate-based evaluation and selection.

D DETAIL OF EXPERIMENTAL PROCEDURES

The experiments evaluate LLAMBO on Bayesmark and HPOBench tasks using seeded comparisons, component analyses, and end-to-end procedures. The method combines language-model-based candidate generation and surrogate evaluation with expected improvement selection, while warmstarting results assess how prompt context affects search.

  • Benchmarks: Bayesmark evaluation uses 25 built-in tasks plus 30 private and synthetic tasks, while component tests use five RandomForest tasks.The built-in tasks combine five datasets with five machine-learning models; all tasks support end-to-end evaluation.
  • Benchmarks: HPOBench evaluation covers 24 dataset-model tasks with five seeds and 25 trials, using precomputed tabular validation-loss evaluations.The benchmark includes eight OpenML datasets and three models: XGBoost, RandomForest, and MLP.
  • End-to-end procedure: End-to-end LLAMBO samples candidate points, scores them with a surrogate and expected improvement, then evaluates the highest-scoring point.The procedure iterates over candidate generation, surrogate-based acquisition scoring, and next-point selection.
  • Implementation: The reported instantiation samples M = 20 candidates, uses α = −0.1 for exploration, and draws K = 10 Monte Carlo predictions.The implementation uses gpt-3.5-turbo version 0301 with default hyperparameters.
  • Warmstarting: Increasing warmstart prompt informativeness from no context to partial and full context improved search performance for both GP and TPE.The analysis also reports higher correlations between warmstarting points recommended by an LLM.
  • Surrogate comparison: A constrained single-run comparison across public Bayesmark tasks found the discriminative surrogate stronger for end-to-end evaluation, while the generative version was a close second until late trials.The authors caution that definitive conclusions from one seed are difficult and identify sensitivity to τ as a future research topic.

E.3 ADDITIONAL RESULTS ON BAYESMARK

Additional results compare LLAMBO with a broader set of BO baselines across Bayesmark and HPOBench. LLAMBO shows strong overall tuning performance, particularly in early search stages, but performance varies across model types and incurs higher surrogate-computation time.

  • Bayesmark comparisons: Additional Bayesmark comparisons include optimized TPE, DNGO, STO, Turbo, and HEBO alongside the original BO baselines.These methods broaden evaluation across Bayesian neural-network, neural-network, trusted-region GP, and other BO approaches.
  • Bayesmark results: LLAMBO consistently achieves the best overall average rank, excelling on DecisionTree and RandomForest across public and private benchmarks but performing less well on SVMs.The authors leave the cause of this model-dependent variability speculative and relate it to black-box-function characteristics.
  • HPOBench results: HPOBench results show LLAMBO achieves the best tuning performance across 24 tasks and excels earlier in the search when fewer observations are available.The individual results report task metrics, normalized regret, and rank averaged over five seeded searches.
  • Runtime: LLAMBO incurs higher average clock time per iteration than the compared algorithms when measuring surrogate-model computation alone.Runtime measurements were collected on an Intel i7-1260P laptop and exclude black-box query time.
  • Per-task analyses: The supplementary figures report individual Bayesmark and HPOBench outcomes using task metrics, normalized regret, and average rank, generally averaged over five runs.The metrics use accuracy or negative MSE for Bayesmark task performance and lower-is-better regret or rank measures.
Loading 2402.03921v2…