PaperlayerView exact revision on arXiv

Source-linked AI summary

Pre-training of Bayesian Optimization Algorithm through Bayesian Optimization

Satoshi Katayama, Shoyo Hunt, Shintaro Masuda, Masayuki Karasuyama

arXiv:2610.10186v1cs.LGstat.ML

TL;DR

BO algorithms require advance parameter choices that can substantially affect performance. The paper selects configurations by estimating expected cumulative regret from GP sample paths and optimizing that estimate with outer BO; experiments show it can identify high-performing configurations.

  • Problem

    BO algorithms often require parameters to be specified before optimization, and those choices can substantially affect performance.

    Source: §1 · p. 1

  • Method

    The framework estimates expected cumulative regret by running BO on sampled objective functions, then uses outer BO to optimize the parameter configuration.

    Source: §1 · p. 2

  • Results

    Experiments on TuRBO and LSBO demonstrated that the method can identify high-performing configurations.

    Source: §7 · p. 12

  • Takeaways & Limitations

    The results suggest that prior knowledge about objective functions can inform BO algorithm configuration as well as surrogate modeling.

    Source: §7 · p. 12

    The theoretical analysis leaves the continuous-configuration case for future work, as it may require additional regularity conditions.

    Source: §3.3 · p. 7

Abstract

from arXiv · show

Bayesian optimization (BO) is widely used as a standard approach for expensive black-box optimization. However, BO algorithms often involve parameters that must be specified in advance, and their performance can strongly depend on these choices. We propose a framework for optimizing such parameters using sample paths drawn from a Gaussian process (GP) inferred from the information available at the start of BO. We use cumulative regret as the performance metric for a BO algorithm. By running the BO algorithm on the generated sample paths, we obtain an empirical estimate of its expected cumulative regret for a given parameter configuration. Optimizing this estimate allows us to identify parameter configurations that, given the currently available information, are expected to achieve low cumulative regret. Since this parameter optimization is itself a black-box optimization problem, we employ another BO procedure to solve it, which we refer to as outer BO. Through experiments, we demonstrate that the proposed framework can effectively select parameter configurations that achieve strong performance among a range of candidate configurations.

1 Introduction

The paper selects BO algorithm parameters by minimizing estimated finite-horizon Bayes cumulative regret under the currently available distribution over objective functions. It uses outer BO for this parameter search and reports theoretical guarantees and strong performance among candidate configurations.

1 Introduction

  • The framework estimates expected cumulative regret by running a candidate BO algorithm on objective-function sample paths drawn from the initial distribution.This targets finite-horizon performance given the information available before BO begins.Source: §1 · p. 2
  • The approach uses objective-function prior knowledge for both surrogate prediction and algorithm configuration while retaining existing BO structure and tuning only a small number of parameters.The distribution can come from an initial-data GP posterior or a GP prior trained on related tasks.Source: §1 · p. 2
  • Outer BO optimizes this estimated regret because evaluating configurations is a costly black-box problem.Each evaluation requires repeated finite-horizon BO runs, and the regret estimate is generally difficult to express analytically or differentiate.Source: §1 · p. 2
  • The framework also extends to multi-fidelity evaluations using shorter inner-BO horizons and to multi-stage configurations with different parameter values across optimization stages.These extensions are stated as part of the proposed framework.Source: §1 · p. 2
  • For finite candidate sets, the paper guarantees the expected-regret gap of the MC-selected configuration is of the same order as the estimator error.The MC cumulative-regret estimator also has a convergence result; the guarantee does not depend on the acquisition function.Source: §1 · pp. 2–3
  • Experiments show the framework identifies configurations with strong finite-horizon BO performance among candidates.The paper reports experiments on TuRBO and LSBO.Source: §1 · p. 3

2 Bayesian Optimization (BO)

This section introduces GP-based BO and the algorithm parameters that may require configuration before optimization. It outlines how acquisition choices and search-strategy designs can affect the procedure.

2.1 Preliminary

  • A GP models the unknown objective, and its posterior mean and covariance are conditioned on observations collected during BO.The next observation point is selected by maximizing an acquisition function.Source: §2.1 · p. 3
  • The paper assumes a maximum of T BO iterations and names expected improvement and GP-UCB as example acquisition functions.GP-UCB is expanded in the passage as Gaussian process Upper Confidence Bound.Source: §2.1 · p. 3

2.2 Parameter Configuration of BO

  • BO algorithm parameters θ must often be set before optimization, and they are distinct from GP hyperparameters that can be estimated by marginal likelihood.Such algorithm parameters are often chosen heuristically because standard GP hyperparameter tuning does not apply to them.Source: §2.2 · pp. 3–4
  • Configuration examples include an arbitrary EI improvement threshold and TuRBO trust-region settings, while LSBO has VAE loss-balance and latent-dimension choices.These choices can affect optimization results or performance.Source: §2.2 · pp. 3–4

3 Proposed Method: Pre-trained BO

Pre-trained BO chooses algorithm parameters by minimizing expected cumulative regret under the initial distribution over objective functions. It estimates this target with GP sample paths and outer BO, and establishes error guarantees under stated conditions.

3.1 Evaluation of BO Algorithm

  • Pre-trained BO selects θ by minimizing expected cumulative regret under the initial posterior, which can incorporate expert knowledge or data from related tasks.The posterior can be estimated from the initial dataset and enriched with a prior mean and covariance.Source: §3.1 · pp. 4–5

3.2 Monte-Calro Approximation and Optimization

  • It estimates this target by running T-step BO on N GP sample paths, then uses outer BO to optimize the resulting black-box estimate.Evaluating a configuration requires repeated finite-horizon BO runs, making exhaustive evaluation impractical.Source: §3.2 · pp. 5–6

3.3 Theoretical Properties

  • Under stated regularity conditions, estimator error is O_p(T/√N), without restricting the inner BO acquisition function.The bound has standard Monte Carlo dependence on N and first-order dependence on T.Source: §3.3 · pp. 6–7
  • For finite Θ, the selected configuration’s expected-regret gap from the optimum has the same order; extending this guarantee to continuous Θ requires uniform convergence.The complete continuous-case analysis is left for future work, and may require additional regularity conditions.Source: §3.3 · p. 7

4 Extensions

The extensions use shorter BO runs as cheaper fidelities for outer BO and allow the algorithm’s configuration to change during a run. For a one-switch configuration, a correlated surrogate accommodates different input dimensions across fidelities.

4.1 Multi-fidelity Approach for Outer BO

  • For fixed θ, regret at t<T serves as a cheaper low-fidelity approximation to the T-step target, allowing MFBO to choose θ and fidelity jointly.The described setup uses a two-output GP for the low- and high-fidelity objectives.Source: §4.1 · p. 7

4.2 Optimizing Multi-Stage Configurations

  • A one-switch configuration jointly tunes θ(1) and θ(2), while the mid-run regret objective depends only on θ(1).The switch occurs at Tmiddle, separating the earlier and later inner-BO iterations.Source: §4.2 · p. 8
  • A co-kriging-inspired surrogate models correlated low- and high-fidelity objectives despite their different input dimensions.This accommodates the low-fidelity objective’s dependence on θ(1) alone.Source: §4.2 · p. 8

5 Related Work

The paper relates its approach to learning-to-optimize methods and transferable function priors. It retains established BO algorithms while tuning their parameters, and treats learned objective-function distributions as complementary inputs.

Learning optimization policies

  • Unlike L2O and meta-learning methods that learn optimizers or acquisition policies, this approach retains an established BO algorithm and optimizes its pre-specified parameters.The structured search is intended to leverage established acquisition rules and their exploration–exploitation principles while adapting to a target task distribution.Source: §5 · pp. 8–9

Learning distributions over objective functions

  • Learned function priors and this framework are complementary: a pre-trained GP prior can supply the objective-function distribution used to sample paths and tune BO parameters.The framework can also be combined with other probabilistic surrogate models that support function sampling.Source: §5 · p. 9

6 Experiments

The experiments evaluate Pretrained BO and its multi-stage extension on TuRBO and LSBO configurations. Across the reported settings, the method selects high-performing configurations, while outer BO and multi-fidelity outer BO search more efficiently than random search and standard BO, respectively.

6 Experiments

  • The experiments compare the framework on GP sample-path functions using TuRBO and on a synthetic CNN-based latent-space optimization task using LSBO.The TuRBO setup generated training and test functions from a GP posterior conditioned on an initial dataset.Source: §6 · p. 9, §6.1 · p. 9, §6.2 · pp. 11–12

6.1 TuRBO

  • TuRBO experiments optimized seven parameters over 3,888 candidates; the multi-stage version used three parameters per stage across 11,025 candidates.The multi-stage configuration space was reduced for computational tractability.Source: §6.1 · pp. 9, 11
  • When Pretrained BO fell below all 1,000 random configurations, its selection could roughly be regarded as belonging to the candidate set’s top 0.1%; it also beat standard BO and MetaBO in many settings.The comparison covers test-function performance; Pretrained BO-MS substantially improved results in several settings and was comparable to Pretrained BO in others.Source: Figure 1 · p. 10, §6.1 · p. 10
  • Outer BO generally searched more efficiently than random search, while MFBO further improved search efficiency over standard BO.The comparison uses cumulative inner-BO values against total evaluation cost; Fig. 2 evaluates BO for Pretrained BO and MFBO for Pretrained BO-MS.Source: Figure 2 · p. 11, §6.1 · p. 11

6.2 LSBO

  • In the Fashion-MNIST latent-space task, Pretrained BO-MS identified a high-performing configuration among candidates, with expected score as the reported outcome.BO optimized a 20-dimensional VAE latent space, and the configuration included VAE hyperparameters.Source: Figure 3 · p. 11, §6.2 · p. 12

7 Conclusion

The paper concludes that Pretrained BO selects configurations by minimizing expected cumulative regret estimated from GP sample paths, with theoretical guarantees and encouraging experiments. For multi-stage MFBO, its acquisition optimization handles the requirement for a full configuration even at low fidelity.

7 Conclusion

  • Pretrained BO configures BO by minimizing expected cumulative regret estimated from GP sample paths, and the framework extends to multi-fidelity and multi-stage settings.Experiments on TuRBO and LSBO demonstrated identification of high-performing configurations.Source: §7 · p. 12
  • The conclusion identifies GP-posterior misspecification as a limitation that may lead to suboptimal BO parameter choices.Evaluating robustness under model misspecification is named as future work.Source: §7 · p. 12

A Details of MFBO for Multi-Stage Configurations

  • For MFSKO’s low-fidelity acquisition, the procedure first optimizes the full configuration at high fidelity, then optimizes low-fidelity parameters with the high-fidelity parameters fixed.The same procedure is stated to apply to MFMES.Source: A Details of MFBO for Multi-Stage… · p. 16

B.1 Setting of TuRBO Experiment

The appendices detail candidate settings for TuRBO and LSBO, multi-fidelity modeling choices, and results under different initial sample sizes.

B.1 Setting of TuRBO Experiment

  • TuRBO candidates vary success/failure tolerances, initial, maximum, and minimum trust-region sizes, and the number of regions maintained in parallel.The listed candidate values are discrete settings for each parameter.Source: §B.1 · p. 16
  • TuRBO multi-stage candidates include success/failure tolerances and initial and minimum region sizes; inner and outer GP kernel parameters are fitted by marginal log-likelihood.Training functions use 1,000 random Fourier feature basis functions.Source: §B.1 · p. 17

B.2 Setting of LSBO Experiment

  • The LSBO VAE objective uses separate weights for mutual-information, total-correlation, and dimension-wise KL penalties, controlling retained information, latent independence, and marginal-prior matching.These weights are α, β, and γ, respectively.Source: §B.2 · p. 17
  • LSBO’s finite candidate set combines α, β, and γ values on a log-scale grid with latent dimensions from 4 to 32.The latent dimensions considered are 4, 8, 16, 24, and 32.Source: §B.2 · p. 17

B.3 Additional Details of MFBO

  • For multi-fidelity modeling, Pretrained BO uses a standard multi-output GP with shared input dimensions, while Pretrained BO-MS uses a model that handles different input dimensions.The Pretrained BO model employs a linear model of coregionalization with a shared latent kernel across fidelities.Source: §B.3 · p. 18

C.1 Results on Different n0

  • The additional TuRBO results use initial sample sizes of 50 and 300, in contrast to the main setting.Source: §C.1 · p. 18

C.2 Results with Error Bars

The appendix reports standard errors for the results in Figures 6–8.

C.2 Results with Error Bars

  • Figures 6–8 show standard errors; each of the 1,000 random configurations has an error estimate based on 10 runs.Source: §C.2 · p. 18

D Proof of Proposition 3.1

The proof establishes a conditional variance bound for cumulative regret and the posterior-path regularity needed for it. It also bounds the canonical pseudometric for Matérn kernels by a power of input distance, with the exponent depending on smoothness.

D Proof of Proposition 3.1

  • Dudley’s entropy bound and Borell–TIS establish almost-sure boundedness of the centered posterior process and a finite conditional second moment.Almost-sure boundedness permits applying Borell–TIS; the proof then shows E[M0^2 | D0] is finite.Source: D Proof of Proposition 3.1 · pp. 18, 25, 26, 27

N Var(RT | D0)

  • Var(R_T | D0) ≤ 4C_D0T^2 for a BO run.The proof uses this bound in controlling the Monte Carlo estimate of expected cumulative regret.Source: D Proof of Proposition 3.1 · p. 28

E Bounds of Canonical Pseudo-Metric

F Proof of Corollary 3.1

For a finite set of candidate configurations, the corollary proof uses a union bound to establish the comparison between the selected configuration and the optimum under estimated regret.

F Proof of Corollary 3.1

  • The estimated regret at the selected configuration is no greater than at θ⋆: dBRT(bθN) ≤ dBRT(θ⋆).The proof considers a finite candidate set and takes uniform constants across its configurations using a union bound.Source: F Proof of Corollary 3.1 · p. 30
Loading 2610.10186v1…