PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization

Michael Volpp, Lukas P. Fröhlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, Christian Daniel

arXiv:1904.02642v5stat.MLcs.AIcs.LG

TL;DR

Costly black-box optimization often has no gradients, and universal Bayesian optimization acquisition functions make few assumptions about the task. The paper meta-learns a neural acquisition function from related tasks using reinforcement learning; experiments report that MetaBO consistently outperforms existing methods, including in simulation-to-real and hyperparameter-search settings.

  • Problem

    Global black-box optimization often involves expensive evaluations without gradients, while general-purpose acquisition functions make few assumptions about the target objective.

    Source: §1 · p. 1

  • Method

    MetaBO uses reinforcement learning to meta-learn neural acquisition functions on source tasks, encoding implicit structural knowledge within Bayesian optimization.

    Source: §1 · p. 2

  • Results

    MetaBO consistently outperforms existing methods, including in simulation-to-real settings and hyperparameter search tasks.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The approach is presented as applicable to practical problems with either scarce or abundant source data.

    Source: §6 · p. 10

    The intended use is on unseen functions drawn from the training distribution.

    Source: §5 · p. 9

Abstract

from arXiv · show

Transferring knowledge across tasks to improve data-efficiency is one of the open key challenges in the field of global black-box optimization. Readily available algorithms are typically designed to be universal optimizers and, therefore, often suboptimal for specific tasks. We propose a novel transfer learning method to obtain customized optimizers within the well-established framework of Bayesian optimization, allowing our algorithm to utilize the proven generalization capabilities of Gaussian processes. Using reinforcement learning to meta-train an acquisition function (AF) on a set of related tasks, the proposed method learns to extract implicit structural information and to exploit it for improved data-efficiency. We present experiments on a simulation-to-real transfer task as well as on several synthetic functions and on two hyperparameter search problems. The results show that our algorithm (1) automatically identifies structural properties of objective functions from available source tasks or simulations, (2) performs favourably in settings with both scarse and abundant source data, and (3) falls back to the performance level of general AFs if no particular structure is present.

1 INTRODUCTION

Global black-box optimization often involves costly, low-dimensional functions without gradients, while general-purpose Bayesian optimization methods make few assumptions about target functions. The paper proposes meta-learning a neural acquisition function from related tasks to encode task structure and improve data-efficiency without requiring objective-function gradients.

1 INTRODUCTION

  • The target setting includes relatively low-dimensional functions with expensive evaluations and typically no available gradients.Applications mentioned include hyperparameter tuning, control-parameter identification, and system design.Source: §1 · p. 1
  • General-purpose acquisition functions use minimal assumptions about target objectives, although Bayesian optimization uses a Gaussian-process surrogate to generalize from data and balance exploration with exploitation.The acquisition function encodes how that trade-off is performed.Source: §1 · p. 1
  • Repeated optimization in similar settings, such as tuning one model across datasets, motivates specialized optimizers that transfer knowledge across tasks.The paper identifies this as a practical motivation for transfer learning.Source: §1 · pp. 1–2
  • The proposed method meta-learns a neural acquisition function on source tasks using reinforcement learning, encoding implicit task structure while retaining Bayesian optimization’s Gaussian-process surrogate.The training procedure does not require objective-function gradients.Source: §1 · p. 2
  • The method is demonstrated on simulation-to-real control, two hyperparameter optimization problems, and synthetic functions.The authors present this as evidence of efficiency and practical applicability.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses meta-learning local optimization and transfer learning for Bayesian optimization, but the paper distinguishes its approach by learning a neural acquisition function for global black-box optimization without gradients. It also contrasts this approach with transfer acquisition methods that combine source and target information explicitly.

2 RELATED WORK

  • Learning to learn, meta-learning, and transfer learning describe improving performance or convergence on tasks through experience with similar tasks.The passage notes a long history of research on these ideas.Source: §2 · p. 2
  • Meta-learning research also includes learned update rules, step-size controllers, neural-network update strategies, and initialization methods for solving new learning tasks with few gradient steps.These approaches primarily address local optimization or training machine-learning models.Source: §2 · p. 2
  • The cited prior work on meta-learning global black-box optimization assumes gradients and uses supervised learning, unlike the present setting where gradients are typically unavailable.That prior method represents the optimizer as a recurrent network that outputs the next query from optimization-history statistics.Source: §2 · p. 2
  • Earlier transfer approaches combine source and target data in Gaussian processes or ensembles, while many rely on hand-engineered dataset features to assess source-task relevance.Alternative surrogate models have also been proposed to address Gaussian processes’ cubic scaling.Source: §2 · pp. 2–3
  • The closest related method, Wistuba et al. (2018), combines source-task predicted improvements with target-task expected improvement in a transfer acquisition function, whereas this method learns a neural acquisition function whose weighting adapts online.The learned network encodes source-dataset knowledge in its weights rather than storing and evaluating many source Gaussian processes, which the authors say permits larger amounts of source data than TAF.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries describe optimization from noisy evaluations, Bayesian optimization’s surrogate-and-acquisition loop, and reinforcement learning. They also introduce expected improvement and transfer acquisition functions as reference strategies.

3 PRELIMINARIES

  • The optimization setting chooses each query from past evaluations of an unknown objective, which may be noisy; the optimizer has no access to objective gradients.Performance is assessed using simple regret, defined in the passage as the gap between the optimum and the best evaluated point, though its displayed expression is truncated.Source: §3 · p. 3
  • The optimizer’s iterate is the best evaluation found through the current step, as used in the simple-regret definition.The supplied regret description states this role but truncates the displayed formula.Source: §3 · p. 3
  • Bayesian optimization fits a probabilistic surrogate to the optimization history, then uses an acquisition function to score candidate points and select the next query.With a Gaussian-process surrogate, the posterior has a mean and variance used to balance exploration and exploitation.Source: §3 · p. 3
  • Expected improvement is the experiment’s uninformed baseline acquisition function, and it has a closed form when the objective value is Gaussian.Other general-purpose choices mentioned are probability of improvement and GP-upper confidence bound.Source: §3 · p. 3
  • The transfer acquisition function combines target-task expected improvement with predicted improvements from source tasks.It uses separate Gaussian-process surrogate models for source and target tasks; its weights can be based on predicted variances or pairwise performance-rank comparisons.Source: §3 · pp. 3–4
  • Reinforcement learning frames an agent’s interaction as a Markov decision process, with actions leading to new states and scalar rewards that the policy seeks to maximize cumulatively.The policy maps states to probability distributions over actions.Source: §3 · p. 4

4 METABO ALGORITHM

MetaBO replaces the hand-designed acquisition function in Bayesian optimization with a neural acquisition function meta-trained on source tasks, while retaining the GP surrogate. Its reinforcement-learning training procedure uses an adaptive candidate set to learn a data-efficient strategy for related tasks.

4 METABO ALGORITHM

  • MetaBO meta-trains a neural acquisition function on source tasks to exploit shared objective structure while retaining the Gaussian-process surrogate in the Bayesian optimization loop.The learned strategy is intended to improve data-efficiency on new instances from the same function class.Source: §4 · p. 4
  • During offline meta-training, cheaper approximations of a function class can supply source tasks from simulations or prior optimization runs.The procedure adapts the acquisition-function parameters to the class's implicit structure.Source: §4 · p. 4
  • The neural acquisition function receives the GP posterior mean and variance, input location, optimization step, and budget as features.This pointwise design is based on GP posterior predictions and is described as scalable and usable as a plug-in feature in BO frameworks.Source: §4 · pp. 4–5
  • During training, PPO treats neural acquisition-function outputs on candidate locations as categorical-policy logits and samples the next evaluation location.At test time, the method selects the location maximizing the acquisition function, as in classical BO.Source: Figure 1 · p. 5, §4 · pp. 5–6
  • The candidate set combines a coarse global Sobol grid with local maxima found by optimizing the neural acquisition function from its top grid evaluations.The local candidates support exploitation, while the global grid maintains exploration.Source: §4 · p. 6
  • Meta-training rolls out episodes on sampled source functions, records state-action-reward tuples, and updates the policy using PPO; negative simple regret is the experiment reward.The true maximum is needed only during training, and a coarse-grid approximation may be used when it is unknown.Source: §4 · p. 6

5 EXPERIMENTS

Across synthetic benchmarks, simulation-to-real control, and hyperparameter search, MetaBO learned data-efficient acquisition strategies that outperformed standard and transfer baselines. Its performance also remained at least on par with EI on function classes with little exploitable structure.

Global Optimization Benchmark Functions

  • The experiments compared MetaBO with EI and both ranking-based and mixture-of-experts TAF, reporting median simple regret over 100 runs unless stated otherwise.The synthetic benchmark set used Branin, Goldstein-Price, and Hartmann-3 functions with random translations and scalings.Source: §5 · pp. 6–7
  • MetaBO-50 outperformed EI early and both TAF variants over wide optimization-budget ranges on translated and scaled benchmark functions.It used initial non-greedy evaluations to identify target-function properties; TAF-20 results are in the appendix.Source: Figure 2 · p. 7, §5 · p. 7
  • Using the full available source-task set produced performance comparable to MetaBO-50 on benchmark functions, and full-source training was beneficial on the more complex simulation-to-real task.Unlike TAF, MetaBO did not require manual selection of a restricted source-task set.Source: §5 · pp. 7–8

Simulation-to-Real Task

  • On hardware, MetaBO consistently found stabilizing controllers in fewer than ten BO iterations, while benchmark acquisition functions did not reliably do so within 25 steps.The method transferred from simulation and adjusted its strategy to target-specific properties.Source: Figure 3 · p. 8, §5 · p. 9

Hyperparameter Optimization

  • On both SVM and AdaBoost hyperparameter tasks, MetaBO learned data-efficient strategies that surpassed EI and TAF using the same 35 source datasets as TAF.The remaining 15 datasets were held out for evaluation.Source: §5 · p. 9, Figure 4 · p. 9

General Function Classes

  • On general function classes, MetaBO performed better than or at least on par with EI across D = 3, 4, and 5 without retraining.Training used D = 3 tasks and excluded the x-feature; tests included RBF and Matern-5/2 kernel function classes.Source: §5 · p. 10, Figure 5 · p. 10

6 CONCLUSION AND FUTURE WORK

MetaBO transfers task structure into neural acquisition functions within Bayesian optimization, and the experiments show strong performance across several practical tasks. The authors identify multi-task, multi-fidelity optimization as a direction for future work.

6 CONCLUSION AND FUTURE WORK

  • MetaBO injects prior knowledge into BO’s optimization strategy through flexible meta-learning of neural acquisition functions.The experiments report consistent outperformance of existing methods, including in simulation-to-real and hyperparameter-search settings.Source: §6 · p. 10
  • The experiments support applicability across settings with scarce and abundant source data.The resulting neural acquisition functions can represent search strategies beyond weighted combinations of source-data priors and standard acquisition functions.Source: §6 · p. 10
  • The authors propose multi-task, multi-fidelity optimization as future work and expect MetaBO’s sample efficiency to be valuable there.Source: §6 · p. 10

A.1 INTERPRETATION OF NEURAL AF SEARCH STRATEGIES

MetaBO learns neural acquisition functions that use source-task structure to guide Bayesian optimization beyond simple combinations of standard acquisition functions and priors. On toy function classes, it learns an informative non-greedy first evaluation that identifies the target instance and helps locate the global optimum efficiently.

Emergence of Non-Greedy Search Strategies

  • The Rhino-1 class translates a two-bump objective, while Rhino-2 varies a wide bump’s height and places a sharp peak at that height.MetaBO’s learned strategy uses information from the wider bump to infer the sharp global optimum’s location.Source: §A.1 · p. 14
  • On Rhino-1 and Rhino-2, a fixed non-greedy first evaluation reveals the target instance, allowing the global optimum to be found in the second step under noiseless evaluations.For Rhino-1, the probe samples where the smaller, wider bumps overlap; for Rhino-2, it is near x = 0.2.Source: §A.1 · p. 14, Figure 6 · p. 15, Figure 7 · p. 16

Additional Baseline Methods

  • GMM-UCB combines a fitted GMM over source-task best designs with UCB, while EPS-GREEDY samples source-task best designs or uses EI.The GMM-UCB mixture weight and component count, and the EPS-GREEDY probability, were selected by grid search on the test set.Source: §A.1 · pp. 14, 17
  • MetaBO outperforms GMM-UCB and EPS-GREEDY on Branin, Goldstein-Price, and Hartmann-3, beyond a simple combination of standard acquisition functions with a prior over D.The comparison uses 50 source tasks and the baseline parameter configurations selected for those methods.Source: §A.1 · p. 17, Figure 8 · p. 17

A.2 DEPENDENCE ON THE NUMBER OF SOURCE TASKS

The number of source tasks needed for strong performance depends on task complexity: a small set suffices for Branin, whereas stabilization benefits from much more source data. MetaBO’s evaluation runtime is independent of source-task count, and its out-of-distribution generalization is better for heavier, longer pendula than for lighter, shorter ones.

A.2 DEPENDENCE ON THE NUMBER OF SOURCE TASKS

  • MetaBO can process abundant source data by accumulating it in neural acquisition-function weights rather than storing all source data in GP models, avoiding manual selection of a small task subset.It also performs favorably with baseline-sized source sets when tasks do not require large amounts of source data.Source: §A.2 · p. 17
  • A small number of source tasks suffices for strong Branin performance, while the more complex stabilization task needs much more data to be solved reliably.The source-task study plots steps to a median-regret threshold over 100 test functions, with budgets of 30 for Branin and 50 for stabilization.Source: §A.2 · p. 17, Figure 9 · p. 18
  • MetaBO’s evaluation runtime is independent of source-task count, whereas TAF-ME scales linearly with that count and quadratically with data points per task, and TAF-R has an even stronger dependence on task count.The runtime comparison is reported for BO episodes with budget T = 30; absolute runtimes depend on hardware.Source: §A.2 · pp. 17–18, Table 3 · p. 18

A.3 GENERALIZATION BEHAVIOR

  • Outside the training distribution, neural acquisition functions generalize better to heavy, long pendula than to lightweight, short ones, where the optimization landscape changes more markedly.The study also evaluates translated and scaled global-optimization functions; the intended use case remains functions inside the training distribution.Source: §A.3 · p. 19

A.4 FULL SET OF RESULTS FROM MAIN PART

The additional benchmark results include TAF with 20 source tasks, while the simulation-to-real results include TAF-50. The reported benchmark results show that TAF performance does not necessarily improve with more source data.

Global Optimization Benchmark Functions

  • The global-optimization benchmark results in Figure 12 include TAF with M = 20 and show that TAF’s performance does not necessarily increase with more source data.Source: §A.4 · p. 19
  • On global optimization benchmark functions, Fig. 12 reports TAF with M = 20 and shows that TAF’s performance does not necessarily increase with more source data.Source: §A.4 · p. 19

Simulation-to-Real Experiment

  • The full simulation-to-real results include TAF-50, in addition to the results shown in the main part of the paper.Source: §A.4 · p. 19

B EXPERIMENTAL DETAILS

The experimental details include out-of-distribution tests, additional benchmark comparisons, and simulation-to-real transfer results. In the hardware task, MetaBO’s learned acquisition functions produce stabilizing controllers in fewer than ten Bayesian-optimization iterations.

Experimental Details

Generalization to the Simulation-to-Real Task

  • The simulation-to-real generalization test varies measured physical parameters, plots steps to reach regret R = 1.0, and finds the threshold unreachable within 30 steps for f ≤ 0.5.The authors associate this range with lightweight, short pendula that are much harder to stabilize than the training distribution.Source: Figure 11 · p. 20

Global Optimization Benchmark Functions

  • On three global-optimization benchmark functions with random translations sampled uniformly from [−0.1, 0.1]D and scalings from [0.9, 1.1], MetaBO outperformed EI especially in early stages, then outperformed both TAF flavors over wide optimization-budget ranges.The comparison used TAF-R-M and TAF-ME-M with M ∈ {20, 50}; MetaBO-50 used the same source-task set as TAF-50.Source: Figure 12 · p. 20

Simulation-to-Real Experiment

  • On the simulation-to-real hardware task, MetaBO reliably yields stabilizing controllers in fewer than ten BO iterations, while the listed TAF variants and EI explore too heavily.The simulation-to-real results average ten BO runs; MetaBO also shows strong early-time performance and online adaptation.Source: Figure 13 · p. 21

B.1 GENERAL IMPLEMENTATION DETAILS

The experiments use a consistent MetaBO configuration, with Gaussian-process surrogate settings, baseline choices, and acquisition-function maximization procedures specified for evaluation and training.

B.1 GENERAL IMPLEMENTATION DETAILS

  • The same MetaBO hyperparameters were used across all experiments, and their values are summarized in Table 4.The supplied table passage identifies its scope but does not provide the parameter values.Source: §B.1 · p. 21, Table 4 · p. 22
  • Gaussian processes used squared-exponential kernels, or Matérn-5/2 kernels for general function classes, with hyperparameters fitted offline by type-2 maximum likelihood.The method is compatible with online hyperparameter optimization, which was omitted for a consistent comparison with fewer confounding factors.Source: §B.1 · p. 21
  • The baselines were parameter-free EI and both ranking-based TAF-R and product-of-experts TAF-ME; EI used a midpoint initial design, while TAF used source tasks to warmstart.MetaBO also works without an initial design.Source: §B.1 · p. 21
  • AF maxima were found by hierarchical Sobol gridding for continuous domains, while the discrete HPO domain allowed exact maximization.The gridding procedure used multistart points followed by local searches around the top evaluations.Source: §B.1 · p. 21
  • Neural AFs were trained with PPO; the four-layer, 200-unit ReLU networks took about 30 minutes to 10 hours to train, depending on objective-evaluation complexity.A separate network estimated value from optimization step and budget to reduce variance in PPO gradient estimates.Source: §B.1 · pp. 21–22
  • When the true maximum was unknown, training reward was negative regret relative to an approximate maximum; otherwise it was negative log10 simple regret.The approximate maximum came from grid search for GP samples and an LQR controller for the Furuta pendulum task.Source: §B.1 · p. 22

Related papers

Loading 1904.02642v5…