Source-linked AI summary
Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly
Kirthevasan Kandasamy, Karun Raju Vysyaraju, Willie Neiswanger, Biswajit Paria, Christopher R. Collins, Jeff Schneider, Barnabas Poczos, Eric P. Xing
TL;DR
Modern expensive black-box optimisation creates scalability and robustness challenges for conventional Bayesian optimisation, especially across high-dimensional, multi-fidelity, structured, parallel, and constrained settings. Dragonfly integrates methods for these settings and reports better or competitive performance across synthetic benchmarks and real-world tasks, with consistent performance across a wide array of problems.
Problem
Modern optimisation tasks pose challenges that conventional Bayesian optimisation approaches do not handle well, while black-box evaluations can be expensive and hyperparameter search is often inefficient or manual.
Method
Dragonfly is an open-source Python library integrating scalable and robust Bayesian-optimisation methods for high-dimensional, multi-fidelity, structured, parallel, and complex constrained domains.
Results
Dragonfly performs better or competitively across synthetic benchmarks and real-world computational-astrophysics and model-selection tasks, consistently performing well across a wide array of problems.
Takeaways & Limitations
Integrating these methods enables Bayesian optimisation to be applied across a broad range of challenging real-world optimisation problems.
Abstract
from arXiv · showhide
Bayesian Optimisation (BO) refers to a suite of techniques for global optimisation of expensive black box functions, which use introspective Bayesian models of the function to efficiently search for the optimum. While BO has been applied successfully in many applications, modern optimisation tasks usher in new challenges where conventional methods fail spectacularly. In this work, we present Dragonfly, an open source Python library for scalable and robust BO. Dragonfly incorporates multiple recently developed methods that allow BO to be applied in challenging real world settings; these include better methods for handling higher dimensional domains, methods for handling multi-fidelity evaluations when cheap approximations of an expensive function are available, methods for optimising over structured combinatorial spaces, such as the space of neural network architectures, and methods for handling parallel evaluations. Additionally, we develop new methodological improvements in BO for selecting the Bayesian model, selecting the acquisition function, and optimising over complex domains with different variable types and additional constraints. We compare Dragonfly to a suite of other packages and algorithms for global optimisation and demonstrate that when the above methods are integrated, they enable significant improvements in the performance of BO. The Dragonfly library is available at dragonfly.github.io.
1. Introduction
Dragonfly is an open-source Python library designed to make Bayesian optimisation scalable and robust for modern, expensive black-box optimisation tasks. It integrates methods for challenging domains and reports better or competitive performance across synthetic benchmarks and real-world tasks.
- Motivation: Black-box optimisation repeatedly evaluates noisy, non-convex functions without gradients while seeking an optimum with as few expensive evaluations as possible.Hyperparameter tuning and model selection are prominent applications, but practical methods often rely on inefficient grid, random, or manual search.
- Motivation: Bayesian optimisation uses a posterior model and acquisition function to select future evaluations, trading computation for fewer evaluations of an expensive objective.It has been applied to policy search, industrial design, scientific experimentation, and model selection.
- Dragonfly: Dragonfly targets scalable and robust Bayesian optimisation through an open-source Python library for modern optimisation settings.Scalability is the primary focus, with robustness as a secondary focus.
- Scalability: Dragonfly incorporates methods for high-dimensional, multi-fidelity, neural-architecture, parallel, and constrained mixed-variable optimisation.Evolutionary algorithms optimise acquisitions over complex domains with different variable types and fairly general constraints.
- Robustness: Randomised Dragonfly approaches sample among acquisition choices and Bayesian-model parameters instead of relying on a single value throughout optimisation.This adds robustness with lower computational overhead than existing approaches.
- Evaluation: Dragonfly performs better or competitively across synthetic benchmarks and real-world computational-astrophysics and model-selection tasks.The authors report consistent performance across a wide array of problems.
2. A Brief Review of Gaussian Processes and Bayesian Optimisation
Gaussian processes provide Bayesian models of unknown functions, while Bayesian optimisation uses their posterior to construct an acquisition function and select the next evaluation. The review highlights GP-UCB as balancing exploitation of high posterior means with exploration of uncertain regions.
- Gaussian Processes: A Gaussian process is a random process characterised by a mean function and covariance kernel, with each function value normally distributed.Common prior kernels include squared exponential and Matérn kernels.
- Gaussian Processes: After noisy observations, the Gaussian-process posterior remains a Gaussian process with updated mean and covariance.The updates use the observation vector, kernel evaluations, Gram matrix, noise variance, and identity matrix.
- Bayesian Optimisation: Bayesian optimisation constructs an acquisition function from the posterior and evaluates its maximiser as the next point.The acquisition represents the utility of evaluating a point according to current posterior beliefs.
- Model and Acquisition Choices: Gaussian-process priors and acquisitions have multiple alternatives, including neural networks or random forests for priors and GP-UCB or Thompson sampling for acquisitions.Other acquisitions include probability of improvement, expected improvement, knowledge gradient, top-two expected improvement, and entropy-based methods.
- GP-UCB: GP-UCB combines posterior mean and uncertainty in an upper confidence bound, with β_t controlling the exploration–exploitation trade-off.The posterior mean favours regions believed valuable, while posterior standard deviation favours uncertain regions.
- Model and Acquisition Choices: Prior mean and kernel parameters are typically selected using empirical-Bayes procedures such as maximum likelihood or posterior marginalisation.These choices are discussed further in the paper’s robustness section.
3. Scaling up Bayesian Optimisation
Dragonfly scales Bayesian optimisation to challenging settings by addressing high-dimensional domains, multi-fidelity evaluations, neural architecture search, and parallel evaluations. The section describes additive models, fidelity-aware methods, architecture-specific distances and kernels, and asynchronous or batch execution.
- High-Dimensional BO: High-dimensional BO is difficult statistically and computationally, so Dragonfly uses additive function models with bounded-size groups.The additive decomposition makes regression and acquisition optimisation more tractable while remaining more expressive than searching a fixed low-dimensional subspace.
- Dragonfly Scope: Dragonfly incorporates scalable methods for high dimensions, multi-fidelity evaluations, neural architectures, complex domains, and parallel evaluations.These methods address both statistical and computational challenges in applying BO beyond conventional low-dimensional settings.
- High-Dimensional BO: With additive kernels, regret improves from exponential to linear in dimension, while acquisition optimisation requires only O(Mp+1ζ^-p) calls for ζ accuracy.The computational bound applies when the problem has M groups whose dimensionality is at most p.
- Multi-Fidelity BO: Multi-fidelity BO uses cheap approximations to discard low-value regions and reserve expensive evaluations for promising regions.BOCA selects fidelities using information gaps and posterior-variance thresholds that trade off cost against information.
- Neural Architecture Search: For neural architecture search, OTMANN measures architectural dissimilarity through optimal transport over layer computation, operation types, and structural positions.Dragonfly uses exponentiated OTMANN distances in kernels and evolutionary algorithms to optimise the acquisition function.
- Parallelisation: Dragonfly supports parallel BO by using asynchronous execution when evaluation times vary greatly and batch execution when evaluations take similar times.Asynchronous execution redeploys workers immediately after completion, whereas batch execution waits for all workers before issuing new queries.
4. Robust Bayesian Optimisation in Dragonfly
Dragonfly improves robustness by randomising over acquisition functions and GP hyperparameter-selection strategies, adapting their weights when they find better values. These combinations are intended to avoid dependence on a single potentially unsuitable choice while keeping computational overhead low.
- 4.1 Choice of Acquisition: Dragonfly adaptively samples among multiple acquisitions rather than selecting one acquisition for the entire optimisation process.The strategy initially samples acquisitions equally, then progressively favours those performing better on the problem.
- 4.1 Choice of Acquisition: Figure 3 compares individual acquisitions, the combined sampling method, and random sampling using simple regret, averaged over 10 independent runs with one-standard-error bars.Lower simple regret is better.
- 4.1 Choice of Acquisition: The acquisition portfolio includes GP-UCB, GP-EI, TS, and TTEI by default, adds Add-GP-UCB for entirely Euclidean domains, and excludes PI because it consistently underperformed.The combined approach is reported as robust across problems and competitive with the best acquisition on each problem.
- 4.2 GP Hyperparameters: GP hyperparameter selection is difficult because marginal-likelihood maximisation can overfit early, whereas posterior integration is computationally burdensome.The paper contrasts maximum likelihood (ML) with posterior sampling (PS) as alternative strategies.
- 4.2 GP Hyperparameters: ML generally works better for smooth functions, while PS tends to work better for less smooth functions; Figure 4 reports that their combination performs as well as or better than the stronger individual strategy.The paper attributes this pattern to ML’s potential overfitting on non-smooth functions and PS’s randomness preventing poor hyperparameter choices.
- 4.2 GP Hyperparameters: Dragonfly randomises between ML and PS at each iteration, updating their weights when selected strategies produce improved function values.After every ncyc evaluations, it fits a GP by ML and samples ncyc hyperparameter values from the posterior.
5. BO Implementation in Dragonfly
Dragonfly implements BO across mixed variable domains, constrained spaces, neural network architectures, and asynchronous workers. Its implementation combines domain-specific kernels and acquisition optimisers with adaptive model and decomposition choices.
- Domains and initialisation: Dragonfly supports Euclidean, integral, discrete, discrete numeric, discrete Euclidean, and neural network variable types, with Boolean domain constraints.Constraints return True exactly for points belonging to the domain, and rejection sampling checks sampled points.
- Kernels: Dragonfly uses squared exponential and Matérn kernels for numeric variables, Hamming kernels for discrete variables, and also implements OTMANN and exponential-decay kernels.Matérn-2.5 is the default for Euclidean, integral, and discrete numeric variables.
- Optimising the Acquisition: Acquisition optimisation uses DiRect or PDOO in unconstrained Euclidean spaces and an evolutionary algorithm for other cases.DiRect is used up to dimensionality d = 64; PDOO is used for larger dimensions.
- Initialisation: Initial BO evaluations use Latin hypercube sampling for Euclidean and integral variables, uniform sampling for discrete variables, and feed-forward architectures for neural network variables.The default initialisation count is ninit = 5d, capped at 7.5% of the optimisation budget.
- Asynchronous BO: Algorithm 1 maintains observations and updates acquisition and GP-hyperparameter weights when a completed evaluation exceeds the current best value.With asynchronous workers, the procedure waits for workers, incorporates their observations, selects strategies, and redeploys evaluations.
- Add-GP-UCB: For Add-GP-UCB, Dragonfly partially searches the combinatorial space of additive decompositions and uses pmax = 6 and k = 25 by default.The paper treats the additive model as a useful small-sample approximation rather than assuming the objective is truly additive.
- Multi-fidelity optimisation: In multi-fidelity settings, queries include both a fidelity z and an input x, and the product kernel can retain an additive form.The acquisition and GP-parameter-selection strategy are defined over the resulting query.
6. Experiments
Dragonfly is evaluated across synthetic benchmarks, constrained and noisy settings, astrophysical likelihood problems, and model-selection tasks. It performs competitively across these settings, with particular advantages in higher-dimensional, non-Euclidean, and multi-fidelity problems.
- Experimental setup: Dragonfly is compared with random search, evolutionary algorithms, PDOO, HyperOpt, SMAC, Spearmint, and GPyOpt across synthetic and real-world optimisation tasks.The synthetic benchmarks include Euclidean, non-Euclidean, constrained, and noisy functions; real-world tasks cover computational astrophysics and model selection.
- Synthetic benchmarks: On lower-dimensional Euclidean tasks, Dragonfly and Spearmint perform consistently well, while Spearmint is prohibitively expensive in high dimensions.Dragonfly is described as the most competitive on the higher-dimensional tasks.
- Synthetic benchmarks: Dragonfly performs consistently well on non-Euclidean domains, whereas GPyOpt and Dragonfly each perform very well on some problems but poorly on others.For constrained domains, comparisons are made against RAND and EA because the other methods do not support arbitrary constraints of the specified form.
- Synthetic benchmarks: Multi-fidelity improvements are modest in some synthetic cases because early low-fidelity spending delays highest-fidelity queries and complicates Gaussian-process estimation.The reported simple regret remains infinite until the highest fidelity is queried in those cases.
- Real-world tasks: On the Luminous Red Galaxies task, Spearmint, GPyOpt, and Dragonfly perform well; on model-selection datasets, Dragonfly can outperform NASBOT by exploiting cheaper evaluations and robust hyperparameter techniques.The Luminous Red Galaxies experiment uses nine Euclidean cosmological parameters, while the model-selection experiments include blog feedback, indoor location, and slice localisation datasets.
- Real-world tasks: For gradient-boosted regression, Bayesian optimisation using only a fraction of the training data outperforms Bayesian optimisation using the entire training set within the specified time budget.The lower-fidelity version performs more evaluations while still obtaining good predictions from 2,000 training points.
7. Conclusion
The paper integrates methods for scaling and improving the robustness of Bayesian optimisation in Dragonfly, an open source platform. It reports that this integrated approach outperforms existing Bayesian optimisation platforms across varied applications.
- Dragonfly integrates techniques for scaling Bayesian optimisation to modern large-scale applications and improving its robustness.
- The integrated platform outperforms existing Bayesian optimisation platforms across a variety of applications.
- Dragonfly is implemented as an open source platform for Bayesian optimisation.
- The authors identify multi-objective optimisation and customised probabilistic-programming models as future integration directions.