PaperlayerView exact revision on arXiv

Source-linked AI summary

BOHB: Robust and Efficient Hyperparameter Optimization at Scale

Stefan Falkner, Aaron Klein, Frank Hutter

arXiv:1807.01774v1cs.LGstat.ML

TL;DR

Expensive training and sensitivity to many hyperparameters make vanilla Bayesian optimization difficult, while random-search bandit methods lack guidance for fast convergence. BOHB combines Bayesian optimization with Hyperband’s multi-fidelity resource allocation, and the paper reports robust, scalable performance across diverse benchmarks, outperforming both approaches overall.

  • Problem

    Modern models are costly to train and highly sensitive to hyperparameter settings, while Bayesian optimization is computationally expensive and random-search bandit methods lack guidance toward the best configurations.

  • Method

    BOHB combines a robust Bayesian optimization variant with Hyperband-style multi-fidelity successive halving and parallel resource allocation.

  • Results

    BOHB consistently outperformed Bayesian optimization and Hyperband across a diverse set of benchmarks, with strong anytime and final performance.

  • Takeaways & Limitations

    BOHB is presented as a robust, flexible, scalable, conceptually simple, and easy-to-use method for practical hyperparameter optimization.

  • Takeaways & Limitations

    BOHB can be slower than random search by a bounded factor when lower fidelities are misleading, although it remains guaranteed to converge eventually.

Abstract

from arXiv · show

Modern deep learning methods are very sensitive to many hyperparameters, and, due to the long training times of state-of-the-art models, vanilla Bayesian hyperparameter optimization is typically computationally infeasible. On the other hand, bandit-based configuration evaluation approaches based on random search lack guidance and do not converge to the best configurations as quickly. Here, we propose to combine the benefits of both Bayesian optimization and bandit-based methods, in order to achieve the best of both worlds: strong anytime performance and fast convergence to optimal configurations. We propose a new practical state-of-the-art hyperparameter optimization method, which consistently outperforms both Bayesian optimization and Hyperband on a wide range of problem types, including high-dimensional toy functions, support vector machines, feed-forward neural networks, Bayesian neural networks, deep reinforcement learning, and convolutional neural networks. Our method is robust and versatile, while at the same time being conceptually simple and easy to implement.

1. Introduction

Modern machine-learning models require many costly-to-tune hyperparameters, motivating HPO methods that perform well quickly, eventually find strong configurations, use parallel resources, and scale. The paper combines Bayesian optimization and Hyperband to target these desiderata.

  • Modern deep-learning models are increasingly expensive to train while depending strongly on many correctly set hyperparameters.
  • Practical HPO must provide strong anytime performance under small budgets and strong final performance under larger budgets.It should also use parallel resources effectively and scale to many hyperparameters.
  • BOHB combines Hyperband with a robust Bayesian optimization variant to achieve fast initial progress and rapid convergence to the global optimum.Figure 1 illustrates this behavior for optimizing six neural-network hyperparameters.

2. Related Work on Model-based Hyperparamter Optimization

Related HPO methods offer useful capabilities but face trade-offs in scalability, flexibility, guidance, or convergence. Bayesian optimization provides model-based guidance, whereas Hyperband efficiently allocates multiple budgets but samples configurations randomly.

  • Bayesian optimization has been used successfully to tune neural-network architectures and hyperparameters across several datasets and challenges.
  • Gaussian-process Bayesian optimization can struggle with high-dimensional spaces, complex configuration spaces, cubic data-point complexity, and carefully specified hyperpriors.
  • Multi-fidelity Bayesian optimization methods accelerate HPO by exploiting cheaper objective evaluations or correlations between tasks and data subsets.
  • Hyperband dynamically allocates resources through successive halving, providing strong anytime performance and scalability while relying on random configuration sampling.Because it does not learn from previous configurations, it can have worse final performance than model-based approaches.
  • Concurrent methods also combined Bayesian optimization with Hyperband, but the cited approaches did not achieve BOHB’s consistent large speedups or avoid Gaussian-process limitations.

3. Bayesian Optimization and Hyperband

The paper formulates HPO as noisy optimization and introduces Bayesian optimization and Hyperband as complementary components. Bayesian optimization models observed data to select promising configurations, while Hyperband evaluates many random configurations at different budgets and progressively retains the best.

  • HPO seeks a hyperparameter configuration x⋆ minimizing validation performance f(x) over a space containing discrete and continuous dimensions.
  • Because machine-learning training is intrinsically random, the observed objective is modeled as noisy rather than directly observed.
  • Bayesian optimization fits a probabilistic model, maximizes an acquisition function, evaluates the selected configuration, and refits using the augmented data.
  • The Tree Parzen Estimator models densities over configurations, selects candidates by maximizing l(x)/g(x), and supports mixed spaces with linear data-point scaling.
  • Cheap approximate objectives parameterized by budget b become the full objective at bmax and typically improve in quality as b increases.Experiments encode budgets using iterations, data points, MCMC steps, or reinforcement-learning trials.
  • Hyperband repeatedly applies successive halving to randomly sampled configurations, aggressively testing many configurations cheaply and allocating larger budgets to top performers.It ranks configurations at each budget and continues the best-performing η^-1 fraction on a budget η times larger.

4. Model-Based Hyperband

BOHB combines Hyperband’s resource allocation and successive halving with model-based configuration selection, aiming to retain strong anytime performance while improving final performance. Its design emphasizes scalability, robustness, flexibility, simplicity, computational efficiency, and effective parallelization.

  • BOHB uses Hyperband to determine evaluation budgets and successive halving to evaluate and eliminate configurations, while replacing random sampling with model-based search.
  • The Bayesian-optimization component uses a single multidimensional KDE, unlike TPE’s hierarchy of one-dimensional KDEs, to better handle interaction effects.
  • BOHB increasingly relies on the largest budget with enough observations, reducing the risk of conclusions from lower-fidelity evaluations.
  • The method samples from a bandwidth-expanded KDE around promising configurations to encourage exploration and improve late-stage convergence.
  • A constant fraction ρ of configurations is sampled uniformly at random, preserving global exploration and Hyperband’s theoretical guarantees.
  • BOHB parallelizes successive-halving runs through one shared worker pool, sharing observations and models across runs while prioritizing smaller-budget waiting runs.
  • With two and four workers, speedup is close to linear; for one versus 32 workers, reaching regret 10^-2 takes approximately 2000s versus 130s, a speedup of about 15.

5. Experiments

Across toy, surrogate, Bayesian neural-network, reinforcement-learning, and convolutional-network benchmarks, BOHB combines Hyperband-style budget allocation with model-based configuration selection. It generally matches strong early performance while improving convergence or final performance, with a high-dimensional exception where TPE and SMAC eventually outperform it.

  • Experimental scope: BOHB was evaluated across high-dimensional toy functions, support vector machines, feed-forward and Bayesian neural networks, reinforcement learning, and convolutional neural networks.The experiments used surrogate benchmarks for several tasks and included a CIFAR-10 convolutional-network study.
  • Artificial Toy Function: Counting Ones: In the 16-dimensional counting-ones benchmark, BOHB matched Hyperband early, then quickly improved and was the only method to converge within the time budget.Random search performed poorly, while SMAC and TPE initially dominated it; at 64 dimensions, TPE and SMAC eventually performed better than BOHB.
  • Support Vector Machine on MNIST: On the two-dimensional SVM-on-MNIST surrogate, BOHB performed similarly to Fabolas and slightly better than Hyperband.The benchmark used the SVM regularization parameter C and RBF-kernel parameter γ, with training-data size as the budget.
  • Feed-Forward Neural Networks on OpenML Datasets: On feed-forward neural-network surrogates, BOHB achieved Hyperband’s final performance 100 times faster and ultimately surpassed the other Bayesian-optimization methods.Hyperband initially outperformed vanilla Bayesian optimization and was roughly three times faster than random search; BOHB also consistently outperformed HB-LCNet.
  • Bayesian Neural Networks: For Bayesian neural networks, BOHB converged faster than Hyperband and TPE and found a better configuration than both baselines on the Boston housing dataset.This experiment optimized five hyperparameters of a two-layer Bayesian neural network trained with SGHMC.

6. Conclusions

BOHB is presented as a robust, flexible, scalable HPO method that achieves strong anytime and final performance across diverse benchmarks.

  • BOHB combines robustness, flexibility, and scalability to both high-dimensional spaces and parallel resources.
  • BOHB achieves both strong anytime performance and strong final performance.
  • The method outperformed a wide range of other state-of-the-art approaches across diverse benchmarks.
  • An open-source implementation is provided to support use on new problems.

Supplementary material for: BO-HB: Robust and Efficient Hyperparameter Optimization at Scale

The supplementary material describes SuccessiveHalving, BOHB’s model-based evaluation strategy, implementation choices, and comparisons with related Hyperband–Bayesian optimization combinations.

  • Reproducibility: The authors provide open-source BOHB and Hyperband code, along with benchmarks and scripts used to produce the paper’s data.The implementation is hosted at HpBandSter, with benchmark materials in the icml_2018 branch.
  • Related methods: Bertrand et al. combine Hyperband with Gaussian-process modeling, treating budget as an ordinary search-space dimension.The authors expect squared-exponential kernels may extrapolate poorly in this setting.
  • Related methods: Wang et al. rebuild a TPE model at the start of every SuccessiveHalving run, so later iterations do not use evaluations from earlier iterations.BOHB instead retains evaluations across iterations for its budget-specific models.
  • Method: BOHB aggregates evaluations across budgets and models future evaluations using the largest budget with enough observations.The authors identify this aggregation as a crucial part of BOHB’s strong performance.
  • Method: SuccessiveHalving evaluates configurations on increasing budgets while retaining only the best 1/η after each stage.It starts with the smallest budget and progressively allocates more resources to promising configurations.
  • Implementation: The implementation uses factorized multivariate KDEs with Gaussian kernels for continuous parameters and Aitchison-Aitken kernels for categorical parameters.Scott’s rule is used for bandwidth estimation.

E. Performance of all methods on all

Across surrogate benchmarks, BOHB and Hyperband improve quickly by exploiting smaller budgets, while BOHB begins outperforming Hyperband after only a few iterations.

  • Overall performance: BOHB starts outperforming Hyperband after the first couple of iterations on all surrogate benchmarks, sometimes earlier.The letter dataset is cited as an example where this happens especially quickly.
  • Overall performance: Hyperband, HB-LCNet, and BOHB improve more quickly than random search because they use smaller budgets.Random search is described as the worst optimizer once model-based methods can leverage their models.
  • Overall performance: Traditional Bayesian optimization can retain an advantage at very large budgets on some surrogate benchmarks.The letter dataset illustrates this behavior.

F. Performance of parallel runs

Parallel BOHB achieves nearly linear speedups with small worker counts, while larger pools evaluate more random configurations before model construction and can reduce performance.

  • Parallel performance: BOHB speedups are almost linear with 2-8 workers across the surrogate benchmarks.The authors describe the speedups as quite consistent across datasets.
  • Parallel performance: Even with 32 workers, linear speedups can be achieved for some targets, including reaching regret 2 × 10−3 on letter.This is presented as an example rather than a universal result across datasets.
  • Parallel performance: With more workers, additional random configurations are evaluated before the first model is built, degrading performance.This explains why scaling becomes less favorable as the worker count increases.
  • Experimental setup: Parallel-performance experiments simulated real-time evaluations with workers waiting for each surrogate budget before returning results.The poker dataset was not evaluated with fewer than 16 workers because each such run would have taken more than a day.

G. Evaluating the hyperparameters of BOHB

BOHB’s sensitivity is evaluated across acquisition-sampling, random-search, η, and bandwidth settings. The experiments also emphasize that minimum and maximum budgets are problem-specific inputs shared with Hyperband.

  • The study varies BOHB’s acquisition-function sample count, random fraction ρ, scaling parameter η, and bandwidth factor.
  • η = 3 is used for both HB and BOHB because BOHB is relatively insensitive to η within a reasonable range.
  • Minimum and maximum budgets are problem-specific inputs to both HB and BOHB, with the minimum budget needing to provide information about larger-budget quality.

H. Counting Ones

The counting-ones experiment evaluates BOHB and competing optimizers across dimensions under a fixed budget. BOHB converges faster than the model-based baselines in lower dimensions but struggles in the very high-dimensional case.

  • The experiment compares applicable methods in dimensions d = 8, 16, 32, and 64 using a budget of 8192 full function evaluations.
  • RS and HB perform poorly because random configuration sampling is ineffective in this problem space.
  • SMAC and TPE substantially outperform RS and HB, especially with larger budgets, but require more samples than BOHB to converge to the true optimum.
  • BOHB struggles in very high dimensions because higher noise requires larger budgets to build a good model, creating overhead from small-budget evaluations.
  • Because the problem is perfectly separable, the authors expect TPE’s univariate KDE to outperform BOHB’s multivariate KDE.

I.1. Constructing the Surrogates

The surrogate benchmarks approximate classification error and training time from random-forest models trained on extensive configuration evaluations. They enable repeated, inexpensive optimizer comparisons but omit several properties of real training benchmarks.

  • Constructing the Surrogates: For each dataset, 10 000 random configurations are trained for 50 epochs, recording classification error after each epoch and total training time.
  • Constructing the Surrogates: Two independent random forests predict classification error and training time from hyperparameter configurations, enabling classification-error prediction as a function of time.
  • Constructing the Surrogates: The surrogates make benchmarking inexpensive enough to run each algorithm 256 times, while differential evolution searches for the true surrogate optimum over 10 000 iterations.
  • Limitations: The surrogate results report validation performance without test performance, so they provide no test-based indication of overfitting.
  • Limitations: Unlike stochastic-gradient-descent training, the surrogates do not represent evaluation-to-evaluation performance noise, potentially making them easier to optimize than the real benchmark.
  • Limitations: The surrogate minimizer may differ from the real benchmark’s minimizer because deterministic surrogate data can contain small classification-error fluctuations.
  • Additional Benchmarks: The benchmark includes Bayesian neural-network regression on Boston Housing and Protein Structure, using MCMC steps as the budget and validation log-likelihood after 100 sampled parameter vectors.
Loading 1807.01774v1…