Source-linked AI summary
Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020
Ryan Turner, David Eriksson, Michael McCourt, Juha Kiili, Eero Laaksonen, Zhen Xu, Isabelle Guyon
TL;DR
The paper studies whether derivative-free optimizers can improve machine-learning hyperparameter tuning, addressing limited evidence from real ML problems. It reports results from a hidden-objective competition using standardized baselines and evaluation, finding strong performance for surrogate-assisted and Bayesian optimization approaches over random search.
Problem
The paper addresses limited evidence comparing black-box optimizers for machine-learning hyperparameter tuning, despite widespread use of manual, random, and grid search.
Method
The authors ran a NeurIPS 2020 competition evaluating optimizers on real ML optimization problems with hidden objectives, random search, and open-source package baselines.
Results
All top-20 participants used surrogate-assisted optimization, and 61 of 65 teams beat the random-search baseline on previously hidden final problems.
Takeaways & Limitations
The challenge demonstrates the benefits of Bayesian optimization and surrogate-assisted approaches for ML-related black-box optimization under limited evaluations.
Takeaways & Limitations
The paper notes that further analysis and ablation studies are needed to understand why ensembling works so well for Bayesian optimization.
Abstract
from arXiv · showhide
This paper presents the results and insights from the black-box optimization (BBO) challenge at NeurIPS 2020 which ran from July-October, 2020. The challenge emphasized the importance of evaluating derivative-free optimizers for tuning the hyperparameters of machine learning models. This was the first black-box optimization challenge with a machine learning emphasis. It was based on tuning (validation set) performance of standard machine learning models on real datasets. This competition has widespread impact as black-box optimization (e.g., Bayesian optimization) is relevant for hyperparameter tuning in almost every machine learning project as well as many applications outside of machine learning. The final leaderboard was determined using the optimization performance on held-out (hidden) objective functions, where the optimizers ran without human intervention. Baselines were set using the default settings of several open-source black-box optimization packages as well as random search.
1 Introduction
The paper motivates black-box optimization for expensive machine-learning hyperparameter tuning and introduces a challenge designed to compare optimizers across many ML problems. It focuses on whether Bayesian optimization and related methods outperform random search under limited evaluations.
- Black-box optimization: Black-box optimization minimizes an expensive function without derivative information, typically allowing only a few hundred evaluations.The domain is commonly a hyper-rectangle, and the setting is also called derivative-free optimization.
- Applications: Machine-learning hyperparameters commonly require black-box optimization, while Bayesian optimization is also used in signal processing and other applications.BO uses a probabilistic surrogate model that provides uncertainty, often through a Gaussian process.
- Challenge motivation: The challenge was the first to seek the best black-box optimizers specifically for machine-learning-related problems across many different problems.Its motivation included limited adoption of BO despite widespread hyperparameter tuning in NeurIPS and ICLR papers.
- Broader relevance: Black-box optimization supports applications including chemical engineering, materials discovery, manufacturing design, control systems, drug discovery, and web-interface A/B testing.The passage also notes that major cloud platforms offer hyperparameter-tuning tools.
- Alternative optimizers: Evolutionary algorithms often require thousands of evaluations to compete with more sample-efficient methods such as Bayesian optimization.This evaluation burden can be impractical when objective-function evaluations are computationally expensive.
2 Background
The paper distinguishes its ML-focused black-box optimization challenge from earlier synthetic benchmarks and AutoML competitions. Its benchmark assigns the search space and evaluates optimizers on hidden objective functions rather than giving algorithms training data.
- Related competitions: Earlier COCO and most BBComp competitions primarily used synthetic black-box functions, unlike this challenge’s real-world machine-learning tasks.The paper contrasts synthetic functions with ML problems containing irrelevant dimensions and complex noise patterns.
- AutoML distinction: AutoML algorithms choose the search space and machine-learning method, whereas black-box optimization algorithms receive a user-assigned search space.In black-box optimization, the algorithm may receive only a search-space specification and a black-box objective.
- Search-space example: The competition’s SVM example specifies C, gamma, and tol as real-valued parameters searched on logarithmic ranges.The ranges are C: 1.0–1e3, gamma: 1e-4–1e-3, and tol: 1e-5–1e-1.
- Challenge design: Unlike related competitions that allowed closed-source or human-in-the-loop submissions, this challenge required open-source submissions for prize eligibility.The authors connect this requirement to the goal of advancing the field.
3 Competition Setup
The competition evaluated derivative-free optimizers on collections of model, dataset, and loss-function problems using hidden test phases, standardized scoring, and fixed parallel compute budgets. Baselines included random search and default configurations from several open-source packages.
- Evaluation phases: The competition used local practice, feedback leaderboard, and final leaderboard problems, with the latter two hidden from participants.The final leaderboard therefore evaluated performance on previously unavailable objective functions.
- Baselines: Baselines included default configurations of Hyperopt, Nevergrad, OpenTuner, pySOT, Scikit-Optimize, and TuRBO, with random search as the principal reference point.The baselines were intended to provide participants with a starting point rather than exhaustively represent all available packages.
- Datasets: The benchmark’s practice problems used public scikit-learn datasets and were not a random split of the other problems.This limits how representative the local practice set was of feedback and final evaluation problems.
- Optimization problems: Optimization problems were constructed from combinations of datasets, machine-learning models, and evaluation metrics or loss functions.Examples include GBDT on MNIST with accuracy, logistic regression on MNIST with log loss, and an MLP on Boston housing with RMSE.
- Compute budget: Each submission had a 30-minute optimization-run budget and was evaluated through 16 iterations with batches of 8 parallel suggestions.The setup favored algorithms able to make effective parallel suggestions under wall-clock constraints.
- Scoring: Bayesmark normalized each problem so one random-search suggestion scored 1 and the global optimum scored 0 before averaging across problems.The leaderboard score was 100 × (1 − norm-mean-perf), producing an intuitive unitless scale from 0 to 100.
- Repetition and ranking: The final leaderboard used 60 problems with 30 repeated trials, and the top 20 were rerun with 100 trials for ranking.The feedback leaderboard separately used 60 problems and 10 repeated trials.
4 Learnings and Key Results
The challenge found that surrogate-assisted optimization, especially Bayesian and trust-region methods, consistently outperformed random search on held-out ML problems. Top solutions commonly used ensembles, open-source components, and sometimes meta-learning, although parameter encoding and limited analysis constrain the conclusions.
- Generalization: 61 of 65 teams beat random search on previously unseen final-leaderboard problems.Most teams’ gains persisted on hidden problems, indicating they did not simply overfit practice or feedback tasks.
- Baselines: TuRBO8 scored 88.921, ahead of pySOT, while both trust-region methods outperformed traditional Bayesian-optimization and evolutionary-algorithm defaults.The comparison concerns default package configurations rather than the packages’ absolute capabilities.
- Surrogate-assisted optimization: All top-20 participants used surrogate-assisted optimization, and most top teams combined Gaussian-process models with standard acquisition functions.The results strongly indicate that surrogate modeling and intelligent decision-making improved optimization performance.
- Comparison with random search: Participants achieved orders-of-magnitude greater search efficiency than random search, despite random search sampling uniformly in the warped search space.The comparison used the challenge’s 128-evaluation budget as the reference point.
- Ensemble methods: All top-10 methods used ensembling, combining multiple surrogates, acquisition functions, or complete optimizers to address disparate problems.The paper defines ensemble broadly and reports that teams discovered mixtures of methods were effective.
- Ensemble methods: NVIDIA’s TuRBO-plus-Scikit-Optimize ensemble scored 92.928, exceeding TuRBO’s 88.921 and Scikit-Optimize’s 88.085 individually.The ensemble selected 50% of suggestions from each component and was interpreted as potentially avoiding failed models.
- Open-source tools and parameter types: All top-20 submissions included open-source elements, while the challenge’s continuous encoding limited natural handling of integer and categorical parameters.The paper identifies this encoding choice as a scope limitation for surrogate-assisted methods.
- Trust-region methods: Six of the top-10 teams used TuRBO, suggesting trust-region optimization was effective for these relatively low-dimensional ML hyperparameter problems.The paper notes this prevalence may reflect nonsmooth function landscapes favoring local models.
5 Discussion
The discussion identifies scope boundaries in the competition and proposes extensions involving additional problem information, multi-fidelity evaluations, asynchronous parallelism, and constraints.
- The challenge addressed only one part of automated machine-learning development, motivating future competitions that cover neglected aspects.
- The competition deliberately withheld knowledge about the machine-learning problems, while future challenges could reveal model type or modeling circumstances.Such information may create opportunities for improved performance even without access to training data.
- Multi-fidelity computations were unavailable, although practical settings may offer cheaper lower-fidelity signals such as partial data or early learning curves.These signals could let optimizers reject poor configurations earlier and use computational budgets more intelligently.
- The batch-based competition emphasized parallelism, but future settings could require asynchronous suggestions conditioned on outstanding evaluations.The discussion also identifies black-box constraints and multiobjective settings as relevant extensions beyond unconstrained single-objective optimization.
6 Conclusions
The challenge established a large ML-focused benchmark for derivative-free optimization and reported strong advantages for Bayesian optimization over random search.
- 65 teams and hundreds of participants engaged with the challenge, which also became an ongoing CodaLab benchmark.
- The challenge was the first optimization competition evaluating derivative-free optimizers on machine-learning-related problems.
- Top submissions achieved over 100× sample-efficiency gains compared to random search.
- All top teams used some form of Bayesian-optimization ensemble, including strategies that alternated surrogates, acquisition functions, or complete optimizers.
- Warm starting from loosely related problems often produced large performance gains.
A Warm-starting
The warm-start leaderboard compares leading submissions with random search and shows AutoML.org ranked first after leveraging solutions from related problems with the same search space.
- AutoML.org ranked first on the top-20 warm-start leaderboard.
- Figure 3 compares top warm-start submissions with random search using the same analysis as Figure 1.
- AutoML.org’s intensive use of warm starting allowed it to pull ahead of the other methods.
- AutoML.org, DeepWisdom, dangnguyen, and Tiny, Shiny & Don made big gains by warm starting from parameter names and solutions from problems sharing the same search space.
B How scoring works
The scoring system aggregates optimizer evaluations through batchwise minima, cumulative performance, baseline clipping, normalization, and averaging across problems and trials.
- Batch performance uses the minimum function evaluation among suggestions, while cumulative minima over batches represent trial performance.
- Mean scoring uses clipped scores with a baseline because crashes can produce infinite objective values and infinite random-search loss.
- Scores are normalized by linearly rescaling performance between optimal and random-search performance, then aggregated across objective functions as a grand mean.
- The leaderboard uses 16 batches of size 8 and transforms scores onto a 0-to-100 scale.
- A score of 0 matches a single random-search guess, whereas a score of 100 always finds the best known global optimum.