Source-linked AI summary
SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization
Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhopf, René Sass, Frank Hutter
TL;DR
Choosing effective machine-learning hyperparameters is crucial, while Bayesian Optimization is sample-efficient but sensitive to its design choices and task characteristics. SMAC3 addresses this with a flexible package offering multiple Bayesian Optimization approaches, facades, and presets across optimization settings. Its use in successful AutoML tools and a winning BBO challenge demonstrates value beyond academic research.
Problem
Bayesian Optimization can be brittle to its design choices, and different Hyperparameter Optimization tasks require different approaches.
Method
SMAC3 combines multiple Bayesian Optimization approaches, task-specific facades and presets, surrogate models, intensification, and multi-fidelity strategies for varied optimization problems.
Results
SMAC3 performs as well as Hyperband initially, best in the middle, and eventually catches up through pure Bayesian Optimization with random forests, while consistently outperforming Dragonfly.
Takeaways & Limitations
SMAC3 provides a robust, flexible tool for Hyperparameter Optimization, low-dimensional continuous black-box optimization, and algorithm configuration across problem instances.
Takeaways & Limitations
SMAC3 does not exploit task-landscape structure, and choosing its own hyperparameters can remain challenging.
Abstract
from arXiv · showhide
Algorithm parameters, in particular hyperparameters of machine learning algorithms, can substantially impact their performance. To support users in determining well-performing hyperparameter configurations for their algorithms, datasets and applications at hand, SMAC3 offers a robust and flexible framework for Bayesian Optimization, which can improve performance within a few evaluations. It offers several facades and pre-sets for typical use cases, such as optimizing hyperparameters, solving low dimensional continuous (artificial) global optimization problems and configuring algorithms to perform well across multiple problem instances. The SMAC3 package is available under a permissive BSD-license at https://github.com/automl/SMAC3.
1. Introduction
SMAC3 addresses the brittleness of Bayesian Optimization design choices by providing a flexible package for different hyperparameter-optimization tasks. Its use in AutoML tools and a winning BBO challenge illustrates value beyond academic research.
- Correctly setting machine-learning hyperparameters is crucial for achieving top performance on a dataset.
- Bayesian Optimization is commonly used for Hyperparameter Optimization because of its sample efficiency, but remains brittle to its own design choices.
- SMAC3 implements several Bayesian Optimization approaches and provides facades that hide unnecessary complexity for different Hyperparameter Optimization tasks.
- SMAC3’s successful use in auto-sklearn, Auto-PyTorch, and the winning solution to the latest BBO challenge demonstrates value beyond academic research.
2. Different Use Cases and Modes of SMAC3
SMAC3 supports multiple Bayesian-optimization use cases through facades and pre-sets tailored to continuous black-box optimization, structured HPO, multi-fidelity tasks, and algorithm configuration.
- SMAC4BB: For low-dimensional continuous spaces, SMAC4BB combines a Sobol initial design, a GP with a 5/2-Matérn kernel, and EI acquisition.
- SMAC4HPO: For CASH, SMAC3 jointly selects an algorithm and its hyperparameters, modeling conditional subspaces with random forests.
- SMAC4MF: For expensive evaluations, SMAC4MF combines Hyperband with Bayesian optimization and uses cheaper budgeted evaluations as proxies for full-cost evaluations.
- SMAC4AC: For algorithm configuration, SMAC3 targets robust performance across instances and uses aggressive racing to allocate evaluations selectively.
3. Brief Empirical Comparison
SMAC3 is compared with random search, Hyperband, Dragonfly, and BOHB on surrogate HPO, joint HPO+NAS, and NAS benchmarks.
- SMAC3’s multi-fidelity approach performs as well as Hyperband initially, performs best in the middle, and is later matched by its pure random-forest Bayesian optimization.
- SMAC3 consistently outperforms Dragonfly throughout the comparison and later also outperforms BOHB.
4. Related Work
Related work includes several Bayesian-optimization and evolutionary tools, while SMAC3 emphasizes random forests for complex spaces and flexible combinations of optimization strategies.
- Recent follow-up tools extend Bayesian optimization for hyperparameter optimization, alongside evolutionary algorithms for efficient black-box optimization.
- SMAC3’s stated advantages are random-forest surrogates for higher-dimensional complex spaces and flexible combinations of Bayesian-optimization and intensification strategies.
- Choosing the right optimizer for a task remains an open problem, motivating systems that schedule several optimizers sequentially.
5. Outlook
SMAC3 is robust on many HPO tasks but does not exploit their landscape structure, and selecting SMAC3’s own settings can remain challenging.
- SMAC3 does not exploit HPO landscape structure, motivating planned integration of local Bayesian-optimization approaches.
- Although SMAC3 provides an easy-to-use API and facades, choosing its own hyperparameters may still be challenging.
- The authors plan adaptive selection of SMAC3 settings through mechanisms such as bandits or reinforcement learning.