PaperlayerView exact revision on arXiv

Source-linked AI summary

HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO

Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik, Matthias Feurer, René Sass, Aaron Klein, Noor Awad, Marius Lindauer, Frank Hutter

arXiv:2109.06716v3cs.LG

TL;DR

HPO research lacks realistic, diverse, affordable, and standardized benchmarks, particularly for multi-fidelity methods. HPOBench addresses this gap with a broad, containerized suite supporting reproducible evaluations, and its study demonstrates compatibility with many optimizers and tools.

  • Problem

    HPO research lacks realistic, diverse, computationally cheap, and standardized benchmarks, especially for multi-fidelity HPO, complicating reproducible comparisons.

  • Method

    HPOBench packages more than 100 multi-fidelity problems across 12 benchmark families with unified interfaces, containers, and raw, surrogate, and tabular versions.

  • Results

    The authors evaluate 13 optimizers from 6 optimization tools and report that advanced HPO methods outperform random search and Hyperband, while multi-fidelity extensions improve over black-box versions.

  • Takeaways & Limitations

    HPOBench provides a reproducible, extensible basis for developing and comparing multi-fidelity HPO methods.

  • Takeaways & Limitations

    HPOBench currently focuses on stateless benchmarks started in a single container and does not immediately guarantee replicability of optimization results across seeds and hardware.

Abstract

from arXiv · show

To achieve peak predictive performance, hyperparameter optimization (HPO) is a crucial component of machine learning and its applications. Over the last years, the number of efficient algorithms and tools for HPO grew substantially. At the same time, the community is still lacking realistic, diverse, computationally cheap, and standardized benchmarks. This is especially the case for multi-fidelity HPO methods. To close this gap, we propose HPOBench, which includes 7 existing and 5 new benchmark families, with a total of more than 100 multi-fidelity benchmark problems. HPOBench allows to run this extendable set of multi-fidelity HPO benchmarks in a reproducible way by isolating and packaging the individual benchmarks in containers. It also provides surrogate and tabular benchmarks for computationally affordable yet statistically sound evaluations. To demonstrate HPOBench's broad compatibility with various optimization tools, as well as its usefulness, we conduct an exemplary large-scale study evaluating 13 optimizers from 6 optimization tools. We provide HPOBench here: https://github.com/automl/HPOBench.

1 Introduction

HPOBench addresses the lack of realistic, diverse, reproducible, and computationally affordable multi-fidelity HPO benchmarks by providing a unified, containerized benchmark suite. It supports broad optimizer evaluation through raw, surrogate, and tabular benchmarks.

  • Containerized benchmarks preserve dependencies to improve longevity, maintainability, and extensibility despite evolving software ecosystems.
  • Raw and tabular benchmark versions support computationally efficient yet statistically sound large-scale evaluations.
  • HPOBench provides 12 benchmark families containing more than 100 multi-fidelity HPO problems under a unified interface, including traditional HPO and NAS.
  • HPOBench supports multi-objective optimization and transfer-HPO across datasets, including combinations with multiple fidelities.
  • The paper demonstrates HPOBench with 13 optimizers from 6 optimization tools to assess advanced methods and multi-fidelity HPO.

2 Background on Hyperparameter Optimization

The paper frames HPO as black-box optimization over hyperparameter configurations and describes multi-fidelity methods as using cheaper evaluations at lower resource levels. It positions Hyperband and hybrid methods as important examples of this approach.

  • 2.1 Black-box Hyperparameter Optimization: BBO searches for a minimum of a black-box objective using queried configurations without assuming gradients, smoothness, convexity, or noise properties.
  • 2.1 Black-box Hyperparameter Optimization: BBO methods span exploratory search, evolutionary algorithms, and model-based approaches such as Bayesian optimization.
  • 2.2 Multi-fidelity Hyperparameter Optimization: Multi-fidelity optimization allows cheaper queries at lower fidelities, including dataset subsets, feature subsets, and fewer training epochs.
  • 2.2 Multi-fidelity Hyperparameter Optimization: Hyperband randomly samples configurations and repeatedly applies successive halving to allocate more resources to promising configurations.
  • 2.2 Multi-fidelity Hyperparameter Optimization: Transfer-HPO across datasets is another approach for accelerating HPO and can be evaluated across datasets within HPOBench benchmark families.

3 Related Work

Prior work shows that shared benchmark suites can guide optimization research, but HPOBench focuses on supplying a flexible, concrete environment for reproducible development and comparison of HPO methods.

  • Benchmarking HPO is difficult because implementation choices, tuning practices, and uncontrolled variance can produce misleading comparisons and hinder reproducibility.
  • HPOBench aims to simplify HPO development and improve reproducibility and comparability by providing concrete benchmarks.
  • Existing efforts provide optimization benchmarks for areas including algorithm configuration, continuous optimization, Bayesian optimization, experiment planning, and transfer-HPO.
  • Unlike competitions, HPOBench emphasizes a flexible benchmarking environment rather than fixed experimentation protocols.

4 HPOBench: A Benchmark Suite for Multi-Fidelity Hyperparameter Optimization benchmarks

HPOBench is a unified framework for evaluating HPO, especially multi-fidelity methods, while addressing benchmark standardization, efficiency, reproducibility, and flexibility. It combines raw, tabular, and surrogate benchmarks with containerized execution and diverse new benchmark families.

  • 4 HPOBench: A Benchmark Suite for Multi-Fidelity Hyperparameter Optimization benchmarks: HPOBench defines benchmarks through a common interface for hyperparameter spaces, fidelity dimensions, and loss functions, standardizing key evaluation components.Benchmarks specify relevant settings, dependencies, datasets, splits, and loss computation.
  • 4.2 Efficiency: Tabular and surrogate benchmarks make HPO evaluation cheaper: tabular versions use recorded function values, while surrogates predict values over the original hyperparameter space.Tabular benchmarks discretize the hyperparameter space; surrogate benchmarks preserve it but depend on model and training-data quality.
  • 4.3 Reproducibility: Containerizing each benchmark and its dependencies isolates environments, mitigating dependency conflicts and supporting long-term benchmark use.HPOBench uses Singularity containers to separate benchmarks from one another and from the host system.
  • 4.4 Flexibility: HPOBench includes 22 existing multi-fidelity benchmarks from 7 families and 88 new benchmarks from 5 families, supporting a broad range of HPO studies.The framework targets validation and development of HPO algorithms, with emphasis on multi-fidelity methods.
  • 4.4.1 Existing Community Benchmarks: Existing community benchmarks remain limited for transfer HPO because families contain only 1–6 benchmarks and provide a single fidelity dimension and metric.HPOBench augments these benchmarks with five new families to broaden their applicability.
  • 4.4.2 New Benchmarks: The new benchmark families expand coverage with up to 20 datasets per family, multiple metrics, and multiple fidelity dimensions, enabling transfer-HPO and multifidelity research beyond single-fidelity settings.Four new families allow up to two fidelity dimensions, while the MLP family reaches hyperparameter dimensionality up to 5.

5 Experiments

The study evaluates 13 optimizers across 12 benchmark families to compare advanced methods with random or Hyperband baselines and to test multi-fidelity benefits under fixed time budgets. Advanced methods generally outperform their baselines, while multi-fidelity methods are strongest at small budgets and black-box methods can catch up with sufficient time.

  • Experimental Setup: The experiments use 32 seeded repetitions per benchmark and optimizer, with trajectories based on best-seen values under fidelity-aware evaluation costs.New multi-metric benchmarks minimize 1−accuracy, and the study uses tabular versions for those benchmarks to avoid unnecessary computation and CO2 emissions.
  • RQ1: Do advanced methods improve over random search?: HEBO substantially improves over all other black-box methods, while some methods that rank poorly overall perform well on individual benchmarks.The rank-over-time analysis covers existing community and new benchmarks, and reports DF6 as an example of strong per-benchmark performance despite weaker aggregate ranking.
  • RQ1: Do advanced methods improve over random search?: Four of five black-box methods significantly outperform random search, whereas only two of four multi-fidelity methods significantly outperform Hyperband.The comparison uses wins, ties, losses, and sign tests on existing community benchmarks; new benchmarks show similar results.
  • RQ2: Do multi-fidelity methods improve over black-box methods?: Hyperband is initially competitive, then advanced multi-fidelity methods and later black-box methods overtake it; this pattern is weaker on new benchmarks with tighter time limits.The multi-fidelity advantage is more pronounced early, while black-box methods catch up later under longer budgets.
  • RQ2: Do multi-fidelity methods improve over black-box methods?: Multi-fidelity optimizers outperform black-box optimizers for relatively small compute budgets, but black-box methods become competitive given enough budget.At the full budget, advanced multi-fidelity methods are no longer statistically better than their black-box counterparts, although they remain competitive by wins and losses.
  • Conclusion: DEHB and SMAC-HB work reliably across the benchmark collection, but other multi-fidelity optimizers do not consistently improve over Hyperband.The conclusion also notes that black-box methods can outperform multi-fidelity methods at very high budgets and that random search can win on individual benchmarks.

6 Discussion and Future Work

HPOBench provides a unified, containerized library for multi-fidelity HPO benchmarks and supports rapid prototyping and community contributions. Its evaluation demonstrates compatibility with diverse tools and advantages for advanced and multi-fidelity methods, while identifying extensions for future work.

  • HPOBench provides a unified API and containerized multi-fidelity benchmarks to support rapid prototyping and method development.
  • HPOBench is open-source, and the authors invite contributions to keep the library current and evolving.
  • Future work includes connecting HPOBench with other frameworks, containerizing optimizers, and supporting stateful or parallel benchmarks.
  • The benchmark set would benefit from providing raw, tabular, and surrogate versions for every benchmark and automatically generating derived versions.
  • A large-scale study evaluated 13 algorithm implementations and found advanced HPO methods preferable to random search and hyperband baselines.
  • Multi-fidelity extensions of popular optimizers improved over their black-box versions in the reported study.

B Benchmarking efforts

Prior benchmarking efforts span global optimization, algorithm configuration, reinforcement learning, continuous optimization, Bayesian optimization, supervised classification, and experiment planning. The cited libraries provide established domain-specific comparison resources.

  • These resources represent long-running or published efforts to compare methods across related optimization and machine-learning domains.
  • Existing benchmarking libraries cover global optimization, algorithm configuration, reinforcement learning, continuous optimization, Bayesian optimization, supervised classification, and experiment planning.
  • The surveyed efforts include HPOLib, ACLib, OpenAI Gym, COCO, Bayesmark, OpenML suites, Olympus, and HPO-B.

C Benchmarking competitions

Benchmarking competitions provide additional venues for evaluating HPO and related optimization methods. The paper also documents dependency conflicts encountered while collecting existing benchmarks, illustrating maintenance challenges.

  • HPO-related competitions include the AutoML and AutoDL challenges, the NeurIPS 2020 black-box optimization challenge, and the KDD Cup.
  • Related competition venues also include the Challenges in Machine Learning workshop series and the BBOB workshop series.
  • Benchmark collection encountered incompatible dependencies, changed APIs, altered interfaces, and missing complete dependency lists in existing community benchmarks.
  • Examples include Net becoming incompatible with a newer scikit-learn version, Cartpole failing after a TensorForce API change, and NB201 changing its interface and underlying data.

D.2 Existing Community Benchmarks

HPOBench includes diverse existing community benchmarks spanning reinforcement learning, Bayesian neural networks, neural networks, joint NAS-HPO, and tabular NAS. The benchmarks expose varied configuration and fidelity spaces across raw, surrogate, and lookup-based formats.

  • The existing benchmarks include Cartpole, BNN, Net, NBHPO, NB101, NB1Shot1, and NB201, covering HPO, NAS, and joint NAS-HPO tasks.
  • Cartpole uses repetitions as fidelity, BNN uses MCMC sampling steps, and Net uses neural-network training epochs.
  • Net is a surrogate benchmark trained from 10K randomly sampled configurations across six OpenML datasets.
  • NB101 offers nearly 423k DAG-based architectures, while NB1Shot1 provides three search spaces of around 6k, 29k, and 300k architectures.
  • NB201 uses a four-node fixed-cell DAG space of around 15k architectures and lookup results for Cifar-10, Cifar-100, and ImageNet16-120.
  • The new benchmark families cover SVM, logistic regression, XGBoost, random forest, and MLP models with raw and tabular variants evaluated across AutoML datasets.

E Details on Hardware Used for Experiments

Experiments primarily used Intel Xeon Gold 6242 CPUs, with one CPU and up to 6GB RAM per job unless benchmarks required more memory.

  • The benchmark study used Intel(R) Xeon(R) Gold 6242 CPUs @ 2.80GHz on a compute cluster.
  • NB101, NB1Shot1, and NB201 could use up to 12GB RAM, while tabular-data collection used Intel(R) Broadwell E5-2630v4 CPUs @ 2.2GHz.

F Details on Runtime

The appendix reports computational cost, runtime exceptions, optimizer configurations, and the implementation details needed to interpret the benchmark study.

  • Runtime: Tabular and surrogate benchmarks reduce the estimated evaluation cost from more than 1500 CPU years to 22.5 CPU years.
  • Runtime: Most optimizers spent less than two hours of simulated optimization budget, while BOGP and DF often ran until the 4-day budget.
  • Runtime: DF evaluated only 90 NB1Shot1 configurations because it stopped after its initial design when it could not construct a model.
  • Optimizers: The study used default settings for the optimization tools and included random-search, Hyperband, Bayesian, evolutionary, and multi-fidelity optimizers.
  • Multi-fidelity optimizers: BOHB combines Bayesian optimization with Hyperband by modeling fidelities independently and using the highest-fidelity model available.
  • Multi-fidelity optimizers: DEHB maintains a separate differential-evolution subpopulation per fidelity and transfers good lower-fidelity configurations through its mutation strategy.

H More Results

The appendix expands the empirical study with final-performance tables, statistical tests, runtime summaries, and rank- and performance-over-time figures across benchmark families.

  • Final results: The appendix reports final performance for black-box and multi-fidelity optimizers across existing community benchmarks and five new model-specific benchmark families.
  • Statistical tests: Tables 22 and 23 provide sign-test p-values for advanced methods versus RS or HB and for multi-fidelity methods versus black-box counterparts.
  • Rank over time: Figure 5, Figure 6, and Figure 7 show median rank over time for all, black-box, and multi-fidelity optimizers across benchmark families.
  • Performance over time: Figure 8, Figure 9, and Figure 10 show median performance over time for all, black-box, and multi-fidelity optimizers.
Loading 2109.06716v3…