PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comparative study of Hyper-Parameter Optimization Tools

Shashank Shekhar, Adesh Bansode, Asif Salim

arXiv:2201.06433v1cs.LG

TL;DR

Choosing hyper-parameters matters for model performance, while manual selection can be costly across algorithms with differing sensitivities. The paper compares four HPO libraries on two benchmarks and finds Optuna best suited to the CASH trade-off and HyperOpt to the MLP problem.

  • Problem

    Manual hyper-parameter selection can be costly because algorithms differ in their settings and sensitivity to those settings.

    Source: §I · p. 1

  • Method

    The study compares Optuna, HyperOpt, Optunity, and SMAC on two HPO benchmarks.

    Source: §VI · p. 5

  • Results

    For the CASH benchmark, Optuna is the better option considering runtime and performance trade-offs, while HyperOpt is better for the MLP problem.

    Source: §VI · p. 5

  • Takeaways & Limitations

    For industry applications balancing performance and runtime, the authors favor Optuna; they also identify Optunity as a possible choice for very large data with parallel computation.

    Source: 1) Results and Observations · p. 5

Abstract

from arXiv · show

Most of the machine learning models have associated hyper-parameters along with their parameters. While the algorithm gives the solution for parameters, its utility for model performance is highly dependent on the choice of hyperparameters. For a robust performance of a model, it is necessary to find out the right hyper-parameter combination. Hyper-parameter optimization (HPO) is a systematic process that helps in finding the right values for them. The conventional methods for this purpose are grid search and random search and both methods create issues in industrial-scale applications. Hence a set of strategies have been recently proposed based on Bayesian optimization and evolutionary algorithm principles that help in runtime issues in a production environment and robust performance. In this paper, we compare the performance of four python libraries, namely Optuna, Hyper-opt, Optunity, and sequential model-based algorithm configuration (SMAC) that has been proposed for hyper-parameter optimization. The performance of these tools is tested using two benchmarks. The first one is to solve a combined algorithm selection and hyper-parameter optimization (CASH) problem The second one is the NeurIPS black-box optimization challenge in which a multilayer perception (MLP) architecture has to be chosen from a set of related architecture constraints and hyper-parameters. The benchmarking is done with six real-world datasets. From the experiments, we found that Optuna has better performance for CASH problem and HyperOpt for MLP problem.

I. INTRODUCTION

Model performance depends not only on learned parameters but also on hyper-parameters that control the learning process. The paper compares HPO tools and formalizes optimization over a hyper-parameter space using a black-box objective.

I. INTRODUCTION

  • Hyper-parameters control learning and cannot be inferred like model parameters during fitting, so choosing them is part of achieving good model performance.Prior studies cited by the authors report that optimal hyper-parameters improve model performance [2].Source: §I · p. 1
  • Manual selection can be costly because algorithms vary in their hyper-parameters and sensitivity, motivating systematic tuning that considers time and performance.Source: §I · p. 1
  • The study compares four HPO libraries on real-world problems in performance and runtime characteristics.The tools are Optuna, HyperOpt, Optunity, and SMAC.Source: §I · p. 1

II. HYPER-PARAMETER OPTIMIZATION – THE FORMAL DEFINITION

  • The formal setup represents an algorithm by hyper-parameters x drawn from a product space λ and evaluates model performance using training and test data.The model is denoted M := Aλ, with D = {Dtrain, Dtest}.Source: §II · p. 2
  • The optimization objective is treated as a black-box function whose analytical form is unknown.Source: §II · p. 2

A. Challenges in Hyper-parameter tuning

Hyper-parameter tuning must accommodate time constraints, varied parameter types, and dependencies among settings. The section contrasts grid and random search, including their practical limitations.

A. Challenges in Hyper-parameter tuning

A. Grid Search

  • Grid search divides the parameter space into a grid and evaluates every grid combination, but is time-consuming in larger spaces and limited to tuning one model.The authors restrict its use to low-dimensional spaces such as 1-D or 2-D.Source: A. Grid Search · p. 2

B. Random Search

  • Random search samples hyper-parameters independently and can converge faster than grid search, but costly evaluations make it time-consuming.It tunes a single model and does not exploit knowledge of well-performing regions as Bayesian optimization does.Source: B. Random Search · p. 2

C. Bayesian Hyper-parameter Optimization

Bayesian optimization addresses black-box hyper-parameter objectives by using observed evaluations to guide subsequent choices. Its acquisition function balances exploration of uncertain regions with exploitation of promising ones.

1) Overview:

  • Bayesian optimization is used because the objective is a black box with unknown analytical form and derivatives and may be non-convex.Source: 1) Overview · p. 2
  • Observed configuration-performance pairs update a posterior estimate, and an acquisition function selects the next configuration to evaluate.The acquisition balances exploring uncertain regions with exploiting configurations associated with favorable objective values.Source: 1) Overview · p. 2
  • Bayesian optimization can require only a few evaluations, which is useful when evaluating the objective is expensive.Source: 1) Overview · p. 2
  • Across four iterations, Figure 1 depicts a Gaussian-process objective approximation and lower acquisition plots, with sampling guided by predicted value and uncertainty.The far-left region remains unsampled because it is predicted to offer little improvement despite high uncertainty [14]].Source: Figure 1 · p. 3
  • A Gaussian process is introduced as a distribution over functions parameterized by a mean function and a covariance function.Source: 1) Overview · p. 3

2) Surrogate functions:

The section describes Gaussian processes as a surrogate approach for estimating function values and lists covariance-function and surrogate alternatives.

2) Surrogate functions:

  • The squared exponential is presented as a common choice for the covariance function.The passage gives a covariance expression involving xi and xj, and notes β is a hyper-parameter.Source: 2) Surrogate functions · p. 3
  • Given prior observations, the Gaussian-process posterior is used to estimate the function value at a new evaluation point.The passage describes the prior function values as drawn from a normal distribution with covariance matrix K.Source: 2) Surrogate functions · p. 3
  • The alternatives listed for Gaussian processes are random forests and Tree-structured Parzen Estimators.The section also names automatic relevance determination and Matérn kernels as covariance-function choices.Source: 2) Surrogate functions · p. 3

3) Acquisition functions:

An acquisition function selects the next evaluation point by maximizing a score intended to move optimization toward a maximum. Expected improvement is one such choice.

3) Acquisition functions:

  • The next evaluation point is chosen by maximizing the acquisition function, which is intended to move optimization toward a maximum.The section names expected improvement (EI) as a popular acquisition-function choice.Source: 3) Acquisition functions · p. 3
  • Expected improvement is evaluated relative to the best observed function value, f(x+), where x+ is the best point among those already evaluated.The passage introduces an improvement function and states that the next point is found by maximizing EI.Source: 3) Acquisition functions · p. 3
  • The EI expression uses φ and Φ, identified as the probability distribution and cumulative distribution functions of the standard normal distribution.The evolutionary-algorithm discussion is omitted for brevity.Source: 3) Acquisition functions · p. 3

IV. HYPER-PARAMETER OPTIMIZATION TOOLS

The tools section introduces HyperOpt, a Python library based on sequential model-based optimization associated with Bayesian optimization. It supports configurable search spaces, objectives, and optimization algorithms.

A. HyperOpt

  • HyperOpt uses sequential model-based optimization (SMBO) associated with Bayesian optimization.The library lets users configure the variable search space, evaluation function, and loss function.Source: A. HyperOpt · p. 3
  • HyperOpt can combine algorithms, data-preprocessing modules, and their hyper-parameters into one large optimization problem.Source: A. HyperOpt · p. 3
  • HyperOpt's search domain can use continuous, ordinal, or categorical variables, and its objective can be a user-defined Python function returning a loss.Available search algorithms are random search, TPE, and adaptive TPE; parallel implementation can use Apache Spark and MongoDB.Source: A. HyperOpt · p. 3

B. Optuna

Optuna is described through its dynamic search-space API, sampling and pruning features, and distributed-computing support. Optunity offers several solvers, with particle swarm optimization used in this paper's experiments.

B. Optuna

  • Optuna's define-by-run API constructs search spaces dynamically, unlike HyperOpt's explicit per-parameter search-space specification.Its objective function receives a trial object embedding the parameter space and function to optimize.Source: B. Optuna · p. 4
  • Optuna monitors intermediate objective values and can terminate runs that fail predefined conditions through asynchronous successive halving.The passage also notes distributed-computing capabilities.Source: B. Optuna · p. 4

C. Optunity

  • Optunity offers basic, grid, random, and evolutionary solvers; the experiments use particle swarm optimization (PSO).Its features include configurable evaluation limits and box constraints for hyper-parameters [5].Source: C. Optunity · p. 4

D. Sequential model based algorithm configuration (SMAC)

SMAC combines sampling and runtime prediction to select configurations in large mixed numerical and categorical spaces. The experiments use six real-world datasets and evaluate classifiers with three-fold cross-validation and F1-score.

D. Sequential model based algorithm configuration (SMAC)

  • SMAC models algorithm runtime to reduce evaluation runs and selects promising configurations in large mixed numerical/categorical spaces.It also supports categorical parameters and uses performance across multiple instances in its sampling mechanism.Source: D. Sequential model based algorithm… · p. 4

V. EXPERIMENTS

  • The experiments section covers datasets, system setup, implementation details, and classification-task metrics.The classifier configuration space is summarized in Table II by categorical, continuous, and total hyperparameter counts.Source: §V · p. 4, Table II · p. 4

A. Datasets

  • The experiments use six publicly available OpenML datasets chosen to cover probable real-world scenarios.Dataset details are listed in Table I.Source: A. Datasets · p. 4, Table I · p. 4

B. System setup and Implementation

  • The classification experiments use three-fold cross-validation, F1-score, and 50 evaluation iterations.The classifiers were built in Python 3 on a 3.50 GHz Intel Xeon i7-4771 system with 32 GB RAM.Source: B. System setup and Implementation · p. 4

C. Benchmark-I

Across the experiments, the authors favor Optuna when balancing performance and runtime, while noting dataset-specific trade-offs. The section also describes the CASH configuration space and the MLP architecture task used in the second benchmark.

C. Benchmark-I

  • The CASH algorithm’s configuration space contains 12 classifiers and 58 total hyper-parameters; separately, a NeurIPS black-box optimization challenge chose an MLP architecture by optimizing over hyper-parameters.Source: C. Benchmark-I · p. 4, D. Benchmark-II · p. 5

1) Results and Observations:

  • Optuna runs faster on all datasets except semeion, but scores vary: Optunity PSO scores highly on electricity and gas drift and lower on semeion.The electricity and gas drift datasets have high sample size and moderately high dimensionality; Optunity PSO takes longer than Optuna there, while Optuna's semeion score comes with a slight runtime increase over Optunity PSO.Source: 1) Results and Observations · pp. 4–5
  • HyperOpt and SMAC have similar performance scores across datasets, with SMAC taking longer on most; Optuna and HyperOpt scores are similar, while Optuna is usually faster except on semeion.The runtime difference between Optuna and HyperOpt is described as not significant.Source: 1) Results and Observations · p. 5
  • TPE performs slightly better than random search in the comparison of HyperOpt and Optuna searches.Source: 1) Results and Observations · p. 5
  • Considering performance and runtime together, the experiments identify Optuna as a better HPO choice overall, while retaining Optunity as a possible choice for very large data.The authors note that Optunity runtime could be further optimized through parallel computation.Source: 1) Results and Observations · p. 5

1) Results and Observations:

Across two benchmarks, Optuna is the better option for CASH, while HyperOpt is better for the MLP problem. The results also distinguish score, runtime, and iteration-to-iteration stability.

1) Results and Observations:

  • Optuna has the highest performance score across datasets, while HyperOpt has the best runtime; SMAC matches Optuna on some datasets but runs comparatively longer.The score difference between Optuna and HyperOpt is not considerable except on gas drift and nomao.Source: 1) Results and Observations · p. 5
  • Optunity PSO has the lowest performance score, with runtime between Optuna and HyperOpt and better than SMAC.The experiments identify HyperOpt as the better option for neural network applications.Source: 1) Results and Observations · p. 5
  • Optunity and HyperOpt show oscillatory F1-score behavior across iterations, while Optuna is more stable.The plots cover the best-performing model for each dataset in benchmarks I and II.Source: 1) Results and Observations · p. 5

VI. CONCLUSION

  • For CASH, Optuna offers the better trade-off between runtime and performance; for the MLP problem, HyperOpt is the better option.The paper compares HyperOpt, Optuna, Optunity, and SMAC on both benchmarks.Source: §VI · p. 5
  • Table IV lists the MLP tuning configuration space by parameter type, search-space nature, and value range.The passage identifies these as the table's columns but does not provide their values.Source: Table IV · p. 5

Related papers

Loading 2201.06433v1…