Sherpa: Robust Hyperparameter Optimization for Machine Learning

1. Motivation and significance

• Manual trial and error can consume days or weeks and depends on researchers interpreting results and choosing settings, motivating automation.Hyperparameters cannot be directly learned from training data and are needed for optimal model performance.Source: §1 · pp. 1–2
• Most Bayesian optimization packages provide standalone implementations but do not provide infrastructure for parallel training.The paper groups existing tools into Bayesian optimization, bandit and evolutionary algorithms, framework-specific libraries, and all-round software.Source: §1 · p. 2
• Framework-specific libraries ease setup through tight integration but can limit researchers who later need a different machine-learning framework.Source: §1 · p. 2

2.1. Hyperparameter Optimization

• Automated optimization combines allowed hyperparameter ranges and prior results with an algorithm that suggests a new trial for model training.Each training run yields an objective value that is added to the results for subsequent suggestions.Source: §2.1 · p. 3

2.2. Components

• A Sherpa Study combines parameter objects with an algorithm and stores results, while each suggested run is represented by a Trial object.Figure 1 depicts the Study class.Source: §2.2 · p. 4, Figure 1 · p. 4

2.3. API Mode

• In API mode, users request a trial, train a model with its parameters, submit its objective, and finalize it; this mode cannot evaluate trials in parallel.Invalid observations are automatically excluded from results.Source: §2.3 · p. 4

2.4. Parallel Mode

• Parallel mode uses a server script to define ranges, algorithm, scheduler, and ML command, while client scripts obtain trials and submit objectives.Sherpa schedules jobs when resources are available, updates results from a database, and finalizes completed trials; if a script submits no objective, it continues to the next trial.Source: §2.4 · p. 5, Figure 2 · p. 5

3.1. Available Hyperparameter Types

• Sherpa supports float, integer, unordered categorical, and ordered categorical hyperparameter types, with ranges specified as bounds or categories.Source: §3.1 · p. 6

3.2. Diversity of Algorithms

• Its algorithm options span uniform random and grid search, Bayesian optimization, early-stopping bandits, population-based training, and local search.Grid search is recommended for one or two hyperparameters, while local search is useful when a strong starting setting is available.Source: §3.2 · pp. 6–7

3.3. Accounting for Random Variation

• The Repeat algorithm runs each hyperparameter setting multiple times so variation between training runs can be considered.Source: §3.3 · p. 7

3.4. Visualization Dashboard

• The interactive dashboard supports real-time monitoring, exploring hyperparameter–objective relationships, checking training progress, and canceling unpromising trials.Its parallel-coordinates plot, progress chart, and completed-trial table support reviewing ranges, instability, parameter impact, and consistency of strong settings.Source: §3.4 · pp. 7–8

3.5. Scaling up with a Cluster

• Parallel mode supports local CPU/GPU execution and cluster schedulers for Sun Grid Engine and SLURM, with MongoDB handling worker concurrency.Parallel mode requires MongoDB to be installed.Source: §3.5 · pp. 8–9

4.1. Handwritten Digits Classification with a Neural Network

• The MNIST example tunes Adam learning rate, hidden-layer width, and activation for a one-hidden-layer Keras classifier.Source: §4.1 · p. 9

4.1.1. API Mode

• In API mode, a Study combines hyperparameter ranges with GPyOpt, and a loop trains and evaluates each yielded trial while recording observations.The example maximizes classification accuracy and records loss as additional observation context.Source: §4.1.1 · p. 9

4.1.2. Parallel Mode

• In parallel mode, a server script configures the search and scheduler while a trial script obtains settings and sends training metrics through a Client.The example runs two evaluations concurrently; unlike API mode, it needs no finalize call.Source: §4.1.2 · p. 12

4.2.1. Introduction

• The cloud-resolving-model network maps 94 input features to a 65-dimensional output containing heating, moistening, radiative-flux, and precipitation quantities.Source: §4.2.1 · p. 14

4.2.2. General Hyperparameter Optimization

• The initial random search favored low dropout, Leaky ReLU coefficients mostly around 0.3 or higher, learning rates mostly near 0.002, eight layers, and batch normalization.The number of units did not appear to have a large impact.Source: §4.2.2 · p. 14

4.2.3. Optimization of the Learning Rate Schedule

• A 50-trial random search found a best initial learning rate of 0.001196 and a best decay factor of 0.843784.The learning-rate range was set around the earlier best value, and the decay factor multiplies the learning rate after each epoch.Source: §4.2.3 · p. 14

4.2.4. Results

• The Sherpa-tuned model consistently outperforms Rasp et al., including at latitudes where their model fails.Its loss also decreases further after the Rasp et al. model converges, which the authors attribute to learning-rate fine-tuning.Source: §4.2.4 · p. 15

5. Impact

• Sherpa aims to require minimal learning for users to get started.Source: §5 · p. 15
• The software aims to support users as they need parallel evaluations or less common optimization algorithms.Source: §5 · p. 15
• Sherpa had been downloaded more than 6000 times from PyPi and had over 160 GitHub stars.Source: §5 · p. 15

• Sherpa provides flexible, open-source software for robust hyperparameter optimization of machine-learning models.Source: §6 · p. 15
• Its interchangeable optimization algorithms can be useful at different stages of model development.Source: §6 · p. 15
• The interactive dashboard supports real-time monitoring and analysis of multiple optimization runs and helps users assess hyperparameter-setting robustness.Source: §6 · p. 15
• Sherpa provides tools for hyperparameter exploration and optimization on a laptop or distributed cluster, rather than returning only one setting as a black box.Source: §6 · pp. 15–16

Appendix A. Deep learning for Cloud Resolving Models

• The appendix reports an initial random search over the DNN hyperparameters listed in Table A.2.Source: Appendix A · p. 21, Table A.2 · p. 21
• After that search, the dashboard showed good configurations generally had low dropout, leaky ReLU coefficients around 0.3 or larger, learning rates near 0.002, and often eight layers with batch normalization.The number of units did not seem to have a large impact.Source: Appendix A · p. 21, Figure A.6 · p. 21
• The schedule discovered by Sherpa allowed the DNN to keep learning and further reduce loss, unlike the rapid decay and early loss plateau under the schedule in.Source: Appendix A · p. 22
• At all pressures and latitudes, the Sherpa model outperformed the previously published model and achieved a new state of the art for this dataset.Figure A.7 compares the optimized model with the model developed by Rasp et al. [41].Source: Appendix A · p. 22, Figure A.7 · p. 23