Hyper-Parameter Optimization: A Review of Algorithms and Applications

• Manual hyperparameter selection is experience-dependent and often yields workable rather than optimal sets, while larger or novel models make trial-and-error costly in time and computation.Source: §1 · pp. 2–3
• HPO automates search for structure and training settings as an AutoML component, reducing expert menial work and targeting better accuracy, efficiency, and reproducibility at computational cost.Source: §1 · p. 2
• HPO is increasingly necessary as deep-learning models scale up or become lightweight: complexity increases settings to tune, while constrained designs demand strict ranges for reproducing accuracy.Source: §1 · pp. 2–3
• Demand from research and industry, including inexperienced users, motivates ready-to-use HPO services and toolkits, which the survey compares for design insights and target users.Source: §1 · p. 3
• The survey categorizes structure- and training-related hyperparameters and compares HPO algorithms by accuracy, efficiency, applicability, and limitations across scenarios.Source: §1 · p. 3
• The paper identifies HPO research directions across algorithms, applications, and techniques, and highlights efficient model evaluation under limited computational resources.Source: §1 · pp. 3–4

2. Major Hyper-Parameters and Search Space

• Hyper-parameters with stronger effects on training weights receive priority, but their importance is difficult to quantify and is generally judged from previous experience.Source: §2 · p. 4

2.1 Learning Rate

• Learning rates may be constant, manually adjusted, adaptive, or scheduled; linear and exponential decay use initial rate and decay settings, and EfficientNet illustrates that these can be difficult to choose by hand.The passage gives exponential-decay settings of epochsdrop=2.4 and floor=0.97 for EfficientNet trained on ImageNet and says automatic tuning is appropriate in that case.Source: §2.1 · pp. 4–6
• Learning-rate schedules require advance choices based on experience, may apply one rate across all layers, and should be tuned with the optimizer because rates that are too small slow convergence while large rates may prevent it.Automated HPO or cyclical learning rates can address advance schedule choices; layer-wise adaptive rate scaling gives layers local rates related to their weights and gradients (You et al., 2017).Source: §2.1 · p. 6

2.2 Optimizer

• Mini-batch gradient descent speeds training on large datasets and reduces noise relative to batch size 1, while batch size depends on device memory; powers of 2 and 32 are suggested, though vanilla mini-batch GD can converge slowly.The passage also suggests re-optimizing batch size and learning rate after other hyper-parameters are fixed (Masters and Luschi, 2018).Source: §2.2 · p. 8
• Adam is suggested as a default and typically needs little tuning beyond learning rate, while SGD with momentum may take longer to find an optimum; optimizer settings should be adjusted alongside the learning-rate schedule.Suggested momentum values are 0.9, 0.99, or 0.999; Adam settings include β1=0.9, β2=0.999, and 10−8, with a starting learning rate of 0.001.Source: §2.2 · pp. 8–10

2.3 Model Design-Related Hyper-Parameters

• More hidden layers can yield more complex features and higher accuracy, while layer width must balance underfitting from too few neurons against overfitting and added training time from too many.Tan and Le (2019) created networks using depth and width settings to achieve high accuracy with low computational resources and few parameters.Source: §2.3 · p. 11
• Regularization reduces model complexity and overfitting, but λ that is too large oversimplifies a network and λ that is too small insufficiently reduces weights; L2 is more computationally efficient, whereas L1 better supports sparsity.L1-generated models are described as simpler and interpretable, while L2 provides superior predictions.Source: §2.3 · pp. 11–12
• Data augmentation creates training examples through transformations to reduce overfitting, dropout randomly deactivates neurons with a suggested rate of 20%-50%, and activation-function structure and settings can be searched automatically.The listed activation functions include sigmoid, tanh, ReLU, Maxout, and Swish; a dropout rate that is too large oversimplifies the model, while a small rate has little effect.Source: §2.3 · pp. 12–13
• Sigmoid outputs lie between 0 and 1 but can cause vanishing gradients and difficult optimization, whereas softmax is used for multinomial outputs whose probabilities sum to 1.Sigmoid is used for binary classification; softmax is used for multinomial classification and gives higher probability to a higher input relative to other classes.Source: §2.3 · pp. 13–14
• Tanh is zero-centered, has a steeper derivative than sigmoid, and is described as more efficient, but it still has vanishing gradients and cannot be used in many-layer networks.Source: §2.3 · p. 14
• ReLU is widely used and suggested as a simple, fast default for deep networks, while leaky ReLU can keep updates alive when vanishing gradients occur.ReLU avoids vanishing gradients but can produce unbounded activations and inactive neurons for negative inputs; leaky ReLU uses a small negative slope, such as 0.01x.Source: §2.3 · pp. 14–15
• Maxout generalizes ReLU and leaky ReLU, is described as fixing ReLU’s problems, and often works with dropout, but doubles the number of parameters.Each neuron adds a weight and bias.Source: §2.3 · p. 15
• Swish was proposed as a ReLU competitor and claimed to outperform it with similarly simple calculations, but it is less popular and offers a smooth curve that may suit smooth landscapes.It is bounded below and unbounded above, capturing small negative inputs while mapping large negative inputs toward zero.Source: §2.3 · p. 15

3. Search Algorithms and Trial Schedulers on Hyper-Parameter Optimization

• HPO adjusts hyper-parameters from coarse to fine, evaluates models across parameter sets, and seeks an optimal combination.This study focuses on the adjustment and model-evaluation steps of that process.Source: §3 · p. 15
• Hyper-parameters can be integer, floating-point, categorical, or binary values, and affect both network structure and training accuracy.HPO seeks a configuration that minimizes loss or maximizes accuracy.Source: §3 · p. 15
• To optimize the objective function, the model is trained on Dtrain and validated on Dvalid.The objective is to maximize or minimize the V function using algorithm Aλ applied to the hyper-parameters.Source: §3 · p. 16
• Search algorithms sample hyper-parameter configurations, while trial schedulers chiefly use early stopping to evaluate models.The section discusses popular search algorithms first, followed by state-of-the-art early-stopping strategies.Source: §3 · p. 16

3.1.1 Grid Search

• Grid search exhaustively tests user-generated candidates and can find the optimum with sufficient resources, but is practical mainly for narrow spaces and at most three simultaneous hyper-parameters.Its independent trials are easy to parallelize, while resource needs grow exponentially with the number of hyper-parameters.Source: §3.1.1 · p. 16

3.1.2 Random Search

• Random search samples hyper-parameters from distributions, allocates budget across the search space, and is often more effective than grid search, especially when parameters are not uniformly distributed.It is suggested for early coarse-to-fine narrowing, but remains computationally intensive and is often used as an HPO baseline.Source: §3.1.2 · pp. 16–17

3.1.3 Bayesian Optimization and Its Variants

• Bayesian optimization iteratively updates a probabilistic surrogate and acquisition function to balance exploration and exploitation, often using Gaussian processes and expected improvement to find optima in fewer trials.Expected improvement is described as the most common acquisition function, while other options include probability of improvement, GP-UCB, and predictive entropy search.Source: §3.1.3 · pp. 18, 19, 21
• Bayesian optimization’s sequential trials and computational demands limit parallelism and deep-learning use, while Gaussian processes scale poorly with high dimensions or many observations.The paper describes batch and acquisition-function parallelization, predictive termination and freeze-thaw methods, and applications in image classification and natural language processing.Source: §3.1.3 · pp. 21–22

3.1.4 Tree Parzen Estimators

• Tree Parzen Estimators use a tree-structured surrogate to handle conditional variables and more data types; HyperOpt parallelizes TPE, and BOHB combines it with HyperBand.TPE is more popular than the original Bayesian method in deep learning, but Gaussian processes perform better when hyper-parameters have strong interactions.Source: §3.1.4 · p. 23

3.2 Optimization with an Early-stopping Policy

• Early-stopping policies free computational resources by terminating trials before full training, directing the budget toward promising hyperparameter sets.They can be used alongside a search algorithm or independently for both searching and halting.Source: §3.2 · pp. 23–24

3.2.1 median stopping

• Median stopping halts a trial when its best objective by a given step is worse than the median running-average objective of completed trials at that step.It is model-free, needs no user-set hyperparameters, and is supported by Google Vizier, Tune, and NNI.Source: §3.2.1 · p. 24

3.2.2 curve fitting

• Curve fitting predicts a trial’s final objective from its learning curve and stops it when the prediction is sufficiently worse than a tolerant historical optimum.The approach fits and weights candidate curves, predicts outcomes, and reassesses when predictions have not converged; unlike median stopping, it is a parameterized model with its own training process. Freeze-Thaw BO combines Bayesian optimization with curve fitting.Source: §3.2.2 · pp. 24–25

3.2.3 Successive Halving and HyperBand

• Successive halving (SHA) repeatedly drops poorly performing trials and increases resources for the survivors, while HyperBand (HB) tests different trial-count and budget tradeoffs.SHA requires users to choose the initial number of trials and total budget, risking premature stopping or too few options; HB varies these tradeoffs and uses resource limit R and elimination proportion η, with η defaults of 2 for SHA and 3 or 4 for HB.Source: §3.2.3 · pp. 26–27

3.2.4 Asynchronous Successive Halving and Bayesian OptimizationHyperBand

• 20-times acceleration over random search is reported for BOHB and HyperBand with limited budgets; BOHB reaches 55-times acceleration over random search with sufficient budgets.ASHA performs similarly to HyperBand in sequential or small-scale settings and exceeds most state-of-the-art HPO methods at large scale. BOHB combines HyperBand scheduling with guided Bayesian sampling, but budget selection can make it slower than vanilla TPE or less efficient in extreme cases.Source: §3.2.4 · pp. 27–29, Figure 15 · p. 28

3.2.5 Population-Based Training

• Population-based training (PBT) adapts hyperparameters during training by combining parallel search with sequential optimization.It periodically exploits strong configurations and explores through random hyperparameter perturbations, enabling warm starts for large models; its optimality is unproven and its exploitation and exploration strategies remain limited.Source: §3.2.5 · pp. 29–30

4. Toolkits for Hyper-parameter Optimization

• Toolkits combine tuning algorithms, trial scheduling, and deep-learning framework compatibility to speed HPO and lower barriers to research and development.The review assesses contemporary tools and services by resource handling, support for state-of-the-art optimization algorithms, and trial scheduling.Source: §4 · p. 31

4.1 Overview

• Open-source libraries target varied tasks and algorithms, while cloud services offer massive parallel resources; common library limitations in algorithms, framework support, and scheduling impede efficient deep-network search.The review frames tool selection around ease of use, state of the art, availability, scalability, and flexibility.Source: §4.1 · p. 31

4.2 Google Vizier

• Google Vizier simplifies HPO setup and scales to millions of experiments and billions of trials, but its cloud infrastructure is closed-source and may be less flexible.It automates resource management and trial scheduling, supports GP-regression transfer learning from prior jobs, and offers a GUI; users can customize algorithms, though the review notes limitations in search algorithms and early stopping.Source: §4.2 · pp. 32–33

4.3 Automatic model tuning in Amazon SageMaker

• Amazon SageMaker’s managed model-building service integrates HPO, training, and deployment, with parallel tuning, console monitoring, early stopping, and warm starts, but search is limited to random search and Bayesian optimization.Users provide a model and training data, and can add their own algorithms.Source: §4.3 · p. 34

4.4 Neural Network Intelligence

• NNI is an extensible open-source AutoML and HPO toolkit with broad framework and deployment support, but it requires users to deploy it and bring model-training experience.It includes most state-of-the-art algorithms except PBT, supports user-written algorithms, and offers a Web UI for monitoring; unlike Vizier, that interface cannot interactively change experiment settings, and its debugging information could be clearer.Source: §4.4 · pp. 34–35

4.5 Ray.Tune: An open-source toolkit for hyper-parameter search

• Ray Tune supports parallel search and PBT, while NNI offers an easier Web UI; both are free, extensible open-source toolkits suited to experienced developers but require deployment and configuration.Tune lacks Kubernetes-based Docker support, and both toolkits lack interactive GUIs and cannot match cloud services’ resource scale for thousands of simultaneous trials.Source: §4.5 · pp. 36–38

5.1 Comparability between different algorithms

• Multi-armed bandits save computation by terminating poor trials early and parallelize readily, while random search offers parallelism and can gain efficiency from early stopping.SHA, HB, BOHB, and ASHA allocate and adjust trial budgets; BOHB combines guided search with early stopping, and ASHA supports asynchronous deployment. Budget design remains challenging, and early stopping can eliminate models that improve later.Source: §5.1 · pp. 38–40
• PBT tunes hyper-parameters during parallel training, using warm starts and inheritance to avoid waiting for convergence; the review calls it the most efficient algorithm and schedule.Its nonstationary hyper-parameters suit expensive models such as GANs and transformers, but changing computation graphs can complicate use with predefined graphs and prevent a single best or stationary hyper-parameter set.Source: §5.1 · pp. 39–41
• Grid search becomes infeasible as dimensionality grows, random search cannot guarantee an optimum, and Bayesian optimization is less naturally parallel and conceptually complex.Grid search remains applicable when users narrow the range, while random search suits early exploration; TPE handles conditional hyper-parameters and parallelization. ASHA suits massive parallel systems, BOHB suits sequential or distributed systems, and PBT suits expensive models with nonstationary hyper-parameters (e.g., LR).Source: §5.1 · pp. 40–41

5.2 Evaluation of results

• Direct evaluation trains each configuration to convergence and is most accurate but slow; reduced-fidelity evaluation is faster, while subsampling can add noise and uncertainty.Subsampling trains on a subset whose size must be chosen carefully; transfer series expansion combines predictors to accelerate evaluation.Source: §5.2 · pp. 41–42
• Early stopping saves training time by terminating models that underperform at checkpoints, but can discard a model that improves later.Median stopping and curve-fitting can accompany random search or Bayesian methods; multi-armed bandit and population-based algorithms use their own checkpoints and criteria. Early stopping is common in AutoML frameworks.Source: §5.2 · p. 42
• Transfer learning reuses knowledge or parameters from similar tasks or configurations to reduce target-task training, but its noise and local-optimum risks are less predictable.Its suitability depends on source and target model similarity; it has been used with large image datasets such as ImageNet (Sharif Razavian et al., 2014), while most open-source AutoML frameworks lack a transfer-learning module.Source: §5.2 · pp. 42–43
• Meta learning predicts configuration performance with surrogate models built from prior configurations, but predictions can be biased by model choice and sampling data.Approaches include linear regression, curve-fitting, support vector machines, Bayesian neural networks, and multilayer perceptrons; prediction accuracy depends on available sampling points and their accuracy.Source: §5.2 · p. 43

• The review introduces important hyper-parameters for training and model structure, then surveys state-of-the-art HPO search algorithms and schedulers.Source: §6 · p. 43
• It reviews HPO services and open-source frameworks, including their pros, cons, and applications.Source: §6 · p. 43
• It compares optimization algorithms and evaluation methods for feasibility with large models and computational-resource consumption.Source: §6 · p. 43