A survey on multi-objective hyperparameter optimization algorithms for Machine Learning

• HPO is necessary because hyperparameters influence ML performance but are specified before training and are not optimized within the training algorithm.Source: §1 · p. 2
• Non-convex, non-differentiable objectives, mixed and conditional hyperparameters, and expensive model training make HPO difficult and demand computationally efficient search methods.Source: §1 · p. 2
• HPO differs from broader automatic algorithm configuration and from multi-task learning because it optimizes hyperparameters for one task and dataset, typically across conflicting objectives.Source: §1 · pp. 2–3
• Although advanced HPO methods have largely addressed single objectives, multi-objective HPO considers competing goals including error-based performance, inference time, model size, and energy consumption.Source: §1 · p. 3
• The survey aims to provide the first comprehensive review of multi-objective HPO for ML, covering literature approaches, objective measures, challenges, and future research directions.Earlier reviews mainly addressed single-objective HPO or only mentioned multi-objective approaches without discussing them in detail.Source: §1 · pp. 3–4

Search strategy

• The search covered 2014–2020 publications and used Web of Science terms spanning multi-objective and single-objective HPO, surrogates, metamodels, deep learning, constraints, and overfitting.The search was restricted to computer-science-related Web of Science categories and included conference proceedings, articles, book chapters, and meeting abstracts.Source: §2 · pp. 4–5
• The candidate set was supplemented by Google Scholar alerts and reference searches limited to electronic, Web of Science-indexed journals, conferences, and workshops.Accepted preprints of forthcoming Web of Science journal articles were included.Source: §2 · p. 5

Eligibility screening

• Manual title-and-abstract screening excluded work on industrial-process optimization, meta-learning, internal-parameter optimization, unrelated AutoML, and NAS unless framed as HPO.More specific NAS studies, such as Negrinho et al. (2019), were outside the survey’s scope.Source: §2 · pp. 5–6

Final selection

• 48 relevant articles remained after full reading and reference searching; about 60% were published in conferences or workshops, with journal publication increasing in 2020.The 2020 journal articles were mainly in Q1/Q2 computer-science journals.Source: §2 · p. 6, Figure 2 · p. 6

3.1 HPO: Concepts and terminology

• HPO optimizes hyperparameter configurations through an outer process, while the target ML algorithm is trained and validated inside it using protocols such as holdout or cross-validation.The hyperparameter configuration space is Λ = Λ1 × Λ2 × ... × ΛN, and performance is typically estimated by minimizing a validation loss.Source: §3.1 · pp. 6–7
• Multi-objective HPO seeks diverse, convergent Pareto-optimal solutions under trade-offs among objectives such as model performance, training time, and error-based measures.Scalarization can simplify optimization but may fail to detect non-convex parts of the Pareto front.Source: §3.1 · p. 8

3.2 Multi-objective HPO: Typical objectives

• Common objectives combine error-based measures with model complexity, feature count, time, computational cost, energy consumption, memory utilization, and other task-specific measures.Complexity may be represented by parameters, features, support vectors, model count, or FLOPs, while hardware costs are especially relevant for edge devices.Source: §3.2 · pp. 8–11

3.3 Quality metrics for comparing multi-objective HPO algorithms

• Quality assessment primarily uses Pareto-front metrics such as hypervolume, dominated hypervolume, distance to reference fronts, coverage, shape, spacing, and spread, supplemented by runtime and evaluation-based measures.Additional measures include execution time, the number of performance evaluations, CPU utilization, non-objective performance measures, and algorithm-specific search indicators.Source: §3.3 · pp. 11–12

4 Multi-objective HPO algorithms: categorization

• Multi-objective HPO algorithms are divided into metaheuristic-based methods, metamodel-based methods using acquisition functions, and hybrids combining metamodels with metaheuristic search.The categorization is based on how algorithms search for optimal solutions.Source: §4 · p. 12

4.1 Metaheuristic-based HPO algorithms

• NSGA-II is the most popular metaheuristic, with lower temporal complexity than the original NSGA and elitist selection preserving high-quality, diverse solutions.NSGA-II has complexity O(MN^2) versus O(MN^3) for the original NSGA, where M is the number of objectives and N is the population size.Source: §4.1 · p. 14
• Metaheuristic studies report improved accuracy over manual hyperparameter selection, while comparisons among methods are conditional: NSGA-II slightly underperformed ParEGO on generalization-dominated hypervolume, and CMA-ES produced smaller, lower-error networks than NSGA-II.NSGA-II is suggested over ParEGO when evaluations are cheap and marginal performance degradation is acceptable; the CMA-ES comparison with single-objective CMA-ES is considered inappropriate because network size was not optimized there.Source: §4.1 · pp. 14, 20

4.2 Metamodel-based HPO algorithms

• Metamodel-based HPO trains a surrogate on initial input/output observations, uses an acquisition function to select new evaluations, updates the surrogate, and repeats until stopping.Gaussian Processes are the most common metamodel; their predictions and uncertainty guide acquisition-function search for new configurations.Source: §4.2 · pp. 22–23
• POI significantly outperformed EI and UCB for a weighted three-objective SNN optimization, while the BO approach required significantly fewer evaluations than evolutionary strategies such as SADE.Expected Improvement is the most popular acquisition function and is usually applied after scalarizing objectives in multi-objective problems.Source: §4.2 · pp. 27–28
• PABO outperformed NSGA-II in search speed by fitting objective-specific metamodels, sharing nondominated infill evaluations across objectives, and using dominance rank as its infill criterion.The approach models an error-based and an energy-related objective independently, then evaluates each eligible suggested configuration for both objectives.Source: §4.2 · p. 28
• Correlation-aware modified EHV accelerated convergence to Pareto-optimal solutions, while MOBO-PC directed search toward Pareto solutions satisfying user-defined preference-order constraints.Shah and Ghahramani (2016) model accuracy, memory consumption, and training time jointly; Abdolsh et al. (2019) modify EHI to account for preference constraints.Source: §4.2 · p. 28
• Confidence-bound strategies improved CPU utilization within the same time budget, while RF surrogates enabled SMS-EGO and ParEGO to outperform random sampling and NSGA-II.Chin et al. (2020) reduced the complexity of the studied neural networks but did not compare performance with other multi-objective HPO algorithms.Source: §4.2 · pp. 28–29
• PESMO and PESMOC achieved superior hypervolume and evaluation efficiency in neural-network and constrained decision-tree HPO, while decoupled PES improved solutions under limited resources but could prioritize objectives.PESMO minimizes uncertainty in the Pareto-set location, whereas PESMOC incorporates constraints; the decoupled approach selects configurations using the maximum acquisition value across objectives.Source: §4.2 · p. 29
• Noise is commonly ignored in multi-objective HPO, but reevaluating promising configurations performed best under heterogeneous noise, whereas single evaluations performed worse than repeated random search.Horn et al. (2017) assumed homogeneous noise but found that reevaluation improved reliability; the survey attributes this to noise magnitude varying across the search space.Source: §4.2 · p. 31
• For noisy SVM evaluations, SMS-EGO and SExI-EGO combined with reinterpolation produced comparable hypervolume results.Koch et al. (2015) compared reinterpolation with static resampling; both algorithms used EHI-based infill criteria, although SExI-EGO required more runtime.Source: §4.2 · p. 31

4.3 Hybrid HPO algorithms

• Hybrid HPO algorithms are relatively scarce and combine metamodels with population-based search, producing near-literature performance at lower energy, architecture-size, or search-time costs.The survey summarizes these approaches and their experimental comparisons in Tables 8 and 9.Source: §4.3 · pp. 31–32, Table 8 · p. 33, Table 9 · p. 34
• NSGANetV2 matched state-of-the-art NAS performance on CIFAR-10, CIFAR-100, and ImageNet at reduced search cost, while sequential HPO and feature selection remained suboptimal.Lu et al. (2020) selected among MLP, CART, RBF, and GP metamodels per iteration; Martinez-de Pison et al. (2017) optimized HPO and feature selection sequentially rather than jointly.Source: §4.3 · pp. 31–32

Scope and limitations

• Published papers rarely detail computational complexity, input-space support, parallelization, algorithm components, experimental designs, or final Pareto solutions, limiting systematic comparison.The review attributes this lack of detail largely to application-focused papers in which HPO was not the main contribution.Source: §5 · p. 34

Computational complexity

• The review estimates worst-case HPO running time using the number of evaluated configurations, represented as g(n, I, N) for initial configurations, iterations, and new configurations per iteration.Training and validation dominate HPO cost because they are performed for every configuration evaluated.Source: §5 · pp. 34–35, Table 10 · p. 37
• Metamodel-based methods usually evaluate fewer costly configurations than metaheuristics, but MADE and GP + GA Parsimony can remain expensive because they generate many candidates or add feature selection.GP + GA Parsimony evaluates substantially more configurations than methods such as NSGA-II and GP-based optimization because feature selection follows HPO in a separate phase.Source: §5 · p. 35

Parallelization

• Metaheuristics commonly parallelize population evaluations, whereas Bayesian optimization is inherently serial but supports limited parallelism through initial batches or batch BO.Parallelism can also occur during cross-validation or metamodel training, including Random Forests and Gaussian Processes.Source: §5 · p. 35

Search-space support

• Mixed-space handling requires specialized encoding or operators for many metaheuristics and suitable GP kernels, while Random Forest and TPE metamodels handle mixed variables more straightforwardly.The review classifies high-dimensional search-space ability as poor, good, or unknown based on findings from other studies.Source: §5 · p. 36

• Multi-objective HPO can optimize trade-offs among performance measures, and minimizing the number of training features may improve task performance without compromising loss-based performance (Binder et al., 2020).Source: §6 · p. 39
• Metaheuristic-based approaches dominate current literature despite expensive configuration evaluations, making hybrid algorithms promising for combining inexpensive metamodel evaluations with heuristic search.Hybrid research remains scarce, but hybrid methods could avoid the challenge of optimizing an acquisition function while reducing computational cost.Source: §6 · pp. 39–40
• Ensembles of optimal configurations can improve results (Ekbal & Saha, 2015, 2016; Sopov & Ivanov, 2015; C. Zhang et al., 2016), while future work could ensemble metamodels, acquisition functions, or optimization procedures.Such ensembles increase the number of required hyperparameter evaluations.Source: §6 · p. 40
• Low-fidelity evaluations, bandit methods, and early stopping are promising extensions from single-objective HPO, but none had yet been applied to multi-objective HPO.These approaches could use partial training data, fewer iterations, or learning-curve predictions to allocate computational budgets more intelligently; early stopping may also reduce overfitting.Source: §6 · p. 40
• Performance-measure uncertainty is usually ignored; future methods should model heterogeneous noise and dynamically allocate replications to configurations yielding the most information.Stochastic algorithms may support dynamic replication decisions (Binois, Huang, Gramacy, & Ludkovski, 2019; Gonzalez, Jalali, & Van Nieuwenhuyse, 2020).Source: §6 · p. 40
• Future studies should use individual objectives, choose scalarization carefully, report search spaces, algorithms, objectives, and final solutions, and benchmark against existing methods.The Augmented Tchebycheff function is recommended for non-convex Pareto fronts; only about half of reviewed papers benchmarked against existing algorithms.Source: §6 · pp. 39–41, Table 11 · p. 39

Appendix scope

• The pseudocode headings specify each algorithm’s hyperparameter-evaluation count, while the procedures were reconstructed from surveyed papers and original algorithm descriptions.This appendix documents implementation details rather than introducing new algorithms.Source: Appendix A · p. 42

Metamodel-based optimization

• Metamodel-based optimization evaluates g(n, I, N = 1) = n + I configurations by sampling an initial design and optimizing an acquisition function using metamodel predictions.The procedure returns the hyperparameter configurations on the Pareto front.Source: Appendix A · p. 42

Population-based and hybrid algorithms

• NSGA-II, OMOPSO, PSO, SPEA-II, MO-RACACO, CoDeepNEAT, ENS-MOEA/D, MOEA/D, scalarized-objective GA, and hybrid models with NSGA-II use g(n, I, N) = n+IN.These procedures generate N new configurations per iteration from a population of size n over I iterations, then return Pareto-front configurations; selection uses non-dominated and crowding-distance sorting for specified methods, while particle and pheromone updates apply to others.Source: Appendix A · p. 43