Source-linked AI summary
A survey on multi-objective hyperparameter optimization algorithms for Machine Learning
Alejandro Morales-Hernández, Inneke Van Nieuwenhuyse, Sebastian Rojas Gonzalez
TL;DR
Multi-objective HPO addresses the challenge of optimizing conflicting objectives for ML models, where traditional methods are often ill-suited and prior reviews largely focused on single-objective HPO. This paper systematically reviews multi-objective HPO literature from 2014 to 2020, categorizes its approaches, and identifies methodological shortcomings and future research directions.
Problem
HPO is challenging because ML objectives are often non-convex and non-differentiable, while practical model selection may require optimizing conflicting measures such as error, inference time, model size, and energy consumption.
Method
The paper presents a comprehensive review of multi-objective HPO approaches, categorizing them as metaheuristic-based, metamodel-based, or hybrid methods.
Results
The review finds that metaheuristic-based HPO approaches are the most popular, despite requiring evaluations of many hyperparameter configurations.
Takeaways & Limitations
Future HPO research should improve reproducibility and comparability by reporting search spaces, algorithmic details, performance objectives, and final optimal solutions clearly.
Takeaways & Limitations
The reviewed papers often lack explicit methodological detail, so the discussion remains general and relies partly on independently reconstructed rough pseudocodes.
Abstract
from arXiv · showhide
Hyperparameter optimization (HPO) is a necessary step to ensure the best possible performance of Machine Learning (ML) algorithms. Several methods have been developed to perform HPO; most of these are focused on optimizing one performance measure (usually an error-based measure), and the literature on such single-objective HPO problems is vast. Recently, though, algorithms have appeared that focus on optimizing multiple conflicting objectives simultaneously. This article presents a systematic survey of the literature published between 2014 and 2020 on multi-objective HPO algorithms, distinguishing between metaheuristic-based algorithms, metamodel-based algorithms, and approaches using a mixture of both. We also discuss the quality metrics used to compare multi-objective HPO procedures and present future research directions.
1 Introduction
HPO is challenging because ML hyperparameters affect performance, while search spaces and objectives can be complex and conflicting. This survey addresses the gap by systematically reviewing multi-objective HPO approaches and their evaluation.
- Hyperparameters must be configured before training and can substantially influence ML algorithm performance.
- HPO is difficult because objectives are often non-convex and non-differentiable, while hyperparameters may be discrete, categorical, continuous, or conditional.
- Automatic configuration is broader than HPO, which typically seeks a well-performing hyperparameter configuration for one task and dataset.
- Multi-objective HPO jointly addresses conflicting objectives such as predictive error, inference time, model size, and energy consumption.
- The survey reviews approaches, performance measures, and remaining challenges, claiming to provide the first comprehensive review focused on multi-objective HPO.
- The article covers literature search methodology, HPO formalization and measures, method categorization, algorithm strengths and weaknesses, and future findings.
2 Methodology
The survey searched and filtered literature published from 2014 to 2020 using Web of Science and supplementary manual searches. The final selection contained 48 relevant articles, mostly from conferences or workshops.
- The review focused on research published between 2014 and 2020 because HPO publications had surged since 2014.
- The initial Web of Science search used terms covering HPO, multi-objective optimization, scalarization, surrogates, metamodels, neural networks, overfitting, and constraints.
- The authors supplemented database results by scanning Google Scholar alerts and searching references within electronic collections indexed on Web of Science.
- Candidate papers were manually filtered by title and abstract to exclude industrial-process optimization, meta-learning, internal-parameter optimization, and unrelated AutoML work.
- Neural Architecture Search papers were included only when addressed as HPO, excluding work focused on more specific NAS aspects.
- 48 relevant articles remained after full reading and reference searching; about 60% appeared in conferences or workshops, with journal publication increasing in 2020.
3 HPO: Concepts and performance measures
Multi-objective HPO evaluates hyperparameter configurations through an inner training and validation process, balancing error-based and non-error-based objectives. The section also describes Pareto-based goals and the metrics used to compare resulting algorithms.
- HPO concepts: HPO wraps target-model training and validation in an outer optimization loop that maps hyperparameters to performance measures.The target ML algorithm performs the inner optimization, while HPO searches for hyperparameters producing favorable outcomes.
- Multi-objective HPO: Multi-objective HPO extends single-objective HPO by optimizing multiple performance measures, including error, complexity, time, cost, and energy-related objectives.These objectives can conflict, requiring explicit consideration of trade-offs.
- Multi-objective HPO: Pareto-optimal solutions cannot improve one objective without worsening another; they form the Pareto set in decision space and the Pareto front in objective space.These solutions are also called non-dominated solutions.
- Multi-objective HPO: Scalarizing multiple objectives into one function can simplify optimization but may fail to detect non-convex parts of the Pareto front.The choice of scalarization approach therefore affects which trade-offs can be identified.
- Performance measures: The literature uses error-based measures alongside model-complexity, time-based, computational-cost, hardware-cost, memory, energy, and feature-count objectives.Complexity may be represented by neurons, support vectors, file size, model count, FLOPs, parameters, or number of features.
- Quality metrics: Quality comparisons evaluate Pareto fronts using hypervolume, distance to reference fronts, coverage, shape, and diversity metrics.Hypervolume measures the area or volume enclosed by a Pareto front and a user-specified reference point.
4 Multi-objective HPO algorithms: categorization
Multi-objective HPO algorithms are categorized into metaheuristic-based and metamodel-based approaches, with methods combining both also represented. The literature includes diverse search procedures, acquisition functions, objective models, and strategies for handling computational cost and noisy evaluations.
- Metaheuristic-based HPO algorithms: NSGA-II is the most popular metaheuristic-based algorithm in multi-objective HPO and uses elitist selection to preserve strong, well-spread solutions.Its temporal complexity is O(MN^2), compared with O(MN^3) for the original NSGA algorithm.
- Metaheuristic-based HPO algorithms: Nelder-Mead applications scalarize multiple objectives with weighted sums, while their local search can become trapped in local minima.Different starting points or modified procedures can reduce this risk, but modifications may require many iterations.
- Metamodel-based HPO algorithms: Metamodel-based HPO iteratively trains a surrogate on initial observations, selects new configurations with an acquisition function, and updates the model.Most surveyed metamodel-based methods use Gaussian Processes, while Random Forests are also used as surrogate models.
- Metamodel-based HPO algorithms: Expected Improvement is the most popular acquisition function, although multi-objective studies also use POI, UCB, EHI, PES, and related criteria.Expected Improvement is often applied after scalarizing objectives, whereas PES-based methods can target Pareto-set uncertainty directly.
- Metamodel-based HPO algorithms: PABO fits separate metamodels for individual objectives and outperforms NSGA-II in search speed, while MOBO-PC focuses on Pareto solutions satisfying preference-order constraints.PABO uses dominance rank as an infill criterion; MOBO-PC modifies EHI to incorporate user-defined preferences.
- Metamodel-based HPO algorithms: Most surveyed HPO algorithms ignore noisy performance measures, and studies that model noise commonly assume it is homogeneous despite evidence that reevaluation can improve reliability.Noise can arise from randomized learning procedures and from repeated cross-validation splits; heterogeneous noise may vary across the search space.
5 Multi-objective HPO algorithms: Pros and cons
The review finds substantial variation in the computational, search-space, and parallelization properties of multi-objective HPO algorithms, but methodological reporting is often insufficient for firm comparisons.
- Evaluation criteria: The review evaluates computational complexity, high-dimensional input spaces, mixed input spaces, and ease of parallel computation.The authors note that studied papers rarely provide explicit details on these aspects.
- Reporting quality: The papers frequently omit methodological details, including hyperparameter types, genetic operators, initial populations, experimental designs, final Pareto solutions, pseudocode, and novel metrics.Consequently, the review’s discussion remains general and relies partly on independent analysis of methodological papers.
- Computational complexity: The analysis estimates configuration evaluations with g(n, I, N), determined by initial configurations n, iterations I, and new configurations per iteration N.Table 10 summarizes these estimates across the algorithms studied.
- Computational complexity: Metamodel-based methods typically evaluate fewer costly configurations than metaheuristic-based methods because they usually assess one new solution per iteration.This advantage does not guarantee lower total execution time when additional stages, such as feature selection, substantially increase evaluations.
- Search spaces and parallelization: Parallelization commonly evaluates novel configurations concurrently within each metaheuristic population generation.The review also notes that mixed search spaces are not straightforward for some metaheuristics, while Random Forest and TPE metamodels can accommodate mixed variables.
6 Conclusions and future research
The survey categorizes multi-objective HPO literature and finds that the approach is practically relevant despite being less abundant than single-objective HPO. It identifies reporting gaps and several directions for improving methods, comparisons, reproducibility, and interpretability.
- Conclusions: The survey categorizes multi-objective HPO algorithms into metaheuristic-based, metamodel-based, and hybrid approaches.It presents this categorization as its review of the existing literature.
- Conclusions: Multi-objective HPO can optimize trade-offs among performance measures, and minimizing training features can improve ML performance without necessarily compromising loss-based performance.The review cites this pattern as evidence of practical relevance.
- Future research: Metaheuristic-based HPO is most popular despite requiring many costly hyperparameter evaluations, while hybrid research remains scarce.The authors describe hybrid methods as promising because they may combine inexpensive metamodel evaluations with heuristic search.
- Future research: Future work could extend low-fidelity methods, such as bandit-based approaches, from single-objective to multi-objective HPO.The survey identifies cheaply available, lower-fidelity information as a potential research opportunity.
- Future research: The literature commonly ignores uncertainty in performance measures, and existing noise-handling approaches assume homogeneous noise likely unlike practical settings.The authors suggest adapting metamodels or algorithmic approaches to account for noise.
- Interpretability: The authors recommend using individual performance measures rather than aggregate objectives such as F-measure or AUC to improve interpretability.Aggregate measures encode fixed relationships among their component measures.
- Reporting and evaluation: The review recommends reporting search spaces, algorithmic details and pseudocode, performance objectives, final solutions, and quality metrics to improve reproducibility.It also advocates benchmarking new algorithms against existing methods because only about half of reviewed papers do so.
Appendix A Pseudocodes of MO-HPO algorithms
The appendix presents pseudocode for metamodel-based and population-based multi-objective HPO algorithms, parameterized by initial configurations, iterations, and configurations generated per iteration.
- Algorithm families: The pseudocode covers metamodel-based optimization and a broad group of population-based, particle-based, evolutionary, ant-colony, and hybrid algorithms.The population-based group includes NSGA-II, OMOPSO, PSO, SPEA-II, MO-RACACO, CoDeepNEAT, ENS-MOEA/D, MOEA/D, scalarized GA, and metamodel-plus-NSGA-II hybrids.
- Metamodel-based optimization: Metamodel-based optimization samples an initial design, then obtains one new configuration per iteration by optimizing an acquisition function using metamodel predictions.Its evaluation count is expressed as g(n, I, N = 1) = n + I.
- Population-based optimization: Population-based algorithms generate N new hyperparameter configurations per iteration and use algorithm-specific update and selection operations.The shared evaluation-count expression is g(n, I, N) = n+IN.
- Selection mechanisms: NSGA-II, CoDeepNeat, GA, and the hybrid algorithm select configurations using non-dominated and crowding-distance sorting.This selection step differs from the non-dominated sorting used by MOEA/D and ENS-MOEA/D.
- Algorithm-specific updates: MOEA/D and ENS-MOEA/D use non-dominated sorting, while OMOPSO and PSO update particle velocity and position during the search.The pseudocode also includes pheromone-path updates for the ant-colony-based branch.
- Output: All listed procedures return the hyperparameter configurations on the Pareto front after completing their optimization loops.The pseudocode explicitly terminates with a Pareto-front return step.