Source-linked AI summary
Performance assessment and exhaustive listing of 500+ nature inspired metaheuristic algorithms
Zhongqiang Ma, Guohua Wu, Ponnuthurai N. Suganthan, Aijuan Song, Qizhang Luo
TL;DR
Metaheuristics have not been rigorously tested on comprehensive, representative benchmark suites or compared with one another. This paper reviews and categorizes more than 500 metaheuristics and compares selected algorithms, finding EBCM similar to four state-of-the-art methods while ten recent algorithms are inferior.
Problem
Metaheuristics are neither rigorously tested on comprehensive and representative benchmark suites nor compared with established methods.
Method
The paper provides a comprehensive review, collects more than 500 proposed and improved metaheuristics, proposes a taxonomy, and tests parameter configurations across functions.
Results
EBCM performs similarly to four state-of-the-art algorithms, whereas ten recent algorithms are inferior on the CEC 2017 benchmark suite.
Takeaways & Limitations
The findings distinguish EBCM from most recent algorithms while indicating that the four state-of-the-art methods remain stronger on CEC 2017.
Takeaways & Limitations
The paper calls for thorough theoretical analysis to explain why these metaheuristics perform better or worse.
Abstract
from arXiv · showhide
Metaheuristics are popularly used in various fields, and they have attracted much attention in the scientific and industrial communities. In recent years, the number of new metaheuristic names has been continuously growing. Generally, the inventors attribute the novelties of these new algorithms to inspirations from either biology, human behaviors, physics, or other phenomena. In addition, these new algorithms, compared against basic versions of other metaheuristics using classical benchmark problems without shift/rotation, show competitive performances. In this study, we exhaustively tabulate more than 500 metaheuristics. To comparatively evaluate the performance of the recent competitive variants and newly proposed metaheuristics, 11 newly proposed metaheuristics and 4 variants of established metaheuristics are comprehensively compared on the CEC2017 benchmark suite. In addition, whether these algorithms have a search bias to the center of the search space is investigated. The results show that the performance of the newly proposed EBCM (effective butterfly optimizer with covariance matrix adaptation) algorithm performs comparably to the 4 well performing variants of the established metaheuristics and possesses similar properties and behaviors, such as convergence, diversity, exploration and exploitation trade-offs, in many aspects. The performance of all 15 of the algorithms is likely to deteriorate due to certain transformations, while the 4 state-of-the-art metaheuristics are less affected by transformations such as the shifting of the global optimal point away from the center of the search space. It should be noted that, except EBCM, the other 10 new algorithms proposed mostly during 2019-2020 are inferior to the well performing 2017 variants of differential evolution and evolution strategy in terms of convergence speed and global search ability on CEC 2017 functions.
2 Literature overviews
Metaheuristics have rapidly expanded across inspiration sources and algorithm families, motivating taxonomies and comprehensive performance comparisons. The review emphasizes population-based methods and reports that established evolutionary variants remain strong benchmarks for newer algorithms.
- Research activity: Differential evolution, particle swarm optimization, simulated annealing, ant colony optimization, and artificial immune systems each exceeded 10,000 publications in the surveyed data.
- More than 500 metaheuristics are collected, and a taxonomy is proposed to organize them.
- Taxonomy of metaheuristics: Metaheuristics are classified by inspiration source and by search structure into population-based and single-solution-based algorithms.Population-based algorithms use multiple cooperating individuals, whereas single-solution methods generally use one individual.
- Population-based optimization algorithms: Population-based algorithms employ information sharing, interactive learning, and stochastic operators such as mutation and crossover.
- Population-based optimization algorithms: Population-based methods are further divided into evolutionary, swarm-intelligence, and physics- or chemistry-based algorithms.
- Swarm intelligence algorithms: Swarm-intelligence algorithms offer general frameworks, retained search information, and relatively few parameters, but can face premature convergence, local optima, and exploration–exploitation trade-offs.
3 Experimental analyses
The study compares 11 newly proposed and 4 state-of-the-art metaheuristics using unified parameter configuration, origin-bias tests, and analyses of search behavior and performance.
- 11 newly proposed and 4 state-of-the-art metaheuristics are selected for comparative evaluation.
- The irace framework automatically configures the parameters of all 15 comparative algorithms.
- The experiments investigate whether the algorithms have a search bias toward the origin.
- Convergence, diversity, and the exploration–exploitation trade-off are analyzed for all 15 algorithms.
- Nonparametric statistical methods, including Friedman, Wilcoxon signed-rank, and Bayesian signed-rank tests, are used to analyze the results.
3.1 Experiment setup
The experiments compare 15 algorithms on CEC 2017 functions across three dimensionalities, using repeated runs, parameter tuning, statistical tests, and shifted-function analyses.
- The benchmark contains 30 bound-constrained functions spanning unimodal, multimodal, hybrid, and composition categories.
- CEC 2017 functions use upper and lower bounds of 100 and -100, respectively, with global minima at function-index-scaled values.
- The maximum evaluation budget is 10000 ∗D, and reported experimental results average over 31 runs.
- Parameters are tuned on CEC 2017 functions before performance, origin-bias, and transformation experiments are conducted.
- Friedman, Bayesian rank-sum, Wilcoxon signed-rank, and Bayesian signed-rank tests assess statistical differences, with p ≤ 0.05 treated as significant.
3.2 Automatic parameter tuning
Automatic parameter tuning uses irace to evaluate configurations across training functions and iteratively focus sampling on better-performing settings.
- The iterated racing method automatically configures algorithm parameters for fair comparison.
- irace samples configurations, races them to select better performers, and updates its sampling distribution toward those configurations.
- Training instances comprise eight functions covering all CEC 2017 function types.
- irace executes algorithms with different parameter configurations and tests configurations across all functions to identify the best-performing configuration.
- The tuned parameter settings are reported for the 15 algorithms, while adjustment parameters are based on published papers and author-provided code.
3.3 Experimental Results and Discussion
The experimental analysis compares 11 recent and 4 state-of-the-art algorithms on CEC 2017 functions at three dimensionalities, with results summarized in tables and supplemented by nonparametric tests.
- 11 recent and 4 state-of-the-art algorithms are compared on CEC 2017 functions with 10, 30, and 50 variables.
- Experimental results are summarized in Tables S1–S3.
- Friedman, Bayesian signed-rank, and Wilcoxon signed-rank tests are used for statistical analysis.
- The main text presents analysis results for functions with 30 variables because of space limitations.
- Results for functions with 10 and 50 variables are provided in Sections 3 and 4 of Appendix A in the supplementary material.
3.3.1 Benchmark functions with 30 variables
On CEC 2017 functions with 30 variables, EBCM is the strongest of the 11 newly proposed algorithms and performs competitively with established state-of-the-art variants. Most other recent algorithms show weaker convergence, global search ability, or exploration–exploitation trade-offs.
- Overall comparison: EBCM is the best-performing algorithm among the 11 newly proposed methods and is competitive with the four state-of-the-art algorithms.It completely surpasses HSES and is comparable to ED-EB and LS-SPA.
- Overall comparison: 21/14/14/5 functions are improved by EBCM over HSES, ED-EB, LS-SPA, and NLSHADE, respectively.
- Algorithm-specific results: GSK is superior to HSES, ED-EB, LS-SPA, and NLSHADE on 15, 3, 5, and 2 functions, respectively.GSK is equivalent to the four state-of-the-art algorithms except NLSHADE on F22 and is superior or similar to all five comparison algorithms on F4 and F28.
- Algorithm-specific results: MPA surpasses all four state-of-the-art algorithms on composition function F26.MPA achieves better results than the four baselines on seven, three, five, and one functions, respectively.
- Algorithm-specific results: EO, AO, HGSA, IGOA, MSCA, and SDCS are superior to the four state-of-the-art algorithms on fewer than three functions.These algorithms are described as less efficient on the 30-variable CEC 2017 functions.
- Convergence and search behavior: The four state-of-the-art algorithms generally converge faster and search globally better than the other 11 new algorithms on most selected functions.EBCM and the four state-of-the-art algorithms obtain better solutions on selected functions, while several newer methods converge more slowly or have poorer global search ability.
3.3.2 Results of CEC 2017 functions considering nonshifted and shifted
The shifted-versus-nonshifted experiments show that all 15 algorithms have search bias toward the origin, but the four state-of-the-art algorithms are less affected by shifting. Several recent algorithms improve on nonshifted functions yet deteriorate substantially when the optimum is shifted.
- Search bias: SDCS, MSCA, and MFLA are significantly affected by the shift operator, indicating origin-biased searches.AO, SDCS, MSCA, and MFLA also show significantly improved performance on nonshifted functions with 10 and 30 variables.
- Search bias: All 15 algorithms show search bias toward the origin, with the four state-of-the-art algorithms less affected by shifting than the 11 recent algorithms.
- Interpretation: The authors conclude that search mechanisms substantially influence performance and identify this relationship for future investigation.
4 Issues and suggestions for future research
The paper identifies reproducibility, mechanism-centered design, automated configuration, fair comparisons, machine learning integration, domain knowledge, and real-world scalability as priorities for future metaheuristic research.
- Reproducibility and comparison: Fair evaluation requires strong established variants, representative benchmarks with rotated and shifted problems, and rigorous statistical tests.The paper notes that stochastic algorithms can produce different-quality results across runs.
- Reproducibility and comparison: Incomplete algorithm descriptions can cause inaccurate replication and inconsistent computational results.The paper recommends making source codes publicly available.
- Algorithm design: Future algorithms should be improved from search behaviors and optimization mechanisms, not only from their inspiration sources.Neighborhood structures, operators, search direction, and exploration–exploitation balance are identified as important mechanisms.
- Automated configuration: Automatic design and configuration can reduce human effort and adapt algorithm components to specific problem characteristics.Operators, neighborhoods, parameters, information sharing, and learning mechanisms may be selected and combined from a component pool.
- Machine learning integration: Machine learning may recommend algorithms, model and decompose problems, tune parameters dynamically, and configure multiple operators.
- Real-world applications: Real-world optimization remains challenging because problems are often large-scale, constrained, high-dimensional, and mixed continuous–discrete.The paper suggests combining metaheuristics with surrogate models, parallel acceleration, and simulation optimization.
5 Conclusions
The paper catalogs more than 500 metaheuristics, proposes a taxonomy, and evaluates 15 algorithms under unified tuning on CEC2017. EBCM is comparable to four state-of-the-art variants, while most other recent algorithms perform worse and show stronger limitations under transformations.
- More than 500 newly proposed and improved metaheuristics are collected and organized into a taxonomy based on inspiration sources and essential solution-generating operators.
- The study compares 11 newly named metaheuristics with four state-of-the-art algorithms on CEC2017 using unified irace-based parameter configuration and nonparametric statistical tests.Convergence, diversity, exploration–exploitation trade-offs, and origin bias are also analyzed.
- EBCM performs similarly to the four compared algorithms across convergence, diversity, exploration, and exploitation trade-offs.
- All 15 algorithms show some search bias toward the origin, but the four state-of-the-art algorithms are less affected by shift transformations.
- Except for EBCM, the other 10 new algorithms are inferior to the four state-of-the-art algorithms in convergence speed and global search ability on most CEC2017 functions.They also exhibit rougher exploration–exploitation trade-offs and diversity.
- The authors identify further needs for broader benchmark and real-world comparisons, theoretical analysis, and study of parameters or strategies affecting origin bias.