Source-linked AI summary
A Prescription of Methodological Guidelines for Comparing Bio-inspired Optimization Algorithms
Antonio LaTorre, Daniel Molina, Eneko Osaba, Javier Del Ser, Francisco Herrera
TL;DR
Bio-inspired optimization proposals often lack fair benchmarks, competitive reference algorithms, principled validation, or adequate justification of their contribution. This paper reviews these problems and proposes four methodological guidelines, illustrated through case studies, to improve proposal design and evaluation. It concludes that novelty should be accompanied by rigorous comparisons and explicit evidence supporting the proposal’s usefulness.
Problem
Many new proposals do not adequately select benchmarks or reference algorithms, validate results, or demonstrate a significant advance over existing outcomes.
Method
The paper reviews literature and bad practices, then proposes four guidelines covering benchmarks, validation, component and parameter analysis, and usefulness.
Results
The guidelines are illustrated through case studies, including analyses of component contributions and comparisons with competitive algorithms.
Takeaways & Limitations
Successful proposals should justify their relevance through fair experimental design, appropriate comparisons, and evidence supporting their claimed contributions.
Takeaways & Limitations
Claims about exploration–exploitation balance require empirical analysis, and tuning only the new proposal can give it an unfair comparative advantage.
Abstract
from arXiv · showhide
Bio-inspired optimization (including Evolutionary Computation and Swarm Intelligence) is a growing research topic with many competitive bio-inspired algorithms being proposed every year. In such an active area, preparing a successful proposal of a new bio-inspired algorithm is not an easy task. Given the maturity of this research field, proposing a new optimization technique with innovative elements is no longer enough. Apart from the novelty, results reported by the authors should be proven to achieve a significant advance over previous outcomes from the state of the art. Unfortunately, not all new proposals deal with this requirement properly. Some of them fail to select appropriate benchmarks or reference algorithms to compare with. In other cases, the validation process carried out is not defined in a principled way (or is even not done at all). Consequently, the significance of the results presented in such studies cannot be guaranteed. In this work we review several recommendations in the literature and propose methodological guidelines to prepare a successful proposal, taking all these issues into account. We expect these guidelines to be useful not only for authors, but also for reviewers and editors along their assessment of new contributions to the field.
1. Introduction
The paper reviews recurring weaknesses in bio-inspired optimization proposals and prescribes four methodological guidelines for credible, community-accepted contributions. It illustrates the guidelines through use cases and a case study covering proposal design, evaluation, component analysis, and usefulness.
- The mature field receives many proposals, but inadequate proof of algorithm quality can obscure their true contribution and significance.
- Existing recommendations address specific experimental issues, yet the paper identifies a need to consider multiple threats to fair performance comparisons together.
- The authors propose four guidelines covering benchmark selection, result validation, component and parameter analysis, and justification of an algorithm’s usefulness.
- A case study simulates proposing and evaluating a new algorithm while analyzing component contributions and discussing the proposal’s usefulness.
- The contribution combines a literature review of bad practices with four methodological guidelines intended to help authors produce contributions adopted by the community.
- The paper organizes its detailed guidelines in Sections 3–6 and presents case studies in Section 7 before concluding in Section 8.
2. Relevant issues for the proposal of methodological guidelines
The reviewed issues concern biased or incomplete experimentation, benchmark design, validation, reproducibility, component justification, and parameter tuning. Together, they motivate methodological guidance specific to bio-inspired optimization.
- Correct and unbiased experimental conditions are necessary because small design choices can inadvertently benefit particular algorithm types.
- The literature review covers incomplete algorithm descriptions, search-process bias, benchmark selection, and other issues relevant to evaluating bio-inspired methods.
- Reproducibility requires implementation details, dependencies, parameter values, and clear differences from any pre-existing base method.
- Benchmark design can favor newly proposed methods through centered optima, coordinate-system sensitivity, separability, dimensionality, multimodality, or ill-conditioning.
- Standardized benchmarks and tools support fairer comparisons, whereas author-selected function subsets make selection bias difficult to assess.
- New components added to existing algorithms require explicit contribution analysis because otherwise they may increase complexity while contributing only marginally.
- Parameter-tuning tools such as F-RACE and I-RACE evaluate candidate configurations iteratively and discard candidates using statistical comparisons.
3. Guideline #1: Benchmarks
Guideline #1 requires selecting benchmarks and reference algorithms that support fair, informative comparisons. Benchmark design should expose relevant algorithm characteristics while comparisons should include competitive and structurally similar methods.
- Benchmark and reference-algorithm selection are the two central decisions for testing a new proposal and supporting significant conclusions.
- Selection of the benchmark: Benchmark choice strongly affects conclusions because testbeds may contain features that favor particular algorithmic biases.
- Selection of the benchmark: Standard benchmarks and comparison tools provide controlled, homogeneous testbeds, including COCO, TACO, CEC benchmarks, and IEEE CEC LSGO.
- Selection of the benchmark: Author-selected function subsets are problematic because their performance bias is difficult to determine and many alternative benchmark functions exist.
- Selection of the benchmark: When ad-hoc instances are unavoidable, researchers should detail the generation process, share instances, and make the benchmark realistic and general.
- Selection of the benchmark: Benchmarks should test bias avoidance, local-optima sensitivity, performance measures, and comparable stopping effort across algorithms.
- Selection of the reference algorithms: Reference comparisons should include component-based algorithms, best-so-far methods, algorithms from relevant special sessions, and similar or base methods rather than outdated classics.
4. Guideline #2: validation of the Results
Guideline #2 recommends principled validation through statistical analysis and comparative visual analysis. It emphasizes matching tests to data assumptions, controlling accumulated error, and using visualizations that expose rankings, convergence, and run-to-run variation.
- Validation approach: Validation should combine statistical analysis with comparative visual analysis to assess algorithm performance fairly.The guideline treats both approaches as complementary tools for validating benchmark results.
- Test selection: Parametric tests require checking assumptions such as normality and equal variances; otherwise, non-parametric alternatives should be used.The workflow checks normality and homocedasticity before applying Student’s t-test, while Welch’s t-test or Wilcoxon’s signed-rank test is recommended when assumptions are only partly or not satisfied.
- Test selection: Student’s t-test is recommended when both normality and equal-variance conditions are satisfied.Normality can be assessed with Shapiro-Wilk or Kolmogorov-Smirnov tests, and equal variances with Levene’s test.
- Test selection: If no distributional assumptions hold, the Wilcoxon signed-rank test is recommended instead.This non-parametric choice avoids assuming particular characteristics of the underlying data distribution.
- Multiple-algorithm comparison: Algorithm comparisons should first use Friedman ranking across the benchmark, followed by corrected pairwise comparisons when differences are significant.The reference algorithm is typically the method with the best average ranking or the newly proposed method.
- Comparative visualization: Visualizations summarize comparative results through rankings, best-result fractions, convergence speed, effort-to-optimum plots, and boxplots of run variation.These views can be easier to interpret than raw tables and can expose different aspects of algorithm behavior.
5. Guideline #3: Components Analysis and Parameter Tuning of the Proposal
Guideline #3 requires authors to state hypotheses and analyze how search phases, components, complexity, and parameter choices contribute to results. The paper also stresses that tuning and component comparisons must support fair, interpretable conclusions.
- 5.1. Origin, hypotheses and proposal: Authors should state the proposal’s hypotheses before experimentation and revisit them after validating the results.
- 5.2. Search phases identification: Exploration and exploitation claims require empirical evidence rather than unsupported assertions about search behavior.The paper notes that such claims should be checked quantitatively, often using indirect measures such as solution diversity.
- 5.3. Components analysis and simplicity/complexity: Each component and complexity contribution should be analyzed individually, with statistically validated evidence that every addition improves overall behavior.Components whose contribution is small should be considered for removal to preserve simplicity.
- 5.3. Components analysis and simplicity/complexity: Hybrid algorithms should be compared with their composing methods; MOS-SOCO2011 was reported to reach the best component’s performance and normally outperform its components.The cited comparison used MOS-SOCO2011, DE, and MTS-LS1 on 1000-D functions, with significant p-values reported for both pairwise comparisons.
- 5.4. Parameter tuning and analysis: Parameter tuning is critical because incorrect values can make even a well-designed algorithm perform poorly, and tuning choices can bias comparisons.The paper discusses self-adaptation, offline tuning, partial tuning, independent tuning, and the need to tune reference algorithms comparably when feasible.
6. Guideline #4: Why is my algorithm useful?
Guideline #4 asks authors to explain why their algorithm is useful, whether through validated state-of-the-art performance, novelty with competitiveness, or methodological relevance. A proposal can remain significant without overall superiority when its benefits or components are clearly justified.
- A proposal has clear scientific value when it outperforms state-of-the-art methods and validates that advantage through principled analysis.
- Failure to outperform the best-so-far algorithms does not by itself make a proposal insignificant, but its benefits must then be stated clearly.
- Acceptable contributions may involve particularly strong results on characterized problem subsets, credible novelty with competitiveness, or a relevant methodological component.The paper specifically identifies algorithmic building blocks such as local optimizers as potentially valuable even when the complete algorithm is not fully competitive.
7. Case Studies
The case studies apply the proposed methodology to both comparisons of existing algorithms and the design of a new algorithm. They emphasize fair comparison, informed evidence of competitiveness, and advantages over previous methods.
- One case-study scenario compares existing algorithms to identify their advantages, drawbacks, and the circumstances under which each can be recommended.
- A second scenario evaluates a new algorithm by providing evidence of its competitiveness and advantages over previous algorithms.
- The case-study section first fairly compares recent real-parameter algorithms, then examines a new algorithm for large-scale global optimization.
7.1. Several modern bio-inspired algorithms for real-parameter optimization
The case study compares modern bio-inspired algorithms across benchmarks, dimensions, evaluation budgets, and statistical procedures. ELSHADE-SPACMA is generally the strongest recent proposal, while performance depends on dimensionality and evaluation budget.
- Comparative performance: ELSHADE-SPACMA is the only new proposal that consistently competes with previous winners, especially on higher-dimensional problems.It is reported as statistically better than previous winners for dimension 100 and as outperforming jSO and EBOwithCMAR overall.
- Comparative performance: ELSHADE-SPACMA is best from 40% of the evaluation budget for dimensions 30, 50, and 100, although EBOwithCMAR attains the best final results.This distinguishes performance during most of the search from the final outcome.
- Comparative performance: For lower evaluation budgets, GSKA performs best, whereas EBOwithCMAR, jSO, or ELSHADE-SPACMA become preferable when more evaluations are available.The preferred algorithm therefore changes with the effort allocated to the search.
- Statistical validation: Statistical testing finds significant overall algorithm differences, but pairwise differences among ELSHADE-SPACMA, EBOwithCMAR, and jSO occur only at dimension 100.The Friedman p-values are 3.61e-08, 1.51e-07, 7.58e-7, and 9.62e-10 for dimensions 10, 30, 50, and 100; the lower-dimensional comparisons show no significant differences among these three algorithms.
- Comparative performance: The CEC’2017 analysis shows two performance groups for dimensions 30, 50, and 100: EBOwithCMAR, ELSHADE-SPACMA, and jSO versus GSKA, SSA, PSO, and AEO.The rope analysis also reports no sampled cases where GSKA outperforms jSO under the selected threshold.
- Comparative performance: GSKA performs better than AEO and SSA and is especially advantageous at small dimensionalities, but its advantage decreases as dimensionality grows.The broader conclusions also report that the newer proposals remain competitive with classic PSO while failing to rival modern competition winners.
7.2. SHADE-ILS for large-scale global optimization
The SHADE-ILS case study applies the proposed methodology by selecting and evaluating a suitable standard benchmark, comparing related and state-of-the-art algorithms, and validating results statistically. SHADE-ILS achieves strong average rankings and significantly outperforms IHDELS and DECCG, while differences from MLSHADE-SPA and MOS-CEC2013 are not statistically significant at the stated levels.
- Benchmark selection: CEC’2013 LSGO was selected because it satisfies most methodological requirements, including varied local optima, differing variable interrelations, and publicly available implementations.The benchmark also uses consistent experimental conditions and avoids placing optima at the domain center.
- Algorithm comparison: SHADE-ILS was compared with similar algorithms and reference methods, including IHDELS, MLSHADE-SPA, MOS-CEC2013, and DECCG.Including IHDELS addressed the recommendation to compare modified proposals with conceptually related prior methods.
- Performance results: SHADE-ILS achieves a slightly better average ranking than MLSHADE-SPA and MOS-CEC2013, and a much better ranking than IHDELS and DECCG.The ranking is computed from the relative positions of the four algorithms across benchmark functions.
- Statistical validation: The Friedman rank-sum test reports p-value 4.87e-03, indicating significant differences at α = 0.05.Pairwise analysis finds significant differences involving SHADE-ILS, DECCG, and IHDELS, but not between SHADE-ILS and MLSHADE-SPA.
- Components and tuning: SHADE-ILS’s outperforming behavior is attributed to all its novel contributions, while those changes do not add complexity to the overall search process.The study also recommends automatic parameter tuning, but SHADE-ILS used manual tuning because of computational constraints.
- Usefulness of the proposal: The case study justifies SHADE-ILS through improved results over a previous hybridization and competitive performance against established methods, although some differences lack statistical significance.The proposed combination of local search and SHADE is presented as a possible stimulus for further research.
8. Conclusions and Outlook
The paper reviews recurring methodological flaws in bio-inspired optimization comparisons and organizes recommended practices into a unified framework. It illustrates the framework through two case studies and argues that shared standards for testing and comparison are needed for future progress.
- Contribution: The work critically reviews experimental recommendations and practices for proposing and validating new bio-inspired metaheuristics.Its framework aims to address common mistakes and flaws in benchmark-based experimental studies.
- Framework: The proposed framework seeks fairness, coherence, and soundness in future studies through a set of practical methodological techniques.The techniques are graphically summarized in Figure 9.
- Case studies: Two case studies demonstrate how the prescribed methodology can be applied to the design and evaluation of meta-heuristic algorithms.Both case studies use well-known benchmarks, while the authors envision broader use for real-world optimization.
- Recommendations: The guidelines recommend standard benchmarks when appropriate, checking benchmark properties, using problem-linked performance measures, and selecting suitable reference algorithms.These recommendations address benchmark bias, insufficiency, modality, and comparison quality.
- Outlook: The paper argues that scientific advancement requires agreement on systematic practices for testing and comparing algorithms.The guidelines are intended to support researchers, particularly newcomers entering the field.