A Prescription of Methodological Guidelines for Comparing Bio-inspired Optimization Algorithms

Motivation

• Many bio-inspired algorithm proposals fail to establish their contribution because experimental flaws, insufficient validation, and weak result discussions undermine confidence in their conclusions.The paper argues that rigorous experimental practices are needed for the community to accept conclusions and achieve meaningful advances.Source: §1 · pp. 2–3

Contribution

• The paper reviews literature on bad practices in algorithm analysis and provides four pragmatic methodological guidelines to help researchers prepare higher-quality proposals.The guidelines target common experimental mistakes and are intended especially to support new researchers entering the field.Source: §1 · pp. 3, 5

Guideline #1: Benchmarks

• Benchmark selection must account for test-problem characteristics because biased benchmarks can favor particular algorithms and compromise fair comparisons.The benchmark may represent a real-world problem or provide a test bed for comparing reference algorithms, but conclusions depend deeply on its design.Source: §1 · p. 3

Guideline #2: Validation of the results

• Result validation should combine appropriate statistical tests with visualization, since raw full-page tables alone cannot adequately establish comparative performance.Parametric tests should be used only when their assumptions are satisfied, while visualizations can condense information for rapid interpretation.Source: §1 · pp. 3–4

Guideline #3: Components analysis and parameter tuning of the proposal

• Proposal analysis should state hypotheses clearly and examine search phases, component contributions, complexity, parameter tuning, and statistical comparisons with state-of-the-art algorithms.The hypotheses should be revisited after results are validated, with individual components assessed for their contribution to the overall method.Source: §1 · p. 4

Guideline #4: Why is my algorithm useful?

• Authors should explain why their algorithm is useful, including competitiveness with state-of-the-art methods or methodological contributions that stimulate further research.The case study demonstrates the proposed process by selecting a CEC’2013 LSGO benchmark, ranking measure, reference algorithms, statistical validation, visualizations, component analysis, and usefulness discussion.Source: §1 · pp. 4–5

2.1. Inappropriate description of the algorithm

• Reproducibility requires complete algorithm descriptions, including implementation details, dependencies, parameter values, differences from base methods, and preferably source-code release.Missing implementation details can prevent replication, especially for advanced methods.Source: §2.1 · p. 6

2.2. Bias in the search process

• Benchmarks should avoid favoring algorithms through search bias, including central optima or coordinate-system sensitivity, while exploiting known problem-specific bias can be appropriate [7].Shifting optima and rotating functions support fairer comparisons, whereas real-world solvers should exploit relevant prior information.Source: §2.2 · pp. 6–7

2.3. Relevant features of benchmarks

• Fair benchmark selection should address separability, dimensionality, multimodality, and conditioning because these landscape properties affect problem difficulty and solver evaluation [22, 23, 24, 25, 26, 27, 28, 29, 30, 31].Benchmarks should reflect the proposal’s target characteristics, including subcomponents, dimensionality, local optima, and ill-conditioning.Source: §2.3 · pp. 7–9

2.4. Validation of the results

• Comparisons should include competitive state-of-the-art references rather than only basic or similar algorithms, because reported rankings depend strongly on the problem.Statistical validation should use methods appropriate to the data, including non-parametric tests when parametric assumptions fail.Source: §2.4 · pp. 9–10
• Existing experimental-design recommendations complement these guidelines, which specifically address bio-inspired optimization beyond broader frameworks or narrower advice on problems, measures, and benchmarking.CEC special sessions and COCO address experimental frameworks from a different but complementary perspective.Source: §2.4 · p. 10

2.5. Components analysis and parameter tuning

• New components must be justified through component analysis, since additions or replacements can create complex algorithms with only marginal performance contributions.Parameter tuning is itself an optimization problem; established tools include F-RACE, I-RACE, REVAC, and ParamILS.Source: §2.5 · pp. 10–11

3. Guideline #1: Benchmarks

• Researchers should choose both the benchmark and the reference algorithms before evaluating a new optimization contribution.These decisions are presented as crucial to guaranteeing the significance of resulting conclusions.Source: §3 · p. 11

3.1. Selection of the benchmark

• Using standard benchmarks and tools is preferable because author-defined function subsets can introduce bias and make independent assessment of algorithm quality nearly impossible.Ad-hoc instances are justified only for previously unstudied problems or specific real-world constraints; then generation must be detailed, instances shared, and the benchmark realistic and general.Source: §3.1 · pp. 13–14
• Benchmarks should match the algorithmic characteristics under study and define comparable experimental conditions, including performance measures and a common stopping criterion.Relevant characteristics include search bias, the number of local optima, evaluation effort, and measures such as mean error, running time, or memory footprint, with external measurement biases considered.Source: §3.1 · pp. 14–15

3.2. Selection of the reference algorithms

• Comparisons should include component algorithms underlying the proposal, best-so-far methods for the selected benchmark, and leading algorithms from the benchmark’s originating special session.This enables assessment of individual contributions and informed comparison with competitive counterparts.Source: §3.2 · p. 15
• Authors should compare new methods with similar algorithms, base methods, and related approaches rather than classic algorithms already clearly outperformed by newer ones.The guideline identifies comparisons using obsolete baselines as misleading and scientifically questionable.Source: §3.2 · p. 16

4. Guideline #2: validation of the Results

• Validation should combine statistical analysis with comparative visual analysis to support principled assessment of benchmark results.The guideline presents these as complementary tools, covered through statistical tests and visualization techniques for comparing meta-heuristics.Source: §4 · p. 16

4.1. Statistical analysis: non-parametric tests and beyond

• Check assumptions before choosing a parametric test: use Student’s t-test when normality and equal variances are satisfied; otherwise, non-parametric tests should be used instead.Assess normality with the Shapiro-Wilk or Kolmogorov-Smirnov test, using Shapiro-Wilk for smaller samples, and assess equal variances with Levene’s test.Source: §4.1 · p. 17

4.2. Visualization techniques for comparative analysis

• Visualizations make comparisons easier to interpret and can summarize tables through rankings, best-result frequencies, convergence curves, solved-problem ratios, and boxplots.Different visual idioms support different analyses: convergence plots can reveal changing performance over time, while boxplots show both mean error and variation across runs; visualizations should complement other reporting methods.Source: §4.2 · pp. 19, 21, Figure 1 · p. 20, Figure 2 · p. 21

5.1. Origin, hypotheses and proposal

• Authors should state the proposal’s hypotheses, explain how it targets its objectives, and identify every intended contribution before presenting benefits.The main contribution may be improved benchmark performance, but any other relevant contribution must also be clearly highlighted.Source: §5.1 · p. 23

5.2. Search phases identification

• Claims about exploration–exploitation balance require quantitative empirical evidence rather than unsupported assertions about an algorithm’s superiority.Evidence can use explicit evolutionary operators or indirect search feedback, building on the taxonomy and measurement analysis in.Source: §5.2 · pp. 23–24

5.3. Components analysis and simplicity/complexity

• Each added or modified component must be statistically tested for a significant contribution, and weak improvements should be removed to preserve simplicity.Analysis of L-SHADE-EpSin found that only initializing F to 0.5 during the first half of the search significantly improved the base L-SHADE, while local-search additions did not.Source: §5.3 · p. 25
• MOS-SOCO2011 solved 14 functions to maximum precision versus 1 for DE and 4 for MTS-LS1, generally matching or outperforming its components except on Schwefel 2.21 and Schwefel 1.2.The component results are presented in Table 2, and significant p-values were reported for MOS-SOCO2011 versus both DE and MTS-LS1 [79, 88, 89].Source: §5.3 · pp. 25–26, Table 2 · p. 26

5.4. Parameter tuning and analysis

• Parameter tuning should account for interacting parameters, test configurations rather than parameters independently, and use statistical tests when comparing configurations because average fitness alone is insufficient.Source: §5.4 · pp. 28–29

Purpose of the discussion

• A thorough results discussion must explain why the algorithm is useful and clearly state the benefits of adopting it, especially when it does not outperform the best-so-far methods.Claims should be rigorous, unambiguous, and supported by principled validation.Source: §6 · pp. 30–31

Quality of results

• Quality can justify acceptance through strong overall results or particularly good performance on a clearly identified and characterized subset of problems.Such cases do not exempt authors from the remaining methodological guidelines.Source: §6 · p. 31

Novelty

• Novelty may justify a proposal if it can become competitive, but a new metaphor alone is insufficient and the method must remain competitive on some problems.The authors advocate standardized descriptions of meta-heuristic components and interfaces to assess similarities and differences objectively [106].Source: §6 · p. 31

Methodological relevance

• Methodological relevance can reside in an important algorithmic component or framework even when the complete algorithm is not fully competitive.Authors should select benchmarks and frameworks that expose the intended contribution, such as CEC’2013 for subcomponent identification, and explicitly state these benefits.Source: §6 · pp. 31–32

7.1 Existing algorithms

• The first scenario compares existing algorithms using multiple criteria to determine the circumstances under which each method can be recommended.Subsection 7.1 applies the guidelines to several recent bio-inspired algorithms for real-parameter optimization.Source: §7 · p. 32

7.2 New algorithm

• The second scenario evaluates a new algorithm by providing informed evidence of its competitiveness and advantages over previous algorithms.Subsection 7.2 considers a new algorithm for large-scale global optimization.Source: §7 · p. 32

7.1.1. Selecting the benchmark as per Guideline #1

• The study selects CEC’2017 and COCO because their complementary designs assess final error, evaluation efficiency, solved-problem counts, and other performance perspectives without unexpected benchmark bias.CEC’2017 reports errors at evaluation milestones, whereas COCO tracks the fraction of functions reaching the optimum as evaluations increase.Source: §7.1.1 · p. 35, §7.1.2 · p. 36

7.1.2. Selecting the performance measure as per Guideline #1

• Performance is reported per function and aggregated through average rankings, with lower rankings indicating better algorithms and final scores supplemented by search-evolution measures.The analysis emphasizes the 100% evaluation milestone while retaining information about earlier progress.Source: §7.1.2 · p. 36

7.1.3. Selecting the reference algorithms as per Guideline #1

• The comparison includes each algorithm, classic PSO, and competitive references: jSO and EBOwithCMAR for CEC’2017 and a previous COCO competition winner.Because the evaluated algorithms have different inspirations and behavioral patterns, no additional similar algorithms are included beyond PSO.Source: §7.1.3 · p. 37

7.1.4. Experimenting and validating the results as per Guideline #2

• ELSHADE-SPACMA is the only recent proposal outperforming CEC’2017 winners jSO and EBOwithCMAR, while GSKA and SSA follow among recent methods and PSO generally performs worse except against AEO.These conclusions come from average rankings across dimensions, with ELSHADE-SPACMA particularly strong for D50 and D100.Source: Table 3 · p. 38, §7.1.4 · pp. 38–39
• For dimensions 30 and 50, GSKA is much faster and clearly better than all other proposals up to 10% of the fitness evaluations, including EBOwithCMAR, jSO, and ELSHADE-SPACMA.Source: §7.1.4 · p. 39
• CEC’2017 differences are statistically significant at all dimensions, with Friedman p-values of 3.61e-08, 1.51e-07, 7.58e-7, and 9.62e-10.The values correspond to dimensions 10, 30, 50, and 100, respectively, all below α = 0.05.Source: §7.1.4 · p. 44
• ELSHADE-SPACMA outperformed EBOwithCMAR and jSO overall, becoming the state-of-the-art method on CEC’2017, while COCO’s 2009 winner remained superior.On CEC’2017, statistical differences among the leading methods appeared only at dimension 100 and were confirmed after Holm correction; on COCO, ELSHADE-SPACMA remained competitive but did not surpass the 2009 winner.Source: §7.1.4 · pp. 47, 48, 49, 51, Table 12 · p. 48
• GSKA performed best at low evaluation budgets and small dimensions, whereas EBOwithCMAR and jSO improved with larger budgets and ELSHADE-SPACMA strengthened as dimensionality increased.Overall, GSKA exceeded AEO and SSA but lost its advantage as dimension increased; recent proposals remained competitive with PSO yet generally did not match modern competition-winning solvers.Source: §7.1.4 · pp. 50–51

7.1.5. Components analysis and Tuning as per Guideline #3

• SSA tuning favored attraction factor 0.7 and swarm size 100, improving results in dimensions 10 and 30 but not changing SSA’s relative ranking.The tuned configuration was best for dimensions 10 and 50 and second-best for dimension 30; larger swarms and medium-range attraction factors generally performed better, while dimension 100 was not tuned.Source: §7.1.5 · pp. 54–55

7.1.6. Justifying the usefulness of the algorithm as per Guideline #4

• The algorithms’ usefulness rests on ELSHADE-SPACMA’s quality, GSKA’s low-budget performance, and distinct mechanisms in ELSHADE-SPACMA, SSA, and GSKA, despite no methodological contribution.SSA is notably simple but may converge prematurely; its restart and category-based mutation strategies, alongside the other algorithms’ diversity, selection, and neighborhood mechanisms, provide ideas for future search methods.Source: §7.1.6 · pp. 55–56

7.1.7. Summary of the use case

• The use case applied accepted CEC’2017 and COCO benchmarks, competitive competition-winning solvers and PSO, non-parametric and Bayesian validation, tuning, and guideline-based justification.It also showed that stopping criteria and parameter values can bias conclusions; only ELSHADE-SPACMA surpassed existing competitive algorithms, while GSKA excelled at low evaluation counts.Source: §7.1.7 · pp. 56–57

7.2.1. Selecting the benchmark as per Guideline #1

• The CEC’2013 benchmark is selected because it is popular, openly implemented, shifted away from centered optima, and includes varied local optima and variable interrelations.Its experimental conditions are clearly defined and identical across algorithms, although its functions are not rotated.Source: §7.2.1 · pp. 58–59

7.2.2. Selecting the performance measure as per Guideline #1

• Performance is summarized primarily at the maximum evaluation budget using final fitness error and aggregate measures such as average ranking.The study notes that analogous analysis should be performed at each accuracy level, and also considers the benchmark’s ranking score.Source: §7.2.2 · pp. 59–60

7.2.3. Selecting the reference algorithms as per Guideline #1

• The comparison includes IHDELS as a similar method, DECCG as a reference algorithm, and MOS-CEC2013 and MLSHADE-SPA as competitive algorithms.MOS-CEC2013 had dominated competitions, while MLSHADE-SPA was reported to outperform it in 2018; the comparison focuses on methods available by 2018.Source: §7.2.3 · p. 60

7.2.4. Testing and validating the results as per Guideline #2

• SHADE-ILS slightly outperforms MLSHADE-SPA and MOS-CEC2013, and substantially outperforms IHDELS and DECCG on the CEC’2013 LSGO benchmark.Its average ranking is significantly better than IHDELS, despite IHDELS previously performing worse than MOS-CEC2013.Source: Table 17 · p. 61, §7.2.4 · p. 61
• The Friedman test finds significant overall differences (p = 4.87e-03), while pairwise differences are significant versus DECCG and IHDELS but not versus MLSHADE-SPA or MOS-CEC2013 after Holm correction.At a 10% confidence level, the difference with MOS-CEC2013 becomes statistically significant.Source: §7.2.4 · p. 62, Table 18 · p. 62
• SHADE-ILS achieves the best average ranking at 600,000 and 3,000,000 fitness evaluations, with differences increasing as evaluations rise.Figure 8 compares average rankings across fitness-evaluation counts; MLSHADE-SPA and SHADE-ILS improve from 600,000 evaluations onward.Source: Figure 8 · p. 63, §7.2.4 · p. 63

7.2.5. Components analysis and tuning as per Guideline #3

• SHADE-ILS’s improved performance results from both the SHADE-based Differential Evolution modification and the restart mechanism, rather than one component alone.The component comparison indicates that these contributions jointly produce the outperforming behavior without increasing overall search-process complexity.Source: §7.2.5 · pp. 63–64, Table 19 · p. 64

7.2.6. Justifying the usefulness of the algorithm as per Guideline #4

• SHADE-ILS’s usefulness is supported by quality, technical novelty, and methodological contribution, including improved hybridization of Differential Evolution with local search and a simple model.Its parameter tuning was manual because of computational constraints, while the method retains fewer parameters than other proposals.Source: §7.2.5 · p. 64, §7.2.6 · p. 65, §7.2.7 · p. 66

• The methodology is intended to guide both benchmark studies and the design of meta-heuristic algorithms for real-world optimization problems, following the tutorial.For real-world applications, statistical validation should be complemented by other perspectives and practical measures such as memory consumption.Source: §8 · p. 67
• The framework recommends standard or appropriately characterized benchmarks, problem-linked performance measures, and suitable reference algorithms for comparisons.These practices address benchmark bias, insufficiency, modality, and the need for fair comparisons.Source: §8 · p. 66
• Figure 9 summarizes the methodological framework proposed to ensure fairness, coherence, and soundness in future meta-heuristic studies.Two case studies exemplify applying the prescribed guidelines and discuss the results of each guideline.Source: Figure 9 · p. 67, §8 · p. 67
• The authors call for community agreement on systematic testing and comparison practices as a basis for future scientific advances and newcomers’ work.The recommendations consolidate practices reviewed and critically analyzed across the literature, spanning experimentation through results presentation.Source: §8 · pp. 66–68