PaperlayerView exact revision on arXiv

Source-linked AI summary

Recent Trends in the Use of Statistical Tests for Comparing Swarm and Evolutionary Computing Algorithms: Practical Guidelines and a Critical Review

J. Carrasco, S. García, M. M. Rueda, S. Das, F. Herrera

arXiv:2002.09227v1cs.NEstat.ME

TL;DR

Comparing evolutionary and swarm algorithms requires statistical analyses whose assumptions and interpretations support reliable conclusions. The paper surveys frequentist, non-parametric, and Bayesian methods, applies them to CEC’2017 results, and develops recommendations for choosing and combining tests. Its results show that statistical tests can leave a lead group of equivalent algorithms, while Bayesian analyses can clarify differences when non-parametric tests are inconclusive.

  • Problem

    Algorithm comparisons need reliable statistical conclusions, but different tests produce different results and have distinct assumptions, interpretations, and drawbacks.

  • Method

    The paper surveys statistical proposals and theoretical backgrounds, analyzes CEC’2017 single-objective optimization results, and compares frequentist, non-parametric, Bayesian, and confidence-curve procedures.

  • Results

    Statistical tests identify a lead group whose equivalence cannot be discarded across dimension scenarios, while in 100 dimensions DES beats the remaining algorithms with probabilities from 0.59 to 0.97.

  • Takeaways & Limitations

    Parametric tests should be used only when normality and homoscedasticity prerequisites are fulfilled, and non-parametric and Bayesian tests should be used jointly for a fuller comparison.

  • Takeaways & Limitations

    NHST p-values are often misinterpreted as the probability that the null hypothesis is true or as reproducibility probabilities.

Abstract

from arXiv · show

A key aspect of the design of evolutionary and swarm intelligence algorithms is studying their performance. Statistical comparisons are also a crucial part which allows for reliable conclusions to be drawn. In the present paper we gather and examine the approaches taken from different perspectives to summarise the assumptions made by these statistical tests, the conclusions reached and the steps followed to perform them correctly. In this paper, we conduct a survey on the current trends of the proposals of statistical analyses for the comparison of algorithms of computational intelligence and include a description of the statistical background of these tests. We illustrate the use of the most common tests in the context of the Competition on single-objective real parameter optimisation of the IEEE Congress on Evolutionary Computation (CEC) 2017 and describe the main advantages and drawbacks of the use of each kind of test and put forward some recommendations concerning their use.

1 Introduction

The paper surveys changing statistical approaches for comparing evolutionary and swarm algorithms, emphasizing assumptions, interpretation, and reproducible procedures. It contrasts frequentist, non-parametric, and Bayesian perspectives and introduces effect sizes, confidence curves, and multiple-error concerns.

  • Statistical comparison workflow: Algorithm results are treated as samples from unknown distributions whose estimated parameters or performance differences are compared statistically.The procedure aggregates run results into statistics before drawing conclusions.
  • Statistical comparison workflow: Statistical tests should function as a toolbox for extracting relevant information rather than confirming a previously stated conclusion.The authors stress impartiality and reproducibility of the procedure.
  • Frequentist and Bayesian paradigms: Frequentist NHST compares hypotheses about algorithm populations, using α as a threshold against which the p-value is evaluated.A p-value below α leads to rejection of the null hypothesis; α is commonly 0.05.
  • Frequentist and Bayesian paradigms: Parametric NHST can reject equal means even when fitted distributions overlap, motivating effect size as a measure of practical difference.The example compares DYYPO and TLBO-FL on one benchmark under Gaussian assumptions.
  • Frequentist and Bayesian paradigms: Non-parametric tests avoid assuming normal input data, while Bayesian analysis estimates the distribution of performance differences directly.Bayesian statements concern the parameter distribution rather than a single probability and can clarify differences when NHST finds no significance.
  • Paper scope: The paper synthesizes statistical viewpoints and adapts confidence-curve calculation non-parametrically to assess test appropriateness across situations.It also discusses convergence, robustness across runs, confidence intervals, and multiple-comparison errors.

2 Survey on Statistical Analyses Proposed

The survey organizes statistical proposals by their nature, underlying idea, comparison scenario, and publication year. It shows a historical movement from parametric frequentist methods toward non-parametric and Bayesian approaches, while identifying limited coverage of repeated optimization runs.

  • Survey structure: Table 2 surveys statistical proposals by statistic nature, underlying idea, and comparison scenario.The proposals are organized as an extensive overview of machine-learning and optimization algorithm analyses.
  • Historical trends: The proposals are sorted by publication year to highlight changes from parametric frequentist methods to non-parametric and later Bayesian approaches.Bayesian proposals remain dependent on result distributions and focus on classifier comparisons.
  • Coverage and gaps: Most surveyed proposals compare classifiers rather than optimization algorithms.The paper notes that optimization guidelines are used in the literature despite this distribution of proposals.
  • Coverage and gaps: No specific test in the surveyed proposals accounts for multiple runs on the same benchmark function while estimating correlations between those runs.The stated gap concerns repeated-run structure in optimization experiments.

3 Frequentist tests

This section presents parametric and non-parametric frequentist tests for comparing evolutionary optimization algorithms, together with their assumptions, inputs, and uses for pairwise, multiple-algorithm, and convergence analyses.

  • Parametric tests: Parametric tests assume observations come from a known distribution family, usually Gaussian, described by a small number of parameters.Normality and homoscedasticity are key prerequisites examined before using tests such as t-test or ANOVA.
  • Parametric tests: The t-test compares two algorithm samples, assuming random extraction and normally distributed populations.Its input is the run observations for two algorithms on one problem.
  • Parametric tests: ANOVA compares k algorithms through the null hypothesis that all means are equivalent, avoiding repeated pairwise t-tests that increase type I error.Its input matrix contains algorithms by columns, benchmark functions by rows, and mean run performance in cells.
  • Non-parametric tests: Non-parametric tests use less restrictive assumptions and are more robust and less sensitive to dirty data than parametric tests.They may still require conditions such as symmetry for the Wilcoxon signed-rank test.
  • Non-parametric tests: The sign test counts wins, whereas the Wilcoxon signed-rank test ranks performance differences and is used when t-test prerequisites are not fulfilled.For two algorithms across benchmarks, these tests compare medians rather than means.
  • Multiple and convergence tests: Repeated pairwise comparisons inflate family-wise error, so multiple-algorithm analyses use alternatives such as Friedman, aligned-ranks, Quade, and multiple sign tests.Friedman ranks algorithm performance per problem and compares average ranks; aligned-ranks addresses conservatism when few algorithms are compared.
  • Multiple and convergence tests: The Page trend test evaluates convergence by testing ordered trends in performance differences at equidistant search cut points.Rejecting the null for A−B can indicate faster convergence by B, while testing B−A gives the analogous interpretation for A.
  • Multiple and convergence tests: Convergence conclusions can depend on the sign of initial differences, allowing a faster reduction rate to appear superior despite worse starting performance.This limitation concerns tests focused on trends rather than absolute performance levels.

3.3 Post-hoc Procedures

Post-hoc procedures adjust comparisons among multiple algorithms after an omnibus analysis, controlling error while exploiting logical constraints among hypotheses. Different procedures trade simplicity and power when comparing all algorithm pairs.

  • Purpose and hypothesis families: Omnibus multiple-comparison tests detect group differences but do not identify which algorithms differ, requiring a family of pairwise hypotheses.Comparisons may involve a control algorithm or all k algorithms, producing k−1 or k(k−1)/2 hypotheses.
  • Adjustment procedures: Adjusted p-values are required because singular p-values in multiple comparisons lose control of the family-wise error rate.Bonferroni-Dunn, Li, Holm, Holland, Hochberg, and Rom procedures are listed as adjustment methods.
  • Logical constraints: Bergmann–Hommel rejects hypotheses using exhaustive sets of hypotheses that could simultaneously be true.Its acceptance-set formulation accounts for logical relationships among the hypothesis family.
  • All-pairs procedures: Critical Difference plots order algorithms by mean rank and connect pairs whose equivalence is not rejected by the post-hoc test.The plots show discarded equivalences and groups without sufficiently different performance, but the method is less powerful than others.
  • Shaffer procedures: Shaffer’s static procedure uses logical constraints to set ti, the maximum number of hypotheses that can remain true after ordered rejections.Its correction depends on the given hypotheses rather than the p-values themselves.
  • Shaffer procedures: Shaffer’s dynamic procedure further increases power by making t*i depend on hypotheses already rejected.Its adjusted p-values use the current assignment of potentially true hypotheses.

3.4 Confidence Intervals and Beyond

The paper presents confidence intervals and confidence curves as alternatives or complements to p-values for comparing algorithm performance, while proposing a non-parametric interval based on Wilcoxon rankings.

  • Confidence intervals provide a suitable range for the parameter of interest, with narrower intervals indicating greater certainty about the real difference.They are commonly defined at a fixed confidence level, usually 95%, and incorporate effect-size information through interval width.
  • The proposed modification computes a non-parametric confidence interval for optimization comparisons from a ranking perspective using the Wilcoxon test.The interval estimates the difference between two medians from pairwise sample differences.
  • Confidence curves plot confidence intervals for all levels around the observed difference, reducing reliance on an arbitrary significance level.Their x-axis represents null hypotheses and their y-axis the associated p-values; wider curves indicate less certainty.
  • A single confidence-curve comparison combines information from NHST, a classic confidence interval, effect size, and intervals at other significance levels.Interpreting this richer output requires more attention than interpreting a single p-value.

4 Known Criticisms to Null Hypothesis Statistical Tests

The paper identifies several limitations of NHST for comparing optimization algorithms, including p-value misinterpretation, sensitivity to sample size, and ambiguity when the null hypothesis is not rejected.

  • A p-value gives P(D|H0), the probability of data at least as extreme under the null hypothesis, not P(H0|D).Confusing these probabilities leads researchers to treat a p-value as the probability that the null hypothesis is true.
  • Large samples can make very small differences statistically significant, while insufficient data can conceal differences that exist.This can encourage adding many benchmark functions until random and practically negligible differences produce rejection of the null hypothesis.
  • A lower p-value does not imply a higher probability that replicated experiments will produce the same results.After rejecting H0, the probability of obtaining significance depends on α, the true effect size, and test-set size rather than the p-value.
  • Failure to reject the null hypothesis provides no information that the compared algorithms have equivalent performance.The absence of a significant difference is therefore not evidence of equivalence.

5 Bayesian Paradigm and Distribution Estimation

The Bayesian paradigm estimates distributions of algorithm-performance differences rather than relying only on null-hypothesis decisions. The paper describes Bayesian procedures for pairwise and multiple-algorithm comparisons, including sign, signed-rank, and Friedman tests.

  • Bayesian comparison assigns credibility to candidate parameter values, including no difference, using observations and Bayesian inference.For optimization comparisons, this avoids reintroducing NHST objections through a null value in Bayesian model comparison.
  • Bayesian analysis combines a data model, a prior distribution, and Bayes’ rule to obtain the posterior distribution.The posterior represents updated information about performance differences between algorithms.
  • The posterior distribution can distinguish differing results from evidence that one algorithm consistently outperforms another.For example, an algorithm may perform better on one problem and worse on another, a distinction not supplied by a frequentist equivalence decision alone.
  • The Bayesian sign test estimates posterior probabilities that the performance difference lies below, within, or above a practical-equivalence region.The three regions are defined relative to the ROPE limit r, with probabilities computed from weighted observations.
  • Bayesian signed-rank probabilities use paired observations and can be estimated with Monte Carlo sampling.Unlike the sign test, this version does not have a simple distribution for the probabilities.
  • The Bayesian Friedman test generalizes Bayesian sign testing to comparisons among m ≥3 algorithms using a symmetric credible region for expected ranks.A difference is inferred with probability 1−γ when the equal-rank null point is excluded from the credible region.

6 Multiple Measures Tests

Multiple-measures comparisons address multi-objective algorithm evaluation, where differences across objectives and Pareto dominance matter. The paper contrasts a parametric multivariate test with a non-parametric and Bayesian dominance-based proposal.

  • Multi-objective comparisons can aggregate objective scores with a weighted sum or examine Pareto-frontier dominance.Pareto selection retains algorithms that are not worse than others across all criteria.
  • Hotelling’s T^2 compares two algorithms’ results across multiple measures under a multivariate Gaussian assumption.The normality assumption can be checked with a generalized Shapiro-Wilk test.
  • Rejecting Hotelling’s null hypothesis indicates that at least one measure differs between the algorithms.This result does not identify which measure differs or establish Pareto superiority.
  • The Multiple Measures Test represents each algorithm comparison across problems and objectives as a dominance statement.Each entry records whether one algorithm is better or worse on a particular performance measure.
  • Its null hypothesis states that the most frequently observed dominance configuration is no more probable than the second-most probable configuration.The Bayesian version estimates posterior probabilities of dominance statements with a Dirichlet prior and Monte Carlo sampling.

7 Experimental Framework

The experiments illustrate statistical comparisons using the CEC 2017 single-objective real-parameter optimisation competition, its benchmark functions, and thirteen contestant algorithms.

  • Competition and benchmarks: The framework applies previously defined statistical tests to results from the CEC 2017 single-objective real-parameter optimisation competition.The competition seeks minima of shifted, scalable, and rotated functions in dimensions 10, 30, 50, and 100.
  • Experimental data: The experiments use aggregated final results across runs, except for the Page test, which studies convergence.Mean final results are reported for the 10-dimension scenario, and the source data come from the organiser’s repository with a correction for LSHADE-cnEpSin.
  • Competition and benchmarks: The benchmark suite includes simple multimodal, hybrid, and composition functions, while the Sum of Different Power Function is discarded for unstable cross-language behavior.The composition functions combine basic functions using weights and a bias; the discarded function produced inconsistent behavior for the same algorithm in different languages.
  • Algorithms: The contestant list contains thirteen algorithms ordered by their competition ranking and assigned short names for tables and plots.Examples include EBOwithCMAR, jSO, LSHADE variants, CMA-ES variants, MOS, PPSO, DYYPO, and TLBO-FL.

8 Experiments and Results

The experiments show that non-normal results require non-parametric or Bayesian analyses, whose conclusions can differ in interpretability and certainty across comparisons.

  • Parametric analysis: Multivariate normality is rejected for every algorithm, stopping the parametric analysis when multiple dimensions are treated as multiple measures.The multivariate Shapiro-Wilk generalisation is used before considering Hotelling’s T^2 test.
  • Pairwise comparisons: The Wilcoxon Rank-Sum test rejects equality of medians in the EBO–jSO comparison, whereas the Sign and Wilcoxon tests do not.The reported Wilcoxon statistics R+ and R− are both high, indicating no significant difference in the ranking of observations where either algorithm performs better.
  • Multiple comparisons: Multiple-algorithm tests reject median equivalence, but including lower-performing algorithms can produce rejection unrelated to the focal algorithm’s differences from relevant competitors.The paper recommends comparing against state-of-the-art algorithms rather than automatically including all thirteen contestants.
  • Multiple comparisons: The n × n comparison finds no algorithm significantly different from all others after Holland adjustment, although jSO and EBO differ significantly from MOS, PPSO, and RBI.Differences are not significant for LSHADE variants or MM, and the adjustment makes differences harder to detect with many algorithms.
  • Multiple comparisons: The Nemenyi critical-difference plot groups the top algorithms through RBI, showing that differences are difficult to identify among similarly performing algorithms.The overlapping groups include up to the seventh-ranked algorithm in the 10-dimension scenario.
  • Bayesian analyses: Bayesian analyses distinguish performance direction, practical equivalence, and uncertainty rather than only rejecting equivalence.For EBO-CMAR versus jSO, the Bayesian tests assign similar probabilities across hypotheses, with high rope probability indicating several benchmark ties.
  • Bayesian analyses: The imprecise Dirichlet Process estimates EBO-CMAR’s probability of outperforming jSO between 0.59 and 0.45, below the 0.95 decision threshold.The result does not establish that EBO-CMAR outperforms jSO with 95% probability.

9 Summary of results in CEC’2017

The CEC’2017 analysis compares algorithms across dimensions using non-parametric critical-difference plots and Bayesian pairwise tests. Results generally align with summary rankings, while Bayesian analyses expose uncertainty and dimension-dependent dominance patterns.

  • The competition analysis uses non-parametric tests with post-hoc procedures and critical-difference plots for four dimension scenarios.
  • Critical-difference plots and summary scores place the same algorithms among the leading positions.
  • LSSPA, DES, LSCNE, MM, jSO and EBO form a lead group whose equivalence with the winner cannot be discarded in any dimension scenario.
  • As competition error increases, more comparisons become significant and fewer ties are detected.
  • In the 50-dimensional scenario, the lead group is EBO, jSO, LSCNE and DES, with lower tie probabilities than earlier scenarios.
  • In 100 dimensions, DES wins against every remaining algorithm, with probabilities from 0.59 against LSCNE to 0.97 against PPSO.

10 Discussion and Lessons Learnt

The discussion reviews limitations of frequentist and Bayesian testing, emphasizing that statistical conclusions depend on interpretation, design choices and the range of summaries reported. It therefore supports contextualized use of multiple tests and transparent analysis.

  • Failure to reject an NHST null hypothesis should not be interpreted as evidence of no difference between algorithms.
  • NHST’s dichotomous decisions are criticized, while Bayesian analyses also retain researcher-dependent choices involving priors and other parameters.
  • Reliable and repeatable results require diverse data summaries and clear understanding of the underlying process rather than reliance on a single index.
  • Bayesian tests are more complex, which contributes to their lower use in experimentation.
  • Bayesian conclusions depend on choices such as the prior distribution and the rope parameter, which may produce a model with low data representation.
  • Because tests can be misused and produce spurious conclusions, using different tests can help place results in context.

11 Conclusions

The paper surveys statistical procedures for comparing optimisation algorithms and illustrates them through CEC’17 competition results. It concludes by recommending complementary non-parametric and Bayesian analyses for a fuller comparison perspective.

  • The paper provides an extensive set of statistical procedures and examples for comparing optimisation-algorithm results, particularly in competition settings.
  • Its coverage ranges from the Binomial Sign test to Bayesian techniques such as the Imprecise Dirichlet Process, with tools for experimental studies.
  • CEC’17 analyses produced different results depending on the test and circumstances, motivating recommendations about test selection and interpretation.
  • The paper encourages joint use of non-parametric and Bayesian tests because their results can provide complementary perspectives on algorithm comparisons.
Loading 2002.09227v1…