Source-linked AI summary
Should we really use post-hoc tests based on mean-ranks?
Alessio Benavoli, Giorgio Corani, Francesca Mangili
TL;DR
This technical note examines whether mean-ranks post-hoc tests are appropriate after Friedman tests for comparing algorithms across multiple data sets. It analyzes their dependence on the full algorithm pool and recommends pairwise alternatives whose decisions depend only on the two algorithms compared. The note concludes that mean-ranks tests should be avoided because they can produce pool-dependent decisions and have additional Type I error-control drawbacks.
Problem
Mean-ranks post-hoc comparisons are widely used after Friedman tests, but their decisions can depend on the performance of algorithms other than the pair being compared.
Method
The note analyzes the mean-ranks test's rank-based statistic, sampling assumption, pool dependence, and illustrative classifier comparisons, then contrasts it with sign and Wilcoxon signed-rank tests.
Results
The mean-ranks test can change its decision for the same algorithm pair when the pool of alternative algorithms changes.
Takeaways & Limitations
Pairwise post-hoc comparisons should use tests whose decisions depend only on the two algorithms compared, such as the sign test or Wilcoxon signed-rank test.
Takeaways & Limitations
Mean-ranks post-hoc analysis does not guarantee family-wise Type I error below α when comparing equivalent algorithms in the described setting.
Abstract
from arXiv · showhide
The statistical comparison of multiple algorithms over multiple data sets is fundamental in machine learning. This is typically carried out by the Friedman test. When the Friedman test rejects the null hypothesis, multiple comparisons are carried out to establish which are the significant differences among algorithms. The multiple comparisons are usually performed using the mean-ranks test. The aim of this technical note is to discuss the inconsistencies of the mean-ranks post-hoc test with the goal of discouraging its use in machine learning as well as in medicine, psychology, etc.. We show that the outcome of the mean-ranks test depends on the pool of algorithms originally included in the experiment. In other words, the outcome of the comparison between algorithms A and B depends also on the performance of the other algorithms included in the original experiment. This can lead to paradoxical situations. For instance the difference between A and B could be declared significant if the pool comprises algorithms C, D, E and not significant if the pool comprises algorithms F, G, H. To overcome these issues, we suggest instead to perform the multiple comparison using a test whose outcome only depends on the two algorithms being compared, such as the sign-test or the Wilcoxon signed-rank test.
1. Introduction
The Friedman test is commonly used to compare multiple algorithms across data sets, followed by mean-ranks post-hoc comparisons. This note argues that mean-ranks decisions depend on the full algorithm pool and recommends pairwise tests whose decisions depend only on the compared algorithms.
- The Friedman test is recommended for comparing multiple algorithms across data sets because it is non-parametric, does not require commensurable measures, does not assume normality of sample means, and is robust to outliers.
- After Friedman rejects no difference among algorithms, pairwise post-hoc comparisons assess significant differences while adjusting significance to control family-wise Type I error.
- The mean-ranks test is widely recommended and used for multiple pairwise comparisons in nonparametric statistics and machine learning.
- Mean-ranks comparisons between A and B depend on the performance of the other algorithms in the original experiment, allowing contradictory significance decisions across alternative algorithm pools.
- The note recommends Wilcoxon signed-rank or sign tests for pairwise post-hoc comparisons, while retaining family-wise Type I error control.
2. Friedman test
The Friedman procedure organizes algorithm performances across data sets into a matrix, ranks algorithms within each data set, and tests the resulting rank structure. Under its null hypothesis, the average row sum has a specified value and the statistic follows a chi-squared distribution.
- Algorithm performances are organized in an m-by-n matrix, with Xij denoting algorithm i's performance on data set j and columns assumed independent.
- Each data set is ranked column-by-column, replacing every performance Xij with its rank relative to the other algorithms on that data set.
- The rank Rij denotes algorithm i's rank on data set j, and the sum of algorithm i's ranks depends on its performance relative to the other m −1 algorithms.
- Under the Friedman null hypothesis, the average rank-sum value is n(m+1)/2.
- The Friedman statistic has a chi-squared distribution with m−1 degrees of freedom under the null hypothesis, and for m = 2 the test corresponds to the sign test.
3. Mean ranks post-hoc test
Following a rejected Friedman test, all pairwise algorithm comparisons require multiplicity control. The mean-ranks post-hoc test uses a rank-difference statistic and an equally probable-ranks assumption that conflicts with the preceding rejection.
- If all m algorithms are compared pairwise after Friedman rejects, the analysis requires m(m −1)/2 pairwise comparisons.
- Multiple comparisons require controlling the family-wise error rate, the probability of at least one erroneous null-hypothesis rejection among the pairwise comparisons.
- The mean-rank test declares algorithms i and j significantly different using a comparison based on their mean-rank difference and a corrected normal quantile.
- Its large-sample approximation assumes all (m!)^n rank configurations are equally probable, yielding variance m(m + 1)/6n for the mean-rank difference statistic.
- That equal-probability assumption is untenable because post-hoc analysis is conducted precisely after the Friedman null hypothesis has been rejected.
4. Inconsistencies of the mean-ranks test
The examples show that mean-ranks post-hoc decisions can change when other algorithms are added or removed, even when the compared algorithms' performances remain unchanged. This pool dependence can amplify apparent differences, reduce power, and produce inconsistent significance decisions.
- Examples of inconsistency: For A and B with symmetric differences and equal mean performance, t-test, Wilcoxon, sign-test, and mean-ranks procedures all return p = 1.Adding C, D, and E leads the Friedman test to p ≈10^-10 and the mean-ranks post-hoc test to reject for A and B.
- Examples of inconsistency: The mean-ranks test can declare A and B significantly different when tested with other algorithms, but not when tested alone.With C, D, and E included, the test rejects for A and B; comparing A and B alone does not reject.
- Examples of inconsistency: Other algorithms can artificially amplify rank differences between two classifiers when their relative performance varies across datasets.The global rank difference need not correspond to a large accuracy difference, and the reverse can also occur.
- Examples of inconsistency: With m = 5, the mean-ranks test estimates five times larger variance than the sign test, reducing its power as the algorithm pool grows.In the reported simulation, sign-test power is 0.94 whereas mean-ranks power is 0.046.
- Maximum type I error: The mean-ranks test may fail to control the maximum family-wise Type I error when comparing equivalent algorithms within a larger pool.The paper notes that this guarantee is not ensured when all but one algorithm have equal accuracies.
5. A suggested procedure
The paper recommends replacing the mean-ranks post-hoc test with pairwise tests whose decisions depend only on the two algorithms compared, while adjusting significance levels for family-wise Type-I error.
- The sign test and Wilcoxon signed-rank test avoid dependence on the pool of classifiers included in the original experiment.
- The sign test is more robust because it assumes only that observations are identically distributed, but it has low power.
- The Wilcoxon signed-rank test is generally recommended because it is more powerful, but it additionally assumes symmetric differences.
- The choice between the two tests depends on whether symmetry is tenable for the analyzed data.
- Multiple comparisons should adjust the significance level to control family-wise Type-I error.
6. Software
The paper provides MATLAB scripts for the examples through an online download.
- MATLAB scripts for the examples can be downloaded from the project website.
7. Conclusions
The paper discourages use of the mean-ranks post-hoc test and points readers to pairwise and Bayesian alternatives.
- The authors recommend avoiding the widely used mean-ranks post-hoc test because of its drawbacks.
- They recommend the sign test or Wilcoxon signed-rank test because each decision depends only on the two algorithms compared.
- The paper also highlights Bayesian counterparts of these tests as alternatives to null-hypothesis significance testing.
Table of accuracies used in example 3
Table 2 reports classifier accuracies on different data sets for the paper’s example.
- Table 2 reports the accuracy of classifiers on different data sets.