Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis

Alessio Benavoli; Giorgio Corani; Janez Demsar; Marco Zaffalon

Source-linked AI summary

Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis

Alessio Benavoli, Giorgio Corani, Janez Demsar, Marco Zaffalon

arXiv:1606.04316v3Jun 14, 2016stat.MLcs.LG

TL;DR

The paper argues that NHST is a poor fit for comparing classifier performance because it does not provide the probabilities researchers actually seek. It develops Bayesian alternatives based on likelihoods, priors, and posterior distributions, while emphasizing that posterior summaries and frequentist thresholds can discard information. The authors conclude that machine learning should move away from NHST and p-values.

Problem
NHST does not provide the probability of one classifier outperforming another or the probability of hypotheses given observed data.
Method
The paper models data with likelihoods and priors, combines them through Bayes’ rule, and queries the resulting posterior distribution for classifier comparisons.
Results
Frequentist analyses yield thresholded significance decisions while leaving effect magnitude, uncertainty, and hypothesis probabilities unspecified; Bayesian analysis can distinguish practical equivalence from meaningful differences.
Takeaways & Limitations
The authors recommend replacing NHST and p-values with Bayesian analysis that retains graded information about comparative performance.
Takeaways & Limitations
The signed-rank test can be miscalibrated for nonsymmetric distributions, while multiple-comparison corrections depend on which comparisons the analyst chooses.

Abstract

from arXiv · show

The machine learning community adopted the use of null hypothesis significance testing (NHST) in order to ensure the statistical validity of results. Many scientific fields however realized the shortcomings of frequentist reasoning and in the most radical cases even banned its use in publications. We should do the same: just as we have embraced the Bayesian paradigm in the development of new machine learning methods, so we should also use it in the analysis of our own results. We argue for abandonment of NHST by exposing its fallacies and, more importantly, offer better - more sound and useful - alternatives for it.

1. Introduction

As machine learning shifted from novel methods toward empirical comparisons, statistical validation became important, but NHST does not answer whether one classifier is better than another. The paper advocates Bayesian analysis because it directly addresses posterior questions about comparative performance.

Empirical comparisons of competing machine-learning methods increased the need for statistical validation.As practical applications grew, researchers increasingly compared methods on real-world datasets.
The paper argues that machine learning should move beyond NHST because its shortcomings have prompted criticism in statistics and other scientific fields.The authors cite an American Statistical Association statement against p-values and the declining favor of NHST elsewhere.
NHST evaluates the probability of observing the data under a null hypothesis of equivalence, not the probability that one classifier outperforms another.This mismatch means a reported confidence level does not directly quantify the probability that a classifier is more accurate.
Bayesian methods naturally provide posterior probabilities for questions such as whether method A is better than method B.The paper focuses on Bayesian alternatives to frequentist NHST and discusses their inference, results, and supporting software libraries.

2. Frequentist analysis of experimental results

The paper applies frequentist testing to classifier comparisons and shows that NHST answers a different question from the posterior probabilities researchers often want. Its examples expose sensitivity to sample size, uncertainty, practical effect size, non-rejection, multiple comparisons, and arbitrary thresholds.

Experimental setup: The comparison uses accuracy differences from 10 runs of 10-fold cross-validation across 54 datasets, after selecting classifiers, datasets, and a comparison metric.The classifiers are naive Bayes (nbc) and averaged one-dependence estimator (aode).
Frequentist testing: Overlapping training sets make cross-validation differences dependent, so the usual t-test is miscalibrated and has an inflated Type I error rate.The correlation cannot be estimated without bias; a heuristic correlation parameter is introduced for the correlated t-test.
NHST pitfalls: NHST p-values do not provide the probability of a hypothesis, the effect magnitude, or its uncertainty; they instead describe data extremeness assuming the null hypothesis.Researchers may want probabilities such as whether one method is better or better by more than 1%, which are not supplied by NHST.
NHST pitfalls: With 15 observations, p = 0.077 leaves the null unrejected, whereas all 100 observations give p = 0.048 despite a very small accuracy difference.The example shows that statistical significance can change with added data without indicating practical significance.
NHST pitfalls: Failure to reject the null does not establish equivalence or truth, and the α threshold lacks a principled basis while multiple-comparison corrections depend on the analyst’s comparison plan.The paper also notes that point-wise classifier equality is practically always false and that enough data can confirm arbitrarily small effects.

3. Bayesian analysis of experimental results

The paper replaces NHST with Bayesian estimation, using posterior probabilities, a practical-equivalence region, effect sizes, and uncertainty to compare classifiers. Across classifier comparisons, the Bayesian approach supports more informative decisions, including practical equivalence, while reducing claimed significances relative to NHST.

Bayesian alternatives: Bayesian model comparison evaluates which null or alternative model is more credible, whereas Bayesian estimation queries the posterior distribution of parameter values.The paper focuses on Bayesian estimation for subsequent analysis.
Bayesian alternatives: The matching prior makes the Bayesian posterior numerically equivalent to the frequentist correlated t-test while yielding different inferences.For one dataset, the posterior is a Student distribution with 99 degrees of freedom, mean −0.0194, and scale parameter 0.000030.
Practical interpretation: The rope estimates the posterior probability of practical equivalence and identifies differences with practical meaning, while HDIs represent effect-size uncertainty.For squash-unsorted, P(nbc = aode) = 0.086, P(nbc ≪aode) = 0.788, and P(nbc ≫aode) = 0.126.
Practical interpretation: Across 54 datasets, Bayesian posteriors identified 12 cases where aode was practically better than nbc and 6 cases where the classifiers were practically equivalent.The posterior-based analysis also included cases where practical equivalence or difference could not be clearly determined.
Comparison with NHST: Bayesian analysis made decisions in 21 datasets compared with the NHST decision pattern, including equivalence decisions that NHST cannot provide.Among NHST non-rejections, Bayesian analysis declared equivalence in 6 datasets; among NHST rejections, it identified practical differences in 14 and equivalence in 1.
Comparison with NHST: The Bayesian test reduced claimed significances by 30% relative to NHST but made more decisions when practical equivalence was counted, 222 versus 199.For NHST-significant comparisons, Bayesian analysis confirmed practical differences in 142/199 cases, made no conclusion in 51/199, and found equivalence in 6/199.

4. Comparing two classifiers on multiple data sets

Comparing classifiers across multiple datasets exposes a gap in NHST: it does not directly answer which classifier is better across datasets or quantify the probability and practical size of differences. The section describes the usual signed-rank procedure and illustrates its binary conclusions.

Multiple-dataset comparison: NHST has no direct test that takes cross-validation differences from multiple datasets and decides which classifier is better across all datasets.The common procedure first averages cross-validation differences within each dataset, then applies NHST to those mean differences.
Multiple-dataset comparison: For q = 54 datasets, the recommended NHST is the signed-rank test applied to the vector of per-dataset mean accuracy differences.The differences are denoted z = {z1, . . . , zq}.
Signed-rank test: The Wilcoxon signed-rank test assumes i.i.d. observations from a symmetric distribution, and is miscalibrated when that distribution is asymmetric.Strict use would first test symmetry, but the paper follows common practice and does not perform that preliminary test.
Signed-rank test: The test ranks absolute differences and compares the ranks of positive and negative differences, while ignoring their absolute magnitudes beyond rank order.For sufficiently large samples such as q > 10, the statistic is treated as approximately normally distributed for the two-sided test.
Results: p-value=10−6 for nbc versus aode leads NHST to reject the null, while also leaving effect magnitude, uncertainty, and the probability of the null unspecified.The same analysis marks nbc-aode, nbc-hnb, and j48-j48gr as statistically significantly different.

4.2 The Bayesian analysis approach

The paper proposes Bayesian alternatives for comparing two classifiers across multiple datasets, extending the signed-rank approach and using a hierarchical model. This approach is motivated by the limits of frequentist comparisons summarized in the section.

Bayesian alternatives: The paper presents two approaches for comparing two classifiers across multiple datasets: a nonparametric extension of the Wilcoxon signed-rank test and a hierarchical model.These alternatives are introduced as ways to analyze comparisons after cross-validation on multiple datasets.
Bayesian alternatives: The Bayesian counterpart of the frequentist sign and signed-rank tests is based on a Dirichlet process.The cited approach is attributed to Benavoli et al. (2014).

Bayesian sign and signed-tank test

The Bayesian sign and signed-rank tests model the distribution of classifier differences with a Dirichlet process and estimate probabilities for worse, equivalent, and practically better outcomes. Unlike the frequentist signed-rank test, the Bayesian signed-rank test does not require symmetry.

Bayesian model: A Dirichlet process prior is placed on the probability distribution of the scalar difference variable z, using prior strength s and prior mean z0.The observed vector z = {z1, . . . , zq} consists of i.i.d. observations.
Bayesian model: The posterior distribution is a mixture of Dirac deltas at the observations and prior pseudo-observation, with Dirichlet-distributed weights.This hierarchical model represents the posterior distribution of the probability distribution of z.
Bayesian sign test: The Bayesian sign test partitions mean differences into left, rope, and right regions, yielding probabilities θl, θe, and θr for those outcomes.The rope is the interval [−r, r], while the outer regions represent differences below −r and above r.
Bayesian signed-rank test: The Bayesian signed-rank test computes the corresponding posterior probabilities by Monte Carlo sampling Dirichlet weights and also accounts for the rope [−r, r].Unlike the Bayesian sign test, its distribution does not have a simple closed form.
Bayesian signed-rank test: The Bayesian signed-rank test does not require symmetry because the Bayesian estimation approach estimates the distribution from data.The paper explicitly states that it also works when the observations are asymmetrically distributed.
Experiments: With r = 0, s = 0.5, and z0 = 0, the reported posterior for θl represents the probability that aode is better than nbc.The posterior samples are shown as a histogram of 150′000 samples.
Experiments: aode is practically better than nbc with probability 90%, while rope is selected in the remaining 10%.The paper reports additional posterior comparisons, including practical equivalence for j48 and j48gr.
Computational cost: Comparing two classifiers with 150′000 Bayesian sign-rank samples takes less than one second on a standard computer.

4.3 Choice of the prior

The prior uses a strength and location parameter whose influence is examined through extreme prior locations. The reported probabilities change only slightly with the prior location, while the section also contrasts this multi-dataset analysis with single-dataset correlated t-tests.

Prior choice: The selected prior parameters s = 0.5 and z0 = 0 correspond, in rank terms, to one pseudo-observation located inside the rope.
Prior sensitivity: Using z0 = −∞ and z0 = ∞ produces only a minor change in the classifier-comparison probabilities.The paper presents these extreme cases as an example of robust Bayesian analysis.
Scope: The Bayesian correlated t-test is limited to a single dataset, whereas the multi-dataset analysis addresses comparisons across datasets.
Scope: The usual NHST based on per-dataset mean differences discards the cross-validation correlation ρ and the sample standard deviation σ̂i for each dataset.The standard deviation indicates the accuracy of each mean-difference estimator and can vary substantially across datasets.

Bayesian hierarchical correlated t-test

The hierarchical correlated t-test models dataset-specific uncertainty and population-level differences between classifiers, using Bayesian inference to estimate posterior probabilities for future comparisons.

Model specification: Equation (12) models each dataset’s cross-validation measures as jointly multivariate normal with shared mean, variance, and correlation.The correlation represents dependence from overlapping training sets.
Model specification: The hierarchical model treats dataset-specific mean differences as drawn from a high-level Student distribution centered on the population mean difference µ0.Its heavy tails accommodate datasets whose effects differ substantially and make conclusions more cautious.
Priors and uncertainty: Dataset-specific standard deviations are drawn from a common distribution, representing different uncertainty levels across datasets.The model uses a uniform high-level distribution for σi and bounded priors for other parameters, with bounds adapted to measures such as accuracy, AUC, precision, and recall.
Inference: Joint estimation applies shrinkage to dataset-specific sample means, estimating µi more accurately than the independently analyzed means used by other tests.This result is stated to hold under general assumptions, including severe misspecification being addressed by the hierarchical model.
Experiments: The experiment estimates posterior probabilities that one classifier is better, equivalent, or worse on the next unseen dataset using 4′000 Monte Carlo samples.For nbc versus aode, aode is better in almost 100% of cases; the hierarchical and Bayesian signed-rank tests otherwise substantially agree, except for comparisons involving aode versus j48 and j48gr.
Experiments: The analysis uses 10 runs of 10-fold cross-validation on 54 datasets, totaling 5400 observations, and takes about three minutes on a standard computer.The hierarchical model is implemented in Stan.

4.4 Choice of the hyper-priors parameters

A sensitivity analysis found that changing the hyper-prior constants produced essentially identical results for the posterior distributions.

Sensitivity analysis: Changing the constants in the top-level Gamma and uniform hyper-priors produced essentially identical posterior results.The analysis tested whether different hyper-prior settings had any notable influence on the resulting posterior distribution.

4.5 Bayesian signed rank or hierarchical model?

The paper prefers the hierarchical model because it uses all available cross-validation information when estimating classifier differences across datasets.

Model comparison: The paper frames the choice between the Bayesian signed-rank test and hierarchical model as a comparison of two methods for classifiers evaluated on multiple datasets.The hierarchical model is presented as the preferred option.
Model comparison: The hierarchical model is preferred because it uses repeated k-fold cross-validation results to estimate each dataset’s mean accuracy difference from sample means, variability, and training-set correlation.The Bayesian signed-rank test uses less of this information.

5. Comparisons of multiple classifiers

The paper highlights multiple-hypothesis testing as a problem for NHST and describes multilevel Bayesian analysis as an alternative for comparisons involving multiple classifiers.

Multiple testing: With ten pairwise comparisons, Bonferroni correction changes a 0.05 overall significance level to an individual threshold of 0.005.The correction divides the original p-value by the number of tests and implicitly assumes independent test statistics.
Multiple testing: Bonferroni correction depends on how the analyst defines the set of comparisons, producing 0.0125 for four comparisons instead of 0.005 for ten.The paper identifies this dependence as a critical problem inherited from Type I error reasoning.
Bayesian alternative: The Bayesian approach generally avoids multiple-comparison concerns associated with Type I error and suggests multilevel analysis to mitigate false alarms.For this setting, the suggested multilevel approach is a hierarchical Bayesian model on multiple classifiers.

6. Software and available Bayesian tests

The paper provides freely available implementations of its Bayesian classifier-comparison tests and points to broader parametric and nonparametric alternatives.

R and Python code is available for all tests presented in the paper.
Replication materials include open-source IPython notebooks implemented in Python and Julia.The software may be freely used, changed, and shared.
Researchers can also use general Bayesian parametric tests described by Kruschke and Gelman et al.
The paper specializes Bayesian tests for correlated data and implements several Bayesian nonparametric tests for algorithm comparison.

7. Conclusions

The authors recommend replacing frequentist NHST in machine learning with Bayesian analysis, especially for comparing classifier performance. They present three Bayesian tests and make their code freely available for practical use.

The authors discourage frequentist NHST for comparing classifier performance and argue that machine learning should move beyond NHST and p-values.
The paper presents Bayesian correlated t-test, Bayesian signed rank test, and Bayesian hierarchical model for comparing algorithm performance.
All code for these tests is freely available, allowing researchers to use them in their analyses.
The authors state that Bayesian tests can replace NHST tests used inside machine-learning algorithms as well as in performance comparisons.They give racing algorithms and Bayesian-network structure learning as examples of internal NHST use.