PaperlayerView exact revision on arXiv

Source-linked AI summary

Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning

Sebastian Raschka

arXiv:1811.12808v3cs.LGstat.ML

TL;DR

Machine learning research needs reliable ways to estimate generalization, select models, and compare algorithms, especially when data are limited. The article reviews evaluation, selection, resampling, cross-validation, and statistical-comparison techniques, then recommends context-dependent practices, including cross-validation for small datasets. It concludes that cross-validation is not a silver bullet but is the best available non-parametric tool for testing model generalization.

  • Problem

    Researchers and practitioners need to estimate future performance, choose models, and identify suitable algorithms without confusing these distinct tasks or relying on unreliable estimates.

  • Method

    The article synthesizes theoretical and empirical evidence on holdout evaluation, bootstrap uncertainty estimation, cross-validation, statistical tests, multiple-comparison strategies, and algorithm-selection methods.

  • Results

    Cross-validation is described as the best available non-parametric method for testing model generalization, although it is not a silver bullet.

  • Takeaways & Limitations

    For very small datasets, k-fold cross-validation with a large k or leave-one-out cross-validation can replace setting aside a separate test set, subject to their trade-offs.

  • Takeaways & Limitations

    Holdout evaluation can be problematic on small datasets because splitting data can introduce substantial variance and pessimistic bias.

Abstract

from arXiv · show

The correct use of model evaluation, model selection, and algorithm selection techniques is vital in academic machine learning research as well as in many industrial settings. This article reviews different techniques that can be used for each of these three subtasks and discusses the main advantages and disadvantages of each technique with references to theoretical and empirical studies. Further, recommendations are given to encourage best yet feasible practices in research and applications of machine learning. Common methods such as the holdout method for model evaluation and selection are covered, which are not recommended when working with small datasets. Different flavors of the bootstrap technique are introduced for estimating the uncertainty of performance estimates, as an alternative to confidence intervals via normal approximation if bootstrapping is computationally feasible. Common cross-validation techniques such as leave-one-out cross-validation and k-fold cross-validation are reviewed, the bias-variance trade-off for choosing k is discussed, and practical tips for the optimal choice of k are given based on empirical evidence. Different statistical tests for algorithm comparisons are presented, and strategies for dealing with multiple comparisons such as omnibus tests and multiple-comparison corrections are discussed. Finally, alternative methods for algorithm selection, such as the combined F-test 5x2 cross-validation and nested cross-validation, are recommended for comparing machine learning algorithms when datasets are small.

1 Introduction: Essential Model Evaluation Terms and Techniques

The article distinguishes model evaluation, model selection, and algorithm selection as related but different tasks requiring different approaches. It frames these tasks around estimating generalization, improving predictive performance, and identifying suitable algorithms while highlighting pitfalls of data splitting and performance estimation.

  • Three evaluation and selection tasks: Model evaluation estimates predictive performance on future unseen data, while model selection chooses a high-performing model within a hypothesis space.Algorithm selection compares different learning algorithms and may also select the best model within each algorithm’s hypothesis space.
  • Three evaluation and selection tasks: The article reviews distinct techniques for model evaluation, model selection, and algorithm selection because the three tasks require different approaches.It presents these methods within a broader machine learning workflow.
  • Performance estimation: Training-set accuracy is optimistically biased because it overestimates a model’s true accuracy on unseen data.The article defines prediction bias as the difference between expected and true prediction accuracy.
  • Holdout validation: Splitting off test data can create pessimistic bias when the model has not reached capacity, because withholding training examples prevents learning from additional data.Refitting on the full dataset removes the possibility of estimating that refitted model’s generalization performance with the already-used test set.

2 Bootstrapping and Uncertainties

Resampling methods address bias, variance, and stability in model-performance estimates, while repeated holdout validation reduces sensitivity to a single train/test split. The section also introduces bootstrap-based uncertainty estimation as an alternative evaluation approach.

  • Bias and variance: Learning curves illustrate that softmax classifiers trained on smaller MNIST subsets generalize less well because limited training data increases susceptibility to noise.The experiment used a stratified 5000-sample MNIST subset, with classifiers evaluated on a fixed 1500-sample test set.
  • Bias and variance: Performance estimates trade off bias and variance: large test sets can be pessimistic, whereas smaller test sets increase variance across data splits.The variance depends on which instances enter the training and test sets, especially with small datasets.
  • Repeated holdout validation: Repeated holdout validation averages performance across k random train/test splits, producing a more robust estimate and information about model stability.Each repetition fits and evaluates the model on a different split.
  • Repeated holdout validation: On Iris, 50 stratified 50/50 splits produced 95% average accuracy, while 50 stratified 90/10 splits produced 96% average accuracy for 3-nearest neighbors.The 50/50 splits used 75 test samples, whereas the 90/10 splits used 15 test samples.
  • Bootstrap uncertainty: Bootstrap methods provide an alternative approach for estimating uncertainty in model-performance estimates after repeated holdout evaluation.The section presents different bootstrap variants for predictive-model evaluation.

3 Cross-validation and Hyperparameter Optimization

Hyperparameter choices produce different models, so model selection requires evaluation procedures that balance bias, variance, computational cost, and dataset size. The section presents three-way holdout and k-fold cross-validation, emphasizing empirical trade-offs and practical guidance for choosing among them.

  • 3.2 About Hyperparameters and Model Selection: Hyperparameters control algorithm behavior, and changing them can produce different models; selecting the best-performing model across settings is model selection.Examples include k and the distance metric in k-nearest neighbors, or optimization iterations and regularization strength in logistic regression.
  • 3.3 The Three-Way Holdout Method for Hyperparameter Tuning: The three-way holdout method separates training, validation, and test data so hyperparameters can be tuned without using the independent test set.The training set fits models, the validation set supports hyperparameter selection, and the test set remains independent for evaluation.
  • 3.4 k-fold Cross-Validation: k-fold cross-validation iterates over k splits, using one part for validation and the remaining k −1 parts for training so every sample is tested.This differs from repeated holdout because k-fold cross-validation guarantees that each sample serves as validation at least once.
  • 3.5 Special Cases: 2-Fold and Leave-One-Out Cross-Validation: There is no universal unbiased estimator of k-fold cross-validation variance, making k selection an unavoidable bias-variance trade-off.The cited theorem establishes that no estimator is universally valid and unbiased across all distributions.
  • 3.6 k-fold Cross-Validation and the Bias-Variance Trade-off: LOOCV has small bias but large variance, whereas experiments suggest 10-fold cross-validation often provides the best bias-variance trade-off.Repeating k-fold cross-validation can increase estimate precision while maintaining a small bias; LOOCV may be preferable to holdout when computationally feasible.

4 Algorithm Comparison

This section outlines statistical approaches for comparing model performances, while emphasizing that conventional proportion tests rely on assumptions often violated in practice. It presents a general hypothesis-testing workflow and explains why paired tests such as McNemar’s test may be preferable.

  • Overview: Algorithm comparisons may use statistical tests based on independent test-set predictions or models evaluated through cross-validation.The section introduces these testing frameworks as alternatives for comparing model and algorithm performance.
  • Testing the Difference of Proportions: A conventional difference-of-proportions test compares two estimated accuracies using confidence intervals or a z-score.The procedure pools the two accuracies, computes a standard deviation and z-score, and compares the result with a significance threshold.
  • Testing the Difference of Proportions: Empirical results indicate that the z-score test can have a high false positive rate, so it is not recommended in practice.A paired test may be more appropriate, with McNemar’s test identified as a more robust alternative.
  • Testing the Difference of Proportions: The standard workflow formulates hypotheses, chooses a significance threshold, computes a test statistic and p-value, and accepts or rejects the null hypothesis.The outlined example tests whether two proportions are equal or different.
  • Testing the Difference of Proportions: The difference-of-proportions test assumes independent test errors, an assumption usually violated when both models use the same test set.The z-score rejects equality at α = 0.05 when |z| exceeds 1.96, but shared test examples undermine independence.

4.3 Comparing Two Models with the McNemar Test

McNemar’s test compares two classifiers using paired predictions organized in a 2x2 contingency table. Its off-diagonal disagreements determine whether the models’ performances differ, with a continuity-corrected variant commonly used.

  • Test rationale: McNemar’s test is a non-parametric paired-comparison test for evaluating whether two classifiers differ in performance.It is preferred over the difference-of-proportions test for paired classifier comparisons.
  • Contingency-table setup: The test uses a 2x2 table of paired predictions, where cells B and C count examples classified correctly by only one of the two models.Cells A and D count cases where both models are correct or both are wrong, while the off-diagonal cells drive the comparison.
  • Contingency-table setup: 11:1 disagreements favor Model 1 more clearly than 25:15, illustrating why paired disagreement counts can clarify nearly identical accuracies.Both scenarios involve Model 1 and Model 2 accuracies of 99.7% and 99.6%, respectively.
  • Hypothesis testing: The null hypothesis states that the probabilities of the two discordant outcomes are equal, meaning neither model performs better.A p-value below the selected α-level leads to rejection of equal performance.
  • Hypothesis testing: Scenario A yields χ2 = 8.3 and p = 0.0039, whereas Scenario B yields χ2 = 2.5 and p = 0.1138.The first result rejects equal performance at α = 0.05; the second does not.
  • Continuity correction: Edwards’s continuity-corrected McNemar test is the more commonly used variant and is intended to improve usefulness and accuracy for discrete frequencies.The correction reduces the absolute difference B − C by one.

4.4 Exact p-Values via the Binomial Test

This section recommends the exact binomial test when McNemar’s chi-squared approximation may be unreliable because discordant-cell counts are small. The continuity-corrected approximation closely matches exact p-values when both counts exceed 50.

  • When to use the exact test: The exact binomial test is computationally more expensive but appropriate when cells B and C contain relatively few observations.Small discordant counts can make the chi-squared approximation inaccurate.
  • Exact p-value calculation: Under the null hypothesis, McNemar’s exact calculation is equivalent to a binomial test with proportion 0.5.The binomial sample size is n = B + C, distinct from the overall test-set size.
  • Approximation accuracy: Figure 21 compares regular and continuity-corrected McNemar approximations with exact binomial-test p-values.The figure uses a heat map to display differences among these calculations.
  • Approximation accuracy: When both B and C exceed 50, continuity-corrected McNemar p-values are almost identical to exact binomial-test p-values.The MLxtend implementation provides options for regular, corrected, and exact calculations.

4.5 Multiple Hypotheses Testing

Comparing more than two models creates a multiple-hypothesis problem when pairwise tests are repeated. The recommended strategy is to test overall differences first, then conduct corrected pairwise comparisons if warranted.

  • Problem and workflow: Repeated pairwise comparisons among more than two models create a multiple-hypotheses testing problem.The issue arises when models are compared using estimated generalization performance on an independent test set.
  • Problem and workflow: The proposed workflow first applies an omnibus test, then uses adjusted pairwise post hoc tests if the omnibus null hypothesis is rejected.McNemar’s test is given as an example of a possible pairwise post hoc test.
  • Omnibus tests: Cochran’s Q test is a possible omnibus test for three or more classifiers because it generalizes McNemar’s test.Omnibus tests assess whether group performances differ overall.
  • Omnibus tests: An omnibus test can establish that models differ but cannot identify which model pairs differ.Post hoc testing is therefore used after rejecting the omnibus null hypothesis.
  • Post hoc testing: Post hoc procedures can require all pairwise comparisons and therefore need multiple-comparison corrections such as Bonferroni’s correction.Using a significance threshold suitable for one comparison is not suitable for multiple pairwise tests.

4.6 Cochran’s Q Test for Comparing the Performance of Multiple Classifiers

Cochran’s Q test compares three or more classifiers evaluated on the same dataset, testing whether their classification accuracies differ. A significant omnibus result can be followed by corrected pairwise tests to identify differing models.

  • Cochran’s Q test generalizes McNemar’s test to compare three or more classifiers on paired nominal data.It indicates whether any models differ, but not which specific groups differ.
  • The test evaluates the null hypothesis that classifiers tested on the same dataset have equal classification accuracies.
  • Classifier predictions are organized in a binary n × M matrix, with entries marking correct and incorrect predictions for each test example.Rows represent test examples and columns represent classifiers.
  • In the worked example, the Q statistic gives p ≈ 0.023 with 2 degrees of freedom, so equal performance is rejected at α = 0.05.
  • After rejecting the omnibus null, corrected post hoc pairwise tests such as McNemar’s test with Bonferroni correction can identify which classifier pairs differ.Multiple comparisons are difficult, and statistical tests should not receive excessive emphasis when assessing evidence in data.

4.7 The F-test for Comparing Multiple Classifiers

The F-test provides a procedure for comparing multiple classifiers evaluated on the same dataset. It tests equal classification accuracies through sums of squares, an F statistic, and a subsequent p-value decision.

  • The F-test compares multiple classifiers tested on the same dataset under the null hypothesis that their classification accuracies are equal.
  • Under equal performance, the F statistic follows an F distribution with (M −1) and (M −1) × n degrees of freedom.
  • The calculation defines average classifier accuracy and decomposes performance variation into sums of squares for classifiers, objects, and their interaction.
  • Mean-square terms are computed from the sums of squares, and their ratio produces the F statistic used for inference.
  • A significant F-test can be followed by pairwise tests such as McNemar tests with Bonferroni correction to determine which population proportions differ.An implementation is available through MLxtend.

4.8 Comparing Algorithms

Algorithm comparison evaluates sets of models fitted to different training sets, making the effect of training-set variation relevant. The section contrasts statistical tests and recommends 5x2cv when repeated fitting is possible.

  • Comparing model sets fitted to different training sets shifts the task from model comparison toward algorithm comparison.This supports choosing an algorithm for deployment across datasets from a similar problem domain.
  • Dietterich’s 5x2-Fold Cross-Validation method is presented as a common technique for comparing supervised learning algorithms.
  • The resampled paired t-test has a high false positive rate and is computationally very expensive, while the difference in proportions test also has a high false positive rate.
  • McNemar’s test is a good choice for relatively large datasets or when model fitting can occur only once.
  • The 5x2cv test considers varying or resampled training sets and is recommended when repeated model fitting is possible.It is somewhat more powerful than McNemar’s test but requires substantially more computation.

4.9 Resampled Paired t-Test

The resampled paired t-test repeatedly splits a labeled dataset, fits two classifiers on shared training sets, and compares their performance differences. It is not recommended because its dependence structure violates Student’s t-test assumptions.

  • The resampled paired t-test repeatedly uses typically 2/3 training data and 1/3 test data to compare two classifiers across k resampling iterations.
  • Each iteration fits both classifiers on the same training set, evaluates them on the same test set, and records their accuracy difference.
  • A p-value below the chosen significance level, such as α = 0.05, leads to rejecting the null hypothesis of no significant difference between the models.
  • The method is not recommended because shared test sets make accuracies dependent and overlapping resampled test sets make differences non-independent, violating Student’s t-test assumptions.

4.10 k-fold Cross-validated Paired t-Test

The k-fold cross-validated paired t-test improves on resampled paired testing but remains unsuitable in practice because its training sets overlap. It is essentially a resampled paired t-test using k-fold cross-validation.

  • Limitation: Overlapping training sets violate an important assumption, so the method is not recommended for practical use.This limitation remains despite addressing some drawbacks of the resampled paired t-test.
  • Context: The method is presented as a common technique in older literature for comparing model performances.The procedure was also made available through MLxtend for comparison studies.
  • Procedure: The procedure replaces simple resampling with k-fold cross-validation when computing the paired t statistic.The number of cross-validation rounds is denoted by k.

4.11 Dietterich’s 5x2-Fold Cross-Validated Paired t-Test

Dietterich’s 5x2cv paired t-test compares two models through five repeated half-splits, evaluating both train/test orientations in each iteration. Its resulting statistic supports a significance test for performance differences.

  • Purpose: The 5x2cv paired t-test compares two classifiers or regressors while addressing shortcomings of resampled and k-fold cross-validated paired t-tests.It was proposed by Dietterich as an alternative to those procedures.
  • Procedure: Five iterations each split the data into 50% training and 50% test subsets.The procedure repeats this splitting five times.
  • Procedure: Each iteration evaluates both classifiers, then swaps the training and test subsets to obtain two performance differences.The two difference measures come from the original and rotated splits.
  • Statistic: The procedure estimates the mean and variance of the differences before computing the t statistic.The variance across the five iterations is used in the statistic.
  • Inference: Under equal model performance, the t statistic approximately follows a t distribution with 5 degrees of freedom.A p-value below a chosen significance level, such as α = 0.05, leads to rejecting the null hypothesis.

4.12 Alpaydin’s Combined 5x2cv F-test

The combined 5x2cv F-test is presented as a more robust alternative to Dietterich’s 5x2cv paired t-test for comparing model performance. The surrounding discussion places such comparisons within small-data evaluation and selection problems.

  • Purpose: The combined 5x2cv F-test compares classifier or regressor performance and was proposed as a more robust alternative to Dietterich’s paired t-test.It was proposed by Alpaydin.
  • Procedure: The method computes an F statistic for two classifiers using notation reused from the preceding 5x2cv procedure.The supplied passage introduces the computation but does not reproduce the equation.
  • Inference: The resulting statistic is approximately F-distributed with 10 and 5 degrees of freedom.The combined test is available through MLxtend.
  • Interpretation: Statistical significance should be distinguished from practical significance because large samples can make very small effects appear significant.The article notes that effect sizes depend on the specific problem, task, and question.
  • Small-data setting: Small datasets motivate alternatives such as nested cross-validation because reserving an independent test set may be infeasible.Nested cross-validation is described as having low bias in small-data settings.
  • Nested cross-validation: Nested cross-validation uses an inner loop for model selection and an outer loop for estimating generalization accuracy.It consists of two nested k-fold cross-validation loops.

4.15 Conclusions

The conclusions present recommendations as problem-dependent guidance for model evaluation, model comparison, and algorithm comparison. They emphasize cross-validation for limited datasets while cautioning that its estimates and statistical tests are not definitive.

  • Recommendations: The article concludes with a diagram summarizing recommendations based on the reviewed concepts and literature.The recommendations address model evaluation, model selection, and algorithm selection techniques.
  • Statistical testing: Parametric performance tests often violate independence assumptions because models and performance estimates share training or test data.With limited datasets, the article treats statistical tests as heuristics for decision making.
  • Dataset size: The recommended techniques depend on the problem and dataset size rather than applying universally.Large test datasets can provide reliable generalization estimates, whereas single splits can be problematic with few records.
  • Small datasets: For very small datasets, large-k cross-validation or leave-one-out cross-validation can evaluate generalization when holding out test data is infeasible.These procedures estimate average performance across test folds, but they compare algorithms producing different fold-specific models rather than identical models.
  • Model comparison: For model comparisons without multiple independent test sets, k-fold cross-validation, the 5x2cv method, and nested cross-validation are available alternatives.Cross-validation is described as the only non-parametric method available for testing model generalization, though not a perfect solution.
  • Caution: Cross-validation is useful but not a silver bullet, so its estimates and assumptions require careful interpretation.The article presents it as the best available tool under practical data constraints.
Loading 1811.12808v3…