Systematic Misestimation of Machine Learning Performance in Neuroimaging Studies of Depression

1 Introduction

• Large-sample neuroimaging studies report weaker performance than small-sample studies, challenging the validity and generalizability of many proof-of-concept findings.Machine learning aims to make individual predictions that could support clinical utility.Source: §1 · p. 2
• In PAC 2018, models classified MDD and healthy controls from structural MRI with 60%–65% accuracy in a large independent test set, versus 80% or more in smaller studies.The competition drew about 170 machine learners in 49 teams.Source: §1 · p. 2
• Across schizophrenia, MDD, and ADHD studies, sample size was inversely related to balanced accuracy; a review of 200 studies likewise found lower reported accuracy in larger samples.The reported correlations were r = −0.34, −0.32, and −0.43, respectively.Source: §1 · p. 2
• Prior work links high-dimensional, complex classification to a need for large training samples and flags possible cross-validation biases, while empirical tests of sample-size effects remained limited.The study separately examines training and test sizes, tests 48 additional pipeline configurations, and repeats the procedure with a dummy classifier.Source: §1 · p. 2

2.1 Data description

• The PAC dataset included 2,240 participants from two independent studies, with structural MRI and clinical diagnoses; analyses used a balanced sample of 934 MDD cases and 934 controls.The data came from the Münster Neuroimaging Cohort and FOR2107; MRI images were preprocessed into grey-matter segments.Source: §2.1 · pp. 2–3, §2.2 · p. 3

2.2 Machine learning pipeline

• The pipeline prevented information leakage, downsampled images to 3 mm voxels, retained 58,255 features, standardized them, and trained a linear SVM with default parameters.Random under-sampling balanced the classes; scanner-distribution effects were negligible.Source: §2.2 · p. 3

2.3 Overall sample size effects

• For overall sample sizes N = 4–150, the researchers generated 1,000 balanced samples per size and estimated accuracy distributions using LOOCV.They trained 11,315,000 SVMs in total.Source: §2.3 · p. 3

2.4 Training set size effects

• To isolate training-set size, they trained models on 1,000 balanced samples per N = 4–150 and evaluated each on a fixed balanced test set of N = 300.They estimated overestimation probabilities from the resulting distributions.Source: §2.4 · p. 3

2.5 Test set size effects

• To isolate test-set size, they evaluated one SVM on 1,000 balanced test samples per N = 4–150 and compared results with a separate performance estimate.The classifier was trained on 627 MDD cases and 627 controls; a separate sample of 157 per class provided the comparison estimate.Source: §2.5 · p. 3

2.6.1 Alternative pipeline configurations

• Forty-eight additional configurations tested linear and radial-basis SVMs, Random Forests, PCA, and F-test feature selection.These alternatives assessed whether findings depended on the original pipeline or its large predictor set.Source: §2.6.1 · p. 5

2.6.2 Dummy classifier

• A dummy classifier used the training-set class prevalence as its prior; balancing made its ground-truth accuracy exactly 50%.This comparison assessed sample-size-related misestimation independently of the PAC data characteristics and analysis pipeline.Source: §2.6.2 · p. 5

3.1 Overall sample size effects

• 13% of N = 20 LOOCV samples reached at least 70% accuracy, versus 2% at N = 100; accuracy ranges narrowed from 10%–95% to 35%–81%.The effect was also present with the dummy classifier and remained under adjusted SVM regularization.Source: §3.1 · p. 5, Figure 2 · p. 6

3.2 Training set size effects

• With a fixed test set of N = 300, training sets of N = 4–150 showed no systematic misestimation; accuracy ranged from 32%–69% at N = 20 and 51%–70% at N = 100.Larger training sets reduced the chance of very low estimates; the dummy-classifier results showed the effect was general.Source: §3.2 · p. 5, Figure 3 · p. 7

3.3 Test set size effects

• 30% of N = 20 test samples reached at least 70% accuracy, versus 13% at N = 100; mean accuracy on the full dataset was 61%.Accuracy ranges narrowed from 35%–95% at N = 20 to 51%–79% at N = 100; the pattern also appeared with a dummy classifier and across 48 pipeline configurations.Source: §3.3 · pp. 5, 8, Figure 4 · p. 8

4 Discussion

• Small test sets, not training-set size, primarily drive the risk of artificially high performance estimates in the PAC analyses.The authors report that test sets of N = 100 reduced the probability of accuracy of 70% or higher to 13%.Source: §4 · p. 9
• For the linear SVM, performance stopped increasing once training-set size reached 30, while more complex rules may benefit from more training data.The authors note that high-complexity models did not yield higher performance when trained with approximately 1,000 samples in the PAC competition.Source: §4 · p. 9
• All 48 additional pipeline configurations showed the same systematic misestimation pattern, as did dummy classifiers that ignored participant characteristics and imaging inputs.The additional configurations included linear and nonlinear classifiers, dimensionality reduction, and feature selection.Source: §4 · p. 9
• Random sampling and dummy classifiers suggest that small-test-set variation, or sampling error, can produce above-chance performance even without patient-specific patterns.This contrasts with a prior explanation based on more homogeneous patient characteristics in small samples.Source: §4 · pp. 9–10
• Data leakage is another issue likely contributing to systematic overestimation in the literature.The authors also call for model code and more elaborate evaluation frameworks.Source: §4 · p. 10
• The authors conclude that large test sets are a straightforward remedy and advise evaluating promising models on them rather than discarding models because training N is low.They qualify that conclusion by noting that simple, low-complexity linear SVMs did not gain from larger training sets in this analysis.Source: §4 · p. 10

FOR2107

• Funding included German Research Foundation grants and support from the Münster medical faculty’s Interdisciplinary Center for Clinical Research.Source: FOR2107 · p. 10
• The work is part of the German multicenter FOR2107 consortium on the neurobiology of affective disorders, funded by the German Research Foundation.Source: FOR2107 · p. 10
• FOR2107 responsibilities span brain imaging, animal phenotyping, miRNA, immunology and mitochondria, genetics, multi-method analytics, and biobank and administration coordination.Source: FOR2107 · p. 10
• All principal investigators take responsibility for the integrity of their respective study data and components, and all authors and coauthors had full access to the data.Source: FOR2107 · p. 10

Conflict of Interest

• Tilo Kircher received unrestricted educational grants from five named pharmaceutical companies; the other listed authors declared no conflicts of interest.Source: Conflict of Interest · p. 11

• The passage refers to neuroimaging studies of depression.Source: SUPPLEMENTAL MATERIALS Systematic… · p. 14

A Summary statistics for the final study sample

• 934 MDD patients and 934 HC made up the final study sample.The sample summary also covers sex, scanner site, age, and total intracranial volume.Source: A Summary statistics for the final… · p. 14, Table A.1 · p. 14

B Predictive Analytics Competition 2018

• PAC 2018 invited teams worldwide to classify MDD patients and healthy controls using structural MRI data.Source: B Predictive Analytics Competition 2018 · p. 14
• The study sample was drawn from the Predictive Analytics Competition 2018 dataset.Source: B Predictive Analytics Competition 2018 · p. 14, A Summary statistics for the final… · p. 14

Data

• PAC 2018 provided structural MRI data from 2,240 subjects, with age, gender, TIV, and scanner site available as covariates.Images were preprocessed with CAT-12 and quality checked.Source: Data · p. 14
• The competition data were split into training and test sets in advance, and the test set was reserved for selecting the winner.Source: Data · p. 14, Table B.2 · p. 15

Timeline

• The competition opened registration on February 1, 2018, and announced its award winner on June 21, 2018.Training data were available March 1; test data became available April 27; prediction uploads ran May 4–31.Source: Timeline · pp. 14–15

Results

• The winning team had the highest balanced accuracy on the held-back test set.The team was “paranoidandroid.”Source: Results · p. 15, Table B.3 · p. 15

C Additional tabular presentations

• The tables report accuracy distributions and threshold-exceedance probabilities across varying overall sample sizes for linear SVM and dummy-classifier analyses.Table C.4 covers a linear SVM using LOOCV; Table C.5 covers a dummy classifier.Source: Table C.4 · p. 16, C Additional tabular presentations · pp. 16–17, Table C.5 · p. 17
• Additional tables present accuracy distributions and threshold probabilities for varying training and test sample sizes.The training-size tables use a fixed validation set of n = 300; the test-size tables evaluate fixed pretrained classifiers.Source: Table C.6 · p. 18, C Additional tabular presentations · p. 18, Table C.7 · p. 19

D Influence of the scanner distribution

• Scanner distribution significantly influenced test-set accuracy but explained only 0.79% of its variance.The analysis related scanner-distribution Gini indices to accuracy on a hold-out test set of n = 300.Source: D Influence of the scanner distribution · p. 22

E Adjustment of SVM regularization based on sample size

• Adjusted SVM regularization did not match the results obtained with C = 1, and the default C value was not judged to increase misestimation as sample size decreased.The adjustment set C = k/sample size, with k = 75; for n = 100, C = 1.33.Source: E Adjustment of SVM regularization… · p. 22, Figure E.1 · p. 22, Figure E.2 · p. 23
• The adjustment used C = k/sample size, with k = 75 to approximate default C = 1 on average.At sample size n = 100, the adjusted parameter was C = 1.33.Source: E Adjustment of SVM regularization… · p. 22

F Alternative Machine Configurations

• The alternative configurations combined feature-space reduction and selection with linear and RBF SVMs or a random forest.The classifiers used default parameters; preprocessing combinations are listed in Table F.10.Source: F Alternative Machine Configurations · p. 23, Table F.10 · p. 23
• The alternative preprocessing and classifier combinations yielded 48 configurations, and the analyses were repeated for test-set size.The authors state that the observed effect was limited to test-set size.Source: F Alternative Machine Configurations · p. 23

F.1 Results

• Results were comparable to the original linear-kernel SVM configuration without preprocessing, underlining the findings’ generally valid character.The passage references Figures F.3–F.50 and Tables F.11–F.58.Source: §F.1 · p. 24
• The PCA configuration with 10 components and an RBF-kernel SVM was an outlier.The passage references Figure F.22 and Table F.30.Source: §F.1 · p. 24
• Overfitting was offered as an explanation for the outlier: the model constantly predicted one class.Source: §F.1 · p. 24

F.1.1 Graphical results representation

• The figures plot results against test-set size for fixed Random Forest, RBF-kernel SVM, and linear SVM classifiers.The captions specify default classifier parameters.Source: Figure F.3 · p. 24, Figure F.4 · p. 25, Figure F.5 · p. 25
• ANOVA feature-selection plots vary the number of retained features from 10 to 50,000 and show results across the three classifiers.Feature selection is performed inside the pipeline using ANOVA F-values.Source: Figure F.6 · p. 26, Figure F.7 · p. 26, Figure F.8 · p. 27
• PCA-based plots show results with 10, 50, or 500 retained components for Random Forest and both SVM classifiers.The figures vary test-set size while holding each classifier fixed.Source: Figure F.21 · p. 33, Figure F.22 · p. 34, Figure F.23 · p. 34
• Additional figures combine 50-component PCA with ANOVA selection of the top 10 features before classification.They include Random Forest, RBF-kernel SVM, and linear SVM plots.Source: Figure F.27 · p. 36, Figure F.28 · p. 37, Figure F.29 · p. 37
• The Random Forest results are represented as a function of test sample size with all PCA components and the top 1,000 ANOVA features retained.The figure caption specifies default Random Forest parameters and 100 estimators.Source: §F.1.1 · p. 47, Figure F.48 · p. 47
• The SVM with an RBF kernel is shown as a function of test sample size with all PCA components and the top 1,000 ANOVA features retained.The figure caption gives C = 1.0 and γ = 1/nfeature.Source: §F.1.1 · p. 47, Figure F.49 · p. 47
• The linear SVM is shown as a function of test sample size with all PCA components and the top 1,000 ANOVA features retained.The figure caption gives C = 1.0.Source: §F.1.1 · p. 48, Figure F.50 · p. 48

F.1.2 Tabular results representation

• Tables F.11–F.38 report accuracy-threshold probabilities and accuracy distributions as test-set size varies with a fixed classifier.The tables pair probabilities of exceeding an accuracy threshold with summaries of accuracy distributions for selected sample sizes; no numerical outcomes are supplied in the passages.Source: §F.1.2 · p. 49, Table F.11 · p. 49
• ANOVA feature selection uses top 10 to 50,000 features, paired with Random Forest, RBF SVM, or linear SVM classifiers.The listed feature counts include 10, 100, 1,000, 10,000, and 50,000.Source: Table F.14 · p. 52, Table F.17 · p. 55, Table F.20 · p. 58
• PCA variants retain 10, 50, or 500 components and evaluate Random Forest, RBF SVM, and linear SVM classifiers.The tables also include a PCA model retaining 50 components followed by ANOVA selection of the top 10 features.Source: Table F.29 · p. 67, Table F.32 · p. 70, Table F.35 · p. 73
• Results are represented as probabilities of exceeding accuracy thresholds related to test-set size, with PCA retaining 500 components and an RBF-kernel SVM in the example.The passage gives no threshold probabilities or accuracy values.Source: Table F.39 · p. 77, §F.1.2 · p. 78
• Accuracy-value distributions are also summarized for selected sample sizes, as illustrated for the same fixed-classifier setup.The supplied passage describes the summary but provides no distribution values.Source: Table F.39 · p. 77, §F.1.2 · p. 78
• Across tables evaluating variable test-set sizes with a fixed classifier, PCA retained 500 components; ANOVA-based selection used the top 10 or top 100 features, with linear SVM, RBF SVM, or Random Forest.Source: Table F.40 · p. 78, Table F.41 · p. 79, Table F.42 · p. 80

G Abbreviations

• The abbreviations define MDD as major depressive disorder, PAC as Predictive Analytics Competition, MRI as magnetic resonance imaging, and SVM as support-vector machine.They also define LOOCV as leave-one-out cross-validation and PCA as principal-component analysis.Source: G Abbreviations · p. 97
• Other listed abbreviations are ADHD for attention deficit hyperactivity disorder, TIV for total intracranial volume, and RBF for radial basis function.Source: G Abbreviations · p. 97