Assessing and tuning brain decoders: cross-validation, caveats, and guidelines

Gaël Varoquaux; Pradeep Reddy Raamana; Denis Engemann; Andrés Hoyos-Idrobo; Yannick Schwartz; Bertrand Thirion

Source-linked AI summary

Assessing and tuning brain decoders: cross-validation, caveats, and guidelines

Gaël Varoquaux, Pradeep Reddy Raamana, Denis Engemann, Andrés Hoyos-Idrobo, Yannick Schwartz, Bertrand Thirion

arXiv:1606.05201v2Jun 16, 2016stat.ML

TL;DR

The paper addresses how to evaluate predictive power and tune neuroimaging decoders despite limited, high-dimensional, and correlated data. It combines theoretical discussion with extensive empirical studies and finds that repeated random splits are preferable to leave-one-out, while parameter defaults can outperform nested tuning for some non-sparse decoders.

Problem
Neuroimaging decoding requires reliable estimates of generalization and principled parameter choices despite scarce data, high dimensionality, and model-weight interpretability concerns.
Method
The paper reviews cross-validation theory and practice and evaluates common decoders using multiple neuroimaging datasets, simulations, and nested cross-validation.
Results
Repeated random splits with blocks of correlated observations left out provide better predictive-power estimates than leave-one-out, while sane defaults often outperform nested tuning for non-sparse decoders.
Takeaways & Limitations
Best practice is repeated learning-testing with 20% of the data left out, with parameter tuning nested to limit optimistic bias and decoder choices guided across several datasets.
Takeaways & Limitations
Cross-validation estimates in neuroimaging have large uncertainty, with empirical confidence intervals typically extending more than 10 points up and down because of data scarcity.

Abstract

from arXiv · show

Decoding, ie prediction from brain images or signals, calls for empirical evaluation of its predictive power. Such evaluation is achieved via cross-validation, a method also used to tune decoders' hyper-parameters. This paper is a review on cross-validation procedures for decoding in neuroimaging. It includes a didactic overview of the relevant theoretical considerations. Practical aspects are highlighted with an extensive empirical study of the common decoders in within-and across-subject predictions, on multiple datasets --anatomical and functional MRI and MEG-- and simulations. Theory and experiments outline that the popular " leave-one-out " strategy leads to unstable and biased estimates, and a repeated random splits method should be preferred. Experiments outline the large error bars of cross-validation in neuroimaging settings: typical confidence intervals of 10%. Nested cross-validation can tune decoders' parameters while avoiding circularity bias. However we find that it can be more favorable to use sane defaults, in particular for non-sparse decoders.

1. Introduction: decoding needs model evaluation

Decoding requires empirical evaluation because predictive performance must generalize beyond observed brain data, while high-dimensional models require regularization choices that affect both prediction and highlighted brain features.

Predictive power must be assessed on prospective data rather than only on the observations used to fit the decoder.
High-dimensional neuroimaging data create many estimates with similar training errors, making regularization necessary for choosing among solutions.
Regularization and decoder-parameter choices can affect both prediction performance and the brain features emphasized by the model.
Cross-validation is presented as the standard tool for measuring predictive power, selecting among decoders, and tuning decoder parameters.

2. A primer on cross-validation

Cross-validation estimates how well neuroimaging decoders generalize by separating training from unseen test data, while also supporting hyper-parameter tuning. Its design must balance bias, variance, dependence among observations, and the competing goals of predictive accuracy and interpretable, stable model weights.

2.1. Cross-validation: estimating predictive power: Cross-validation trains a decoder on one data split and estimates predictive power on held-out observations that were unseen during training.This separation addresses the difficulty of measuring generalization to new data.
2.1. Cross-validation: estimating predictive power: Train-test independence requires temporal separation for autocorrelated fMRI data and test samples that represent uncontrolled sources of variability.Leaving out part of each session can be easier than predicting an entirely new session.
2.1. Cross-validation: estimating predictive power: Leave-one-out cross-validation preserves training data but maximizes test-set variance and can produce unbalanced test sets containing only one class.Small test sets generally provide weak power for estimating prediction error, so cross-validation balances training size against test-set variance.
2.2. Hyper-parameter selection: Regularization is necessary in high-dimensional neuroimaging, but too little causes overfitting while too much causes underfitting and harms predictive power and weight estimation.The best bias-variance tradeoff depends on data amount and task signal-to-noise ratio.
2.2. Hyper-parameter selection: Nested cross-validation tunes regularization inside an inner loop while reserving an outer loop for unbiased performance assessment.Model averaging and bagging provide additional strategies for stabilizing predictions across suitable models.
2.3. Model selection for neuroimaging decoders: Decoder selection in neuroimaging must consider weight-map stability alongside prediction because sparse models can improve prediction while producing less stable maps.The empirical comparison reported better prediction but worse weight-map stability for sparse models, whereas non-sparse SVM with C = 1 was a good stability choice.

3. Empirical studies: cross-validation at work

Empirical studies across MRI, MEG, and simulations show that cross-validation strategy strongly affects predictive estimates, while parameter-tuning choices trade prediction against map stability.

Empirical design: Experiments across MRI, MEG, and simulated data evaluated cross-validation reliability and decoder parameter-tuning strategies.The study examined within- and across-subject decoding and compared prediction accuracy with stability of decoder maps.
Cross-validation reliability: Cross-validation estimates show substantial discrepancies from validation-set performance, including pessimism for high-accuracy models and optimism for poorly performing models.The estimates also display large variance across nested cross-validation splits and validation sets.
Cross-validation strategies: 10 to 50 repeated random splits leaving out 20% of data blocks reduce variance and estimate accuracy better than successively leaving out each block.The repeated-split strategy also costs less computation when the number of blocks is large.
Cross-validation strategies: Leave-one-sample-out cross-validation can be optimistic when samples are correlated, whereas block-based splits reduce this bias.In simulated temporally autocorrelated data, leave-one-sample-out was strongly optimistic; leaving blocks out with many splits gave better results.
Parameter tuning: Non-sparse models are generally insensitive to regularization tuning, so nested cross-validation does not generally outperform a default hyper-parameter choice.Their tuning curves have wide plateaus, whereas sparse models have narrower peaks and can benefit from refitting after nested tuning when prediction accuracy is good.
Parameter tuning: For sparse models, default C = 1 performs well, while high C values can produce poor performance; tuning choices also affect map stability.Model averaging gives the most stable maps, followed by refitting after nested cross-validation, and non-sparse models are more stable than sparse ones.

4. Discussion and conclusion: lessons learned

Cross-validation is useful but uncertain in small neuroimaging samples, so evaluation and parameter choices require careful validation across correlated data and datasets.

Lessons learned: Small neuroimaging samples limit the reliability of cross-validation estimates, whose confidence intervals typically extend more than 10 points up and down.This uncertainty affects predictive-power assessments used in decoding and hypothesis testing.
Lessons learned: Decoder and parameter choices should be guided by several datasets because small samples produce substantial variance.
Lessons learned: Leave-one-out cross-validation should be avoided because it produces more variable results.Validation should leave out blocks of correlated observations rather than individual observations.
Lessons learned: Repeated random splits leaving out 20% of the data provide better estimates with less computation by increasing the number of cross-validations without shrinking test sets.
Lessons learned: Parameter tuning should use nested cross-validation to limit optimistic bias, although sane defaults can outperform tuning for non-sparse decoders.Parameter selection can also substantially change weight maps and their stability.
Lessons learned: Across 31 decoding tasks and 8 datasets, sparse models—especially ℓ1 SVM with model averaging—gave better prediction but worse weight-map stability than non-sparse classifiers.For stable weight maps, non-sparse SVM with C = 1 appeared to be a good choice.

Appendix A.1. Dataset simulation

The simulations generate two Gaussian classes in 100 dimensions with controlled separability and temporally correlated noise, while using 2D views only for visualization.

Dataset simulation: Each simulated sample belongs to one of two classes, each modeled by a 100-dimensional Gaussian with identity covariance.
Dataset simulation: Class centers are vectors of all µ or all −µ, so increasing µ increases expected predictive accuracy.
Dataset simulation: Gaussian smoothing with σ = 2 is applied along the sample direction to mimic temporal dependence in neuroimaging noise.This makes successive samples exhibit correlated shifts.
Dataset simulation: The simulations vary µ across (.05, .1, .2) to create datasets with predefined separability.The actual experiments use 100 features, whereas Figure A1 shows 2D data for visualization.

Appendix A.2. Experiments: error varying separability

Simulation experiments compare cross-validation strategies against large-test-set performance across separability levels and examine parameter-tuning behavior for sparse and non-sparse models.

Experiments: error varying separability: The simulations use 200 training samples and a separate 10,000-sample test set to compare cross-validation estimates with prediction performance.The procedure is repeated 100 times using the same decoders as in the brain-imaging analyses.
Experiments: error varying separability: Leave-one-sample-out cross-validation is strongly biased by correlations across samples.
Experiments: error varying separability: Figure A2 compares leave-one-sample-out, leave-one-block-out, and repeated random-split strategies using 3, 10, or 50 splits.
Experiments: error varying separability: All strategies tend to overestimate prediction for low accuracy and underestimate it for high accuracy.This pattern is observed after setting aside leave-one-sample-out cross-validation.
Experiments: error varying separability: For low predictive power, setting C=1 in non-sparse models is preferable to cross-validation, whereas cross-validation is similarly efficient at high predictive power.
Experiments: error varying separability: The comparisons aggregate points from different studies and classification tasks, whose observed trends are fairly homogeneous.

Appendix C. Results without variance-normalization

Without variance normalization, cross-validation remains no more reliable, while nested parameter tuning becomes more important for prediction, especially because defaults transfer less consistently across datasets.

Results without variance-normalization: Figures A4–A6 examine error correspondence, prediction performance, and weight-stability effects of parameter-tuning strategies without variance normalization.
Results without variance-normalization: Without variance normalization, cross-validation is not more reliable than with variance-normalized neuroimaging data.
Results without variance-normalization: Nested cross-validation becomes more important for good prediction when neuroimaging data are not variance-normalized.
Results without variance-normalization: Variance normalization makes datasets more comparable, increasing the likelihood that a default parameter value works well across them.
Results without variance-normalization: Variance normalization can be important, particularly for non-sparse SVM.

Appendix D.1. Details on stability–prediction results

Figures 11, A7, and A8 examine how decoding strategy affects prediction accuracy and weight stability across datasets and tasks. The main variation in accuracy comes from dataset difficulty, while penalty choice and model averaging shape stability.

Figures A7–A8: Figure A7 provides task-level data linking weight stability to changes in prediction accuracy relative to each split’s average.Each point represents a specific prediction task.
Figures A7–A8: Figure A8 measures weight stability as the correlation across validation splits and compares decoder and parameter-tuning choices across datasets and tasks.A lowess local regression indicates the overall tendency.
Across datasets: Dataset difficulty is the main source of variation in prediction accuracy, although classification tasks differ in ease and stability.The figures therefore capture interactions between task difficulty and decoding strategy.
Decoder choices: Penalty choice is the most important decoding decision, with logistic regression and SVM showing overall similar behavior.The comparison concerns their prediction–stability tradeoff across the study’s tasks.
Decoder choices: Higher prediction accuracy generally corresponds to greater weight stability, except for the overly penalized sparse model with C=1000.For non-sparse models, averaging models after cross-validation is especially beneficial when prediction is good.

Appendix E. Details on datasets used

The experiments use multiple prediction tasks assembled from several neuroimaging datasets. In Haxby, the tasks are visual-stimulus classification pairs selected to vary in difficulty.

Haxby dataset: The Haxby dataset supplies visual-stimulus pairs with differing task difficulty, while unsuccessful pairs such as scissors versus bottle were excluded.Table A1 lists the studies and specific prediction tasks used in the experiments.

Appendix F.1. fMRI data

The fMRI analyses include intra-subject Haxby data and inter-subject open fMRI datasets. Haxby preprocessing adds 6 mm spatial smoothing, while inter-subject preprocessing follows a standard SPM8 pipeline.

Intra-subject prediction: For intra-subject prediction, the study uses the Haxby dataset distributed through the PyMVPA website.
Intra-subject prediction: The Haxby data receive no additional preprocessing beyond the public dataset, except isotropic Gaussian spatial smoothing with 6 mm FWHM.The preprocessing uses nilearn 0.2 and Python 2.7.
Inter-subject prediction: Inter-subject prediction uses datasets from open fMRI processed with SPM8 through slice-time correction, motion correction, coregistration, and template-space normalization.The preprocessing pipeline is orchestrated with Nipype.

Appendix F.2. Structural MR data

The study uses structural MR data from the Oasis dataset and MEG data from an HCP working-memory experiment. Structural analyses use VBM, while MEG analyses use temporal windows across sensor-array measurements.

Structural MR data: Structural MR prediction uses Voxel Based Morphometry on Oasis data, with modulated grey-matter intensity as the predictive-model input.Preprocessing includes tissue segmentation and DARTEL deformation-field estimation.
Evaluation setup: Table A1 reports validation performance for parameter tuning using 10 random splits followed by refitting with the best parameter.
MEG data: The MEG analysis uses 52 subjects and two runs from an HCP N-back working-memory experiment, with each run validating the other.A temporal-window approach converts sensor-array fields within fixed intervals into variable sets.
MEG data: The MEG analyses use cleaned single-trial outputs from the HCP tmegpreproc pipeline recorded with a whole-head MAGNES 3600 magnetometer system.Environmental magnetic contamination was addressed using remotely located reference sensors.

Appendix G. Performance on each classification task

Figure A9 compares prediction accuracy across decoding strategies and datasets, showing mostly dataset-driven variation with a few strategy-specific exceptions.

Appendix G. Performance on each classification task: Prediction accuracy is presented for each decoding strategy across all datasets in Figure A9.The figure’s x positions denote different decoding strategies.
Appendix G. Performance on each classification task: Prediction-accuracy variations between strategies are mostly reported across datasets, with the plotted lines roughly parallel.
Appendix G. Performance on each classification task: SVM ℓ1 with C = 1000 is a notable exception because some datasets show a strong decrease in prediction accuracy.
Appendix G. Performance on each classification task: ℓ1 models tend to perform better on the Haxby dataset, the source of intra-subject classification tasks.The text describes this as a weaker variation than the dataset-driven pattern.
Appendix G. Performance on each classification task: For non-sparse models, using a large C without parameter tuning is described as an overall safe choice.For sparse models, model averaging, refitting, and C = 1 do not offer a clear win, although model averaging is comparatively less variable.