PaperlayerView exact revision on arXiv

Source-linked AI summary

Assessing and tuning brain decoders: cross-validation, caveats, and guidelines

Gaël Varoquaux, Pradeep Reddy Raamana, Denis Engemann, Andrés Hoyos-Idrobo, Yannick Schwartz, Bertrand Thirion

arXiv:1606.05201v2stat.ML

TL;DR

Decoding studies need to estimate how well models generalize to unseen data, often with few observations relative to features. This review combines cross-validation theory with empirical evaluations and finds sparse decoders predicted better but had less stable weight maps than non-sparse classifiers.

  • Problem

    Decoding requires estimating generalization to unseen data, while neuroimaging often has far more features than observations and may require regularization tuning.

    Source: §1 · p. 1, §2.1 · p. 2, §2.3 · p. 3

  • Method

    The paper reviews cross-validation and empirically evaluates decoder performance across 31 tasks and eight datasets, including nested cross-validation.

    Source: §4 · p. 9

  • Results

    Across 31 decoding tasks, sparse models—especially ℓ1 SVM with model averaging—predicted better but had worse weight-map stability than non-sparse classifiers.

    Source: §4 · p. 9

  • Takeaways & Limitations

    Repeated random splits with 20% of data held out and blocks of correlated observations left together provide better, less computationally costly estimates.

    Source: §4 · p. 9

    Cross-validation accuracy estimates typically had empirical confidence intervals extending more than 10 points in either direction across MRI, MEG, and simulations.

    Source: §4 · p. 8

Abstract

from arXiv · show

Decoding, ie prediction from brain images or signals, calls for empirical evaluation of its predictive power. Such evaluation is achieved via cross-validation, a method also used to tune decoders' hyper-parameters. This paper is a review on cross-validation procedures for decoding in neuroimaging. It includes a didactic overview of the relevant theoretical considerations. Practical aspects are highlighted with an extensive empirical study of the common decoders in within-and across-subject predictions, on multiple datasets --anatomical and functional MRI and MEG-- and simulations. Theory and experiments outline that the popular " leave-one-out " strategy leads to unstable and biased estimates, and a repeated random splits method should be preferred. Experiments outline the large error bars of cross-validation in neuroimaging settings: typical confidence intervals of 10%. Nested cross-validation can tune decoders' parameters while avoiding circularity bias. However we find that it can be more favorable to use sane defaults, in particular for non-sparse decoders.

1. Introduction: decoding needs model evaluation

Decoding predicts behavior or phenotypes from brain data and supports clinical, cognitive, and brain-mapping applications, all of which depend on measuring predictive power. This paper reviews cross-validation as the standard tool for evaluating and tuning decoders, and reports that its estimates are uncertain, should respect correlated data blocks, and generally favor repeated random splits over leave-one-out.

1. Introduction: decoding needs model evaluation

  • Decoding predicts behavior or phenotypes from brain images or signals and supports clinical diagnosis or prognosis.The passage also describes its use in studying cognition, encoding models, and the functions implied by activations.Source: §1 · p. 1
  • Decoder assessment and tuning are difficult because they require prospective-data evaluation and, in high-dimensional settings, regularization choices that can affect both prediction and highlighted brain features.When features outnumber samples, many estimates can yield the same prediction error; regularization encodes a prior, and its amount may require tuning.Source: §1 · p. 1
  • Cross-validation results carry large uncertainty; evaluation should use full blocks of correlated data, and repeated random splits should be preferred to leave-one-out.The study also provides guidelines for decoder parameter choice in terms of prediction performance and stability.Source: §1 · p. 1

2. A primer on cross-validation

Cross-validation estimates how well a brain decoder generalizes to unseen data. This primer covers split design, regularization and tuning, and neuroimaging-specific model-selection concerns.

2.1. Cross-validation: estimating predictive power

  • Valid estimates require unseen, independent test data—time separation for fMRI and splits matching session or subject generalization—while larger test sets reduce variance but leave less training data.Movement or other confounds should be balanced between training and test sets; small datasets also risk negative bias because each model is fit on only a fraction of the data.Source: §2.1 · p. 2
  • Leave-one-out maximizes test-set variance and yields unstable accuracy estimates; holding out 10% to 20% and using more repeated random splits is preferred [18, chap. 7.12].Repeated random splits should respect dependencies among observations and can be stratified to avoid class imbalance.Source: §2.1 · p. 2

2.2. Hyper-parameter selection

  • Because selecting regularization on test performance makes that estimate unreliable, nested cross-validation tunes the parameter internally and evaluates performance on external held-out splits; the best tradeoff depends on data amount and signal-to-noise.Regularization is needed to constrain high-dimensional models, but too much causes underfitting while too little risks overfitting.Source: §2.2 · pp. 2–3
  • Averaging predictions across suitable models can reduce variance when their errors are sufficiently independent; for linear decoders, this also amounts to averaging weight maps [44, chap. 35].Bagging generates models from bootstrap resamplings and averages their predictions.Source: §2.2 · p. 3

2.3. Model selection for neuroimaging decoders

  • In high-dimensional, data-scarce neuroimaging, ℓ2 regularization often has little effect on decoder maps, whereas ℓ1 regularization changes voxel selection visibly.Correlated neighboring voxels make ℓ1-selected features unstable, motivating approaches such as averaging maps across folds.Source: §2.3 · pp. 3–4
  • Model selection must consider both prediction performance and decoder-map stability, but stability is only an indirect proxy for map estimation error and does not ensure that selected regions are predictive.The challenge is to find a good prediction-stability tradeoff.Source: §2.3 · p. 4

3. Empirical studies: cross-validation at work

Across MRI, MEG, and simulated decoding tasks, the experiments compare cross-validation strategies for estimating predictive power and tuning decoder parameters. They also assess how tuning choices affect the stability of model weights.

3.1. Experiments on real neuroimaging data

  • The real-data study spans seven fMRI datasets, VBM gender decoding, and 31 classification tasks, with 82% of cross-validation data points from within-subject settings.Tasks range from easy to difficult; decoder sample sizes range from 80 to 400 observations with balanced classes. Examples include Haxby and OASIS.Source: §3.1 · pp. 4–5

MEG data.

  • MEG experiments decode image content or working-memory role within 52 subjects, using two runs and four classification analyses per subject.Each run validates the other; features are sensor-signal windows from 50 ms before to 300 ms after event onset.Source: MEG data · p. 5

3.2. Results: cross-validation to assess predictive power

Cross-validation Difference in accuracy measured strategy by cross-validation and on validation set

  • 10–50 repeated random splits holding out 20% of data blocks estimate accuracy better than successive block holdouts at lower compute cost; leave-one-sample-out is optimistic within subjects.Leaving out blocks largely removes the bias, with remaining discrepancies appearing mostly as variance.Source: Cross-validation Difference in… · p. 6, Figure 6 · p. 6
  • In temporally autocorrelated simulations, leave-one-sample-out is strongly optimistic; across MEG and simulations, block holdout with many splits (e.g. 50) gives the best results.MEG estimates are slightly pessimistic and highly variable, likely because of sample inhomogeneities.Source: Cross-validation Difference in… · pp. 6–7, Figure 7 · p. 6

3.3. Results on cross-validation for parameter tuning

  • For non-sparse ℓ2 models, nested tuning generally does not outperform default parameters; for sparse ℓ1 models, default C = 1 suits difficult tasks, while tuning and averaging help when accuracy is good.ℓ2 tuning curves show broad plateaus, whereas ℓ1 curves have narrower peaks, making the selected optimum less likely to carry over to validation data. Refitting can benefit non-sparse models when signal-to-noise ratio or sample count is high.Source: §3.3 · p. 7, Figure 9 · p. 7, Figure 8 · p. 8

3.4. Results: stability of model weights

  • Model averaging yields the most stable weight maps, while ℓ2 models are more stable than ℓ1 models; for non-sparse models, averaging trades a couple of percentage points of accuracy for stability.With sparse models, averaging offers a slight stability advantage and performs well for SVM prediction; C = 1000 gives sparse models a poor prediction–stability tradeoff. Averaging brings larger stability gains for non-sparse models when accuracy is high, and for sparse models when accuracy is poor.Source: Figure 10 · p. 7, §3.4 · p. 8, Figure 11 · p. 8

4. Discussion and conclusion: lessons learned · Appendix A. Experiments on simulated data

The review recommends repeated held-out testing and cautious interpretation of cross-validation’s high variance in small neuroimaging samples. Its empirical results distinguish prediction performance from weight-map stability, with decoder and tuning choices depending on which matters most.

4. Discussion and conclusion: lessons learned

  • Cross-validation directly probes prediction on new data with few assumptions, but small neuroimaging samples constrain its assessment [12].Source: §4 · p. 8
  • Cross-validated accuracy confidence intervals typically extend more than 10 points up and down, warranting caution for predictive-power tests such as searchlight and generalization.Source: §4 · p. 8
  • Because larger inter-subject cohorts such as ADNI and HCP are emerging, the empirical conclusions may not readily transfer to those settings.Source: §4 · p. 8
  • Use permutation to define null hypotheses, and validate new methods across several datasets because cross-validation variance is substantial.Source: §4 · p. 9
  • Repeated random splits leaving out 20% of data give better estimates at lower computational cost; avoid leave-one-out and hold out correlated blocks for unbiased estimates.Source: §4 · p. 9
  • For variance-normalized data, default parameters often outperform nested-CV tuning for weakly sensitive non-sparse ℓ2 decoders, while tuning with model averaging can stabilize sparse ℓ1 maps and improve prediction.Default parameters also give good prediction performance for sparse decoders.Source: §4 · p. 9
  • Cross-validation is not a silver bullet or the sole metric; tune only within nested cross-validation to limit optimistic bias, and select decoders using several datasets.Source: §4 · p. 9
  • Across 31 decoding tasks, ℓ1 SVM with model averaging predicted better but had less stable weight maps; non-sparse SVM with C = 1 suited stability-focused use.The validation covered 8 datasets and almost 1,000 nested cross-validation splits.Source: §4 · p. 9

Appendix A.1. Dataset simulation

The appendix simulates two-class data in 100 dimensions, varying class separation and adding time-correlated noise to approximate neuroimaging data.

Appendix A.1. Dataset simulation

  • Each class is modeled as a 100-dimensional Gaussian with identity covariance.Source: Appendix A.1 · p. 10
  • Class means are (µ, . . . , µ) and (−µ, . . . ,−µ); µ varies across .05, .1, and .2, with larger µ expected to yield higher predictive accuracy.Source: Appendix A.1 · p. 11
  • Gaussian smoothing of the noise along the sample direction (σ = 2) creates time dependence, so successive simulated samples show similar shifts.Figure A1 illustrates separability in a 2D visualization, while the experiments use 100 features.Source: Appendix A.1 · p. 11, Figure A1 · p. 11

Appendix A.2. Experiments: error varying separability · Appendix B. Comparing parameter-tuning strategies

Simulations show cross-validation bias varies with true prediction accuracy, while tuning comparisons suggest that sensible defaults can outperform cross-validation in some low-accuracy settings. The sections also compare tuning strategies across studies and describe how simulated performance and refitted-model errors are evaluated.

Appendix A.2. Experiments: error varying separability

  • The simulation compares cross-validation with actual prediction performance using 200 training samples, 10,000 test samples, the same decoders, and 100 repetitions.For strategies using sample blocks, the data are divided into 10 continuous blocks.Source: Appendix A.2 · p. 11
  • Figure A2 compares leave-one-sample-out, leave-one-block-out, and random splits leaving out 20% of blocks, with 3, 10, or 50 splits.The figure shows cross-validation accuracy alongside accuracy on a very large test set across different simulated separabilities.Source: Figure A2 · p. 11
  • Except for leave-one-sample-out, cross-validation tends to overestimate low-accuracy and underestimate high-accuracy performance as separability changes.The simulated data vary in separability, making the true prediction accuracy visible alongside the cross-validation estimates.Source: Appendix A.2 · p. 11, Figure A2 · p. 11

Appendix B. Comparing parameter-tuning strategies

  • At low predictive power, C=1 can beat cross-validation for nonsparse models, while high-power performance is similar; sparse averaging may beat refitting, and C=1 helps low-accuracy logistic regression.For sparse models, model averaging can be preferable to refitting; the reported trends are small.Source: Appendix B · p. 11, Figure A3 · p. 12
  • Tuning-strategy trends are fairly homogeneous across studies and classification tasks, with no standout regions, so the conclusions are driven by all studies.Source: Appendix B · p. 11
  • Figure A4 compares predictor error from inner-loop cross-validation with error on a left-out validation set after refitting with the best parameters.Source: Figure A4 · p. 12

Appendix C. Results without variance-normalization · Appendix D. Stability–prediction trends

Without variance normalization, cross-validation is no more reliable than with normalization, while nested parameter tuning matters more for good prediction. Variance normalization can be important, particularly for non-sparse SVM.

Appendix C. Results without variance-normalization

  • Without variance normalization, nested cross-validation is more important for good prediction.Figure A5 presents the impact of tuning strategy on prediction accuracy without feature standardization.Source: Appendix C · p. 12, Figure A5 · p. 12
  • Cross-validation on non-variance-normalized neuroimaging data is not more reliable than on variance-normalized data.The passage refers to figure A4 for this comparison.Source: Appendix C · p. 12
  • Variance normalization makes datasets more comparable, making default parameter values more likely to work well.Source: Appendix C · p. 12
  • Variance normalization can be important, particularly for non-sparse SVM.Source: Appendix C · p. 12

Appendix D.1. Details on stability–prediction results · Appendix D.2. Prediction and stability interactions

Across datasets and tasks, prediction–stability patterns depend strongly on the penalty and prediction difficulty: higher accuracy generally corresponds to more stable weights, with an exception for overly penalized sparse models. The appendices provide task-level data and examine how decoder choices and parameter tuning relate to stability.

Appendix D.1. Details on stability–prediction results

  • Figure 11 summarizes decoding-strategy effects on the prediction–stability tradeoff, and Figure A7 gives the underlying task-specific data points.A7 plots weight stability relative to the split average against the change in prediction accuracy.Source: Appendix D.1 · p. 12, Figure A7

Appendix D.2. Prediction and stability interactions

  • Figure A8 plots weight stability against validation-set prediction accuracy by dataset, task, decoder, and parameter-tuning strategy, with LOWESS indicating the trend.Weight stability is the correlation across validation splits.Source: Figure A8 · p. 13, Appendix D.2 · p. 13
  • Dataset choice is the main source of prediction-accuracy variation because prediction-task difficulty differs across datasets.Some classification tasks are easier or more stable than others.Source: Appendix D.2 · p. 13
  • Higher prediction accuracy generally corresponds to more stable weights, except for overly penalized sparse models (C=1000).The relationship is reported across validation splits.Source: Appendix D.2 · p. 13
  • Logistic regression and SVM show overall the same behavior, while the penalty is the most important decoding choice.Source: Appendix D.2 · p. 13
  • For non-sparse models, averaging models after cross-validation is particularly beneficial when prediction is good.Source: Appendix D.2 · p. 13

Appendix E. Details on datasets used · Appendix F. Details on preprocessing

The dataset appendix identifies the studies and prediction tasks used in the experiments and describes the Haxby stimulus-pair selection.

Appendix E. Details on datasets used

  • In Haxby, the experiments used visual-stimulus pairs of varying difficulty and excluded pairs with unsuccessful decoding, such as scissors versus bottle.Table A1 lists the studies and their specific prediction tasks.Source: Appendix E · p. 13

Appendix F.1. fMRI data

The fMRI appendix describes intra-subject prediction using the Haxby dataset and inter-subject prediction using openfMRI datasets, with distinct preprocessing procedures for each.

Intra-subject prediction

  • Intra-subject prediction used the Haxby dataset from PyMVPA, with no additional preprocessing except 6 mm FWHM Gaussian smoothing.The original paper reported no spatial smoothing; the appendix applied isotropic Gaussian smoothing.Source: Appendix F.1 · p. 13

Inter-subject prediction

  • Inter-subject prediction used openfMRI datasets, standard SPM8 preprocessing, and session-level GLMs designed according to each study.Preprocessing included slice-time and motion correction, coregistration, and normalization; Nipype orchestrated the pipeline, with Nipy used for GLMs.Source: Appendix F.1 · p. 13

Appendix F.2. Structural MR data · Appendix F.3. MEG data

The appendix specifies how structural-MRI and MEG datasets were prepared and analyzed. Structural MRI used modulated grey-matter maps, while MEG analyses used cleaned single-trial data from a working-memory experiment.

Appendix F.2. Structural MR data

  • Structural-MRI inputs were modulated grey-matter intensity maps derived by VBM on the Oasis dataset using SPM8 segmentation and DARTEL deformation fields.The maps were available through Nilearn’s Oasis VBM dataset-fetching function.Source: Appendix F.2 · pp. 13, 15

Appendix F.3. MEG data

  • MEG analyses used data from 52 subjects and two runs, with each run serving as the other’s validation set.The data came from the HCP N-back working-memory experiment.Source: Appendix F.3 · p. 15
  • Four two-class analyses per subject tested image content (faces versus tools) or working-memory role (target versus distractors).The distractor class combined low- and high-level distractors.Source: Appendix F.3 · p. 15
  • MEG features were sensor fields within temporal windows spanning 50 ms before to 300 ms after event onset.The window choice emphasized visual components.Source: Appendix F.3 · p. 15
  • MEG preprocessing used cleaned single-trial HCP tmegpreproc outputs, bandpass filtering at 1.3–150 Hz, and notch filtering around 60 and 120 Hz.The recordings came from a whole-head MAGNES 3600 magnetometer system in a magnetically shielded room; reference sensors were used to account for environmental fields.Source: Appendix F.3 · p. 15
  • The HCP pipeline detected remaining artifacts and FastICA components associated with cardiac and ocular activity were identified and projected out.The semi-automatic component identification was assisted by ECG and EOG recordings; datasets had fewer than 248 sensors depending on bad-channel classification.Source: Appendix F.3 · p. 15
  • MEG data were accessed using MNE-Python and its MNE-HCP library.The passage directs readers to Larson-Prior et al. and the HCP reference manual for pipeline details.Source: Appendix F.3 · p. 15

Appendix G. Performance on each classification task

Across classification tasks, decoder-accuracy differences were mostly consistent across datasets, with a few exceptions. Large C is a safe untuned choice for non-sparse models, while sparse-model tuning strategies show no clear overall win.

Appendix G. Performance on each classification task

  • Accuracy variations across decoding strategies were mostly consistent across datasets, but SVM ℓ1 with C = 1000 decreased strongly on some datasets.Figure A9 presents prediction accuracy for each decoding strategy on all datasets; its results are differential effects removing the contribution of the dataset.Source: Appendix G · p. 15, Figure A9 · p. 15
  • ℓ1 models tended to perform better on Haxby, the source of intra-subject classification tasks.The authors suggest sparse maps may be less robust to inter-subject variability as a possible explanation.Source: Appendix G · p. 15
  • For non-sparse models, using a large C without parameter tuning is an overall safe choice; for sparse models, averaging, refitting, and C = 1 offer no clear win, though averaging is less variable.These conclusions apply to both intra- and inter-subject settings.Source: Appendix G · p. 15

Related papers

Loading 1606.05201v2…