Classical Statistics and Statistical Learning in Imaging Neuroscience

• Scientists should be open to using a wide variety of tools.Source: Main Text · p. 3

1. Introduction

• The paper aims to disentangle classical inference and generalization inference by comparing their historical trajectories, modeling philosophies, conceptual frameworks, and performance metrics.Source: §1 · p. 3
• Neuroimaging analysis has expanded from null-hypothesis testing toward statistical learning, with flexible, data-driven models that scale to high-dimensional data and rely more on numerical optimization.Source: §1 · p. 4
• The introduction treats inference as encompassing both null-hypothesis falsification and out-of-sample generalization, while noting that classical statistics and statistical learning may yield different inference categories.The text cites Jacob Cohen (1990) and G. Gigerenzer & Murray (1987) in connection with this broader conception of inference.Source: §1 · p. 4

2.1 The origins of classical hypothesis testing and learning algorithms

• Around 1900, standard deviation, goodness of fit, and the “p < 0.05” threshold emerged; Fisher developed ANOVA and null-hypothesis testing (R. A. Fisher, 1925; Ronald A. Fisher, 1935; Ronald A. Fisher & Mackenzie, 1923), while Neyman and Pearson proposed an alternative framework introducing power.Source: §2.1 · p. 6
• Kernel methods, neural networks, decision trees, nearest neighbors, and graphical models originated in the computer science community and mostly continued to develop in the statistical learning community.Source: §2.1 · p. 7
• Changing data from n > p to n < p, increased computing power, and cheaper memory helped revive statistical learning research and applications around 2000 (Tibshirani, 1996; Efron & Tibshirani, 1991).Source: §2.1 · p. 8
• With n << p, sparsity assumptions treated only subsets of features as relevant and enabled reproducible, interpretable statistical relationships in high-dimensional data (Trevor Hastie, Tibshirani, & Wainwright, 2015; Bühlmann & Van De Geer, 2011).Source: §2.1 · p. 8
• Deep neural networks improved automated feature selection and exceeded human-level performance in several tasks; one algorithm mastered 49 games from pixel input alone (Bengio, Courville, & Vincent, 2013; LeCun et al., 2015; Mnih et al., 2015).Source: §2.1 · p. 8

2.2 Related spotlights in the history of neuroimaging analysis methods

• Neuroimaging commonly used mass-univariate GLM analyses across voxels, and the resulting multiple-comparisons problem prompted methods including false discovery rate (Friston et al., 1994; Genovese, Lazar, & Nichols, 2002).Source: §2.2 · pp. 9–10
• Because p > n made GLM solutions non-unique, sparsity-inducing Lasso and ElasticNet later offered a principled way to handle high-dimensional neuroimaging analyses (Tibshirani, 1996; Trevor Hastie et al., 2015; Jenatton, Audibert, & Bach, 2011).Source: §2.2 · p. 9
• Random-field inference addressed multiple comparisons by modeling spatially neighboring activations during inference, rather than incorporating voxel dependencies into GLM estimation.The approach treated discrete voxel activations as topological units with continuous height and extent.Source: §2.2 · p. 10
• Early multivariate work used principal component analysis to study neural activity and Alzheimer’s disease, while canonical correlation analysis linked task-free activity with schizophrenia symptoms (Friston, Liddle, Frith, Hirsch, & Frackowiak, 1992).These early brain-behavior approaches did not spark a major research trend.Source: §2.2 · p. 10
• Searchlight analysis enabled whole-brain assessment of local neighborhoods with predictive activity patterns, while the statistical position of decoding models was seldom made explicit (N. Kriegeskorte, Goebel, & Bandettini, 2006; Friston et al., 2008).Prediction was increasingly understood as out-of-sample generalization, unlike in-sample linear correlation.Source: §2.2 · pp. 10–11
• As neuroimaging datasets grow in dimensionality and complexity, statistical learning may become more useful, but larger data can reveal subtle effects while making false positives a major concern (Jordan et al., 2013; Meehl, 1967).Structured sparsity penalties can incorporate local spatial or spatial-temporal dependence as neurobiological priors (Gramfort, Thirion, & Varoquaux, 2013; Michel, Gramfort, Varoquaux, Eger, & Thirion, 2011; Gramfort, Strohmeier, Haueisen, Hamalainen, & Kowalski, 2011).Source: §2.2 · p. 11
• Deep neural networks have entered neuroimaging despite incompletely understood mathematical properties and missing formal guarantees, and may help approximate neural processing (de Brebisson & Montana, 2015; Güçlü & van Gerven, 2015; Plis et al., 2014; D. D. Cox & Dean, 2014; Yamins & DiCarlo, 2016).Their application is described as atheoretical and more empirically justified.Source: §2.2 · p. 11

3.1 Two different modelling goals in statistical methods

• CS treats data-generating mechanisms as partly known and states model assumptions explicitly, whereas SL treats phenomena as complex or unknowable and learns models from data.CS tends toward analytical rigor, while SL seeks useful approximations with fewer explicit assumptions.Source: §3.1 · pp. 13–15
• CS typically yields point or interval estimates, such as p values and confidence intervals, whereas SL often yields functions such as centroids or classifiers applicable to new data.SL approaches often rely on iterative numerical optimization, while many CS methods use closed-form computations.Source: §3.1 · p. 14
• CS is more often associated with confirmatory analysis of controlled experimental data, whereas SL is more often associated with exploratory analysis of observational data.The former commonly concerns structured perturbations; the latter commonly studies systems left unperturbed.Source: §3.1 · p. 14
• CS mainly explains preselected constructs retrospectively and evaluates significance; SL predicts future observations and evaluates generalizability, while explanatory power need not imply predictive power.The approaches emphasize different outcomes and do not make the same kind of claim.Source: §3.1 · pp. 14–15
• The contrast is a didactic simplification: CS was largely fashioned for small samples and investigator-chosen models, while SL targets many variables and data-derived functions.The passage frames these as typical problem settings rather than strict categories.Source: §3.1 · p. 15
• The Bayesian–frequentist distinction is orthogonal to CS versus SL, and Bayesian approaches can belong to either family.The section cautions that the CS–SL polarization is a didactic simplification.Source: §3.1 · pp. 15–16
• Parametric approaches can be more successful with few observations, whereas non-parametric approaches can capture patterns emerging from larger datasets.Non-parametric classifiers can form increasingly complex decision boundaries as training data grow.Source: §3.1 · p. 17
• Neither classical statistics nor statistical learning is generally superior; investigators choose an approach suited to the neurobiological phenomenon and research object.This reflects the no-free-lunch principle: no single statistical strategy performs best in all circumstances.Source: §3.1 · p. 17

3.2 Different modelling goals in neuroimaging

• Encoding models fit neural activity from stimulus, task, or behavioral indicators, whereas decoding models predict indicators from neural data.Encoding fits a general linear model; decoding trains and tests classifiers on separate data splits.Source: §3.2 · pp. 17–18
• Encoding treats indicators as causes and asks whether activity follows them (forward inference), while decoding treats indicators as consequences and asks what process activity implies (reverse inference).The distinction concerns the direction of inference between brain space and indicator space.Source: §3.2 · p. 18
• Spatial smoothing in classical analyses aids multiple-comparison correction, reduces anatomical variability, and increases sensitivity, but discards fine-grained activity patterns.Those fine-grained patterns may carry information about mental operations.Source: §3.2 · pp. 19–20
• Decoding pursues an informational agenda by demonstrating generalization to new acquisitions; inference on brain information patterns reduces to model comparison.Source: §3.2 · p. 20
• Multivariate decoding can harvest spatially distributed information, and some brain-behavior associations emerge across voxel groups but disappear in single-voxel GLMs.Source: §3.2 · p. 20
• Learning-based analyses ask where information is processed and whether a category is reflected in neural activity, including distributed patterns and topographically similar but neurally distinct processes.This extends interpretation beyond increases or decreases in activity.Source: §3.2 · pp. 20–21
• Classifier analyses of neural time series ask when information is generated, processed, and bound, and can characterize computational properties such as regional versus network processing.The temporal scope includes the beginning, evolution, and end of cognitive facets; computational questions also include isolated versus partially shared processing.Source: §3.2 · p. 21
• The properties of a chosen learning algorithm, such as a linear versus non-linear support vector machine, are not convincing evidence for reverse-engineering neural processing mechanisms.The passage presents this as an important caveat.Source: §3.2 · p. 21

4.1 Diverging ways to formalize statistical inference

• A domain-informed alternative should contradict the current state of the art, and data with probability <=5% under the null hypothesis are deemed significant.The test is framed as an attempt to disprove the null hypothesis in favor of the alternative.Source: §4.1 · p. 22
• Failure to reject the null yields no conclusive result, not evidence for a null result.Statistical significance is often misconstrued as the probability of the null or alternative hypothesis, the result, or the null given the result.Source: §4.1 · p. 22
• As model complexity rises, bias tends to fall while variance and overfitting risk rise, trading approximation of the target for generalization to unseen data.Complex models can fit sample-specific noise, whereas simpler models generalize better but approximate the target less well.Source: §4.1 · pp. 23–24
• VC dimensions formalize model capacity and generalization, but are explicitly computable only for simple models, often leaving investigators with approximate bounds.This limits their usefulness for theoretical considerations.Source: §4.1 · p. 25
• For models with finite VC dimensions, sufficient data can yield performance approximating expected performance on unseen data; infinite VC dimensions preclude generalization conclusions.VC dimensions formalize a model's capacity to learn distinctions and predict on new data.Source: §4.1 · p. 25

4.2 The impact of diverging inference categories on neuroimaging analyses

• In neuroimaging, classical-statistical analyses center on the multiple-comparisons problem, whereas statistical-learning analyses foreground dimensionality and overfitting.The passage identifies the curse of dimensionality as a concern from the statistical-learning stance.Source: §4.2 · p. 25
• Model choice can encode assumptions about spatial structure: discrete local compartments suit brain regions, whereas overlapping continuous compartments suit brain networks.The paper gives k-means or spatially constrained Ward clustering for the former and independent component analysis or sparse principal component analysis for the latter.Source: §4.2 · p. 26
• Without complexity restrictions, high-dimensional models struggle to generalize, while exploiting effective dimensions can reduce bias and variance and improve generalization.The paper links multiple comparisons more closely to encoding and overfitting to decoding studies; restricting analysis to a region of interest can alleviate both problems.Source: §4.2 · pp. 26–27
• Classical double dipping occurs when the same data guide voxel selection and a subsequent test, making the tests non-independent and corrupting the null hypothesis (N. Kriegeskorte et al., 2009).The described example selects voxels using a behavioral correlation, then tests those same voxels against that measure again (Lieberman, Berkman, & Wager, 2009; Vul et al., 2008).Source: §4.2 · p. 27
• Iterative feature pruning, such as recursive feature extraction, is valid in statistical learning, while preprocessing the entire dataset instead of separately on training and test sets can produce optimistic cross-validation estimates and ultimately invalidate VC-dimension error bounds.Source: §4.2 · pp. 27–28
• Both errors involve compressing high-dimensional data, introducing unauthorized prior information, and biasing estimates, potentially producing unjustified confidence in findings (Tal Yarkoni & Westfall, 2016).The paper notes that these errors can arise subtly and may be avoidable with relatively small effort.Source: §4.2 · p. 28
• The authors present cross-validation with k = 5 or 10 splits as probably optimal for neuroimaging: more splits can increase estimate variance, whereas fewer splits can increase estimate bias (Leo Breiman & Spector, 1992; Kohavi, 1995).Source: §4.2 · p. 28
• Leave-one-out validation may be justified when samples are particularly few and fit is the priority, but its computational load is often much higher than 5- or 10-fold validation.The paper also notes that leave-one-out does not always select the best model, even when that model is known.Source: §4.2 · p. 29

5.1 Diverging performance metrics can quantify the behavior of statistical models

• P values are deductive inferential measures, whereas effect sizes are descriptive; effect sizes can reveal statistically significant effects that are not practically relevant.P values do not establish replication and can decrease as sample size increases.Source: §5.1 · pp. 30–31
• Confidence intervals express uncertainty around point estimates, while statistical power measures a test’s probability of detecting a true effect and is meaningful within formal hypothesis falsification.Power depends on the significance threshold, population effect size and variation, sample size, and experimental design.Source: §5.1 · pp. 31–32
• Cross-validation estimates a model’s ability to generalize by repeatedly training on larger data subsets and testing predictions on held-out data assumed to share the same distribution.Predicted and true labels can then be compared using accuracy, precision, recall, and F1 score.Source: §5.1 · p. 32
• Accuracy, precision, and recall capture different aspects of classification, while Fβ scores weight precision and recall; no single measure is optimal in every context.Precision and recall can ignore true negatives and be susceptible to estimator bias.Source: §5.1 · p. 33
• Learning curves evaluate model fit and prediction as sample size grows, illustrating how simple models may underfit while generalizing well and complex models may behave differently.The supplied passage begins its description of complex models but does not provide the comparison’s conclusion.Source: §5.1 · p. 33
• With scarcer training data, model effectiveness is more challenged; overfitting can yield low in-sample error but poor extrapolation and high out-of-sample error.Source: §5.1 · pp. 33–34

5.2 Outcome metrics of statistical models used in neuroimaging

• In-sample effect sizes are typically optimistic, whereas out-of-sample effect sizes are unbiased estimates of the true effect; reporting both from identical data risks double dipping.Source: §5.2 · p. 34
• In wide neuroimaging data, judging significance by p values is challenging, while accuracy alone is affected by hemodynamic responses, design efficiency, data folds, and feature number.Source: §5.2 · pp. 34–35
• Bootstrapping can support model-complexity-independent population inference, confidence intervals for estimates, and non-parametric null-hypothesis testing.Source: §5.2 · p. 35
• For multiclass prediction, accuracy can be ambiguous, so class-specific F1, precision, and recall provide important complementary measures.Source: §5.2 · p. 35
• Beyond predictive scores, learning models should be evaluated for neurobiological plausibility, including whether fitted mappings reflect task-related features and can reconstruct stimulus or task aspects.Source: §5.2 · pp. 36–37

• In an age-prediction example, five-fold cross-validation of a Lasso selects a few grey-matter voxels whose volumes best predict age.Source: §6 · p. 38
• Voxel-wise testing would treat voxels independently and require multiple-comparison correction, while ignoring Lasso's stochastic variable-selection process.Source: §6 · pp. 38–39
• Naive classical tests and confidence intervals after data-guided selection are invalid: selection alters parameter-estimate sampling distributions and biases p-values downward.Source: §6 · p. 39
• Selective inference computes selection-adjusted p-values for chosen predictors, requiring stronger predictor-target associations for equal significance and prompting extensions to classical theory (Trevor Hastie et al., 2015).Source: §6 · p. 39
• Data splitting performs selection on one split and p-value computation on another, but may be infeasible, reduce power, and complicate interpretation.Source: §6 · p. 40

7. Case study two: Classical inference and subsequent generalization

• Brain scans’ >100,000 voxels are reduced to 10,000 most relevant grey-matter features using a univariate significance test with male–female labels; support vector machine classification predicts gender from each scan.Source: §7 · p. 40
• Computing feature selection within each training fold keeps test-set feature construction independent and does not invalidate cross-validation, even when class labels guide selection.Cross-validation evaluates the entire prediction process, including nested dimensionality reduction.Source: §7 · p. 41
• Using class labels to select variables before null-hypothesis testing on the same sample invalidates the null and constitutes double-dipping.This contrast reflects cross-validation's out-of-sample estimates versus hypothesis testing's in-sample estimates.Source: §7 · p. 41
• Univariate filtering can miss voxels informative only in ensembles and promote classifier patterns that are not neurobiologically meaningful, weakening whole-brain interpretation.The filter retains independently relevant voxels, while the classifier learns complex patterns among those selected features.Source: §7 · pp. 41–42
• Statistical association does not necessarily imply good generalization, and vice versa; univariate selection with linear classification is often favored for prediction over interpretability.Source: §7 · p. 42
• Selecting fewer than 500 voxel variables from 500 subjects yields n > p data, enabling a general linear model and classical testing instead of predictive classification (Brodersen, Haiss, et al., 2011).Source: §7 · p. 42

8. Case study three: Structure discovery by clustering algorithms

• ROI voxels are clustered using their whole-brain connectivity fingerprints, grouping voxels with similar connectivity patterns that may be functionally distinct.The voxels are the elements and connection strengths are their features for assessing similarity.Source: §8 · pp. 42–43
• Classical significance testing is difficult because clustering lacks a meaningful null hypothesis and may require an arbitrary hypothesis to falsify (Vogelstein et al., 2014; Jain, Murty, & Flynn, 1999).Classical effect size and power notions do not seem to apply to brain parcellation.Source: §8 · p. 43
• The true cluster shape is usually unknown, and determining an optimal number of clusters remains unresolved in brain imaging and statistics (Handl, Knowles, & Kell, 2005; Jain et al., 1999).Unsupervised estimation is generally recognized as difficult (Bishop, 2006; Zoubin Ghahramani, 2004; T. Hastie et al., 2001).Source: §8 · pp. 43–44
• Heuristic cluster-validity criteria assess fit, but algorithms always find subregions according to their optimization objective, whether or not those structures exist in nature.Criteria commonly favor low within-cluster and high between-cluster differences.Source: §8 · p. 44
• Connectivity clusters are candidate brain modules whose neurobiological existence requires further scrutiny (S. B. Eickhoff et al., 2015; Thirion et al., 2014).Clustering can produce relevant-looking subregions whether or not they truly exist in nature.Source: §8 · p. 44
• Clustering narrows high-dimensional data and broadens hypotheses, enabling exploratory findings to guide future investigations and later formal testing.Brain parcellation outlines relevant data structure as an exploratory unsupervised method, not a substitute for testing a concrete hypothesis.Source: §8 · p. 44

9. Conclusion

• The two statistical cultures have different historical trajectories and theoretical foundations, yet both use mathematical models to extract knowledge from data; this gap may contribute to confusion about their relationship.Source: §9 · p. 45
• A statistically significant effect need not generalize to future samples, and successful out-of-sample generalization need not be statistically significant because the relevant effect distributions differ.Source: §9 · p. 45
• Classical inference judges an entire sample, whereas predictive inference can apply to individual datapoints, so the adopted framework conditions the goals and permissible conclusions.Source: §9 · p. 45
• Imaging neuroscience should routinely pair inductive learning to discover candidate structures with deductive hypothesis testing of those candidates.The paper points to drug development, where predictive inference is used in early discovery and classical inference in clinical trials, as an example of such back-and-forth.Source: §9 · p. 45