Source-linked AI summary
Classical Statistics and Statistical Learning in Imaging Neuroscience
Danilo Bzdok
TL;DR
The paper examines how statistical-learning practices and classical inference can support different conclusions about neuroimaging effects and generalization. It contrasts out-of-sample performance with p-value-based inference and highlights constraints from data reuse and preprocessing.
Problem
Statistically significant effects do not necessarily generalize to future data, while successful out-of-sample generalization does not by itself establish classical inference.
Method
The paper contrasts classical inference with statistical-learning procedures including cross-validation, recursive feature extraction, and nested dimensionality reduction.
Results
Statistical significance and successful out-of-sample generalization can support different conclusions about an observed effect.
Takeaways & Limitations
Interpreting neuroimaging findings requires distinguishing p-value-based association from evidence that a model generalizes to future samples.
Takeaways & Limitations
Using information related to the hypothesis can loosen and ultimately invalidate VC-dimension error bounds, while naive classical inference after selection can bias parameter estimates.
Abstract
from arXiv · showhide
Neuroimaging research has predominantly drawn conclusions based on classical statistics, including null-hypothesis testing, t-tests, and ANOVA. Throughout recent years, statistical learning methods enjoy increasing popularity, including cross-validation, pattern classification, and sparsity-inducing regression. These two methodological families used for neuroimaging data analysis can be viewed as two extremes of a continuum. Yet, they originated from different historical contexts, build on different theories, rest on different assumptions, evaluate different outcome metrics, and permit different conclusions. This paper portrays commonalities and differences between classical statistics and statistical learning with their relation to neuroimaging research. The conceptual implications are illustrated in three common analysis scenarios. It is thus tried to resolve possible confusion between classical hypothesis testing and data-guided model estimation by discussing their ramifications for the neuroimaging access to neurobiology.
Main Text
Scientific work benefits from remaining open to a wide variety of tools.
- Using a wide variety of tools is presented as part of being a scientist.
1. Introduction
The paper examines classical statistics and statistical learning as distinct but related approaches whose differing foundations shape neuroimaging inference. It aims to disentangle their forms of inference and clarify their implications for interpreting neurobiological data.
- The paper juxtaposes classical inference and generalization inference across historical trajectories, modelling philosophies, conceptual frameworks, and performance metrics.
- Growing neuroimaging datasets now involve greater scanning resolution, sample size, complexity, multimodality, and repository-based reanalysis.
- Neuroimaging has expanded toward statistical-learning methods that use flexible, data-driven models and scale naturally to high-dimensional data.
- Classical statistics and statistical learning can be understood as different categories of statistical inference, including null-hypothesis falsification and out-of-sample generalization.
2. Different histories
Classical statistics and statistical learning developed through partly independent histories, vocabularies, and institutional communities. Changes in data dimensionality and computing helped statistical learning become increasingly prominent in modern neuroimaging.
- The two communities use different basic terminology, including independent or predictor variables versus features, and dependent or response variables versus target variables.
- Classical statistics developed around experimental design, null-hypothesis testing, p values, t-tests, ANOVA, and competing Fisher and Neyman–Pearson frameworks.
- Statistical learning grew from computing, artificial intelligence, pattern recognition, and exploratory data analysis, with methods including neural networks, kernel methods, trees, and graphical models.
- Increasingly wide data, greater computing power, and cheaper memory encouraged renewed statistical-learning research and applications from approximately 2000 onward.
- Sparsity assumptions gained importance because dense high-dimensional models that treat all features as relevant are difficult for existing methods.
- Modern deep neural networks improved automated feature selection and exceeded human-level performance in several tasks, including a 49-game pixel-input demonstration.
3. Different philosophies
Classical statistics and statistical learning represent different, partly overlapping positions on a continuum, differing in assumptions, modelling direction, outputs, and evaluation goals. In neuroimaging, these distinctions shape whether analyses address explanatory relationships, predictive generalization, or mappings between brain activity and indicators.
- Classical statistics emphasizes confirmatory, explanatory modelling and significance, whereas statistical learning emphasizes exploratory, predictive modelling and generalization to unseen data.The two approaches therefore evaluate different outcomes: quantified relationships under a pre-specified model versus robustness or predictiveness across new observations.
- Classical statistics often targets small samples and few investigator-chosen parameters, while statistical learning targets many variables and uses data-created functions, sometimes with implicit assumptions.Statistical learning may favor simple tractable models despite consciously false assumptions because sufficiently large data quantities are expected to remedy them.
- Classical statistics typically pre-specifies and tests an interpretable model, whereas statistical learning estimates a model from data with fewer explicit assumptions.Classical statistics treats the data-generating process as partly known; statistical learning treats the phenomenon as complex, largely unknown, and partly unknowable.
- Neither classical statistics nor statistical learning is generally superior, so investigators should choose the approach suited to the neurobiological phenomenon and research object.The paper presents the two families as methodological poles that are useful for comparison but not as universally ranked alternatives.
- In neuroimaging, encoding and decoding differ in inference direction between brain space and indicator space, while information-pattern analyses extend questions beyond increases or decreases in activity.The paper frames information-pattern inference as modelling from neural-activity features to indicators and identifies questions such as where an information category is neurally processed.
4. Different theories
Classical statistics and statistical learning arise from distinct theoretical frameworks: hypothesis testing emphasizes falsification, whereas learning emphasizes model complexity, generalization, and prediction. In neuroimaging, both confront high-dimensional data, model-selection risks, and related forms of biased inference.
- Classical statistics: Classical hypothesis testing frames mutually exclusive hypotheses and uses significance thresholds to evaluate evidence against the null, not to establish the alternative directly.A non-rejected null yields no conclusive result, and substantive theories require more than statistical testing.
- Statistical learning: Statistical learning balances model bias and variance, with increasing complexity generally reducing bias while increasing variance and overfitting risk.Simple models generalize better but approximate poorly; complex models approximate better but may fit noise and generalize less well.
- Statistical learning: Cross-validation and VC dimensions frame learning performance through generalization error, while neuroimaging-specific dimensionality knowledge can improve the bias-variance balance.The passages connect effective dimensionality reduction with reduced model bias and variance and improved generalization.
- Neuroimaging implications: In neuroimaging, multiple-comparisons problems are associated more closely with encoding studies, whereas overfitting is associated more closely with decoding studies.Restricting analysis to a region of interest can simultaneously reduce multiple comparisons and overfitting, corresponding to small-volume correction and feature selection.
- Neuroimaging implications: Double dipping and data snooping share information leakage, biased estimation, and subtle failure modes across classical and learning-based analyses.Both compress high-dimensional data, introduce unauthorized prior information, and can invalidate inferential guarantees.
5. Different currencies
Classical statistics and statistical learning use different currencies for judging models: inferential quantities such as p values, effect sizes, confidence intervals, and power versus out-of-sample predictive performance. These measures support different interpretations of evidence, uncertainty, and model usefulness.
- Classical statistics: P values do not indicate replication probability and tend to become smaller with increasing sample size, so effect size helps assess practical relevance.Effect size can identify marginal effects that meet significance thresholds without being practically important.
- Classical statistics: Classical statistics reports inferential and descriptive quantities such as p values, effect sizes, confidence intervals, and statistical power.P values quantify extremeness under a null, effect sizes describe relationship strength, confidence intervals express ranges, and power concerns long-run detection of true effects.
- Statistical learning: Statistical learning uses cross-validation to estimate generalization beyond the observed sample by training on one subset and evaluating predictions on held-out data.Predictions can be summarized with accuracy, precision, recall, and F1 score, including out-of-sample performance.
- Comparison: The two frameworks therefore evaluate different targets: classical analysis emphasizes population inference, whereas learning emphasizes predictive performance on unseen data.The passages distinguish in-sample from out-of-sample quantities and contrast classical point or interval estimates with model assessment.
- Neuroimaging implications: Neuroimaging applications may benefit from evaluating whether learned feature-label mappings have neurobiological plausibility, not only predictive accuracy.Reverse-engineering fitted models is proposed as an evaluation metric for explaining why a model achieves its performance.
6. Case study one: Generalization and subsequent classical inference
The case study contrasts Lasso-based predictive voxel selection with classical voxel-wise inference. It shows that data-driven selection changes the inferential problem, making naive post-selection p values and confidence intervals invalid without selection adjustment.
- Predictive model: Five-fold cross-validation identifies a small set of non-zero grey-matter voxel coefficients whose volumetric information best predicts individual age.The Lasso solves automatic variable selection in a high-dimensional pattern-learning setting and assesses sparse models through generalization performance.
- Classical inference: Voxel-wise p values would recast the predictive model as mass-univariate testing and require multiple-comparison correction while ignoring the preceding selection process.That recasting treats voxel relevance independently even though the predictive model selected a reduced set jointly.
- Classical inference: Data-driven selection alters parameter-estimate distributions, so naive classical inference assumes a non-adaptive model and cannot generally be used with Lasso or arbitrary selection procedures.The resulting conflict reflects the difference between cross-validated model selection and confirmatory inference.
- Post-selection inference: The conflict between predictive selection and confirmatory inference is an active statistical-development frontier rather than a settled equivalence between the two approaches.The paper identifies post-selection inference and conceptual extensions of classical theory as responses to adaptive testing.
- Post-selection inference: Selective inference replaces naive p values with selection-adjusted p values for features chosen as meaningful by an adaptive procedure.Data splitting is an alternative, but it may be unavailable and can reduce power or complicate interpretation.
7. Case study two: Classical inference and subsequent generalization
This case study contrasts feature selection for predictive learning with the stricter requirements of classical hypothesis testing. Training-set-only preprocessing preserves cross-validation validity, whereas using the same selected variables for null-hypothesis testing creates circular analysis and can weaken interpretation.
- Pipeline: The example reduces more than 100,000 voxels to 10,000 ANOVA-ranked features before applying a support vector machine to predict gender from brain scans.The selected features are ranked by variance explaining male–female differences and then used for classification.
- Validity of predictive estimation: Cross-validation remains valid when supervised feature selection is performed separately within each training fold and the resulting feature space is applied identically to training and test data.The estimator evaluates the entire prediction process, including nested dimensionality reduction, without using concealed test-set properties.
- Validity of classical inference: Using class labels to select variables immediately before null-hypothesis testing on the same sample constitutes double-dipping and invalidates the null hypothesis.This distinction reflects the contrast between out-of-sample cross-validation estimates and in-sample classical inference.
- Interpretation: Univariate selection can miss voxel ensembles whose information emerges jointly, while encouraging classifier patterns that are difficult to interpret neurobiologically.The resulting model may learn complex relationships among individually selected voxels, weakening claims about whole-brain multivariate patterns.
- Prediction versus interpretation: Whole-brain univariate selection followed by linear classification is often effective for prediction, but statistical association and generalization performance need not coincide.The approach is described as favorable when optimized prediction matters more than optimized interpretability.
8. Case study three: Structure discovery by clustering algorithms
This case study treats connectivity-based brain parcellation as an unsupervised structure-discovery problem rather than a straightforward classical significance test. Clustering can suggest candidate modules and hypotheses, but their neurobiological existence requires further scrutiny.
- Structure discovery: Connectivity-based parcellation groups ROI voxels by similarities in their whole-brain connectivity fingerprints to identify cortical modules with potentially similar connectivity patterns.The clustering elements are voxels, and connectivity strengths provide the features used for similarity assessment.
- Classical inference: Classical significance testing is difficult because parcellation requires a meaningful null hypothesis, while cluster shape and the optimal number of clusters remain unresolved.Heuristic validity criteria instead favor solutions with low within-cluster and high between-cluster differences.
- Validation: Clustering algorithms can always find subregions according to their optimization objective, whether or not those structures exist in nature.This makes heuristic cluster-validity measures necessary but insufficient for establishing biological structure.
- Interpretation: The discovered connectivity clusters are hints to candidate brain modules whose existence in neurobiology requires further scrutiny.They should therefore be interpreted as exploratory outputs rather than formally validated biological entities.
- Scientific use: Brain parcellation is an exploratory unsupervised method that can narrow high-dimensional structure and broaden hypotheses for subsequent formal testing.Unexpected candidate regions can motivate future experimental investigations.
9. Conclusion
The conclusion argues that classical significance and predictive generalization answer different inferential questions and therefore support different conclusions. Imaging neuroscience should move between exploratory learning and deductive testing while recognizing the scope and limits of each framework.
- Different inferential targets: A statistically significant effect need not generalize to future samples, and an effect that generalizes successfully need not be statistically significant.The distributions supporting statistical significance and successful generalization are not identical.
- Different inferential targets: Classical inference judges an entire data sample, whereas predictive inference can be applied to individual datapoints.The adopted framework conditions both the goal of inference and the conclusions that are permissible.
- Integrated workflow: The paper proposes a routine back-and-forth between inductive learning algorithms and deductive hypothesis testing of candidate structures.This alternation is presented as a way to investigate discovered phenomena while respecting each framework’s inferential scope.
- Implications: Awareness of the divide between statistical cultures matters as neuroimaging repositories provide increasingly granular information.The conclusion connects this awareness to the changing scale and information content of imaging data.
- Conceptual synthesis: Statistical inference is heterogeneous, spanning classical hypothesis falsification and generalization inference based on extrapolating complex patterns.The paper frames these approaches as distinct points within a broader continuum rather than interchangeable procedures.