Source-linked AI summary
Criteria for Classifying Forecasting Methods
Tim Januschowski, Jan Gasthaus, Yuyang Wang, David Salinas, Valentin Flunkert, Michael Bohlke-Schneider, Laurent Callot
TL;DR
Forecasting methods are often labeled “statistical” or “ML,” although the paper argues these categories do not reflect fundamental methodological differences. It proposes alternative forecasting dimensions, highlights overlaps and complementarities, and concludes that cross-pollination between communities is promising.
Problem
The ML-versus-statistics distinction can mislead judgments about forecasting methods because it implies a deeper difference than the methods actually have.
Method
The paper examines forecasting methods through alternative objective and subjective dimensions, including uncertainty, computational cost, and data-driven versus model-driven approaches.
Results
The paper concludes that ML and statistical labels have no added value for classifying forecasting methods and identifies interaction between data-driven and model-driven methods as a promising direction.
Takeaways & Limitations
Meaningful forecasting conclusions should use method characteristics and restrict empirical claims to the datasets and forecasting tasks evaluated.
Takeaways & Limitations
Empirical conclusions should remain bounded by the domains and forecasting tasks represented in the benchmark data.
Abstract
from arXiv · showhide
Classifying forecasting methods as being either of a "machine learning" or "statistical" nature has become commonplace in parts of the forecasting literature and community, as exemplified by the M4 competition and the conclusion drawn by the organizers. We argue that this distinction does not stem from fundamental differences in the methods assigned to either class. Instead, this distinction is probably of a tribal nature, which limits the insights into the appropriateness and effectiveness of different forecasting methods. We provide alternative characteristics of forecasting methods which, in our view, allow to draw meaningful conclusions. Further, we discuss areas of forecasting which could benefit most from cross-pollination between the ML and the statistics communities.
1. Introduction
The paper argues that labeling forecasting methods as “statistical” or “ML” exaggerates their differences and can mislead practical conclusions. It proposes more meaningful distinctions and encourages common ground between forecasting communities.
- The labels “statistical” and “ML” imply a deeper qualitative distinction than the methods actually exhibit.The authors describe these labels as community-assigned terms rather than intrinsic attributes of methods.
- Using broad labels can encourage sweeping claims about the relative accuracy of ML and statistical methods.
- The paper argues that no meaningful grouping matching the original meanings of “statistical” and “ML” can be established.It instead favors distinctions along other dimensions for practical conclusions.
- The discussion focuses on forecasting and treats cooperation between statistics and ML communities as more constructive than artificial boundaries.The authors emphasize that both communities have much to learn from each other.
2. Connotations of the Terms “Statistics” and “ML”
The paper presents “statistics” and “ML” as inadequate method categories for forecasting. It instead organizes the discussion around alternative dimensions and distinguishes methodological properties from community-oriented perspectives.
- Attempts to draw clear demarcation lines between statistics and ML are described as thought-provoking but scientifically limited.
- In practice, “statistical methods” often denotes exponential smoothing and ARIMA variants, while “ML methods” may denote everything else or neural networks and random forests.
- Breiman distinguishes data modeling focused on in-sample fit from algorithmic modeling focused on out-of-sample fit.
- Forecasting combines formal probabilistic methods with statistical guarantees and algorithmic approaches, evaluating models primarily through out-of-sample accuracy.
- The paper’s proposed dimensions are a non-comprehensive starting point, with objective mathematical properties separated from subjective methodological or cultural dimensions.
3. Objective Dimensions for Classifying Forecasting Methods
The paper proposes objective forecasting dimensions—including parameter sharing, uncertainty representation, computational cost, linearity, and convexity—instead of relying on ML-versus-statistics labels. These dimensions expose overlaps and trade-offs across methods.
- Local and global methods: Local methods estimate parameters independently for each time series, whereas global methods estimate them jointly across available series.Hybrid methods estimate some parameters globally and others locally.
- Local and global methods: The local-global distinction concerns parameter estimation and is complementary to assumptions about dependence or multivariate forecasting.
- Uncertainty quantification: Probabilistic forecasts quantify predictive uncertainty, while point forecasts provide a single best prediction relative to an error metric.
- Uncertainty quantification: Probabilistic methods expose an often-intractable joint predictive distribution through intervals, sample paths, or parametric distribution parameters.
- Uncertainty quantification: M4 results showed that ML-derived prediction intervals can be highly accurate despite lacking a theoretical underpinning.
- Computational complexity and costs: Computational cost should be weighed against predictive accuracy and application constraints, including the costs of expert supervision and tuning.
- Computational complexity and costs: On a large weekly retail dataset, ets required about 30 ms per series for model selection, training, and prediction, while an RNN required about 100 ms after one hour of training.The paper notes that periodic forecasting can amortize training time.
- Linearity and convexity: Linearity does not separate statistical from ML methods: statistical methods can be nonlinear, while ML methods can be linear.Examples include damped or multiplicative exponential smoothing, linear support vector regression, and matrix factorization.
4. Subjective Dimensions For Classifying Forecasting Methods
The paper argues that forecasting methods are better distinguished by methodological dimensions than by the broad labels “ML” and “statistical.” It examines data-driven versus model-driven approaches, ensembles, discriminative versus generative models, empirical versus theoretical validation, and interpretability.
- Subjective dimensions are less well-defined than objective ones but may capture the principal methodological and cultural differences between ML and statistics.
- Data-driven and Model-Driven Methods: Data-driven models learn complex patterns with few structural assumptions but are data-hungry and vulnerable to over-fitting.Model-driven methods use parsimonious parameterizations and can be data-efficient when their assumptions are appropriate, but they are more rigid.
- Data-driven and Model-Driven Methods: Global data-driven models suit large operational forecasting problems, whereas model-driven methods can be efficient for few series when researchers can specify appropriate models.Model-driven approaches often require careful feature engineering and model specification; data-driven models can extract complex patterns with little researcher intervention.
- Single Models and Combination Models: Single-model and combination-model methods form a continuum because model selection, feature selection, hyperparameter optimization, and mixtures can involve multiple fitted models.This dimension separates forecasting-model assessment from combination techniques and supports comparisons of end-to-end learning with staged and ensemble approaches.
- Discriminative and Generative Models: Discriminative and generative forecasting models are not mutually exclusive categories, and no general theory establishes one approach as superior in a given setting.Discriminative models may be more accurate in some settings but can require retraining when the forecast horizon changes.
- Explanatory Modeling and Interpretability: Interpretability is limited because feature engineering and nonlinear preprocessing can make both model inputs and estimated relationships difficult to interpret.Even parsimonious linear models can change parameter interpretations when correlated covariates are added.
5. Opportunities & Challenges
The paper identifies opportunities for stronger forecasting research through better theory, software, empirical rigor, and practical modeling choices. It emphasizes collaboration between ML and statistics communities and restricting empirical conclusions to evaluated datasets and tasks.
- Software Frameworks: Open-source forecasting ecosystems built on deep-learning frameworks could improve reproducibility, model exchange, and adoption.These frameworks combine fast tensor computation, automatic differentiation, optimization, and probabilistic tools.
- Empirical Rigor: ML forecasting research should adopt the empirical rigor of forecasting competitions and compare with state-of-the-art methods and accepted baselines.The paper also notes that competition findings on ensembles and data-driven methods deserve greater attention in academic literature.
- Empirical Rigor: Empirical conclusions should be limited to the datasets and forecasting tasks on which methods are evaluated.The paper contrasts operational settings, where data-driven methods may perform well, with strategic settings, where model-based methods may perform well.
- Practical considerations: Careful training-scheme design and appropriate covariate handling are important practical determinants of forecasting performance.The paper highlights covariate availability and handling, particularly for global models.
- Theory: Neural-network methods are useful across many applications, but their theoretical understanding lags, creating an opportunity for collaboration between ML and statistics.The paper notes that this gap is beginning to close for neural networks and random forests.
6. Conclusion
The paper concludes that classifying forecasting methods as ML or statistical adds little value and can obscure a larger need for interaction between their communities. It points to hybrid research combining probabilistic models, neural networks, ML-enhanced forecasting, and mixed-method pipelines as a promising direction.
- 6. Conclusion: Classifying forecasting methods as ML or statistical adds no value, while the distinction highlights insufficient interaction between communities studying the same problem.The authors encourage researchers to move beyond their disciplinary comfort zones.
- 6. Conclusion: Current research increasingly combines probabilistic models with neural networks and uses hybrid methods spanning ML and statistics.Examples include gradient-boosted trees selecting classical methods and pipelines combining ML and statistical components.
- 6. Conclusion: The paper expects substantial progress from hybrid forecasting work, including ML approaches for improving multi-step-ahead forecasts.It also identifies mixed ML-statistical model pipelines as part of this direction.