Source-linked AI summary
Kaggle forecasting competitions: An overlooked learning opportunity
Casper Solheim Bojer, Jens Peder Meldgaard
TL;DR
The paper addresses the limited academic attention to Kaggle forecasting competitions involving real-life business tasks. It reviews six competitions, analyzes their datasets and top solutions, and finds higher intermittence and entropy than in M-competitions alongside strong performance from global ensembles and adapted machine-learning models.
Problem
Kaggle competitions with real-life business forecasting tasks have been largely overlooked, limiting attention to their potential lessons for forecasting research.
Method
The paper reviews six Kaggle competitions by analyzing dataset characteristics, benchmarking submitted solutions, and examining top-performing methods.
Results
Global ensemble models outperform local single models, while gradient boosted decision trees and neural networks perform strongly after adaptation to forecasting tasks.
Takeaways & Limitations
Kaggle competitions offer the forecasting community lessons about daily and weekly business time series, external information, cross-learning, and machine-learning adaptations.
Takeaways & Limitations
Because post-competition test data are unavailable, the review cannot test significant performance differences or evaluate alternative error measures.
Abstract
from arXiv · showhide
Competitions play an invaluable role in the field of forecasting, as exemplified through the recent M4 competition. The competition received attention from both academics and practitioners and sparked discussions around the representativeness of the data for business forecasting. Several competitions featuring real-life business forecasting tasks on the Kaggle platform has, however, been largely ignored by the academic community. We believe the learnings from these competitions have much to offer to the forecasting community and provide a review of the results from six Kaggle competitions. We find that most of the Kaggle datasets are characterized by higher intermittence and entropy than the M-competitions and that global ensemble models tend to outperform local single models. Furthermore, we find the strong performance of gradient boosted decision trees, increasing success of neural networks for forecasting, and a variety of techniques for adapting machine learning models to the forecasting task.
1 Introduction
Forecasting competitions provide empirical tests of forecasting methods in realistic settings, but Kaggle competitions involving real-life business tasks have received limited attention. This paper reviews six such competitions to identify lessons for the forecasting community and compare them with established M-competitions.
- Forecasting supports business planning processes including financial planning, inventory management, and capacity planning.
- Forecasting competitions are widely regarded as the community standard because they evaluate ex-ante forecasts in real-life settings.
- Practitioners questioned M4’s representativeness because high-frequency series and external information such as exogenous variables and hierarchies were underrepresented.
- M5 was announced on Kaggle in response to concerns about high-frequency data and access to external information.
- The paper reviews six Kaggle competitions by analyzing their datasets, benchmarking solutions, and examining top-performing methods against M3 and M4 findings.
2 Background
M-competitions established empirical forecasting comparison as a central community practice and used broad participation to support fairer method comparisons. M4 also examined complex machine-learning methods, while imposing data-selection criteria that excluded intermittent and incomplete series.
- M-competitions shifted attention toward empirical forecast accuracy and enabled diverse contestants to compare methods more fairly.
- M4 included business-domain continuous series but excluded intermittent series and missing values, requiring more than three full seasonal periods at most frequencies.
- M4 findings covered complex versus simple models, cross-learning, prediction uncertainty, and ensembling.
- The ML-versus-statistical distinction is debated because top M4 methods included neural networks and gradient boosted decision trees.
- Kaggle adds public leaderboard feedback, multiple submissions, and private evaluation on unseen data to business-oriented forecasting competitions.
3 Selection of Competitions
The authors selected six Kaggle competitions covering retail, web traffic, and restaurant forecasting tasks, with varied aggregation levels and information structures. These datasets are narrower than M4’s coverage but more representative of particular business contexts because external variables and hierarchies are available.
- Selection procedure: The initial screening retained forecasting competitions from 2014 onward, producing a pool of seven competitions.
- Selection procedure: The Grupo Bimbo competition was excluded because many series had only one observation and at most seven weekly observations.
- Selected competitions: The six selected competitions include four retail tasks plus web-traffic and restaurant domains, with substantially different numbers of series and aggregation levels.
- Selected competitions: The remaining disaggregate tasks forecast daily webpage visits and restaurant visitors, with the restaurant data including reservations and Wikipedia providing a forecast hierarchy.
- Representativeness: The selected competitions cover a limited subset of business forecasting tasks compared with M4 but provide exogenous variables and business hierarchies.
4 Analysis of Competition Datasets
The paper represents Kaggle, M3, and M4 series in a shared feature-based instance space after preprocessing irregular competition data. Kaggle datasets include more intermittent and higher-entropy series than M-competitions, while aggregation level and domain shape distinct patterns.
- Representativeness analysis: Kaggle datasets were positioned relative to M3 and M4 using feature vectors that summarize each time series in a two-dimensional instance space.
- Data preprocessing: Preprocessing set negative or missing values to zero, removed unsuitable series, regularized intervals, and removed leading zeros.
- Representativeness analysis: The feature set measures spectral entropy, trend, seasonality, seasonal period, first-order autocorrelation, and an optimal transformation.
- Instance-space findings: Highly aggregated Rossmann, Recruit Restaurant, and Walmart Store Sales series share density peaks and show lower trend and autocorrelation but higher spectral entropy than most M4 series.
- Instance-space findings: More than 98% of series in Corporación Favorita, Walmart Stormy Weather, Recruit Restaurant, and Rossmann exhibit some intermittency.
- Instance-space findings: Approximately 16% of Walmart Store Sales series and 26% of Wikipedia series exhibit intermittency.
5 Benchmarking Kaggle Solutions
The paper tests whether Kaggle competition solutions add value beyond simple forecasting benchmarks. Across the reviewed competitions, top solutions outperform naïve and seasonal naïve methods.
- The benchmark compares Kaggle solutions against naïve and seasonal naïve forecasting methods.These methods were selected because they are simple, robust to missing data, and commonly used as forecasting workhorses.
- 25%: first-place solutions improve on the better simple benchmark by more than 25% across all competitions.
- 10%: 50th-place solutions beat the simple benchmarks by at least 10% across all competitions.
6 Competition Review
The review examines methods used by top contestants across six Kaggle forecasting competitions. It finds strong roles for adapted machine-learning models, global ensembles, gradient-boosted decision trees, and increasingly successful neural networks, while performance varies by competition.
- Review approach: Top-25 solution descriptions were reviewed from Kaggle forums, but missing method reports limit what can be learned from the competitions.The review collected textual descriptions and code, while blank or absent descriptions were common enough to be identified as a limitation.
- Competition-specific patterns: Conventional time-series models dominated another competition, while machine-learning models mainly supplemented ensembles after holiday alignment.The available temperature, fuel-price, CPI, unemployment, and markdown variables did not prove useful for accurate forecasts in that competition.
- Machine-learning adaptations: XGBoost won one competition after adaptation with time-series and exogenous-variable features plus ridge-based trend adjustment.The feature set included rolling statistics, event counters, weather information, and multiple seasonal indicators.
- Machine-learning adaptations: Global XGBoost ensembles generally used event counters and hierarchical statistics, with holiday and promotion variables essential for high performance in that competition.
- Machine-learning adaptations: 11%: the winning solution beat a time-series hybrid by 11%, and 31%: it beat a median-by-store benchmark by 31%.The comparison came from a competition where more complex models yielded much better solutions.
- Neural networks: Neural-network solutions used recurrent, convolutional, and feedforward architectures, with feature engineering and validation approaches varying across contestants.The reviewed neural-network solutions included historical lags, categorical variables, scaling, and seasonality-related inputs.
- Neural networks: Top solutions did not always favor long histories: one winner selected only one, three, or five months of recent data despite multiple seasons being available.
- Competition-specific patterns: Boosted decision trees outperformed recurrent and convolutional neural-network variants in one competition, where those variants were around 2% less accurate than first place.
7 Discussion
Across six Kaggle forecasting competitions, ensembles and global models consistently performed well, while the strongest methods varied across competitions. Dataset characteristics and practical constraints shape how broadly these findings can be interpreted.
- General modeling strategies: Ensembles won all six competitions, while global models appeared among every winner, sometimes combined with local models.This supports the continued value of ensembling and cross-learning across domains and forecasting tasks.
- Method performance: No single method dominated all competitions: the earliest two favored time-series or statistical methods, whereas four later competitions favored gradient boosted decision trees or neural networks.The authors relate this shift partly to method maturity and the availability of forecasting-specific adaptations.
- Dataset characteristics: Intermittency, hierarchy information, and predictive exogenous variables characterized the later datasets and provided conditions suited to global machine-learning methods.The Walmart Store Sales and Walmart Stormy Weather datasets differed in continuity, exogenous-variable usefulness, dataset size, hierarchy information, or forecast horizon.
- Method performance: Neural networks outperformed gradient boosted decision trees in the Wikipedia competition, while both methods placed among the top solutions in three other later competitions.The Wikipedia dataset was very large and contained no useful exogenous variables.
- Validation: A hold-out dataset matching the forecast horizon was successfully used across all six competitions to validate models and prevent overfitting.The described competition workflow also included training, validation, a second validation set with summary feedback, and a final test set.
- Practical applicability: Top solutions generally improved on seasonal-naïve benchmarks by 20% to 74%, but their greater computational cost and time require consideration of decision cost structures and labor requirements.Simple methods may require substantial human labor for forecast adjustments, whereas complex methods may shift work toward exception management.
- Limitations: Kaggle results have interpretive limits because researchers cannot access the post-competition test set or fully analyze performance across dataset subsets.These restrictions prevent significance testing, alternative error-measure evaluation, and detailed analysis of method strengths and weaknesses.
- Limitations: The review is also constrained by contestants’ uneven public sharing of solutions and code, so its conclusions should be tested across additional datasets.The authors identify the upcoming M5 competition as an initial testing ground for these hypotheses.
8 Conclusions
The six Kaggle competitions provide lessons for forecasting daily and weekly business time series. Their results show underrepresented M4-like series, benefits from external information, and strong performance from global ensembles and machine-learning methods.
- Kaggle competitions offer the forecasting community substantial lessons about daily and weekly business time series.
- M4 contains time series resembling Kaggle data, but these characteristics are underrepresented in M4.
- External information, including exogenous variables and business hierarchy, led to significant improvements in forecast accuracy.
- Global ensemble models outperform local single models, while machine-learning methods outperformed conventional statistical methods in four recent Kaggle competitions.
- Top Kaggle solutions resemble leading M4 solutions through their use of gradient boosted decision trees or neural networks, with adaptations required for forecasting.
- The authors encourage the forecasting community to develop machine-learning strategies further and view M5 as an opportunity to apply these lessons.
Appendix A: Preprocessing of Kaggle Competition Datasets
The Kaggle datasets were prepared without predefined time-series filters, using preprocessing to regularize values, groupings, dates, and series boundaries. The appendix also references Figure 1's adjusted-frequency comparison.
- Unlike M4, the Kaggle datasets had no predefined filters applied to their time series.
- The preprocessing replaced NA or negative values with zeros and removed series containing only zeros.
- Training data were restricted to groupings also present in available test sets, except for the Recruit Restaurant and Wikipedia competitions.
- Missing dates were filled and forecast units set to zero to mitigate irregular time series, including store closures in Rossmann.
- Leading and trailing zeros were removed from time series that began with consecutive zeros.
- Figure 1 compares M4 features using original and adjusted frequency.