PaperlayerView exact revision on arXiv

Source-linked AI summary

Forecast Evaluation for Data Scientists: Common Pitfalls and Best Practices

Hansika Hewamalage, Klaus Ackermann, Christoph Bergmeir

arXiv:2203.10716v2cs.LGstat.ME

TL;DR

Forecasting concepts are not yet mainstream among ML practitioners, creating a need for reliable evaluation guidance. The paper compiles pitfalls and best practices across forecast-evaluation steps, including partitioning and error calculation. It concludes with formalized guidelines intended to support better evaluation practices in the ML community.

  • Problem

    ML practitioners often lack established forecasting evaluation knowledge and best-practice guidelines, contributing to confusion and spurious conclusions.

  • Method

    The article provides a tutorial-like compilation of forecast-evaluation pitfalls and best practices covering data partitioning, error measures, statistical testing, and related steps.

  • Results

    The paper formalizes guidelines for forecast evaluation, including when k-fold cross-validation is valid for pure autoregressive setups and when temporal-order-preserving validation is appropriate for non-stationary series.

  • Takeaways & Limitations

    Correct evaluation principles can support more reliable conclusions and further development of ML-based forecasting systems.

  • Takeaways & Limitations

    Global or pooled models create scale-related issues during model building because normalization choices may conflict with evaluation on the original series scales.

Abstract

from arXiv · show

Machine Learning (ML) and Deep Learning (DL) methods are increasingly replacing traditional methods in many domains involved with important decision making activities. DL techniques tailor-made for specific tasks such as image recognition, signal processing, or speech analysis are being introduced at a fast pace with many improvements. However, for the domain of forecasting, the current state in the ML community is perhaps where other domains such as Natural Language Processing and Computer Vision were at several years ago. The field of forecasting has mainly been fostered by statisticians/econometricians; consequently the related concepts are not the mainstream knowledge among general ML practitioners. The different non-stationarities associated with time series challenge the data-driven ML models. Nevertheless, recent trends in the domain have shown that with the availability of massive amounts of time series, ML techniques are quite competent in forecasting, when related pitfalls are properly handled. Therefore, in this work we provide a tutorial-like compilation of the details of one of the most important steps in the overall forecasting process, namely the evaluation. This way, we intend to impart the information of forecast evaluation to fit the context of ML, as means of bridging the knowledge gap between traditional methods of forecasting and state-of-the-art ML techniques. We elaborate on the different problematic characteristics of time series such as non-normalities and non-stationarities and how they are associated with common pitfalls in forecast evaluation. Best practices in forecast evaluation are outlined with respect to the different steps such as data partitioning, error calculation, statistical testing, and others. Further guidelines are also provided along selecting valid and suitable error measures depending on the specific characteristics of the dataset at hand.

1. Introduction

Forecasting is increasingly assigned to ML practitioners, but evaluation concepts and best practices remain less established in ML than in fields such as NLP and Computer Vision. This article addresses the resulting pitfalls by compiling guidance for point forecast evaluation.

  • Forecasting is increasingly becoming a responsibility of data scientists with ML backgrounds, despite its traditional development within statistics and econometrics.
  • Forecast evaluation supports model selection, ensemble development, and final testing on held-out data.
  • Common mistakes include unsuitable benchmarks, error measures, visualisations, and data leakage during evaluation.
  • The article compiles best practices for point forecast evaluation to raise awareness among ML practitioners.

2. Problem Definition and Terminology

The paper defines univariate forecasting around past information, a forecast origin, and a future horizon, then distinguishes local and global models and fixed versus rolling-origin evaluation. These concepts establish how forecasting inputs, model training, and out-of-sample testing are organized.

  • Point forecasting predicts a selected statistic of the future distribution, while this article focuses on univariate forecasting using a series’ history and optional exogenous variables.
  • The forecast origin is the last known data point, and the forecast horizon is the future interval being predicted.
  • Without external variables, inputs are the target series’ past values; with known external variables, inputs may also include their past or future values.
  • An autoregression uses a fixed number of previous lags, arranged in an embedded matrix whose order is the model order.
  • Local models train parameters on one series, whereas global models train parameters across multiple series while predicting each series from its corresponding lags.
  • Out-of-sample forecasting evaluation uses fixed-origin or rolling-origin setups, with the latter moving the forecast origin forward across evaluations.

3. Motivation and Common Pitfalls

Forecast evaluation is complicated by time-series non-stationarities, non-normalities, and leakage risks that can mislead model comparisons. The paper emphasizes benchmarks, context-appropriate error measures, and evaluation procedures over visually appealing forecasts.

  • 3.1. Characteristics of Time Series: Time-series forecasting is harder than many ML tasks because non-stationarities and non-normalities are common.These characteristics include trends, unit roots, structural breaks, non-symmetric distributions, outliers, intermittency, and short histories.
  • 3.1. Characteristics of Time Series: Unit-root series can cause autoregressive ML models without preprocessing to forecast poorly beyond the training-value range.Figure 4 describes a continuously increasing series whose stochastic trend changes its mean, while unprocessed models have limited extrapolation capacity.
  • 3.2. Benchmarks for Forecast Evaluation: The naïve forecast is theoretically best for a random walk with no predictable properties beyond its unit root.Differenced ML models show behavior similar to the naïve benchmark and add no value under the stated data-generating process.
  • 3.3. Forecast Plots: Forecast plots can mislead because visual proximity may reflect horizontal rather than evaluation-relevant vertical distances.Rolling-origin naïve forecasts follow newly observed values and can look appealing even when visual inspection obscures the relevant errors.
  • 3.3. Forecast Plots: In random-walk settings, apparent accuracy improvements over the naïve method can be spurious, so plots should mainly support sanity checking.The paper recommends basing decisions primarily on error measures rather than forecast plots.
  • 3.4. Data Leakage in Forecast Evaluation: Forecasting leakage can arise from preprocessing, feature extraction, rolling-origin implementations, or misaligned series that expose one series’ future to another.The paper recommends splitting before preprocessing or feature extraction, verifying rolling-origin code, and checking temporal alignment within series sets.

4. Overview of the Forecast Evaluation Process

Forecast evaluation comprises data partitioning, error calculation, optional significance testing, and optional model selection or ensembling. Unlike ordinary regression evaluation, forecasting offers many choices without established standards, making identified pitfalls especially important.

  • Evaluation steps: Forecast model building and evaluation typically include data partitioning, error-measure calculation, statistical significance testing, and optional model selection.Instead of selecting one model, practitioners may also combine forecasts into an ensemble.
  • Evaluation steps: Ordinary regression evaluation applies fitted models to validation targets, computes prediction errors, summarizes them with measures such as RMSE, and selects the best model.The passage presents this as a relatively straightforward process with established error-measure practices.
  • Evaluation steps: Forecast evaluation has many options for each step, but no standards have yet been established despite known pitfalls in some setups.The article discusses data partitioning, model selection, error calculation, and statistical significance testing as commonly used evaluation activities.

5. Data Partitioning

Forecast evaluation must preserve the temporal structure and deployment conditions of time series while matching the dataset’s stationarity, serial correlation, and length. The paper contrasts fixed-origin, rolling-origin, and randomized cross-validation setups and provides selection guidance.

  • Evaluation setups: Out-of-sample evaluation can use fixed-origin or rolling-origin setups, evaluating forecast steps separately or averaging across the test period.The suitable choice depends on the forecasting scheme and the evaluation objective.
  • Fixed Origin Setup: Fixed-origin evaluation is fast and avoids leakage, but a single series yields few forecasts and may reflect only particular horizon phases.Long horizons can mix short-term dynamics with trend and seasonality, while short testing periods may not generalize across business phases.
  • Rolling Origin, Time Series Cross-Validation and Prequential Evaluation Setups: Rolling-origin evaluation updates the forecast origin across multiple periods, allowing performance monitoring as new actual data become available.It is therefore typically more practical for real-world applications and can support detection of concept drift in streaming settings.
  • Rolling Origin, Time Series Cross-Validation and Prequential Evaluation Setups: Expanding windows increase the training region as the origin advances, whereas rolling windows keep its size constant and discard the oldest observations.Expanding windows suit small datasets, while rolling windows may benefit autoregressive ML models that use only a fixed history.
  • (Randomised) Cross-Validation: Randomized k-fold cross-validation uses shuffled train-validation partitions efficiently but breaks temporal order and can introduce future observations into training.It may also turn forecasting into missing-value imputation and mishandle autocorrelation or non-stationarity.
  • Data partitioning for non-stationary data: Blocked or non-dependent cross-validation can address serial-correlation concerns, while tsCV is generally preferred for non-short series and temporally ordered non-stationary data.For short series, k-fold CV may be used if non-stationarity and autocorrelation are handled adequately.

6. Error Measures for Forecast Evaluation

Forecast error measures differ in their error bases, scaling procedures, and aggregation operators, so evaluation requires matching the measure to the series and statistic of interest. The paper catalogs these choices and highlights limitations of point-only evaluation.

  • Bias and variance: Forecast bias retains the sign of errors, revealing whether forecasts systematically overestimate or underestimate despite having small absolute errors.This directional information can matter for business decisions.
  • Common measures: MSE and MAE are simple regression-style measures, while forecasting requires many additional measures to enable comparisons across series.Scaling procedures cannot work reliably for every possible time-series non-stationarity and non-normality.
  • Data-dependent validity: Error measures often rely on unstated series preconditions, so a measure suitable for one domain may fail under different trends, level shifts, or distributions.Smart-meter and wind-power series differ from growing-business series in these characteristics.
  • Target statistic: Squared-error measures such as MSE and RMSE optimize for the mean, whereas other measures target different distributional statistics.The intended statistic should be distinguished for each point forecast error measure.
  • Beyond point forecasts: Point forecast evaluation alone does not assess the uncertainty of the overall predictive distribution.Distributional evaluation methods include randomized probability integral transforms and proper scoring rules.
  • Aggregation: Aggregating errors across forecast steps and series produces many model errors that must be summarized consistently in global forecasting evaluation.The paper discusses aggregation across available series and forecast steps.
  • Error-measure taxonomy: Forecast error measures vary by squared or absolute errors, scale-free transformations, and operators used to summarize errors.The paper organizes these measures and provides guidance for selecting them under different data characteristics.

6.1. Categorisation of Error Measures

The article categorizes forecast error measures by how errors are scaled and summarizes the assumptions, interpretations, and limitations associated with each family.

  • Measure families: Error measures are categorized following Hyndman and Koehler’s framework, including scale-dependent, scale-invariant, percentage-based, and relative or scaled measures.The categorization supports selecting measures according to the characteristics of the forecasting task and series.
  • Scale-dependent measures: RMSE remains on the original data scale, while squared-error losses produce unbiased point forecasts by targeting the mean.The passage contrasts RMSE with MSE and notes the statistical implication of squared errors.
  • Percentage-based measures: Percentage measures scale errors by actual values and are meaningful when the series has a meaningful zero and percentage interpretation.They were originally developed for high-volume, non-intermittent inventory series.
  • Percentage-based measures: sMAPE is not fully symmetric and penalizes underestimates more than overestimates for equal actual values.This asymmetry can favor slightly overestimating models.
  • Percentage-based measures: msMAPE applies a fixed scaling factor below a threshold, but the threshold depends on series scale and the measure lacks a theoretical foundation.The authors also note that winsorising skews the error distribution and leaves some sMAPE symmetry issues unresolved.
  • Weighted measures: WAPE scales errors using out-of-sample values across the forecast horizon, while sWAPE modifies its denominator to avoid exchange-related asymmetry.WRMSPE uses squared errors in the numerator, and related variants can aggregate errors across multiple series.

6.1.3. Measures based on Relative Errors

Relative-error measures divide model errors by benchmark errors at corresponding time steps, making performance interpretable relative to a reference forecast.

  • Definition and purpose: Relative errors divide each model error by the benchmark error at the same time step, commonly using naïve or seasonal-naïve forecasts.This scaling compares the forecasting model with a benchmark on the same series and time point.
  • Measure variants: Relative-error measures include MRAE, MdRAE, RMRSE, GMRAE, and RGRMSE.The article lists these measures as alternatives within the relative-error family.
  • Measure variants: GMRAE and RGRMSE are mathematically equivalent despite slightly different definitions.The equivalence concerns the geometric mean absolute and relative geometric root mean squared formulations.
  • Interpretation and limitations: Benchmark-based scaling improves interpretability and cross-dataset comparability, but benchmark competence changes the apparent magnitude of model errors.A weak benchmark can reduce the apparent impact of model errors, while a very strong benchmark can produce the opposite effect.

6.1.4. Relative Measures

Relative measures summarize forecast accuracy by comparing model errors with benchmark errors, providing intuitive thresholds but imposing multi-step and benchmark-consistency requirements.

  • Definition and variants: Relative measures divide a forecasting model’s error by the corresponding benchmark error, with examples including RelMAE, RelMSE, and RelRMSE.The framework also includes RSE and AvgRelMAE variants.
  • Specific formulation: RSE scales aggregate out-of-sample root squared errors by deviations from the out-of-sample mean, effectively using a mean forecast benchmark.The benchmark is computed across out-of-sample points and series.
  • Aggregation: Geometric means may be preferred over arithmetic means for summarizing relative measures across series.This preference is attributed to Davydenko and Fildes for cross-series aggregation.
  • Interpretation: Values below 1 indicate that the model is more accurate than the benchmark, whereas values above 1 indicate that the benchmark is better.This threshold-based interpretation makes relative measures comparatively intuitive.
  • Limitations: Relative measures require more than one-step forecasts to compute per-series MAE; otherwise they reduce to stepwise relative errors.The authors identify this as a limitation of applying the measures to one-step forecasts.

6.1.5. Measures based on Scaled Errors

Scaled-error measures normalize forecast errors by an in-sample benchmark error, offering symmetric and interpretable comparisons while requiring horizon- and benchmark-aware scaling.

  • Definition and purpose: Scaled errors divide a forecasting method’s error by the in-sample MAE of a benchmark such as the naïve method.The approach was introduced as an alternative to relative errors and relative measures.
  • Measure variants: The scaled-error family includes MASE, MdASE, and RMSSE, with RMSSE used in the M5 Forecasting Competition.These measures use scaled errors as their base quantity, with RMSSE using an MSE-based denominator.
  • Interpretation: Scaled measures treat positive and negative errors, and equal-sized errors at high- and low-valued points, symmetrically.This property distinguishes their error penalization from several percentage-based measures.
  • Interpretation: MASE values below 1 indicate improvement over the in-sample one-step-ahead naïve method, while values above 1 indicate the opposite.The interpretation is explicitly tied to that benchmark.
  • Limitations: One-step in-sample benchmarks can produce very large errors for multi-step forecasts because they represent an easier forecasting problem.The recommended remedy is to use in-sample multi-step benchmark forecasts matching the evaluation horizon.
  • Limitations: Scaled errors cannot be compared across datasets or series when different underlying benchmarks are used.Benchmark choice may also need to vary between naïve and seasonal-naïve methods across series.

6.1.6. Measures based on Ranks/Counting

Rank- and count-based measures make forecast evaluation scale-free by comparing model performance across series or time steps. Their rankings and benchmark comparisons, however, depend on the competing methods or chosen benchmark.

  • Rank-based measures summarize model rankings across series or horizon steps, producing scale-free evaluation even when absolute errors are used.
  • Rankings depend on the other competing methods, so the measured result can change when the comparison set changes.
  • Percentage Better counts how often a method outperforms a benchmark such as random walk across series and time steps.
  • The Percentage of Critical Event for Margin X reports the percentage of forecasts whose error exceeds a specified margin X.

6.1.7. Measures based on a Transformation

Transformation-based measures use logarithms to obtain scale-invariant error summaries, particularly for non-negative series. These measures are symmetric and convey both differences and ratios, while optimizing for the geometric mean.

  • Logarithmic error measures add 1 to non-negative actuals and predictions before applying the natural logarithm, avoiding logarithms of zero.
  • Logarithmic error measures optimize for the geometric mean of the distribution and provide information about both differences and ratios.
  • RMSLE and its weighted variant NWRMSLE are logarithmic measures used in several forecasting competitions.RMSLE was used for Walmart Stormy Weather and Recruit Restaurant Visitor forecasting; NWRMSLE was used for Corporación Favorita Grocery Sales.
  • Log transformation makes data approximately normal through a monotonic rescaling, but it is not bounded to a predefined range.
  • RMSLE is symmetric because interchanging actual and predicted values does not change the measure.

6.1.8. Other Error Measures in the Literature

The literature includes rate-based, weighted, and correlation-based error measures. Their suitability depends on the decision context, weighting scheme, scale differences, and whether correlation can reward biased forecasts.

  • Rate-based measures such as MSR and MAR are introduced as alternatives based on rate errors.
  • Rate-based errors are generally suitable for inventory decisions where maintaining optimal inventory and avoiding underestimations matter more than exact time-step accuracy.
  • WMAE uses weights to increase or decrease the impact of particular series or time steps according to their importance.
  • WMAE can still be dominated by high-scale series even when identical weights are assigned, unless weights are adjusted for scale or a scale-free base error is used.
  • CORR may select a biased forecast because perfect correlation can occur between actual values and biased predictions.

6.2. Problems of the Error Measures and Guidelines

No error measure is universally appropriate: evaluation must account for scale, non-stationarity, seasonality, intermittency, and business objectives. The paper therefore recommends matching measures and benchmarks to series characteristics and decision context.

  • Scale-dependent measures are suitable across similarly scaled series but can be dominated by high-volume series when scales differ.
  • Scale-free measures support comparisons across differently scaled series, but ranking-based evaluation may favor a method that wins often rather than one that performs consistently well.
  • Global models introduce scale concerns during model building because normalization helps optimization while evaluation may require original meaningful scales.
  • Evaluating forecasts with multiple measures provides a practical sanity check, while forecasting accuracy does not necessarily translate linearly into business value.
  • The article limits its scope by excluding exhaustive analysis of complex loss functions that combine forecasting with downstream optimization.
  • Percentage measures can underestimate errors at seasonal peaks and overstate errors when actual values are small; WAPE, ND, or NRMSE can reduce this problem.
  • Strong trends can make aggregated-scale measures inappropriate, whereas rolling-window scaling or in-sample benchmark scaling can address non-stationarity.

7. Statistical Tests for Significance

Forecast error measures rank competing methods but do not establish whether performance differences are statistically significant. The paper discusses tests, post-hoc comparisons, and how data volume and model selection affect significance conclusions.

  • Error measures rank methods but cannot determine whether the best method’s advantage is statistically significant or likely to generalize.
  • The Diebold-Mariano and Wilcoxon rank-sum tests compare two forecasts, while the former is parametric and the latter is non-parametric.The Giacomini-White test also compares two forecasts and can assess conditional predictive ability.
  • Multiple comparisons require corrections such as Bonferroni when testing more than two forecasts.
  • F-tests and t-tests compare mean error distributions under normality assumptions, whereas the Friedman test assesses significance across multiple methods.The Friedman test requires a post-hoc procedure to identify which method pairs differ after rejecting its null hypothesis.
  • Significance-test results depend on the ranking level, amount of data, and set of models being compared.With many series, the critical difference can become very small, making small performance differences statistically significant.
  • A critical-difference diagram can show groups of methods that are not significantly different, while insufficient data may leave some pairwise differences unresolved.In the example, A, B, and C are indistinguishable, D, E, and F are worse than those three, and comparisons involving E remain unresolved.

8. Conclusions

The paper compiles forecast-evaluation pitfalls and best-practice guidelines for ML practitioners, covering benchmarks, validation, leakage, error measures, non-stationarity, and significance testing. It emphasizes that evaluation choices must match the data, model, and practical utility.

  • Forecast evaluation lacks universally established best practices, leaving many ML practitioners unfamiliar with its pitfalls and contributing to confusion.
  • Models should always be compared with appropriate simple benchmarks, especially naïve and seasonal naïve forecasts.Without a naïve comparison, the quality of more complex models cannot be meaningfully assessed.
  • Data leakage must be explicitly prevented during rolling-origin evaluation and preprocessing steps such as smoothing, decomposition, and normalization.
  • k-fold cross-validation is valid and data-efficient for pure autoregressive setups without underfitting, while time-series cross-validation is preferred with enough data.For models with continuous states, such as RNNs and ETS, temporal ordering makes time-series cross-validation potentially the only applicable strategy.
  • No single evaluation measure is valid for every scenario; error, summarization, and scaling choices should reflect data characteristics, user expectations, and business utility.For meaningful real-world scales, scale-dependent measures remain valuable when scales reflect business utility such as price or volume.
  • Several evaluation measures can be used for sanity checking, but one main measure must be selected because forecasts cannot be optimized for all measures simultaneously.
  • Statistical testing should balance model diversity against the number of data points to reduce spurious similarity or difference conclusions.
  • The article highlights moving-average smoothing and STL decomposition as techniques for capturing non-stationarities.
Loading 2203.10716v2…