Forecast Evaluation for Data Scientists: Common Pitfalls and Best Practices

• The article focuses on point-forecast evaluation, an important forecasting step increasingly undertaken by ML-oriented data scientists unfamiliar with forecasting research.Forecasting’s sequential and self-supervised nature creates pitfalls across the process, motivating focused guidance on evaluation.Source: §1 · p. 2
• Forecast evaluation is complicated by time-series non-independence, non-stationarities, non-normalities, alternative data-partitioning schemes, and error measures that can fail under specific conditions.Choices also include whether to aggregate errors across series or evaluate forecast-horizon steps separately.Source: §1 · p. 2
• Inadequate benchmarks, unsuitable metrics, unfair loss-function comparisons, weak empirical demonstrations, and misunderstood preprocessing practices produce misleading results and have slowed progress in ML forecasting.The introduction notes examples including MAPE on series containing negative values, comparisons using MAE when only one method is trained with L1 loss, and failure to address seasonal or scaling-related non-stationarity appropriately.Source: §1 · p. 3
• The article provides a comprehensive tutorial and guideline covering forecast-evaluation terminology, pitfalls, data partitioning, evaluation measures, metric selection, and statistical significance testing.Its stated goal is to raise awareness among ML practitioners and summarize best practices for the overall point-forecast evaluation process.Source: §1 · pp. 3–4

Forecasting scope

• The article focuses on point forecasts of a mean or median for univariate series, using past target values and potentially time-varying exogenous variables.Multivariate regression predicting many target series jointly is outside scope.Source: §2 · p. 4

Forecasting terminology

• A forecast origin is the last known observation, while the forecast horizon is the future period length being predicted from available inputs.The forecasting function may be nonlinear and non-parametric, with parameters θ and inputs Xt available through the forecast origin.Source: §2 · p. 4, Figure 1 · p. 5

Autoregressive models

• In an autoregression of order p, lagged values 1 through p predict the next target, producing n − p training instances for a series of length n.Each training row contains p + 1 consecutive observations, with the target shifted one time step across rows.Source: §2 · p. 4, Figure 2 · p. 5

Local and global models

• Local models train parameters on one series, whereas global models train across many series while using the corresponding series’ lags for each prediction.With many series, local modeling requires one model per series; global modeling pools data across series.Source: §2 · pp. 4, 6

Out-of-sample evaluation

• Out-of-sample evaluation uses fixed-origin or rolling-origin setups: the former keeps training data and origin fixed, while the latter rolls the origin with a fixed horizon.Rolling-origin evaluation, also called time series cross-validation or prequential evaluation, creates multiple test periods and can leak future information into later training if mishandled.Source: §2 · p. 6, Figure 3 · p. 6

3.1. Characteristics of Time Series

• Non-stationarities and non-normalities make forecasting harder than typical ML tasks and require evaluation choices suited to each series.Relevant characteristics include trends, unit roots, structural breaks, non-symmetric distributions, short histories, fat tails, outliers, and intermittency; there is no universal evaluation rule.Source: §3.1 · pp. 7–8

3.2. Benchmarks for Forecast Evaluation

• The naïve forecast is an essential benchmark because it is theoretically optimal for a random walk and makes apparent gains by complex models potentially spurious.For a random walk with zero-mean normal innovations, more complex methods have no predictive power beyond the naïve forecast; seasonal series should instead also use a seasonal naïve benchmark.Source: §3.2 · pp. 8, 9, 11

3.3. Forecast Plots

• Forecast plots can mislead evaluation, so model decisions should rely primarily on error measures and appropriate benchmarks rather than visual appeal or apparent fit.Rolling-origin naïve forecasts can visually follow actuals because horizontal distances obscure the vertical errors, while a fixed-origin naïve forecast may look unrealistic despite minimizing error given current information.Source: §3.3 · pp. 11–12

3.4. Data Leakage in Forecast Evaluation

• Forecasting makes data leakage especially difficult to avoid because rolling-origin retraining, preprocessing, feature extraction, and cross-series timing can expose unavailable information.Leakage may arise when transformations or time-varying features use the full series, when unaligned global-model series reveal another series’ future, or when a forecast uses data unavailable at the stated horizon.Source: §3.4 · p. 13
• Leakage prevention requires verifying rolling-origin implementations, partitioning before preprocessing and feature extraction, and ensuring series do not contain future information about one another.These safeguards are emphasized because external evaluation can be difficult in self-supervised forecasting workflows.Source: §3.4 · pp. 13–14

• Forecast evaluation includes data partitioning, error measure calculation, optional statistical significance testing, and optional model selection.Source: §4 · p. 14
• Unlike straightforward regression evaluation, forecasting has no established standards because multiple evaluation options and setup-specific pitfalls exist.In regression, predictions are compared with validation targets, errors are summarized using measures such as RMSE, and the best model is selected by the final validation error.Source: §4 · p. 14
• Model selection is optional: practitioners may deploy an ensemble of all fitted models instead of selecting a single best model.Source: §4 · p. 14

5. Data Partitioning

• Forecast evaluation may score each horizon separately or average across the test period, and model-selection choices should follow the underlying forecasting scheme.Reliable cross-validation is essential for estimating generalization error under the series’ serial correlations, non-stationarities, and amount of data, as emphasized in M5.Source: §5 · p. 14, §5.6 · p. 21

5.1. Fixed Origin Setup

• Fixed-origin evaluation is simple and leakage-resistant but may provide too few forecasts, mix incompatible horizons, and fail to generalize across business phases.A single series yields one forecast per horizon step, while short testing periods and long horizons can confound short-term dynamics with trend or seasonality.Source: §5.1 · p. 15

5.2. Rolling Origin, Time Series Cross-Validation and Prequential Evaluation Setups

• Rolling-origin evaluation generates multiple forecasts per horizon step and better reflects operational updating, with expanding windows suited to short series and rolling windows retaining only recent observations.Rolling windows can benefit autoregressive ML models and streaming settings, whereas expanding-window prequential evaluation may overestimate validation error; hybrid windows can help short series.Source: §5.2 · pp. 15–17, Figure 8 · p. 16

5.3. (Randomised) Cross-Validation

• Randomized k-fold CV uses data efficiently but can violate temporal order, mishandle autocorrelation and seasonality, leak future information, and turn forecasting into missing-value imputation.Blocked or non-dependent CV can address dependence; standard k-fold CV is valid for sufficiently competent pure AR models when residual autocorrelation is absent, especially for short series.Source: §5.3 · pp. 17–18

5.4. Data partitioning for non-stationary data

• For non-stationary series, temporally ordered tsCV is generally safest, but structural breaks can defeat every scheme and transformations may make CV valid after stationarity is achieved.Gradual distributional change favors end-of-series validation, deterministic components can be modeled separately, and no transformation is universally best across datasets.Source: §5.4 · pp. 19–20

5.5. Other model selection methods

• Information criteria use all observations for training but are mainly suitable for small datasets because likelihood comparability and model complexity are problematic across diverse ML models.AIC is asymptotically equivalent to minimizing OOS one-step MSE under stated conditions, but cannot generally compare models with different orders, families, or differencing.Source: §5.5 · pp. 20–21

5.6. Summary and guidelines for data partitioning and model selection

• For non-short series, tsCV is usually preferred over k-fold CV; for short series, k-fold CV is appropriate when non-stationarity and residual autocorrelation are controlled.These recommendations are summarized in the data-partitioning guidelines, while the appropriate strategy should estimate final test-set error without under- or overestimation.Source: §5.6 · p. 21, Figure 10 · p. 21

• Bias should be retained with its sign to show systematic over- or underestimation, while RMSE equals the error standard deviation for an unbiased model.Mean Error provides a scale-dependent bias measure, and bias and variance can imply different business decisions.Source: §6 · pp. 21–22
• Error-measure selection must be mathematically and practically robust to the data because scaling procedures can fail under different non-stationarities and non-normalities.Measures may target specific business needs or correct earlier measures while introducing new limitations; interpretability and business alignment are additional requirements outside this work’s focus.Source: §6 · pp. 22–23
• Squared-error measures such as MSE and RMSE target the mean, whereas absolute-error measures such as MAE and MASE target the median, making measure choice consequential for skewed or intermittent series.Petropoulos et al. (2020) caution against evaluating identical forecasts with measures targeting different statistics, while Koutsandreas et al. (2021) report that large samples can produce agreement among measures on consistently dominating methods.Source: §6 · p. 23
• Point-forecast evaluation alone does not quantify predictive uncertainty adequately, so overall predictive distributions and proper scoring rules can also be considered.Kolassa (2016) discusses randomized probability integral transforms and scoring rules including logarithmic, Brier, and ranked probability scores.Source: §6 · pp. 23–24
• Errors should be summarised across series and forecast horizons with deliberate operators, since medians and geometric means reduce outlier sensitivity but geometric means can reward catastrophic performance on most series.Using multiple summary operators can reveal problems in individual errors, and horizon weights or different aggregation orders can produce different final measures.Source: §6 · p. 24
• Table 1 maps error measures to scaling types and data conditions, while Figure 11 provides a requirement- and data-characteristic-based selection flow chart, including cautious use under problematic circumstances.The table covers no scaling, actual-value scaling, benchmark-error scaling, and per-step, per-series, or all-series scaling; its outlier selections emphasize robustness rather than capturing outliers.Source: §6 · pp. 24–25

6.1. Categorisation of Error Measures

• The categorisation follows Hyndman and Koehler (2006) and organizes forecast error measures into distinct families for evaluation.Source: §6.1 · p. 25

6.1.1. Scale-Dependent Error Measures

• Scale-dependent measures use the series scale and a common base error et, including MSE, MAE, RMSE, RMdSE, MdAE, GRMSE, and GMAE.GRMSE was proposed by Syntetos and Boylan (2005).Source: §6.1.1 · p. 25
• RMSE remains on the original data scale and supports unbiased mean forecasts under squared-error loss, whereas geometric measures can reward one exceptionally good series despite poor performance elsewhere.GMAE and GRMSE are mathematically equivalent despite slightly different definitions.Source: §6.1.1 · p. 28
• Scale-invariant measures enable comparisons across differently scaled series, but global or constant scaling can be problematic under time-series non-stationarities and non-normalities.Chen et al. (2017) favor scaling closely related to the series scale at specific observation points, including per-step scaling.Source: §6.1.1 · p. 28

6.1.2. Measures based on Percentage Errors

• Percentage errors scale base errors by actual values and were introduced for high-volume, non-intermittent inventory series because percentage loss is intuitive and communicable.Examples include MAPE, MdAPE, RMSPE, RMdSPE, sMAPE, sMdAPE, msMAPE, MAAPE, WAPE, WRMSPE, RTAE, ND, and NRMSE.Source: §6.1.2 · pp. 28–29
• Percentage measures are unsuitable without a meaningful zero, can exceed 100%, and are asymmetric because exchanging actual and forecast values changes the error.These properties can make a zero forecast appear preferable under 100% MAPE.Source: §6.1.2 · p. 29
• sMAPE reduces MAPE’s exchange asymmetry but still penalizes underestimates more, may favor overforecasting, and has limited interpretability.Its asymmetry may nevertheless suit domains where underestimates cost more than overestimates (Armstrong, 2001).Source: §6.1.2 · p. 29
• The proposed sWAPE avoids differing errors when actual and forecast values are exchanged by modifying WAPE’s denominator, while msMAPE is a winsorised but ad-hoc sMAPE variant.msMAPE uses a fixed scaling factor below a threshold, but its threshold depends on series scale and its statistical properties remain unexplored.Source: §6.1.2 · pp. 30–31

Definition

• Relative errors divide each time-step forecast error by the corresponding error from a benchmark, commonly the naïve method.This compares model performance against a benchmark on the same series while standardising for forecastability.Source: §6.1.3 · pp. 32–33

Measures

• The measures include MRAE, MdRAE, RMRSE, GMRAE, and RGRMSE; GMRAE and RGRMSE are mathematically equivalent despite different definitions.Source: §6.1.3 · p. 33

Advantages

• Relative measures provide direct benchmark-based interpretations and enable comparisons across datasets, unlike unbounded measures whose values lack context without an explicit benchmark.For example, a MAPE value alone does not indicate whether performance is high or low for a particular series.Source: §6.1.3 · pp. 33–34

Limitations

• Benchmark choice is critical because weak benchmarks dampen model errors, whereas exceptionally strong benchmarks can exaggerate them and obscure models that perform well on those series.Source: §6.1.3 · p. 34

Relative Measures

• Relative measures divide a forecasting model’s error by a benchmark’s error, with values below 1 favoring the model and values above 1 favoring the benchmark.They provide an intuitive interpretation of comparative forecast accuracy.Source: §6.1.4 · pp. 34–35
• RelMAE, RelMSE, and RelRMSE extend relative evaluation across MAE, MSE, and RMSE, while analogous forms can use MdAE, MAPE, and sMAPE.These measures compare each selected error metric with the corresponding benchmark metric.Source: §6.1.4 · p. 34
• RSE scales pooled out-of-sample root squared error by deviations of actuals from their mean, making the mean forecast the benchmark across series and test points.The scaling uses all out-of-sample time steps across all series.Source: §6.1.4 · pp. 34–35
• AvgRelMAE summarizes relative errors across series, with Davydenko and Fildes (2013) arguing that the geometric mean is more suitable than the arithmetic mean.The formulation accounts for the number of series and testing steps per series.Source: §6.1.4 · p. 35
• Relative measures require multiple-step forecasts to compute each series’ MAE; one-step forecasts otherwise revert to relative errors, while scale-dependent metrics require similarly scaled series.Computing MAE across multiple series can address the one-step case, but metric-specific pitfalls remain.Source: §6.1.4 · p. 35

Definition and variants

• Scaled errors divide a forecasting method’s error by a benchmark’s in-sample error, commonly the naïve method’s MAE, and support MASE and MdASE.Using the benchmark’s MSE instead yields RMSSE, which was used in the M5 Forecasting Competition.Source: §6.1.5 · pp. 35–36

Interpretation and properties

• Scaled-error measures penalize positive and negative errors, and errors of equal magnitude at high- and low-valued series points, symmetrically.MASE values below 1 indicate better performance than the in-sample one-step-ahead naïve method, whereas values above 1 indicate worse performance.Source: §6.1.5 · p. 36

Practical limitations

• Scaled errors are difficult to communicate in business applications because they compare against in-sample benchmark errors, so they are used mainly in research.Although Hyndman and Koehler (2006) describe them as applicable across many forecasting scenarios, Koutsandreas et al. (2021) note their limited practical interpretability.Source: §6.1.5 · p. 36

Horizon dependence

• For multi-step forecasts, one-step-ahead naïve scaling can produce very large errors far ahead because the benchmark solves an easier problem.Using in-sample multi-step benchmark forecasts matching the evaluation horizon gives a more reasonable comparison, but implementation and interpretation become harder.Source: §6.1.5 · p. 36

Benchmark selection

• Scaled-error results cannot be compared across datasets or series when different underlying benchmarks are used.The benchmark may need to be one-step naïve for some series and seasonal naïve for others.Source: §6.1.5 · p. 36

6.1.6. Measures based on Ranks/Counting

• Rank-based measures summarize model rankings across series or horizon steps, producing fully scale-free errors even with absolute error, but depend on the competing methods.The M and M3 competitions used rankings among participating methods.Source: §6.1.6 · p. 37
• Percentage Better counts how often a method outperforms a benchmark such as random walk across series and time steps, reporting the result as a percentage.Hyndman and Koehler (2006) call this the PB score and define it using base errors such as MAE.Source: §6.1.6 · p. 37
• The Percentage of Critical Event for Margin X measures the percentage of forecasts whose error exceeds X, indicating deviation severity and whether action may be needed.The error measure E in Equation 52 can be any measure defined according to user requirements.Source: §6.1.6 · p. 37

• Logarithmic error measures add 1 to non-negative actuals and predictions before applying the natural logarithm, avoiding problems with zero values.Source: §6.1.7 · p. 37
• Logarithmic errors optimize for the geometric mean and provide information about both differences and ratios, similar to percentage-based measures.Source: §6.1.7 · p. 37
• RMSLE, also called Log Accuracy Ratio, was used in the Walmart Stormy Weather and Recruit Restaurant Visitor Forecasting Competitions (Bojer and Meldgaard, 2020).Source: §6.1.7 · pp. 37–38
• NWRMSLE was used in the Corporación Favorita Grocery Sales Forecasting Competition, weighting perishable items at 1.25 and all others at 1.00 (Bojer and Meldgaard, 2020).Source: §6.1.7 · p. 38
• Log transformation approximately normalizes data and RMSLE is symmetric, while the measures balance over-forecasts and under-forecasts to produce unbiased forecasts.Source: §6.1.7 · p. 38

Rate-based Error Measures

• Rate-based measures, including Mean Squared Rate (MSR) and Mean Absolute Rate (MAR), use c_t as the base error and suit inventory decisions prioritizing optimal stock over per-step accuracy.They are generally appropriate when avoiding underestimation matters more than exact accuracy at each time step. Introduced by Kourentzes (2014).Source: §6.1.8 · pp. 38–39

Weighted Mean Absolute Error (WMAE)

• Weighted Mean Absolute Error (WMAE) emphasizes important series or time steps, such as promotion periods or holiday weeks, through user-assigned weights.In the Walmart Store Sales Forecasting Competition, holiday-week days received weight 5 and other days weight 1; used by Bojer and Meldgaard (2020).Source: §6.1.8 · p. 39
• WMAE weights can remain distorted when series have different scales, so scale-adjusted weights or a scale-free base error may be needed.A high-scale series can contribute more error than a low-scale series despite identical weights.Source: §6.1.8 · p. 39

Empirical Correlation Coefficient (CORR)

• Empirical Correlation Coefficient (CORR) measures linear association across actuals and forecasts, but higher values are better and can favor biased models.Proposed by Lai et al. (2018), CORR selected heavily overestimating model A with 4.55 over model B with 4.48, although model B was better suited.Source: §6.1.8 · p. 39, Figure 12 · p. 40

6.2. Problems of the Error Measures and Guidelines

• Error measures should be selected according to series characteristics, scale comparability, and business objectives rather than applied universally.Scale-dependent measures suit similarly scaled series and meaningful original scales, while scale-free measures support comparisons across differently scaled series; multiple measures can sanity-check business relevance.Source: §6.2 · pp. 40, 42

6.2.1. Count Data Well above Zero with Stationarity

• For bounded series without trends or seasonality, error measures are generally applicable, but benchmark-scaled measures require caution when naïve forecasts perform poorly on highly fluctuating AR series.In scenario A, MAE is slightly higher at 0.41 than in scenario B, yet MRAE is 0.25 versus 4.46 because the naïve forecast performs poorly in scenario A.Source: §6.2.1 · p. 43

6.2.2. Seasonality

• Seasonal series can make percentage measures understate uncaptured peaks and overstate errors at small values, so WAPE, ND, or NRMSE may be preferable.Benchmark-scaled measures should use a benchmark capable of capturing seasonality, such as a seasonal naïve method.Source: §6.2.2 · pp. 43, 45, Figure 14 · p. 44

6.2.3. Trends

• Strong trends distort aggregated or benchmark-based scales, whereas rolling-window statistics or MASE can better accommodate changing levels; global scaling and log errors remain problematic.Log transformation can greatly reduce the impact of errors on exponentially trended series.Source: §6.2.3 · p. 45, Figure 15 · p. 46, Figure 16 · p. 47

6.2.4. Unit Roots

• Unit-root series resemble trended series, making aggregate-scale and relative measures vulnerable, while naïve benchmarks and scaled errors such as MASE remain important.Per-step measures such as MAPE and RMSPE may still miss peak points, similarly to seasonal series.Source: §6.2.4 · pp. 45–46

6.2.5. Heteroscedasticity

• Heteroscedasticity is generally less problematic than level non-stationarity, but MAPE, RMSPE, and log-transformed errors can suppress or misrepresent high-variance peaks.Variance-driven peaks and troughs create issues similar to those in seasonal series.Source: §6.2.5 · p. 46

6.2.6. Structural Breaks (with Level Shifts)

• Structural breaks invalidate assumptions of stable scales, distorting WAPE, in-sample-scaled measures, and benchmark-relative scores unless the benchmark and scale remain comparable out of sample.A high or low scaled-error value need not indicate good or bad performance when in-sample benchmark errors poorly represent the forecast horizon.Source: §6.2.6 · pp. 48–50, Figure 17 · p. 48, Figure 18 · p. 49

6.2.7. Intermittent Series

• Intermittent series favor squared-error measures such as RMSE, WRMSPE, NRMSE, RMRSE, and RMSSE because absolute or percentage measures can reward constant-zero forecasts or become undefined.MAPE, sMAPE, and MAAPE mishandle zero actuals, while WAPE is more robust but can still favor zeros; squared-error variants better capture nonzero spikes.Source: §6.2.7 · pp. 50–52
• Rate-based errors compare cumulative mean actual demand with Croston-type forecasts, better matching intermittent-demand inventory decisions than time-step comparisons.MSE and MAE may bias forecasts toward zero in highly sparse series.Source: §6.2.7 · pp. 52–53

6.2.8. Outliers

• Outlier handling depends on business needs: squared errors with means capture outliers, whereas absolute errors, medians, quantiles, or geometric means improve robustness.Anomaly detection can remove unimportant outliers, and geometric means retain tail values unlike medians.Source: §6.2.8 · p. 53
• MAPE and WAPE can be distorted by low or extreme outliers, while MdAPE and bounded sMAPE are more robust; scaling can be improved by excluding anomalies or computing it globally.A single extreme point can dominate WAPE across the horizon, whereas global scaling largely mitigates point-specific effects.Source: §6.2.8 · pp. 53–54, Figure 20 · p. 54
• Relative measures are also vulnerable to outlier-affected benchmarks, but MdRAE, GMRAE, MRAE, and MASE are generally more robust than their mean-, squared-, or RMSSE-based alternatives.Geometric means require excluding zero errors for both benchmark and evaluated forecasts; logarithmic transformations can further de-emphasise very large outlier errors.Source: §6.2.8 · pp. 54–55, Figure 21 · p. 55

Motivation

• Error metrics alone cannot show whether the top-ranked method is significantly better or whether alternatives are interchangeable based on simplicity or computational efficiency.Significance testing distinguishes chance differences on the observed series from differences likely to persist in other samples.Source: §7 · pp. 55–56

Pairwise and multiple testing

• Pairwise tests frame non-significance as the null hypothesis: Diebold–Mariano is time-series-specific and parametric, whereas Wilcoxon is rank-based and non-parametric; Giacomini–White assesses conditional predictive ability.Comparisons of more than two forecasts require multiple-testing corrections such as Bonferroni; Li et al. (2022) addresses conditional superior predictive ability against a benchmark under general serial dependence.Source: §7 · pp. 56–57

Multiple-method tests

• For multiple methods, F-tests and t-tests compare mean errors under parametric assumptions, while Friedman detects significance from method ranks without requiring those assumptions.The Central Limit Theorem supports approximate normality of sample means for sufficiently large samples (n ≥30), but this qualification applies to measures such as MSE and MAE, not RMSE.Source: §7 · p. 57

Post-hoc analysis

• When Friedman rejects equal performance, post-hoc procedures identify which methods differ; Nemenyi uses critical-distance bounds, while Bonferroni-Dunn and Hochberg are preferable for comparisons against one control.CD diagrams display average ranks and connect groups whose differences are not statistically significant.Source: §7 · p. 57

Interpretation and scope

• Significance results depend on ranking granularity, data volume, and the comparison set: many series can make tiny differences significant, whereas many poor methods can enlarge the critical distance and obscure intermediate differences.Reasonable data volume and a sufficiently diverse, reasonable number of models help avoid spurious conclusions about significance.Source: §7 · pp. 57–58, Figure 23 · p. 58

• Because existing evaluation measures can break under particular non-stationarities and established best practices remain lacking, the paper compiles guidelines to support more correct forecasting evaluation.The authors propose combination-based measures analogous to Huber loss as a potential direction for robustly combining MAE and RMSE strengths.Source: §8 · pp. 58–60
• Forecast evaluation should compare models with appropriate simple benchmarks, especially naïve and seasonal naïve methods, rather than relying on forecast-plot appearance.Benchmarks and error measures matter more than visual appeal when judging forecasts.Source: §8 · p. 59
• Avoid data leakage explicitly during rolling-origin evaluation and preprocessing operations including smoothing, decomposition, and normalization.Source: §8 · p. 59
• Use k-fold cross-validation for pure autoregressive models when they do not underfit, particularly for short series, and diagnose underfitting with residual serial-correlation tests.If models underfit, improve them before applying any cross-validation technique.Source: §8 · p. 59
• Use tsCV when sufficient data are available and for continuous-state models such as RNNs and ETS, where temporal order may make it the only applicable validation strategy.Source: §8 · p. 59
• No evaluation measure is universally valid: select measures according to data characteristics, business utility, and choices of error type, aggregation, scaling, and meaningful scale.Evaluate the same forecasts with several measures for sanity checking, but choose one main measure because forecasts cannot be optimized for all measures simultaneously.Source: §8 · pp. 59–60
• Statistical testing should balance diversity among compared models with the number of data points to reduce spurious findings of similarity or difference.Source: §8 · p. 59
• The work highlights methods for handling non-stationarity, including moving-average smoothing and STL decomposition, while formalizing forecast-evaluation guidance for the ML community.The authors argue that insufficient knowledge has led ML studies to struggle to demonstrate competitiveness or reach spurious conclusions.Source: §8 · pp. 59–60