PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating time series forecasting models: An empirical study on performance estimation methods

Vitor Cerqueira, Luis Torgo, Igor Mozetic

arXiv:1905.11744v1cs.LGstat.ML

TL;DR

Reliable performance estimation is difficult for time-series forecasting because observations are dependent and no consensual evaluation approach exists. The paper compares cross-validation and out-of-sample variants across stationary synthetic and real-world series, finding that stationary and non-stationary settings differ: cross-validation is competitive for stationary data, while out-of-sample methods are most accurate in real-world settings.

  • Problem

    Time-series dependence complicates performance estimation, and the appropriate method remains unsettled despite its importance for estimating unseen-data loss and model generalization.

  • Method

    The paper conducts a comparative study of cross-validation and out-of-sample methods using synthetic stationary series and 62 real-world series with potential non-stationarities.

  • Results

    Results differ across scenarios: cross-validation is competitive for stationary time series, whereas repeated holdout and holdout provide the best estimation ability for real-world and non-stationary series.

  • Takeaways & Limitations

    For real-world time series with non-stationary variation, estimates that preserve temporal order are most accurate; cross-validation remains applicable to stationary series.

  • Takeaways & Limitations

    Earlier evidence motivating cross-validation relied mainly on stationary, often artificial series and one-shot out-of-sample evaluation.

Abstract

from arXiv · show

Performance estimation aims at estimating the loss that a predictive model will incur on unseen data. These procedures are part of the pipeline in every machine learning project and are used for assessing the overall generalisation ability of predictive models. In this paper we address the application of these methods to time series forecasting tasks. For independent and identically distributed data the most common approach is cross-validation. However, the dependency among observations in time series raises some caveats about the most appropriate way to estimate performance in this type of data and currently there is no settled way to do so. We compare different variants of cross-validation and of out-of-sample approaches using two case studies: One with 62 real-world time series and another with three synthetic time series. Results show noticeable differences in the performance estimation methods in the two scenarios. In particular, empirical experiments suggest that cross-validation approaches can be applied to stationary time series. However, in real-world scenarios, when different sources of non-stationary variation are at play, the most accurate estimates are produced by out-of-sample methods that preserve the temporal order of observations.

1 Introduction

Performance estimation is essential for judging predictive models, but dependent time-series observations complicate standard cross-validation. The paper compares cross-validation and out-of-sample methods across stationary synthetic and potentially non-stationary real-world series.

  • Performance estimation estimates unseen-data loss and supports model selection, hyper-parameter tuning, and generalization assessment.
  • Cross-validation is typically appropriate for i.i.d. data because it uses available data efficiently, whereas time-series dependence raises caveats.
  • The study compares out-of-sample and cross-validation methods across 62 real-world series and a stationary synthetic environment.
  • Out-of-sample methods hold out later observations, preserving temporal order but using less available data.
  • The paper expands earlier work through a larger experimental setup, stationarity analysis, and a more critical discussion of results.

2 Background

Time-series performance estimation methods preserve temporal structure in different ways, with out-of-sample, prequential, and cross-validation variants offering distinct trade-offs. Earlier evidence favored cross-validation on stationary series, motivating broader comparisons involving real-world non-stationarity.

  • Time-series estimators commonly test models on observations future to those used for training, including out-of-sample and cross-validation variants.
  • Out-of-sample approaches: Out-of-sample holdout trains on an initial period and tests on later data, while repeated windows can provide more robust estimates.
  • Prequential approaches: Prequential methods test each sequential block before incorporating it into training, using growing, sliding, or gap-window variants.
  • Cross-validation approaches: Standard cross-validation randomly shuffles observations into K folds, trains on K-1 folds, and tests on the remaining fold.
  • Cross-validation approaches: Blocked, modified, and hv-blocked cross-validation variants preserve contiguity or remove neighboring observations to address dependence.
  • Prior evidence and motivation: Prior stationary-series studies found cross-validation more accurate than one-shot out-of-sample estimation, but their evidence was limited by evaluation design and artificial stationarity.
  • Prior evidence and motivation: The paper therefore compares multiple cross-validation and out-of-sample variants using real-world and synthetic univariate forecasting data.

2.4 Related work on performance estimation for dependent data

Performance estimation for dependent data remains unsettled across data-stream and spatio-temporal settings. Existing work considers sequential evaluation, blocking, and both cross-validation and out-of-sample approaches.

  • Dependent-data performance estimation is relevant beyond ordinary time series, including data streams and spatio-temporal forecasting.
  • Spatio-temporal data: For spatio-temporal forecasting, prior results suggest both cross-validation and out-of-sample methods can be applicable, with blocking recommended for dependencies.
  • Data streams: Data-stream mining evaluates evolving models as observations arrive, commonly using prequential estimators with forgetting mechanisms.
  • Data streams: The paper’s fixed-dataset setting estimates loss on future unseen observations, unlike sequential analysis with an unfixed sample size.

3 Materials and methods

The study’s materials and methods define the forecasting task, describe the time-series datasets, formalize performance estimation, and specify the experimental design.

  • The methods section defines the prediction task before describing datasets, performance-estimation methodology, and experimental design.

3.1 Predictive task definition

The forecasting task predicts the next time-series value from previous observations using a purely autoregressive model. Time-delay embedding represents each prediction example through the previous p lags.

  • A time series is a temporal sequence Y = {y1, y2, . . . , yt}, where yi is its value at time i and t is its length.
  • Forecasting predicts yt+1 from previous observations using a purely autoregressive modelling approach.
  • Time-delay embedding constructs feature vectors xi ∈ X ⊂ Rp from the previous p lags and targets yi ∈ Y ⊂ R.
  • The regression objective is to construct a model f : X → Y that maps lag-based feature vectors to predicted values.
  • The first embedded row uses {y1, . . . , yp} to predict yp+1, assuming no time dependencies larger than p.

3.2 Time series data

The study uses real-world and synthetic time series to examine stationarity and forecasting-estimation behavior. Real-world series vary across domains and may require differencing, while the synthetic processes are stationary.

  • Two case studies comprise real-world time series and a synthetic setting previously used to study performance estimation for forecasting.
  • The real-world study analyzes 62 series from different domains with varied granularity, length, and unknown dynamics.
  • Differencing removes changes in level to stabilize the mean and accounts for trend and seasonality; I records differences needed for trend-stationarity.
  • The wavelet spectrum test detects non-stationarity through large Haar-wavelet coefficients assessed with multiple-hypothesis corrections.
  • The synthetic data-generating processes are stationary and include a lag-3 autoregressive process, lag-1 moving-average process, and seasonal autoregressive process.
  • Each synthetic use case has 1000 Monte Carlo simulations, with a time series of 200 values generated per repetition.

3.3 Performance estimation methodology

The methodology compares performance-estimation procedures by separating estimation from validation. Estimates produced on an estimation set are evaluated against loss on subsequent unseen observations.

  • Performance estimation uses data to estimate the loss a model will incur on unseen data, supporting model comparison and hyper-parameter tuning.
  • Training and evaluating on the same data is biased by overfitting, motivating procedures that split available data into training and test sets.
  • Standard cross-validation efficiently uses data but assumes independent observations, creating theoretical problems for dependent time-series data.
  • The study splits each series into an estimation part and a subsequent validation part, using the latter to compute the model’s true loss on unseen observations.
  • Within the estimation set, each procedure splits training and test data, fits the model on training data, and computes an estimated loss on the test data.
  • The objective is for each estimate ĝi to approximate the ground-truth loss Lm as closely as possible.

3.4 Experimental design

The experimental design compares cross-validation and out-of-sample estimators across forecasting settings. It evaluates estimated losses against validation-set losses using APAE and PAE, with RMSE measuring forecasting accuracy.

  • The experiments ask how cross-validation estimates compare with out-of-sample estimates for time-series forecasting tasks.
  • The study hypothesizes that preserving temporal order is important because real-world series may contain complex non-stationary variation and future-related cues.
  • False Nearest Neighbours estimates each series’ embedding dimension, while the synthetic study fixes the dimension at 5.
  • Each series uses its first 70% as the estimation set and the subsequent 30% as the validation period.
  • Eleven estimation methods are divided into cross-validation variants and out-of-sample approaches, including randomized, blocked, modified, and hv-blocked cross-validation.
  • Out-of-sample methods include Holdout and Rep-Holdout, while prequential methods use growing, sliding, or gap-block strategies.
  • APAE measures estimation-error size, whereas PAE measures whether the estimate underestimates or overestimates the true error.
  • Forecasting accuracy is evaluated with RMSE, and Cubist is the primary regression system, with lasso and random forest producing similar conclusions.

4 Empirical experiments

Experiments reveal different estimation-method rankings across synthetic and real-world time series. Cross-validation performs well in stationary settings, while out-of-sample methods are strongest for non-stationary real-world series.

  • Synthetic case study: Cross-validation approaches generally outperform simple Holdout across synthetic scenarios S1, S2, and S3.
  • Synthetic case study: Preq-Bls-Gap achieves the best average rank in S1, while Rep-Holdout has a better rank than cross-validation but high variability.
  • Synthetic case study: CV-hvBl has the best average rank in S2, whereas prequential variants perform worst and Rep-Holdout has a large standard deviation.
  • Synthetic case study: S3 produces less differentiated results, with methods showing closer average ranks and large standard deviations.
  • Real-world case study: Rep-Holdout and Holdout rank best on the real-world case study, while CV-Bl is strongest among cross-validation procedures.
  • Real-world case study: Cross-validation tends to underestimate error, whereas prequential and out-of-sample variants tend to overestimate it.
  • Controlling for stationarity: For stationary real-world series, CV ranks best; for non-stationary series, CV is among the worst and Holdout and Rep-Holdout rank best.
  • Descriptive model: The descriptive tree is dominated by Rep-Holdout and Holdout leaves, with Rep-Holdout as the root-node method across most of the 62 series.

5 Discussion

The discussion contrasts synthetic stationary findings with complex real-world data and examines practical trade-offs. It argues that temporal-order-preserving methods are generally preferable in non-stationary settings, despite using less data.

  • Synthetic stationary results do not fully reflect real-world results, where Rep-Holdout generally provides the most accurate performance estimates.
  • For stationary time series, CV remains competitive with Rep-Holdout.
  • Real-world time series contain complex structures and non-stationary variations, motivating Rep-Holdout over cross-validation in those scenarios.
  • Out-of-sample methods preserve temporal order but leave less data available for performance estimation, potentially disadvantaging small data sets.
  • Varying synthetic series length from 100 to 3000 observations produced no evidence that size noticeably affected estimation error.
  • The study focuses on univariate numeric series, next-value prediction with immediate feedback, and half-hourly, hourly, or daily sampling.

6 Final remarks

The paper addresses the lack of a settled performance-estimation approach for time-series forecasting by comparing out-of-sample and cross-validation methods. Its conclusions depend strongly on stationarity.

  • The study compares out-of-sample and cross-validation methods across synthetic stationary and real-world potentially non-stationary case studies.
  • Cross-validation variants are competitive for stationary series but systematically estimate worse than out-of-sample approaches when non-stationarity is present.
  • For real-world series with potential non-stationarities, the paper recommends methods preserving temporal order, particularly Rep-Holdout across multiple testing periods.

Appendix

The appendix captions summarize the data-set metadata and its continuation.

  • Table 1 summarizes the time-series data sets and defines p, I, and S as embedding dimension, differencing count, and detrended stationarity indicator.
  • Table 2 provides the continuation of Table 1.
Loading 1905.11744v1…