PaperlayerView exact revision on arXiv

Source-linked AI summary

Are We Really Benchmarking Forecasting Models? The Impact of Preprocessing on Time Series Performance

Guilherme Afonso Galindo Padilha, Paulo Salgado Gomes de Mattos Neto, Rafael Menelau Oliveira e Cruz

arXiv:2610.09096v1cs.LG

TL;DR

Benchmark comparisons shape research progress and adoption, while forecasting performance depends on preprocessing as well as architecture. This paper evaluates models across preprocessing pipelines on M4 series. Fixed global pipelines favor models with internalized preprocessing, whereas per-series selection yields larger gains for models without it.

  • Problem

    Benchmark comparisons shape perceived research progress and adoption, while forecasting performance depends on preprocessing as well as model architecture.

    Source: §1 · p. 1

  • Method

    The paper benchmarks 11 forecasting algorithms across 16 reversible preprocessing pipelines on around 29,000 M4 series.

    Source: §1 · p. 3

  • Results

    Four of five non-IP-Models improve by more than 50% under Oracle per-series selection, while fixed global pipelines favor IP-Models.

    Source: §5 · p. 7, §6 · p. 10

  • Takeaways & Limitations

    The authors conclude that forecasting benchmarks should treat preprocessing as an explicit evaluation dimension, and PATM supports future meta-models that recommend pipelines from series metafeatures.

    Source: §6 · p. 10, A LIMITATIONS AND FUTURE WORKS · p. 13

    The benchmark is limited to univariate time series, so whether its conclusions remain stable with multivariate inputs or covariates remains for future work.

    Source: A LIMITATIONS AND FUTURE WORKS · p. 13

Abstract

from arXiv · show

While established literature underscores the pivotal role of preprocessing in forecasting accuracy, this stage remains largely overlooked in current research. Modern benchmarks typically resort to simple scaling, failing to account for critical transformations required to address nonstationarity, such as differencing. This omission creates a significant structural preprocessing bias that favors models with built-in data treatments while obscuring the true potential of simpler architectures. We study this effect through a preprocessing-aware benchmark that evaluates 11 forecasting models across 16 reversible preprocessing pipelines on 29,000 M4 time series. Our results identify preprocessing as a key driver of forecasting performance. Optimizing preprocessing per series yields gains of approximately 27\% to 87\% across all evaluated models, with architectures lacking internalized preprocessing experiencing the most substantial improvements. This allows simpler architectures to become highly competitive with complex, state-of-the-art models in modern forecasting benchmarks. All resources and experimental results from this benchmark are stored in a comprehensive metadataset to support future metalearning tasks.

1 INTRODUCTION

Forecasting benchmarks shape perceptions of progress and adoption, but performance depends on preprocessing as well as model architecture. The paper proposes a preprocessing-aware benchmark to examine how pipelines affect model performance and rankings.

1 INTRODUCTION

  • Without preprocessing, forecasts in the examples deteriorate under trend, changing variance, random-walk behavior, seasonality, and interacting nonstationarity.The figure is illustrative rather than a general proof; both models track the stationary example well.Source: Figure 1 · p. 2, §1 · p. 2
  • Many benchmarks report preprocessing only partially or use minimal transformations, leaving nonstationarity-focused pipelines outside systematic evaluation.Table 1 summarizes the benchmark landscape.Source: §1 · p. 2, Table 1 · p. 2
  • Models with internalized preprocessing, such as ARIMA with differencing or PatchTST with RevIN, may have preprocessing capacity unavailable to other models.The paper calls these Internalized Preprocessing Models (IP-Models).Source: §1 · p. 3
  • Structural preprocessing bias arises when benchmarks give models unequal access to preprocessing capacity, making rankings reflect more than architectural merit.The paper notes that this does not invalidate existing benchmarks, but limits interpreting their leaderboards as architecture-only comparisons.Source: §1 · p. 3
  • The benchmark evaluates 11 forecasting algorithms with 16 reversible preprocessing pipelines on around 29,000 M4 series.The algorithms span statistical, machine-learning, and transformer-based methods.Source: §1 · p. 3

2 TIME SERIES PRELIMINARIES

The preliminaries describe time-series structure and preprocessing transformations, then distinguish models that embed preprocessing from those using external transformations. The selected transformations are invertible so forecasts can be returned to the original scale.

2.1 Time Series Characteristics

  • Time series are ordered observations indexed by time, and descriptors such as trend, seasonality, and stationarity summarize their temporal structure.The study uses such descriptors to compare series structures.Source: §2 · p. 3

2.2 Time Series Preprocessing

  • Preprocessing often aims to achieve weak stationarity, for example by differencing a unit-root series to improve suitability for models that assume stationarity.The paper groups transformations into variance-stabilizing, difference-based unit-root removal, and scaling; the selected techniques are invertible.Source: §2 · p. 4

Internalized-Preprocessing Models

  • An Internalized-Preprocessing Model embeds a preprocessing pipeline within the model’s computational flow before its core forecasting algorithm.The embedded transformation may address nonstationarity; this differs from an external step and from structural features such as patching or convolutions.Source: §2 · p. 4

3 PREPROCESSING-AWARE TIME SERIES BENCHMARK

The benchmark crosses a formalized preprocessing search space with forecasting algorithms and records experiment metadata and results in the Preprocessing-Aware Time Series Metadataset. Series descriptors provide context for analyzing how preprocessing and data characteristics relate to model performance.

3 PREPROCESSING-AWARE TIME SERIES BENCHMARK

  • The benchmark’s metadata and results are saved in the Preprocessing-Aware Time Series Metadataset (PATM).The framework is shown in Figure 2.Source: §3 · p. 4
  • Each dataset is contextualized with descriptive features representing structural properties of its time series.These descriptors support analysis of how series characteristics interact with preprocessing and model success.Source: §3 · p. 4
  • The benchmark formalizes four preprocessing categories: variance stabilization, unit-root removal, seasonality removal, and scaling.The pipeline space is their Cartesian product, and each dataset is processed through every pipeline.Source: §3 · p. 4
  • The benchmark executes a standardized training and inference protocol for forecasting algorithms across preprocessed dataset variants.The resulting metadataset records performance metrics, cost metrics, and configuration information for each experiment.Source: §3 · p. 5

4 EXPERIMENTAL PROTOCOL

The experimental protocol uses selected univariate M4 series, 16 combinations of established preprocessing operations, and 11 forecasting algorithms with fixed hyperparameters and lag settings. It records five performance metrics and reports MASE as the primary metric.

4 EXPERIMENTAL PROTOCOL

  • The evaluation uses univariate quarterly, weekly, daily, and hourly M4 series, excluding yearly series because they lack a seasonal period.The univariate setting is intended to isolate preprocessing effects; M4 provides series from diverse domains and temporal granularities.Source: §4 · p. 5
  • The grid evaluates 16 pipelines combining none or Box-Cox variance stabilization, none or differencing, none or seasonal differencing, and none or z-score scaling.Operations are applied in the order variance stabilization, seasonal removal, unit-root removal, then scaling.Source: §4 · p. 5
  • The 11 algorithms span statistical, machine-learning, deep-learning, transformer-based, and synthetic-pretrained foundation-model approaches.Hyperparameters and lag windows are fixed from established benchmark practices to isolate preprocessing effects.Source: §4 · pp. 5–6
  • Five metrics are recorded, with MASE reported as primary because it is scale-independent across series of differing magnitudes.The other metrics are MSE, RMSE, MAE, and sMAPE.Source: §4 · p. 6

5 RESULTS

Per-series preprocessing substantially changes forecast performance, especially for models without internalized preprocessing, and can alter relative model standings. Preprocessing preferences also vary by model and series properties, while optimized pipelines can improve accuracy with little added training time.

Fixed preprocessing

  • A single global pipeline can mismatch series properties; combining variance stabilization with unit-root and seasonal differencing can make errors explode on non-seasonal series.Over-differencing can amplify noise, and inverse Box-Cox transformation can exponentially amplify erratic forecasts when predictions return to the original scale.Source: §5 · p. 6
  • Under no preprocessing and standard scaling, IP-Models achieve lower MASE than all non-IP-Models, while LR, XGBoost, and Informer improve substantially under differencing.The best IP-Model results remain concentrated around simple or internally compatible pipelines, whereas non-IP-Models require more specific choices.Source: §5 · p. 6, Table 3 · p. 7

Per-series preprocessing

  • Four of five non-IP-Models improve by more than 50% under oracle preprocessing, with MLP and Informer improving by over 75%.IP-Models show oracle gains of approximately 35% to 55%; in some cases, non-IP-Models close the performance gap.Source: §5 · p. 7, Figure 3 · p. 7

Model rankings

  • Preprocessing-dependent rankings shift favorably for several non-IP-Models under the oracle setting, although the comparison omits model-specific hyperparameter search.Table 4 compares models under the same preprocessing configuration; under the commonly used Q pipeline, IP-Models dominate the best ranks.Source: §5 · p. 8, Table 4 · p. 8

Preprocessing preferences

  • Under high trend or non-stationarity, non-IP-Models often favor first-order or seasonal differencing, while several IP-Models retain simpler preferences.Preferences also vary among non-IP-Models: U benefits LR but is extremely harmful for N-BEATS; Table 5 reports best pipelines across data characteristics.Source: §5 · p. 8, Table 5 · p. 9

Cost and performance

  • Optimal preprocessing makes classical models such as LR and THETA, and simpler machine-learning models, highly competitive with a fraction of deep-learning models’ computational cost.The reported performance boost comes with virtually no practical increase in training time; Figure 5 relates mean MASE to mean training time and peak GPU memory.Source: §5 · p. 9, Figure 5 · p. 9

6 CONCLUSION

The conclusion reports that preprocessing strongly shapes forecasting performance and model rankings, with global pipelines favoring models that internalize preprocessing. Per-series preprocessing benefits models without those mechanisms, and the PATM metadataset supports further analysis.

6 CONCLUSION

  • Global pipelines favor models with internalized preprocessing, while per-series preprocessing benefits models without those mechanisms more.The authors call this unequal access to preprocessing capacity structural preprocessing bias.Source: §6 · p. 10
  • Preprocessing preferences depend on both model class and series properties, with trend and seasonality shifting non-IP-Models toward differencing-based pipelines.The conclusion recommends treating preprocessing as an explicit benchmark dimension.Source: §6 · p. 10
  • The authors release the PATM metadataset to support study of preprocessing sensitivity and development of adaptive preprocessing selection methods.The conclusion describes the resource as a basis for future work.Source: §6 · p. 10

A LIMITATIONS AND FUTURE WORKS

The benchmark controls its preprocessing search space, series type, and model tuning to isolate preprocessing effects. These choices constrain coverage and mean the reported rankings are not a fully optimized leaderboard.

A LIMITATIONS AND FUTURE WORKS

  • The search covers a small set of established, interpretable, reversible transformations, excluding methods such as moving-average smoothing, learned normalization, and decomposition-based preprocessing.The authors propose expanding the set while distinguishing reversible, lossy, and representation-changing transformations.Source: A LIMITATIONS AND FUTURE WORKS · p. 13
  • All experiments use univariate series, so the findings do not establish whether structural preprocessing bias remains stable with multivariate inputs or covariates.The authors also identify long-term forecasting as a useful extension.Source: A LIMITATIONS AND FUTURE WORKS · p. 13
  • Largely fixed model hyperparameters isolate preprocessing effects, but the resulting rankings are not a fully optimized leaderboard.Tuning could change model rankings or a model's best preprocessing pipeline.Source: A LIMITATIONS AND FUTURE WORKS · p. 13
  • PATM provides millions of evaluations that can support meta-models recommending preprocessing from time-series metafeatures.The authors describe this as groundwork for future meta-learning and automated forecasting pipelines.Source: A LIMITATIONS AND FUTURE WORKS · p. 13

B BENCHMARK LITERATURE REVIEW

The literature review identifies incomplete and inconsistent preprocessing as a limitation of forecasting benchmarks. The accompanying feature and preprocessing descriptions specify how series properties are represented and how reversible transformations target them.

B BENCHMARK LITERATURE REVIEW

  • Many forecasting benchmarks lack a comprehensive, standardized preprocessing pipeline, complicating fair model comparisons and allowing undisclosed preprocessing to cloud architectural comparisons.Prior work explores models and modules, but not systematically how external nonstationarity preprocessing affects model categories.Source: B BENCHMARK LITERATURE REVIEW · p. 14

C TIME SERIES FEATURES

  • The study extracts 40 time-series features with tsfeatures, including autocorrelation, differenced-series autocorrelation, stationarity tests, trend, seasonality, and distribution descriptors.The feature set extends Monash archive metadata with descriptors including seasonal strength, ADF p-value, and skewness.Source: C TIME SERIES FEATURES · pp. 14–15
  • STL decomposition separates trend, seasonality, and remainder; trend and seasonal strengths above 0.75 are high, while values below 0.25 are low.Trend describes long-term movement, while seasonality captures patterns recurring at a fixed period.Source: C TIME SERIES FEATURES · p. 15
  • Weak stationarity requires a constant mean and lag-dependent covariance, and the study uses the ADF test to assess it.The paper treats weak stationarity as generally sufficient for standard time-series modeling.Source: C TIME SERIES FEATURES · p. 16
  • Spectral entropy above 0.75 marks high complexity, while values below 0.25 mark low complexity; shifting features summarize abrupt changes in level, variance, or distribution.The shifting measures aggregate maximum divergence and level and variance shifts across rolling windows.Source: C TIME SERIES FEATURES · p. 16

D CATEGORIES OF PREPROCESSING

  • The benchmark sequences variance stabilization, seasonal removal, unit-root removal, and scaling; its experimental groups isolate nonstationarity components and their removal.Operations follow a fixed mathematical order, including testing scaling alone versus variance stabilization combined with differencing.Source: D CATEGORIES OF PREPROCESSING · p. 16
  • Box-Cox transforms target variance that changes with the series level; λ = 0 gives the natural logarithm and constrains forecasts to positive original-scale values.Varying λ addresses a range of heteroscedastic behaviors.Source: D CATEGORIES OF PREPROCESSING · p. 16
  • First differences target level changes and linear trends, while seasonal differences can remove seasonal unit roots; differenced-scale forecasts are reintegrated by summation.The stated stationarity effect of first differencing applies under standard conditions.Source: D CATEGORIES OF PREPROCESSING · pp. 16–17

E HARDWARE SPECIFICATIONS

The experiments use specified server hardware and fixed model settings to isolate preprocessing effects. Statistical models use automated selection, while machine-learning baselines follow standard forecasting configurations.

E HARDWARE SPECIFICATIONS

  • Experiments ran on an Intel Xeon Silver 4310 server with 512GB RAM and an NVIDIA RTX A6000 GPU.Classical models used statsforecast; machine-learning and deep-learning models used scikit-learn, XGBoost, and PyTorch.Source: E HARDWARE SPECIFICATIONS · p. 17

F MODEL HYPERPARAMETERS AND TRAINING CONFIGURATIONS

  • Fixed hyperparameters and lookback settings isolate preprocessing as the independent variable, but the authors note that joint tuning of these settings remains future work.Lookback windows follow Monash specifications and align with temporal frequency and dominant seasonality.Source: F MODEL HYPERPARAMETERS AND TRAINING… · p. 17

F.1 STATISTICAL MODELS

  • ETS, ARIMA, and Theta use automated procedures to select their respective components, lag orders, and smoothing parameters.The procedures are implemented with statsforecast and seasonal-strength detection.Source: §F.1 · p. 17

F.2 MACHINE LEARNING BASELINES

  • Linear Regression and XGBoost are configured following standard forecasting benchmarks.The cited table is titled “Hyperparameters for Machine Learning Baselines.”Source: §F.2 · p. 18, Table 7 · p. 18

F.3 DEEP LEARNING AND TRANSFORMER ARCHITECTURES

The evaluation aligns neural-model capacity broadly, follows original recommended configurations for transformer baselines, and compares global training with local runs that isolate per-series preprocessing effects. It defines MASE and describes the evaluation metrics used to assess forecast accuracy.

F.3 DEEP LEARNING AND TRANSFORMER ARCHITECTURES

  • Neural models are configured with roughly aligned capacity, except transformer baselines, which follow configurations recommended in their original publications.PatchTST and Informer use standard configurations from their original implementations.Source: §F.3 · p. 18, Table 8 · p. 18, §F.4 · p. 18

G LOCAL VS. GLOBAL MODELS

  • Global training is the primary setup for machine-learning and deep-learning models, while local runs isolate per-series preprocessing effects and yield fundamentally similar result patterns.Local models train on one series; global models train jointly across a collection of series. Figure 6 and Table 10 report local-model results.Source: G LOCAL VS. GLOBAL MODELS · pp. 18–19, Figure 6 · p. 19, Table 10 · p. 19

H EVALUATION METRICS

  • The evaluation metrics capture different aspects of predictive accuracy, including sensitivity to large deviations, scale dependence, and robustness across series magnitudes.The notation defines y_i as the true value, ŷ_i as the prediction, and N as the number of evaluated predictions.Source: H EVALUATION METRICS · p. 19

H.1 MEAN ABSOLUTE SCALED ERROR (MASE)

  • MASE measures absolute forecast error relative to a naive baseline, using a seasonal scaling term based on the training sequence.When m = 1, the denominator is the average one-step naive forecasting error; ϵ is a small positive constant for numerical stability.Source: §H.1 · p. 20

H.2 MEAN SQUARED ERROR (MSE)

This section defines four error metrics used to evaluate forecasts. Their accompanying tables report model performance across preprocessing pipelines.

H.2 MEAN SQUARED ERROR (MSE)

  • MSE measures average squared prediction deviation and penalizes large errors more heavily; lower MSE indicates better accuracy.Table 11 reports MSE across preprocessing pipelines and models.Source: §H.2 · p. 20, Table 11 · p. 20

H.3 ROOT MEAN SQUARED ERROR (RMSE)

  • RMSE is the square root of MSE, expressed in the target variable’s units, and it assigns relatively large penalties to large errors.Table 12 reports RMSE across preprocessing pipelines and models.Source: §H.3 · p. 20, Table 12 · p. 21

H.4 MEAN ABSOLUTE ERROR (MAE)

  • MAE measures average absolute prediction deviation and is less sensitive to extreme errors than MSE or RMSE.Lower MAE indicates better forecasting performance; Table 13 reports it across preprocessing pipelines and models.Source: §H.4 · p. 21, Table 13 · p. 21

H.5 SYMMETRIC MEAN ABSOLUTE PERCENTAGE ERROR (SMAPE)

  • sMAPE measures prediction error relative to the magnitudes of true and predicted values and is scale-independent, commonly reported as a percentage.A small positive ϵ avoids division by zero; lower sMAPE indicates better accuracy, and Table 14 reports it across pipelines and models.Source: §H.5 · p. 22, Table 14 · p. 22

I COMPLEMENTARY RESULTS

Across complementary metrics, the patterns distinguish robust models from those more sensitive to preprocessing. The section also examines preprocessing preferences and costs, then describes the metadataset organized to support further analyses.

I COMPLEMENTARY RESULTS

  • Across MSE, RMSE, MAE, and sMAPE, IP-Models are typically strongest with minimal preprocessing or simple scaling, while non-IP-Models generally prefer differencing-based pipelines.The pattern holds for both scale-dependent and scale-independent metrics.Source: I COMPLEMENTARY RESULTS · p. 22

I.1 GRANULAR PREPROCESSING PREFERENCES; I.3 PER-MODEL RANKING ANALYSIS

  • IP-Models often remain competitive across simple pipelines, whereas non-IP-Models show concentrated preferences that shift toward differencing under high trend or nonstationarity.Across entropy conditions, models tend toward simpler pipelines in high-entropy series and can leverage complex pipelines more successfully in low-entropy series; rank frequencies also vary by model.Source: §I.1 · p. 23, §I.3 · p. 23, Figure 7 · p. 24

I.2 NO PIPELINE X ORACLE DELTAS

  • Models lacking internal preprocessing mechanisms show the largest relative gains from No Pipeline to Oracle, while IP-Models show comparatively smaller gains.Table 15 reports MASE for these configurations and percentage gain; the gains vary substantially across time-series domains.Source: §I.2 · p. 23, Table 15 · p. 23

I.4 QUANTIFYING COST-RELATED TRADE-OFFS

  • Training-time differences between Oracle and No Pipeline are marginal for local models and negligible for almost all global models, whose joint training shares cross-series information.Figure 9 plots global-model mean MASE against mean training time, with marker size indicating peak GPU memory.Source: §I.4 · pp. 24–25, Table 16 · p. 25, Table 17 · p. 25

J PATM; J.1 FILE ORGANIZATION

  • PATM records represent time-series, preprocessing-pipeline, and model experiment tuples, with separate frequency-specific files for experiment results and series features.Separating the files avoids repeating series features that remain invariant across models and pipelines.Source: J PATM · p. 25, §J.1 · p. 25

J.2 PATM SCHEMA; J.3 EXAMPLE RECORD

  • The PATM schema captures predictive performance and computational cost for multi-objective evaluation, and its example record includes a PatchTST forecast using first-order differencing.Table 18 summarizes the schema; the example identifies a series, model, pipeline, metrics, forecast, time, and GPU memory.Source: §J.2 · p. 26, Table 18 · p. 26, §J.3 · p. 26

J.4 FEATURE FILES

  • Series-feature files can be joined to PATM records by series_id, supporting analyses of how series characteristics relate to preprocessing effectiveness and model performance.The resulting relational structure supports meta-learning, preprocessing recommendation, and performance modeling.Source: §J.4 · pp. 26–27
Loading 2610.09096v1…