PaperlayerView exact revision on arXiv

Source-linked AI summary

How Good Can Linear Models Be for Time-Series Forecasting?

Lang Huang, Jinglue Xu, Luke Darlow

arXiv:2606.27282v2cs.LG

TL;DR

Standard forecasting evaluations often fix preprocessing, leaving unclear how competitive simple linear models can be when context and normalization are tuned. This paper searches those choices for Ridge regression across eight benchmarks, finding it achieves the best average MSE on seven datasets and beats nonlinear baselines on six.

  • Problem

    Fixed preprocessing in standard evaluations can make linear forecasters appear weaker, leaving their competitiveness under tuned input representations insufficiently assessed.

  • Method

    The paper searches Ridge regression’s context length, local normalization, regularization, and augmentation per horizon and series across eight benchmarks.

  • Results

    Best average MSE is achieved on seven of eight datasets, while nonlinear baselines are beaten on six of eight benchmarks.

  • Takeaways & Limitations

    Optimized hyperparameters diagnose dataset structure through series-specific lookback scaling, local normalization preferences, and dataset-specific cross-series sharing.

  • Takeaways & Limitations

    The findings are limited to standard numeric long-horizon benchmarks and Ridge with searched preprocessing, while nonlinear comparisons use published configurations.

Abstract

from arXiv · show

Time-series forecasting research has been moving steadily toward larger architectures, from specialized transformers to general-purpose foundation models, on the assumption that capacity is what unlocks accuracy. We take the opposite position: most of the gap can be closed at far lower cost by tuning preprocessing rather than scaling models. We use Ridge regression as the testbed, since it has a closed-form solution and interpretable weights, which let the optimal hyperparameters be read off the search directly. We search over context length, local normalization, regularization, and augmentation on eight standard benchmarks and find three patterns. (1) Optimal lookback is strongly series-specific and often non-monotonic in forecast horizon, with fitted power-law exponents ranging from $+0.46$ on ETTm2 to $-0.19$ on Exchange and Traffic, challenging the convention that longer horizons need longer history. (2) Normalizing over a learned trailing fraction of the context, rather than its entirety, is almost universally preferred. (3) Series within the same dataset often disagree on hyperparameters; the optimal degree of cross-series sharing varies from fully shared to fully per-series. The resulting models beat prior linear forecasters on most dataset-horizon entries and exceed Transformer, MLP, and CNN baselines on six of eight benchmarks. The optimized hyperparameters also serve as a diagnostic on the data itself, revealing structures that larger models absorb silently into their learned parameters. We provide an accompanying interactive online demonstration and the code at https://sakanaai.github.io/SearchCast/.

1 Introduction

The paper argues that preprocessing choices, rather than larger architectures, explain much of time-series forecasting performance. A systematic Ridge search reveals dataset-specific lookback, normalization, and model-comparison patterns while using transparent, low-cost models.

  • Motivation: Preprocessing choices can systematically disadvantage low-capacity models when context length, normalization, and augmentation remain fixed across benchmarks.The paper argues that limited models cannot absorb suboptimal input representations through learned parameters as effectively as larger architectures.
  • Key observations: Optimal lookback is dataset-specific and often non-monotonic in forecast horizon, with power-law exponents from +0.46 on ETTm2 to −0.19 on Exchange and Traffic.This contradicts the assumption that longer horizons require longer history.
  • Contributions: Carefully tuned Ridge regression outperforms prior linear forecasters across most dataset-horizon entries and matches or exceeds Transformer and MLP architectures on six of eight benchmarks.The paper describes Ridge as orders of magnitude cheaper to train.
  • Contributions: The optimal hyperparameter landscape of a transparent linear model encodes structural properties of time-series data, including scaling behavior and normalization preferences.The paper presents hyperparameter tuning as a way to expose data structure rather than leaving it absorbed silently in learned parameters.
  • Key observations: Adapting context per horizon yields up to +16% MSE improvement over a global baseline across four datasets.Median optimal lookback varies by series and horizon.

2 Related Work

Prior work spans transformer forecasters, simpler MLP/CNN architectures, linear models, and zero-shot time-series foundation models. This work builds on linear-model unification by fixing Ridge regression and optimizing preprocessing across horizons and series on eight benchmarks.

  • Transformer-based forecasters: Transformer forecasters use frequency-domain operators, patching, or inverted attention to capture seasonal structure, channel-wise patterns, and cross-channel dependencies.PatchTST remains the strongest transformer baseline on long-term benchmarks.
  • MLP and CNN alternatives: MLP and CNN alternatives include 2D convolutional reshaping, encoder-decoder MLPs, temporal-resolution mixing, and attention-free mixing.TSMixer reaches competitive accuracy with an attention-free MLP mixer.
  • Linear models: Linear-model extensions combine trend-seasonal decomposition, normalization, reversible instance normalization, or parameter sparsity, while related variants are functionally equivalent to unconstrained linear regression over augmented features.DLinear, NLinear, RLinear, and SparseTSF exemplify these extensions.
  • Positioning: Rather than proposing another linear variant, the paper fixes Ridge regression and searches context length, local normalization, regularization, and augmentation per horizon and series across eight benchmarks.The optimized hyperparameters are also used diagnostically to study how optimal context scales with horizon and whether normalization should use a trailing context fraction.

3 Method

The method uses channel-independent Ridge regression with a closed-form solution, making large-scale hyperparameter search efficient. Its modeling flexibility comes primarily from jointly tuning preprocessing, regularization, and the granularity of sharing across forecast horizons and series.

  • Ridge regression: Ridge regression maps an L-step context to an H-step forecast with weights obtained in closed form, avoiding iterative optimization.Each variate is modeled independently, and the closed-form solution enables effectively instantaneous training on a single GPU.
  • Preprocessing search: The search tunes four preprocessing axes: normalization, context length, augmentation, and regularization.Normalization includes global or local statistics and standardization or robust scaling; augmentation may use time- or frequency-domain Gaussian perturbations, or none.
  • Preprocessing search: Context length L is searched logarithmically from 32 to 2048, subject to caps imposed by data size and forecast horizon.Regularization evaluates 21 logarithmically spaced α values from 10^-6 to 10^3 and selects the value minimizing validation loss.
  • Search granularity: Horizon grouping shares hyperparameters within contiguous blocks of size g_h, spanning fully per-step tuning at g_h = 1 to the global baseline at g_h = H.This design interpolates between homogeneous tuning and independent horizon-step searches without requiring a separate search for every step.
  • Search granularity: The pair (g_h, g_s) partitions the horizon×series space into independent Optuna search cells, with chronological 3-fold expanding-window cross-validation.The global and fully local models are special cases, and the held-out test set is excluded from hyperparameter search.

4 Experiments

Experiments evaluate SearchCast on eight multivariate benchmarks using chronological splits, fixed-sample validation across searched contexts, and comparisons with linear and nonlinear baselines. SearchCast achieves the best average MSE among linear models on seven datasets and ties FITS on ETTh2, with reductions over OLS reaching 14.8%.

  • Datasets: Eight benchmarks span electricity temperature, weather, electricity consumption, traffic, and exchange-rate series across hourly, 15-minute, and 10-minute granularities.ETTh1, ETTh2, ETTm1, ETTm2, Weather, Electricity, Traffic, and Exchange are evaluated.
  • Datasets: Datasets use chronological training, validation, and test splits of 6:2:2 for ETT and 7:1:2 for the other benchmarks.This follows the stated standard protocol.
  • Evaluation Protocol: Evaluation reports MSE at horizons H ∈{96, 192, 336, 720}, using 20 Optuna trials per cell and 3-fold cross-validation.Validation folds add pre-validation history for candidate contexts while keeping validation target points and intervals fixed.
  • Baselines: SearchCast is compared with linear OLS, FITS, and DLinear and nonlinear PatchTST, iTransformer, TimeMixer, TimesNet, and Autoformer baselines.The linear comparisons isolate preprocessing-search gains from model capacity, while nonlinear baselines span Transformer, MLP, and CNN families.
  • Results: 7 of 8 datasets have the best average MSE within the linear class, while SearchCast ties FITS on ETTh2 at 0.334 vs. 0.333.The reported reductions over OLS reach 4.8% on ETTm2, 4.3% on ETTh1, and 14.8% on Exchange.

5 Analysis

The search reveals that forecasting behavior is strongly data- and series-dependent: optimal context, sharing, normalization, and augmentation vary across datasets, horizons, and channels. These choices materially alter predictions and weights while exposing interpretable temporal structure.

  • 5.1 Lookback versus horizon: Optimal lookback scales in opposite directions across datasets, with exponents from +0.46 on ETTm2 to −0.19 on Exchange and Traffic.ETTm2 strongly favors more history for longer horizons, while Weather and Electricity are nearly flat and Exchange and Traffic favor shorter contexts.
  • 5.1 Lookback versus horizon: Per-series exponents vary substantially within datasets, so a single shared lookback can be insufficient even when aggregate trends appear mild.ETTh2, Electricity, and Traffic contain channels that prefer both growing and shrinking context as horizon increases; Weather is the tightest cluster.
  • 5.2 Hyperparameter sharing: The best degree of series sharing is dataset-specific: fully shared settings win on both ETT benchmarks, whereas fully per-series settings win on Weather with up to 10% degradation at gs = 21.On the ETT benchmarks, per-series search degrades MSE by 0.5–4%, suggesting related variates can regularize one another.
  • 5.3 Forecasts and weights: Tuning prevents Ridge from drifting toward the mean at long horizons, while selected weights reveal recent-memory, calendar, lag-0/1, and phase-matched historical structures.Exchange concentrates weight on recent lags; Traffic shows weekly bands, ETTh2 daily structure, Electricity mixes near and remote lags, and Weather and ETTm often use distant lags.
  • 5.3 Forecasts and weights: Horizon grouping changes MSE by at most 0.4% but can create abrupt weight changes at gh = 48 bin boundaries.Grouping nearby horizons preserves separate tuning near the reported evaluation cutoffs while changing the weights abruptly at shared-setting boundaries.
  • 5.4–5.6 Preprocessing choices: Local normalization is selected in 62–100% of dataset–horizon cells, and its optimal trailing fraction is almost always below full-window normalization.The learned ratio r restricts statistics to the last r · L steps, with r = 1 recovering full-window normalization; Weather and Exchange also show wider optimal augmentation-noise spreads than several other datasets.

6 Conclusion

The conclusion argues that linear models appear weak mainly because of undertuned preprocessing, not insufficient capacity. A carefully tuned Ridge regression is competitive across standard benchmarks while also exposing useful properties of the data.

  • Undertuned preprocessing, rather than limited model capacity, explains linear models’ reputation as uncompetitive long-term forecasting baselines.
  • Ridge regression tuned over context length, local normalization, regularization, and augmentation matches or exceeds prior linear, Transformer, MLP, and CNN baselines on most standard benchmarks.
  • The optimized hyperparameters provide a diagnostic lens on the underlying data in addition to improving forecasting accuracy.

A Long-range linear autocorrelation in the benchmark series

Across six benchmark datasets, substantial linear dependence remains at lag 720 even after daily and weekly seasonality is removed, supporting the use of long contexts for genuinely long-range information rather than simple cycle re-representation.

  • Long-range dependence: 0.39 to 0.73: all six datasets retain substantial autocorrelation at lag 720 after harmonic deseasonalization.The confidence intervals are far above zero, and the values are channel-averaged Pearson correlations.
  • Robustness to deseasonalization: Four deseasonalization methods agree closely at long lags, so the surviving correlation is not specific to one removal procedure.The methods are raw, per-position-in-period, harmonic, and STL.
  • Interpretation: 720 steps in the past still carry a real and moderately strong linear association with the current value after obvious daily and weekly cycles are removed.Autocorrelation measures the Pearson correlation between a series and its lagged values, ranging from −1 to 1.
  • Interpretation: The long contexts selected by the search therefore exploit genuine long-range information rather than merely re-representing cycles captured by simpler seasonal methods.Per-dataset selected context lengths range from a few dozen steps on Exchange to roughly a thousand on Weather and Electricity.

B How much accuracy comes from context length alone

Holding normalization, regularization, and augmentation fixed shows that context length alone explains substantial but dataset-dependent accuracy gains. Longer context helps strongly on some datasets, saturates or barely matters on others, and requires dataset-specific search.

  • Method: The analysis varies only L under one universal-default preprocessing configuration, sweeping L from 24 to 2000 at H = 720.The default uses local mean normalization, the standard regularization grid, and no augmentation.
  • Dataset regimes: 36% to 53% error reduction occurs as L grows to 2000 on ETTh1, ETTh2, and Weather.These datasets form the strong regime, where error continues decreasing through the largest tested context.
  • Dataset regimes: 16% error reduction is achieved on ETTm2 by L ≈192–336, after which performance saturates, while ETTm1 is almost insensitive to additional context.These correspond to the plateau and flat regimes, respectively.
  • Implication: No single context length is best for every dataset, so identifying the optimum requires searching each dataset’s value.The study isolates context length specifically to distinguish its contribution from other preprocessing choices.

C Does a nonlinear model capture structure the linear model misses?

The analysis tests whether nonlinear models capture predictable structure that Ridge misses by comparing residual randomness with paired BDS statistics. PatchTST leaves less non-random residual structure on the standard benchmarks, but this advantage generally does not improve accuracy over the tuned linear model.

  • Residual-structure test: Paired BDS differences compare each architecture with Ridge under matched dataset, context length, horizon, seed, and channel conditions.A negative Δ means the architecture’s residuals are closer to noise; significance requires the paired 95% confidence interval to exclude zero.
  • Standard benchmarks: −7.83 mean Δ: PatchTST leaves less non-random residual structure than Ridge on the six standard benchmarks, significantly in 18 of 24 cells.The result indicates that PatchTST captures nonlinear patterns that Ridge cannot represent.
  • Accuracy implications: On the same six benchmarks, PatchTST’s additional captured structure does not change the accuracy ranking: tuned Ridge still matches or outperforms it.This directly addresses whether nonlinear residual structure necessarily translates into lower forecast error.
  • Dataset-specific result: On Weather, PatchTST’s residuals are more structured than Ridge’s at three of four horizons, with the difference significant.Here the nonlinear model does not even leave less residual structure than the linear model.
  • Large datasets: On the two large datasets, only one of eight comparisons is significant, so PatchTST’s lower error there is not explained by captured nonlinear structure.The residual-structure test therefore provides little evidence that nonlinear-pattern capture accounts for the accuracy advantage on those datasets.

D Limitations

The study’s findings are specific to standard numeric long-horizon forecasting benchmarks and should not be treated as universal. Reported MSE margins and nonlinear-baseline comparisons also have important interpretive limitations.

  • Scope: The reported lookback-scaling, locality, and heterogeneity patterns apply to standard numeric long-horizon forecasting benchmarks, not universally.The authors explicitly limit these findings to this forecasting regime.
  • Evaluation: Very small MSE margins should be treated as ties because evaluation uses point-estimate MSE following prior work.The study does not use probabilistic or uncertainty-aware evaluation for these comparisons.
  • Comparisons: Nonlinear-baseline comparisons reflect published configurations rather than a jointly retuned preprocessing protocol.The nonlinear baselines were quoted from their original publications.

E Broader Impact

The work is a methodological study on public numeric time-series benchmarks, offering a reproducible, lower-compute forecasting baseline without human-subject data or operational deployment. Potential harms are generic to forecasting research because accurate forecasts could be misused on sensitive operational signals, although such deployments are not studied.

  • Scope: The study uses public numeric benchmarks and does not target individuals, use human-subject data, or evaluate deployment in a real operational system.
  • Positive impact: Closed-form Ridge models with searched preprocessing provide a strong, reproducible baseline at substantially lower compute cost than many nonlinear forecasters.
  • Potential risks: More accurate long-horizon forecasts could be misused on energy, traffic, or financial signals, affecting privacy, resource allocation, infrastructure operation, or markets.
  • Potential risks: The paper does not study these deployments, and its released pipeline is a reusable forecasting method rather than a system targeted at individuals or groups.

F Licenses and Terms of Use for Existing Assets

The study uses public benchmark datasets, quoted baseline results, author-released implementations, and standard scientific Python libraries under their stated terms. Dataset licensing is identified where verified, otherwise the canonical repository governs use.

  • Benchmark datasets: Eight multivariate forecasting benchmarks come from the public distribution accompanying Autoformer, under the MIT License.The listed original sources include Informer’s ETDataset repositories, UCI Electricity under CC BY 4.0, Caltrans PeMS, the Max Planck Weather dataset, and LSTNet’s Exchange Rate repository under MIT.
  • Baseline numbers and reference implementations: Linear baseline numbers for OLS, FITS, and DLinear are quoted from [20], while nonlinear results for PatchTST, iTransformer, TimeMixer, TimesNet, and Autoformer come from their original publications.The corresponding author-released reference implementations are also identified for these baselines.
  • Software libraries: The pipeline uses unmodified open-source libraries, including PyTorch, NumPy, Pandas, SciPy, scikit-learn, statsmodels, Matplotlib, Seaborn, and Optuna, within their respective licenses.Most listed libraries use BSD-3-Clause; Matplotlib uses its PSF-style license, and Optuna uses the MIT License.
Loading 2606.27282v2…