PaperlayerView exact revision on arXiv

Source-linked AI summary

An Analysis of Linear Time Series Forecasting Models

William Toner, Luke Darlow

arXiv:2403.14587v2cs.LG

TL;DR

Deep forecasting models can offer only marginal benefits over simple linear models, motivating analysis of whether popular linear variants genuinely differ. The paper characterizes their function classes, shows they can be reformulated as augmented-feature linear regression with closed-form solutions, and finds that the models learn nearly identical solutions while closed-form methods outperform them across 72% of test settings.

  • Problem

    Popular linear forecasting variants claim advantages through normalization, decomposition, or frequency-domain processing, but their functional differences from standard linear regression require analysis.

  • Method

    The paper mathematically characterizes several linear architectures, reformulates them as unconstrained linear regression over augmented features, and analyzes their MSE optimization and closed-form solutions.

  • Results

    Closed-form ordinary least-squares solutions are superior forecasters across 72% of test settings, while the analyzed models learn nearly identical solutions.

  • Takeaways & Limitations

    Popular linear variants largely retain the behavior of standard linear regression, and simpler closed-form solutions are generally as good as or better than trained alternatives.

  • Takeaways & Limitations

    The experiments analyze FITS mode 1 only and exclude its additional reconstruction-loss training mode.

Abstract

from arXiv · show

Despite their simplicity, linear models perform well at time series forecasting, even when pitted against deeper and more expensive models. A number of variations to the linear model have been proposed, often including some form of feature normalisation that improves model generalisation. In this paper we analyse the sets of functions expressible using these linear model architectures. In so doing we show that several popular variants of linear models for time series forecasting are equivalent and functionally indistinguishable from standard, unconstrained linear regression. We characterise the model classes for each linear variant. We demonstrate that each model can be reinterpreted as unconstrained linear regression over a suitably augmented feature set, and therefore admit closed-form solutions when using a mean-squared loss function. We provide experimental evidence that the models under inspection learn nearly identical solutions, and finally demonstrate that the simpler closed form solutions are superior forecasters across 72% of test settings.

1. Introduction

Time-series forecasting has motivated increasingly complex models, yet recent work finds that simpler linear models can match or outperform deep alternatives. This paper analyzes popular linear variants and shows they are mathematically equivalent or weakly constrained forms of linear regression, with experiments supporting near-identical solutions and stronger closed-form OLS performance.

  • The analysis responds to evidence that deep forecasting models often offer marginal benefits over simpler linear models, which remain attractive for their efficiency and explainability.
  • Several popular linear forecasting variants are mathematically equivalent or functionally indistinguishable from unconstrained or weakly constrained linear regression.
  • The paper fully characterizes the functions expressible by these architectures and reframes their variants as linear regression problems.
  • Experiments show that the models tend toward the same solution when trained on the same data, differing only in the bias parameter.
  • Closed-form ordinary least-squares solutions are typically superior to existing variants trained with stochastic gradient descent.

2. Related Work

Related work established linear models as strong forecasting baselines and introduced variants based on normalization, channel independence, and frequency-domain processing. These studies motivate the paper’s analysis of whether such architectural changes materially alter linear forecasting models.

  • DLinear and NLinear became widely used baselines after showing that linear models can match or outperform complex transformer architectures.
  • Reversible instance normalization standardizes input features before modeling and reverses the transformation afterward, optionally using a learnable affine transformation.
  • RLinear combines linear mapping with reversible instance normalization, while channel independence learns distinct models for each variate.
  • FITS operates in frequency space by applying a complex linear map between a real discrete Fourier transform and its inverse, with reported near-state-of-the-art performance under optimal settings.

3. Analysis of Linear Time Series Forecasting Models

This section characterizes several linear forecasting architectures as function classes, showing how decomposition, frequency transforms, and normalization affect their expressivity. Under stated conditions, multiple variants coincide with unconstrained or constrained linear regression.

  • 3.1.1. DLinear: DLinear’s trend-seasonal decomposition still reduces algebraically to an affine map A⃗x +⃗b, and every affine map can be represented by DLinear.The moving-average operator is absorbed into the resulting weight matrix.
  • 3.1.2. FITS: FITS applies Fourier-domain linear transformations, yet when L ≥ T −2 its resulting matrix and bias are entirely unconstrained.The forecast is obtained from the output after discarding the reconstructed context components.
  • 3.1. Analysis of Linear Models: When L ≥ T −2 and without a low-pass filter, DLinear, Linear, and FITS have the same model class of affine linear functions.FITS is unconstrained in both its matrix and bias under this context-length condition.
  • 3.2. Instance Normalization: Instance normalization changes the function class by centering and scaling the context before forecasting, then reversing that transformation on the prediction.The numerical-stability constant ε is included in the normalization and inverse transformation.
  • 3.3. Instance Norm: Linear+IN represents functions of the form ˜A⃗x +⃗bσ(⃗x), with every row of ˜A summing to 1.Thus normalization introduces an input-dependent standard-deviation term and a row-sum constraint.
  • 3.5. NLinear: NLinear similarly corresponds to constrained linear regression, while the analyzed normalized variants are grouped into only three related model classes.The NLinear constraint is that each row of the weight matrix sums to 1.

4. Discussion

The analysed linear forecasting architectures can be characterised as constrained or unconstrained linear regression, with equivalent formulations enabling closed-form least-squares solutions. Convexity predicts common optima, while parameterisation and training choices can still affect optimisation behaviour.

  • Model classes: The model classes can be reformulated as unconstrained linear regression over augmented features, making each class solvable in closed form.NLinear imposes row-sum restrictions, while Linear+IN, Linear+RevIN, and FITS+IN share a different bias parameterisation.
  • Convexity: Mean-squared linear regression is convex and therefore has a unique global optimum when the training data is full rank.This motivates the expectation that equivalent architectures trained on the same data should converge to the same solution.
  • Closed-form solutions: Each model class admits a least-squares optimum in closed form, which the paper evaluates as an OLS alternative to iterative training.The closed-form solutions are compared against models trained using the architectures’ usual procedures.
  • Experimental scope: The analysis considers FITS mode 1 only, using forecast-target MSE rather than the additional context-reconstruction loss in mode 2.Both FITS training modes are reported to have similar empirical performance, but the experiments restrict attention to mode 1.
  • Optimisation effects: Different parameterisations and initialisations leave open whether early stopping changes generalisation despite equivalent convex optima.The discussion specifically examines whether FITS’s parameterisation induces a lower effective learning rate for its bias term.

5. Experiments

Across convergence and forecasting experiments, trained linear variants learned highly similar weights and forecasts, while closed-form OLS generally performed best across benchmark settings.

  • 5.1. Convergence: The learned weight matrices of four trained variants were near identical to the closed-form OLS+IN solution after 50 epochs.The comparison used RLinear, NLinear, DLinear+IN, FITS+IN, and OLS+IN on ETTh1 with context 720 and horizon 336.
  • 5.1. Convergence: All model weight matrices converged toward their OLS counterparts, although parameterisation affected convergence rate and trajectory.Cosine similarity to OLS was tracked during training; a value of one denotes exact matrix equality.
  • 5.1. Convergence: Forecasts remained clearly and pervasively similar despite subtle model differences that produced marginal forecast changes.Figure 3 compares five instance-normalised models on ETTh1 with T = 336.
  • 5.1. Convergence: DLinear+IN, RLinear, and OLS+IN learned the same bias terms, whereas FITS+IN learned a considerably smaller and different bias.The difference occurs despite equivalent model classes; FITS performs particularly well on the small hourly ETT datasets, where its bias restriction may reduce overfitting.
  • 5.2. Performance: OLS solutions were superior across 23 of 32 (72%) settings.Table 2 reports MSE values across the evaluated datasets and horizons; green and blue highlighting mark OLS superiority and results within one standard deviation.

6. Conclusion

The paper concludes that common linear forecasting modifications remain functionally close to standard linear regression and generally behave similarly to one another. Closed-form solutions usually outperform these trained variants, supporting simpler approaches for the studied settings.

  • 6. Conclusion: Common forecasting modifications barely deviate from standard linear regression from a functional standpoint.The paper considers trend-residual decomposition, instance normalisation, and Fourier-space processing as modifications whose model classes remain close to linear regression.
  • 6. Conclusion: The models behave and perform similarly to each other and generally worse than their closed-form solutions.

A.1. FITS

FITS applies Fourier-domain filtering and a learnable complex linear transformation before reconstructing the forecast. Despite this specialized architecture, it is affine linear and, under the stated length condition, can express any affine map.

  • FITS model definition: FITS transforms the context with an RFT, optionally low-pass filters frequencies, applies a complex linear layer, pads with zeros, and uses an inverse RFT to produce the output.The analysis later ignores the low-pass filter because it degrades performance in the settings studied.
  • Fourier transforms: The DFT is defined as a matrix transformation, while the RFT truncates redundant conjugate-frequency components and reconstructs real signals through an inverse RFT.For real inputs, conjugate DFT components contain duplicate information, motivating the truncated representation.
  • Expressive power: When L ≥ T −2, the FITS model class is precisely the space of affine linear functions A x + b.The proof characterizes the Fourier-domain matrices and shows that the resulting real affine maps are unrestricted under this condition.
  • Affine representation: Because FITS composes affine linear operations, every FITS model is an affine map f(x) = Ax + b, with its real bias obtained by applying the inverse RFT to the complex-layer bias.The linear component can be written as iRFT ◦ W ◦ RFT.
  • Proof strategy: The proof constructs the required matrix representation by translating between Fourier-domain complex multiplications and real affine maps through projection and inverse-transform operators.The matrix characterization uses conjugacy and zero-padding constraints induced by the real Fourier representation.

Proof of Lemma 3.3

The proof shows that FITS can realize arbitrary forecast matrices once the context is sufficiently long relative to the prediction horizon. Fourier-domain structural constraints are overcome through a rank argument.

  • Forecast-map surjectivity: FITS imposes no restriction on the bias, and when L ≥ T −2 any real T × L forecast matrix A_T can be attained by selecting its complex weight matrix.Thus the forecast component is unrestricted under the theorem’s length condition.
  • Matrix characterization: The attainable FITS matrices are characterized through Fourier columns: endpoint columns are real, intermediate columns obey conjugacy, and high-frequency output components are zero.These conditions describe the structured complex matrices induced by the real Fourier transform and inverse transform.
  • Dimension argument: The proof reduces arbitrary forecast realization to showing that a projected inverse-DFT subspace has dimension at least T.The required dimension condition becomes (L/2) + 1 ≥ T, equivalent to L ≥ T −2.
  • Rank argument: A Vandermonde submatrix generated from a root of unity has non-zero determinant and therefore full rank, establishing the needed dimension bound.This full-rank property supplies the final step in showing that every real forecast matrix can be represented.

B. Further Results and Experiments

Figure 5 compares learned biases after 50 ETTh1 training epochs across FITS, Linear, DLinear, and closed-form OLS. Linear, DLinear, and OLS share the same bias, whereas FITS differs substantially.

  • Bias comparison: Linear, DLinear, and closed-form OLS learn the same bias after training on ETTh1 for 50 epochs.The caption states that this matches the theory developed in Section 3.
  • Bias comparison: FITS learns a substantially different bias, which the authors explain through the Fourier-transform normalization used by FITS.The comparison isolates a difference in learned bias despite the related linear model classes.

C. FITS Bias Term - Detailed Breakdown

The bias parameterization in FITS changes how stochastic gradient descent scales updates because the inverse Fourier transform maps the learned complex bias through a non-orthonormal normalization. This makes bias learning much slower than direct bias optimization.

  • Bias parameterization: FITS obtains its real bias through b = iRFT(c), equivalently b = M C after splitting the complex bias into real components.This reparameterization is the basis for analyzing FITS bias updates under SGD.
  • SGD update scaling: The non-orthonormal Fourier normalization makes the induced bias update scale like M M^T b rather than a direct update of b.The transform matrix scales components by approximately 1/√(L+T), so M M^T scales them by approximately 1/(L+T).
  • Optimization consequence: Although FITS’s output scaling partially offsets this effect, the bias learning rate remains approximately 1/L times smaller than under naive bias parameterization.Increasing the learning rate globally would also affect the weight matrix, so it cannot simply fix the bias without disrupting weight learning.
  • Weight updates: The normalization choice does not alter the weight update because normalization in one Fourier transform is offset by the corresponding unnormalized multiplication in the other.The analysis states that alternative RFT normalization standards lead to the same weight update.

D. Further Proofs

The proofs show how moving averages, closed-form least squares, and normalized linear variants can be represented through matrix operations or augmented features. NLinear and RLinear therefore admit global least-squares solutions in closed form.

  • DLinear moving average: DLinear’s padded moving average can be expressed as a matrix multiplication after repeating boundary values to preserve the context dimension.For odd kernel size K, each side is padded with (K−1)/2 repeated entries; the paper notes K=25 for DLinear.
  • Closed-form regression: Least-squares regression has a closed-form solution, but rank-deficient design matrices may lack a unique solution and require a Moore-Penrose pseudo-inverse.Ill-conditioned X^T X can also make the direct formula numerically unstable, motivating a more stable SVD computation.
  • NLinear: NLinear can be solved by centering each input and target row by the input mean, then applying unconstrained least-squares regression to the augmented matrices.This transformation preserves the relevant MSE and yields a solution satisfying the original row-sum constraint.
  • RLinear: RLinear likewise admits a global least-squares solution because its standard-deviation term can be represented by appending the input standard deviation as an augmented feature.The resulting construction parallels the NLinear reduction to regression on a suitable feature set.

E. Experiment Details

The experiments compare multiple linear forecasting architectures and closed-form OLS variants across eight benchmark datasets and standard forecast horizons. Separate weight-comparison experiments examine learned parameters and their convergence toward OLS.

  • Datasets: Experiments use eight standard time-series benchmarks, including hourly, per-minute, electricity, weather, and traffic datasets with varying channel counts.The listed datasets include ETTh1, ETTh2, ETTm1, ETTm2, ECL, Weather, and Traffic; the supplied passage truncates the final dataset description.
  • Models: The study evaluates DLinear, NLinear, RLinear, FITS, Linear, FITS+IN, DLinear+IN, OLS, and OLS+IN using SVD-based closed-form implementations for OLS.Training-based models use the specified Adam setup, while closed-form solutions are implemented with scikit-learn’s LinearRegression.
  • Training and evaluation: Models are evaluated at horizons 96, 192, 336, and 720 with context length 720, using at least three random seeds and reporting MSE standard deviations.Training runs use 50 epochs, learning rate 0.0005, Adam, batch size 128, and validation-loss model selection.
  • Weight comparisons: Weight-comparison experiments inspect weights, biases, and forecasts after 50 epochs, while cosine-similarity analyses run for 350 epochs with learning rate 0.0002.These settings differ from the main benchmark procedure, which uses early stopping based on validation loss.

F.2. Limitations and Future Work

The limitations qualify the equivalence results by model setting and identify unresolved opportunities involving FITS filtering and post hoc compression. These boundaries define where the paper’s conclusions do not yet extend.

  • Normalization settings: Linear+RevIN and Linear+IN share model classes, but separate RevIN affine parameters can produce marginally different biases across channels when linear weights are shared.The distinction arises specifically in the multi-channel setting.
  • FITS filtering: The FITS equivalence result applies without its low-pass filter; applying the filter can restrict the model class and requires further analysis.The paper states that the low-pass filter is omitted when analyzing FITS under optimal settings.
  • Future work: The paper leaves open whether FITS frequency compression can be applied post hoc to closed-form OLS solutions.This follows from the established mapping between FITS models and their underlying affine representations.
Loading 2403.14587v2…