PaperlayerView exact revision on arXiv

Source-linked AI summary

Simple Feedfoward Neural Networks are Almost All You Need for Time Series Forecasting

Fan-Keng Sun, Yu-Cheng Wu, Duane S. Boning

arXiv:2503.23621v1cs.LG

TL;DR

Time-series forecasting has increasingly favored sophisticated architectures, despite uncertainty about whether that complexity is necessary on current benchmarks. The paper evaluates simple feedforward neural networks, finding they can match or exceed state-of-the-art models; it presents them as a strong baseline, while acknowledging they are not universally optimal.

  • Problem

    Recent forecasting work has pursued increasingly sophisticated architectures based on the belief that intricate temporal and spatial patterns are crucial, while the need for that complexity on current benchmarks remains in question.

    Source: §1 · p. 2

  • Method

    The paper introduces simple feedforward neural networks, generally using a shared univariate model and adding simple series-wise mapping when inter-series dependencies are strong.

    Source: §1 · p. 2

  • Results

    SFNNs can match or surpass state-of-the-art models such as iTransformer and DUET on current benchmark datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SFNNs are a strong baseline for future time-series forecasting models, and the proposed evaluation protocol selects look-back length on validation data and reports results over multiple trials.

    Source: §5 · p. 11

    SFNNs are not universally optimal; some datasets may favor specialized models.

    Source: §5 · p. 11

Abstract

from arXiv · show

Time series data are everywhere -- from finance to healthcare -- and each domain brings its own unique complexities and structures. While advanced models like Transformers and graph neural networks (GNNs) have gained popularity in time series forecasting, largely due to their success in tasks like language modeling, their added complexity is not always necessary. In our work, we show that simple feedforward neural networks (SFNNs) can achieve performance on par with, or even exceeding, these state-of-the-art models, while being simpler, smaller, faster, and more robust. Our analysis indicates that, in many cases, univariate SFNNs are sufficient, implying that modeling interactions between multiple series may offer only marginal benefits. Even when inter-series relationships are strong, a basic multivariate SFNN still delivers competitive results. We also examine some key design choices and offer guidelines on making informed decisions. Additionally, we critique existing benchmarking practices and propose an improved evaluation protocol. Although SFNNs may not be optimal for every situation (hence the ``almost'' in our title) they serve as a strong baseline that future time series forecasting methods should always be compared against.

1 Introduction

Long-term forecasting research has increasingly pursued sophisticated Transformer architectures, driven by the view that intricate temporal and spatial patterns are crucial. The authors argue that simple feedforward neural networks can match or exceed state-of-the-art models on current benchmark datasets, and describe their architecture, design choices, and evaluation contributions.

1 Introduction

  • Earlier neural-network forecasting methods were often tailored to specific applications, limiting their direct applicability across diverse time-series domains.The introduction describes weather, traffic, electricity-demand, and economic forecasting as application-specific research areas.Source: §1 · pp. 1–2
  • Recent long-term forecasting work has pursued increasingly sophisticated Transformer models, amid claims that self-attention remains effective and a belief that intricate temporal and spatial patterns matter.The passage notes that linear models may perform better in some comparisons, while later work has demonstrated effective self-attention.Source: §1 · p. 2
  • SFNNs match or surpass state-of-the-art models such as iTransformer and DUET on current benchmark datasets, without handcrafted modules for scales, frequencies, or pattern clusters.The authors generally use a shared univariate SFNN when inter-series dependencies are weak, and add simple series-wise mapping when they are strong.Source: §1 · p. 2
  • The authors introduce a univariate SFNN for long-term forecasting and a multivariate version for settings with strong inter-series dependencies.They state that the univariate model can achieve state-of-the-art performance against Transformer-based models.Source: §1 · p. 2
  • The authors also identify three critical design choices for ablation and practical guidance, and propose an improved evaluation protocol in response to current benchmarking practices.The experiments section describes these as ablations of optional modules and a critique of existing evaluation practices.Source: §1 · p. 2

2 Preliminaries

Forecasting uses a selected span of recent observations to predict future values, with model parameters chosen to minimize mean squared error. Because time series require temporal splits and may have small samples or distribution drift, rigorous validation is important, yet test-set peeking and single validation runs remain common.

2.1 Definitions

  • A forecasting example pairs recent input histories with future target values, using a look-back length L to select the most recent histories.The input contains N series over T time steps, and the forecast horizon H specifies the future values to predict.Source: §2.1 · p. 2

2.2 Problem Statement

  • The model parameters are chosen to minimize mean squared error on the test set, using forecasts generated from the preceding L time steps.The model maps the look-back input to forecasts over the horizon H.Source: §2.2 · pp. 2–3

2.3 Common Practices

  • Time series should be split in temporal order, and out-of-sample K-fold cross-validation uses successive subsets to validate models trained on earlier data.The passage notes that relatively small datasets and distribution drift make method comparisons more challenging.Source: §2.3 · p. 3
  • Common practice may select hyperparameters by peeking at the test set and rely on one validation period in a single run, which can mislead model selection.Out-of-sample K-fold cross-validation is described as more robust for selecting models for future unseen data.Source: §2.3 · p. 3

3 Simple Feedforward Neural Networks (SFNNs)

SFNNs build on a shared univariate mapping, adding linear-ReLU residual blocks to capture nonlinear patterns. Optional modules address trends, inter-series relationships, and normalization, but their benefits vary across datasets.

3 Simple Feedforward Neural Networks (SFNNs)

  • The base SFNN shares a univariate mapping across series and adds linear-ReLU inner blocks with residual connections to capture more complex nonlinear patterns.Figure 1 identifies linear mappings, ReLU activations, and residual connections as the core block.Source: §3 · pp. 3–4, Figure 1 · p. 3

3.1 Input mean centering

  • Input mean centering subtracts the input mean before forecasting and adds it back to the output, adapting the model to trends without explicit trend decomposition.The authors report that relying on the last input value as a baseline diminishes performance in most benchmark cases.Source: §3.1 · p. 4

3.2 Series-wise non-linear mapping

  • Univariate SFNNs work well on most datasets, while series-wise nonlinear mapping can significantly improve results in cases such as Solar Energy.The mapping applies a shared N × N transformation and activation across time steps; SELU performs slightly better than ReLU in this setting, though the reason is uncertain.Source: §3.2 · p. 4

3.3 Layer normalization

  • Layer normalization benefits some datasets but may cause overfitting in others.The passage notes that its bias and gain parameters can contribute to overfitting.Source: §3.3 · p. 4

4 Experimental Results

The experiments benchmark SFNNs against prior methods, ablate optional modules, and propose an improved evaluation protocol. Table 1 reports test-peek-selected results, while Table 2 lists the look-back lengths searched for each dataset.

4 Experimental Results

  • The experiments benchmark SFNNs against prior methods, study three optional modules, and propose an improved evaluation protocol.The section also offers practical guidance on choosing modules for different application characteristics.Source: §4 · p. 4
  • Table 1 reports MSE and standard deviations across 10 runs, marking best models and statistically significant wins.Look-back lengths are selected by peeking at the test set; † denotes significance at p-value below 5%.Source: Table 1 · p. 5
  • Table 2 scans look-back lengths that are multiples of dataset periods, using five for Exchange rate because it lacks a clear periodicity.Source: Table 2 · p. 5

4.1 Results following common practices

Under common benchmarking practices, SFNN outperforms DUET and iTransformer in most reported dataset-horizon cases, though results vary by dataset. The section also compares their runtime, MSE, and parameter counts across datasets and two horizons.

4.1 Results following common practices

  • The comparison uses 10 datasets with training-set z-score normalization, but Table 1 reports seven because ILI, Weather, and Exchange rate have issues.Source: §4.1 · p. 5

4.1.1 Model sizes and run times

  • Figure 2 compares relative runtime, MSE, and parameter counts for DUET and iTransformer against SFNNs at horizons 96 and 720.Source: Figure 2 · p. 6

4.1 Results following common practices

  • Only MSE is reported because the results are largely similar to MAE while avoiding the extra computational effort of tuning separate hyperparameters.Source: §4.1 · p. 6
  • SFNN is best in 19 out of 28 cases, with 14 improvements statistically significant, but performance is suboptimal on Traffic and, less strongly, ETTh2.On Traffic, univariate SFNNs slightly underfit while the multivariate variant overfits.Source: §4.1 · p. 6

4.2 Ablation study

The ablation relates component benefits to dataset properties rather than removing components one at a time. It finds that the value of mean centering, series-wise mapping, and layer normalization depends on trends, inter-series relationships, and scale differences.

4.2 Ablation study

  • The ablation correlates component improvements with dataset properties and uses significance tests to draw data-agnostic conclusions.The authors argue that removing components individually is not ideal given the diversity of time series data.Source: §4.2 · p. 6

4.2.1 Input mean centering

  • Mean centering enables state-of-the-art SFNN performance on ETT and is generally beneficial above a 0.2 mean-squared input mean, with significant positive improvement at p=5%.The mean of the squared input mean serves as a measure of trend strength; the authors describe the threshold as a guideline, not a strict rule.Source: §4.2.1 · pp. 6–7

4.2.2 Series-wise non-linear mapping

  • Series-wise mapping can worsen performance when N > 100, increasing Traffic MSE by over 100%, but strongly cointegrated Solar series can benefit.The risk grows with longer look-back lengths; Solar Energy improves by 20% around a look-back length of 200, and 137 series remain significantly cointegrated through 120 lags.Source: §4.2.2 · p. 8

4.2.3 Layer normalization

  • Layer normalization's benefits increase with inter-series scale differences and are usually positive above an x-axis value of 0.5.The within-dataset trend is weaker and does not hold for Solar, which exhibits strong cointegration.Source: §4.2.3 · pp. 8–9

4.3 Critiques of current benchmarking practices

The section identifies weaknesses in commonly used benchmark datasets and evaluation practices. It recommends model-specific look-back selection, repeated trials, and validation-based hyperparameter selection to reduce bias.

4.3.1 Issues with the ILI, Weather, and Exchange rate datasets

  • ILI has fewer than 1,000 time steps across seven series and substantial train-test drift, so its benchmark results warrant caution.No model achieves MSE below 1, and high variability means no model significantly outperforms the others.Source: §4.3.1 · p. 9
  • Weather and Exchange Rate favor N separate linear models, suggesting they behave like independent univariate series and can lead multivariate models to overfit.SFNNs nevertheless rank second-best on these datasets.Source: §4.3.1 · pp. 9–10

4.3.2 Affects of look-back length and randomness

  • On Electricity and ETTm1, SFNNs may lag with short look-backs but outperform competitors with longer lags, leveraging longer historical dependencies more effectively.Figure 7 compares MSE across horizons and look-back lengths.Source: Figure 7 · p. 9, §4.3.2 · p. 10

4.3.3 Reduce the effect of “peeking”

  • With look-back selection on validation data, SFNN's performance advantage over baseline models becomes more pronounced than in Table 1.Table 4 reports MSE and standard deviation across 10 runs for models using validation-selected look-back lengths.Source: Table 4 · p. 10, §4.3.3 · p. 11

4.3.2 Affects of look-back length and randomness

  • Because initialization, batch shuffling, and dropout affect results, the authors ran each experiment 10 times and reported means, standard deviations, and significance.They advocate allowing each model to select its optimal look-back length and reporting multiple trials.Source: §4.3.2 · pp. 10–11

4.3.3 Reduce the effect of “peeking”

  • To reduce test-set peeking, the authors recommend choosing look-back length on validation data, which better reflects real-world selection and makes evaluation less biased.Out-of-sample K-fold cross-validation is presented as rigorous but often too complex and computationally intensive for broad use.Source: §4.3.3 · p. 11

5 Conclusion and Limitations

SFNNs are an effective, robust, and efficient alternative that can match or exceed more complex models, and the paper proposes a more rigorous evaluation protocol. The authors do not claim SFNNs are universally optimal, but recommend them as a strong baseline.

5 Conclusion and Limitations

  • SFNNs consistently match or exceed more complex models while offering an effective, robust, and efficient alternative.Ablation studies highlight input mean centering, series-wise mapping, and layer normalization as key components, with practical implementation guidelines.Source: §5 · p. 11
  • The proposed evaluation protocol selects look-back length on validation data and reports results over multiple trials for more rigorous comparisons.The authors say this captures statistical significance.Source: §5 · p. 11
  • SFNNs are not universally optimal: specialized models may outperform them on some datasets, so SFNNs are a strong baseline for future comparisons.Source: §5 · p. 11

A Statistics and Descriptions of Datasets

The dataset descriptions are organized by length, number of series, sampling period, data-collection split and year(s), and description.

A Statistics and Descriptions of Datasets

Loading 2503.23621v1…