Source-linked AI summary
SegRNN: Segment Recurrent Neural Network for Long-Term Time Series Forecasting
Shengsheng Lin, Weiwei Lin, Wentai Wu, Feiyu Zhao, Ruichao Mo, Haotong Zhang
TL;DR
Long-term forecasting makes RNNs costly and difficult because long look-back windows and horizons require many recurrent iterations. SegRNN reduces those iterations through segment-wise encoding and PMF decoding, and experiments report stronger forecasting with substantially lower runtime and memory use.
Problem
Long look-back windows and forecast horizons make RNNs difficult to use for LTSF because they require prohibitively many recurrent iterations.
Method
SegRNN replaces point-wise with segment-wise iterations and combines this encoding strategy with Parallel Multi-step Forecasting instead of recurrent multi-step decoding.
Results
SegRNN generally outperformed SOTA Transformer-based models while reducing runtime and memory usage by over 78% in most scenarios.
Takeaways & Limitations
The results provide evidence that RNNs retain strong capabilities for LTSF when recurrent iteration counts are reduced.
Takeaways & Limitations
The approach adopts a channel-independent strategy and uses GRU as its default exemplar, with lower performance reported on larger-channel datasets.
Abstract
from arXiv · showhide
RNN-based methods have faced challenges in the Long-term Time Series Forecasting (LTSF) domain when dealing with excessively long look-back windows and forecast horizons. Consequently, the dominance in this domain has shifted towards Transformer, MLP, and CNN approaches. The substantial number of recurrent iterations are the fundamental reasons behind the limitations of RNNs in LTSF. To address these issues, we propose two novel strategies to reduce the number of iterations in RNNs for LTSF tasks: Segment-wise Iterations and Parallel Multi-step Forecasting (PMF). RNNs that combine these strategies, namely SegRNN, significantly reduce the required recurrent iterations for LTSF, resulting in notable improvements in forecast accuracy and inference speed. Extensive experiments demonstrate that SegRNN not only outperforms SOTA Transformer-based models but also reduces runtime and memory usage by more than 78%. These achievements provide strong evidence that RNNs continue to excel in LTSF tasks and encourage further exploration of this domain with more RNN-based approaches. The source code is coming soon.
1 Introduction
Long-term forecasting increases uncertainty and modeling complexity, exposing the iteration burden of RNNs. SegRNN addresses this with segment-wise iterations and Parallel Multi-step Forecasting, improving accuracy while reducing computational costs.
- Longer forecast horizons increase uncertainty and require models to use more historical context, making LTSF substantially more difficult.
- RNN forecast error rises rapidly with the horizon, while inference time also increases as the horizon grows.
- Transformer-based LTSF models have achieved strong results, but their intricate designs and computational requirements remain concerns.
- SegRNN replaces point-wise with segment-wise iterations and introduces PMF to reduce recurrent iterations while retaining sequential information.
- 78%+ runtime and memory reductions were reported for SegRNN, which outperformed SOTA Transformer-based models in most benchmark scenarios.
2 Related Work
LTSF research has expanded across Transformer, MLP, CNN, and RNN architectures. Although RNNs remain effective for sequential data and conventional forecasting, excessively long LTSF windows and horizons have challenged their use.
- Transformers: Transformers use self-attention to capture long-term dependencies and have demonstrated impressive performance in LTSF.
- MLPs: MLP-based approaches became prominent after DLinear surpassed then-state-of-the-art Transformer models with a simple linear layer and channel-independent strategy.
- CNNs: CNN-based models have shown strong LTSF performance after adapting convolutional methods for local-pattern extraction in time series.
- RNNs: RNNs have long supported sequential forecasting and probabilistic prediction, but excessively long LTSF windows and horizons have made them appear inadequate.
3 Preliminaries
The preliminaries define LTSF as forecasting a multivariate future from historical observations, adopt a channel-independent strategy, and review RNN variants. SegRNN is presented as RNN-cell agnostic but illustrated primarily with GRU.
- 3.1 LTSF Problem Formulation: LTSF predicts Y ∈ R^H×C from historical MTS X ∈ R^L×C, with the forecast horizon H extending as far as 720.
- 3.2 Channel Independent Strategy: The channel-independent strategy maps each univariate historical series X^(i) ∈ R^L to its future series Y^(i) ∈ R^H.
- 3.3 RNN Variants: Vanilla RNNs face vanishing and exploding gradients, while LSTM and GRU modify cell structures to address these training difficulties.
- 3.3 RNN Variants: SegRNN is not restricted to one RNN cell, but the paper uses GRU as its exemplar because of its stable practical performance.
4 Model Architecture
SegRNN reduces recurrent computation by encoding time-series segments instead of individual points and decoding forecast segments in parallel. Its architecture also uses positional embeddings, sequence recovery, instance normalization, and MAE training.
- Encoding: SegRNN replaces point-wise encoding iterations with segment-wise iterations, reducing the count from L to L/w.The input sequence is partitioned into segments of window length w, then projected before recurrent encoding.
- Decoding: PMF duplicates the encoded state across forecast segments and processes them simultaneously with positional embeddings, reducing decoding iterations from H/w to 1.Independent parallel decoding avoids dependence on the previous time step and improves inference speed without iterative error accumulation.
- Encoding: The projected segments are processed by a GRU, whose final hidden state summarizes the encoded sequence for decoding.The GRU applies recurrent updates to the projected segment representations and passes h_n to the decoder.
- Decoding: RMF reduces decoding to H/w iterations but remains recursive, causing error accumulation and limiting parallel computation.Each prediction is fed into the next step, so decoding remains sequential.
- Decoding: Positional embeddings encode each forecast segment’s relative sequence position and channel position during parallel decoding.Channel-position encoding partially compensates for limitations in capturing relationships between variables.
- Output and training: After decoding, projected outputs are transformed into the forecast sequence, while instance normalization subtracts and later restores the input sequence’s last value.The model uses MAE as its loss function.
5 Experiments
Experiments across LTSF benchmarks show that SegRNN is highly competitive with established baselines while reducing computational cost. Ablations indicate that larger appropriate segments and PMF improve accuracy or efficiency, with trade-offs at extreme settings.
- Main results: SegRNN achieved top-two positions in 50 of 56 multivariate metrics, including 45 first-place rankings, across the evaluated scenarios.Its strongest results were on ETT and Weather; performance decreased slightly on larger Electricity and Traffic datasets but remained competitive or superior.
- Main results: 75% and 80% MSE improvements were reported over GRU and DeepAR, respectively.These comparisons indicate substantial gains over the evaluated RNN-based baselines.
- Ablation studies: Increasing segment length generally decreased forecast error and inference time until the segment length reached the look-back length, where prediction error increased.At the extreme, the model degenerates into a multilayer perceptron; therefore, a large but appropriate segment length is preferred.
- Ablation studies: PMF significantly outperformed RMF across forecast horizons, with its advantage becoming more evident as the horizon increased.For horizons below 192, PMF was slightly slower because memory-replication overhead outweighed its parallel-computation benefit.
- Model analysis: SegRNN’s forecast error consistently decreased with longer look-backs and remained robust across shorter look-back lengths.Among tested RNN variants, GRU generally produced lower and more stable forecast errors than alternatives.
- Model analysis: Over 78% lower average training time and over 82% lower average maximum GPU memory consumption were reported versus PatchTST.The comparison used the ETTm1 dataset with a single NVIDIA T4 GPU, a look-back of 720, and a horizon of 192.
6 Conclusion
The conclusion presents SegRNN as an RNN-based LTSF model that reduces recurrent iterations through segment-wise encoding and PMF decoding. It reports improved accuracy and efficiency relative to existing methods, supporting continued investigation of RNNs for LTSF.
- Conclusion: SegRNN replaces point-wise iterations with segment-wise iterations and RMF with PMF to reduce recurrent iterations in LTSF.Segment-wise iterations reduce encoding iterations, while PMF further reduces decoding iterations and mitigates error accumulation associated with RMF.
- Conclusion: SegRNN outperformed current SOTA models in prediction accuracy while reducing training time and memory usage by over 78%.The conclusion attributes these efficiency improvements to the two iteration-reduction strategies.
- Conclusion: The reported outcomes provide evidence that RNNs remain capable of strong performance in LTSF tasks.The paper encourages further exploration of RNN-based methods.
A Appendix
The appendix provides additional experimental material beyond the main results, including further details, univariate evaluations, convergence analyses, and positional-embedding analysis.
- Appendix: The appendix includes additional experimental details and SegRNN results for univariate LTSF.These materials supplement the main multivariate experiments.
- Appendix: The appendix analyzes convergence across iteration schemes and the role of positional embeddings in PMF.Both analyses examine properties of the proposed iteration and decoding strategies.
A.1 Experimental details
The appendix details the datasets, baselines, training configuration, and parameter meanings used for the experiments. It also reports additional univariate benchmark results for SegRNN.
- Datasets: The experiments use multivariate LTSF datasets including ETTs, Electricity, and Traffic.The dataset descriptions include transformer measurements, customer electricity consumption, and California traffic data.
- Baselines: The baselines include SOTA deep-learning methods spanning Transformer, MLP, CNN, and RNN categories.The listed methods include PatchTST, FEDformer, Informer, TiDE, DLinear, MICN, TimesNet, DeepAR, and GRU.
- Configuration: The appendix defines l_back as the historical look-back length, s_len as the segment window length, and d_model as RNN hidden-variable dimensionality.It also specifies channel positional embeddings, dropout, batch size, and learning rate as configuration parameters.
A.2 Univariate Forecasting Results
On the full ETT benchmarks, SegRNN performed exceptionally well in univariate long-term forecasting, frequently ranking among the strongest baselines.
- SegRNN ranked in the top two on 30 of 32 univariate ETT metrics.
- SegRNN achieved 23 first-place rankings across the evaluated univariate scenarios.
- These results reaffirm the competitiveness of RNN methods for both multivariate and univariate long-term forecasting.
A.3 Convergence of Different Iteration Schemes
Segment-wise iterations improved convergence over point-wise iterations, with larger segments producing stronger gains as recurrent iteration counts decreased.
- Segment-wise iterations improved both final convergence error and convergence speed compared with point-wise iterations.
- As segment length increased, the convergence improvement became more pronounced.
- Reducing recurrent iterations as much as possible facilitated RNN convergence and subsequently enhanced performance.
A.4 Effect of Positional Embeddings
Relative positional encoding was important for PMF, while channel positional encoding provided additional gains but remained limited for highly multivariate traffic data.
- Relative Position encoding reduced MSE by 28.8% compared with the scheme without Relative Position encoding.
- Relative positional information was essential because Parallel Multi-step Forecasting loses sequential order between segments.
- Channel Position encoding improved performance, but simple encoding may be insufficient for Traffic with over 800 variables.