Source-linked AI summary
SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction
Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, Qiang Xu
TL;DR
Time-series forecasting models may overlook temporal structure preserved by downsampling and may inadequately represent complex dynamics. SCINet addresses this with recursive downsample-convolve-interact processing and reports consistent gains over existing forecasting methods across real-world datasets.
Problem
Generic forecasting models do not consider that temporal relations are largely preserved when time series are downsampled into two sub-sequences, limiting specialized multiresolution modeling.
Method
SCINet recursively downsamples sequences, applies distinct convolutional filters to sub-sequences, and exchanges information between them across hierarchical temporal resolutions.
Results
SCINet consistently outperforms existing time-series forecasting approaches across various real-world datasets and achieves state-of-the-art performance in most benchmarks and settings.
Takeaways & Limitations
SCINet effectively models complex temporal dynamics by combining features extracted and exchanged across multiple temporal resolutions.
Takeaways & Limitations
SCINet focuses on regular deterministic time series and may be affected by missing data beyond a threshold or by irregular sampling intervals.
Abstract
from arXiv · showhide
One unique property of time series is that the temporal relations are largely preserved after downsampling into two sub-sequences. By taking advantage of this property, we propose a novel neural network architecture that conducts sample convolution and interaction for temporal modeling and forecasting, named SCINet. Specifically, SCINet is a recursive downsample-convolve-interact architecture. In each layer, we use multiple convolutional filters to extract distinct yet valuable temporal features from the downsampled sub-sequences or features. By combining these rich features aggregated from multiple resolutions, SCINet effectively models time series with complex temporal dynamics. Experimental results show that SCINet achieves significant forecasting accuracy improvements over both existing convolutional models and Transformer-based solutions across various real-world time series forecasting datasets. Our codes and data are available at https://github.com/cure-lab/SCINet.
1 Introduction
SCINet is proposed as a hierarchical downsample-convolve-interact framework that uses time-series structure to model complex temporal dynamics. Experiments report consistent improvements over existing forecasting approaches across real-world datasets.
- Motivation: Time-series forecasting supports decisions based on estimated future metrics or events in healthcare, energy management, traffic flow, and financial investment.These applications motivate accurate forecasting methods.
- Motivation: Time-series temporal relations, including trends and seasonal components, are largely preserved when data are downsampled into two sub-sequences.This motivates extracting dynamic temporal features at multiple resolutions.
- Contribution: SCINet is introduced as a hierarchical downsample-convolve-interact framework for time-series modeling and forecasting.The framework iteratively extracts and exchanges information at multiple temporal resolutions.
- Architecture: SCI-Block downsamples inputs into two sub-sequences, applies distinct convolutional filters, and uses interactive learning to compensate for downsampling information loss.The resulting features are exchanged between the two convolutional branches.
- Results: SCINet consistently outperforms existing time-series forecasting approaches across various real-world datasets.It also achieves competitive accuracy on spatial-temporal forecasting tasks despite not explicitly modeling spatial relations.
2 Related Work and Motivation
Existing forecasting models include recurrent, Transformer-based, and temporal convolutional architectures, but generic models may overlook time-series-specific structure. The paper motivates SCINet by identifying limitations in TCN feature diversity and intermediate receptive fields.
- Problem Definition: Time-series forecasting predicts a future horizon from a fixed-length historical window of past observations.The input contains d variates, and the forecast horizon has length τ.
- Related Work: RNN-based forecasting is restricted by gradient vanishing or exploding problems and inefficient training procedures.RNNs recursively update internal memory states with new inputs.
- Related Work: Transformer-based forecasting methods leverage self-attention for long-sequence modeling and commonly target long-term forecasting.The passage describes their effectiveness and efficiency in sequence modeling.
- Temporal Convolution: TCNs use convolutional filters to capture local temporal features and dilated causal convolutions to obtain large receptive fields with few layers.TCNs have been widely applied to time-series forecasting.
- TCN Limitations: TCN limitations include one shared convolutional filter per layer and limited intermediate receptive fields that can cause temporal relation loss.Complex time series therefore require richer feature extraction across temporal dynamics.
- Motivation: These TCN limitations motivate the SCINet design.The proposed method addresses the stated modeling deficiencies with its downsample-convolve-interact structure.
3 SCINet: Sample Convolution and Interaction Network
SCINet uses an encoder-decoder architecture built from hierarchical SCI-Blocks that split, convolve, and interact across temporal resolutions. Its multiresolution representation captures short- and long-term dependencies while maintaining relatively low computational complexity.
- Architecture: SCINet’s encoder hierarchically processes downsampled sub-sequences with rich convolutional filters and interactive learning, then decodes enhanced representations for forecasting.SCI-Blocks are arranged in a binary tree, and the decoder is fully connected.
- SCI-Block: SCI-Block splits an input feature into even and odd sub-sequences at coarser resolution while preserving most of the original information.The split is performed by separating even and odd elements.
- SCI-Block: Separate convolutional kernels extract distinct temporal relations from the two sub-sequences, while interactive learning exchanges information between them.The interaction learns affine scaling parameters from each branch using convolutional modules.
- Design Advantages: Compared with TCN, SCINet provides larger receptive fields at each convolutional layer and aggregates local and global temporal views.Its rich filters avoid the feature-extraction restriction of a single shared TCN filter.
- Multiresolution Modeling: Hierarchical levels accumulate information across resolutions, enabling SCINet to capture both short-term and long-term temporal dependencies.Deeper features receive finer-scale information transmitted from shallower levels.
- Decoding: The rearranged features are added to the original sequence through a residual connection before fully connected decoding.This produces a sequence representation described as having enhanced predictability.
- Configuration: L≤5 generally achieves the best forecasting accuracy in empirical studies, even with T=168, while K≤3 is reported as sufficient.Stacking more SCINets can improve accuracy when training samples are sufficient, but increases model complexity.
- Complexity: O(T log T) worst-case time complexity is reported for SCINet, compared with O(T^2) for vanilla Transformer-based solutions.Its computational cost is usually on par with TCN.
4 Experiments
Experiments across short-term, long-term, multivariate, univariate, and spatial-temporal forecasting tasks show that SCINet generally outperforms established RNN, TCN, Transformer, and graph-based approaches. Ablation results support the importance of interactive learning and distinct convolutional weights, while lower permutation entropy accompanies enhanced representations.
- Experimental setup: Experiments cover 11 real-world datasets under multivariate time-series forecasting settings, using established experimental settings and evaluation metrics for fair comparison.The datasets include ETTh, Traffic, Solar-Energy, Electricity, Exchange-Rate, and four PeMS datasets.
- Short-term forecasting: SCINet outperforms RNN/TCN-based and Transformer-based solutions in most short-term forecasting cases, especially on Solar-Energy and Exchange-Rate.Short-term experiments use an input length of 168 and forecast horizons of 3, 6, 12, and 24.
- Long-term forecasting: 39.89% average MSE improvement is achieved by SCINet across the reported long-term forecasting settings, including 65% average improvement on Exchange-Rate over previous state-of-the-art results.SCINet achieves state-of-the-art performance in most benchmarks and prediction-length settings.
- ETT forecasting: SCINet outperforms the compared models by a large margin on multivariate ETT forecasting and obtains trend and seasonality patterns in qualitative evaluations.The discussion contrasts SCINet with RNN, Transformer, and TCN approaches.
- Spatial-temporal forecasting: SCINet achieves better performance than graph-based spatial-temporal methods on PeMS datasets without explicitly modeling spatial relations.The result is presented as evidence of strong temporal modeling capabilities.
- Representation analysis: Enhanced SCINet representations have lower permutation entropy than original inputs, although lower permutation entropy alone does not guarantee easier prediction across different time series.Prediction accuracy also depends on training data, trend and seasonality, and the predictive model.
- Ablation study: Interactive learning and distinct weights both improve prediction accuracy, with interactive learning more effective for longer look-back windows.The ablation comparison attributes the longer-window benefit to exchanging information between downsampled sub-sequences.
5 Limitations and Future Work
The paper identifies scope limitations involving irregular or incomplete time series, deterministic forecasting, and spatial-temporal modeling. It proposes addressing these boundaries in future SCINet development.
- SCINet may be affected by missing data above a certain ratio because downsampling can introduce biases and poor prediction performance.
- The downsampling mechanism may have difficulty handling data collected at irregular time intervals.
- The work focuses on deterministic forecasting, although many application scenarios require probabilistic forecasts.
- Spatial-temporal forecasting accuracy could be further improved by incorporating dedicated spatial models.
6 Conclusion
The paper proposes SCINet, a hierarchical downsample-convolve-interact architecture motivated by time series properties. Experiments on real-world datasets demonstrate superiority over state-of-the-art methods.
- SCINet hierarchically downsamples, convolves, and exchanges information across temporal resolutions to learn representations with enhanced predictability.
- Extensive experiments on various real-world time series forecasting datasets demonstrate SCINet’s superiority over state-of-the-art methods.
Checklist
The checklist records documentation and reproducibility responses for the paper, including limitations, experimental details, assets, and ethics-related items.
- The checklist states that code and datasets are proprietary in one asset-license response.
- The authors state that the paper describes its limitations in Section 5.
- The paper reports that code, data, training details, and compute information are provided in the appendix or linked materials, but error bars were not reported.
- The checklist indicates that participant instructions, risks, compensation, and personally identifiable information considerations were not applicable.
- The appendix introduces datasets, evaluation metrics, experimental results, network design, training schemes, and hyper-parameter tuning details.
A Datasets and Evaluation Metrics
The experiments use multiple time series datasets with dataset-specific preprocessing, forecasting horizons, data splits, and evaluation metrics. Additional studies examine parameter and operator choices in SCI-Block.
- Experiments cover 11 popular time series datasets, including electricity transformer temperature and traffic data.
- For some datasets, zero-mean normalization is used with MAE and MSE, and train, validation, and test sets contain 12, 4, and 4 months.
- Another dataset uses zero-mean normalization with RMSE and MAPE as evaluation metrics.
- For specified datasets, the look-back window is 168 and future horizons are τ = 3, 6, 12, and 24, evaluated with RSE and CORR.
- Additional experiments study error bars and the effects of parameter and operator combinations in SCI-Block on ETTh1 and PEMS.
B.1 Error Bars Evaluation
SCINet’s ETTh1 results were evaluated across five runs to assess sensitivity to random initialization. The reported standard deviations were small relative to the means, indicating robust performance across initializations.
- 2% to 3% of the mean values: the standard deviations across five ETTh1 runs indicate SCINet is robust to random initialization.
B.2 Evaluation on the Impact of K and L
The study evaluates how SCINet’s stack count K and level count L interact with look-back window size T. Results show a trade-off between K and L, while larger windows benefit from deeper levels.
- Larger L improves prediction accuracy for larger look-back windows T = 128 or 192 when K = 1.Deeper levels can extract information from coarser temporal resolutions when T is large.
- Increasing K improves prediction accuracy for small look-back windows T = 24 or 48 when L = 3.
- The results show a clear trade-off between the number of levels L and stacks K.
- Performance varies with T, highlighting the importance of selecting an appropriate look-back window for forecasting.The prediction horizon in these experiments is fixed at 24.
- L ≤5 and K ≤3 are usually sufficient, and tuning these hyperparameters generally requires limited effort.
- Addition and subtraction are both available interaction operators, and their choice has only a minor performance impact.The model can learn the operation adaptively during training, although parameter initialization affects the final performance.
C Reproducibility
The reproducibility details describe the implementation environment, network modules, training procedure, hyperparameter selection, and dataset-specific loss choices. They also document architectural and supervision details for reproducing the experiments.
- All experiments were conducted with PyTorch on an Nvidia Tesla V100 SXM2 GPU with 32GB memory.
- The four SCI-Block modules φ, ρ, ψ, and η share the same convolutional network architecture.The architecture uses replication padding, two 1D convolutional layers, LeakyRelu, and Dropout.
- For short-term single-step forecasting, the final SCINet in a stacked model uses a revised loss function with two parts.
- Training uses random seed 4321 and at most 150 epochs, with test results reported from the model performing best on validation data.
- Essential hyperparameters are selected through grid search on held-out validation sets, with configurations reported separately for ETT, Traffic, Solar-energy, Electricity, Exchange-rate, and PeMS datasets.
- Weighted loss is applied only to Solar and Exchange-rate data, while a non-causal TCN is obtained by removing chomps and matching padding to dilation.
- For single-step forecasting, all available values in the prediction window are used as supervision signals.