Source-linked AI summary
DSformer: A Double Sampling Transformer for Multivariate Time Series Long-term Prediction
Chengqing Yu, Fei Wang, Zezhi Shao, Tao Sun, Lin Wu, Yongjun Xu
TL;DR
Long-term multivariate time-series prediction is important for understanding future changes, but transformer models often underuse global information, local information, and variable correlations. DSformer addresses this gap with double sampling and temporal-variable attention, and experiments report state-of-the-art performance on nine real-world datasets.
Problem
Transformer-based predictors do not fully exploit global information, local information, and variable correlations in multivariate time series, creating challenges for observing and integrating these features.
Method
DSformer uses double sampling for global and local feature extraction, then parallel temporal and variable attention blocks to mine and integrate these features.
Results
DSformer achieves state-of-the-art multivariate time-series long-term prediction performance on nine real-world datasets and outperforms eight existing state-of-the-art models.
Takeaways & Limitations
The experiments support DSformer as an efficient model for integrating global information, local information, and variable correlations in long-term prediction.
Abstract
from arXiv · showhide
Multivariate time series long-term prediction, which aims to predict the change of data in a long time, can provide references for decision-making. Although transformer-based models have made progress in this field, they usually do not make full use of three features of multivariate time series: global information, local information, and variables correlation. To effectively mine the above three features and establish a high-precision prediction model, we propose a double sampling transformer (DSformer), which consists of the double sampling (DS) block and the temporal variable attention (TVA) block. Firstly, the DS block employs down sampling and piecewise sampling to transform the original series into feature vectors that focus on global information and local information respectively. Then, TVA block uses temporal attention and variable attention to mine these feature vectors from different dimensions and extract key information. Finally, based on a parallel structure, DSformer uses multiple TVA blocks to mine and integrate different features obtained from DS blocks respectively. The integrated feature information is passed to the generative decoder based on a multi-layer perceptron to realize multivariate time series long-term prediction. Experimental results on nine real-world datasets show that DSformer can outperform eight existing baselines.
1 INTRODUCTION
Multivariate time-series long-term prediction matters for understanding future trends, but existing transformer-based models do not fully exploit global information, local information, and variable correlations. DSformer addresses these challenges through double sampling and temporal-variable attention, outperforming eight existing state-of-the-art models on nine real-world datasets.
- Long-term multivariate time-series prediction supports understanding future trends for decision-making across fields including weather, energy, economics, environment, and traffic.
- Transformer-based models analyze sequential context but do not fully use global information, local information, and variable correlations.
- DSformer uses down-sampling and piecewise sampling to derive feature vectors emphasizing global and local information, respectively.
- Temporal attention captures temporal context, while variable attention analyzes correlations among variables; their parallel structure integrates these features.
- DSformer combines the double-sampling block with multiple TVA blocks to learn and integrate global information, local information, and variable correlations.
- DSformer outperforms eight existing state-of-the-art models in comparative experiments on nine real-world datasets.
2 RELATED WORK
Related work covers recurrent and hybrid neural architectures before turning to transformer variants for multivariate time-series long-term prediction. These approaches improve sequence modeling through mechanisms such as attention, convolution, hashing, and residual connections.
- RNNs, LSTMs, and GRUs are established deep-learning approaches studied for multivariate time-series long-term prediction.
- LSTMa combines LSTM with attention, while LSTNet fuses LSTM, CNN, and attention for multivariate time-series long-term prediction.
- Transformer variants have grown rapidly because of their series-modeling capabilities in multivariate time-series long-term prediction.
- Log-Trans uses convolutional self-attention, while another transformer approach combines locality-sensitive hashing attention with reversible residual layers.
3 METHODOLOGY
DSformer models multivariate time series by combining double sampling with parallel temporal-variable attention, targeting global information, local information, and variable correlations.
- Information integration: The two sampled feature tensors are processed by TVA components and then integrated to jointly model global information, local information, and variable correlation.Temporal and variable attention receive the double-sampling outputs as inputs before feature fusion.
- Double sampling block: Downsampling forms subsequences with larger time intervals to emphasize global information while retaining them together for subsequent modeling.The method uses C consistent-length subsequences and represents them as a 3D feature vector.
- Double sampling block: Piecewise sampling divides the historical series into contiguous subsequences so the model can focus on local changes and details.The resulting subsequences are also combined into a 3D feature vector to avoid information loss.
- TVA block: Temporal attention mines context from the temporal dimension, whereas variable attention analyzes implicit relations among variables.The TVA block integrates information from both components through a parallel structure.
- Overall framework: DSformer combines a double sampling block with three temporal variable attention blocks in a parallel modeling framework.The architecture is designed to mine and integrate multiple feature representations.
4 EXPERIMENT AND ANALYSIS
Experiments evaluate DSformer on nine real-world datasets against eight baselines, then analyze its components, hyperparameters, sampling settings, and efficiency. The results support the importance of modeling global information, local information, and variable correlation together.
- Experimental design: Experiments use nine datasets, eight SOTA baselines, and MAE and MSE as the main evaluation metrics.The datasets include ETT, Exchange, ILI, Weather, Electricity, and Traffic; the baselines include PatchTST, Crossformer, TimesNet, DLinear, FEDformer, Pyraformer, Autoformer, and Informer.
- Main results: Compared with other transformer variants, DSformer is evaluated for using global information, local information, and variable correlation together.The reported comparison discusses limitations of methods that focus on only some of these features.
- Ablation experiments: Removing variable attention increases prediction error when variables are correlated, especially on the Weather dataset.The ablation discussion attributes this effect to the large correlation among Weather time series.
- Ablation experiments: Long prediction steps benefit from down sampling, whereas short prediction steps benefit from piecewise sampling.Down sampling emphasizes global information, while piecewise sampling emphasizes local information.
- Hyperparameter analysis experiments: Learning rate and sampling interval strongly affect performance, while the number of multi-head attention heads and loss weight have relatively little impact.Smaller sampling intervals work better for shorter prediction steps, whereas larger intervals can work better for longer steps.
- Sampling interval and history length: Balancing history length with sampling interval can improve prediction accuracy by combining broader historical information with global and local features.With short history, an overly large sampling interval degrades performance; with long history, increasing the interval can improve it.
5 CONCLUSION AND FUTURE WORK
DSformer combines double sampling and temporal variable attention to mine global information, local information, and variable correlation for multivariate time-series long-term prediction. Experiments on nine real-world datasets report state-of-the-art performance, while future work targets adaptive balancing of sampling interval and history length.
- Conclusion: DSformer contains DS and TVA blocks for multivariate time-series long-term prediction.The DS block mines global and local information, while the TVA block integrates these features with variable correlation.
- Conclusion: DSformer achieves state-of-the-art performance for multivariate time-series long-term prediction on nine real-world datasets.
- Future Work: Future work will design a module to adaptively balance sampling interval and history length.The stated goal is to further improve information mining and long-term prediction.