Source-linked AI summary
MSGNet: Learning Multi-Scale Inter-Series Correlations for Multivariate Time Series Forecasting
Wanlin Cai, Yuxuan Liang, Xianggen Liu, Jianshuai Feng, Yuankai Wu
TL;DR
Multivariate forecasting remains limited by insufficient modeling of inter-series correlations that vary across time scales. MSGNet uses frequency-based scale discovery, adaptive graph convolution, and attention to model these dependencies, and experiments report improved forecasting accuracy and generalization.
Problem
Existing deep learning models often fail to capture diverse, time-varying inter-series correlations, including correlations tied to distinct time scales.
Method
MSGNet uses periodicity identified with FFT, adaptive graph convolution for inter-series correlations, and multi-head attention for intra-series correlations.
Results
MSGNet consistently outperforms existing deep learning models on real-world forecasting tasks and shows better generalization capability.
Takeaways & Limitations
The findings underscore the importance of discerning varying inter-series correlations at different time scales in time series analysis.
Takeaways & Limitations
The formulation assumes that varying inter-series correlations can be represented by graphs at selected time scales.
Abstract
from arXiv · showhide
Multivariate time series forecasting poses an ongoing challenge across various disciplines. Time series data often exhibit diverse intra-series and inter-series correlations, contributing to intricate and interwoven dependencies that have been the focus of numerous studies. Nevertheless, a significant research gap remains in comprehending the varying inter-series correlations across different time scales among multiple time series, an area that has received limited attention in the literature. To bridge this gap, this paper introduces MSGNet, an advanced deep learning model designed to capture the varying inter-series correlations across multiple time scales using frequency domain analysis and adaptive graph convolution. By leveraging frequency domain analysis, MSGNet effectively extracts salient periodic patterns and decomposes the time series into distinct time scales. The model incorporates a self-attention mechanism to capture intra-series dependencies, while introducing an adaptive mixhop graph convolution layer to autonomously learn diverse inter-series correlations within each time scale. Extensive experiments are conducted on several real-world datasets to showcase the effectiveness of MSGNet. Furthermore, MSGNet possesses the ability to automatically learn explainable multi-scale inter-series correlations, exhibiting strong generalization capabilities even when applied to out-of-distribution samples.
Introduction
Existing forecasting models often miss inter-series correlations that vary across time scales. MSGNet addresses this gap by combining scale discovery, adaptive graph convolution, and multi-head attention, with experiments showing stronger forecasting performance and generalization.
- Inter-series correlations can vary across time scales, but existing deep learning models often fail to describe them accurately.
- MSGNet uses FFT to project time series into spaces associated with prominent time scales.The method treats periodicity as the source of time-scale information.
- MSGNet captures multi-scale inter-series correlations through a novel structure designed to discover and model them efficiently.
- A combination of multi-head attention and adaptive graph convolution models intra-series and inter-series correlations simultaneously.
- MSGNet consistently outperforms existing deep learning models on real-world forecasting tasks and exhibits better generalization capability.
Related Works
Related work spans classical and deep learning approaches for intra-series forecasting and increasingly uses GNNs to learn inter-series correlations. MSGNet’s architecture combines multi-scale identification with graph-based and attention-based modeling.
- Time Series Forecasting: Classical methods such as VAR and Prophet assume that intra-series variations follow predefined patterns.The paper notes that real-world series can exhibit more complex variations.
- Time Series Forecasting: Deep learning approaches include MLPs, TCNs, RNNs, and Transformer-based models for time series analysis.
- GNNs for Inter-series Correlation Learning: GNNs have been increasingly used to learn inter-series correlations and improve short-term time series prediction.
- GNNs for Inter-series Correlation Learning: MSGNet uses ScaleGraph blocks combining FFT-based scale identification, adaptive graph convolution, and multi-head attention.
Problem Formulation
The forecasting task uses a retrospective window of multivariate observations to predict future values for all variables. The formulation represents inter-series dependencies at multiple time scales as separate graph adjacency matrices.
- Given N variables and a retrospective window Xt−L:t ∈ R^N×L, the task predicts future values across T time steps.
- The predicted sequence is represented as X̂t:t+T ∈ R^N×T, containing values for all variables from t through t + T − 1.
- At each time scale si, inter-series dependencies are represented by a graph Gi with N nodes and weighted edges.
- For k time scales, the formulation identifies k adjacency matrices that capture varying inter-series correlations across scales.
Methodology
MSGNet processes time series across periodic scales, learning scale-specific inter-series dependencies while modeling intra-series correlations and aggregating scale representations for forecasting.
- Scale Learning and Transformation: MSGNet identifies prominent periodicities with FFT and reshapes inputs into representations for multiple time scales.The model treats evolving periodicities as scale sources and uses zero-padding before reshaping.
- Multiple Graph Convolution: At each scale, a learnable projection maps representations back to N variables before adaptive graph convolution captures inter-series correlations.The scale-specific projection uses Wi ∈ R^N×dmodel, while the adaptive adjacency matrix is generated from trainable parameter matrices.
- Multiple Graph Convolution: Mixhop graph convolution combines information across integer powers of each learned adjacency matrix, then an MLP projects the output back into a scale tensor.The set P controls adjacency powers, and column-level connections fuse intermediate outputs.
- Temporal Multi-head Attention: Multi-head attention models intra-series correlations within each scale, with scale transformation converting long time spans into periodic lengths.The authors report that MSGNet maintains performance consistently as input length increases.
- Scale Aggregation and Forecasting: The model reshapes scale outputs, weights scales by FFT-derived amplitudes through SoftMax, and aggregates them before forecasting.Linear projections along variable and time dimensions transform the aggregated representation into N variables over forecast horizon T.
- Scale Aggregation and Forecasting: MSGNet uses linear projections along variable and time dimensions to produce forecasts from the aggregated representation.Ws projects across variables, Wt across time, and T denotes the forecast horizon.
Experiments
MSGNet is evaluated across eight datasets against transformer, graph-convolution, linear, and periodic-decomposition baselines. It achieves strong forecasting accuracy, learns scale-specific airport relationships, and retains performance under an altered COVID-19 data partition.
- Experimental Setup: Experiments cover eight datasets and compare MSGNet with Informer, Autoformer, MTGnn, DLinear, NLinear, and TimesNet.The evaluation uses Flight, Weather, four ETT variants, Exchange-Rate, and Electricity.
- Forecasting Results: MSGNet achieves the best average MSE on five datasets and the second-best performance on two datasets.Across datasets, its average rank also exceeds those of the competing models.
- Forecasting Results: On Flight, MSGNet reduces average MSE by 21.5% from 0.265 to 0.208 and average MAE by 13.7% from 0.372 to 0.321 versus TimesNet.TimesNet is identified as the current state-of-the-art baseline in this comparison.
- Forecasting Results: Figure 3 shows MSGNet closely tracking ground-truth flight values, while other models exhibit pronounced dips during specific periods.The discussed peaks and troughs align with flight-data events, trends, or periodic dynamics.
- Learned Correlations: MSGNet learns distinct adjacency matrices at 24-hour, 6-hour, and 4-hour scales, with airport influences changing across these scales.Airport 6 has stronger influence on Airports 0, 1, and 3 at 24 hours, whereas nearby airports show stronger shorter-scale interactions.
- Ablation Analysis: Ablations show that removing graph learning substantially decreases performance, while multi-scale graph learning significantly improves it.Removing multi-head attention yields marginal gains, and replacing mix-hop convolution makes performance slightly worse.
- Generalization: Under a 4:4:2 Flight partition, MSGNet remains best, shows the least degradation, and demonstrates robust generalization to out-of-distribution samples.The altered split retains the same test set while restricting training data to before the epidemic outbreak.
Conclusion
The paper presents MSGNet as a framework for learning diverse inter-series correlations across time scales by using periodicity as the source of scale information. Experiments on real-world datasets show improved forecasting accuracy and capture of intricate multivariate interdependencies.
- Conclusion: MSGNet uses periodicity as the source of time scales to capture diverse inter-series correlations.The framework is designed to address limitations in existing deep learning models for time series analysis.
- Conclusion: Experiments on various real-world datasets demonstrate that MSGNet outperforms existing models in forecasting accuracy.The conclusion also reports that the model captures intricate interdependencies among multiple time series.
- Conclusion: The findings underscore the importance of distinguishing varying inter-series correlations at different time scales.
Appendix 1 A Mixture-of-Experts Perspective of MSGNet
The appendix interprets MSGNet’s multi-scale graph convolution as a mixture-of-experts system. Fourier-derived, input-dependent gates combine graph-convolution experts associated with different scales, generalizing a single-graph model.
- Mixture-of-Experts Perspective: Mixture-of-experts models train specialized expert functions and combine their outputs using a gating function.The gating function determines each expert’s relative contribution to the final decision.
- Multi-Scale Graph Convolution: MSGNet’s multi-scale graph convolution computes graph-convolution outputs for multiple adjacency matrices and combines them through scale-specific gates.The appendix presents this as a simplified form of the multi-scale graph convolution.
- Multi-Scale Graph Convolution: The gates depend on the input because they are computed from the amplitudes of the time series’ Fourier transformation.
- Single- versus Multi-Expert View: Setting k = 1 reduces the multi-scale graph convolution to Z = H1, equivalent to a single expert model.With multiple graph convolutions, the construction instead combines several expert-like outputs.
2 Representation Power Analysis
The analysis formalizes multivariate time series as graph signals and shows that fixed graph models cannot represent certain multi-hop inter-series features. MSGNet addresses this limitation by learning multiple graph structures across scales with Mixhop convolution.
- Multivariate time series can be represented as graph signals whose nodes are individual time series and whose adjacency matrix encodes their correlations.
- The model defined by Equation 15 cannot represent two-hop Delta Operators.The proof constructs distinct graph operators that produce the same output, violating injectivity.
- Methods lacking advanced graph modeling are constrained to learn fixed inter-series correlation patterns, reducing generalizability when those patterns change.
- MSGNet uses Mixhop to learn multiple graph structures at different scales and represent both Two-hop Delta Operator and layer-wise neighborhood-mixing features.
3 More Details on Experiments
The experiments use diverse real-world datasets, including COVID-19-affected European flight data, and evaluate MSGNet across partitions and hyperparameter settings. Results indicate stable performance, with smaller Mixhop orders and larger scale counts generally favored.
- Datasets: The Flight dataset covers seven major European airports from January 2019 to December 2021 and includes flight changes associated with COVID-19.
- Datasets: COVID-19 caused substantial disruption in flight operations, making the Flight dataset suitable for assessing stability on out-of-distribution data.
- Hyper-Parameters: MSGNet experiments evaluate MSE across Flight, ETTh1, ETTh2, and Weather for prediction lengths {96, 192, 336, 720}, varying k and Mixhop order from 1 to 5.
- Hyper-Parameters: MSGNet maintains consistent performance across k and Mixhop-order selections, with Mixhop order 2 generally producing improved performance.Flight is comparatively insensitive to Mixhop order.
- Hyper-Parameters: Increasing k generally enhances predictive performance by broadening the learned inter-series correlations and enabling more diverse predictions.
4 The detected scale
MSGNet detects multiple salient frequency-derived scales and uses them to represent temporal dependencies. On Flight, the detected scales correspond to recognizable daily and sub-daily flight patterns.
- For Flight predictions over 96 time steps, MSGNet consistently captures diverse scales including 1 day, half day, and morning patterns.
- MSGNet identifies the k most significant frequencies and normalizes their proportions to obtain scale distributions.
- The detected Flight patterns resemble real-world flight behavior and support learning time dependencies close to reality.
5 Performances under Longer Input Sequences
The section examines forecasting with longer review windows, where MSGNet avoids the strong fluctuations seen in other models and generally improves as the window extends. Scale transformation shortens long sequences before Transformer processing.
- Longer review windows change which historical dependencies models can learn, while prior Transformer-based models often become less stable as windows lengthen.
- MSGNet’s Flight evaluation forecasts the next 336 time steps across review windows from 48 to 720, using MSE.
- MSGNet shows an overall decreasing error trend with longer review windows and avoids the significant rebound and strong fluctuations seen in other models.
- Scale transformation reshapes a length-720 sequence with period 24 into a 24 × 30 scale tensor, allowing the Transformer to operate on length 24.
- On ETT datasets, extending the review window improves MSGNet’s performance, supporting scale transformation for longer inputs.
6 Computational Efficiency
MSGNet is evaluated for computational efficiency and forecasting behavior across datasets and prediction settings. It is more efficient than TimesNet in the reported comparison, while remaining computationally heavier than Dlinear and Autoformer.
- Efficiency comparison: MSGNet surpassed TimesNet in operational efficiency on the Electricity dataset evaluation.The comparison considered GPU memory usage, running speed, and MSE ranking across prediction lengths.
- Scaling behavior: MSGNet’s computational complexity remains unchanged as input time increases when the number of scales is fixed.Its multi-head attention operates on short time scales, while graph convolution depends on the number of scales.
- Efficiency trade-offs: MSGNet is computationally heavier than Dlinear and Autoformer.The paper identifies Dlinear’s simpler linear structure and Autoformer’s full-sequence attention as relevant comparison points.
- Prediction visualization: MSGNet better fits trend changes and periodicity than other models in the Flight and ETTm2 visual showcases.The showcases use input length 96 with output length 96 for Flight and output length 336 for ETTm2.