PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Scale Adaptive Graph Neural Network for Multivariate Time Series Forecasting

Ling Chen, Donghui Chen, Zongjiang Shang, Binqing Wu, Cen Zheng, Bo Wen, Wei Zhang

arXiv:2201.04828v2cs.LG

TL;DR

MTS forecasting must handle multi-scale temporal patterns alongside complex intra- and inter-variable dependencies, which single-scale or shared-graph approaches do not fully represent. MAGNN combines hierarchical multi-scale representations with scale-specific adaptive graphs, temporal graph modeling, and scale-wise fusion, and it outperforms state-of-the-art methods across six datasets.

  • Problem

    Existing methods often learn temporal patterns using a single inter-variable dependency structure, although real-world MTS contain multi-scale temporal patterns with potentially different dependencies.

  • Method

    MAGNN uses a multi-scale pyramid, adaptive scale-specific graph learning, multi-scale temporal graph neural networks, and scale-wise fusion to model temporal and inter-variable dependencies.

  • Results

    MAGNN outperforms state-of-the-art methods on six real-world MTS datasets.

  • Takeaways & Limitations

    MAGNN can capture multi-scale temporal patterns and complicated inter-variable dependencies for accurate MTS forecasting.

Abstract

from arXiv · show

Multivariate time series (MTS) forecasting plays an important role in the automation and optimization of intelligent applications. It is a challenging task, as we need to consider both complex intra-variable dependencies and inter-variable dependencies. Existing works only learn temporal patterns with the help of single inter-variable dependencies. However, there are multi-scale temporal patterns in many real-world MTS. Single inter-variable dependencies make the model prefer to learn one type of prominent and shared temporal patterns. In this paper, we propose a multi-scale adaptive graph neural network (MAGNN) to address the above issue. MAGNN exploits a multi-scale pyramid network to preserve the underlying temporal dependencies at different time scales. Since the inter-variable dependencies may be different under distinct time scales, an adaptive graph learning module is designed to infer the scale-specific inter-variable dependencies without pre-defined priors. Given the multi-scale feature representations and scale-specific inter-variable dependencies, a multi-scale temporal graph neural network is introduced to jointly model intra-variable dependencies and inter-variable dependencies. After that, we develop a scale-wise fusion module to effectively promote the collaboration across different time scales, and automatically capture the importance of contributed temporal patterns. Experiments on four real-world datasets demonstrate that MAGNN outperforms the state-of-the-art methods across various settings.

I. INTRODUCTION

MTS forecasting must capture nonlinear temporal and inter-variable dependencies across multiple time scales. MAGNN addresses this with scale-specific graph learning, multi-scale temporal modeling, and cross-scale fusion.

  • Motivation: MTS forecasting jointly requires modeling intra-variable temporal dependencies and inter-variable dependencies from other variables.Traditional methods often rely on strict stationarity and cannot capture nonlinear dependencies among variables.
  • Limitations of Existing Methods: Existing approaches often use one time scale or one shared adjacency matrix, which cannot represent distinct temporal patterns and their differing inter-variable dependencies.Real-world series can contain daily, weekly, monthly, and other periodic patterns.
  • MAGNN: MAGNN decomposes time series hierarchically with a multi-scale pyramid network to preserve temporal dependencies at different scales.Its pyramid layers generate representations at progressively larger scales.
  • MAGNN: An adaptive graph learning module infers scale-specific inter-variable dependencies without requiring predefined graph structures.The framework therefore represents different dependencies for different temporal scales.
  • MAGNN: A multi-scale temporal graph neural network models intra-variable and inter-variable dependencies, while scale-wise fusion captures the importance of scale-specific patterns and cross-scale correlations.The fusion module combines representations across scales rather than simply concatenating them.
  • Evaluation: Experiments on six real-world benchmark datasets report better performance than state-of-the-art methods.The reported evaluation covers multiple datasets and forecasting settings.

B. Graph Learning for MTS

Graph-based MTS forecasting treats variables as graph nodes and inter-variable dependencies as edges. Existing graph construction methods use priors, rules, or learned parameters, while the formulation allows multiple learned graphs.

  • Graph Construction Methods: Prior-knowledge methods construct graphs from domain information such as road networks, physical structures, or feature matrices.Their graph designs are difficult to transfer between different scenarios.
  • Graph Construction Methods: Rule-based methods infer edges using causal discovery, entropy, or similarity measures, but each is limited to a specific dependency type.Examples include Granger causality, transfer entropy, Pearson correlation, and dynamic time warping.
  • Graph Construction Methods: Learning-based methods introduce parameterized modules to automatically learn pairwise inter-variable dependencies and can learn multiple graph structures.Neural relational inference is an example of this learning-based strategy.
  • Problem Formulation: Given observed signals X, MTS forecasting predicts future values at a lookahead horizon using a learnable mapping function.The formulation can incorporate a set of graphs G = {G1, G2, ..., GK} for GNN-based forecasting.
  • Graph Modeling: GNN-based MTS forecasting represents variables as graph nodes and pairwise inter-variable dependencies as weighted edges.This graph view lets GNNs exploit structural information while learning node representations.
  • Problem Formulation: Without prior knowledge, multiple weighted adjacency matrices are learned to represent abundant and implicit inter-variable dependencies.The resulting graph set supports forecasting with multiple graph structures.

B. Graph Neural Networks

MAGNN models multivariate time series by combining temporal multi-scale representations with graph-based inter-variable dependencies. Its pyramid network builds hierarchical temporal features, which are processed by scale-aware temporal GNNs and fused for prediction.

  • Graph modeling: GNNs represent MTS variables as graph nodes and inter-variable dependencies as edges, enabling graph-based modeling of pairwise relationships.Graph convolution aggregates information across neighboring nodes and can incorporate multi-order neighbors by stacking layers.
  • Temporal multi-scale modeling: MAGNN focuses on temporal scales, unlike general multi-scale GNNs that construct hierarchical spatial graphs.It transforms raw time series into representations from smaller to larger temporal scales and learns scale-specific graphs.
  • Multi-scale pyramid network: The multi-scale pyramid network hierarchically transforms raw time series into feature representations at progressively larger scales.Pyramid layers use convolutions to capture local temporal patterns, with different kernel sizes controlling receptive fields.
  • Multi-scale pyramid network: A parallel convolutional path and point-wise additions improve flexibility in preserving temporal dependencies across adjacent scales.The additional path uses a 1 × 1 convolution and 1 × 2 pooling to reduce sensitivity to kernel and stride settings.
  • Framework: Scale-specific representations are weighted and fused before two output convolutions produce the predicted values.The framework therefore combines pyramid features, graph processing, and scale-wise fusion in one forecasting pipeline.

C. Adaptive Graph Learning

MAGNN’s adaptive graph learning module infers inter-variable dependencies separately for each temporal scale. It uses shared node information together with scale embeddings and sparsifies the resulting adjacency matrices.

  • Motivation: A shared adjacency matrix can bias models toward one prominent temporal pattern, motivating multiple scale-specific adjacency matrices.The paper argues that distinct temporal patterns may depend on different inter-variable relationships.
  • Scale-specific graph construction: AGL uses shared node embeddings and scale embeddings to generate scale-specific node representations with fewer parameters than independently learning every graph.The design addresses the parameter and training burden of unique adjacency matrices for each scale, especially with many nodes.
  • Scale-specific graph construction: Pairwise node similarities are computed from the scale-specific embeddings to infer the inter-variable relationships at each scale.Learnable parameters obtain receiver and sender node features before similarity-based graph construction.
  • Graph sparsification: The inferred adjacency values are normalized and sparsified with a TopK strategy to reduce computation and noise.The threshold τ controls the maximum number of neighbors retained for a node, yielding scale-specific adjacency matrices.

D. Multi-Scale Temporal Graph Neural Network

MAGNN’s multi-scale temporal graph neural network jointly models temporal patterns and variable interactions at each scale. It combines temporal convolutions with graph convolutions using the corresponding scale-specific adjacency matrices.

  • MTG design: MTG receives multi-scale feature representations and scale-specific adjacency matrices to capture temporal patterns across time steps and variables.Each scale is processed with a temporal graph neural network tailored to its representation and graph structure.
  • MTG design: Each of MTG’s K networks combines temporal convolutional networks with graph neural networks for scale-specific temporal modeling.The architecture replaces recurrent processing with temporal convolutions while retaining graph-based variable interaction modeling.
  • Motivation: Temporal convolution avoids the step-by-step computation and gradient problems associated with recurrent solutions for long time series.The paper motivates TCNs as an efficient alternative to RNN-based graph forecasting approaches.
  • Joint dependency modeling: MTG uses both an adjacency matrix and its transpose in two GNNs to capture incoming and outgoing information.The two GNN outputs are then added for the scale-specific representation.
  • Joint dependency modeling: Graph convolution explicitly incorporates inter-variable dependencies into the temporal representations learned at each scale.MTG is designed to capture scale-specific temporal patterns across both time steps and variables.

E. Scale-Wise Fusion

The scale-wise fusion module combines representations from multiple temporal scales while accounting for their differing contributions and cross-scale correlations.

  • Motivation: Scale-specific representations capture diverse temporal patterns, but direct concatenation or global pooling treats all scales equally.The paper notes that small scales may matter more for short-term forecasting, whereas large scales may matter more for long-term forecasting.
  • Fusion procedure: The module concatenates the scale-specific representations into a multi-scale matrix H ∈ R^K×N×ds.This operation is defined as H = Concat(h1, ..., hk, ..., hK).
  • Fusion procedure: Average pooling over the scale dimension and two fully connected layers refine information across different time scales.The pooled representation is flattened before entering the refining module.
  • Fusion procedure: A sigmoid-generated importance vector α assigns scores to the different scale-specific representations before weighted aggregation produces the final multi-scale representation.The weighted aggregation combines the scale-specific representations into hm.

F. Output Module & Objection Function

The output module transforms the fused representation into predicted values, while the objective and complexity analyses specify training quantities and computational scaling.

  • Output module: A convolution with a 1 × ds kernel transforms hm into the desired output dimension, followed by a 1 × 1 convolution that obtains predicted values.The fused representation is denoted hm ∈ R^N×ds.
  • Objective function: The objective function is formulated over predicted and ground-truth values for each variable and training sample.bxi,j denotes the prediction and xi,j the ground truth for variable j in sample i.
  • Complexity analysis: The multi-scale pyramid network has Θ(N × T) time complexity when channel dimensions are treated as constants.N is the variable dimension and T is the input sequence length.
  • Complexity analysis: The multi-scale temporal graph module has Θ(K × (m + N)) complexity under constant feature dimensions, where K is the number of scales and m the number of edges.This scaling follows from graph message passing and information aggregation.
  • Complexity analysis: The scale-wise fusion module has Θ(N + K) time complexity when its feature dimensions are treated as constants.The unfactored expression is Θ(N × ds × d1 + d1 × K).

V. EXPERIMENTS

MAGNN is evaluated on six public multivariate time-series benchmarks using chronological train-validation-test splits, automated hyperparameter search, and RSE and CORR metrics.

  • Datasets: The evaluation uses Solar-Energy, Traffic, Electricity, Exchange-Rate, Nasdaq, and METR-LA datasets.These benchmarks cover solar power, road occupancy, electricity consumption, exchange rates, stock prices, and traffic speeds.
  • Datasets and settings: The six datasets are split chronologically into 60% training, 20% validation, and 20% test data.The split follows existing works.
  • Experimental settings: NNI automatically searches hyperparameters instead of exhaustively grid-searching all tunable parameters.The implementation uses PyTorch 1.7.1, one NVIDIA RTX 3090 GPU, a 168-step input window, learning rate 0.001, and Adam optimization.
  • Evaluation metrics: Performance is measured with Root Relative Squared Error (RSE) and Empirical Correlation Coefficient (CORR).Lower RSE is better, whereas higher CORR is better.

B. Methods for Comparison

The comparison includes traditional, recurrent, attentive, and graph-learning baselines, with results drawn from prior literature or reproduced using released implementations and tuned settings.

  • Traditional methods: Traditional baselines include AR, TRMF, GP, and VAR-MLP, representing autoregressive, matrix-factorization, Gaussian-process, and hybrid autoregressive-MLP approaches.Their key hyperparameters are selected from specified search spaces.
  • Recurrent and attentive methods: Recurrent and attentive baselines include RNN-GRU, LSTNet, MTNet, and TPA-LSTM.These methods use recurrent, convolutional, memory, skip, or attention components to model temporal patterns.
  • Graph-learning methods: Graph-learning baselines include Graph WaveNet, AGCRN, MTHetGNN, and MTGNN.They use graph convolutions, adaptive graph inference, heterogeneous graph embeddings, or graph learning with dilated convolution.
  • Evaluation comparison: MAGNN is compared with these methods using RSE and CORR results summarized in Tables III and IV.The tables identify the best results in bold and the second-best results with underlining.
  • Evaluation comparison: For four datasets, results for many baselines are adopted from existing literature, while other results are produced from original code and tuned with NNI.This procedure applies to AGCRN, Graph WaveNet, MTHetGNN on four datasets and all baselines on Nasdaq and METR-LA.

C. Main Results

MAGNN achieves state-of-the-art forecasting results across six datasets, with especially strong performance when data exhibit multi-scale temporal and complicated inter-variable dependencies. Ablations and scale experiments support its multi-scale design.

  • MAGNN achieves state-of-the-art results across the six evaluated datasets.
  • On Traffic and Nasdaq, MAGNN outperforms existing methods at every horizon and metric.
  • Traffic contains clear daily and weekly patterns, whereas Exchange-Rate lacks visible multi-scale dependencies and yields slightly worse MAGNN performance.
  • Traditional methods perform worse than deep learning methods because they cannot capture non-stationary and non-linear dependencies.
  • Deep learning methods without explicit pairwise inter-variable modeling generally underperform graph-based methods, although some perform better on Exchange-Rate.
  • MAGNN outperforms MTGNN in 19 of 24 cases and Graph WaveNet in 22 of 24 cases on RSE and CORR.
  • Increasing the number of scales from 2 to 4 significantly improves MAGNN on Traffic by capturing more diversified short-term and long-term patterns.

F. Effect of the Parallel CNNs in the Multi-Scale Pyramid Network

Ablation studies evaluate the multi-scale pyramid, adaptive graph learning, and temporal graph network components. MAGNN generally performs best, supporting parallel CNNs, scale-specific graphs, asymmetric adjacency, and two GNNs.

  • Parallel CNNs: MAGNN performs best in all Traffic cases against the variant without parallel CNNs, indicating more stable extracted multi-scale features.
  • Adaptive graph learning: MAGNN outperforms MAGNN-full, MAGNN-one, and MAGNN-sym in most cases, supporting sparse, multiple scale-specific, and asymmetric graph designs.
  • Adaptive graph learning: Dynamic adjacency is competitive on Solar-Energy and Exchange-Rate but fluctuates dramatically, making stable performance across horizons and datasets difficult.
  • Multi-scale temporal graph neural network: MAGNN outperforms the one-GNN variant in all cases, possibly because two GNNs exploit more hidden complementary information.

I. Effect of Scale-Wise Fusion

MAGNN’s scale-wise fusion generally outperforms direct concatenation, pooling, and attention-based aggregation, while scale importance varies with forecasting horizon. The broader evaluation also reports favorable computation-performance trade-offs and identifies future extensions.

  • Scale-wise fusion: MAGNN achieves the best performance in most cases against concatenation, pooling, and attention-based fusion variants.
  • Scale-wise fusion: Scale-wise fusion considers the importance of scale-specific temporal patterns and captures cross-scale correlations.
  • Scale importance: Small-scale representations matter more for short-term forecasting, while large-scale representations matter more for long-term forecasting.
  • Hyper-parameters: On Traffic, convolutional channel size 32 gives the best performance among sizes from 4 to 128.
  • Computation cost: Compared with TPA-LSTM and MTGNN, MAGNN runs fastest and achieves the best forecasting results on Traffic.
  • Future work: Future work targets dynamic adjacency regularization, neural architecture search, and graph matching to improve scalability and reduce time complexity.
Loading 2201.04828v2…