PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Query Network for Efficient Multivariate Time Series Forecasting

Shengsheng Lin, Haojun Chen, Haijie Wu, Chunyun Qiu, Weiwei Lin

arXiv:2505.12917v2cs.LG

TL;DR

Accurate multivariate time series forecasting requires modeling inter-variable correlations, but non-stationary disturbances can make sample-level relationships unstable. The paper introduces Temporal Query, which uses periodically shifted learnable queries with input-derived keys and values, and builds the lightweight TQNet around single-layer attention and an MLP. Across 12 real-world datasets, TQNet achieves state-of-the-art performance with efficiency comparable to linear-based methods, while its reliance on data periodicity limits generalization to datasets without clear periodic patterns.

  • Problem

    Non-stationary disturbances can distort sample-level inter-variable correlations, creating a gap between local relationships and global dataset-level patterns important for accurate MTSF.

  • Method

    Temporal Query uses periodically shifted learnable vectors as attention queries for global dependencies, while raw input sequences provide keys and values for local relationships; TQNet combines this technique with single-layer attention and a shallow MLP.

  • Results

    TQNet achieves state-of-the-art forecasting performance across 12 challenging real-world datasets and computational efficiency comparable to linear-based methods, including on high-dimensional datasets.

  • Takeaways & Limitations

    TQNet provides a lightweight approach that captures robust inter-variable correlations while balancing forecasting accuracy and computational efficiency.

  • Takeaways & Limitations

    TQNet relies heavily on data periodicity to choose its hyperparameter W, which may limit generalization to datasets without clear periodic patterns.

Abstract

from arXiv · show

Sufficiently modeling the correlations among variables (aka channels) is crucial for achieving accurate multivariate time series forecasting (MTSF). In this paper, we propose a novel technique called Temporal Query (TQ) to more effectively capture multivariate correlations, thereby improving model performance in MTSF tasks. Technically, the TQ technique employs periodically shifted learnable vectors as queries in the attention mechanism to capture global inter-variable patterns, while the keys and values are derived from the raw input data to encode local, sample-level correlations. Building upon the TQ technique, we develop a simple yet efficient model named Temporal Query Network (TQNet), which employs only a single-layer attention mechanism and a lightweight multi-layer perceptron (MLP). Extensive experiments demonstrate that TQNet learns more robust multivariate correlations, achieving state-of-the-art forecasting accuracy across 12 challenging real-world datasets. Furthermore, TQNet achieves high efficiency comparable to linear-based methods even on high-dimensional datasets, balancing performance and computational cost. The code is available at: https://github.com/ACAT-SCUT/TQNet.

1. Introduction

Accurate multivariate forecasting depends on robust inter-variable correlations, yet non-stationary disturbances make sample-level relationships unstable. The paper proposes TQ and TQNet to combine stable global dependencies with local information using a lightweight attention-based architecture.

  • Inter-variable correlations strongly influence multivariate time series forecasting accuracy.
  • Non-stationary disturbances can make individual-sample correlations diverge from global training-set correlations.The cited disturbances include extreme values, missing data, and noise.
  • TQ uses periodically shifted learnable vectors as queries while raw input sequences provide keys and values for global and local correlation modeling.Its learnable parameters support adaptive relationship representations, and periodic shifting facilitates parameter reuse.
  • TQNet combines TQ with a single-layer multi-head attention mechanism and a shallow MLP.The architecture is designed as a simple yet effective multivariate forecaster.
  • 12 challenging real-world multivariate datasets show that TQNet achieves overall state-of-the-art performance while maintaining efficiency comparable to linear-based models.The lightweight architecture remains efficient even on high-dimensional datasets.

2. Related Work

Prior multivariate forecasting methods differ in how they model channel relationships, trading inter-variable capacity, robustness, and modeling complexity. This paper focuses on attention-based channel dependence and introduces TQ to incorporate periodic structure into robust dependency modeling.

  • Channel Mixing (CM): Channel-mixing methods combine multiple variables at each time step and are used by many early Transformer-based forecasting models.
  • Channel Independence (CI): Channel-independence methods forecast each variable separately with a shared model, reducing inter-variable modeling capacity while enhancing robustness.
  • Channel Dependence (CD): Channel-dependence methods model relationships through graph, convolutional, and MLP-based architectures, among other mechanisms.
  • Paper Focus: The paper targets attention-based channel-dependence methods and aims to design a minimalist, efficient MTSF network.
  • Paper Focus: TQ introduces periodically shifted learnable vectors as attention queries to incorporate inherent periodic structures while capturing robust inter-variable relationships.

3. Methodology

TQNet combines temporal-query-enhanced attention for multivariate correlations with a shallow MLP for temporal dependencies, optionally using instance normalization to mitigate distributional drift.

  • TQNet architecture: TQNet forecasts future sequences from historical multivariate inputs using TQ-MHA, an MLP, and a final linear projection.The TQ-MHA captures multivariate correlations, the MLP models temporal dependencies, and the projection maps hidden representations to the forecasting horizon.
  • Temporal Query: TQ initializes learnable parameters over a dataset periodic length and periodically extracts query segments for each input sample.The starting index is t mod W, and a cyclic segment of length L is selected; samples W time steps apart use identical extracted vectors.
  • Temporal Query: Periodic shifting reuses parameters across time and can mitigate localized noisy perturbations when modeling sequence correlations.The method aligns the query cycle with periodic variations in real-world data.
  • TQ-MHA: TQ-MHA uses TQ-derived queries with input-derived keys and values to combine global correlation patterns with sample-specific relationships.This design balances global consistency and local adaptability in modeling inter-variable correlations.
  • MLP and projection: The MLP follows attention to model temporal dependencies and consists of two fully connected layers with GeLU activations.Residual connections enhance learning stability, and the output projection optionally includes Dropout.
  • Instance Normalization: Instance normalization optionally removes input mean and variance before and after model operation to address distributional shifts.The variance term uses a small constant for numerical stability.

4. Experiments

Experiments across 12 real-world datasets evaluate TQNet’s forecasting accuracy, correlation modeling, portability, hyperparameter sensitivity, and computational efficiency. Results show strong accuracy from integrating global and local correlations, with lightweight computation despite a quadratic attention mechanism.

  • Query-Key Analysis: Combining global learnable correlations with local per-sample correlations yields the best average performance on large-scale multivariate datasets.The TQNet configuration uses TQ vectors for queries and raw data for keys, integrating both correlation types.
  • Ablation Study: Removing the TQ module causes the largest performance degradation, while attention remains important for channel-wise information interaction.A pure MLP performs worst because it lacks a mechanism for modeling channel correlations.
  • Integration Study: Adding TQ consistently improves diverse forecasting architectures, indicating that the technique is portable across existing attention-based and channel-independent models.The integration study enhances existing attention mechanisms or embeds a complete TQ-equipped attention module.
  • Representation Learning: Channels close in the TQ t-SNE representation tend to have similar raw sequence patterns, suggesting that learned representations capture intrinsic channel correlations.The paper links this structure to using correlated channels to stabilize predictions under noisy disturbances.
  • Hyperparameter Sensitivity: W aligned with the dataset’s true periodicity performs best, whereas mismatched values such as W = 23 or W = 167 degrade performance through semantic inconsistencies.For the Electricity dataset, weekly periodicity corresponds to W = 168; W = 336 remains competitive as a multiple of the true period.
  • Computational Efficiency: TQNet combines state-of-the-art accuracy with smaller parameter sizes and shorter training times, while maintaining near-linear overhead across channel counts.The attention mechanism has quadratic complexity, but the lightweight design and GPU parallelization keep practical overhead low, particularly when C < 1000.

5. Limitations

TQNet has practical limitations tied to periodicity assumptions, weak multivariate correlations, and long look-back windows. These conditions can reduce generalization or the benefits of multivariate modeling.

  • TQNet relies on inherent data periodicity to determine W, which may limit generalization on datasets without clear periodic patterns.
  • When underlying multivariate correlations are weak or insignificant, strong multivariate modeling may add complexity and negatively affect performance.
  • The benefits of multivariate modeling diminish when the look-back window becomes sufficiently long.Longer windows can provide richer temporal information but also introduce more noise and increase overfitting risk in correlation modeling.
  • Addressing these periodicity, correlation-strength, and look-back challenges is left for future exploration.

6. Conclusion

TQNet combines the Temporal Query technique with a lightweight forecasting architecture to model global inter-variable correlations. It achieves state-of-the-art performance across 12 datasets while retaining efficiency on high-dimensional data.

  • TQNet uses periodically shifted learnable parameters as attention queries to model global inter-variable correlations.
  • TQNet achieves state-of-the-art performance across 12 challenging real-world datasets.
  • TQNet maintains computational efficiency comparable to linear-based methods on high-dimensional datasets with nearly 1,000 variables.
  • TQNet’s pseudocode initializes periodic query parameters and applies attention, linear layers, an MLP, and output projection to produce forecasts.

A.1. Method Details

TQNet processes historical sequences and cycle indices through periodic learnable queries, attention-based multivariate modeling, temporal MLP processing, and forecast projection. Optional instance normalization addresses distributional shifts before restoring the output scale.

  • TQNet takes a historical look-back sequence and cycle index as inputs, then outputs a predicted sequence.
  • The model initializes learnable query parameters whose width W represents the dataset’s prior cycle length.
  • Optional instance normalization removes each sample’s mean and variance before modeling and reverses this transformation on the output.
  • Periodic indices select a segment of the learnable parameters to form the temporal query vectors for each sample.
  • The temporal query vectors serve as attention queries, while the input sequence supplies keys and values; a residual connection produces the attention representation.
  • A linear layer and two-layer GeLU MLP extract nonlinear temporal dependencies with a residual connection before projecting representations to the forecast horizon.

A.2. Experimental Details

The experiments use standard deep-learning training procedures and select W from domain knowledge or autocorrelation-based periodicity estimates. TQNet’s single-period design handles overlapping cycles better than irregularly interwoven periodicities.

  • Experiments use PyTorch, a single NVIDIA GeForce RTX 4090 GPU, Adam optimization, L2 loss, and splits consistent with prior work.
  • TQNet trains for 30 epochs with validation-based early stopping, while learning rates and batch sizes vary across datasets.
  • The hyperparameter W is selected from domain-specific periodicity knowledge or computational methods such as the autocorrelation function.
  • TQNet uses one periodic length W per dataset, creating challenges for datasets with multiple periodicities.
  • Overlapping periodicities are generally manageable by selecting the longest cycle, which can implicitly capture shorter cycles.
  • Irregularly interwoven periodicities are more difficult because choosing the longest cycle may miss shorter-cycle nuances; multi-W ensembles are a possible compromise.

A.4. Effectiveness analysis

The TQ technique combines globally shared, periodically shifted queries with input-derived keys and values to model stable global and local inter-variable correlations. Periodic query reuse averages correlations across samples, reducing sensitivity to local noise and outliers.

  • Effectiveness analysis: The theoretical analysis explains why TQ is intended to provide more stable and expressive representations of variable correlations.The accompanying comparison reports full multivariate forecasting results across four horizons with look-back length L fixed at 96.
  • TQ mechanism: TQ-MHA uses periodically shifted learnable vectors as queries and raw input sequences as keys and values.This design targets global dependencies through the queries and sample-specific relationships through the keys and values.
  • TQ mechanism: Training encourages each learnable query Q_i to align with its corresponding key K_i, maximizing directional similarity.The alignment is interpreted as learning relevant inter-variable information for forecasting.
  • Periodic reuse: Queries extracted from the shared parameter θ_TQ are identical across samples separated by W time steps.The periodicity is expressed as Corr(Q_i) = Corr(Q_i+nW) for n = 0, 1, . . . , N −1.
  • Periodic reuse: Periodic query reuse makes each TQ query an averaged representation of correlations across multiple periodic samples.The averaging effect mitigates local non-stationary noise or outliers and produces more stable correlation modeling.

B.1. Full Comparison Results

Across 12 real-world multivariate datasets, TQNet consistently achieves state-of-the-art forecasting performance under most experimental settings.

  • Full comparison: TQNet consistently achieves state-of-the-art forecasting performance under most experimental settings across 12 real-world multivariate datasets.Table 5 compares TQNet with several baseline methods using the full multivariate forecasting results.

B.2. Univariate Forecasting Results

TQNet is competitive for multivariate-to-univariate forecasting while using a minimal architecture. Its accuracy-efficiency balance remains strong, and stacking additional modules generally provides little benefit.

  • Univariate forecasting: TQNet shows strong competitiveness against state-of-the-art models designed for multivariate-to-univariate forecasting.This setting uses exogenous variables to predict a single target variable.
  • Architecture: TQNet uses a single-layer attention mechanism and a single MLP to balance forecasting accuracy and computational efficiency.The model achieves state-of-the-art performance with only these essential components.
  • Architecture: Adding three stacked TQ-MHA and MLP modules does not yield significant performance gains and slightly reduces performance on most datasets.Modest improvements are observed on the PEMS datasets.
  • Practical implications: TQNet is presented as striking a trade-off between forecasting accuracy and computational cost, while lightweight designs support interpretability and practical deployment.These are the authors' stated implications of the model's compact design.
  • Comparison with ETS: TQNet demonstrates superior forecasting capabilities compared with ETS across multiple datasets.ETS is a univariate method that cannot leverage multivariate information, while TQNet emphasizes inter-variable dependencies.
  • Comparison with ETS: ETS requires L ≥ 2W, so its comparison uses L = 336 and L = 720 rather than TQNet's default L = 96.The different look-back lengths are imposed to satisfy ETS's period-length constraint.
  • Comparison with ETS: Under long forecasting horizons such as H = 720, ETS errors compound over time and its MSE substantially increases.ETS is primarily designed for short-term forecasting based on statistical extrapolation.
  • Look-back length: With shorter inputs, TQNet outperforms other channel-dependent methods by capturing robust multivariate dependencies.Channel-dependent methods generally retain an advantage over channel-independent methods as input length changes.

B.6. Robustness of TQNet

TQNet is reported to remain stable across random seeds and learning rates, with consistently low standard deviations. The supplied experiments also examine look-back lengths and architectural variants.

  • Robustness: TQNet maintains low standard deviations across runs with different random seeds and learning rates.The authors interpret this as evidence of stability and reliability under initialization and hyperparameter variation.
  • Look-back sensitivity: Figure 8 compares TQNet and other models on Electricity across different look-back lengths with forecast horizon 96.The figure's stated comparison varies the input history while holding the forecast horizon fixed.
  • Architecture robustness: The stacked three-layer TQNet variant is compared with the original model using results averaged across all four prediction horizons.This experiment evaluates whether additional architectural capacity improves performance.
  • Forecasting comparisons: The TQNet-versus-ETS comparison reports multivariate-to-univariate forecasting results averaged across four prediction horizons.This comparison contextualizes robustness across multiple datasets and horizons.
  • Robustness: The robustness evaluation varies both random seeds and learning rates, reporting mean and standard deviation.Table 9 defines mean as the average value and Std as the standard deviation.
Loading 2505.12917v2…