PaperlayerView exact revision on arXiv

Source-linked AI summary

DUET: Dual Clustering Enhanced Multivariate Time Series Forecasting

Xiangfei Qiu, Xingjian Wu, Yan Lin, Chenjuan Guo, Jilin Hu, Bin Yang

arXiv:2412.10859v3cs.LGstat.ML

TL;DR

Multivariate time series forecasting must handle heterogeneous temporal patterns and complex, intertwined channel relationships. DUET addresses these challenges through dual temporal and channel clustering, and experiments show improved performance over state-of-the-art baselines.

  • Problem

    Multivariate time series forecasting is challenged by temporal distribution shifts that create heterogeneous patterns and by complex, intertwined relationships among channels.

  • Method

    DUET combines temporal clustering with distribution-specific pattern extractors and channel soft clustering that models frequency-domain relationships while sparsifying noisy connections.

  • Results

    DUET outperforms state-of-the-art baselines across 25 datasets, including 7.1% lower MSE and 6.5% lower MAE than PDF, and 32.4% lower MSE and 21.7% lower MAE than Non-stationary Transformer.

  • Takeaways & Limitations

    Temporal clustering is particularly useful under substantial distribution changes, while channel soft clustering helps filter irrelevant channels and model useful inter-channel relationships.

Abstract

from arXiv · show

Multivariate time series forecasting is crucial for various applications, such as financial investment, energy management, weather forecasting, and traffic optimization. However, accurate forecasting is challenging due to two main factors. First, real-world time series often show heterogeneous temporal patterns caused by distribution shifts over time. Second, correlations among channels are complex and intertwined, making it hard to model the interactions among channels precisely and flexibly. In this study, we address these challenges by proposing a general framework called DUET, which introduces dual clustering on the temporal and channel dimensions to enhance multivariate time series forecasting. First, we design a Temporal Clustering Module (TCM) that clusters time series into fine-grained distributions to handle heterogeneous temporal patterns. For different distribution clusters, we design various pattern extractors to capture their intrinsic temporal patterns, thus modeling the heterogeneity. Second, we introduce a novel Channel-Soft-Clustering strategy and design a Channel Clustering Module (CCM), which captures the relationships among channels in the frequency domain through metric learning and applies sparsification to mitigate the adverse effects of noisy channels. Finally, DUET combines TCM and CCM to incorporate both the temporal and channel dimensions. Extensive experiments on 25 real-world datasets from 10 application domains, demonstrate the state-of-the-art performance of DUET.

1 Introduction

Multivariate time series forecasting must handle temporal distribution shifts that create heterogeneous patterns and complex channel interrelations. DUET addresses both challenges through dual clustering across temporal and channel dimensions.

  • Multivariate time series forecasting supports applications including economics, traffic, energy, and AIOps.
  • Temporal Distribution Shift changes data distributions over time, producing heterogeneous temporal patterns that are difficult to model.Different intervals can exhibit distinct trends and distributions.
  • Channel strategies trade off robustness, interaction modeling, and flexibility: CI ignores interactions, CD admits noisy channels, and CHC restricts relationships within clusters.
  • DUET uses a Temporal Clustering Module to assign series to fine-grained distributions and apply pattern extractors suited to each cluster.The design targets both stationary and nonstationary data.
  • DUET uses a Channel Clustering Module with frequency-domain metric learning, sparsification, and channel soft clustering to focus on useful relationships.A fusion module based on masked attention combines the learned dimensions.

2 Related Works

Prior work addresses temporal distribution shifts through normalization and channel dependencies through CI, CD, or hard clustering. DUET instead explicitly models heterogeneous patterns and adaptively builds sparse, soft channel groups.

  • 2.1 Temporal Distribution Shift in MTSF: Existing temporal-shift methods include adaptive normalization and instance normalization, while DUET explicitly models heterogeneous patterns under different distributions.
  • 2.2 Channel Strategies in MTSF: Channel-independent methods share weights and forecast independently, whereas channel-dependent methods jointly represent all channels.CI emphasizes robustness, while CD provides greater capacity but can incorporate irrelevant channels.
  • 2.2 Channel Strategies in MTSF: Hard clustering combines dependent modeling within clusters with independent modeling across clusters but restricts relationships to same-cluster channels.
  • 2.2 Channel Strategies in MTSF: DUET adopts Channel-Soft-Clustering with fully adaptive sparsity to dynamically build a group for each channel.The design is presented as a more comprehensive alternative to rigid similarity-based grouping.

3 Preliminaries

The paper defines time series and temporal distribution shift, then formulates multivariate forecasting as predicting future timestamps from historical multichannel observations.

  • 3.1 Definitions: A time series X ∈ R^(N×T) is a time-oriented sequence with N channels and T timestamps; N=1 is univariate and N>1 is multivariate.
  • 3.1 Definitions: X_i,j denotes channel i at timestamp j, while X_n,: denotes the full length-T series of channel n.
  • 3.1 Definitions: Temporal Distribution Shift occurs when sliding-window series can be partitioned into K sets with distinct data distributions.The distributions satisfy P_Di(x) ≠ P_Dj(x) for distinct clusters.
  • 3.1 Definitions: Multivariate Time Series Forecasting predicts the next F timestamps Y ∈ R^(N×F) from historical observations X ∈ R^(N×T).

4 Methodology

DUET combines temporal clustering of heterogeneous distributions with soft channel clustering to model temporal patterns and sparse channel dependencies. Its Fusion Module integrates both outputs through masked attention before forecasting.

  • Temporal Clustering Module: DUET first uses a Distribution Router to assign each channel-independent series to latent distribution clusters and corresponding pattern extractors.The router uses candidate latent distributions, noisy gating, and weighted extractor selection.
  • Fusion Module: The Fusion Module combines temporal features with the channel mask through masked attention, producing fused representations for a linear forecasting predictor.The framework extracts X_temp and M, then uses masked attention to fuse them.
  • Temporal Clustering Module: Each selected linear extractor decomposes a series into trend and seasonal parts, separately transforms them, and fuses the resulting temporal features.The extractor cluster processes series sharing latent distributions, while an aggregator combines extractor outputs using routing weights.
  • Channel Clustering Module: DUET learns channel relationships in frequency space with an adaptive metric and converts them into a sparse channel-mask matrix.The method uses Fourier representations, a learnable Mahalanobis metric, probabilized relationships, and Bernoulli resampling.
  • Channel Clustering Module: The Channel Clustering Module assigns channels through soft relationships so each channel can emphasize beneficial neighbors and reduce irrelevant connections.The learned mask represents probabilistic channel usefulness for prediction and supports sparse connections.

5 Experiments

DUET is evaluated against recent forecasting baselines through unified experiments, ablations, and analyses of its temporal and channel clustering components. It improves forecasting performance overall, while the results support the roles of channel-soft clustering, temporal clustering, frequency-domain distances, and distribution-aware routing.

  • Main Results: DUET’s channel-soft-clustering strategy outperforms channel-independent models and mitigates noise from weakly correlated channels in ETT datasets.Each channel focuses on related channels rather than attending indiscriminately to all channels.
  • Main Results: DUET reduces MSE by 32.4% and MAE by 21.7% compared with Non-stationary Transformer on temporal distribution-shifted series.The result supports the effectiveness of the Temporal Clustering Module for heterogeneous temporal patterns.
  • Ablation Studies: Removing TCM hurts performance more on datasets with substantial distribution changes, while removing CCM affects datasets with strong inter-channel correlations.The ablation results connect each module’s contribution to its targeted type of structure.
  • Distance Metrics: Learnable Mahalanobis distance and frequency-domain channel representations outperform alternative similarity measures and random masks.Replacing the learned metric with Euclidean distance, cosine similarity, or DTW decreases performance.
  • Parameter Sensitivity: The best number of temporal pattern extractors varies by domain: M=4 for ETTh1 and ETTh2, M=2 for ILI, and M=5 for Exchange.Using the appropriate M generally outperforms other variants.
  • Visualization Analysis: The distribution router distinguishes samples using distributional characteristics, while CCM forms flexible channel groupings through masked attention weights.The visual analyses use distribution weights for temporal samples and attention weights for ETTh2 channels.

6 Conclusions

The paper concludes that DUET enhances multivariate time series forecasting through dual clustering over temporal and channel dimensions. TCM models distribution-specific temporal patterns, CCM learns sparse frequency-domain channel relationships, and masked attention fuses the resulting representations.

  • Conclusions: DUET combines temporal and channel dual clustering to enhance multivariate time series forecasting.Its framework integrates TCM, CCM, and a Fusion Module based on masked attention.
  • Conclusions: TCM assigns time series to fine-grained distribution clusters and uses different pattern extractors to model their temporal heterogeneity.The module targets heterogeneous patterns associated with different distributions.
  • Conclusions: CCM applies channel-soft-clustering, frequency-domain metric learning, and sparsification to capture channel relationships.The resulting channel mask is combined with temporal features by the Fusion Module.

A.1 Computational Complexity Analysis

The computational analysis attributes DUET’s main complexity to its Transformer-based Fusion Module. Dual clustering reduces the need for attention over temporal dependencies, so attention primarily models channel correlations.

  • Computational Complexity Analysis: DUET’s main computational complexity comes from the Transformer-based Fusion Module.The dual clustering design extracts temporal patterns efficiently before attention is applied.
  • Computational Complexity Analysis: DUET uses attention primarily to capture channel correlations because dual clustering extracts temporal patterns separately.This design is presented as an efficiency benefit relative to methods that use attention for both temporal and channel dependencies.

A.2 Noisy Gating

The appendix explains the relationship between the reparameterization trick and the original Noisy Gating formulation. It rewrites the stochastic components to show computational equivalence under the stated assumptions.

  • Noisy Gating: The Noisy Gating formulation combines Gaussian perturbations with learned transformations and Softplus-based gating.The displayed expression uses μ, σ, ε, and ε′ in the gating computation.
  • Noisy Gating: The derivation simplifies the stochastic Softplus term into a learnable vector to compare the reparameterized and original formulations.The appendix then analyzes each vector component and its normal distribution.
  • Noisy Gating: The learnable components can be adaptively tuned to establish approximate equivalence between the two formulations.The appendix presents this as computational equivalence to the original Noisy Gating technique.

A.3 Fourier Transform in 𝐿2 Space

The appendix frames the Fourier transform as an orthogonal coordinate transformation in L2 space. The experiments then use varying look-back windows to evaluate DUET against baselines.

  • A.3 Fourier Transform in 𝐿2 Space: Fourier coefficients provide coordinates under the trigonometric-function basis in L2 space.The trigonometric functions form an orthonormal basis whose span is dense in L2(−π, π).
  • A.3 Fourier Transform in 𝐿2 Space: The Fourier transform acts as an orthogonal transformation in L2 space.This interpretation follows from representing functions through coordinates in the trigonometric basis.
  • A.3 Fourier Transform in 𝐿2 Space: The study evaluates forecasting with look-back windows H = 48, 96, 192, 336, 512, 720 and horizons F = 96, 720.Figure 8 reports MSE across ETTm1, Solar, and Electricity.

B.2 The Advantages of Dual Clustering

DUET’s dual clustering addresses distributional changes and channel interactions together. Visualizations indicate that its temporal clustering handles changing patterns while CSC uses information from related channels for more accurate forecasts.

  • B.2 The Advantages of Dual Clustering: DUET models heterogeneous temporal patterns by clustering time series into distributional categories and using parameter-independent backbones.Baselines use a single backbone despite distributional changes.
  • B.2 The Advantages of Dual Clustering: DUET’s CSC strategy leverages the advantages and disadvantages of other channels to improve forecast accuracy.The comparison is presented for visualized forecasts on Electricity, PEMS04, and Weather.
Loading 2412.10859v3…