Source-linked AI summary
Gated Transformer Networks for Multivariate Time Series Classification
Minghao Liu, Shengqi Ren, Siyuan Ma, Jiahui Jiao, Yizhou Chen, Zhiguang Wang, Wei Song
TL;DR
Multivariate time series classification requires modeling correlations across channels and time steps, but Transformer evidence in this setting remains limited. The paper introduces GTN, which gates two Transformer towers for channel-wise and step-wise modeling, and reports competing performance across thirteen datasets with preliminary interpretability analyses.
Problem
Multivariate time series classification involves correlated information across channels and time steps, while Transformer classification studies in this area remain at an early stage.
Method
GTN uses gating to merge two Transformer towers that explicitly model channel-wise and step-wise correlations.
Results
Across thirteen multivariate time series datasets, GTN achieved competing performance with current state-of-the-art deep learning models.
Takeaways & Limitations
The preliminary results establish GTN as a baseline for Transformer networks on multivariate time series classification and include attention-map analyses for interpretability.
Takeaways & Limitations
The attention-map impact was not obvious, and the authors identify GTN as comparatively prone to overfitting.
Abstract
from arXiv · showhide
Deep learning model (primarily convolutional networks and LSTM) for time series classification has been studied broadly by the community with the wide applications in different domains like healthcare, finance, industrial engineering and IoT. Meanwhile, Transformer Networks recently achieved frontier performance on various natural language processing and computer vision tasks. In this work, we explored a simple extension of the current Transformer Networks with gating, named Gated Transformer Networks (GTN) for the multivariate time series classification problem. With the gating that merges two towers of Transformer which model the channel-wise and step-wise correlations respectively, we show how GTN is naturally and effectively suitable for the multivariate time series classification task. We conduct comprehensive experiments on thirteen dataset with full ablation study. Our results show that GTN is able to achieve competing results with current state-of-the-art deep learning models. We also explored the attention map for the natural interpretability of GTN on time series modeling. Our preliminary results provide a strong baseline for the Transformer Networks on multivariate time series classification task and grounds the foundation for future research.
1 Introduction
Multivariate time series classification remains an important, information-rich problem, while Transformer applications to it are still developing. The paper proposes Gated Transformer Networks, evaluates them broadly, and examines learned features and attention maps.
- Multivariate time series contain correlated information across channels and time steps, motivating representation learning for classification.
- GTN extends Transformer networks with gating that merges two towers modeling channel-wise and step-wise correlations.The two towers are designed to capture complementary structures in multivariate sequences.
- GTN was evaluated on thirteen multivariate time series benchmark datasets against state-of-the-art deep learning models with comprehensive ablation studies.
- The experiments showed that GTN achieves competing performance with other deep learning models.
- The study visualized learned features and preliminarily explored attention maps to investigate GTN’s feature extraction and interpretability.
2 Related Work
Prior work includes convolutional and recurrent approaches for time series classification, while Transformer classification studies remain comparatively early. This work positions GTN as a comprehensive Transformer study for multivariate time series classification.
- Earlier multivariate time series work explored deep convolutional networks, while FCN and ResNet achieved state-of-the-art performance on univariate classification.
- Related deep models include hybrid LSTM–FCN architectures that merge features to improve univariate classification, although LSTM has high computational complexity.
- Transformer research is established in NLP and computer vision, but time series Transformer classification remains in an early stage.
- The paper describes its contribution as bridging Transformer research and multivariate time series classification through the first comprehensive study in this setting.
3 Gated Transformer Networks
GTN adapts Transformers for multivariate time series by using continuous-value embeddings, separate step-wise and channel-wise towers, and a learned gating mechanism. The architecture is evaluated against benchmark models, where direct feature concatenation compromises performance.
- GTN adapts the Transformer with a fully connected nonlinear embedding, two correlation-specific towers, and gating for multivariate classification.The architecture is shown in Figure 1.
- Two-tower Transformer: The two-tower design explicitly captures step-wise and channel-wise correlations using attention and masking.
- Two-tower Transformer: Channel-wise attention omits positional encoding because channel order has no relative or absolute correlation in the stated formulation.
- Gating: Concatenating features from both towers compromises the performance of both, motivating the proposed gating mechanism.
- Gating: The gating mechanism projects concatenated tower outputs into softmax weights that attend to the corresponding tower outputs and form the final feature vector.
4 Experiments
Experiments evaluate GTN on 13 multivariate time-series datasets against deep-learning baselines, then analyze its ablations, attention maps, embeddings, and gating behavior. GTN achieves comparable performance to FCN and ResNet, while ablations support learned gating over simple concatenation and show dataset-dependent tower preferences.
- Experiment Settings: GTN was tested on 13 multivariate time-series datasets using the Baydogan archive subset and compared with established deep-learning models.Benchmarks include FCN, ResNet, MLP, Encoder, MCNN, MCDCNN, Time-CNN, t-LeNet, and TWIESN.
- Experiment Settings: GTN achieved comparable results with FCN and ResNet, with no statistically significant difference among the three models.GTN showed superior performance on NetFlow and KickvsPunch, but was more prone to overfitting and used dropout to reduce that risk.
- Ablation Study: Channel-wise-only transformers outperform step-wise-only transformers on most datasets, supporting channel correlations as an important multivariate differentiator.Masked attention also performs slightly better overall than unmasked tower-only transformers and better captures channel-wise features.
- Ablation Study: Dataset preferences differ: step-wise modeling performs better on PEMS, whereas channel-wise modeling outperforms on CMUsubject16.These results indicate that different time series rely differently on channel-wise and step-wise information.
- Ablation Study: Gating outperforms simple concatenation by learning data-driven weights that favor the more useful tower for each input.Simple concatenation can fall between the two towers or perform worse, while gating achieved the best performance in the ablation study.
- Visualization and Analysis of the Attention Map: Attention analyses associate high channel-wise scores with similar channel trends, while low scores can occur for channels with different or inverse trends despite small DTW distance.Step-wise attention is compared with Euclidean distance because time is fixed within a step; the authors note that the distance effects are not fully obvious.
- Analysis of the Embedding Output: Embedding visualizations reveal clustered time-step representations with distinct shapelets, and gated features project into an easily separable space for classification.The dark blue shapelet is described as a plateau followed by a deep down like 7; the gated-feature visualization uses JapaneseVowels.
5 Conclusion
GTN extends Transformer Networks with gating to explicitly learn channel-wise and step-wise correlations in multivariate time series. Experiments, ablations, and visualizations support its competing performance and preliminary interpretability.
- GTN uses gating to merge two Transformer towers modeling channel-wise and step-wise correlations.
- Experiments on thirteen datasets show competing performance with current state-of-the-art deep learning models.
- Ablation studies examine how different modules work together to achieve improved performance.
- Visualizations qualitatively analyze the attention map and other components to improve understanding of GTN's interpretability.
- The preliminary results establish a foundation for future research on Transformer Networks for time series classification.