PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Position Encoding of Transformers for Multivariate Time Series Classification

Navid Mohammadi Foumani, Chang Wei Tan, Geoffrey I. Webb, Mahsa Salehi

arXiv:2305.16642v1cs.LGcs.CV

TL;DR

Transformers need positional information to model time-series order, yet the effectiveness of absolute versus relative encodings for time series was not well studied. The paper proposes tAPE and eRPE and combines them with convolutional input encoding in ConvTran, which achieves state-of-the-art performance on multivariate time-series classification benchmarks.

  • Problem

    The effectiveness of absolute, relative, or combined position encoding methods for transformer-based time-series analysis is not well studied.

  • Method

    The paper develops tAPE and eRPE and integrates them with convolution-based input encoding in the ConvTran multivariate time-series classifier.

  • Results

    ConvTran is significantly more accurate than previous state-of-the-art deep-learning models across 32 benchmark datasets.

  • Takeaways & Limitations

    The proposed position encodings are designed for time-series transformers and can be integrated into transformer blocks for downstream tasks.

  • Takeaways & Limitations

    HC2 remains difficult to scale to large datasets, long time series, or many channels, while ConvTran requires more training samples on some small-data datasets.

Abstract

from arXiv · show

Transformers have demonstrated outstanding performance in many applications of deep learning. When applied to time series data, transformers require effective position encoding to capture the ordering of the time series data. The efficacy of position encoding in time series analysis is not well-studied and remains controversial, e.g., whether it is better to inject absolute position encoding or relative position encoding, or a combination of them. In order to clarify this, we first review existing absolute and relative position encoding methods when applied in time series classification. We then proposed a new absolute position encoding method dedicated to time series data called time Absolute Position Encoding (tAPE). Our new method incorporates the series length and input embedding dimension in absolute position encoding. Additionally, we propose computationally Efficient implementation of Relative Position Encoding (eRPE) to improve generalisability for time series. We then propose a novel multivariate time series classification (MTSC) model combining tAPE/eRPE and convolution-based input encoding named ConvTran to improve the position and data embedding of time series data. The proposed absolute and relative position encoding methods are simple and efficient. They can be easily integrated into transformer blocks and used for downstream tasks such as forecasting, extrinsic regression, and anomaly detection. Extensive experiments on 32 multivariate time-series datasets show that our model is significantly more accurate than state-of-the-art convolution and transformer-based models. Code and models are open-sourced at \url{https://github.com/Navidfoumani/ConvTran}.

1 Introduction

Multivariate time-series classification needs models that capture both local patterns and global temporal order. This paper studies position encoding for time series and proposes tAPE, eRPE, and ConvTran to address limitations of existing approaches.

  • CNNs generalize well and converge quickly, but they struggle with long-range dependencies and global ordering.
  • Self-attention captures long-range dependencies, yet it cannot represent input ordering without explicit positional information.
  • Absolute encodings assign vectors by individual positions, whereas relative encodings represent distances between sequence elements.
  • Existing absolute encodings are ineffective for time series because time-series embeddings are relatively low-dimensional and sequence lengths vary widely.
  • The paper introduces tAPE, which incorporates series length and embedding dimension, and reports improved classification over existing absolute encodings.
  • ConvTran combines tAPE/eRPE with convolution-based input encoding and is reported as significantly more accurate than prior deep-learning models on 32 benchmark datasets.

2 Related Work

Related MTSC work includes convolutional, ensemble, and attention-based models. Existing methods offer strong accuracy or efficient feature extraction, but scalability and long-range modeling remain important concerns.

  • State-of-the-art MTSC Algorithms: ROCKET uses 10,000 random convolution kernels to create 20,000 features per time series before linear classification.
  • State-of-the-art MTSC Algorithms: CIF extracts 25 features from random intervals and builds a forest of 500 trees for classification.
  • State-of-the-art MTSC Algorithms: HIVE-COTE combines classifiers from multiple domains, but multivariate adaptation through per-channel ensembles is computationally expensive.
  • State-of-the-art MTSC Algorithms: HC2 is highly accurate on 26 small benchmark datasets but is not scalable to large datasets, long series, or many channels.
  • CNN Based Models: CNN architectures such as FCN and ResNet efficiently extract latent features and have shown strong MTSC performance.
  • CNN Based Models: Inception-Time uses bottleneck layers, multiscale one-dimensional convolutions, and pooling within an ensemble of five inception networks.
  • Attention Based Models: Attention-based MTSC models apply self-attention to leverage long-term dependencies, while BENDER adapts BERT-style representations for time-series data.

3 Background

The background defines MTSC, self-attention, and positional encoding. It distinguishes absolute and relative position methods and describes how these mechanisms are incorporated into transformer attention.

  • Problem Description and Notation: The task maps multivariate time-series samples X to class labels Y using a neural-network classifier.
  • Self-Attention: Self-attention transforms an input series into an output series by weighting input elements according to attention scores.
  • Self-Attention: Multi-head attention computes attention in parallel across h heads and concatenates the transformed head outputs.
  • Self-Attention: Because self-attention lacks recurrence and convolution, positional encoding supplies temporal ordering information needed for time-series inputs.
  • Absolute Position Encoding: Absolute position encoding adds a position vector p_i to input embedding x_i, with fixed sinusoidal and learnable alternatives.
  • Absolute Position Encoding: Sinusoidal encoding uses sine and cosine components at multiple frequencies to represent positions in d_model dimensions.
  • Relative Position Encoding: Relative encoding represents pairwise distances between elements and embeds those relations into self-attention keys and values.
  • Relative Position Encoding: Shaw’s relative encoding requires O(L^2d) memory, whereas Huang’s skewing method reduces intermediate memory to O(Ld).

4 Position Encoding of Transformers for MTSC

The paper develops position encodings for time-series transformers that address distance awareness, isotropy, efficiency, and varying series lengths. It introduces tAPE and eRPE, then integrates them with convolutional input encoding in ConvTran.

  • Time Absolute Position Encoding (tAPE): Existing absolute position encodings can lose distance awareness or produce similar position vectors when time-series embedding dimensions are low.Lower dimensions may make similarity fail to decrease with positional distance, while anisotropy limits use of the embedding space.
  • Time Absolute Position Encoding (tAPE): tAPE incorporates series length and input embedding dimension into sinusoidal frequency terms to balance distance awareness and isotropy.Using series length produces a smoother monotonic similarity trend, while incorporating embedding dimension alleviates anisotropy.
  • Time Absolute Position Encoding (tAPE): With 128-dimensional embeddings, tAPE maintains a steadier decrease in positional similarity than vanilla APE for series lengths L = 1000 and L = 30.For L = 1000, vanilla APE becomes less steadily decreasing beyond approximately |K| > 50; for L = 30, tAPE vectors are less similar.
  • Efficient Relative Position Encoding (eRPE): The paper introduces eRPE as an input-independent relative position encoding using scalar parameters indexed by pairwise positional displacement.Scalar relative weights preserve translation equivalence while adding minimal computation through incorporation into pairwise attention dot products.
  • ConvTran integration: ConvTran first uses convolutions to reduce series length and encode temporal patterns and multivariate correlations, then adds tAPE before transformer attention.This ordering limits the quadratic cost of global attention on long raw time series.

5 Experimental Results

Experiments evaluate position encodings, ConvTran, and comparisons with state-of-the-art methods across multivariate time-series datasets. Results favor tAPE/eRPE and ConvTran overall, while performance and efficiency vary with dataset size and training-set availability.

  • Evaluation setup: Experiments use ablations, model comparisons, state-of-the-art benchmarks, and efficiency evaluations on UEA and additional multivariate time-series datasets.The UEA archive contains 30 real-world datasets, while ConvTran is evaluated across 32 datasets overall.
  • Position-encoding ablation: tAPE achieves the highest accuracy rank among absolute position encodings and significantly outperforms the alternatives.The authors attribute this to effective embedding-space utilization while retaining distance awareness.
  • Position-encoding ablation: eRPE achieves the highest accuracy rank among relative encodings and significantly outperforms the alternatives.Its lower parameter count is presented as making it less likely to overfit.
  • Position-encoding ablation: The tAPE/eRPE combination is significantly more accurate than other tested combinations, whereas Learn/Vector has the least accurate results.The authors associate the weaker Learn/Vector result with its higher parameter count.
  • Deep-learning comparisons: ConvTran has the lowest average rank and is more accurate than all compared convolutional and transformer-based methods across 32 datasets.It is significantly more accurate than Disjoint-CNN and TST, its convolutional and transformer-based predecessors.
  • State-of-the-art benchmark: ConvTran mostly outperforms HC2, ROCKET, CIF, and Inception-Time on datasets with at least 100 training samples per class, but models with few examples can perform better.On EigenWorms, ConvTran remains the strongest CNN, while other state-of-the-art models outperform it overall.
  • Efficiency: ROCKET trains faster on smaller datasets, whereas ConvTran becomes faster as data quantity increases and has faster inference across datasets.The comparison uses GPU experiments for ConvTran and CPU experiments for ROCKET.

6 Conclusion

The paper studies position encoding for time-series transformers, proposes tAPE and eRPE, and integrates them with convolution in ConvTran. Experiments report state-of-the-art deep-learning performance for multivariate time-series classification.

  • Scope: The paper reviews absolute and relative position encodings and studies their importance for time-series classification.It presents this investigation as the first study of position-encoding importance for time series.
  • Contributions: The authors propose tAPE, eRPE, and ConvTran, which combines the two encodings with a convolution layer in a transformer-based classifier.ConvTran is designed for multivariate time-series classification.
  • Conclusion: Extensive experiments show that ConvTran benefits from position information and achieves state-of-the-art performance in deep-learning multivariate time-series classification.The conclusion reports this outcome without specifying a single benchmark value.
  • Future work: Future work will study the proposed transformer block in other transformer-based classifiers and downstream tasks such as anomaly detection.The stated future tasks also include other downstream applications.

A.1 Empirical Evaluation of Efficiency and Effectiveness

The efficiency evaluation compares ConvTran with ROCKET across dataset sizes. ConvTran is favored on larger datasets, while ROCKET retains a training-time advantage on smaller datasets.

  • Larger datasets: ConvTran outperforms ROCKET in both train time and test accuracy on datasets with more than 10k samples.This result is reported for the larger datasets in Table A1.
  • Smaller datasets: ROCKET has better training time on smaller datasets, although ConvTran still achieves acceptable accuracy within a reasonable training time.ConvTran’s performance improves as dataset size increases, supporting its suitability for larger datasets.

A.2 ConvTran vs non-deep learning SOTA Models

Table A2 compares ConvTran with ROCKET, HC2, and CIF across datasets with varying training sample sizes, including cases where methods were non-runnable. ConvTran outperforms the non-deep learning models on most reported datasets, especially when training sets are larger.

  • 19 out of 32 datasets favor ConvTran over the non-deep learning models.For HC2 and CIF, results are available for 26 datasets, with ConvTran outperforming the other models on 13.
  • ConvTran performs better on datasets with larger training sample sizes, while other models perform better when training data are scarce.InsectWingBeat favors ConvTran, whereas StandWalkJump, with only 12 training samples, favors other models.
  • Some non-deep learning models are non-runnable because of computational complexity or inability to handle varying-length series.HC2 and CIF could not run on larger datasets such as HAR, Ford, and InsectWingBeat, and were not designed for datasets including CharacterTrajectories, SpokenArabicDigits, and JapaneseVowels.
Loading 2305.16642v1…