A Transformer-based Framework for Multivariate Time Series Representation Learning

Introduction

• Because labeled multivariate time-series data are often scarce, expensive, or impractical to obtain, the paper motivates methods that use few labels or leverage abundant unlabeled data.The paper describes multivariate time series as arising across domains including science, medicine, finance, engineering, and industry.Source: §1 · p. 1
• Non-deep methods such as TS-CHIEF, HIVE-COTE, and ROCKET held benchmark records in time-series regression and classification, matching or outperforming InceptionTime and ResNet.The introduction notes that deep learning had challenged and sometimes replaced state of the art in forecasting, regression, and classification, but had not become dominant in time series.Source: §1 · pp. 1–2
• On public multivariate time-series regression and classification benchmarks, the framework reportedly outperforms current state-of-the-art approaches with only hundreds of training samples.The authors also report an advantage over supervised learning without additional unlabeled samples, and say models with at most hundreds of thousands of parameters can be trained on CPUs.Source: §1 · p. 2
• The framework pre-trains a transformer encoder to extract dense time-series representations using an input-denoising, autoregressive objective, then applies them to regression, classification, imputation, and forecasting.The paper presents this as its first investigation of transformer encoders for unsupervised multivariate time-series representation learning.Source: §1 · p. 2

Regression and classification of time series

• Public-benchmark leaders include ROCKET, HIVE-COTE, and TS-CHIEF, ahead of CNN architectures, while HIVE-COTE and TS-CHIEF are costly, scale poorly, lack GPU support, and were evaluated only on univariate series.ROCKET is the best-ranking method on average and trains a linear classifier on features from many random convolutional kernels.Source: §2 · p. 2

Unsupervised learning for multivariate time series

• Unsupervised multivariate time-series work has mainly used reconstruction-trained MLP/RNN autoencoders, including VAE variants for clustering and topology visualization and RNN/LSTM sequence-to-sequence models with reconstruction or forecasting losses.Kopf et al. (2019) and Fortuin et al. (2019) explored VAE variants; Malhotra et al. (2017) used an RNN autoencoder, while Lyu et al. (2018) evaluated reconstruction and forecasting losses.Source: §2 · pp. 2–3
• Other approaches preserve kernel-conditioned similarity for imputation/classification, approximate DTW distances for clustering, or reconstruct correlations for anomaly detection (Zhang et al., 2019).Bianchi et al. use a user-provided kernel prior, Lei et al. use matrix factorization, and Zhang et al. use a convolutional-LSTM network with attention.Source: §2 · p. 3
• Temporal-proximity triplet losses support unsupervised time-series representations; Franceschi et al. (2019) combine them with a dilated causal CNN, the best unsupervised method on UEA/UCR univariate and multivariate classification datasets.The method is outperformed by supervised state-of-the-art methods on univariate classification.Source: §2 · p. 3

Transformer models for time series

• Prior transformer work applied encoder-decoder models to univariate forecasting and multivariate geo-tagged imputation, reporting gains over statistical and RNN baselines on public and competition datasets.Forecasting applications include Li et al. (2019), Wu et al. (2020), and Lim et al. (2020); Ma et al. (2019) reported imputation gains on three public and two competition datasets.Source: §2 · p. 3
• Unlike task-specific generative applications, this work generalizes transformers into an unsupervised pre-training framework readily adapted to varied downstream tasks, analogous to BERT's generic NLP framework (Devlin et al., 2018).The passage contrasts this framework with full encoder-decoder architectures designed for specific generative tasks.Source: §2 · p. 3

3.1 BASE MODEL

• For each time step, the encoder normalizes the multivariate feature vector, projects it to the model dimension, and adds learnable positional encodings before self-attention.The framework uses batch normalization rather than layer normalization, and padding masks let it ignore padded positions when processing variable-length samples in minibatches.Source: §3.1 · pp. 3–5

3.2 REGRESSION AND CLASSIFICATION

• For regression and classification, the model concatenates all time-step representations and feeds them to a linear output layer; fine-tuning updates all weights.Regression uses squared error, while classification applies softmax and uses cross-entropy.Source: §3.2 · pp. 5–6

3.3 UNSUPERVISED PRE-TRAINING

• Pre-training masks variable-wise segments with r = 0.15 and mean masked length lm = 3, then predicts full input vectors while applying MSE only to masked values.Segment masking avoids trivial predictions of very short gaps and encourages attention to preceding and succeeding segments as well as contemporaneous values of other variables.Source: Figure 1 · p. 4, §3.3 · p. 6
• The masked-value objective differs from denoising autoencoders, which reconstruct the entire input, and from input-embedding dropout; all models also use 10% dropout.The mask determines both which input values are corrupted and which predictions contribute to the loss.Source: §3.3 · p. 6

4 EXPERIMENTS & RESULTS

• Experiments used predefined train/test splits and convergence-length training; hyperparameters were selected per dataset via an 80/20 training/validation split, then models were retrained on all training data.The authors note that fully supervised transformer training typically required more epochs than fine-tuning pretrained models.Source: §4 · pp. 6–7

4.1 REGRESSION

• Pretraining on the same training samples improved over supervised-only regression models on 3 of 6 datasets, with the advantage persisting across label availability.On the largest regression dataset, the pre-trained model outperformed the fully supervised model even when all labels were used.Source: §4.1 · p. 7, Figure 2 · p. 8
• TST averaged rank 1.33, ranking first on four of six regression datasets; it achieved 30% lower RMSE than the all-model mean and approximately 16% lower RMSE than XGBoost.Absolute RMSE improvements over other models ranged from approximately 4% to 36%.Source: §4.1 · p. 7, Table 1 · p. 8
• With labels fixed, using more samples for unsupervised pretraining generally reduced regression error; at 10% label availability, error briefly rose after an initial drop before declining again.The 10% and 20% label conditions corresponded to approximately 1,250 and 2,500 labels, respectively.Source: §4.1 · p. 8, Figure 2 · p. 8

4.2 CLASSIFICATION

• TST performed best on 7 of 11 classification datasets, averaging rank 1.7 versus ROCKET’s 2.3.ROCKET outperformed TST on very low-dimensional, 3-dimensional datasets; TST was particularly strong on high-dimensional datasets and also performed well on datasets with only a few hundred samples. On InsectWingBeat, TST reached 0.689 accuracy versus XGBoost’s 0.369.Source: Table 2 · p. 9, §4.2 · p. 9
• Pre-trained transformers outperformed fully supervised models on 8 of 11 classification datasets, despite using no additional samples for pretraining.The reported benefit appears to come from reusing the same samples through an unsupervised objective.Source: §4.2 · pp. 9–10

Execution time for training

• A precise training-time comparison is out of scope, though Appendix Section A.3 demonstrates economical resource use and self-attention has O(w^2) complexity in series length w.Sparse attention, recurrence, and compressed global-local attention are proposed as ways to reduce this bottleneck.Source: §5 · p. 10

Imputation and forecasting

• The unchanged pretraining setup supports imputation, achieving RMSEs very close to 0 on test-set input denoising after training-set unsupervised pre-training.The authors defer systematic comparison with the state of the art; masking the final part of all variables can enable forecasting, with sliding windows suggested for longer series.Source: §5 · p. 10

Extracted representations

• Transformer representations can support time-series similarity, clustering, and visualization, with per-time-step access enabling selective temporal comparison.Users can emphasize the beginning, middle, or end of a series and visualize sample evolution.Source: §5 · p. 10

6 CONCLUSION

• Across public datasets from diverse domains, the framework is the best-performing method for regression and classification, even with only a few hundred training samples, and the only top-performing method based on deep learning.The paper also identifies it as the first unsupervised method shown to push state-of-the-art performance for multivariate time series regression and classification.Source: §6 · p. 10
• The proposed framework learns multivariate time series representations using a transformer encoder architecture and an unsupervised pre-training scheme.Source: §6 · p. 10
• Unsupervised pre-training substantially benefits performance over fully supervised learning, even when it reuses the same data samples without additional unlabeled data.Source: §6 · p. 10

A.1 ADDITIONAL POINTS & FUTURE WORK

• A precise training-time comparison is out of scope, although the appendix demonstrates economical computational-resource use.Source: §A.1 · p. 13
• Sparse attention, recurrence, and compressed global-local attention are proposed as alternatives that can reduce the O(w2) self-attention complexity bottleneck.The complexity is with respect to time-series length w.Source: §A.1 · p. 13
• The Section 3.3 setup performs imputation without modification, and the authors report very good results on input denoising after unsupervised pre-training.They defer a systematic quantitative comparison with the state of the art.Source: §A.1 · p. 13
• For test-set input denoising after unsupervised pre-training, the models reached Root Mean Square Errors very close to 0.This is offered as a rough indication of performance, not a systematic comparison.Source: §A.1 · p. 13
• Concealing the last part of all variables enables forecasting with the same model and setup; longer series can use this process within a sliding window.The authors defer systematic investigation of forecasting to future work.Source: §A.1 · p. 13
• The extracted representations zt can support time-series similarity evaluation, clustering, visualization, and other representation-based use cases.Source: §A.1 · p. 13
• Because transformer representations can be addressed independently at each time step, users can weight the beginning, middle, or end of a series selectively.This enables selective comparisons and visualization of temporal evolution.Source: §A.1 · p. 13

A.2 CRITERIA FOR DATASET SELECTION

• Selection spans time-series dimensionality, length, and sample count, while including datasets where baselines perform well and poorly.The authors describe this as a rationale for selecting diverse multivariate datasets from the Monash University, UEA, and UCR archives.Source: §A.2 · p. 14
• EthanolConcentration is low-dimensional, few-sample, moderate-length, many-class, and challenging; FaceDetection is high-dimensional, large-sample, short, and minimally classed; Handwriting is low-dimensional and few-sample.Source: §A.2 · p. 14
• Heartbeat is high-dimensional and few-sample; JapaneseVowels has heterogeneous sample lengths and few samples despite short series, with strong baselines; InsectWingBeat is high-dimensional, large-sample, short, and challenging.Source: §A.2 · p. 14
• PEMS-SF is extremely high-dimensional with few samples; SelfRegulationSCP1 has few dimensions, few samples, long series, and strong baselines, whereas SCP2 is similar but challenging.Source: §A.2 · p. 14
• SpokenArabicDigits has moderate dimensionality, many samples, heterogeneous lengths, and mostly strong baselines; UWaveGestureLibrary is low-dimensional, few-sample, moderate-length, and has strong baselines.Source: §A.2 · p. 14

A.3 EXECUTION TIME

• On a CPU, transformer training was typically at least an order of magnitude slower; on a GPU, it was typically as fast as or faster than ROCKET and XGBoost, and never exceeded 3 hours.The comparison records training time until maximum accuracy; the GPU runs could span hundreds of epochs.Source: §A.3 · p. 14, Table 7 · p. 15
• LSTMs require O(w) sequential operations per sample, with per-layer complexity scaling as O(N · d^2), where w is series length and d is the hidden-state dimension.The section refers readers to Vaswani et al. (2017) for a detailed comparison of transformers and convolutional neural networks.Source: §A.3 · p. 15

A.4 ADVANTAGES OF TRANSFORMERS

• Transformers can model long contexts by selectively attending to relevant sequence elements without position-dependent bias, unlike RNNs’ endpoint asymmetry and limited hidden-state memory.Even bidirectional RNNs treat middle and endpoint elements differently, and LSTMs/GRUs retain only limited time steps because of the vanishing-gradient problem (Pascanu et al., 2013).Source: §A.4 · p. 19
• Multiple attention heads capture distinct relevance patterns across representation subspaces, unlike RNN attention’s single global relevance mode.For a signal with frequency components 1/T1 and 1/T2, heads can attend to neighboring points or points one of those periods earlier.Source: §A.4 · p. 19
• Transformers redistribute attention after each encoder layer, using progressively more abstract representations, whereas RNN attention typically uses one distribution over a single representation layer.The passage contrasts this with RNN models that use one distribution of attention weights, most typically over hidden states.Source: §A.4 · p. 19

• Table 13 lists fixed hyperparameters common to all transformer models.Source: Table 13 · p. 19
• Table 14 presents a configuration described as performing reasonably well for all transformer models.Source: Table 14 · p. 20
• For multivariate regression, Tables 15 and 16 separately specify supervised and unsupervised TST hyperparameters.Source: Table 15 · p. 20, Table 16 · p. 20
• For multivariate classification, Tables 17 and 18 separately specify supervised and unsupervised TST hyperparameters.Source: Table 17 · p. 20, Table 18 · p. 20