PaperlayerView exact revision on arXiv

Source-linked AI summary

A Transformer-based Framework for Multivariate Time Series Representation Learning

George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, Carsten Eickhoff

arXiv:2010.02803v3cs.LGcs.AI

TL;DR

Multivariate time-series applications often lack sufficient labeled data, motivating methods that learn useful representations from limited supervision or unlabeled samples. The paper develops a transformer-based unsupervised pre-training framework and evaluates it on regression and classification benchmarks, where it achieves leading performance, including with few training samples, while remaining computationally economical.

  • Problem

    Labeled multivariate time-series data are limited because extensive annotation is often prohibitively expensive or impractical, creating demand for accurate methods that use limited labels or unlabeled data.

  • Method

    The framework pre-trains a transformer encoder to extract dense multivariate time-series representations through an input-denoising autoregressive objective, then applies them to downstream tasks.

  • Results

    The framework is reported as the best-performing method for regression and classification across evaluated public datasets, including datasets with only a few hundred training samples.

  • Takeaways & Limitations

    Unsupervised pre-training provides substantial benefits over fully supervised learning even when it reuses the same samples rather than adding unlabeled data.

  • Takeaways & Limitations

    The self-attention layers have O(w^2) complexity with respect to time-series length w, which is identified as the main performance bottleneck.

Abstract

from arXiv · show

In this work we propose for the first time a transformer-based framework for unsupervised representation learning of multivariate time series. Pre-trained models can be potentially used for downstream tasks such as regression and classification, forecasting and missing value imputation. By evaluating our models on several benchmark datasets for multivariate time series regression and classification, we show that not only does our modeling approach represent the most successful method employing unsupervised learning of multivariate time series presented to date, but also that it exceeds the current state-of-the-art performance of supervised methods; it does so even when the number of training samples is very limited, while offering computational efficiency. Finally, we demonstrate that unsupervised pre-training of our transformer models offers a substantial performance benefit over fully supervised learning, even without leveraging additional unlabeled data, i.e., by reusing the same data samples through the unsupervised objective.

1 INTRODUCTION

The paper introduces a transformer encoder framework for unsupervised representation learning of multivariate time series, motivated by limited labeled data and transformers’ success in unsupervised sequence representation learning. It applies the framework to downstream time-series regression and classification.

  • 1 INTRODUCTION: Multivariate time series occur across science, medicine, finance, engineering, and industrial applications, but labeled data are often prohibitively expensive or impractical to obtain.The resulting label scarcity motivates methods that can learn from limited supervision or unlabeled data.
  • 1 INTRODUCTION: Deep learning has challenged the state of the art in forecasting, regression, and classification, although non-deep methods still hold leading benchmark performance.Examples include TS-CHIEF, HIVE-COTE, and ROCKET, which can match or outperform sophisticated deep architectures.
  • 1 INTRODUCTION: The work investigates a transformer encoder for unsupervised multivariate time-series representation learning and for regression and classification.This is presented as the first such use of a transformer encoder for these purposes.
  • 1 INTRODUCTION: Transformers are attractive because unsupervised pre-training has enabled strong representation learning in NLP, while their multi-headed attention mechanism offers advantages for time-series data.The paper connects transformer success in NLP with their proposed use for multivariate time series.
  • 1 INTRODUCTION: The framework is evaluated on multivariate time-series regression and classification tasks using transformer models.The supplied introduction describes the framework and its intended downstream applications, including forecasting and imputation.

2 RELATED WORK

Prior work relied mainly on specialized non-deep ensembles, CNNs, autoencoders, and task-specific transformer architectures. This paper instead frames transformers as a reusable unsupervised pre-training framework for varied time-series tasks.

  • Regression and classification of time series: TS-CHIEF, HIVE-COTE, and ROCKET were benchmark state-of-the-art methods for time-series regression and classification, followed by CNN architectures such as InceptionTime and ResNet.The leading non-deep methods use heterogeneous feature and classifier ensembles, while ROCKET uses random convolutional features with a linear classifier.
  • Unsupervised learning for multivariate time series: Unsupervised multivariate time-series learning predominantly used MLP or RNN autoencoders trained with input reconstruction objectives.Variational autoencoding, forecasting losses, attention, and similarity-preserving priors were explored in related variants.
  • Unsupervised learning for multivariate time series: Other unsupervised approaches used correlation-matrix reconstruction for anomaly detection or temporal-proximity triplet losses for representation learning and classification.The temporal-proximity approach was reported as the best unsupervised method for certain univariate and multivariate classification datasets, despite supervised methods outperforming it on univariate classification.
  • Transformer models for time series: Earlier transformer work addressed specific generative tasks such as univariate forecasting using full encoder-decoder architectures.The cited forecasting work compared favorably with ARIMA, TRMF, DeepAR, and DeepState on four public datasets.
  • Transformer models for time series: This work generalizes transformers from task-specific generative solutions to a reusable framework supporting unsupervised pre-training and multiple downstream tasks.The paper presents this direction as analogous to BERT’s role in converting translation-based transformers into a generic NLP framework.

3 METHODOLOGY

The framework adapts a transformer encoder to multivariate time series by embedding normalized feature vectors, adding positional information, and handling variable-length inputs. It supports supervised regression and classification, while unsupervised pre-training denoises independently masked input segments.

  • 3.1 BASE MODEL: The base architecture uses a transformer encoder without a decoder and adds learnable positional encodings so self-attention can represent temporal order.The model adds Wpos to the embedded input vectors; learnable encodings were reported to perform better than deterministic sinusoidal encodings on the presented datasets.
  • 3.1 BASE MODEL: Each multivariate sample is represented as a sequence of w feature vectors, with each vector normalized and linearly projected into the transformer’s d-dimensional representation space.The projection uses learnable parameters Wp and bp, producing input vectors that become self-attention queries, keys, and values after positional encoding.
  • 3.1 BASE MODEL: For fine-grained sequences, a one-dimensional convolution can produce the input vectors, while self-attention has computational complexity O(w^2) and parameter growth O(w) with sequence length w.The convolution uses kernels spanning k time steps and all m variables, with d output channels; stride or dilation can control temporal resolution.
  • 3.1 BASE MODEL: Variable-length samples are padded to a dataset-wide maximum length, and a padding mask suppresses attention to padded positions before the softmax distribution is computed.This allows samples with different lengths to be processed within the same maximum sequence length w.
  • 3.2 REGRESSION AND CLASSIFICATION: For regression and classification, the model concatenates final representations from all time steps and feeds them to a linear output layer.Regression uses squared error, whereas classification applies softmax and categorical cross-entropy; fine-tuning permits all model weights to train.
  • 3.3 UNSUPERVISED PRE-TRAINING: Unsupervised pre-training masks contiguous segments independently in each variable and trains the model to reconstruct the uncorrupted input using loss only on masked values.The experiments use mean masked-segment length lm = 3 because isolated masked values can often be predicted trivially from neighboring values.

4 EXPERIMENTS & RESULTS

Across regression and classification benchmarks, the transformer models generally ranked highest, while unsupervised pre-training improved performance over fully supervised training even when reusing the same samples.

  • Regression: The models ranked first on 4 of 6 regression datasets and second on the remaining two, achieving an average rank of 1.33.They attained 30% lower RMSE than the mean RMSE among all models.
  • Regression: Pre-trained transformers outperformed fully supervised transformers on 3 of 6 regression datasets without using additional pre-training samples.The reported benefit appears to originate from reusing the same training samples through an unsupervised objective.
  • Regression: Increasing the labeled-data proportion improved performance for both supervised and pre-trained models, while pre-training remained beneficial across the full label-availability range.The comparison used the BeijingPM25Quality dataset and evaluated RMSE against labeled training-set proportion.
  • Regression: Using more samples for unsupervised pre-training generally lowered error at fixed label counts, although the 10%-label setting showed a temporary increase.The 10% setting used approximately 1,250 labels, while the 20% setting used approximately 2,500 labels.
  • Classification: The models ranked first on 7 of 11 classification datasets, with an average rank of 1.7 versus ROCKET’s average rank of 2.3.ROCKET performed best on 3 datasets.
  • Classification: Pre-trained transformers outperformed fully supervised transformers on 8 of 11 classification datasets, again without additional samples for unsupervised pre-training.The reported benefit therefore appears to originate from reusing the same samples.

5 ADDITIONAL POINTS & FUTURE WORK

The framework supports imputation, forecasting, and direct use of learned representations, while computational efficiency and attention complexity remain important considerations.

  • Execution time for training: Standard self-attention has O(w^2) complexity in time-series length w, motivating alternative attention schemes for reducing the main bottleneck.The paper notes that sparse, recurrent, or compressed attention can reduce this complexity.
  • Imputation and forecasting: The same model and training process can perform missing-value imputation without modification.The authors report Root Mean Square Errors very close to 0 for input denoising after unsupervised pre-training.
  • Imputation and forecasting: Masking the final portion of all variables enables forecasting, with sliding windows suggested for longer time series.Systematic quantitative evaluation of forecasting is deferred to future work.
  • Extracted representations: Learned representations can support similarity evaluation, clustering, visualization, and other practical representation-based uses.Transformers also allow representations to be addressed independently at each time step.

6 CONCLUSION

The paper introduces a transformer-encoder framework for multivariate time-series representation learning and evaluates it across diverse regression and classification benchmarks. The framework achieves strong performance with unsupervised pre-training, including when training data are limited.

  • 6 CONCLUSION: The framework uses a transformer encoder for multivariate time-series representation learning with unsupervised pre-training.The authors reuse the same data samples through an unsupervised objective rather than requiring additional unlabeled data.
  • 6 CONCLUSION: Across public multivariate datasets, the framework is reported as the best-performing method for regression and classification.The evaluation covers datasets from various domains and with diverse characteristics.
  • 6 CONCLUSION: The method performs strongly even on datasets with only a few hundred training samples.The paper reports it as the only top-performing method based on deep learning in these evaluations.
  • 6 CONCLUSION: The framework is the first unsupervised method shown to push state-of-the-art performance for multivariate time-series regression and classification.

A.1 ADDITIONAL POINTS & FUTURE WORK

The appendix identifies computational complexity, imputation and forecasting extensions, learned representation uses, and supporting dataset-variability tables as additional considerations.

  • Execution time for training: Alternative attention schemes can reduce the O(w^2) self-attention complexity that forms the main performance bottleneck for sequence length w.The paper names sparse, recurrent, and compressed attention as possible alternatives.
  • Execution time for training: The transformer method is described as economical in its use of computational resources, although precise training-time comparison is out of scope.
  • Imputation and forecasting: The model supports imputation without modification and forecasting through masking patterns, but systematic quantitative investigations are deferred.
  • Extracted representations: Transformer representations can be used for similarity evaluation, clustering, visualization, and other time-series representation applications.Representations can be independently addressed at each time step.
  • Supporting tables: Table 5 reports standard deviations of Root Mean Square Error for Time Series Transformer models on multivariate regression datasets.

A.2 CRITERIA FOR DATASET SELECTION

The datasets were selected to span variation in dimensionality, length, sample count, and difficulty, with individual benchmarks covering markedly different experimental settings.

  • Dataset selection: The selected datasets were intended to vary in dimensionality, time-series length, sample count, and baseline difficulty.The collection includes both “easy” and “difficult” datasets.
  • Dataset selection: EthanolConcentration combines very low dimensionality, very few samples, many classes, and a challenging classification setting.
  • Dataset selection: FaceDetection has very high dimensionality, many samples, very short series, and a minimum number of classes.
  • Dataset selection: The collection also includes low-dimensional, few-sample datasets such as Handwriting, Heartbeat, SelfRegulationSCP1, and UWaveGestureLibrary.Their class counts, lengths, and baseline difficulty vary across datasets.
  • Dataset selection: InsectWingBeat combines very high dimensionality, many samples, very short series, and a very challenging classification setting.
  • Dataset selection: PEMS-SF contains extremely high-dimensional series with very few samples, moderate length, and a moderate number of classes.

A.3 EXECUTION TIME

The section compares transformer execution and training-related design choices across benchmark tasks, masking schemes, normalization, and representation fine-tuning.

  • Execution-time comparisons: ROCKET and XGBoost provide fast CPU baselines, while fully supervised transformer models are timed on a GPU.The comparison uses training time until convergence or maximum recorded accuracy.
  • Execution-time comparisons: LSTMs require O(w) sequential operations per sample, with per-layer complexity scaling as O(N · d^2).Here, w is the time-series length, and d is the internal representation dimension.
  • Unsupervised-learning design: Four masking schemes vary by whether variables are masked separately or synchronously and whether masks use Bernoulli sampling or a Markov chain.The schemes are evaluated on four classification and three regression datasets.
  • Unsupervised-learning design: Separate stateful masking achieves the overall best performance across all examined datasets, with mean masked-sequence length lm = 3 and average masking rate r = 15%.The masking probabilities produce geometrically distributed masked sequences.
  • Model-training choices: The supplementary comparisons include layer normalization versus batch normalization and fine-tuning all layers versus freezing the encoder and training only the output layer.The latter comparison also reports per-epoch GPU training time.

A.4 ADVANTAGES OF TRANSFORMERS

Transformers use multi-headed attention to model long-range, multi-aspect dependencies in time series without position-dependent prior bias, while progressively refining contextual representations.

  • Long-context modeling: Transformers concurrently consider long input contexts and selectively attend to relevant sequence elements without position-dependent prior bias.This contrasts with RNN-based models, whose treatment of sequence positions depends on their distance from endpoints.
  • Multi-aspect attention: Multiple attention heads can represent different relevance subspaces, such as neighboring points and points separated by signal periods T1 or T2.Different heads can therefore attend to distinct temporal structures in the same signal.
  • Progressive contextualization: Attention is redistributed after each transformer encoder layer as information flows toward progressively more abstract representations.RNN attention typically uses a single distribution over one representation layer.

A.5 HYPERPARAMETERS

The supplementary material specifies shared and task-specific hyperparameter configurations for supervised and unsupervised transformer models across regression and classification datasets.

  • Shared settings: Common fixed hyperparameters are specified for all transformer models.These settings provide a shared baseline configuration across model variants.
  • Shared settings: A hyperparameter configuration is provided that performs reasonably well across all transformer models.The configuration is intended as a broadly usable setting rather than a dataset-specific optimum.
  • Regression models: Separate supervised and unsupervised hyperparameter tables are provided for multivariate regression datasets.The configurations distinguish the training objectives used for the two model classes.
  • Classification models: Separate supervised and unsupervised hyperparameter tables are also provided for multivariate classification datasets.These tables specify task-specific configurations for the classification experiments.
Loading 2010.02803v3…