PaperlayerView exact revision on arXiv

Source-linked AI summary

TS2Vec: Towards Universal Representation of Time Series

Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, Bixiong Xu

arXiv:2106.10466v4cs.LGcs.AI

TL;DR

Existing methods may be too coarse for fine-grained time-series tasks, may miss multi-scale context, and may rely on invariance assumptions that do not always hold. TS2Vec addresses these gaps with hierarchical contrastive learning over augmented context views and timestamp-level aggregation for arbitrary sub-series. Across classification, forecasting, and anomaly detection, the paper reports that the learned representations are effective and universal.

  • Problem

    Existing time-series representation methods can be unsuitable for fine-grained tasks, omit multi-scale contextual information, and rely on invariance assumptions that may not hold for changing time-series distributions.

  • Method

    TS2Vec hierarchically contrasts instance-wise and temporal representations across augmented context views, then aggregates timestamp representations to represent arbitrary sub-series.

  • Results

    TS2Vec representations are effective across time-series classification, forecasting, and anomaly detection, outperforming existing SOTAs on the three benchmark tasks.

  • Takeaways & Limitations

    The framework provides a unified representation method for arbitrary sub-series at various semantic levels across the three evaluated time-series tasks.

Abstract

from arXiv · show

This paper presents TS2Vec, a universal framework for learning representations of time series in an arbitrary semantic level. Unlike existing methods, TS2Vec performs contrastive learning in a hierarchical way over augmented context views, which enables a robust contextual representation for each timestamp. Furthermore, to obtain the representation of an arbitrary sub-sequence in the time series, we can apply a simple aggregation over the representations of corresponding timestamps. We conduct extensive experiments on time series classification tasks to evaluate the quality of time series representations. As a result, TS2Vec achieves significant improvement over existing SOTAs of unsupervised time series representation on 125 UCR datasets and 29 UEA datasets. The learned timestamp-level representations also achieve superior results in time series forecasting and anomaly detection tasks. A linear regression trained on top of the learned representations outperforms previous SOTAs of time series forecasting. Furthermore, we present a simple way to apply the learned representations for unsupervised anomaly detection, which establishes SOTA results in the literature. The source code is publicly available at https://github.com/yuezhihan/ts2vec.

1 Introduction

Existing time-series representation methods often provide coarse instance-level features, lack multi-scale contextual modeling, and rely on invariance assumptions that may not hold over time. TS2Vec addresses these issues with hierarchical contrastive learning over augmented context views, producing representations for arbitrary sub-series and supporting classification, forecasting, and anomaly detection.

  • Motivation: Existing instance-level representations are too coarse for timestamp-specific or sub-series tasks such as forecasting and anomaly detection.These tasks require fine-grained representations rather than a single representation of the whole time series.
  • Motivation: Prior methods rarely distinguish contextual information across multiple granularities, limiting their ability to capture scale-invariant information.Multi-scale features may provide different semantic levels and improve representation generalization.
  • Motivation: Cropping- and transformation-invariance assumptions from computer vision and NLP may fail because time-series distributions and semantics can change over time.A cropped sub-sequence may have a distinct distribution from the original series.
  • TS2Vec: TS2Vec hierarchically contrasts positive and negative samples across instance-wise and temporal dimensions to learn contextual representations at multiple resolutions.The framework uses augmented context views and aggregates timestamp representations with max pooling to represent arbitrary sub-series.
  • Evaluation: Experiments evaluate TS2Vec representations on time-series classification, forecasting, and anomaly detection tasks.The introduction reports that these results validate the generality of the learned representations.

2 Method

TS2Vec learns time-series representations across semantic levels by combining contextual consistency with hierarchical contrastive learning. Its architecture produces timestamp-level representations and aggregates them across multiple resolutions for arbitrary sub-series.

  • Model Architecture: TS2Vec samples overlapping subseries and encourages consistent representations on their common segment using temporal and instance-wise contrastive losses.The total loss is summed over multiple scales in a hierarchical framework.
  • Model Architecture: The encoder projects observations into latent vectors, masks randomly selected timestamps, and applies a dilated CNN to extract contextual representations.Timestamp masking is performed on latent vectors rather than raw values, while dilated convolutions provide a large receptive field.
  • Contextual Consistency: Contextual consistency treats representations at the same timestamp in two augmented contexts as positive pairs, avoiding assumptions that subseries or adjacent segments are always similar.Contexts use timestamp masking and random cropping, which preserve the time series magnitude and improve robustness across distinct contexts.
  • Hierarchical Contrasting: Hierarchical contrastive learning applies instance-wise and temporal losses at all granularity levels, using max pooling to recursively form coarser representations.Contrasting at top semantic levels enables instance-level representations, while the joint losses capture complementary series characteristics and temporal dynamics.
  • Hierarchical Contrasting: Compared with prior methods that contrast at only one level or enforce smoothness at a fixed granularity, TS2Vec learns representations across multiple levels.The comparison includes T-Loss, TS-TCC, and TNC.
  • Hierarchical Contrasting: Temporal contrast uses the same timestamp across two augmented views as a positive pair and different timestamps from the same series as negatives.The overlap of the two subseries defines the timestamps used for positive comparisons.

3 Experiments

TS2Vec is evaluated across classification, forecasting, and anomaly detection, using pooled representations, linear regression, and contextual-consistency anomaly scoring. It improves over existing methods across all three tasks, with reported gains on broad classification, forecasting, and anomaly-detection benchmarks.

  • Time Series Classification: TS2Vec uses max pooling over timestamps to obtain instance-level representations for classification, followed by an RBF-kernel SVM.This evaluation follows the protocol used by T-Loss.
  • Time Series Classification: 2.4% higher average classification accuracy on 125 UCR datasets and 3.0% on 29 UEA datasets were reported for TS2Vec.The paper also reports significantly better average ranks in the Critical Difference analysis.
  • Time Series Classification: TS2Vec provides the shortest training time among the compared representation-learning methods.The paper attributes this efficiency to applying contrastive losses across different granularities in one batch.
  • Time Series Forecasting: A linear regression using the last-timestamp representation predicts future observations from the preceding time-series window.The regressor uses an L2 penalty and directly predicts H future values, or FH values for multivariate series.
  • Time Series Forecasting: 32.6% lower average MSE in univariate forecasting and 28.2% lower average MSE in multivariate forecasting were achieved by TS2Vec.The representations can be learned once per dataset and reused across forecasting horizons with separate linear regressions.
  • Time Series Anomaly Detection: 18.2% and 5.5% F1 improvements over the best baselines were reported on Yahoo and KPI in the normal setting, while cold-start gains were 19.7% and 1.0%.Similar scores across settings support transfer from FordA to Yahoo and KPI in the cold-start protocol.

4 Analysis

The analysis examines TS2Vec’s components, robustness to missing data, and learned representations. Its ablations identify the effects of augmentation and backbone choices, while visualizations illustrate timestamp-level contextual behavior.

  • Ablation Study: Adding jitter, scaling, and permutation decreases performance, supporting TS2Vec’s use of contextual consistency under time-series distributions that may change.These augmentations assume invariances that do not hold for diverse and changing time-series distributions.
  • Ablation Study: Replacing the dilated CNN with an LSTM or Transformer significantly decreases accuracy, indicating the dilated CNN is an effective backbone for time series.
  • Robustness to Missing Data: Hierarchical contrasting and timestamp masking support robustness to missing points by inferring representations under incomplete contexts and incorporating long-range information.
  • Robustness to Missing Data: 50% missing values produce almost no accuracy change on UWaveGestureLibraryAll and decreases of 2.1%, 2.1%, and 1.2% on StarLightCurves, HandOutlines, and MixedShapesRegularTrain.Without hierarchical contrast or timestamp masking, accuracy drops rapidly as the missing rate grows.
  • Visualized Explanation: TS2Vec representations distinguish high and low values, reflect evolving volatility, and separate spiked timestamps from normal timestamps across three datasets.The visualizations use the 16 representation dimensions with the largest variances for the first test sample.

5 Conclusion

The conclusion presents TS2Vec as a universal time-series representation framework and summarizes its broad evaluation scope. The supplied passages also situate the work within representation learning, forecasting, and anomaly-detection research.

  • Conclusion: TS2Vec learns scale-invariant representations within augmented context views through hierarchical contrasting and evaluates them on classification, forecasting, and anomaly detection.
  • Related Work: The paper positions TS2Vec within unsupervised time-series representation learning, alongside methods developed for computer vision, natural language processing, speech recognition, and time series.
  • Related Work: Deep learning approaches including RNNs, CNNs, GNNs, and Transformers are widely used for forecasting and outperform classical ARIMA and VAR models in the cited related work.
  • Related Work: Statistical methods, FFT, SPOT, and Twitter are cited as established approaches to unsupervised time-series anomaly detection.

B Supplement of Method

This supplement explains why TS2Vec projects inputs before timestamp masking. The projection creates latent vectors for which the mask value can be distinguished from any projected input.

  • B.1 Input Projection Layer: Raw-input masking with zero is ambiguous because time series may contain segments whose actual values are all zero.
  • B.1 Input Projection Layer: TS2Vec applies an input projection layer before timestamp masking to map input vectors into a higher-dimensional latent space.
  • B.1 Input Projection Layer: The latent vector at each masked timestamp is set to zero, while suitable projection parameters ensure unmasked projected inputs are never zero.The condition F′ > F supports choosing a bias outside the span of the projection matrix.
  • B.1 Input Projection Layer: Applying timestamp masking after the input projection layer achieves better performance than the alternative in the reported ablation.

B.2 Random Cropping

Random cropping aligns overlapping context views while preventing the encoder from relying on absolute position. The analysis links its removal to positional representation collapse.

  • B.2 Random Cropping: TS2Vec samples two overlapping segments and requires their contextual representations to agree on the shared interval.The segments satisfy 0 < a1 ≤ a2 ≤ b1 ≤ b2 ≤ T, with consistency enforced on [a2, b1].
  • B.2 Random Cropping: Random cropping makes it impossible to infer a time point’s absolute position on the overlap, encouraging position-agnostic representations.
  • B.2 Random Cropping: Without random cropping, α approaches 1 and β drops during later training, indicating representation collapse.
  • B.2 Random Cropping: With random cropping, β remains relatively high and the collapse is avoided, whereas the network otherwise learns positional embeddings while overlooking contextual information.

C.1 Data Preprocessing

The experiments standardize inputs with z-score normalization, handle variable lengths and missing observations through padding and masks, and add calendar features only when forecasting timestamps are available. TS2Vec uses fixed empirically chosen representation-learning hyperparameters, while baseline implementations follow task-specific reproduction settings.

  • Univariate datasets are standardized to zero mean and unit variance, while each variable in multivariate datasets is normalized independently.
  • Variable-length series are padded to a common length, with NaNs marking missing observations and a zero mask at each missing position.
  • Forecasting inputs receive available calendar features, including minute, hour, day-of-week, and month-of-year, but classification datasets do not.
  • TS2Vec uses fixed empirical hyperparameters without downstream-task optimization because labels and tasks are unavailable during unsupervised representation learning.
  • The default setup uses batch size 8, learning rate 0.001, 200 or 600 optimization iterations by dataset size, and representation dimension 320.
  • Changing hyperparameters can improve some datasets while worsening others, and batch size notably affects individual-dataset performance.

C.4 Details for Benchmark Tasks

Benchmark protocols evaluate pooled instance representations for classification, timestamp representations for forecasting, and delay-tolerant anomaly detection. Forecasting uses regularized linear prediction and reports test-set metrics averaged across slices and instances.

  • Time Series Classification: Classification evaluates max-pooled instance representations with an RBF-kernel SVM whose penalty C is selected by training-set cross-validation.
  • Time Series Classification: TS2Vec works on all UCR and UEA datasets, while comparisons exclude datasets that certain baselines cannot process because of missing observations.
  • Time Series Forecasting: Forecasting trains ridge regression on learned timestamp representations to predict future values, selecting the L2 regularization term on validation data.
  • Time Series Forecasting: Forecasting uses MSE and MAE, with each dataset score averaged over all slices and instances.
  • Time Series Anomaly Detection: Anomaly detections within seven steps for minutely data or three steps for hourly data count as correct within continuous anomaly segments.

D Full Results

TS2Vec generally establishes a new forecasting state of the art, reducing average MSE by 32.6% in univariate settings and 28.2% in multivariate settings.

  • 32.6% decrease of average MSE is achieved in the univariate setting, compared with a 28.2% decrease in the multivariate setting.

D.2 Time Series Classification

TS2Vec performs strongly across classification benchmarks and supports transfer between datasets. Its average performance is robust across batch sizes, although individual datasets vary, and learned representations separate classes in latent space.

  • TS2Vec achieves the best average accuracy among unsupervised representation methods on 128 UCR datasets and reports results on 30 multivariate UEA datasets.
  • Different batch sizes produce close average scores but notable score differences on individual datasets.
  • 3.8% average accuracy improvement is achieved by transfer TS2Vec† over transfer T-Loss†, while transfer scores remain close to the non-transfer version.
  • The non-ensemble TS2Vec representation with size 320 outperforms T-Loss-4X, whose ensemble representation size is 1280.
  • Visualization: T-SNE visualizations use colors for classes and indicate that learned representations distinguish different classes in latent space.
  • StemGNN fails for all H ≥672 cases because of out-of-memory errors even with batch size 1.
Loading 2106.10466v4…