PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects

Kexin Zhang, Qingsong Wen, Chaoli Zhang, Rongyao Cai, Ming Jin, Yong Liu, James Zhang, Yuxuan Liang, Guansong Pang, Dongjin Song, Shirui Pan

arXiv:2306.10125v4cs.LGcs.AIeess.SPstat.AP

TL;DR

Time series SSL lacks a comprehensive survey despite reducing dependence on labeled data and facing modality-specific challenges. This article reviews existing methods, proposes a three-paradigm taxonomy with ten subcategories, and compiles datasets and future directions. Its synthesis covers generative, contrastive, and adversarial approaches across time series applications and identifies open issues including augmentation selection and non-stationary sample construction.

  • Problem

    A comprehensive and systematic review of time series SSL, including benchmark datasets and research directions, was missing despite limited labeled data and modality-specific challenges.

  • Method

    The article reviews state-of-the-art methods and organizes them into generative-based, contrastive-based, and adversarial-based paradigms with ten subcategories, datasets, and future directions.

  • Results

    The review synthesizes time series SSL methods across generative, contrastive, and adversarial approaches and summarizes applications and widely used datasets.

  • Takeaways & Limitations

    The taxonomy highlights open research directions involving augmentation combinations, contextual positive and negative sampling, collaborative systems, and method development.

  • Takeaways & Limitations

    Choosing suitable positive and negative samples remains challenging because most time series are non-stationary.

Abstract

from arXiv · show

Self-supervised learning (SSL) has recently achieved impressive performance on various time series tasks. The most prominent advantage of SSL is that it reduces the dependence on labeled data. Based on the pre-training and fine-tuning strategy, even a small amount of labeled data can achieve high performance. Compared with many published self-supervised surveys on computer vision and natural language processing, a comprehensive survey for time series SSL is still missing. To fill this gap, we review current state-of-the-art SSL methods for time series data in this article. To this end, we first comprehensively review existing surveys related to SSL and time series, and then provide a new taxonomy of existing time series SSL methods by summarizing them from three perspectives: generative-based, contrastive-based, and adversarial-based. These methods are further divided into ten subcategories with detailed reviews and discussions about their key intuitions, main frameworks, advantages and disadvantages. To facilitate the experiments and validation of time series SSL methods, we also summarize datasets commonly used in time series forecasting, classification, anomaly detection, and clustering tasks. Finally, we present the future directions of SSL for time series analysis.

1 INTRODUCTION

Time series SSL addresses the difficulty of obtaining labeled data, but existing CV/NLP pretext tasks do not transfer straightforwardly because time series have distinctive semantics. The article fills a survey gap with a taxonomy, resource collection, and future research directions.

  • Motivation: Time series applications require informative features, yet collecting sufficient labeled data for supervised learning is often time-consuming.SSL derives supervision from unlabeled data through pretext tasks, reducing dependence on manually labeled data.
  • Challenges: Time series properties such as seasonality, trend, and frequency information make direct transfer of CV/NLP pretext tasks non-trivial.
  • Research gap: Existing surveys do not yet provide a comprehensive, systematic review of time series SSL, benchmark datasets, and potential research directions.Earlier reviews cover only a small part of contrastive learning and leave broader coverage incomplete.
  • Contributions: The article organizes time series SSL into generative-based, contrastive-based, and adversarial-based paradigms, with ten detailed method categories.Generative methods include autoregressive forecasting, autoencoder reconstruction, and diffusion generation; contrastive methods are divided into five categories.
  • Contributions: The review also collects applications and datasets, compares quantitative performance, and discusses future directions from applicative and methodological perspectives.The article covers datasets and applications across common time series tasks and identifies key problems, causes, and possible solutions.

2 RELATED SURVEYS

The related-survey review compares SSL taxonomies based on learning paradigms, pretext tasks, and pipeline components, alongside surveys of time series tasks and modeling elements. It positions this article as broader than prior contrastive-focused or medical-time-series reviews.

  • SSL survey criteria: SSL surveys commonly organize methods by learning paradigms, pretext tasks, or components and modules.Learning-paradigm taxonomies typically include generative-based, contrastive-based, and adversarial-based methods.
  • Pretext tasks: Pretext-task surveys classify pseudo-supervision into families including transformation prediction, masked prediction, instance discrimination, clustering, and contrastive instance discrimination.A broader summary also groups pretexts into context prediction, instance discrimination, and instance generation.
  • Pipeline components: Contrastive-learning pipelines can be analyzed through positive and negative samples, pretext tasks, model architecture, and training loss.Architectures include end-to-end, memory bank, momentum encoder, and clustering designs, while losses include similarity, margin, NCE-based, and mutual-information functions.
  • Time series surveys: Time series surveys also focus on tasks such as classification, forecasting, and anomaly detection, or on modeling components such as augmentation and model structure.
  • Positioning: This article emphasizes general time series SSL and reviews generative-based and adversarial-based methods in addition to contrastive approaches.Prior related work discussed medical time series or only a small part of the broader method landscape.

3 GENERATIVE-BASED METHODS

Generative-based time series SSL learns representations by forecasting, reconstructing, or generating partially observed series. The survey covers autoregressive, autoencoder, and diffusion frameworks, including task-specific constraints and applications.

  • Overview: Generative-based SSL uses forecasting, reconstruction, or masked-series generation as pretext tasks for representation learning.
  • Autoregressive-based forecasting: Autoregressive forecasting predicts K future horizons from t historical time steps and minimizes distance between predicted and ground-truth future windows.K = 1 gives single-step forecasting, whereas K > 1 gives multi-step forecasting; distance is usually measured by MSE.
  • Autoencoder-based reconstruction: Autoencoders map an input series to a representation and decode it into a reconstruction, with training minimizing reconstruction error.After joint training, the decoder can be removed and the encoder representation used for downstream tasks.
  • Autoencoder-based reconstruction: Time series autoencoder variants add task-specific constraints, including spectral analysis, K-means clustering, adversarial training, and graph learning.These additions target synchronous representations, clustering-friendly features, enhanced representation ability, or explicit inter-variable relationships.
  • Autoencoder-based reconstruction: Denoising and mask autoencoders corrupt or mask inputs and reconstruct or predict the affected portions, with masking strategies trading fine-grained information against contextual semantics.Time-step-wise masking captures fine-grained information, while segment-wise masking emphasizes trends or higher-level semantics.
  • Diffusion-based generation: Diffusion-based methods use forward noise injection and reverse generation, supporting time series imputation, forecasting, anomaly detection, and probabilistic forecasting.Examples include CSDI, TimeGrad, D3VAE, ImDiffusion, SSSD, DiffLoad, and DiffSTG.

4 CONTRASTIVE-BASED METHODS

Contrastive-based SSL learns time-series representations by making positive samples similar and negative samples different. The review organizes these methods into five categories based on how samples or views are selected, while highlighting challenges from non-stationarity, representation bias, and time-series-specific semantics.

  • Contrastive methods learn representations by pulling positive samples together and separating negative samples.
  • Sampling contrast: Sampling contrast selects nearby windows or timestamps as positives and distant ones as negatives, but non-stationarity complicates contextual sampling.TNC addresses this with the ADF test and positive-unlabeled learning to handle sampling bias.
  • Prediction contrast: Prediction contrast uses context-target pairs and proposal-distribution samples in InfoNCE-style objectives to preserve mutual information across time.CPC does not directly predict future observations; it aims to capture slow features spanning multiple time steps.
  • Augmentation contrast: Augmentation contrast maximizes agreement between views generated from the same time-series sample while separating views from different samples.Frequency-domain methods use Fourier-based representations or perturbations; TF-C reports better performance than TS2Vec and TS-TCC.
  • Prototype contrast: Prototype contrast introduces cluster centers to reduce computation and encourage cluster-friendly feature distributions instead of uniform instance-level representations.This addresses the mismatch between instance discrimination and class-clustered data distributions when labels are unavailable.
  • Expert knowledge contrast: Expert-knowledge contrast incorporates prior information to address sampling bias and representation bias, including clustering, labels, and semi-supervised guidance.The review identifies the gap between pretext-task objectives and downstream-task goals as a source of representation bias.

5 ADVERSARIAL-BASED METHODS

Adversarial-based SSL uses GAN-style generator–discriminator pretext tasks and is organized around generation/imputation and auxiliary representation enhancement. The reviewed methods apply these strategies across multiple time series tasks.

  • Adversarial-based framework: Adversarial-based SSL uses a generator to create synthetic data and a discriminator to distinguish generated from real samples.The generator seeks to maximize the discriminator’s decision failure rate.
  • Taxonomy: The review divides adversarial-based methods into generation and imputation, and auxiliary representation enhancement.Figure 4 illustrates these adversarial-based SSL categories, while Appendix G compares their characteristics and limitations with other paradigms.
  • Generation: TimeGAN combines GANs with autoregressive modeling to preserve temporal dynamics and emphasizes both static features and temporal characteristics.C-RNN-GAN uses an RNN generator and bidirectional RNN discriminator for time series sample generation.
  • Generation: COSCI-GAN models multivariate generation by using channel-specific GANs and a central discriminator to assess cross-dimensional correlations.Channel GANs generate each dimension independently, while the central discriminator evaluates relationships among dimensions.
  • Auxiliary representation enhancement: Auxiliary adversarial objectives enhance representations without being required for representation extraction, as shown by anomaly detection, retrieval, clustering, forecasting, and ECG applications.USAD uses adversarial training between two autoencoders to amplify reconstruction error, while AnomalyTrans amplifies anomaly scores through association discrepancies.

6 APPLICATIONS AND DATASETS

The review organizes time series applications and datasets around anomaly detection, forecasting, classification, and clustering. It summarizes task descriptions, related methods, evaluation information, and commonly used benchmark datasets.

  • Applications and datasets: The application survey covers anomaly detection, forecasting, classification, and clustering, with datasets summarized by names, dimensions, sizes, sources, and comments.The review also reports task descriptions, related methods, evaluation metrics, examples, and task flows.
  • Anomaly detection: Anomaly detection identifies outlier time points or unexpected sequences relative to a normal signal.Because anomaly labels are difficult to obtain, most methods use unsupervised learning, especially autoregressive forecasting and autoencoder reconstruction.
  • Forecasting: Forecasting predicts future windows or time points, with autoregressive SSL naturally serving as a forecasting pretext task.The review lists ETTh, ETTm, Electricity, and Weather among commonly used forecasting datasets.
  • Classification and clustering: Classification and clustering identify the category to which a time series sample belongs, so their datasets are listed together.Contrastive methods are described as particularly suitable because they distinguish positive and negative samples and learn robust representations.

7 DISCUSSION AND FUTURE DIRECTIONS

The discussion identifies unresolved issues in time series SSL and proposes directions involving augmentation evaluation, inductive bias, irregular data, large-scale pretraining, robustness, benchmarking, and distributed learning.

  • Selection and combination of data augmentation: Different augmentation combinations can produce different recommendations, motivating a more reasonable evaluation framework for time series SSL.A study across 128 classification datasets and six neural-network types found that evaluation frameworks affect recommendations and results.
  • Inductive bias: Future SSL models should incorporate time series inductive biases such as seasonality, periodicity, trend, and frequency-domain structure.Purely data-driven models are described as broadly extensible but data-hungry.
  • Irregular and sparse time series: Irregular and sparse series motivate SSL models that operate without interpolation because interpolation can add noise and computational overhead.These costs generally worsen as the series becomes increasingly sparse.
  • Large-scale pretraining: Large-scale time series pretraining remains limited, making large time series models a promising research direction.The paper frames this as an open question inspired by large language and vision models.
  • Robustness and evaluation: The impact of adversarial examples on time series SSL pretraining tasks remains unknown despite broader concerns about deep-model vulnerability and robustness.The review also calls for reliable benchmark evaluation of SSL properties rather than focusing only on task-specific performance.
  • Distributed learning: Distributed learning is relevant because users can train local models without sharing private data while time series are distributed across system locations.The discussion presents collaborative learning as important for distributed systems.

8 CONCLUSION

The article presents a taxonomy and synthesis of time series SSL methods, applications, datasets, and future directions. Its framework distinguishes generative, contrastive, and adversarial learning paradigms and organizes common pretext-task designs.

  • Conclusion: The review categorizes time series SSL into generative-based, contrastive-based, and adversarial-based paradigms with ten detailed subcategories.The subcategories include forecasting, reconstruction, diffusion generation, several contrastive forms, generation/imputation, and auxiliary representation enhancement.
  • Generative-based methods: Generative methods encode input x into representation z and decode z to reconstruct x by minimizing reconstruction error.This formulation describes the generative-based architecture summarized in Figure 5.
  • Contrastive-based methods: Contrastive methods construct positive and negative samples through augmentation or context sampling and maximize mutual information between positive samples.InfoNCE is given as a commonly used contrastive similarity metric.
  • Adversarial-based methods: Adversarial methods pair a generator that produces fake samples with a discriminator that distinguishes fake from real data.This generator–discriminator structure is the defining architecture summarized for the adversarial paradigm.
  • Contrastive pretext construction: Contrastive pretexts use multisensory signals, augmentation, local-global consistency, or temporal consistency to construct related views and sample relationships.For temporal data, nearby or continuous views are treated as positive pairs, while distant or discontinuous views are treated as negatives.
  • Pretext-task classification: Across pretext tasks, context prediction models contextual relationships, instance discrimination separates instances or groups, and instance generation reconstructs or generates related instances.Table 3 classifies these commonly used pretext-task families.

B.3 Characteristics of four SSL model architectures •

SSL model architectures differ in how they obtain representations, manage negative samples, and measure relationships between views or samples.

  • Model architectures: End-to-end architectures use two same-structure encoders for different views and update both with contrastive loss.They generally favor large batches to accumulate more negative samples.
  • Model architectures: Memory banks reduce the computing demands of large training batches by storing and periodically updating sample embeddings.Representation consistency cannot be guaranteed.
  • Model architectures: Momentum encoders provide a dynamic dictionary lookup for negative-sample encodings during training.
  • Model architectures: Clustering reduces reliance on large batches while grouping similar features into clusters alongside bringing paired samples together.
  • Scoring functions: Scoring functions quantify compatibility through similarity or distance, while energy, NCE, and mutual-information objectives impose different contrastive criteria.Energy functions separate positive and negative pairs by energy; NCE identifies positives among negatives; mutual-information objectives align views of the same instance.

APPENDIX C ADVANTAGES AND DISADVANTAGES OF THREE GENERATIVE-BASED METHODS

Generative-based time-series SSL methods include forecasting, reconstruction, and diffusion, each offering distinct representation-learning benefits and limitations.

  • Autoregressive-based forecasting: Autoregressive forecasting uses raw temporal structure and slicing to create abundant unlabeled training data without manual annotation.It directly builds on temporal dependencies in the series.
  • Autoregressive-based forecasting: Forecasting errors may accumulate over time in long-term prediction, biasing extracted representations.
  • Autoencoder-based reconstruction: Autoencoder reconstruction is easy to use for feature learning and dimensionality reduction, while noise or transformations can improve robustness.
  • Autoencoder-based reconstruction: Autoencoders may fail to capture long-term dependencies and complex seasonality, trends, and noise in large-scale, high-dimensional series.
  • Diffusion-based generation: Diffusion models offer concise, flexible workflows that can integrate with other generative models, but semantic extraction, sampling, and likelihood estimation remain difficult.
  • Model architectures: RNNs capture long-term and global context, CNNs efficiently extract local patterns, and GNNs model variable relationships in multivariate series.
  • Model architectures: RNNs face gradient and efficiency problems, CNNs can miss long-term dependencies and lose information, and GNNs incur complexity and graph-size limitations.

APPENDIX E ADVANTAGES AND DISADVANTAGES OF FIVE CONTRASTIVE-BASED METHODS

Contrastive-based methods differ in how they construct comparisons and incorporate temporal structure, semantic prototypes, prior knowledge, or auxiliary strategies.

  • Sampling contrast: Sampling contrast models local correlations simply but may create false negative pairs when representing long-term dependencies.
  • Prediction contrast: Prediction contrast learns representations by predicting future information and emphasizing slowly changing trends, but may underrepresent long-term dependencies.
  • Augmentation contrast: Augmentation contrast is easy to implement across tasks but struggles with temporal dependencies, augmentation selection, and sampling bias.
  • Prototype contrast: Prototype contrast assigns samples to finite prototype classes and encourages clustered feature distributions, but requires the prototype count beforehand.
  • Adversarial-based methods: Adversarial-based methods provide robust auxiliary strategies and strong sample generation, but can be unstable, collapse, or be rarely used for feature extraction.
  • Expert knowledge contrast: Expert-knowledge contrast can select positive and negative samples accurately, but depends on reliable prior knowledge that is difficult to obtain.Incorrect knowledge can produce biased representations.

APPENDIX F ADVANTAGES AND DISADVANTAGES OF TWO ADVERSARIAL-BASED METHODS

Adversarial-based methods support time-series generation, imputation, and robust representation learning, while introducing data-quality and training-stability constraints.

  • Time-series generation and imputation: GANs can generate high-quality time-series samples and perform imputation according to series seasonality and trends.Insufficient or poor-quality training data can reduce consistency and produce unreasonable outputs.
  • Time-series generation and imputation: GAN training is complex because generator–discriminator balance and mode collapse must be addressed.
  • Auxiliary representation enhancement: Adversarial regularization can improve representation robustness and generalization, but increases training complexity, resource demands, and instability.

APPENDIX G CHARACTERISTICS AND LIMITATIONS OF THREE SSL METHODS

The survey compares SSL methods across anomaly detection, forecasting, classification, and clustering, emphasizing task-specific evaluation and differing empirical strengths.

  • Anomaly detection: Anomaly detection commonly uses precision, recall, and F1-score, with an adjustment strategy that counts all anomalies in a detected abnormal segment as correct.The detector converts a time-series sample into anomaly scores, thresholds them into binary predictions, and annotates anomalies from the comparison.
  • Forecasting: Forecasting evaluates mean squared error and mean absolute error after encoding historical-window context and decoding it into future-window predictions.SSL is applied to improve the encoder’s historical-window representations.
  • Classification and clustering: Classification and clustering commonly use accuracy, but clustering accuracy accounts for label permutations whereas classification accuracy directly compares predicted and true labels.SSL is usually applied during encoder training so representations support downstream label prediction.
  • Benchmark datasets: Five datasets are frequently used for anomaly detection, while ETTh, ETTm, Electricity, and Weather are commonly used for forecasting comparisons.The survey collects results according to downstream tasks and emphasizes comparisons using the same datasets.
  • Anomaly detection results: Generative and contrastive methods are preferred for anomaly detection, with Skip-CPC and Dcdetector achieving the best performance on four datasets.Global/local association modeling and regularization constraints are identified as factors supporting more accurate anomaly detection.
  • Forecasting and classification results: LaST and CoST show significant advantages in forecasting, while classification and clustering results indicate that no single method performs best across all 30 datasets.The survey links forecasting advantages to disentangled representations and recommends selecting models according to data characteristics for classification and clustering.

I.2 Correlation analysis between methods and tasks

The survey relates SSL paradigms to downstream tasks and finds that method suitability varies by task, with hybrid approaches also represented.

  • Method–task correlation: Generative-based SSL is more suitable for anomaly detection and forecasting, whereas contrastive-based SSL is more suitable for classification and clustering.The survey attributes these differences to the task characteristics and objectives associated with each method family.
  • Method–task correlation: Adversarial-based methods are widely used for forecasting and anomaly detection because most include a generator, and they also serve as auxiliary regularization for robust representations.The cited discussion presents adversarial losses as a way to make extracted features more robust and informative.
  • Method–task correlation: Existing results also include hybrid approaches that mix multiple SSL strategies rather than applying only one method family.
Loading 2306.10125v4…