PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-supervised Contrastive Representation Learning for Semi-supervised Time-Series Classification

Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee-Keong Kwoh, Xiaoli Li, Cuntai Guan

arXiv:2208.06616v3cs.LG

TL;DR

Time-series representation learning is difficult when unlabeled data or only a few labeled samples are available. The paper proposes TS-TCC and CA-TCC, combining time-series-specific augmentations with temporal and contextual contrasting, and reports effective representations comparable to fully supervised training in linear evaluation.

  • Problem

    Limited labeled time-series data makes it challenging to learn useful representations for deep models.

  • Method

    TS-TCC uses weak and strong time-series augmentations with temporal and contextual contrasting, while CA-TCC adds pseudo-label-based class-aware contrasting for semi-supervised learning.

  • Results

    Linear evaluation of TS-TCC features achieves performance comparable to fully supervised training, while both variants improve transferability in real-world transfer learning scenarios.

  • Takeaways & Limitations

    The framework provides effective representations for unlabeled, few-label, and transfer-learning scenarios.

Abstract

from arXiv · show

Learning time-series representations when only unlabeled data or few labeled samples are available can be a challenging task. Recently, contrastive self-supervised learning has shown great improvement in extracting useful representations from unlabeled data via contrasting different augmented views of data. In this work, we propose a novel Time-Series representation learning framework via Temporal and Contextual Contrasting (TS-TCC) that learns representations from unlabeled data with contrastive learning. Specifically, we propose time-series-specific weak and strong augmentations and use their views to learn robust temporal relations in the proposed temporal contrasting module, besides learning discriminative representations by our proposed contextual contrasting module. Additionally, we conduct a systematic study of time-series data augmentation selection, which is a key part of contrastive learning. We also extend TS-TCC to the semi-supervised learning settings and propose a Class-Aware TS-TCC (CA-TCC) that benefits from the available few labeled data to further improve representations learned by TS-TCC. Specifically, we leverage the robust pseudo labels produced by TS-TCC to realize a class-aware contrastive loss. Extensive experiments show that the linear evaluation of the features learned by our proposed framework performs comparably with the fully supervised training. Additionally, our framework shows high efficiency in the few labeled data and transfer learning scenarios. The code is publicly available at \url{https://github.com/emadeldeen24/CA-TCC}.

1 INTRODUCTION

The paper addresses limited labeled time-series data by proposing TS-TCC for unlabeled learning and CA-TCC for semi-supervised learning. It combines time-series-specific augmentations with temporal and contextual contrasting, and reports effective representations across real-world datasets.

  • Limited labeling makes training deep models for time-series data challenging despite increasingly collected sensor data.
  • CA-TCC uses pseudo labels from TS-TCC to maximize similarity within classes and minimize similarity across classes during supervised contextual contrasting.
  • TS-TCC learns time-series representations from unlabeled data, while CA-TCC extends it to semi-supervised learning with few labeled samples.
  • Time-series-specific weak and strong augmentations create correlated views for contrastive representation learning.
  • Temporal contrasting uses cross-view prediction of future latent features to learn robust representations, while contextual contrasting learns more discriminative representations.
  • Experiments on ten real-world datasets show that both framework variants learn effective representations across different scenarios.

2 RELATED WORKS

Prior work applies self-supervised, contrastive, frequency-domain, and semi-supervised methods to time-series representation learning. The paper distinguishes its approach by jointly addressing temporal and global features through contrasting different augmented views.

  • Self-supervised learning: Earlier self-supervised approaches for images and videos use manually designed pretext tasks such as colorization, sequence reconstruction, future prediction, and frame-order verification.
  • Contrastive learning: Contrastive methods learn invariant representations from augmented views, with approaches differing in how they select positive and negative samples.
  • Contrastive learning: Image-oriented contrastive methods may not transfer directly to time-series data because time-series signals depend on temporal structure and may not suit image augmentations such as color distortion.
  • Self-supervised learning for time-series: Time-series self-supervised methods include transformation classification, future prediction, relative positioning, temporal shuffling, EEG-specific augmentation, and timestep- or instance-wise contrasting.
  • Frequency-domain methods: Other approaches fuse or contrast time-domain and frequency-domain representations to learn time-series features.
  • Positioning: Unlike approaches using only temporal or global features, this paper addresses both through cross-view temporal and contextual contrasting with time-series-specific augmentations.
  • Semi-supervised learning: Semi-supervised time-series methods use generic regularization or self-training, including temporal ensembling, reconstruction, clustering-based pseudo labels, and self-distillation.

3 METHODS

TS-TCC learns time-series representations by combining weak and strong augmented views with temporal cross-view prediction and contextual contrastive learning. CA-TCC extends this pipeline with pseudo-label-driven supervised contextual contrastive training for semi-supervised data.

  • TS-TCC: TS-TCC generates correlated views using separate weak and strong augmentations, aiming to improve representation robustness and generalization.Weak augmentation preserves signal characteristics, whereas strong augmentation substantially perturbs shape while retaining temporal information.
  • TS-TCC: The encoder maps each augmented view into timestep-wise latent representations that feed the temporal contrasting module.A three-block convolutional encoder produces z = [z_1, ..., z_T], with one d-dimensional feature per timestep.
  • Temporal Contrasting: Temporal contrasting uses autoregressive contexts to predict future timesteps across views, increasing similarity for matching samples and decreasing it for other batch samples.The strong-view context predicts weak-view futures and vice versa through a tough cross-view prediction task.
  • Temporal Contrasting: A Transformer autoregressive model summarizes past latent features with a classification token whose final state becomes the context passed to contextual contrasting.The Transformer applies multi-headed attention and an MLP through stacked layers before extracting the context token.
  • Contextual Contrasting: Contextual contrasting projects contexts and maximizes similarity between views of the same sample while minimizing similarity with other samples in the minibatch.Each context has one positive counterpart from the other view and 2N − 2 negatives from other inputs.
  • Class-Aware TS-TCC: CA-TCC pretrains TS-TCC, fine-tunes its encoder on few labeled samples, generates pseudo-labels, and applies supervised contextual contrasting.The supervised loss treats same-class samples as positives and different-class samples as negatives; its overall objective combines temporal and supervised contextual losses.

4 EXPERIMENTAL SETUP

The experiments use ten real-world time-series datasets spanning human activity, sleep, epilepsy, fault diagnosis, and UCR applications, with transfer learning assessed separately. Data are split into training, validation, and testing sets, and repeated runs report mean and standard deviation.

  • Datasets: The evaluation uses ten publicly available real-world datasets covering diverse time-series applications, plus a fault-diagnosis transferability experiment.The datasets include UCI HAR, Sleep-EDF, Epileptic Seizure Recognition, Fault Diagnosis, and seven UCR datasets.
  • Datasets: The Fault Diagnosis dataset contains four working-condition domains, each with inner-fault, outer-fault, and healthy classes, and is used only for transfer learning.Each domain has different characteristics such as rotational speed and load torque.
  • Datasets: The seven selected UCR datasets permit few-label experiments because 1% of training samples includes all classes.The selected datasets are Wafer, FordA, FordB, POC, PPOC, StarLightCurves, and ElectricDevices.
  • Protocol: Data are split 60%, 20%, and 20% into training, validation, and testing, with subject-wise splitting for Sleep-EDF to avoid overfitting.Experiments are repeated five times with five seeds, and performance is reported using mean and standard deviation.
  • Protocol: The experiments use 40 training epochs for pretraining and downstream tasks, with batch size 128 before reductions for few-labeled settings.Training was stopped at 40 epochs because further training did not improve performance.

5 EXPERIMENTAL RESULTS

The framework is evaluated against supervised, self-supervised, and semi-supervised baselines under limited-label, transfer, and class-aware settings. TS-TCC performs strongly across datasets, while CA-TCC improves average semi-supervised performance and benefits from pseudo-label-based supervised contrastive learning.

  • Comparison with Baseline Approaches: TS-TCC achieves the best overall performance across all 10 datasets with both 1% and 5% labeled data in linear evaluation.It ranks first on six datasets and second on two, with only a small margin behind the best methods on the latter.
  • Comparison with Baseline Approaches: Contrastive methods generally outperform the pretext-based SSL-ECG baseline, while CPC exceeds SimCLR across the reported dataset rankings.CPC is best on one dataset and second best on five datasets.
  • Few-Label Fine-Tuning: 69.5% and 89.2% MF1 are achieved by TS-TCC fine-tuning with 1% labels on HAR and Epilepsy, versus 47.6% and 74.8% for supervised training.With 10% labeled data, TS-TCC fine-tuning reaches performance comparable to supervised training using 100% labels across three datasets.
  • Semi-Supervised Evaluation: CA-TCC exceeds the second-best semi-supervised baseline by 3.8% and 4.9% average F1-score with 1% and 5% labels, respectively.CA-TCC ranks first on seven datasets and second on two others across ten datasets.
  • CA-TCC Model Analysis: TS-TCC-generated pseudo labels are more accurate than those from other baselines, and using TS-TCC in both training phases yields the best performance.The analysis attributes this to TS-TCC representations and the supervised contrastive loss used by CA-TCC.
  • CA-TCC Model Analysis: Supervised contrastive loss improves accuracy and F1-score on HAR and Sleep-EDF by including more positive pairs than unsupervised contrastive loss.Class-aware TS-TCC also consistently outperforms class-aware SimCLR on both datasets.
  • CA-TCC Model Analysis: The framework supports different models across training phases, although the experiments use TS-TCC because it performs best among the self-supervised baselines and supports class-aware training.The framework is therefore evaluated with a flexible multi-phase design but a fixed TS-TCC instantiation.

5.3 Transfer Learning Experiment

The transfer-learning experiments evaluate TS-TCC and CA-TCC across cross-domain scenarios, while sensitivity and augmentation studies examine key design choices. The results indicate that weak–strong augmentation combinations and selected hyperparameters support performance across datasets and domains.

  • 5.3 Transfer Learning Experiment: TS-TCC outperforms supervised pretraining in 8 of 12 cross-domain scenarios, while CA-TCC does so in 9 of 12 scenarios with only 1% source-domain labels.Across winning scenarios, TS-TCC achieves at least approximately 7% improvement in 7 of 8 cases, and CA-TCC achieves at least approximately 8% improvement in 7 of 9 cases.
  • 5.3 Transfer Learning Experiment: TS-TCC and CA-TCC improve transferability over supervised training by approximately 4% and 6% in accuracy, respectively.
  • 5.5 Sensitivity Analysis: Predicting 40% of the feature length performs best for temporal contrasting, while CA-TCC sensitivity analysis selects λ3 = 0.01 and λ4 = 0.7.The model is more sensitive to λ2 than λ1, and larger predicted-future percentages can reduce performance by leaving less past data for autoregressive training.
  • 5.6 Augmentation Selection: Augmentation parameters depend on dataset characteristics, motivating signal normalization and dataset-specific choices such as the number of chunks in strong augmentation.For longer sequences, the selected chunk count should be greater than for shorter sequences; 40% of total feature size performs best across the three datasets.

6 CONCLUSION

The paper presents TS-TCC for unlabeled time-series representation learning and CA-TCC for settings with few labels. Their learned representations support few-label evaluation and transfer-learning scenarios.

  • 6 CONCLUSION: TS-TCC learns time-series representations from unlabeled data using temporal and contextual contrasting with time-series-specific weak and strong augmentations.
  • 6 CONCLUSION: CA-TCC extends TS-TCC with pseudo-label-based class-aware supervised contrastive learning when few labeled samples are available.
  • 6 CONCLUSION: A linear classifier trained with few labels on TS-TCC representations achieves comparable performance to fully supervised training, while 10% labels approach full-label performance.
  • 6 CONCLUSION: Both variants improve representation transferability in real-world transfer-learning scenarios.

DATASETS DESCRIPTION

The evaluation uses diverse time-series datasets spanning industrial sensing, engine diagnostics, bone-age-related outlines, astronomy, and household electricity consumption. Several datasets contain binary, multiclass, noisy, or imbalanced classification settings.

  • Industrial sensing: The Wafer dataset contains sensor measurements from semiconductor fabrication with imbalanced normal and abnormal classes.Only 10.7% of the training set and 12.1% of the test set belong to the abnormal class.
  • Engine diagnostics: FordA and FordB contain 500 engine-noise measurements for symptom classification, with FordB collected under noisier conditions.
  • Bone outlines: ProximalPhalanxOutlineCorrect and PhalangesOutlinesCorrect derive classification problems from hand and bone outlines used in bone-age prediction.The latter concatenates three phalanx-outline classification problems.
  • Astronomy: The astronomical dataset represents starlight brightness curves across classes including Classical Type-I Cepheids, Eclipsing Binaries, RRab, and RRc RRLyrae.
  • Electricity consumption: The household electricity dataset records consumption behavior from 251 UK households.

Ensemble of Multiple Self-supervised Algorithms

The ensemble procedure combines three self-supervised algorithms to generate pseudo labels for semi-supervised training. Averaging their fine-tuned features improves stability, but the reported performance gain is minor and increases complexity.

  • Ensemble procedure: The ensemble pretrains each self-supervised algorithm, fine-tunes the encoders with few labels, and averages their features to generate pseudo labels.The process occurs across three phases before pseudo labels are used in the subsequent training phase.
  • Ensemble results: The ensemble of SimCLR, CPC, and TS-TCC produces more stable pseudo-label performance across runs on HAR and Sleep-EDF.The paper attributes this stability to averaging features from the three fine-tuned models.
  • Ensemble results: The ensemble shows only a minor performance improvement over individual self-supervised algorithms.
  • Ensemble trade-off: The ensemble increases model complexity and can lead to longer training times.
Loading 2208.06616v3…