Source-linked AI summary
Time-Series Representation Learning via Temporal and Contextual Contrasting
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, Cuntai Guan
TL;DR
Limited labeling and temporal dynamics make effective representation learning from unlabeled time-series data challenging. TS-TCC uses strong and weak augmentations, temporal cross-view prediction, and contextual contrasting to learn robust and discriminative representations. Across three datasets, its learned features perform comparably with supervised training and remain effective with few labels and in transfer learning.
Problem
Time-series data are difficult to label because they lack human-recognizable patterns and require specialist annotation, leaving limited labeled data for deep learning.
Method
TS-TCC creates strongly and weakly augmented views, applies temporal cross-view prediction, and adds contextual contrasting to learn robust and discriminative representations.
Results
TS-TCC features support linear evaluation comparable with supervised training, while experiments report effectiveness in few-labeled and transfer learning scenarios.
Takeaways & Limitations
TS-TCC provides an unsupervised time-series representation framework whose learned features remain effective for downstream learning with limited labels and across domains.
Abstract
from arXiv · showhide
Learning decent representations from unlabeled time-series data with temporal dynamics is a very challenging task. In this paper, we propose an unsupervised Time-Series representation learning framework via Temporal and Contextual Contrasting (TS-TCC), to learn time-series representation from unlabeled data. First, the raw time-series data are transformed into two different yet correlated views by using weak and strong augmentations. Second, we propose a novel temporal contrasting module to learn robust temporal representations by designing a tough cross-view prediction task. Last, to further learn discriminative representations, we propose a contextual contrasting module built upon the contexts from the temporal contrasting module. It attempts to maximize the similarity among different contexts of the same sample while minimizing similarity among contexts of different samples. Experiments have been carried out on three real-world time-series datasets. The results manifest that training a linear classifier on top of the features learned by our proposed TS-TCC performs comparably with the supervised training. Additionally, our proposed TS-TCC shows high efficiency in few-labeled data and transfer learning scenarios. The code is publicly available at https://github.com/emadeldeen24/TS-TCC.
1 Introduction
Time-series labeling is scarce and difficult, motivating unsupervised representation learning. TS-TCC addresses this challenge with augmented views, temporal cross-view prediction, and contextual contrasting, and is evaluated across downstream learning settings.
- Time-series data are difficult to label because they lack human-recognizable patterns and require specialists, limiting labeled data for deep learning.
- TS-TCC creates two correlated views using strong and weak augmentations for unsupervised time-series representation learning.
- Its temporal contrasting module learns robust representations through a tough cross-view prediction task using past features to predict another view’s future.
- A contextual contrasting module further learns discriminative representations by increasing similarity among contexts from the same sample and reducing similarity across different samples.
- Experiments on three datasets evaluate TS-TCC for supervised, semi-supervised, and transfer learning settings, reporting effective downstream representations.
2 Related Works
Prior time-series representation learning used heuristic pretext tasks, while contrastive approaches developed invariant representations through transformed views. The cited related work illustrates application-specific strategies rather than a single general approach.
- Self-supervised time-series methods have used pretext tasks to learn representations from transformed data.
- One human-activity-recognition approach trains a binary classifier to distinguish original time series from transformed versions.
- SSL-ECG assigns pseudo-labels according to transformation type while learning ECG representations from six transformations.
3 Methods
TS-TCC creates weak and strong views, extracts temporal features through cross-view future prediction, and makes representations discriminative by contrasting contexts across augmented samples.
- 3.1 Time-Series Data Augmentation: TS-TCC generates two correlated views of each input using separate weak and strong augmentations.Weak augmentation uses jitter-and-scale, while strong augmentation uses permutation-and-jitter.
- 3.1 Time-Series Data Augmentation: A convolutional encoder maps both augmented views into high-dimensional latent representations organized across timesteps.The resulting strong and weak latent sequences are passed to the temporal contrasting module.
- 3.2 Temporal Contrasting: The temporal contrasting module uses autoregressive contexts to predict future timesteps of the opposite augmentation through a cross-view prediction task.Contexts summarize features up to timestep t, and the log-bilinear predictor targets future steps from t+1 through t+K.
- 3.2 Temporal Contrasting: A Transformer autoregressive model summarizes past latent features into a representative context token that is forwarded to contextual contrasting.The Transformer uses multi-headed attention, MLP blocks, residual connections, and an input context token c.
- 3.3 Contextual Contrasting: Contextual contrasting applies a projection head and contrasts 2N contexts, bringing same-sample augmented contexts together while separating contexts from other samples.Each context has one positive counterpart from the other view and 2N−2 negative contexts within the minibatch.
- 3.3 Contextual Contrasting: The overall self-supervised objective combines the two temporal contrasting losses with the contextual contrasting loss using fixed relative weights.The contextual loss uses cosine similarity, an indicator for nonmatching samples, and a temperature parameter.
4 Experimental Setup
The experiments use three public time-series datasets spanning human activity, sleep staging, and seizure prediction, plus a fault-diagnosis transfer dataset. Evaluation uses linear, semi-supervised, and transfer-learning settings with accuracy and macro-F1, under repeated train/validation/test splits.
- Datasets: Three public datasets cover human activity recognition, sleep stage classification, and epileptic seizure prediction.A fault-diagnosis dataset is additionally used to evaluate transferability across working conditions.
- Datasets: The UCI HAR dataset contains sensor readings from 30 subjects performing six activities at 50 Hz.The recordings were collected with a Samsung Galaxy S2 mounted on the waist.
- Datasets: Sleep-EDF provides whole-night PSG recordings using one EEG channel sampled at 100 Hz for five sleep classes.The classes are Wake, N1, N2, N3, and REM.
- Datasets: The seizure dataset contains 500 subjects and is converted from five original labels into binary seizure classification.Each recording covers 23.6 seconds of brain activity.
- Datasets: The fault-diagnosis dataset contains four working conditions treated as separate domains, each with two fault classes and one healthy class.Transfer learning trains on one condition and evaluates across another condition.
5 Results
TS-TCC is evaluated against supervised and self-supervised baselines across linear evaluation, few-label training, transfer learning, and ablation settings. Results show strong representation quality, especially with limited labels and cross-domain transfer, while both contrasting modules contribute to performance.
- Comparison with Baselines: TS-TCC outperforms the three state-of-the-art baselines in linear evaluation and performs best on two of three datasets.With only a linear classifier, it achieves performance comparable to supervised training on the remaining dataset.
- Semi-supervised Training: With only 1% labeled data, TS-TCC fine-tuning significantly outperforms supervised training, reaching around 70% on HAR and 90% on Epilepsy.The figure compares macro-F1 across few-labeled data scenarios.
- Semi-supervised Training: With 10% labeled data, TS-TCC fine-tuning achieves performance comparable to supervised training using 100% labeled data across all three datasets.This result is reported for the semi-supervised setting.
- Transfer Learning: TS-TCC fine-tuning outperforms supervised pretraining in 8 of 12 cross-domain scenarios and improves accuracy by about 4% overall.At least 7% improvement occurs in 7 of the 8 winning scenarios, except D→B.
- Ablation Study: The cross-view prediction task improves performance by more than 5% on HAR and about 1% on Sleep-EDF and Epilepsy.Adding contextual contrasting further improves discriminative representations, while both augmentation types together achieve the best overall performance.
- Sensitivity Analysis: On HAR, predicting 40% of the feature length gives the best performance in sensitivity analysis.Performance declines at larger percentages because less past data remains for training the autoregressive model; λ2 performs best at 0.7 and λ1 at 1.
6 Conclusions
TS-TCC learns unsupervised time-series representations through temporal and contextual contrasting, with performance comparable to supervised training and strong efficiency when labels are scarce.
- TS-TCC combines strong and weak augmentations with temporal and contextual contrasting to learn robust, discriminative time-series features.The temporal module uses cross-view prediction, while the contextual module contrasts contexts from the learned representations.
- TS-TCC performs comparably with supervised training when a linear classifier is trained on its learned features.
- 10% of labeled data enables TS-TCC to achieve close performance to supervised training with full labeled data.