Source-linked AI summary
TimesURL: Self-supervised Contrastive Learning for Universal Time Series Representation Learning
Jiexi Liu, Songcan Chen
TL;DR
Universal time-series representation learning is difficult because temporal data require task-aware handling of augmentations, negatives, and semantic levels. TimesURL combines frequency-temporal augmentation, double Universums, and joint reconstruction with contrastive learning, achieving state-of-the-art performance across six downstream tasks.
Problem
Time-series contrastive learning can introduce inappropriate temporal assumptions, weak easy negatives, and insufficient segment- or instance-level information for universal representations.
Method
TimesURL uses FTAug, double Universums as hard negatives, and joint time reconstruction with contrastive learning to capture temporal and multi-level information.
Results
TimesURL achieves state-of-the-art performance across six benchmark downstream tasks, demonstrating universal and high-quality time-series representations.
Takeaways & Limitations
The framework provides one representation-learning approach evaluated across forecasting, imputation, classification, anomaly detection, and transfer learning.
Abstract
from arXiv · showhide
Learning universal time series representations applicable to various types of downstream tasks is challenging but valuable in real applications. Recently, researchers have attempted to leverage the success of self-supervised contrastive learning (SSCL) in Computer Vision(CV) and Natural Language Processing(NLP) to tackle time series representation. Nevertheless, due to the special temporal characteristics, relying solely on empirical guidance from other domains may be ineffective for time series and difficult to adapt to multiple downstream tasks. To this end, we review three parts involved in SSCL including 1) designing augmentation methods for positive pairs, 2) constructing (hard) negative pairs, and 3) designing SSCL loss. For 1) and 2), we find that unsuitable positive and negative pair construction may introduce inappropriate inductive biases, which neither preserve temporal properties nor provide sufficient discriminative features. For 3), just exploring segment- or instance-level semantics information is not enough for learning universal representation. To remedy the above issues, we propose a novel self-supervised framework named TimesURL. Specifically, we first introduce a frequency-temporal-based augmentation to keep the temporal property unchanged. And then, we construct double Universums as a special kind of hard negative to guide better contrastive learning. Additionally, we introduce time reconstruction as a joint optimization objective with contrastive learning to capture both segment-level and instance-level information. As a result, TimesURL can learn high-quality universal representations and achieve state-of-the-art performance in 6 different downstream tasks, including short- and long-term forecasting, imputation, classification, anomaly detection and transfer learning.
Introduction
TimesURL addresses the difficulty of learning universal time-series representations by adapting contrastive learning to temporal structure and multiple semantic levels. It combines tailored augmentation, hard negatives, and reconstruction, achieving state-of-the-art results across downstream tasks.
- Time-series contrastive learning requires tailored solutions because high dimensionality, temporal characteristics, and diverse task semantics complicate direct transfer from CV and NLP.
- Existing augmentations can impose inappropriate assumptions that distort temporal variations such as trends and peaks.
- Easy negative samples dominate many time-series settings, providing weak gradients and insufficient discriminative information.
- Forecasting, anomaly detection, and imputation need segment-level information, whereas classification and clustering prioritize instance-level information.
- TimesURL combines FTAug, double Universums, and joint time reconstruction with contrastive learning to capture temporal structure and both semantic levels.
- TimesURL consistently achieves state-of-the-art results across a broad range of downstream tasks, demonstrating universal, high-quality representations.
Related Work
Prior time-series representation methods use reconstruction, clustering, random convolutions, and contrastive objectives. TimesURL responds to limitations involving augmentation assumptions, easy negatives, and incomplete semantic coverage.
- Earlier unsupervised methods learn representations through similarity preservation, reconstruction, clustering, or random convolutional features.
- Time-series contrastive methods design pretext tasks, augmentations, contextual representations, or frequency-aware objectives for unannotated data.
- Prior methods remain vulnerable to improper assumptions, excessive easy negatives, and insufficient use of both segment- and instance-level information.
- TimesURL unifies frequency-temporal augmentation, hard-negative construction, and multi-level information modeling to address these limitations.
Proposed TimesURL Framework
TimesURL combines frequency-temporal augmentation, double Universums, hierarchical contrastive learning, and masked time reconstruction to learn representations preserving temporal and instance-level information.
- Framework overview: TimesURL generates augmentation sets for original and masked series, using the former for contrastive learning and the latter for time reconstruction.The encoder maps these sets to representations, encouraging transformation consistency while reconstructing the original data.
- Frequency-temporal augmentation: Frequency mixing exchanges frequency components between training instances, while random cropping forms overlapping views whose shared segment is optimized.FTAug is applied only during training and is intended to preserve temporal relationships and semantic consistency.
- Double Universum learning: Double Universums mix anchor representations with negative features at temporal and instance levels to create additional hard negatives.The mixing coefficients are at most 0.5, so negative samples contribute more than the anchor in each synthetic Universum.
- Double Universum learning: Universums are closer to anchors than ordinary negatives, making the proxy task harder while improving ERing linear classification from 0.896 to 0.985.This supports treating the synthetic Universums as high-quality hard negatives for contrastive learning.
- Contrastive learning: Temporal- and instance-wise contrastive losses use original negatives and double Universums, with hierarchical pooling supplying multi-scale representations.The two losses are complementary: they capture temporal variation and instance-specific characteristics, while pooling supports multi-scale learning.
- Time reconstruction: Masked reconstruction uses the encoder to recover the full series from partial observations and computes MSE on masked timestamps.The reconstruction objective is jointly combined with contrastive learning to preserve temporal variation.
Experiments
TimesURL is evaluated across six downstream tasks using extensive comparisons, including task-specific state-of-the-art baselines and representation-learning methods. Results include strong classification, imputation, forecasting, anomaly-detection, transfer-learning, and ablation performance.
- Experimental scope: Six downstream tasks evaluate TimesURL's representation generality: short- and long-term forecasting, imputation, classification, anomaly detection, and transfer learning.The experiments are designed to assess generality and downstream-task performance.
- Baselines: TimesURL is compared with recent self-supervised methods and task-specific state-of-the-art models, with about 15 baselines included overall.Existing self-supervised methods are used only for tasks they target, alongside specialized baselines.
- Classification: 75.2% average accuracy on 30 UEA datasets and 84.5% on 128 UCR datasets make TimesURL the best-performing classification method in the reported comparison.It surpasses InfoTS, whose reported average accuracy is 71.4%.
- Imputation: TimesURL achieves state-of-the-art performance on three imputation datasets under missing-point ratios of 12.5%, 25%, 37.5%, and 50%.The reconstruction module is extended to imputation and is reported to capture temporal variation from complicated time series.
- Short- and Long-Term Forecasting: TimesURL establishes new state-of-the-art results in most short- and long-term forecasting cases while reusing one learned representation across multiple horizons.Forecasting horizons range from 24 to 48 for short-term and 96 to 720 for long-term evaluation.
- Anomaly Detection and Transfer Learning: TimesURL performs consistently well on KPI and Yahoo anomaly detection benchmarks and reaches average transfer results of 0.864 from CBF and 0.895 from CinCECGTorso.The reported no-transfer average is 0.912, while ablations evaluate frequency mixing, temporal and instance Universums, and reconstruction.
Conclusion
TimesURL is a self-supervised framework for learning universal time series representations across diverse downstream tasks. It combines temporal-consistent augmentation, double Universums for stronger negatives, and joint contrastive learning with time reconstruction.
- TimesURL targets universal time series representations applicable to various downstream tasks.
- FTAug preserves contextual consistency and temporal characteristics during augmentation.
- Double Universums increase the quantity and quality of hard negatives for contrastive learning.
- Joint optimization of contrastive learning and time reconstruction captures both segment-level and instance-level information.
- TimesURL performs strongly across six downstream tasks when augmentation, hard negatives, and information levels are appropriately designed.