PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency

Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, Marinka Zitnik

arXiv:2206.08496v3cs.LGcs.AI

TL;DR

Time-series pre-training is difficult because distribution shifts and varied temporal dynamics limit knowledge transfer, while target datasets are unavailable during pre-training. TF-C addresses this by aligning time- and frequency-based representations through self-supervised contrastive learning, achieving a 15.4% average F1-score improvement over baselines.

  • Problem

    Distribution shifts and varied temporal dynamics limit time-series knowledge transfer, while target datasets are unavailable during pre-training.

  • Method

    TF-C injects time-frequency consistency into pre-training by bringing time-based and frequency-based representations and their local neighborhoods close in latent space using contrastive estimation.

  • Results

    15.4% average F1-score improvement: TF-C outperformed eight state-of-the-art baselines across eight time-series datasets.

  • Takeaways & Limitations

    TF-C performs well across downstream classification, clustering, and anomaly-detection tasks.

  • Takeaways & Limitations

    Pre-training data may contain demographic imbalances that introduce bias into predictions.

Abstract

from arXiv · show

Pre-training on time series poses a unique challenge due to the potential mismatch between pre-training and target domains, such as shifts in temporal dynamics, fast-evolving trends, and long-range and short-cyclic effects, which can lead to poor downstream performance. While domain adaptation methods can mitigate these shifts, most methods need examples directly from the target domain, making them suboptimal for pre-training. To address this challenge, methods need to accommodate target domains with different temporal dynamics and be capable of doing so without seeing any target examples during pre-training. Relative to other modalities, in time series, we expect that time-based and frequency-based representations of the same example are located close together in the time-frequency space. To this end, we posit that time-frequency consistency (TF-C) -- embedding a time-based neighborhood of an example close to its frequency-based neighborhood -- is desirable for pre-training. Motivated by TF-C, we define a decomposable pre-training model, where the self-supervised signal is provided by the distance between time and frequency components, each individually trained by contrastive estimation. We evaluate the new method on eight datasets, including electrodiagnostic testing, human activity recognition, mechanical fault detection, and physical status monitoring. Experiments against eight state-of-the-art methods show that TF-C outperforms baselines by 15.4% (F1 score) on average in one-to-one settings (e.g., fine-tuning an EEG-pretrained model on EMG data) and by 8.4% (precision) in challenging one-to-many settings (e.g., fine-tuning an EEG-pretrained model for either hand-gesture recognition or mechanical fault prediction), reflecting the breadth of scenarios that arise in real-world applications. Code and datasets: https://github.com/mims-harvard/TFC-pretraining.

1 Introduction

Time-series pre-training aims to transfer representations across diverse datasets, but distribution shifts and temporal complexity make generalization difficult. TF-C addresses this challenge by aligning time- and frequency-based representations, and outperforms state-of-the-art baselines across broad evaluation settings.

  • Time-series representation learning is valuable for clinical diagnosis, traffic analysis, climate science, and pre-training, but learning generalizable representations remains challenging.
  • TF-C brings time-based and frequency-based representations, along with their local neighborhoods, close together in latent time-frequency space.
  • Distribution shifts, differing temporal dynamics, semantic meanings, sampling patterns, devices, and subjects can limit knowledge transfer across time-series datasets.
  • Target datasets are unavailable during pre-training, so a useful method must capture properties shared by previously unseen datasets without relying on extensive annotations.
  • 15.4% average F1-score improvement over eight state-of-the-art baselines demonstrates positive transfer across eight time-series datasets.The evaluation includes one-to-one and one-to-many transfer settings spanning neurological healthcare, activity recognition, mechanical fault detection, and physical status monitoring.

2 Related Work

Self-supervised pre-training for time series remains underexplored because generalizable assumptions connecting pre-training and target datasets are unclear. TF-C combines contrastive learning with a time-frequency property and avoids target labels during pre-training.

  • Self-supervised pre-training for time series is underexplored, and assumptions that bridge pre-training and target datasets are not obvious.
  • Unlike domain adaptation, time-series pre-training lacks access to fine-tuning datasets during training and therefore requires a generalizable time-series property.
  • TF-C requires no labels during pre-training, whereas self-supervised domain adaptation still requires labels for model training.
  • Contrastive learning pulls augmented views of the same sample together and pushes views of different samples apart, but suitable time-series augmentations remain difficult to identify.
  • Existing time-series contrastive methods exploit temporal, transformation, contextual, or augmentation-based invariances to construct positive pairs and contrastive objectives.

3 Problem Formulation

The paper formulates pre-training with unlabeled source data and a small labeled target dataset that is unavailable during pre-training. TF-C defines a shared latent property by aligning time- and frequency-based representations and their local augmentations.

  • The formulation focuses on univariate series while noting support for multivariate series with varying lengths across datasets.
  • The pre-training problem uses an unlabeled dataset Dpret with N samples to initialize a model that is fine-tuned on a target dataset Dtune with M ≪ N samples.
  • Only unlabeled Dpret is available during pre-training; labeled Dtune is introduced later for fine-tuning, distinguishing the setup from domain adaptation.
  • Time-domain representations describe how sensor readouts change over time, while frequency-domain representations describe signal content across the spectrum.
  • Representational TF-C requires time-based and frequency-based representations of a sample and its local augmentations to be proximal in latent time-frequency space.
  • The method induces TF-C on source data so optimized parameters can initialize target models and bridge datasets with different temporal dynamics and semantic meanings.
  • TF-C uses a time encoder, frequency encoder, and two cross-space projectors that map both representation types into the same latent space.

4 Our Approach

The approach learns time- and frequency-based representations separately with contrastive objectives, then aligns them in a joint time-frequency space through a consistency loss. The pretrained model combines these components for downstream fine-tuning.

  • Model architecture: The model uses separate time and frequency encoders to produce representations from temporal signals and transformed frequency spectra.The time pathway uses temporal augmentations, while the frequency pathway operates on spectra generated by a transform such as the Fourier transform.
  • Time-based contrastive encoder: Temporal augmentations include jittering, scaling, time-shifts, and neighborhood segments to expose the encoder to varied temporal dynamics.The augmentation bank is designed to produce diverse augmentations and more robust time-based embeddings.
  • Time-based contrastive encoder: The time-based contrastive loss pulls embeddings of positive augmented pairs together and pushes embeddings of negative samples apart using NT-Xent.Cosine similarity measures pairwise similarity, while τ adjusts the loss scale.
  • Frequency-based contrastive encoder: Frequency augmentations add or remove frequency components, using a perturbation budget E and setting α = 0.5 for added components in this work.The perturbations are constrained so the resulting time series remains similar to the original in both frequency and time domains.
  • Frequency-based contrastive encoder: The frequency-based contrastive loss trains embeddings to be invariant to perturbations of the frequency spectrum.Preliminary experiments report little performance effect from τ, which is held constant across experiments.
  • Time-frequency consistency: A consistency loss aligns time- and frequency-based embeddings by encouraging within-sample cross-domain distances to be smaller than margin-separated alternatives.The loss uses triplet-style relative distances in a joint time-frequency space and includes augmented embeddings in each pair.
  • Overall objective: The total pre-training objective combines time contrastive loss, frequency contrastive loss, and consistency loss as LTF-C,i = λ(LT,i + LF,i) + (1 −λ)LC,i.λ controls the relative importance of the contrastive and consistency terms, and the total loss sums across pre-training samples.
  • Fine-tuning: After pre-training, the entire model is stored and fine-tuned samples are represented through the model’s concatenated embedding.The model includes the time and frequency encoders together with their respective projectors.

5 Experiments

Experiments evaluate TF-C across diverse one-to-one and one-to-many transfer settings, additional downstream tasks, and ablations. TF-C generally performs strongly across these scenarios and supports the proposed time-frequency consistency property.

  • Evaluation design: TF-C is compared with 10 baselines across 8 diverse datasets and evaluated in one-to-one, one-to-many, clustering, and anomaly-detection settings.The datasets vary in channels, lengths, sampling rates, scenarios, and signal types.
  • One-to-one evaluation: One-to-one experiments transfer models across substantially different channels, physiology, patients, activities, or mechanical operating conditions.Examples include SLEEPEEG → EPILEPSY, FD-A → FD-B, and HAR → GESTURE.
  • One-to-many evaluation: TF-C earned the best performance in 14 out of 18 settings across three challenging one-to-many scenarios.The model showed higher tolerance to transfer across datasets with large gaps, including EEG, vibration, acceleration, and EMG signals.
  • Ablation: TF-C’s consistency loss outperformed within-time consistency by 5.3% and within-frequency consistency by 7.2% in accuracy.The ablation indicates that the performance gain is not solely attributable to adding a third loss term.
  • Additional downstream tasks: TF-C achieved the best clustering performance, surpassing TS-TCC by 5.4% in Silhouette score.The clustering evaluation used SLEEPEEG → EPILEPSY with K-means and reported Silhouette score, ARI, and NMI.
  • Additional downstream tasks: TF-C outperformed five competitive baselines by 4.5% in F-1 Score for sample-level anomaly detection.The FD-A → FD-B task used one-class SVM on a subset containing undamaged and damaged bearings.

6 Conclusion

The paper introduces TF-C pre-training to support knowledge transfer across time-series datasets by aligning time- and frequency-based representations. It reports broad downstream performance while identifying scope boundaries and possible extensions.

  • Conclusion: TF-C uses self-supervised contrastive estimation to bring time-based and frequency-based representations and their local neighborhoods close in a shared latent space.The approach is designed to support knowledge transfer between time-series datasets.
  • Limitations and future directions: The method currently expects regularly sampled time series, although irregular sampling can be handled with suitable encoders or preprocessing alternatives.The paper mentions encoders such as Raindrop and SeFT, along with resampling, interpolation, and regular or non-uniform FFT operations.
  • Conclusion: TF-C performs well across classification, clustering, and anomaly-detection tasks.The current embedding strategy and loss functions are favorable for classification because they leverage global information.
  • Limitations and future directions: TF-C’s current strategy favors classification over tasks using local context, such as forecasting.The paper identifies additional generalizable properties, including temporal autoregressive processes, as future directions.

Broader Impacts

The paper presents TF-C as a self-supervised pre-training approach intended to improve classification across diverse time-series application scenarios. It also highlights potential societal benefits and warns that demographic and protocol biases may transfer during fine-tuning.

  • TF-C improves classification performance on target datasets across different application scenarios.
  • Transfer learning with time-series data may support more efficient use of data in settings such as clinical diagnosis.The paper gives small labelled medical datasets as a motivating example.
  • Bias in source pre-training data may transfer to fine-tuning and produce biased predictions.Sources include demographic sample imbalance and standardized protocols unsuitable for some physiological attributes.
  • The datasets used are publicly available and are reported as having no privacy or security concerns.

Checklist

The checklist states that the paper addresses its main claims, limitations, responsible-use considerations, reproducibility materials, and reporting details. It also documents asset citations, licensing, and the public nature of the datasets.

  • The paper reports that its main claims are accurately reflected and that its limitations are described in Section 6.
  • The paper reports no theoretical results and no crowdsourcing or human-subject study requiring participant-risk or compensation documentation.
  • Code, implementation details, baseline implementations, and eight datasets are provided through an anonymous link.
  • Experiments were run five times, with average values and standard deviations reported.The checklist points to Tables 1, 4–6, and 10.
  • The authors cite the creators of the eight datasets and six state-of-the-art baselines and report dataset licenses in Appendix D.
  • The datasets are freely downloadable public resources, so re-contacting data curators was not required.

Appendix A Further information on the relationship between our pre-training approach and domain adaptation

The appendix distinguishes self-supervised pre-training from domain adaptation and motivates TF-C through consistency between time- and frequency-domain representations. TF-C directly perturbs spectra and uses their consistency to support transfer across datasets.

  • Self-supervised pre-training uses only the pre-training dataset, whereas domain adaptation requires access to a target dataset during training.
  • Self-supervised pre-training in this paper uses no labels during pre-training, unlike self-supervised domain adaptation as defined in the appendix.
  • Unlike BTSF, TF-C directly perturbs the frequency spectrum rather than applying only time-domain transformations before the FFT.
  • TF-C uses time-frequency consistency to produce generalizable representations, whereas BTSF fuses temporal and spectral features for discriminative embeddings.
  • TF-C maps samples into a time-frequency embedding space and constrains relative embedding relationships using triplet loss.
  • TF-C is presented as the first time-series method using frequency-domain augmentations to enable transfer learning.
  • TF-C assumes that time- and frequency-domain representations of the same or similar sample carry the same information and should have consistent features.

D.1 Datasets

The evaluation uses diverse public time-series datasets spanning healthcare, activity recognition, fault detection, and monitoring, with varied channels, lengths, sampling rates, and signal types. Experiments include small balanced fine-tuning sets and cross-dataset settings.

  • D.1 Datasets: The datasets cover neurological healthcare, human activity recognition, mechanical fault detection, and physical status monitoring using EEG, EMG, ECG, acceleration, and vibration.
  • D.1 Datasets: SLEEPEEG contains univariate EEG segments labelled with five sleep stages.
  • D.1 Datasets: EPILEPSY contains 11,500 one-second EEG samples and uses 60 samples for fine-tuning, 20 for validation, and 11,420 for testing.
  • D.1 Datasets: FD-A and FD-B represent rolling-bearing conditions with three classes: undamaged, inner damaged, and outer damaged.
  • D.1 Datasets: HAR uses nine smartphone-sensor channels to classify six daily activities performed by 30 healthy volunteers.
  • D.1 Datasets: GESTURE classifies eight hand gestures from accelerometer measurements, with 440 samples recoverable through the UCR database.
  • D.1 Datasets: ECG is a single-lead, 300 Hz dataset classifying four cardiac-arrhythmia conditions and is imbalanced toward fewer atrial-fibrillation and noisy samples.

Appendix E TF-C and baseline architectures and implementation details

TF-C is compared with eight state-of-the-art time-series methods and additional non-transfer baselines using implementations aligned with the corresponding papers. Its architecture combines time-based and frequency-based contrastive encoders with a consistency module.

  • TF-C is compared against eight state-of-the-art baselines, plus KNN and a randomly initialized model trained from scratch.
  • The baseline implementations follow their original papers and generally use reported default hyperparameters.
  • TF-C architecture: TF-C uses a 1-D ResNet-based time encoder and a cross-space projector within its time-frequency architecture.
  • Baseline architectures: TS-SD encodes latent features with modified attention and uses denoising or DTW-similarity prediction as pretext tasks.
  • Baseline architectures: TS2vec uses contextual consistency and a hierarchical loss to capture long-range time-series structure.
  • Baseline architectures: CLOCS treats recordings from different subjects as negative pairs for contrastive learning of ECG representations.
  • Baseline architectures: Mixing-up uses convex combinations of two time series and a contrastive loss to predict their mixing proportion.
  • Baseline architectures: TS-TCC combines strong and weak augmentations with contextual information from a transformer-based autoregressive model.

Appendix F Additional results on one-to-one evaluation

Additional one-to-one experiments evaluate transfer across paired datasets, clustering, anomaly detection, and component ablations. The ablation results identify the full TF-C model, especially its consistency loss, as strongest across reported metrics.

  • Evaluation settings: One-to-one evaluation covers four paired pre-training and fine-tuning scenarios spanning identical or similar signal semantics.
  • Evaluation settings: Full fine-tuning optimizes all transferred parameters and performs 5.4% better than partial fine-tuning in F1 score.
  • Additional downstream tasks: Downstream clustering compares TF-C with five baselines, including Random Init., Non-DL, TS-TCC, TNC, and CPC.
  • Additional downstream tasks: Anomaly detection uses an FD-A-pretrained model fine-tuned on imbalanced FD-B data containing 90% normal and 10% abnormal samples.
  • Ablation study: TF-C achieves the highest performance on every reported evaluation metric in the SLEEPEEG-to-GESTURE ablation study.
  • Ablation study: The ablation results indicate that every TF-C component contributes to performance, especially the consistency loss LC.

Appendix I Visualization of embeddings in time-frequency space

Embedding visualizations and cosine-distance analyses examine whether TF-C brings time-based and frequency-based representations together while separating samples from different classes. Across these analyses, the consistency loss is associated with closer cross-space embeddings and greater inter-class distance.

  • Visualization procedure: The embeddings have 128 dimensions and are mapped to two dimensions with UMAP using cosine distance.
  • Figure 3 visualization: Without LC, time- and frequency-based embeddings cluster separately; with LC, they move closer in the time-frequency space.
  • Figure 3 visualization: For one annotated sample, LC reduces cosine distance between time- and frequency-based embeddings from 0.89 to 0.71.
  • Figure 3 visualization: Frequency-based epilepsy and health embeddings overlap without LC but cluster more distinctly when LC is included.
  • Figure 3 visualization: Figure 3 compares EPILEPSY embeddings after SLEEPEEG pre-training with and without the consistency loss LC.Circles denote epilepsy samples, triangles denote healthy samples, and the same annotated sample appears in both panels.
  • Figure 4 analysis: Figure 4 defines lower cosine distance between time- and frequency-based embeddings as higher consistency across one-to-many transfer datasets.
  • Figure 5 analysis: Figure 5 interprets higher inter-class cosine distance as better discrimination and reports an EPILEPSY increase from 0.88 to 1.42, a 61.4% margin.
  • Figure 4 analysis: With LC, averaged EPILEPSY cosine distance decreases 22.2%, from 1.32 to 1.08.

Appendix J Additional information on domain augmentations

TF-C explores frequency-domain augmentations and examines transfer when pre-training data become heterogeneous. The results indicate that augmentation choices depend on signal context, while increasing pre-training heterogeneity reduces one-to-one transfer performance.

  • Frequency augmentations: TF-C claims the first direct perturbation of the frequency domain for contrastive learning of time series.
  • Frequency augmentations: The study evaluates eight frequency-augmentation policies across frequency bands, perturbation budgets, and selection distributions.
  • Frequency augmentations: High-frequency components correspond to fast temporal variation, whereas low-frequency components correspond to slow signal variation.
  • Frequency augmentations: Figure 6 visualizes frequency perturbations and their time-domain effects through the inverse FFT on EPILEPSY data.
  • Frequency augmentations: Multi-component perturbations use budget 5 and single-component perturbations use budget 1, with larger budgets producing larger temporal changes.
  • Frequency-augmentation results: Low-band perturbations outperform high-band perturbations by 0.8% in average F1 across four setups, with the preferred band depending on signal context.
  • Many-to-one setting: Many-to-one pre-training merges multiple datasets before fine-tuning on one pure dataset, creating greater heterogeneity than one-to-one or one-to-many settings.
  • Many-to-one results: AUROC decreases from 0.9819 in one-to-one transfer to 0.7253 in four-to-one transfer as pre-training heterogeneity increases.
Loading 2206.08496v3…