PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer

Wenrui Zhang, Ling Yang, Shijia Geng, Shenda Hong

arXiv:2205.09928v2cs.LG

TL;DR

Time-series labels are often scarce, and existing approaches leave temporal–spectral relations underused while facing sampling instability and masking-induced distribution shifts. CRT combines time- and frequency-domain modeling with dropping-based reconstruction and a progressive dropping curriculum. It is reported to achieve the best effectiveness and robustness across three datasets, with gains of 2%∼9%.

  • Problem

    Existing self-supervised methods neglect spectral information and temporal–spectral correlations, while pair sampling can be unstable and zero-value masking can shift distributions.

    Source: §I · p. 2

  • Method

    CRT uses a Transformer to fuse temporal and spectral features, randomly drops data in both domains for reconstruction, and progressively raises the dropping ratio during pre-training.

    Source: §I · p. 2

  • Results

    CRT achieves the best reported effectiveness and robustness across three datasets, with performance gains of 2%∼9%.

    Source: §I · p. 2

  • Takeaways & Limitations

    CRT provides a unified cross-domain reconstruction framework for learning time-series representations.

    Source: §V · p. 9

Abstract

from arXiv · show

Unsupervised/self-supervised representation learning in time series is critical since labeled samples are usually scarce in real-world scenarios. Existing approaches mainly leverage the contrastive learning framework, which automatically learns to understand the similar and dissimilar data pairs. Nevertheless, they are restricted to the prior knowledge of constructing pairs, cumbersome sampling policy, and unstable performances when encountering sampling bias. Also, few works have focused on effectively modeling across temporal-spectral relations to extend the capacity of representations. In this paper, we aim at learning representations for time series from a new perspective and propose Cross Reconstruction Transformer (CRT) to solve the aforementioned problems in a unified way. CRT achieves time series representation learning through a cross-domain dropping-reconstruction task. Specifically, we transform time series into the frequency domain and randomly drop certain parts in both time and frequency domains. Dropping can maximally preserve the global context compared to cropping and masking. Then a transformer architecture is utilized to adequately capture the cross-domain correlations between temporal and spectral information through reconstructing data in both domains, which is called Dropped Temporal-Spectral Modeling. To discriminate the representations in global latent space, we propose Instance Discrimination Constraint to reduce the mutual information between different time series and sharpen the decision boundaries. Additionally, we propose a specified curriculum learning strategy to optimize the CRT, which progressively increases the dropping ratio in the training process.

I. INTRODUCTION

Time-series labels are often difficult to acquire, while existing self-supervised methods underuse temporal–spectral relations and face sampling or masking drawbacks. CRT combines temporal and spectral modeling with dropping-based reconstruction, and is reported to outperform baselines on three datasets.

  • Time-series representations matter across real-world applications, but labeled time-series samples are usually difficult to acquire.Source: §I · p. 1
  • Existing methods neglect spectral information and temporal–spectral correlations, while contrastive sampling can cause bias and instability and zero-value masking can shift distributions.Source: §I · pp. 1–2
  • CRT uses a Transformer encoder on time- and frequency-domain data to fuse cross-domain features, adding spectral phase alongside magnitude.Source: §I · p. 2
  • Its curriculum progressively increases the dropping ratio during pre-training to improve training robustness.Source: §I · p. 2
  • CRT randomly drops parts in both domains rather than replacing masked values with zeros, preserving the time series’ original distribution.Source: §I · p. 2
  • CRT achieves the best reported effectiveness and robustness across three datasets, with performance gains of 2%∼9%.Source: §I · p. 2

A. Self-supervised learning

Self-supervised learning uses pretext tasks on unlabeled data to learn representations, and time-series methods have largely relied on contrastive sampling. CRT combines temporal and spectral information through cross-domain reconstruction and dropping.

A. Self-supervised learning

  • Self-supervised learning designs pretext tasks on unlabeled data to provide supervision and learn representations expected to benefit downstream tasks.The paper notes that this approach has been especially prominent in image and text research.Source: A. Self-supervised learning · p. 2

B. Self-supervised learning for time series

  • Time-series contrastive methods construct positive and negative pairs from temporal fragments, but time-based methods can miss long-term dependencies and perform poorly on downstream tasks with periodic series.A Transformer-based reconstruction approach also masks values to zero, which can shift the original time-series distribution between pre-training and fine-tuning.Source: B. Self-supervised learning for time… · p. 2

III. CROSS RECONSTRUCTION TRANSFORMER

  • CRT is a Transformer-based time-frequency cross-reconstruction method that learns representations from both domains and transfers them to downstream tasks.It drops data parts rather than zeroing masked values, and combines dropping with temporal-spectral modeling and progressive training.Source: §III · p. 3

A. Dropping Rather Than Masking

The section replaces zero-masking with dropping to avoid altering time-series patterns, then incorporates phase information to model temporal-spectral relations. It motivates phase as useful for preserving shape and describes how dropping retains global context.

A. Dropping Rather Than Masking

  • Masking sets temporal segments to zero, changing the original pattern and distribution and potentially adding noise and a pretraining–fine-tuning gap.The paper notes that shape is an important time-series pattern.Source: A. Dropping Rather Than Masking · p. 3
  • Dropping discards selected patches while feeding the remaining real data to the model, and position information helps preserve global context and long-term dependencies.The input is sliced into equal-length patches and patches are randomly discarded.Source: A. Dropping Rather Than Masking · pp. 3–4

B. Involving Spectral Information

  • Frequency-domain information offers another perspective on time-series patterns, and the section proposes modeling it alongside temporal information to capture cross-domain correlations.Source: B. Involving Spectral Information · p. 4
  • Magnitude alone cannot restore a complex frequency value, so the method introduces phase alongside magnitude to retain full spectral information.Source: B. Involving Spectral Information · p. 4
  • With min-max normalization, reconstruction using phase has Euclidean distance 1.62 from the original, versus 5.05 using magnitude, and tends to preserve its shape.Magnitude-based reconstruction tends to preserve the original value instead.Source: B. Involving Spectral Information · p. 4, Figure 4 · p. 4

C. Dropped Temporal-Spectral Modeling

Dropped Temporal-Spectral Modeling combines time-series values with frequency-domain magnitude and phase, then drops patches and uses a Transformer to reconstruct both domains. An Instance Discrimination Constraint encourages sample-specific, discriminable representations.

C. Dropped Temporal-Spectral Modeling

  • The model forms its input by concatenating the time series with its frequency-domain magnitude and phase vectors.The resulting input has length L.Source: C. Dropped Temporal-Spectral Modeling · p. 5
  • It drops patches from time, magnitude, and phase inputs, pairing each dropped frequency patch with its corresponding phase or magnitude patch for reconstruction.The method states that dropping does not change the original data shape and can reduce the pretraining–fine-tuning gap.Source: C. Dropped Temporal-Spectral Modeling · p. 5
  • Three CNNs extract local features from the patch types before their embeddings, domain-type tokens, and position embeddings enter the Transformer.The three embedding streams have separate learnable [CLS] tokens.Source: C. Dropped Temporal-Spectral Modeling · p. 5
  • The Transformer representations reconstruct the original time, magnitude, and phase data, while the Instance Discrimination Constraint reduces shared information between samples and sharpens decision boundaries.The constraint also seeks greater distance between representations of different samples in latent space.Source: C. Dropped Temporal-Spectral Modeling · p. 5

D. Model Optimization

Model optimization combines reconstruction and instance-discrimination losses, with negligible added optimization complexity. Training begins with a small dropping ratio and progressively raises it.

D. Model Optimization

  • The final objective adds the Instance Discrimination Constraint loss to reconstruction loss, weighted by β.The paper says the frequency data are prepared before pretraining, making the added optimization complexity negligible.Source: D. Model Optimization · p. 5
  • Curriculum learning starts with a relatively small dropping ratio and increases it during pretraining because large initial ratios make dropped patches difficult to reconstruct.The initial and final ratios are validation-set hyperparameters; the schedule holds at rmin, increases, then remains at rmax.Source: D. Model Optimization · p. 5

IV. EXPERIMENTS

The experiments use three public time-series datasets and simulate scarce labels by pretraining on the full training split, then fine-tuning with 20%.

1) Datasets:

  • Three public datasets—PTB-XL, HAR, and Sleep-EDF—are used to evaluate time-series classification.Source: 1) Datasets · pp. 5–6
  • PTB-XL contains 21,837 12-lead ECG recordings, each 10 seconds long, sampled at 500 Hz.Source: 1) Datasets · p. 5
  • HAR contains 10,299 nine-variable activity sequences of length 128 from 30 people, sampled at 50 Hz.Source: 1) Datasets · p. 6
  • Sleep-EDF uses recordings from four subjects, segmented into 30-second samples across three EEG/EOG channels for eight-class sleep-stage classification.Source: 1) Datasets · p. 6
  • Each dataset is split 80/10/10 for training, validation, and testing; pretraining uses the full training set and fine-tuning uses a random 20%.Source: 1) Datasets · p. 6

2) Model Architecture:

CRT uses a Transformer autoencoder with convolutional local feature extraction, processing temporal and spectral representations as separate patch types for reconstruction.

2) Model Architecture:

  • The encoder combines a one-dimensional ResNet-18 CNN local feature extractor with a six-layer Transformer, while the two-layer Transformer decoder reduces pretraining time.Source: 2) Model Architecture · p. 6

3) Data processing:

  • For each channel, preprocessing retains the first half of the FFT sequence and stores its magnitude and phase, then concatenates these with the time sequence.Source: 3) Data processing · p. 6
  • Patches contain only time, magnitude, or phase data, and are randomly dropped with epoch-dependent probability r during pretraining.Source: 3) Data processing · p. 6

4) Baselines:

The study compares CRT with contrastive and reconstruction-based baselines, evaluates fine-tuned models across repeated runs, and reports CRT’s overall advantage and relatively stable results.

4) Baselines:

  • Baselines include contrastive methods for time-series or general data, plus TST, which reconstructs masked time-domain values set to zero.Source: 4) Baselines · p. 6

5) Evaluations:

  • The evaluation fine-tunes the full model with a two-layer classifier, selecting hyperparameters and checkpoints using validation accuracy.Source: 5) Evaluations · pp. 6–7
  • Five pretraining seeds and five fine-tuning seeds per pretrained model yield 25 results per method and dataset, assessed with ROC-AUC, F1-score, and accuracy.Source: 5) Evaluations · p. 7

B. Comparison Results

  • CRT outperforms all reported baselines on ROC-AUC, F1-score, and accuracy, with relatively small standard deviations across downstream results.Source: B. Comparison Results · p. 7
  • Scalable Representation Learning is omitted because it required substantially longer runtime and its results could not be produced within several days.Source: B. Comparison Results · p. 7

C. Analysis of Cross-Domain Reconstruction

The analysis tests dropping against masking and examines the contributions of phase and cross-domain reconstruction, reporting improvements for dropping and phase inclusion.

1) Dropping is Better than Masking:

2) Adding Phase Helps Frequency Learning:

C. Analysis of Cross-Domain Reconstruction

  • The experiments compare time-to-time, frequency-to-frequency, and cross-domain reconstruction, with performance shown for five PTB-XL classes and multiple Sleep-EDF training-set ratios.Source: Table V · p. 8, Figure 7 · p. 8

3) Cross-Domain is Better than Single-Domain:

Cross-domain representations outperform single-domain reconstruction across the reported metrics, classes, and fine-tuning training sizes. The section also compares cross-reconstruction directions and reports reconstruction of dropped trends at a high dropping ratio.

3) Cross-Domain is Better than Single-Domain:

  • Cross-domain reconstruction performs best on all three reported metrics, improves PTB-XL results across all five classes, and yields higher Sleep-EDF accuracy across fine-tuning training sizes.The reported training-set ratios are 20%, 40%, 60%, and 80%.Source: 3) Cross-Domain is Better than… · p. 8
  • Time-domain reconstruction outperforms frequency-domain reconstruction on the three classification tasks, while adding frequency information can supplement temporal patterns.The authors suggest temporal patterns may be more direct for neural networks to discover and that the tasks are strongly related to them.Source: 3) Cross-Domain is Better than… · p. 8

4) Exploring Cross-Domain Learning:

  • T2F reconstructs dropped frequency data from time-domain features, whereas F2T reconstructs dropped time data from frequency-domain features.The comparison uses the same inputs for fine-tuning to evaluate the cross-reconstruction tasks.Source: 4) Exploring Cross-Domain Learning · p. 8
  • With a 0.7 dropping ratio, the model nearly reconstructs dropped-input trends, especially in the PTB-XL ECG case.The authors attribute the ECG reconstruction in part to periodicity and abundant similar patterns in the time series.Source: Figure 8 · p. 8, 4) Exploring Cross-Domain Learning · p. 9

5) CL and IDC Help Learning Cross-Domain Representations:

CRT outperforms existing methods on three datasets, and the paper reports component studies of curriculum learning and instance discrimination. The conclusion identifies out-of-distribution scenarios as future work.

5) CL and IDC Help Learning Cross-Domain Representations:

  • Curriculum learning progressively increases the dropping ratio, helping reconstruction and downstream performance; IDC reduces mutual information shared across time series to encourage discriminable patterns.IDC is not a contrastive loss and does not minimize distances between sample pairs.Source: 5) CL and IDC Help Learning… · p. 9
  • At a 0.7 dropping ratio, the model almost reconstructs dropped-input trends, especially for the PTB-XL ECG case.The authors suggest ECG periodicity and abundant similar patterns may explain the reconstruction result.Source: 5) CL and IDC Help Learning… · p. 9

V. CONCLUSION

  • CRT outperforms existing methods on three widely used datasets.The conclusion also reports experiments verifying the effectiveness of framework components.Source: §V · p. 9
  • Adapting CRT to out-of-distribution scenarios and addressing more challenging problems remain future work.Source: §V · p. 9

Related papers

Loading 2205.09928v2…