PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency

Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, Marinka Zitnik

arXiv:2206.08496v3cs.LGcs.AI

TL;DR

Time-series pre-training must transfer across datasets with differing temporal dynamics without access to target examples during pre-training. TF-C brings time- and frequency-based representations together, and across eight datasets it outperformed all baselines by 15.4% (F1 score) on average.

  • Problem

    Time-series pre-training must transfer across datasets with differing temporal dynamics, while target datasets are unavailable during pre-training.

    Source: §1 · p. 2, §3 · p. 4

  • Method

    TF-C uses self-supervised contrastive estimation to bring time-based and frequency-based representations and their local neighborhoods close together in latent space.

    Source: §6 · p. 10

  • Results

    Across eight time-series datasets, TF-C outperformed all eight state-of-the-art baselines by 15.4% (F1 score) on average.

    Source: §1 · p. 3

  • Takeaways & Limitations

    TF-C provides a pre-training mechanism intended to support knowledge transfer between time-series datasets.

    Source: §6 · p. 10

    TF-C’s current embedding strategy and losses favor classification over tasks that use local context, such as forecasting.

    Source: §6 · p. 10

Abstract

from arXiv · show

Pre-training on time series poses a unique challenge due to the potential mismatch between pre-training and target domains, such as shifts in temporal dynamics, fast-evolving trends, and long-range and short-cyclic effects, which can lead to poor downstream performance. While domain adaptation methods can mitigate these shifts, most methods need examples directly from the target domain, making them suboptimal for pre-training. To address this challenge, methods need to accommodate target domains with different temporal dynamics and be capable of doing so without seeing any target examples during pre-training. Relative to other modalities, in time series, we expect that time-based and frequency-based representations of the same example are located close together in the time-frequency space. To this end, we posit that time-frequency consistency (TF-C) -- embedding a time-based neighborhood of an example close to its frequency-based neighborhood -- is desirable for pre-training. Motivated by TF-C, we define a decomposable pre-training model, where the self-supervised signal is provided by the distance between time and frequency components, each individually trained by contrastive estimation. We evaluate the new method on eight datasets, including electrodiagnostic testing, human activity recognition, mechanical fault detection, and physical status monitoring. Experiments against eight state-of-the-art methods show that TF-C outperforms baselines by 15.4% (F1 score) on average in one-to-one settings (e.g., fine-tuning an EEG-pretrained model on EMG data) and by 8.4% (precision) in challenging one-to-many settings (e.g., fine-tuning an EEG-pretrained model for either hand-gesture recognition or mechanical fault prediction), reflecting the breadth of scenarios that arise in real-world applications. Code and datasets: https://github.com/mims-harvard/TFC-pretraining.

1 Introduction

TF-C addresses the difficulty of transferring time-series representations across datasets with different temporal dynamics by aligning time- and frequency-based representations during self-supervised pre-training. Across eight datasets, it outperformed the evaluated baselines in both one-to-one and one-to-many transfer settings.

Introduction

  • Time-series transfer is difficult because temporal dynamics and other properties vary across datasets, while target examples are unavailable during pre-training and labels may be scarce.The paper motivates a shared latent property that can support transfer to unseen target datasets.Source: §1 · p. 2
  • TF-C brings time- and frequency-based embeddings of the same sample closer than embeddings of different samples using contrastive learning and a consistency loss.Its frequency-based contrastive instance discrimination uses novel augmentations based on frequency-spectrum characteristics; the learned relationship initializes downstream fine-tuning.Source: §1 · p. 2, Figure 1 · p. 2
  • TF-C outperformed eight state-of-the-art baselines by 15.4% on average in F1 score across eight datasets and improved on the strongest baselines by up to 7.2% in F1 score.The evaluation covered one-to-one and one-to-many settings.Source: §1 · p. 3
  • In one-to-many transfer, TF-C improved prior work by 8.4% in precision when pre-training on sleep EEG and fine-tuning on hand-gesture recognition.This setting applies the same pre-trained model to multiple independent fine-tuning datasets.Source: §1 · p. 3

2 Related Work

Time-series pre-training remains underexplored, with no established conceptual basis for generalizable pre-training and no target examples available during pre-training. TF-C addresses these gaps with contrastive learning across multiple invariances, including direct frequency-spectrum perturbations.

Pre-training for time series

  • Time-series self-supervised pre-training remains underexplored, with no established conceptual basis for generalizable pre-training; Shi et al. is the only explicitly designed model, but its pretext-task rationale is unclear.The paper positions TF-C as designed to be invariant across time-series datasets and to produce generalizable pre-training models.Source: §2 · p. 3
  • Unlike domain adaptation [21] [42], which uses target data during training, TF-C pre-trains without target examples or labels, unlike self-supervised domain adaptation [44], which still requires training labels.Source: §2 · p. 3

Contrastive learning with time series

  • Existing time-series contrastive methods such as CLOCS [41] and TNC [46] use temporal neighborhoods; other approaches encode transformation invariance [40] or contextual invariance [47] [48].Source: §2 · p. 3
  • TF-C adds an augmentation bank spanning multiple invariances and directly perturbs frequency spectra, unlike CoST [49] and BTSF [50], whose augmentations remain time-domain.The paper presents direct spectrum perturbation as the first use of frequency invariance for contrastive learning. TF-C applies its consistency objective to two individually contrastive-trained encoders.Source: §2 · p. 3

3 Problem Formulation

The problem is to pre-train on unlabeled time series and transfer the model to a small labeled target dataset that is unavailable during pre-training. TF-C uses the relationship between time- and frequency-domain representations to encourage transferable representations.

3 Problem Formulation

  • Pre-training uses unlabeled Dpret, while fine-tuning uses a small labeled Dtune that is not accessed during pre-training, allowing reuse across target datasets without retraining.The target dataset is independent of the pre-training data, distinguishing this setup from domain adaptation.Source: §3 · p. 4
  • TF-C requires time-based and frequency-based representations of a sample, along with representations of its local augmentations, to be proximal in a shared latent time-frequency space.The time-frequency relationship is grounded in signal-processing theory and is proposed as an invariance across time-series distributions.Source: §3 · p. 4
  • The approach induces TF-C in pre-trained parameters so they can initialize a target model, bridging datasets with discrepancies in temporal dynamics or semantic meaning.The paper presents TF-C's invariant nature as a basis for general pre-training on time series.Source: §3 · p. 4
  • TF-C uses a time encoder, a frequency encoder, and two cross-space projectors to map representations into a shared space and promote time-frequency alignment.The aligned pre-trained model is intended for transfer to target datasets not seen during pre-training.Source: §3 · pp. 4–5, Figure 2 · p. 5

4 Our Approach

TF-C pre-training combines time- and frequency-based contrastive encoders with a consistency objective that brings their representations together. The model is described using univariate series and can also be applied to multivariate series.

4.1 Time-based Contrastive Encoder

  • Time-based contrastive learning uses diverse temporal augmentations and NT-Xent to bring positive-pair embeddings closer while separating negatives.The augmentation bank includes jittering, scaling, time-shifts, and neighborhood segments [40] [48] [41].Source: §4.1 · pp. 5–6

4.2 Frequency-based Contrastive Encoder

  • Frequency-based contrastive learning adds or removes a small number of spectrum components, then trains embeddings to be invariant to these perturbations.The perturbation budget limits changes to preserve similarity to the original in both frequency and time domains; added components are scaled to α · Am, with α = 0.5.Source: §4.2 · p. 6

4.3 Time-Frequency Consistency

  • Projectors map time and frequency embeddings into a joint space, where a triplet-inspired consistency loss brings cross-domain representations together while maintaining relative distance from augmented examples.The loss encourages a smaller distance between a time-based embedding and a frequency-based embedding than between the time-based embedding and an augmented-frequency embedding [59, 60].Source: §4.3 · pp. 6–7

4.4 Implementation and Technical Details

  • The pre-training objective weights the time- and frequency-based contrastive losses together against the consistency loss, and the learned model combines the time and frequency encoders and projectors.λ controls the relative importance of the contrastive and consistency losses; the total loss sums per-sample losses across the pre-training data.Source: §4.4 · p. 7

5 Experiments

The experiments evaluate TF-C across diverse time-series transfer settings and downstream tasks. The reported results show strong performance in one-to-one and one-to-many transfer, clustering, and sample-level anomaly detection.

5 Experiments

  • The evaluation compares TF-C with 10 baselines across eight diverse datasets, using one-to-one and one-to-many transfer as well as clustering and anomaly-detection tasks.Dataset labels are not used during pre-training.Source: §5 · p. 7

5.1 Results: One-to-One Pre-Training Evaluation

  • 16/24 tests: TF-C won 16 and ranked second in 8; it averaged a 15.4% F1-score margin over baselines and beat the strongest baselines by 1.5% across scenarios.For HAR→GESTURE, TF-C achieved 79.91% F1 score versus 74.57% for TS-TCC, a 7.2% margin.Source: Table 1 · p. 8, §5.1 · p. 8
  • The four one-to-one transfers span substantial dataset differences in signal semantics, sampling, channels, and operating conditions, with fine-tuning sets as small as 60 samples.The scenarios are SLEEPEEG→EPILEPSY, FD-A→FD-B, HAR→GESTURE, and ECG→EMG; the reported small target-set counts are EPILEPSY: 60, FD-B: 60, GESTURE: 480, and EMG: 122.Source: §5.1 · p. 8

5.2 Results: One-to-Many Pre-Training Evaluation

  • 14/18 settings: TF-C had the best performance across three challenging one-to-many transfers and beat the strongest baseline by 8.4% in precision on GESTURE.TF-C outperformed non-pre-trained KNN and Random Init by average F1-score margins of 42.8% and 25.1%, respectively.Source: §5.2 · p. 9
  • In the SLEEPEEG→GESTURE ablation, removing LC, LT, and LF reduced precision by 6.1%, 7.2%, and 6.7%, respectively, indicating each component contributes to pre-training performance.Replacing LC with LTT-C or LFF-C yielded 5.3% and 7.2% lower accuracy, respectively.Source: §5.2 · pp. 9–10

5.3 Additional Downstream Tasks: Clustering and Anomaly Detection

  • 5.4% in Silhouette score: TF-C surpassed strongest baseline TS-TCC in SLEEPEEG→EPILEPSY clustering.The evaluation used K-means (K=2) and Silhouette score, Adjusted Rand Index, and Normalized Mutual Information.Source: §5.3 · p. 10
  • 4.5% in F-1 Score: TF-C outperformed five competitive baselines in sample-level anomaly detection for FD-A→FD-B.The FD-B subset contained 1,000 samples—900 undamaged and 100 damaged—and used a one-class SVM on learned representations.Source: §5.3 · p. 10

6 Conclusion

TF-C uses self-supervised contrastive estimation to bring time- and frequency-based representations and their local neighborhoods together, supporting transfer between time-series datasets. The conclusion describes its broad downstream performance and notes limitations and possible extensions.

6 Conclusion

  • TF-C injects time-frequency consistency into contrastive pre-training, bringing time- and frequency-based representations and their local neighborhoods close in latent space to support transfer between time-series datasets.The authors suggest TF-C may serve as a universal pre-training property and that other properties, such as temporal autoregressive processes, may also help.Source: §6 · p. 10

Limitations and future directions

  • Although TF-C expects regularly sampled inputs, irregular series can be handled with encoders such as Raindrop [71] and SeFT [72].For frequency-encoder inputs, resampling or interpolation and regular or non-uniform FFT operations are alternatives.Source: §6 · p. 10

6 Conclusion

  • TF-C performs well across downstream classification, clustering, and anomaly-detection tasks.The conclusion points to results in Section 5.3.Source: §6 · p. 10

Limitations and future directions

  • TF-C’s current embedding strategy and loss functions favor classification, which uses global information, over tasks such as forecasting that use local context.The authors suggest additional generalizable time-series properties as a future direction.Source: §6 · p. 10

Broader Impacts

TF-C uses time-frequency consistency to support task- and domain-agnostic transfer learning, with potential to improve time-series data use across applications. The paper also warns that transferred models may carry demographic or protocol-related bias, particularly in medical settings.

Broader Impacts

  • The authors present time-frequency consistency as a weak, time-series-specific universal property enabling task- and domain-agnostic transfer, and hope to spur research on universal transfer properties and time-series representation learning.They note that time-series representation learning remains underappreciated relative to computer vision and NLP.Source: Broader Impacts · p. 15
  • In medicine, unlabeled data from patients with different diseases but similar physiological conditions could support pre-training when labeled data for a disease are scarce, but transferred predictions may be biased.Bias may arise from demographic imbalances in source data or standardized collection protocols unsuitable for some physiological attributes, and may carry into fine-tuning.Source: Broader Impacts · p. 15
  • The study reports that its publicly available datasets raised no privacy or security concerns and that dataset authors’ responsible-use guidelines were followed.This does not remove the stated risk that source-data biases may transfer to fine-tuning.Source: Broader Impacts · p. 15

Checklist

The checklist affirms that the paper’s claims about TF-C’s scope are supported and reports disclosures on limitations, reproducibility, and data use.

Checklist

  • The checklist says TF-C’s stated generalizability is justified theoretically and empirically, with limitations and potential negative societal impacts addressed in Sections 6 and Broader Impact.Source: Checklist · p. 15
  • The authors report an anonymous code link with implementation details, baseline implementations, and eight datasets, plus training details and hyperparameter settings.Source: Checklist · p. 15
  • Experiments were run five times, with average values and standard deviations reported in Table 1, Tables 4–6, and Table 10.Source: Checklist · p. 15
  • Compute amount and resource details are reported in Appendix E.Source: Checklist · p. 15
  • The work uses eight existing datasets and six contrastive-learning and time-series pre-training baselines, credits their creators, and lists dataset licenses in Appendix D.Source: Checklist · p. 16
  • The authors say the datasets are freely available, require no re-contact with curators, and contain no personally identifiable information or offensive content.Source: Checklist · p. 16

Appendix A Further information on the relationship between our pre-training approach and domain adaptation … Appendix D Additional information on datasets and pre-training evaluation

The appendices distinguish TF-C pre-training from domain adaptation and related contrastive methods, and explain how time-frequency consistency supports transferable representations. They also clarify the self-supervised setting and the rationale for treating time- and frequency-domain views as invariant.

Appendix A Further information on the relationship between our pre-training approach and domain adaptation

  • TF-C uses only pre-training data, transfers to multiple unseen target datasets without per-target retraining, and can accommodate different feature spaces.The appendix contrasts this with domain adaptation, which generally uses target data during training and usually assumes the same feature space across source and target datasets.Source: Appendix A · p. 17
  • TF-C injects time-frequency consistency to capture a generalizable property across datasets despite differences in signal meaning, acquisition conditions, and sampling frequencies.The appendix identifies ECG, EMG, acceleration, and vibration as examples of signals whose semantic meanings may differ.Source: Appendix A · p. 17
  • Self-supervised pre-training here means no labels are available in pre-training, unlike self-supervised or unsupervised domain adaptation, which lacks target labels but still requires pre-training labels.The appendix says this distinction is intended to prevent confusion between the paper’s pre-training problem and domain adaptation.Source: Appendix A · p. 17

Appendix B Detailed differences with CoST and BTSF

  • Unlike BTSF’s same-dataset, instance-dropout invariance, TF-C targets transfer across datasets by capturing consistency across different time series and temporal dynamics.The appendix introduces CoST [49] and BTSF [50] as frequency-involving contrastive methods, and uses BTSF to illustrate the distinctions.Source: Appendix B · p. 17
  • TF-C directly perturbs the frequency spectrum, whereas BTSF augments in the time domain before applying FFT, which can cause information loss.The appendix characterizes TF-C as the first method using frequency-domain augmentations to enable time-series transfer learning.Source: Appendix B · pp. 17–18
  • BTSF fuses temporal and spectral features, while TF-C constrains relationships in a time-frequency embedding space with triplet loss to produce generalizable representations.The appendix describes this consistency-based constraint as TF-C’s advantage for transfer learning across time-series datasets.Source: Appendix B · pp. 17–18

Appendix C Time series invariance between time and frequency domains

  • Because time- and frequency-domain representations carry the same information, enforcing consistency across their local neighborhoods can capture invariance for self-supervised pre-training.The appendix motivates this analogy with image augmentations, where transformed views preserve the underlying object’s information.Source: Appendix C · p. 18

D.1 Datasets · D.2 Varying lengths and dimensionality of time series in pre-training vs. target datasets

The evaluation uses eight diverse time-series datasets across one-to-one and one-to-many transfer scenarios, with small, balanced fine-tuning sets. To handle mismatched sequence lengths and channel counts, it adjusts target data relative to the pre-training data and shares model parameters across channels.

D.1 Datasets

  • Eight datasets span sleep and seizure EEG, bearing faults, human activities and gestures, and cardiac and muscle signals across both transfer settings.Fine-tuning sets are balanced and limited to fewer than 320 samples to evaluate performance with small target datasets.Source: §D.1 · p. 18, Table 3 · p. 19
  • SLEEPEEG provides 371,055 non-overlapping 200-observation sleep-stage samples, while EPILEPSY supplies 11,500 one-second EEG samples with binary seizure labels.For EPILEPSY, 60 samples are used for fine-tuning, 20 for validation, and the remaining 11,420 for testing.Source: §D.1 · pp. 18–19
  • FD-A and FD-B are three-class bearing-condition datasets sampled at 64 kHz, segmented with 5,120-observation windows and shifts of 1,024 or 4,096.The segmentation follows Eldele et al. [48] and is intended to keep samples relatively balanced across classes.Source: §D.1 · p. 19
  • HAR contains nine sensor channels across six activities, while GESTURE has three acceleration channels and 440 recovered samples across eight gestures.For HAR-to-GESTURE transfer, models with channel generalization use all nine HAR channels; other models use only the three acceleration channels.Source: §D.1 · p. 19
  • The imbalanced four-class ECG dataset is segmented into five-second samples, with AUROC and AUPRC reported; EMG provides single-channel 1,500-observation samples from three volunteers.ECG uses those metrics because its atrial-fibrillation and noisy classes are underrepresented; EMG categories correspond to patient disorders.Source: §D.1 · p. 20

D.2 Varying lengths and dimensionality of time series in pre-training vs. target datasets

  • When target and pre-training sequences differ in length, target data are zero-padded or downsampled to match the pre-training anchor; channel-shared encoders and projectors enable multivariate inputs.The model applies at the single-lead level, sharing parameters across channels and concatenating their learned representations into the final sample representation.Source: §D.2 · p. 20

Appendix E TF-C and baseline architectures and implementation details

The appendix specifies TF-C’s time- and frequency-space encoders, training procedure, and fine-tuning classifier, then describes how eight baselines and two additional methods were implemented or adapted for comparison. Baseline settings generally follow the original papers, with adjustments for the study’s datasets and scenarios.

Baseline comparisons

  • The comparison includes eight state-of-the-art baselines plus target-data KNN and a randomly initialized model trained from scratch; baselines follow original-paper settings unless specified.The eight baselines are TS-SD, TS2vec, CLOCS, Mixing-up, TS-TCC, SimCLR, TNC, and CPC; the latter two additional methods use K=2 for KNN and the TF-C model structure for scratch training.Source: Appendix E · pp. 21, 23

TF-C (our model)

  • TF-C uses matched 1-D ResNet-style time and frequency encoders with two-layer projectors, and trains with Adam before initializing fine-tuning from the best-F1 checkpoint.Pre-training uses learning rate 0.0003, weight decay coefficient 0.0005, batch size 64, and 40 epochs; fine-tuning uses a two-layer fully connected classifier and jointly optimizes cross-entropy with the TF-C loss.Source: Appendix E · p. 21

Baseline architectures

  • TS-SD uses convolutional multihead attention and denoising, while TS2vec applies masked dilated-CNN representations and hierarchical contextual-consistency losses.The study retained 12 TS-SD heads and implemented it from the authors’ description because source code was unavailable; TS2vec’s residual blocks were reduced from 10 to 2, with hidden dimension 1 and scenario-dependent batch sizes.Source: Appendix E · p. 22
  • CLOCS treats same-patient ECG views as positive pairs and different-patient samples as negatives, whereas Mixing-up predicts the mixing proportion in convex combinations of two series.CLOCS layer settings were adjusted for input lengths and transfer scenarios; Mixing-up used the beta distribution specified in its original work.Source: Appendix E · p. 22
  • TS-TCC predicts future latent observations across strongly and weakly augmented views, SimCLR contrasts two augmented views, and CPC learns context representations that preserve mutual information with observations.TS-TCC used learning rate 3e-4 and pre-training and fine-tuning batch sizes of 128 and 16; SimCLR applies NT-Xent after ResNet encoding and projection, while CPC uses an encoder and autoregressive context model.Source: Appendix E · pp. 22–23
  • TNC distinguishes observations within a time point’s ADF-defined temporal neighborhood from separated observations, but was limited to scenario 1 one-to-one classification because ADF testing was extremely slow.CPC used the implementation provided by the TNC authors, retaining its default parameters where reasonable.Source: Appendix E · p. 23

Appendix F Additional results on one-to-one evaluation · Appendix G Results on downstream clustering and anomaly detection tasks · Appendix H Further results on ablation study

The appendix extends evaluation to cross-domain one-to-one fine-tuning, downstream clustering and anomaly detection, and ablations; the full TF-C model leads every reported ablation metric in the SLEEPEEG-to-GESTURE example.

Appendix F Additional results on one-to-one evaluation

  • One-to-one evaluation covers four pre-training/fine-tuning dataset pairs whose signals share or have similar semantic meanings, including SLEEPEEG-to-EPILEPSY EEG recordings taken under different physical conditions.Scenario 4 pairs ECG and EMG, both physiological signals.Source: Appendix F · p. 23
  • 5.4% in F1 score: full fine-tuning performed slightly better than partial fine-tuning in the one-to-one Scenario 1 preliminary results.All model parameters were optimized during full fine-tuning; the final model was evaluated on a test set using accuracy, macro-averaged precision, recall, F1-score, one-versus-rest AUROC, and AUPRC.Source: Appendix F · p. 23
  • The appendix reports one-to-one results across four scenarios in Tables 4–6 alongside Table 1 in the main paper.The evaluation first trains each baseline on the pre-training dataset, transfers its converged parameters to the target fine-tuning dataset, and evaluates the final model on a test set.Source: Appendix F · p. 23

Appendix G Results on downstream clustering and anomaly detection tasks

  • Downstream clustering evaluation pre-trains on SLEEPEEG and independently fine-tunes on EPILEPSY, comparing TF-C with Random Init., Non-DL, TS-TCC, TNC, and CPC.TS-TCC is identified as the best-performing classification baseline; the table caption provides no clustering outcome values.Source: Table 7 · p. 24, Appendix G · p. 24
  • Downstream anomaly detection pre-trains on FD-A and fine-tunes on an FD-B subset with 90% normal and 10% abnormal samples, evaluating precision, recall, F1 score, and AUROC.The supplied passages identify the comparison table but do not provide its cell values or establish a winning method.Source: Table 8 · p. 24, Appendix G · p. 24

Appendix H Further results on ablation study

  • In SLEEPEEG → GESTURE one-to-one ablations, full TF-C achieves the highest performance on every evaluation metric, with the consistency loss especially contributing to performance.The study compares model components using SLEEPEEG for pre-training and GESTURE for fine-tuning.Source: Appendix H · p. 24

Appendix I Visualization of embeddings in time-frequency space

The visualizations show that the consistency loss brings paired time- and frequency-based embeddings closer together and increases separation between classes in the fine-tuning data.

Appendix I Visualization of embeddings in time-frequency space

  • In the one-to-one EPILEPSY visualization, consistency loss grouped paired time- and frequency-based embeddings together, reduced the marked sample’s cosine distance from 0.89 to 0.71, and better clustered epilepsy versus health samples.Without the loss, time- and frequency-based embeddings clustered separately, while the two classes’ frequency-based embeddings largely overlapped.Source: Appendix I · pp. 24–25, Figure 3 · p. 26
  • 22.2%: In one-to-many EPILEPSY testing, consistency loss reduced average time–frequency cosine distance from 1.32 to 1.08.Higher distance was observed when the consistency loss was removed; lower distance indicates greater consistency between the embeddings.Source: Figure 4 · p. 26, Appendix I · pp. 26–27
  • 61.4%: In one-to-many EPILEPSY testing, consistency loss increased inter-class cosine distance from 0.88 to 1.42.The paper reports this as evidence of greater embedding distinctiveness.Source: Figure 5 · p. 27, Appendix I · p. 27

Appendix J Additional information on domain augmentations · Appendix K Discussion on many-to-one setting

Appendix J examines frequency-domain augmentation choices for TF-C, while Appendix K reports declining EPILEPSY transfer AUROC as pre-training datasets become more heterogeneous and frames many-to-one pre-training as an open challenge.

Appendix J Additional information on domain augmentations

  • The authors describe their work as the first to directly perturb time-series frequency domains for contrastive learning and analyze eight augmentation policies.The policies vary frequency band, number of perturbed components, and random versus distribution-based selection; the augmentations perturb magnitude while leaving phase unchanged.Source: Appendix J · pp. 27, 29, Figure 6 · p. 28, Table 11 · p. 29
  • Low-band perturbations performed better for physiological signals, whereas high-band perturbations outperformed low-band ones by 0.8% in F1 for FD-A → FD-B.Across four physiological setups, low-band perturbations had a 0.8% change in F1 performance on average relative to high-band perturbations; the authors use perturbations across the entire spectrum for broad generalization.Source: Appendix J · pp. 27, 29
  • Perturbing many frequency components generally degraded performance, so the paper uses single-component perturbations.The analysis used a budget of 5 for multi-component perturbations and 1 for single-component perturbations; the authors attribute poorer performance to larger temporal changes that make augmented samples easier to distinguish contrastively.Source: Appendix J · p. 29
  • The authors found no clear performance advantage for random versus Gaussian component selection and use random sampling in the paper.The two sampling strategies produced very similar F1 scores, and the optimal choice may depend on the signal and prediction task.Source: Appendix J · p. 29, Table 11 · p. 29

Appendix K Discussion on many-to-one setting

  • Unlike single-source transfer setups, many-to-one pre-training combines potentially heterogeneous datasets whose misalignments can hinder learning shared time-series patterns.The authors identify universal representations from multi-source time-series datasets as an open challenge and a direction for future research.Source: Appendix K · pp. 29–30
  • For many-to-one experiments, TF-C was pre-trained on merged datasets and fine-tuned for EPILEPSY classification.In the four-to-one setup, SLEEPEEG, FD-A, HAR, and ECG were merged; preprocessing aligned channels and lengths, and 25% of each dataset yielded 108,300 samples across 18 classes. The two- and three-to-one setups merged SLEEPEEG and FD-A, and those datasets plus HAR, respectively.Source: Appendix K · p. 30
  • AUROC fell from 0.9819 (one-to-one) to 0.9525 (two-to-one), 0.8607 (three-to-one), and 0.7253 (four-to-one) as pre-training heterogeneity increased.The authors interpret this decline as consistent with the difficulty of learning patterns across heterogeneous datasets; whether a framework can overcome this remains an open question.Source: Appendix K · p. 30

Related papers

Loading 2206.08496v3…