Source-linked AI summary
Mixing Up Contrastive Learning: Self-Supervised Representation Learning for Time Series
Kristoffer Wickstrøm, Michael Kampffmeyer, Karl Øyvind Mikalsen, Robert Jenssen
TL;DR
Learning useful time-series representations without labels is challenging, especially in medicine, where data may be abundant but labeling is costly. The paper predicts mixup weights as soft targets in a self-supervised contrastive framework, reporting strong comparisons on univariate and multivariate datasets and benefits for clinical transfer.
Problem
Learning useful representations from time series without labels is challenging, while clinical data can be abundant but costly to label.
Source: §1 · p. 1Method
The framework mixes two time-series samples and trains a contrastive model to predict the mixing component as a soft target.
Source: §1 · p. 2, §2 · p. 2, §2.1 · p. 3Results
The learned representations perform best among the compared baselines on univariate and multivariate datasets, and pretraining improves performance on most clinical datasets.
Source: §3.1 · p. 4, §3.2 · p. 5Takeaways & Limitations
The framework offers a self-supervised representation approach evaluated for transfer learning on clinical time series, including echocardiogram classification.
Source: §1 · p. 2, §3.2 · p. 5, §4 · p. 6
Abstract
from arXiv · showhide
The lack of labeled data is a key challenge for learning useful representation from time series data. However, an unsupervised representation framework that is capable of producing high quality representations could be of great value. It is key to enabling transfer learning, which is especially beneficial for medical applications, where there is an abundance of data but labeling is costly and time consuming. We propose an unsupervised contrastive learning framework that is motivated from the perspective of label smoothing. The proposed approach uses a novel contrastive loss that naturally exploits a data augmentation scheme in which new samples are generated by mixing two data samples with a mixing component. The task in the proposed framework is to predict the mixing component, which is utilized as soft targets in the loss function. Experiments demonstrate the framework's superior performance compared to other representation learning approaches on both univariate and multivariate time series and illustrate its benefits for transfer learning for clinical time series.
1. Introduction
Learning useful time-series representations without labels is challenging, particularly in medicine, where data may be abundant but labeling is costly. The paper introduces a self-supervised contrastive framework that predicts mixup strength and evaluates representation quality and clinical transfer.
1. Introduction
- Unlabeled time-series representations could support clinical transfer learning, where abundant data may coexist with costly labeling.Transfer learning transfers knowledge from a source domain to a target domain and can use large unlabeled datasets to train systems on smaller labeled datasets.Source: §1 · p. 1
- Time-series augmentation is challenging because heterogeneous datasets can have different invariances, and noise-based augmentations are not generally applicable.Contrastive learning has had encouraging results for images, where applicable augmentation schemes are more readily available.Source: §1 · p. 1
- The framework’s learned representations perform best among the compared baselines on both univariate and multivariate datasets.The introduction also reports improved performance when transferring to clinical time series, including echocardiogram classification.Source: §1 · p. 2
- The framework creates mixed samples from two data points and predicts the mixing strength, using it as a soft target motivated by label smoothing.The approach is based on mixup, which generates augmented samples through convex combinations of data points.Source: §1 · p. 2
2. Mixup Contrastive Learning
The framework mixes training time series and uses their mixing component as a soft target for contrastive representation learning. Its MNT-Xent loss predicts the amount of mixing rather than identifying only a hard positive pair.
2. Mixup Contrastive Learning
- Because time-series invariances are rarely known in advance, careless transformations can bring unlike samples close together or alter a series’ trend.Source: §2 · p. 2
- Mixup forms each augmented series as a convex combination of two randomly drawn training series, with λ sampled from Beta(α, α) controlling their contributions.Source: §2 · p. 2
- The mixing parameter supplies soft targets λ and 1−λ, motivated by label smoothing and intended to avoid dataset-specific noise-parameter tuning.The paper says label smoothing can improve performance and reduce overconfidence in deep learning.Source: §2 · p. 2
2.1. A Novel Contrastive Loss for Unsupervised Representation Learning of Time Series
- Two minibatches and their mixed batch pass through encoder f, then projection head g produces task-dependent representations for the contrastive loss.Source: §2.1 · p. 3
- MNT-Xent replaces hard positive-pair identification with prediction of the mixup amount in a new contrastive loss.The loss is applied to projection-head representations.Source: §2.1 · pp. 2–3
3. Experiments and Results
The experiments evaluate MCL on univariate and multivariate time-series benchmarks and test encoder pretraining for clinical time-series classification. MCL leads the representation benchmarks, while pretraining improves results on most ECG datasets, with some exceptions and overfitting.
3.1. Evaluating Quality of Representation
- MCL yields the best representations on both univariate and multivariate datasets, outperforming all compared baselines across aggregate accuracy and ranking metrics.The per-dataset comparison also places most results in MCL's favor.Source: Table 1 · p. 4, §3.1 · p. 4
- Handcrafted features perform poorly even against raw inputs; contrastive features are comparable to raw inputs, while autoencoder features improve slightly.Source: §3.1 · p. 4
3.2. Transfer Learning for Clinical Time Series
- Clinical transfer experiments pretrain an encoder on larger pretext datasets, then use its weights to initialize a supervised FCN for echocardiogram classification with limited training data.The pretext domains are synthetic, dissimilar, and similar to the ECG target data.Source: §3.2 · p. 5
- Pretrained weights improve most ECG datasets and converge faster, but random initialization wins on ECG200 and similar- or dissimilar-domain pretraining overfits after 50 epochs.The transfer table reports test accuracy; the convergence comparison averages five training runs.Source: Table 2 · p. 5, §3.2 · p. 5, Figure 5 · p. 5
- Pretext-domain choice matters, and a dissimilar domain performs comparably to a similar one; the authors note that Swedish Leaf is more complex and erratic than periodic ECG5000.Source: §3.2 · p. 5
- Synthetic pretraining also improves performance on some datasets, suggesting generated data may help initialize models when little target data and no pretext task are available.Source: §3.2 · p. 5
4. Discussion and Conclusion
The paper contrasts noise-based and temporal-information approaches to time-series contrastive learning, then identifies combining them as future work. It also points to memory-based feature merging as a possible way to improve clinical transfer learning.
4. Discussion and Conclusion
- Noise-based contrastive learning may succeed on periodic data, while temporal information can be useful when it is discriminative.The paper notes that temporal approaches can encounter challenges with periodic data.Source: §4 · p. 5
- Combining noise-based and temporal-based contrastive learning is proposed as a promising direction for future research.Source: §4 · p. 6
- Memory-based merging of source- and target-domain features is suggested as a possible way to improve transfer-learning performance.Source: §4 · p. 6
- The framework was evaluated on numerous datasets and demonstrated good results for clinical time-series transfer learning.Source: §4 · p. 6
Appendix A. Results on individual datasets
Appendix tables report accuracy results for methods across the UCR and UEA datasets, with learning-method scores averaged over five training runs.
Appendix A. Results on individual datasets
- Tables A.3, A.4, and A.5 report accuracy for all evaluated methods on the UCR and UEA datasets.Source: Appendix A · p. 6
- For AE, CL, and MCL, reported scores are averages across five independent training runs.Source: Appendix A · p. 6
- Results from all five runs and individual-dataset ranks are available with the code.Source: Appendix A · p. 6