Source-linked AI summary
Toward a Foundation Model for Time Series Data
Chin-Chia Michael Yeh, Xin Dai, Huiyuan Chen, Yan Zheng, Yujie Fan, Audrey Der, Vivian Lai, Zhongfang Zhuang, Junpeng Wang, Liang Wang, Wei Zhang
TL;DR
Prior time-series pre-training research had not tested methods with multi-domain data. This paper repurposes the UCR Archive to compare self-supervised methods and architectures, finding that pre-training benefits downstream classification and that TimeCLR with a Transformer outperforms alternatives.
Problem
Prior time-series self-supervised methods had not been tested on pre-training datasets containing time series from multiple domains.
Source: §1 · p. 1Method
The authors repurpose the UCR Archive and evaluate four existing and one proposed self-supervised pre-training method across time-series neural network architectures.
Source: §1 · pp. 1–2, §2 · p. 2Results
Takeaways & Limitations
Abstract
from arXiv · showhide
A foundation model is a machine learning model trained on a large and diverse set of data, typically using self-supervised learning-based pre-training techniques, that can be adapted to various downstream tasks. However, current research on time series pre-training has mostly focused on models pre-trained solely on data from a single domain, resulting in a lack of knowledge about other types of time series. However, current research on time series pre-training has predominantly focused on models trained exclusively on data from a single domain. As a result, these models possess domain-specific knowledge that may not be easily transferable to time series from other domains. In this paper, we aim to develop an effective time series foundation model by leveraging unlabeled samples from multiple domains. To achieve this, we repurposed the publicly available UCR Archive and evaluated four existing self-supervised learning-based pre-training methods, along with a novel method, on the datasets. We tested these methods using four popular neural network architectures for time series to understand how the pre-training methods interact with different network designs. Our experimental results show that pre-training improves downstream classification tasks by enhancing the convergence of the fine-tuning process. Furthermore, we found that the proposed pre-training method, when combined with the Transformer model, outperforms the alternatives.
1 INTRODUCTION
Prior time-series self-supervised methods had not been tested with multi-domain pre-training data. The authors repurpose the UCR Archive to evaluate architectures and pre-training methods for downstream classification.
1 INTRODUCTION
- The study addresses a gap: prior time-series pre-training work had not tested methods on a dataset spanning multiple domains.Source: §1 · p. 1
- The repurposed UCR Archive combines time series from domains including power consumption, heartbeats, and food spectrographs.Source: §1 · p. 1
- The experiments examine whether multi-domain pre-training helps downstream single-domain classification, and which architectures and self-supervised methods work best.Source: §1 · p. 1
- Figure 1 depicts initial training on a large multi-domain time-series dataset followed by fine-tuning on a smaller task-specific training set.Source: Figure 1 · p. 1
- Pre-training improved downstream classification in the experiments, with smoother fine-tuning convergence; TimeCLR with a Transformer was the best-performing combination.Source: §1 · p. 2
2 MODEL ARCHITECTURE
The study uses four backbone architectures for its time-series foundation models: LSTM, GRU, ResNet, and Transformer. Their designs and shared training components are illustrated in Figure 2.
2 MODEL ARCHITECTURE
- The four backbones are LSTM and GRU networks, ResNet, and Transformer; the figure also depicts a projector and classifier used in pre-training and fine-tuning.Source: §2 · p. 2, Figure 2 · p. 2
- The LSTM and GRU designs use two bidirectional recurrent layers, while the Transformer uses fixed positional encoding and a [start] token for the time-series representation.Source: §2 · p. 2
- The Transformer consists of four encoder layers, each with eight heads and 64-dimensional input and output representations.Source: §2 · p. 2
- Layer normalization is used for all normalization layers.Source: §2 · p. 2
3 PRE-TRAINING METHOD
The paper compares four existing self-supervised pre-training methods with proposed TimeCLR. The methods differ in how they form positive pairs, construct representations, and calculate their contrastive objectives.
3 PRE-TRAINING METHOD
- SimCLR forms two randomized augmented views of each time series, processes them through a backbone and projector, and trains with NT-Xent loss on positive pairs.Source: §3 · pp. 2–3
- TS2Vec forms positive pairs from overlapping crops and applies hierarchical temporal and instance-wise contrastive losses across time granularities.Source: §3 · p. 3
- For TS2Vec, the backbone is modified to produce per-time-step representations; removed blocks are restored for classification fine-tuning with a classifier and cross-entropy loss.Source: §3 · p. 3
- MixingUp predicts the mixing weight from representations of two time series and their mixture, while TF-C contrasts representations from time and frequency domains.Source: §3 · p. 3
- TimeCLR extends SimCLR with additional time-series augmentations and uses one augmentation function, rather than all functions, to generate each positive pair.The authors say applying all augmentations could make the resulting series bear little visual similarity to the original.Source: §3 · p. 3
- TF-C uses a separate backbone and projector for each domain, concatenating their projector outputs before classification; the other methods use Figure 3.a.Source: §3 · p. 3, Figure 3 · p. 3
4 EXPERIMENT
Experiments compare 20 pre-training configurations and baselines across 128 UCR downstream classification tasks. Pre-training generally helps after fine-tuning, with TimeCLR plus Transformer performing best overall.
- For each of 128 datasets, half the samples formed the pre-training set; the remaining samples were split 3:1:1 into mutually exclusive training, validation, and test sets.The pre-training set contained samples from all 128 datasets and no label information.Source: §4 · p. 4
- The study crossed five pre-training methods with four architectures, then evaluated task-specific test accuracy across 128 tasks against no-pre-training and ED/DTW baselines.Table 1 reports average ranks, with lower ranks indicating better performance.Source: §4 · p. 4, Table 1 · p. 4
- Without pre-training, ResNet and Transformer outperformed the other methods, while LSTM and GRU performed worse than ED and DTW baselines.Source: §4 · p. 4
- 14 of 20 configurations benefited from pre-training; TimeCLR performed best across backbone types, with TimeCLR plus Transformer achieving the best overall performance.Source: §4 · p. 4
- On harder ArrowHead, TimeCLR pre-training yielded smoother fine-tuning convergence, while pre-trained and scratch models behaved similarly on easier CBF.The authors identify this convergence difference as a likely main reason for downstream gains.Source: §4 · p. 4, Figure 4 · p. 4
5 CONCLUSION
The paper reports benefits from self-supervised foundation-model training for time series using multiple domains. TimeCLR performed best with a Transformer backbone; the authors identify several directions for future work.
- The paper demonstrates benefits from self-supervised learning for time-series foundation models trained on multiple domains, with TimeCLR plus Transformer outperforming alternatives.Source: §5 · pp. 4–5
- Future work includes combining pre-training methods [26] [32] [38] [40], using data compression [35] [36], and testing a 2D ResNet [33] [34].Source: §5 · p. 5