Source-linked AI summary
Unsupervised Scalable Representation Learning for Multivariate Time Series
Jean-Yves Franceschi, Aymeric Dieuleveut, Martin Jaggi
TL;DR
Sparse labels and variable-length time series make scalable, general-purpose representation learning difficult. The paper combines a causal dilated-convolution encoder with a fully unsupervised triplet loss using time-based negative sampling, and reports universal, transferable embeddings that achieve strong classification performance across diverse settings.
Problem
Time-series representation learning must handle sparse labels, unequal lengths, and scalability across short and long inputs, while prior methods provide limited general-purpose evidence.
Method
The method trains a variable-length encoder based on dilated causal convolutions with a novel fully unsupervised triplet loss using time-based negative sampling.
Results
Experiments report universal and transferable representations that outperform concurrent unsupervised methods and achieve state-of-the-art classification performance across diverse tasks and datasets.
Takeaways & Limitations
The resulting embeddings can be used efficiently for diverse tasks such as classification and regression, including variable-length and multivariate time series.
Takeaways & Limitations
The analysis notes a possible training-behavior issue in which representation norms could increase without bound, although the phenomenon was not observed in practice.
Abstract
from arXiv · showhide
Time series constitute a challenging data type for machine learning algorithms, due to their highly variable lengths and sparse labeling in practice. In this paper, we tackle this challenge by proposing an unsupervised method to learn universal embeddings of time series. Unlike previous works, it is scalable with respect to their length and we demonstrate the quality, transferability and practicability of the learned representations with thorough experiments and comparisons. To this end, we combine an encoder based on causal dilated convolutions with a novel triplet loss employing time-based negative sampling, obtaining general-purpose representations for variable length and multivariate time series.
1 Introduction
The paper addresses unsupervised, general-purpose representation learning for multivariate time series with sparse labels, unequal lengths, and demanding scalability requirements. It proposes a scalable encoder and time-based triplet loss, then evaluates transferability across diverse datasets and tasks.
- Sparse labels, unequal lengths, and the need to process both short and long series make general-purpose time-series representation learning challenging.
- The proposed method learns fixed-length representations for variable-length multivariate time series using a scalable dilated-convolution encoder.
- Its novel triplet loss uses time-based negative sampling and is presented as the first fully unsupervised triplet loss for time series.
- Experiments assess representation quality and transferability on UCR, UEA multivariate, and real-life very long time-series datasets.
- The learned representations are reported as general and transferable, outperforming concurrent unsupervised methods and matching state-of-the-art non-ensemble supervised classification techniques.
2 Related Work
Prior unsupervised time-series representation methods are limited by scalability, evaluation breadth, code availability, or comparison quality. The paper positions its scalable model and extensive analysis as addressing these gaps while introducing a fully unsupervised time-series triplet-loss approach.
- Few prior works explicitly study general-purpose unsupervised representation learning for time series without structural assumptions on non-temporal data.
- Existing methods may fail to scale to long series because recurrent models are sequential or DTW has quadratic complexity in input length.
- Other prior studies use few standard datasets, provide no public code, or lack sufficient comparisons for assessing representation quality.
- Triplet losses have seen limited use for time series, and existing applications generally assume class labels or annotations rather than fully unsupervised training.
- Dilated convolutions had been applied to time-series classification and forecasting, motivating their use in a scalable representation-learning architecture.
3 Unsupervised Training
The training procedure replaces decoder-based representation learning with an encoder-only network trained by a time-based triplet loss. It samples nested positive subseries and temporally independent negative subseries so similar inputs converge while dissimilar ones separate.
- The method trains an encoder-only architecture with a novel fully unsupervised triplet loss, avoiding the computational cost of jointly training a decoder.
- A reference subseries xref is paired with a positive subseries xpos drawn from it and a negative subseries xneg sampled from another series or a sufficiently long nonstationary series.
- The loss assimilates representations of xref and xpos while distinguishing xref from xneg.
- The training cost is O(K · c(f)), where K is the number of negative examples and c(f) is the cost of evaluating and backpropagating through the encoder.
- For equal-length datasets, setting negative-example lengths equal to spos can speed training through computation factorizations.
- Training across input lengths from one to the longest training series supports meaningful representations for unequal-length inputs.
4 Encoder Architecture
The encoder uses exponentially dilated causal convolutions to combine variable-length handling with efficient computation and long-range temporal coverage. Global max pooling and a linear transformation produce fixed-size embeddings independent of input length.
- The architecture uses deep exponentially dilated causal convolutions to extract information efficiently while accepting variable-length inputs.
- Unlike sequential recurrent networks, convolutional networks allow efficient parallelization on hardware such as GPUs.
- Exponential dilation increases the receptive field with constant depth, helping capture long-range dependencies.
- Causal convolutions compute each output time step using only input values up to that same time step, excluding future inputs.
- Each layer combines causal convolutions, weight normalization, leaky ReLUs, and residual connections, with exponentially increasing dilation.
- Global max pooling aggregates temporal information into a fixed-size vector, followed by a linear transformation whose output size is independent of input length.
5 Experimental Results
Experiments evaluate the learned representations across classification, sparse-label, transfer, metric-space, multivariate, and long-series settings. The method consistently performs strongly while retaining efficiency and applicability across varying input scales.
- Evaluation scope: The evaluation covers classification, sparse labeling, transferability, metric-space comparison, multivariate classification, and long time series.Experiments use UCR and UEA archives, sparse-label comparisons with ResNet, nearest-neighbor comparisons with DTW, and regression tasks on household electricity data.
- Univariate classification: Average rank 2.92 makes the method second-best among compared supervised non-neural classifiers, behind HIVE-COTE at 1.71 and equivalent to ST at 2.95.Fully supervised ResNet outperforms the method on 63% of 71 UCR datasets, while the method remains close to the best supervised neural network overall.
- Sparse labeling: With 1.5% labeled data, the method reaches 81% accuracy versus 26% for ResNet, and exceeds 99% accuracy from 11% labeled data.ResNet requires more than 50% labeled data to reach the same 99% accuracy level on TwoPatterns.
- Metric space and multivariate series: The representations outperform DTW on 66% of UCR datasets, match or outperform DTWD on 69% of UEA datasets, and support dimensionality-reduction and clustering analyses.The UEA comparison is limited because the archive was recently released and lacked broader state-of-the-art comparisons.
- Long time series: Across day- and quarter-scale inputs, representations cause only slightly degraded regression performance while providing a large efficiency improvement over raw time series.A single encoder produces useful representations for different input lengths and tasks beyond classification.
6 Conclusion
The paper presents scalable, unsupervised time-series embeddings generated by dilated-convolution encoders and efficient triplet loss, with experiments on the IHEPC dataset.
- The IHEPC experiments report results obtained on a long household electricity-consumption dataset.
- The method produces scalable, high-quality, easy-to-use embeddings for time series.The embeddings are generated by dilated convolutions that accept variable-length inputs.
- The encoder uses novel time-based negative sampling within an efficient triplet loss.
- The representations support diverse tasks, including classification and regression.Experiments report state-of-the-art performance for classification.
Appendices
The appendices document training procedures, hyperparameter choices, and comprehensive experimental results for the proposed method and concurrent methods.
- The appendices provide detailed training procedures for classification tasks.
- They describe hyperparameter choices used in the presented experiments.
- They report accuracy scores for method variants across the UCR archive and comparisons with concurrent methods.
S1 Training Details
The supplementary training details cover preprocessing, SVM fitting, training behavior, and numerical stability of the learned representations.
- Preprocessing: Datasets are normalized to zero mean and unit variance, independently per dimension for UEA datasets.
- Classification training: The SVM penalty C is selected by cross-validation using only training representations and labels.For small training sets, C = ∞ is used, corresponding to no regularization.
- Training behavior: Test accuracy increases during unsupervised encoder training as optimization steps accumulate.The reported training procedure stops after the marked 2000 optimization steps.
- Numerical stability: The risk is an expectation of the loss over randomly selected sequences, and scaling representations can create numerical-stability concerns.The text notes that representation norms could grow under certain conditions, although this phenomenon is not observed in practice.
S2 Hyperparameters
The hyperparameter appendix examines the number of negative samples and lists the fixed optimization and architecture settings used across experiments.
- Influence of K: K = 1 yields significantly lower UCR scores than K > 1, while other K values differ mainly across individual datasets.The method therefore combines encoders trained with different K values.
- Detailed choices of hyperparameters: The encoder uses Adam with learning rate α = 0.001 and decay rates β = (0.9, 0.999).
- Detailed choices of hyperparameters: The causal network uses 40 intermediary channels, depth 10, kernel size 3, and 320 pre-pooling output channels.The listed representation dimension is 160.
- Detailed choices of hyperparameters: The IHEPC setting reduces optimization steps to 400, intermediary channels to 30, and pre-pooling output channels to 160.
S3 Univariate Time Series
The univariate evaluation reports results across UCR datasets and compares the method with unsupervised, supervised, and ensemble baselines. Evaluation uses the original dataset splits, while random-seed variation is assessed without reporting standard deviations.
- Evaluation scope: Results cover the first 85 UCR datasets and additional remaining datasets, excluding three DodgerLoop datasets with missing values.The first 85 datasets are reported in Table S1, while Table S4 covers the remaining 43 eligible datasets.
- Evaluation protocol: The reported baseline scores use each UCR dataset’s original train/test split rather than the 100 random resamples used in another evaluation.The authors state that comparable resampling was beyond this work’s scope because it would require substantially more computation.
- Variability: Random-seed accuracy variation for the combined method is below 0.01 in order of magnitude across several example datasets.The reported standard variations are 0.0056 for DiatomSizeReduction, 0.0091 for CricketX, and 0.0053 for UWaveGestureLibraryX.
- Baselines: The combined method is compared with DTW, ST, BOSS, HIVE-COTE, EE, ResNet, TimeNet, and RWS when results are available.The comparisons include unsupervised methods, supervised methods, supervised ensembles, and a supervised deep-learning baseline.
S4 Multivariate Time Series
The multivariate evaluation reports accuracy scores for method variants across UEA datasets and compares them with DTWD. Results use the archive’s unique train/test split, with bold entries marking the best-performing method.
- Evaluation: Accuracy scores for variants of the method are reported on all UEA datasets alongside DTWD.The comparison is compiled in Table S5.
- Evaluation protocol: The UEA evaluation uses the unique train/test split provided in the archive.No alternative resampling protocol is described for this evaluation.
- Reading the results: Bold scores in Table S5 indicate the best-performing method.The table compares variants of the proposed method with DTWD.
S5 Discussion of the Choice of Encoder
The paper chooses a causal CNN encoder rather than an LSTM because scalability is a central aim. On a restricted UCR subset, the proposed encoder outperforms the LSTM by a large margin.
- Encoder choice: The authors exclude an LSTM as the main encoder because the work aims to provide a scalable representation-learning method.They nevertheless conduct a restricted comparison using a two-layer LSTM with the same optimization hyperparameters.
- Comparison: On the restricted experiment set, the causal CNN encoder outperforms the LSTM encoder by a large margin.The comparison results are compiled in Table S6.
- Interpretation: The authors interpret the restricted comparison as indicating that the causal CNN encoder is more adapted to the task and training method.This interpretation is limited to the experiments described.