Source-linked AI summary
TimeNet: Pre-trained deep recurrent neural network for time series classification
Pankaj Malhotra, Vishnu TV, Lovekesh Vig, Puneet Agarwal, Gautam Shroff
TL;DR
Labeled time-series data is costly, motivating representations learned from abundant unlabeled data across domains. The paper trains TimeNet as an unsupervised seq2seq-based deep RNN feature extractor and finds its embeddings effective for classification, often outperforming domain-specific RNN embeddings and DTW-C.
Problem
Labeled time-series data is costly, while unlabeled sensor data is abundant; the paper therefore examines whether unlabeled data can support useful cross-domain time-series representations.
Method
TimeNet is a multilayered RNN trained as the encoder of a sequence auto-encoder on unlabeled time series from diverse domains.
Results
For several time-series classification datasets not used in training, TimeNet embedding classifiers outperform classifiers using data-specific RNN embeddings and DTW-C; they match or exceed DTW-C on 83% (25/30) of datasets.
Takeaways & Limitations
A pre-trained TimeNet can serve as a generic off-the-shelf feature extractor whose embeddings remain competitive even with significantly less labeled training data.
Abstract
from arXiv · showhide
Inspired by the tremendous success of deep Convolutional Neural Networks as generic feature extractors for images, we propose TimeNet: a deep recurrent neural network (RNN) trained on diverse time series in an unsupervised manner using sequence to sequence (seq2seq) models to extract features from time series. Rather than relying on data from the problem domain, TimeNet attempts to generalize time series representation across domains by ingesting time series from several domains simultaneously. Once trained, TimeNet can be used as a generic off-the-shelf feature extractor for time series. The representations or embeddings given by a pre-trained TimeNet are found to be useful for time series classification (TSC). For several publicly available datasets from UCR TSC Archive and an industrial telematics sensor data from vehicles, we observe that a classifier learned over the TimeNet embeddings yields significantly better performance compared to (i) a classifier learned over the embeddings given by a domain-specific RNN, as well as (ii) a nearest neighbor classifier based on Dynamic Time Warping.
1. Introduction
The paper investigates whether abundant unlabeled sensor time series can train a domain-general representation model for downstream analysis. It proposes TimeNet, an unsupervised deep RNN feature extractor, and evaluates its embeddings for classification across datasets and domains.
- Motivation: Industrial event detection supports critical design, engineering, and operational decisions, but practice commonly relies on unsupervised, domain-specific heuristics because labeled time series are scarce.The paper highlights diagnostic trouble codes, anomaly-detection methods, and hard-stop rules as examples.
- Approach: TimeNet is a deep recurrent neural network trained on unlabeled time series to transform sequences into representations useful for further analysis.Its intended use is as a generic, off-the-shelf feature extractor rather than a domain-specific model.
- Approach: The approach adapts the generic-feature-extractor idea from deep image networks to time series by using multilayered RNNs and seq2seq models.The encoder maps a sequence to a fixed-dimensional vector, while the broader motivation comes from successful generic image representations.
- Evaluation: Across several UCR time series classification datasets, TimeNet embeddings are compared with domain-specific SAE embeddings and a Dynamic Time Warping nearest-neighbor classifier.The evaluation also includes qualitative t-SNE analysis and datasets outside the data used to train TimeNet.
- Findings: Even with significantly less labeled training data, classifiers using TimeNet embeddings achieve competitive performance against DTW-C, suggesting robust representations.This result is reported as a contribution of the paper.
- Findings: t-SNE visualizations show well-separated embedding clusters for different classes and for time series from different domains.The visualization is presented as evidence that TimeNet captures important time-series characteristics.
2. Preliminary: Multilayered RNN with Dropout
The preliminary model is a multilayered RNN with Gated Recurrent Units and dropout. Its hidden states are updated iteratively across time and layers using gated transformations, with dropout applied to nonrecurrent connections.
- Multilayered RNN: Each hidden state in the multilayered RNN is computed from the previous hidden state of the preceding layer.The formulation treats the network as processing information hierarchically across layers.
- Temporal recurrence: The hidden-state transition function is applied iteratively for time steps t = 1 to T.This defines recurrent processing over the full input sequence.
- Gated updates: The recurrent transformations use dropout, vector concatenation, and element-wise operations to update gated hidden states.The Hadamard product denotes element-wise multiplication, while D(.) is the dropout operator.
- Inputs and parameters: For the first hidden layer, the input at time step t is z_t, while weight matrices W_r, W_u, and W_p parameterize the recurrent transformations.The sigmoid and tanh activation functions are applied element-wise.
3. Learning Timenet using Sequence Auto-encoder
Timenet is learned as a sequence auto-encoder whose encoder transforms variable-length time series into fixed-dimensional embeddings. The jointly trained decoder reconstructs each input, making the final encoder state a representation for downstream analysis.
- The sequence auto-encoder uses multilayered GRU encoder and decoder RNNs trained jointly to minimize reconstruction error on diverse unlabeled time series.
- The encoder processes a time series and its final hidden state serves as the vector embedding of that series.
- The decoder is initialized with the encoder’s final state and reconstructs the input sequence in reverse order.
- The reconstruction objective compares reconstructed values with the original time-series values across N instances.
- During training, the decoder receives only the encoder’s final hidden state and the required number of decoding steps, rather than the input sequence at each step.
- For an L-layer model with c recurrent units per layer, embedding computation applies recurrent matrix operations across layers and time steps.
4. Experimental Evaluation
The evaluation tests Timenet on diverse unseen time-series classification datasets using visualization and classifier comparisons. Timenet embeddings generally separate classes and domains and often outperform data-specific SAE embeddings and DTW-based classification, including with reduced labels.
- Experimental setup: The evaluation uses UCR datasets plus an industrial telematics dataset covering normal and abnormal engine behavior.
- Experimental setup: Timenet training uses 18 UCR datasets, validation uses 6, and evaluation uses 30 unseen test datasets spanning 151 classes.
- Experimental setup: The selected Timenet has 3 hidden layers, 60 units per layer, and a 180-dimensional embedding.
- Embedding visualization: T-SNE visualizations show well-separated clusters for different classes within datasets and for time series drawn from different datasets.
- Classification results: TN-C and SAE-C match or exceed DTW-C on 83% (25/30) and 73% (22/30) of datasets, respectively, while TN-C exceeds SAE-C on 60% (18/30).
- Classification results: TN-C outperforms PROP on 4 of 15 test datasets with reported PROP results.
- Classification results: Using two-thirds of labeled training data, TN-C2/3 performs better than DTW-C on 66% (20/30) of datasets.
- Layer analysis: For short time series, a single Timenet layer can perform reasonably or outperform the full embedding, whereas longer series favor the full Timenet classifier.
5. Related Work
Prior time-series classification work used distance-based, statistical, and domain-specific deep-learning approaches. This paper positions TimeNet as a generic unsupervised encoder for diverse, varying-length time series without windowing.
- Time-series classification commonly uses time-warping or edit-distance variants in nearest-neighbor classifiers.
- The paper claims to be the first to use unlabeled, varying-length time series from diverse domains to train a multilayered recurrent feature extractor.
- Earlier deep-learning approaches included RNNs, restricted Boltzmann machines, time-delay networks, convolutional auto-encoders, and deep belief networks.
- Unlike static convolutional auto-encoders and deep belief networks, TimeNet uses a seq2seq model to learn representations without fixed-length windowing.
- Related work explored generic neural representations for audio and data mixing across similar-length time series, whereas this work uses varying-length series from diverse domains.
6. Discussion
TimeNet addresses the cost of labeled data by learning representations from diverse unlabeled time series. Its embeddings are effective for classification, while domain-specific encoders may overfit when training instances are scarce.
- TimeNet uses an unsupervised sequence auto-encoder on diverse unlabeled time series to produce fixed-dimensional embeddings with a deep recurrent network.
- TimeNet embeddings are effective for time series classification, and t-SNE visualizations support their ability to produce meaningful representations.
- Domain-specific sequence auto-encoder encoders also provide useful classification representations but may overfit when few training instances are available.
- Evaluating TimeNet embeddings for anomaly detection and clustering is identified as a plausible direction for future work.
Appendix A. Datasets used for training Timenet
Table 2 describes the datasets used to train and validate TimeNet through their time-series length, class count, and number of time series.
- Table 2 covers the training and validation datasets used for TimeNet.
- T denotes time series length, C denotes number of classes, and N denotes number of time series.