Source-linked AI summary
Transfer learning for time series classification
Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, Pierre-Alain Muller
TL;DR
Transfer learning had not been thoroughly investigated for time series classification, despite CNNs’ strong performance and overfitting on small datasets. The paper evaluates CNN transfer across the 85-dataset UCR archive and uses Dynamic Time Warping to select source datasets, finding that similar sources generally support better fine-tuning and improved accuracy on 71 datasets.
Problem
Transfer learning for deep CNNs had not been thoroughly investigated for time series classification, although models can overfit, especially with small training sets.
Method
The paper pre-trains fully convolutional neural networks across UCR datasets and uses Dynamic Time Warping to measure inter-dataset similarity for selecting transfer sources.
Results
71 of 85 datasets improved with DTW-guided source selection, while transfer could improve or degrade accuracy depending on the source dataset.
Takeaways & Limitations
Fine-tuning a CNN pre-trained on a similar source dataset can work better than training from scratch, including across time series datasets of varying length.
Takeaways & Limitations
Because of space limitations, the paper presents only visually interpretable cases for the full 85-dataset dissimilarity analysis; the complete analysis is provided on a companion webpage.
Abstract
from arXiv · showhide
Transfer learning for deep neural networks is the process of first training a base network on a source dataset, and then transferring the learned features (the network's weights) to a second network to be trained on a target dataset. This idea has been shown to improve deep neural network's generalization capabilities in many computer vision tasks such as image recognition and object localization. Apart from these applications, deep Convolutional Neural Networks (CNNs) have also recently gained popularity in the Time Series Classification (TSC) community. However, unlike for image recognition problems, transfer learning techniques have not yet been investigated thoroughly for the TSC task. This is surprising as the accuracy of deep learning models for TSC could potentially be improved if the model is fine-tuned from a pre-trained neural network instead of training it from scratch. In this paper, we fill this gap by investigating how to transfer deep CNNs for the TSC task. To evaluate the potential of transfer learning, we performed extensive experiments using the UCR archive which is the largest publicly available TSC benchmark containing 85 datasets. For each dataset in the archive, we pre-trained a model and then fine-tuned it on the other datasets resulting in 7140 different deep neural networks. These experiments revealed that transfer learning can improve or degrade the model's predictions depending on the dataset used for transfer. Therefore, in an effort to predict the best source dataset for a given target dataset, we propose a new method relying on Dynamic Time Warping to measure inter-datasets similarities. We describe how our method can guide the transfer to choose the best source dataset leading to an improvement in accuracy on 71 out of 85 datasets.
I. INTRODUCTION
Transfer learning is investigated as a way to improve CNN generalization for time series classification, where performance can suffer from overfitting and source-dataset choice is underexplored. The paper evaluates transfers across UCR datasets and uses DTW to guide source selection.
- Motivation: CNNs can overfit small time series training sets, as shown by 30% FCN accuracy versus 96% for 1NN-DTW on DiatomSizeReduction.The authors motivate transfer learning as one technique for addressing this generalization problem.
- Research gap: Transfer learning has been rarely applied to time series because no large general-purpose dataset analogous to ImageNet or OpenImages exists.The paper also notes that deep learning was only recently shown to work well for TSC.
- Study design: The study tests every source-target pairing among 85 UCR datasets, producing more than 7140 trained models.Each dataset is used for pre-training and then transferred to the other datasets.
- Findings: The experiments indicate that time series contain low-level features usable for transfer learning and that transfer can reduce training time by requiring fewer convergence epochs.The training-time reduction is reported alongside evidence that transferable features exist.
- Proposed guidance: DTW is used to quantify inter-dataset similarity and predict which source dataset should be considered for transfer to a given target.The proposed guidance is intended to identify beneficial dataset transfers for new TSC problems.
II. BACKGROUND AND RELATED WORK
The background defines time series classification and reviews deep learning and transfer-learning concepts for temporal data. It positions CNNs as effective TSC models while highlighting the open question of reusing knowledge across datasets.
- Definitions: A time series is an ordered set of real values, and its length equals the number of values it contains.The notation represents a series as X = [x1, x2, ...xT].
- Definitions: TSC trains a classifier on labeled time series to map inputs to a probability distribution over class values.A dataset consists of time series paired with corresponding label or class vectors.
- Deep learning for TSC: Deep learning has been applied to TSC, including FCNs with convolutional layers, global average pooling, and a softmax classifier.The reviewed FCN achieved state-of-the-art results on 44 UCR datasets and is selected for this paper’s transfer-learning study.
- Deep learning for TSC: CNNs learn temporal invariant features, supporting their use across time series domains with sufficient data.The review cites applications spanning forecasting, wearable sensors, and electronic health records.
- Transfer-learning motivation: The background asks whether knowledge learned from one time series dataset can boost deep-network performance on another, even for unrelated data.This question motivates examining transfer between datasets rather than training each model independently.
C. Transfer learning for Time Series Classification
Transfer learning for TSC reuses a model trained on source data, unlike domain adaptation, which jointly trains on source and target data. Prior work motivates but does not fully resolve source selection for target classification.
- Concept: Transfer learning trains on a source dataset and then fine-tunes the learned model on a target dataset.Domain adaptation differs because it jointly trains using source and target datasets.
- Related applications: Prior time-series transfer methods have transferred examples or selected series for anomaly detection and classification-related pipelines.The reviewed work includes source-to-target example transfer for constructing dissimilarity matrices.
- Related TSC work: The closest prior TSC study jointly pre-trained an attention-based CNN on several themed source datasets before target fine-tuning.Its differing source themes limited source-dataset choice to one setting.
- Positioning: This paper instead uses an existing deep model without architectural modification or added regularizers to isolate transfer learning’s effect on accuracy.The comparison is designed to distinguish transfer from changes to the model or regularization.
III. METHOD
The method transfers a one-dimensional FCN across time series datasets and uses DTW-based inter-dataset similarities to guide source selection. The architecture accepts variable-length series and produces class probabilities through a dataset-specific softmax layer.
- Transfer framework: The proposed framework computes DTW-based similarities between datasets and uses them to guide transfer learning.The method first introduces the FCN, adapts it for transfer, and then computes similarities for source selection.
- Architecture: The selected network is a one-dimensional FCN chosen for robustness and prior state-of-the-art performance on 44 UCR datasets.The transfer method itself is described as independent of the chosen network architecture.
- Architecture: The FCN accepts variable-length time series and outputs a probability distribution over the dataset’s C classes.Its three convolutional layers use ReLU activations and batch normalization before global average pooling and classification.
- Architecture: Global average pooling averages each learned feature series over time, reducing model parameters and enabling class activation maps.This operation follows the third convolutional layer.
- Network adaptation: Training from scratch and fine-tuning use the same architecture and hyperparameters, except for adapting the final fully connected layer.The architecture shown in Fig. 3 is identical to the previously proposed FCN architecture.
B. Network adaptation
The network is adapted by replacing the source-specific output layer with a target-specific softmax layer and fine-tuning the full network. The framework uses DTW-based dataset similarity to guide source selection while avoiding test-set information.
- Network adaptation: 85 trained networks share the same architecture except for the softmax output layer, whose size depends on the dataset’s number of classes.
- Network adaptation: The source model’s final layer is replaced by a randomly initialized target softmax layer before fine-tuning on the target dataset.
- Network adaptation: Fine-tuning the whole network was chosen because restricting backpropagation to the new output layer caused the network to fail to converge.
- Inter-datasets similarity: DTW similarity guides source-dataset selection because each target dataset has 84 potential sources, making trial-and-error transfer costly.
- Inter-datasets similarity: Dataset similarity is computed from one DBA prototype per class, using the minimum DTW distance between class prototypes as the dataset distance.
- Inter-datasets similarity: Similarity computation uses only training-set time series, eliminating bias from exposure to the test-set distribution.
IV. EXPERIMENTAL SETUP
The framework is evaluated on the UCR archive using its 85 datasets, predefined train/test splits, and z-normalized time series.
- Experimental setup: The UCR archive contains 85 datasets from various real-world domains and provides the default training and testing splits used in the experiments.
- Experimental setup: All UCR time series are z-normalized to mean zero and standard deviation one before evaluation.
B. Experiments
The experiments test transfer in both directions for every pair of UCR datasets, requiring thousands of models and substantial GPU computation.
- Experiments: Each unordered pair of UCR datasets is tested in both source-target directions, producing 7140 experiments across the 85-dataset archive.
- Experiments: A 60-GPU cluster reduced the approximately 168-day sequential workload to less than one week, with models and code released for reproducibility.
- Experiments: The study computes an N × N dataset-similarity matrix as the output of its inter-datasets similarity algorithm.
- Experiments: For each dataset class, the algorithm initializes a prototype with a medoid and refines it using DTW Barycenter Averaging.
- Experiments: Pairwise dataset distance is assigned as the minimum DTW distance across every pair of class prototypes.
V. RESULTS
Transfer learning usually leaves CNN accuracy nearly unchanged, but source choice can produce substantial gains or losses. DTW-based selection is introduced to avoid harmful source choices and identify beneficial transfers.
- Transfer learning accuracy variation matrix: Fig. 4 encodes source datasets by rows and target datasets by columns, with red indicating deterioration, blue improvement, and white little or no change.
- Transfer learning accuracy variation matrix: Fine-tuning almost never hurts CNN performance, as shown by the heat map’s dominance of white cells.
- Transfer learning accuracy variation matrix: For a given target dataset, changing the source dataset can either deteriorate or improve CNN performance.
- Naive transfer learning: Minimum, median, and maximum transferred accuracies are compared with training from scratch to summarize worst, typical, and best source choices.
- Naive transfer learning: The minimum-accuracy comparison shows that a bad source can decrease performance, whereas the maximum-accuracy comparison shows that another source can increase it.
- Naive transfer learning: OliveOil accuracy fell from 93.3% to 16.7% when MALLAT and FaceFour were chosen as source datasets, respectively.
- Naive transfer learning: Blind transfer can cause negative transfer, motivating a similarity measure to quantify source-target relatedness and predict suitable sources.
C. Smart transfer learning
The paper selects source datasets for transfer learning by measuring inter-dataset similarity with Dynamic Time Warping. This strategy generally outperforms random source selection, although the nearest neighbor is not always optimal.
- The method computes a nearest neighbor for each target dataset and uses it as the selected source dataset without requiring prior domain knowledge.
- 71 of 85 datasets improved, 0 tied, and 14 worsened when the DTW-selected source replaced random selection.The comparison used random-selection accuracy averaged over 1000 iterations.
- The DTW-based selection significantly outperformed random source selection on average, with p < 10^-7 using the Wilcoxon signed-rank test.
- For DiatomSizeReduction and Wine, the second nearest neighbor improved accuracy from 3.3% to 46.7% and from 51.9% to 77.8%, respectively.These cases show that the nearest neighbor was not always the best source choice.
- The broader analysis of accuracy across source-dataset dissimilarity was limited to visually interpretable cases, with results for all 85 datasets provided on the companion web page.
D. Interesting case studies
Case studies show that transfer accuracy varies with source-dataset similarity across targets. The strongest gains occurred for small or structurally related datasets, while the best source could depend on latent shape or domain similarities.
- The case studies were selected to represent small training sets, relationships to shapelets, and variation in transfer-learning accuracy.
- ShapeletSim: For ShapeletSim, the top three neighbors reached 93% average accuracy versus 76% originally, and accuracy decreased for more distant datasets.ShapeletSim has only 20 training instances, and its closest source, RefrigerationDevices, shared similar spiky subsequences.
- HandOutlines: For HandOutlines, the most similar source produced about a 10% accuracy advantage over the most dissimilar source.The strongest sources, 50words and WordsSynonyms, also contain word outlines extracted from images.
- Meat: For Meat, the top three closest datasets reached almost 95% accuracy, with performance decreasing as source-target similarity decreased.The closest sources included Strawberry and Beef, both spectrograph datasets; 50words was the second closest.
VI. CONCLUSION
The conclusion reports that source-dataset choice materially affects CNN generalization and that DTW guidance favors similar source datasets. It recommends fine-tuning pretrained CNNs while identifying synthetic-data generation as future work.
- A bad source dataset can significantly impair generalization by trapping optimization in a local optimum, a phenomenon known as negative transfer learning.
- DTW-guided transfer works best when the CNN is fine-tuned from a model pretrained on a similar source dataset.The finding links classic DTW similarity with features learned by convolutional neural networks.
- Practitioners are encouraged to fine-tune pretrained models rather than train time-series classifiers from scratch.Properly designed CNNs can be adapted across datasets with varying lengths.
- Future work will use synthetic data generated with Weighted DTW Barycenter Averaging to further reduce neural-network overfitting.
- Existing source datasets that resemble a target dataset provide an appealing basis for transfer learning as data repositories expand.