Source-linked AI summary
InceptionTime: Finding AlexNet for Time Series Classification
Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F. Schmidt, Jonathan Weber, Geoffrey I. Webb, Lhassane Idoumghar, Pierre-Alain Muller, François Petitjean
TL;DR
TSC needs accurate classifiers that can handle growing datasets, but existing deep-learning methods have lagged in accuracy while highly accurate alternatives scale poorly. The paper introduces InceptionTime, an ensemble of Inception-based CNNs, and reports state-of-the-art TSC accuracy with substantially greater scalability than HIVE-COTE.
Problem
TSC requires accurate classification of increasingly large and long time-series datasets, while prior deep-learning approaches had lagged behind leading methods in accuracy.
Method
InceptionTime ensembles five randomly initialized one-dimensional Inception networks that learn from raw time series using multiple filter lengths.
Results
InceptionTime produces state-of-the-art results on the 85 UCR datasets and is two orders of magnitude faster than models such as HIVE-COTE.
Takeaways & Limitations
InceptionTime is competitive with HIVE-COTE in accuracy while deep-learning models can more readily leverage GPU parallelization.
Takeaways & Limitations
Residual connections should be studied independently for each case because TSC has relatively little labeled data compared with computer vision.
Abstract
from arXiv · showhide
This paper brings deep learning at the forefront of research into Time Series Classification (TSC). TSC is the area of machine learning tasked with the categorization (or labelling) of time series. The last few decades of work in this area have led to significant progress in the accuracy of classifiers, with the state of the art now represented by the HIVE-COTE algorithm. While extremely accurate, HIVE-COTE cannot be applied to many real-world datasets because of its high training time complexity in O(N2 * T4) for a dataset with N time series of length T. For example, it takes HIVE-COTE more than 8 days to learn from a small dataset with N = 1500 time series of short length T = 46. Meanwhile deep learning has received enormous attention because of its high accuracy and scalability. Recent approaches to deep learning for TSC have been scalable, but less accurate than HIVE-COTE. We introduce InceptionTime - an ensemble of deep Convolutional Neural Network (CNN) models, inspired by the Inception-v4 architecture. Our experiments show that InceptionTime is on par with HIVE-COTE in terms of accuracy while being much more scalable: not only can it learn from 1,500 time series in one hour but it can also learn from 8M time series in 13 hours, a quantity of data that is fully out of reach of HIVE-COTE.
1 Introduction
Time series datasets are growing in both length and quantity, increasing the need for classifiers that can operate at scale. InceptionTime addresses this gap with an ensemble designed for accurate and scalable TSC.
- Time series data is expanding across industries in both length and quantity, increasing dependence on automatic classification at scale.
- TSC must harness temporal information in signals, although its state-of-the-art algorithms differ substantially from those in computer vision.
- Deep learning is established in computer vision but only recently gained popularity in TSC, where ResNet became a state-of-the-art neural-network architecture.
- InceptionTime is a novel deep-learning ensemble that achieves state-of-the-art UCR accuracy while scaling beyond its strongest competitor.
- Each of five InceptionTime models uses cascaded Inception modules with multiple filter lengths to extract features from short and long time series.
- The paper also analyzes depth, filter length, filter count, bottlenecks, and residual connections to explain architectural choices.
2 Related work
Prior TSC research includes distance-based, frequency-based, shapelet, ensemble, and deep-learning approaches, but many accurate methods have limited scalability. Inception-based one-dimensional CNNs offer an end-to-end alternative for raw time series.
- TSC algorithms include distance-based, frequency-based, shapelet, ensemble, and deep-learning families.
- Whole series: Elastic Ensemble achieves state-of-the-art accuracy, but some parameter searches have training complexity O(N^2 · T^3), limiting use on large datasets.
- Shapelets: Shapelet Transform has training complexity O(N^2 · T^4), giving it little potential to scale to large datasets.
- Ensembles: COTE improves accuracy by ensembling classifiers over different time-series representations, while hierarchical voting further improves ensemble decisions.
- Deep learning: Early CNN architectures for domain-agnostic TSC borrowed image-recognition modules, but progressive pooling hindered accuracy by discarding information.
- Deep learning: The proposed Inception-based network learns end-to-end from raw time series with multiple convolution lengths, including filters ten times longer than recent image-recognition Inception variants.
3 InceptionTime: an accurate and scalable time series classifier
InceptionTime is an ensemble of five randomly initialized Inception networks for time series classification. Its modules combine multiscale filters, bottlenecks, residual connections, and receptive-field analysis to extract features across temporal resolutions.
- Inception Network: a novel architecture for TSC: Residual shortcuts transfer each block’s input directly to the next block, allowing direct gradient flow and mitigating vanishing gradients.The network uses two residual blocks, each composed of Inception modules, followed by Global Average Pooling.
- Inception Network: a novel architecture for TSC: The bottleneck reduces an M-dimensional multivariate time series to m ≪ M dimensions, lowering model complexity and mitigating overfitting on small datasets.It uses m filters of length 1 with stride 1.
- Inception Network: a novel architecture for TSC: Each Inception module applies multiple filters simultaneously, including filters of varying lengths, to extract features from short and long time series.The proposed module uses filter lengths 10, 20, and 40, with MaxPooling included.
- InceptionTime: a neural network ensemble for TSC: InceptionTime consists of five randomly initialized Inception networks whose predictions are averaged to form the ensemble output.The ensemble addresses variability from random initialization and stochastic optimization.
- Receptive field: For temporal data, the receptive field measures a network’s maximum one-dimensional field of view and is used to reason about detecting longer patterns.The paper hypothesizes that longer time series patterns require larger receptive fields.
4 Experimental setup
The experiments use a controllable synthetic time series classification dataset alongside the UCR archive. Deep networks are trained on GPU hardware, and comparisons use median accuracy and nonparametric statistical tests.
- Synthetic dataset: The synthetic dataset varies time series length, class count, and class distribution while generating instances from noise with class-specific injected patterns.Patterns have amplitude 1.0 and length equal to 10% of the time series.
- Benchmark evaluation: The UCR archive is used to evaluate global accuracy and computational complexity because it is the largest publicly available archive for time series classification.The passage identifies UCR as the benchmark archive for these evaluations.
- Computational setup: Deep neural networks are trained using a remote cluster of more than 60 GPUs, while a Quadro P6000 is used to compute model training time.The cluster includes GTX 1080 Ti and Tesla K20, K40, and K80 GPUs.
- Statistical analysis: Comparisons with published state-of-the-art results use median test accuracy across runs and Friedman testing followed by Wilcoxon signed-rank post-hoc analysis.This procedure follows recommendations cited by the authors.
- Reproducibility: The source code and pretrained deep learning models are publicly available to support verification and transfer learning.The repository is provided as a companion to the experiments.
5 Experiments: InceptionTime
Across the 85 UCR datasets, InceptionTime reaches competitive accuracy with HIVE-COTE while offering substantially better scalability and outperforming ResNet(5). The experiments also examine ensemble size and show that five individual networks are sufficient without significant improvement from larger ensembles.
- Accuracy: InceptionTime reaches competitive accuracy with HIVE-COTE on the 85 UCR datasets, sharing its critical-difference clique.HIVE-COTE is an ensemble of 37 TSC algorithms, whereas InceptionTime benefits from trivial GPU parallelization.
- Accuracy: 40/6/39 Win/Tie/Loss favors InceptionTime over HIVE-COTE, although the difference is not statistically significant.The largest observed underperformance occurs on the Wine and Beef datasets.
- Scalability: InceptionTime’s training complexity increases almost linearly with series length, while HIVE-COTE is almost two orders of magnitude slower on InlineSkate.This comparison used exponential resampling of the InlineSkate dataset.
- Scalability: InceptionTime is an order of magnitude faster than HIVE-COTE as training-set size grows on the approximately one-million-series SITS dataset.The SITS series have length 46 and belong to 24 possible land-use classes.
- Scalability: InceptionTime accuracy continues to increase with larger training sets, while HIVE-COTE would take 100 times longer to run.This result further evaluates scalability on the SITS dataset.
- Neural-network comparison: 54/8/23 Win/Tie/Loss favors InceptionTime over ResNet(5), with p-value < 0.01.ResNet(5) is an ensemble of five ResNet networks, and similar results hold with its original batch size.
- Ensemble analysis: No significant improvement appears when the ensemble contains x ≥5 individual networks, motivating the choice of five classifiers.The tested ensemble sizes are x ∈ {1, 2, 5, 10, 20, 30}.
6 Architectural Hyperparameter study
The study examines how InceptionTime’s architectural and training hyperparameters affect accuracy, runtime, complexity, and overfitting. Results show that receptive-field choices depend on series length and data availability, while some components offer complexity benefits without significant accuracy changes.
- Batch size: Batch size 64 provides a small performance advantage over alternatives, with no significant differences across 85 UCR datasets.The comparison uses a critical difference diagram evaluated on the UCR archive.
- Bottleneck and residual connections: Bottleneck layers do not significantly change accuracy, but nearly halve the parameters learned by InceptionTime.The Wilcoxon Signed-Rank test reports p-value > 0.1; the authors therefore retain the bottleneck for complexity reduction.
- Bottleneck and residual connections: Residual connections have minimal overall accuracy impact, yet removing them significantly reduces accuracy on ShapeletSim because of severe overfitting.Across the 85 UCR datasets, the effect is minimal with p-value > 0.2.
- Depth: Increasing depth does not necessarily improve performance on small-training-set datasets, although sufficient depth can provide a receptive field long enough for good results without overfitting.Deeper networks can overfit the UCR datasets, while shallower networks may suffer from smaller receptive fields.
- Filter length: Longer filters improve learning of longer patterns when enough training data mitigates overfitting, but can reduce accuracy on some datasets by increasing parameters.The study finds that longer time series require longer filters or receptive fields, while long filters can overfit small datasets.
- Number of filters: Datasets with more classes require more filters, but naively increasing filter count drastically raises complexity and can eventually cause overfitting.The wider InceptionTime variant contains four times as many parameters as the original implementation.
- Sensitivity analysis: Replacing each selected hyperparameter with its second-best value produces only a minimally and non-significantly lower ranking than the default configuration.The post-hoc test fails to reject the null hypothesis, with p-value ≈0.71.
7 Conclusion
The paper introduces InceptionTime to address the gap between scalable deep learning and the stronger experimental and architectural development of neural networks for image recognition. It reports state-of-the-art TSC results and scalability, while identifying further exploration of multivariate architectural advances as future work.
- Conclusion: InceptionTime produces state-of-the-art results on the 85 UCR archive datasets and is two orders of magnitude faster than models such as HIVE-COTE.The speedup is reported as consistent across Big Data TSC repositories and longer time series with high sampling rates.
- Conclusion: The paper investigates CNN hyperparameters using simulated data and frames the analysis around the receptive field for TSC.The study includes architectural designs with long filters beyond standard image-based practices.
- Conclusion: Applying InceptionTime to multivariate TSC and incorporating ongoing computer-vision advances in neural-network architecture remain areas for further exploration.The paper states that extension to multivariate data is straightforward but that additional architectural advances should be explored.