Source-linked AI summary
Benchmarking Multivariate Time Series Classification Algorithms
Alejandro Pasos Ruiz, Michael Flynn, Anthony Bagnall
TL;DR
MTSC involves classifying cases represented by multiple ordered real-valued series, but it has been studied less than univariate TSC. The paper benchmarks bespoke MTSC algorithms and dimension-independent ensembles across 26 equal-length UEA problems, finding independent HIVE-COTE most accurate while DTW remains competitive.
Problem
MTSC is common when multiple series share one label, yet it has received less attention and had fewer benchmarking resources than univariate TSC.
Method
The paper compares twelve bespoke MTSC algorithms and dimension-independent adaptations on 26 equal-length problems from the 30-problem UEA archive.
Results
Independent HIVE-COTE is the most accurate approach, while DTW remains competitive; MUSE performs as well as HIVE-COTE and significantly better than DTW.
Takeaways & Limitations
DTW remains a competitive MTSC benchmark, while HIVE-COTE and MUSE are strong alternatives despite MUSE’s substantial memory overhead.
Takeaways & Limitations
The evaluation covers only the 26 equal-length problems among the 30 datasets in the UEA MTSC archive, whose development was still ongoing.
Abstract
from arXiv · showhide
Time Series Classification (TSC) involved building predictive models for a discrete target variable from ordered, real valued, attributes. Over recent years, a new set of TSC algorithms have been developed which have made significant improvement over the previous state of the art. The main focus has been on univariate TSC, i.e. the problem where each case has a single series and a class label. In reality, it is more common to encounter multivariate TSC (MTSC) problems where multiple series are associated with a single label. Despite this, much less consideration has been given to MTSC than the univariate case. The UEA archive of 30 MTSC problems released in 2018 has made comparison of algorithms easier. We review recently proposed bespoke MTSC algorithms based on deep learning, shapelets and bag of words approaches. The simplest approach to MTSC is to ensemble univariate classifiers over the multivariate dimensions. We compare the bespoke algorithms to these dimension independent approaches on the 26 of the 30 MTSC archive problems where the data are all of equal length. We demonstrate that the independent ensemble of HIVE-COTE classifiers is the most accurate, but that, unlike with univariate classification, dynamic time warping is still competitive at MTSC.
1 Introduction
MTSC assigns one label to multiple time series, yet has received less attention than univariate TSC despite being common in applications. Using the UEA archive, the paper benchmarks bespoke MTSC methods against dimension-independent adaptations and finds HIVE-COTE strongest overall while DTW remains competitive.
- Motivation: MTSC associates multiple series with one class label, unlike univariate TSC, which uses a single series per case.Human activity recognition, ECG, EEG, MEG diagnosis, and systems monitoring are inherently multivariate.
- Motivation: MTSC has received substantially less consideration than univariate TSC, partly because few multivariate resources existed until recently.The UEA archive of 30 MTSC problems released in 2018 made algorithm comparison easier.
- Benchmark: The benchmark compares recently proposed bespoke MTSC algorithms with simple dimension-independent adaptations on 26 equal-length UEA problems.The comparison includes twelve algorithms spanning deep learning, bag of words, and shapelet-based approaches.
- Findings: The independent ensemble of HIVE-COTE classifiers is the most accurate approach evaluated, while dynamic time warping remains competitive for MTSC.The study reports that two algorithms significantly improve benchmark accuracy, while DTW remains difficult to beat.
2 Background
MTSC extends time series classification to multiple series per case, where discriminative information may lie in cross-dimensional interactions. The paper reviews distance-, shapelet-, dictionary-, and deep-learning approaches, including independent adaptations of univariate classifiers.
- Problem: MTSC represents each case with multiple ordered series sharing one class label, rather than a single series.Its added complexity includes interactions between dimensions and the larger volume of data.
- Distance-based approaches: DTW adapts to MTSC through independent warping, which computes a separate distance matrix per dimension and sums the resulting distances.The paper contrasts this with dependent approaches that measure Euclidean distances between multivariate vectors.
- Distance-based approaches: The adaptive DTW strategy selects dependent or independent distance using a threshold estimated by cross-validation on training data.This approach is intended to determine whether a problem is independent or dependent.
- Independent approaches: Independent ensembles adapt univariate classifiers by ignoring relationships among dimensions; HIVE-COTE is presented as an accurate univariate baseline for MTSC.This provides a straightforward baseline for comparison with bespoke multivariate algorithms.
- Bespoke MTSC approaches: The reviewed bespoke methods include generalized random shapelet forests, MUSE, MLCN, and TapNet.MUSE selects discriminatory words across dimensions before χ2 feature selection and logistic-regression classification; MLCN combines LSTM and FCN components with multivariate dependency modeling.
- Bespoke MTSC approaches: TapNet combines random dimension permutation, multivariate time-series encoding, and attentional prototype learning to integrate traditional and deep-learning strengths.Its permutation module forms groups of randomly selected dimensions before encoding.
3 Data
The study uses 26 equal-length multivariate datasets from the UEA archive for its experiments, with dataset characteristics summarized in Table 1.
- The experiments are restricted to the 26 UEA multivariate datasets whose series are all equal length.
- Table 1 summarizes the main characteristics of each dataset used in the experimentation.
4 Results
The results compare multivariate classifiers under completion and accuracy constraints across UEA problems. HIVE-COTE performs strongest overall, while DTW remains competitive and resource limitations affect some methods.
- 4 Results: TapNet completed 23 datasets, while gRSF tuning completed only 9 of 26 experiments within seven days and MUSE failed on four problems under 500GB memory.
- 4.1 Comparison of Classifiers: HIVE-COTE is the only classifier significantly better than the DTWD benchmark on the 26-problem comparison.
- 4.1 Comparison of Classifiers: MUSE joins HIVE-COTE in the top clique and is significantly more accurate than DTWD on the 21 datasets completed by all classifiers.
- 4.1 Comparison of Classifiers: TapNet and MLCN perform worse than the shapelet algorithms and are no better than any of the three DTW algorithms.
- 4.1 Comparison of Classifiers: Normalizing each dimension makes no significant difference to HIVE-COTE or the DTW classifiers, although all three DTW variants are worse on average.
- 4.2 Analysis by Problem: HIVE-COTE beats DTWD on 19 of 26 problems, with a median test-accuracy difference of 2.46%.
- 4.2 Analysis by Problem: On six problems, few classifiers outperform majority-class prediction, and AF and SWJ are often no better than random guessing.
5 Conclusions
MTSC remains less mature than univariate TSC: DTW is still competitive, while newer methods show mixed trade-offs in accuracy, memory, and reliability. The UEA archive is still developing, leaving scope for future algorithms that model cross-dimension interactions.
- MTSC is at an earlier stage of development than univariate TSC, with DTW still competitive against newer alternatives.
- HIVE-COTE is significantly better than DTW, although the DTW comparison uses a full warping window and unnormalised data.The authors report that tuning the warping window and normalising data could improve DTW.
- MUSE performs as well as HIVE-COTE and significantly better than DTW, but averages 26GB of memory versus 1.6GB for HIVE-COTE.The memory comparison covers 21 problems requiring less than 500GB of memory, and MUSE overhead grows with problem dimensions.
- Deep learning methods were not consistently state-of-the-art and sometimes completely failed in these experiments.TapNet and MLCN were reported to be in the middle of the pack overall.
- The UEA MTSC archive is fairly new and needs more development, with additional datasets planned.
- Future algorithms that explicitly model interactions between dimensions are expected to outperform the algorithms evaluated here.