PaperlayerView exact revision on arXiv

Source-linked AI summary

HIVE-COTE 2.0: a new meta ensemble for time series classification

Matthew Middlehurst, James Large, Michael Flynn, Jason Lines, Aaron Bostrom, Anthony Bagnall

arXiv:2104.07551v1cs.LG

TL;DR

HIVE-COTE 2.0 addresses the need for accurate time series classifiers by redesigning HIVE-COTE around heterogeneous representations and adding new component ensembles. It significantly outperforms previous best methods on both univariate and multivariate classification problems, while retaining scalability limits on very large datasets.

  • Problem

    Time series classification requires predicting discrete targets from time series, motivating reproducible comparison and accurate algorithms across diverse applications.

  • Method

    HIVE-COTE 2.0 combines Shapelet Transform Classifier, Arsenal, Temporal Dictionary Ensemble, and Diverse Representation Canonical Interval Forest using accuracy-weighted probability estimates.

  • Results

    HIVE-COTE 2.0 significantly outperforms previous best methods on both univariate and multivariate time series classification problems.

  • Takeaways & Limitations

    HC2’s performance reflects complementary discriminatory features across data domains, with each component making a significant contribution to the ensemble.

  • Takeaways & Limitations

    HC2 does not scale well to very large problems, where build times can become excessive for thousands of long series.

Abstract

from arXiv · show

The Hierarchical Vote Collective of Transformation-based Ensembles (HIVE-COTE) is a heterogeneous meta ensemble for time series classification. HIVE-COTE forms its ensemble from classifiers of multiple domains, including phase-independent shapelets, bag-of-words based dictionaries and phase-dependent intervals. Since it was first proposed in 2016, the algorithm has remained state of the art for accuracy on the UCR time series classification archive. Over time it has been incrementally updated, culminating in its current state, HIVE-COTE 1.0. During this time a number of algorithms have been proposed which match the accuracy of HIVE-COTE. We propose comprehensive changes to the HIVE-COTE algorithm which significantly improve its accuracy and usability, presenting this upgrade as HIVE-COTE 2.0. We introduce two novel classifiers, the Temporal Dictionary Ensemble (TDE) and Diverse Representation Canonical Interval Forest (DrCIF), which replace existing ensemble members. Additionally, we introduce the Arsenal, an ensemble of ROCKET classifiers as a new HIVE-COTE 2.0 constituent. We demonstrate that HIVE-COTE 2.0 is significantly more accurate than the current state of the art on 112 univariate UCR archive datasets and 26 multivariate UEA archive datasets.

1 Introduction

HIVE-COTE 2.0 updates a heterogeneous ensemble for time series classification with new components and usability features. It is reported as significantly more accurate than leading alternatives on univariate and multivariate benchmarks.

  • Motivation: Time series classification predicts a discrete target variable from a possibly multivariate time series.Applications include seizure detection, earthquake monitoring, insect classification, and predictive maintenance.
  • Contribution: HC2 combines TDE, DrCIF, Arsenal, and STC, with each component representing a different data representation.The components cover dictionary, interval, convolutional, and shapelet-based approaches.
  • Results: HC2 is significantly more accurate than HC1, ROCKET, InceptionTime, and CIF on the UEA multivariate archive.The paper describes HC2 as significantly more accurate than these algorithms on multivariate time series classification.
  • Analysis: Each HC2 component contributes significantly to the ensemble overall, despite substantial performance variability across datasets.The study also finds that a simple CAWPE weighted structure performs as well as more complex stacking and selection schemes.
  • Usability: HC2 remains computationally expensive on large problems, although contracted runs generally converge quickly to reasonable results.The paper presents contraction as a way to obtain an estimate within a controlled runtime.

2 Background

The background frames time series classification through its data representations and reviews major algorithm families and HIVE-COTE’s evolution. It positions HC2 within comparisons involving heterogeneous ensembles, tree ensembles, deep learning, and fast convolutional methods.

  • Problem Setting: Time series classification uses equal-length univariate or multivariate series to predict discrete class labels.Multivariate cases contain d-dimensional vectors observed over m time points.
  • Representations: Distance, dictionary, interval, and shapelet methods categorize algorithms by the data representation they exploit.Dictionary methods use recurring symbolic patterns, interval methods derive temporal features, and shapelet methods find phase-independent discriminatory subseries.
  • Approaches: Current TSC approaches include modular heterogeneous ensembles, tree-based homogeneous ensembles, deep learning algorithms, and single-representation methods.These categories differ in whether representations are combined across modules, embedded in trees or networks, or used individually.
  • HIVE-COTE Evolution: HIVE-COTE evolved from five representation-specific constituent ensembles into HC1, which retained accuracy while requiring orders of magnitude less time.The original HIVE-COTE prioritized accuracy without concern for computational resources, while HC1 targeted utility and scalability.
  • Related Methods: InceptionTime, ROCKET, TS-CHIEF, and multivariate variants of ROCKET, InceptionTime, and CIF form important comparison approaches.ROCKET is described as a fast convolutional method with state-of-the-art accuracy, while TS-CHIEF embeds multiple representation types in trees.

3 HIVE-COTE 2.0 (HC2)

HIVE-COTE 2.0 updates the ensemble with four independently trained components, standardized accuracy estimation, and probability-weighted aggregation. It adds new dictionary, interval, and ROCKET-based capabilities while improving usability through contracting and checkpointing.

  • Ensemble structure: HC2 replaces three HC1 classifiers with Shapelet Transform Classifier, Arsenal, TDE, and DrCIF components.Each component is trained independently and produces class-probability estimates plus an accuracy estimate for unseen data.
  • Ensemble structure: CAWPE constructs a tilted class-probability distribution by exponentiating component accuracy estimates with α = 4 by default.The controller weights each module’s class probabilities according to estimated predictive quality.
  • Accuracy estimation: HC2 standardizes train-data accuracy estimation through a hybrid approach because out-of-bag estimates are acceptable but full-data bagged classifiers are more accurate.The hybrid approach avoids relying solely on bagged models built from resampled data.
  • Usability: HC2 supports concurrent component training, time contracts, checkpointing, and multivariate time series classification.Under a time contract, each component builds as many base classifiers as possible within the allotted time, but very large data can still exceed short contracts.
  • Temporal Dictionary Ensemble: TDE represents series with SFA-derived word and bigram histograms, spatial pyramids, histogram intersection, and diverse parameterized 1-NN classifiers.Its candidates are evaluated with LOOCV, while later parameter choices use a Gaussian-process regressor; the retained ensemble uses exponential accuracy weights.
  • Temporal Dictionary Ensemble: TDE significantly outperforms WEASEL and S-BOSS, which in turn outperform BOSS and cBOSS on the reported dictionary-classifier comparison.The comparison covers 106 UCR datasets.
  • Diverse Representation Canonical Interval Forest: DrCIF builds diverse information-gain trees from intervals across raw, differenced, and periodogram representations using a 29-feature candidate pool.The pool combines seven summary statistics with catch22 features, while trees receive different intervals and feature subsets.
  • The Arsenal: Arsenal uses an ensemble of smaller ROCKET classifiers to provide more useful probabilities for HC2, though it builds more slowly than ROCKET.The unaltered Arsenal improves HC2 significantly, whereas Arsenal itself does not improve default ROCKET accuracy.

4 Experimental Structure

The experiments evaluate HIVE-COTE 2.0 on selected univariate and multivariate archives using repeated resampling, multiple performance metrics, and statistical tests. Classifier configurations and computing environments are specified to support reproducibility.

  • Datasets and resampling: Experiments use 112 UCR datasets and 26 equal-length UEA datasets, averaging performance over 30 resamples.UCR datasets with unequal lengths, missing values, and Fungi are excluded; all 26 selected UEA datasets are used.
  • Classifier configurations: Table 2 defines classifier configurations in terms of series length m, dimensionality d, and DrCIF representation lengths r_m.These parameters organize the experimental settings across classifiers and data types.
  • Computing setup: Non-deep-learning experiments use Java tsml implementations, while deep-learning experiments use sktime-dl on specified HPC and GPU resources.tsml jobs run on one CPU core with seven-day and 500GB limits; GPU jobs use 12GB cards without a time limit.
  • Evaluation measures: Performance is evaluated with accuracy, AUC, and negative log-likelihood to assess predictions, rankings, and probability estimates.For multiclass problems, one-vs-many AUC is averaged using class-frequency weighting.
  • Statistical analysis: Statistical comparisons use pairwise Wilcoxon signed-rank tests and critical difference diagrams with Holm-corrected cliques.The procedure supports comparisons between two classifiers and among multiple classifiers across datasets.

5 Results

HIVE-COTE 2.0 significantly outperforms established classifiers across univariate and multivariate benchmarks, while its accuracy, probability quality, and computational cost differ from those of individual components and baselines.

  • Univariate results: HC2 is significantly better than the state of the art on 112 UCR equal-length datasets across accuracy, negative log-likelihood, and AUROC.Critical difference diagrams compare HC2, its four components, and four benchmark algorithms.
  • Univariate results: HC2 consistently outperforms HC1, with average accuracy more than 1% higher and lower performance variance between the two ensembles.Differences from the other classifiers show greater variation, particularly for ROCKET.
  • Runtime and memory: Runtime comparisons are conditioned by hardware and threading differences, including GPU execution for InceptionTime and single-thread restrictions for ROCKET and HC2.Memory measurements are approximate, and threaded execution would trade higher memory for lower runtime.
  • Runtime and memory: ROCKET trains all 112 datasets in under 3 hours, while HC2 is faster than HC1 and TS-CHIEF scales substantially worse on the slowest problems.TS-CHIEF takes ten times longer than HC2 on five particularly slow datasets.
  • Multivariate results: HC2 is significantly better than DTW-D, ROCKET, HC1, and CIF on all three metrics across 26 multivariate UEA datasets.These results support HC2 as a new state of the art for multivariate time series classification.

6 Inside HC2: An Ablative Study

The ablation study examines HC2’s error estimation, component contributions, and ensemble schemes. Results support the full four-component CAWPE ensemble as accurate and useful despite estimation and runtime considerations.

  • Error estimation: Separate bagging models provide an order-of-magnitude speedup over cross-validation for DrCIF and Arsenal error estimation.Using out-of-bag estimates directly reduced unseen-data accuracy for every module, so separate estimation models were retained.
  • Error estimation: HC2 has the lowest average deviation and MSE deviation between estimated and observed test accuracy.The ensemble compensates for small component-level biases through averaging and reused component estimates.
  • Error estimation: 94 of 112 datasets would lead train estimates to choose HC2 correctly over ROCKET.This comparison uses the ratio of train-based estimates and test-set performance in a Texas Sharpshooter plot.
  • Component ablation: The full four-component HC2 is significantly more accurate than every other component combination.The result demonstrates that each component contributes to the overall ensemble.
  • Ensembling Methods: CAWPE over HC2’s heterogeneous components is significantly better on average than the alternative ensemble schemes.Train-based selection performs significantly worse than HC2, while combining representations is better than selecting the best component on 31 datasets.

7 HC2 Usability

HC2 provides reproducible implementations and controllable runtime, but sequential training is slower than alternatives and large experiments can be constrained by time and memory.

  • Implementation: HC2 is available in open-source Java and Python toolkits with reproducible datasets and experiment instructions.The implementations are provided through tsml and sktime.
  • Runtime control: HC2 is slower than current state-of-the-art methods when run sequentially, particularly ROCKET.The authors do not recommend HC2 where models must be trained extremely quickly.
  • Runtime control: A four-hour HC2 contract achieves 98% of final accuracy, while a 12-hour contract achieves 99% on the longest-running problems.Contracting lets users specify an approximate maximum runtime.
  • Checkpointing: Checkpointing enables training continuation and accuracy assessment at saved states under imposed runtime limits.The mechanism periodically saves model state and permits reloading for continued training or testing.
  • Accuracy over time: DrCIF, STC, and TDE reach 80% of their achieved accuracy in less than half of recorded training time.Their accuracy rises quickly with added constituents before diminishing returns; Arsenal instead shows more erratic changes.
  • Accuracy over time: Only Arsenal on FruitFlies completed among eight large-dataset combinations.Other experiments were limited by a seven-day testing cap or Arsenal’s 700 GB memory requirement on InsectSounds.

8 Conclusion

HC2 combines four diverse classifiers to capture discriminatory features across data domains, achieving higher accuracy than prior methods on univariate and multivariate problems. Its main limitation is scalability, while contracting and open-source implementations improve practical usability.

  • Conclusion: HC2 significantly outperforms the previous best in accuracy on both univariate and multivariate time series classification problems.The ensemble contains four classifiers designed to capture different discriminatory features.
  • Conclusion: HC2 is better than any individual constituent, and each component makes a significant contribution to overall performance.The authors attribute this strength to discriminatory features occurring in multiple data domains.
  • Limitations: HC2 does not scale well to very large problems, where full builds can take weeks.The conclusion notes that current state-of-the-art methods, including ROCKET, have similar usability limitations at scale.
  • Future work: Further improvements could target STC, component threading, smarter contracting, uncertainty in train estimates, and instance-specific weights.These are identified as areas for future HC2 development.
  • Usability: HC2 remains available in two open-source toolkits and supports approximate maximum runtimes through contracting.The experiments are described as easily reproducible.
Loading 2104.07551v1…