PaperlayerView exact revision on arXiv

Source-linked AI summary

Bake off redux: a review and experimental evaluation of recent time series classification algorithms

Matthew Middlehurst, Patrick Schäfer, Anthony Bagnall

arXiv:2304.13029v3cs.LG

TL;DR

The paper asks whether conclusions from the earlier TSC bake off still hold as archives and algorithms have expanded, and whether comparisons generalise beyond heavily reused benchmark datasets. It revisits category-based benchmarking on the 112-problem UCR archive and 30 new univariate datasets, finding MR-Hydra and HC2 generally strongest, with their relative advantage depending on dataset characteristics.

  • Problem

    The earlier bake off and UCR archive no longer covered the field’s expanded algorithms and datasets, leaving updated and less development-biased evidence needed.

  • Method

    The study groups algorithms into eight representation-based categories, compares category leaders and newer methods on 112 UCR problems, and evaluates selected algorithms on 30 new univariate datasets.

  • Results

    MR-Hydra and HC2 generally perform best; HC2 is significantly better on the current UCR archive, while the difference is less observable on the 30 new problems.

  • Takeaways & Limitations

    Hybrid algorithms remain stronger on average than single-domain approaches on the UCR archive, while performance differences vary with imbalance, train-set size, class balance, and series length.

  • Takeaways & Limitations

    The study is restricted to univariate classification, and deep-learning research commonly risks biased model selection by repeatedly optimizing on test data.

Abstract

from arXiv · show

In 2017, a research paper compared 18 Time Series Classification (TSC) algorithms on 85 datasets from the University of California, Riverside (UCR) archive. This study, commonly referred to as a `bake off', identified that only nine algorithms performed significantly better than the Dynamic Time Warping (DTW) and Rotation Forest benchmarks that were used. The study categorised each algorithm by the type of feature they extract from time series data, forming a taxonomy of five main algorithm types. This categorisation of algorithms alongside the provision of code and accessible results for reproducibility has helped fuel an increase in popularity of the TSC field. Over six years have passed since this bake off, the UCR archive has expanded to 112 datasets and there have been a large number of new algorithms proposed. We revisit the bake off, seeing how each of the proposed categories have advanced since the original publication, and evaluate the performance of newer algorithms against the previous best-of-category using an expanded UCR archive. We extend the taxonomy to include three new categories to reflect recent developments. Alongside the originally proposed distance, interval, shapelet, dictionary and hybrid based algorithms, we compare newer convolution and feature based algorithms as well as deep learning approaches. We introduce 30 classification datasets either recently donated to the archive or reformatted to the TSC format, and use these to further evaluate the best performing algorithm from each category. Overall, we find that two recently proposed algorithms, Hydra+MultiROCKET and HIVE-COTEv2, perform significantly better than other approaches on both the current and new TSC problems.

1 Introduction

The paper revisits time series classification as the field has expanded across domains, archives, toolkits, and algorithmic approaches. It compares newer methods with earlier bake-off results using explicit selection criteria and an expanded experimental scope.

  • Motivation: TSC maps ordered real-valued observations to discrete response variables, covering univariate and multivariate series across many application domains.Examples include whale-call classification from audio and human activity recognition from motion coordinates.
  • Motivation: The original bake off and renewed archives helped increase interest in TSC algorithms and applications.Open-source toolkits such as aeon and tslearn further made comparison and reproduction easier.
  • Contributions: The review places new TSC algorithms in the original bake-off context and compares their performance on current UCR datasets.It also analyses factors associated with performance and discusses the merits of different approaches.
  • Contributions: The study releases 30 new univariate datasets and evaluates the best algorithm from each category on them.The datasets were donated through the TSC GitHub repository or assembled from other sources and reformatted for TSC.
  • Study design: Algorithms were selected using publication quality, prior evaluation on UCR/UEA data, and accessible source-code criteria.The code had to be usable or readily wrappable in the Java or Python environments used by the study.

2 Definitions and Terminology

This section defines time series, datasets, subseries, sliding windows, convolutions, and dilated subseries used throughout the study. It distinguishes univariate from multivariate observations and describes how transformations produce features.

  • Core objects: A time series is an ordered sequence of m data points, while a dataset is a collection of n labelled time series.Dataset labels come from a predefined discrete class set.
  • Core objects: A multivariate time series contains m observations, each represented by a vector of d aligned channels.The vector formulation makes simultaneous observation across dimensions explicit.
  • Subseries operations: A subseries is a contiguous segment of a time series, and a sliding window extracts all such segments of a chosen length.With increment 1, a series of length m has m − l + 1 windows of length l.
  • Convolution: Convolution applies a kernel through a sliding dot product to create an activation map that can supply summary features.The paper illustrates this with the kernel ω = [−1, 0, 1].
  • Dilation: A dilated subseries selects l non-contiguous points beginning at offset i and taking every d-th value.Dilation enlarges the receptive field while keeping the number of filter values constant and is used in convolution-, shapelet-, and dictionary-based models.
  • Data assumptions: The study commonly assumes equal sampling frequency, while unequal-length series may be padded with trailing zeros during preprocessing.Some algorithms can handle unequal lengths internally.

3 Experimental Procedure

The redux evaluates classifiers on 112 equal-length UCR problems and 30 additional datasets using repeated resampling, multiple performance measures, and statistical comparisons. The new datasets broaden series lengths but retain similar train sizes and class counts, with more sensor and motion data than older datasets.

  • Evaluation design: The original bake off evaluated 18-style classifier comparisons on 85 UCR datasets using 100 train/test resamples and average test accuracy.It progressed from standard classifiers to category comparisons, reproducibility checks, and hybrid methods.
  • Evaluation design: The redux compares 1-NN DTW, prior category leaders, and newer algorithms on 112 equal-length UCR problems.Using the same archive problems supports fair comparison because proposed approaches were evaluated on them or subsets.
  • Scope: The study restricts analysis to univariate classification because relatively few methods were adapted for multivariate data and multivariate evaluation is computationally costly.Many deep-learning algorithms were proposed solely for multivariate time series.
  • Resampling: 30 resamples are used instead of 100, with stratification, seeded resample IDs, and the original archive split retained as resample 0.The authors regard 30 resamples as sufficient to reduce rank changes from small test-accuracy fluctuations while improving computational feasibility.
  • Metrics and statistics: Classification accuracy is primary, supplemented by balanced accuracy, negative log likelihood, AUROC, and corrected pairwise Wilcoxon-based clique comparisons.Results also include pairwise plots, violin plots, accuracy tables, and heatmaps because critical-difference diagrams can hide relationships and effect sizes.
  • New datasets: The 112 archive problems may encode development-set overfitting, motivating a final comparison on newly gathered datasets.The new collection includes direct donations, discretised regression data, audio-classification data, and reformatted unequal-length or missing-value datasets.
  • New datasets: Some new datasets require padding unequal-length series or removing cases with missing values, while four TSER datasets require manual discretisation of continuous responses.The missing-value processing removes cases, and the discretised datasets show original labels alongside newly assigned classes.
  • New datasets: The 30 new datasets span broader series lengths than the existing 112 while having similar train-set sizes and class counts.The new collection is slightly biased toward sensor and motion data, whereas older datasets are dominated by image outlines.

4 Time Series Classification Algorithms

The paper extends the TSC taxonomy and evaluates newer algorithms across distance, feature, interval, shapelet, dictionary, convolution, and deep-learning approaches. Across categories, the strongest methods are generally ensembles or feature-extraction pipelines paired with linear classifiers.

  • The taxonomy is extended to reflect recent developments, while retaining distance, interval, shapelet, dictionary, combination, and model-based categories.
  • The most accurate classifiers generally use homogeneous or heterogeneous ensembles, or extract features before applying a linear ridge classifier.
  • Distance-based approaches: PF is over 2.5% better than competing distance-based approaches in test accuracy and balanced test accuracy, with higher AUROC and lower NLL.
  • Feature-based approaches: FreshPRINCE combines the full TSFresh feature set without FRESH selection and a Rotation Forest classifier; this pipeline is over 0.05 more accurate than using Random Forest.
  • Interval-based approaches: QUANT is selected as the best interval-based method because it is significantly faster than DrCIF and RSTSF, whose accuracies are highly similar.
  • Shapelet and dictionary approaches: RDST leads the shapelet classifiers by more than 1% over MrSQM, while WEASEL 2.0 improves dictionary performance by 4% over BOSS on average.
  • Convolution and deep-learning approaches: MultiROCKET-Hydra is the top convolutional method and is nearly 2% better than ROCKET, while H-InceptionTime is the selected deep-learning best in class.

5 Results

Across the 112 existing and 30 new UTSC problems, HC2 and MR-Hydra consistently emerge as the strongest classifiers, while comparisons also expose limitations of rank-based summaries.

  • HC2 and MR-Hydra are the top-performing algorithms on the 112 UCR problems.The analysis restricts comparison to the best classifier in each of eight categories and averages results over 30 resampled train/test splits.
  • HC2 is about 0.5% more accurate than MR-Hydra, over 6% more accurate than PF, and over 12% more accurate than 1NN-DTW.
  • On 30 new problems, the top accuracy clique contains MR-Hydra, HC2, RDST, QUANT, and FreshPRINCE.Balanced-accuracy results are similar.
  • Across all 142 problems, HC2 and MR-Hydra form a significantly better accuracy clique than the other six classifiers.
  • Critical difference diagrams can disagree with pairwise comparisons, so the study supplements them with multiple comparison matrices and other plots.The MCM reports pairwise accuracy differences, wins/draws/losses, and Wilcoxon p-values; examples show rank ordering can differ from pairwise wins.

6 Analysis

The analysis examines how classifier performance varies with dataset characteristics, training time, and performance stability. HC2 and MR-Hydra lead accuracy, while MR-Hydra offers a much faster alternative with a comparable average accuracy.

  • Performance by data characteristics: HC2 and MR-Hydra rank first or second across series lengths, training-set sizes, class counts, and most problem types.HC2 tends to perform better on longer series, whereas MR-Hydra performs better with larger training sets.
  • Runtime: MR-Hydra is on average as accurate as HC2 but orders of magnitude faster to train.HC2 is clearly much slower, partly because of configuration and implementation choices such as an unoptimised TDE component.
  • Runtime: QUANT is an order of magnitude faster and is recommended for very large problems if it scales accordingly.
  • Runtime: HC2 is preferable when smaller training sets or probability estimates and prediction orderings are required.
  • Performance stability: HC2 and MR-Hydra have tightly grouped performance deviations, while H-IT, FreshPrince, QUANT, and PF show wider variation.PF performs relatively poorly more often than it performs well.
  • Design factors: Using dilation and first differences is associated with significantly different average accuracy-rank distributions in the one-factor analyses.The analyses found no significant distribution difference for several other grouped design factors reported in the passage.

7 Conclusions

The redux study finds substantial progress in time series classification, with MR-Hydra and HC2 generally strongest, while performance depends on dataset characteristics and important scalability and scope limitations remain.

  • MR-Hydra and HC2 generally perform best across the 112 UCR problems and 30 newly introduced datasets.HC2 performs significantly better on the current UCR archive, while the difference is less observable on the 30 new problems.
  • HC2 performs worse than MR-Hydra on imbalanced data and larger training sets, but better with greater class balance, smaller training sets, and longer series.
  • The authors do not recommend always using MR-Hydra or HC2, although they suggest both as sensible starting points without prior information.QUANT is presented as a possible default for exploratory analysis because it is fast.
  • Deep learning has not produced a universally dominant TSC classifier, partly because TSC lacks one common underlying structure for neural networks to exploit.H-InceptionTime improves relative to other algorithms as training-set size increases but often performs poorly, reducing its overall performance.
  • Hybrid algorithms remain stronger on average than single-domain approaches, while dilation improves RDST and WEASEL 2.0.ROCKET and HIVE-COTE combine convolution or shapelet approaches with dictionary-based representations; linear-classifier pipelines are competitive and more scalable than ensembles but produce weaker probability estimates.
  • None except QUANT scale particularly well for large data, and multivariate TSC and unequal-length series remain insufficiently understood.The authors identify limited memory and runtime scalability, a lack of principled workflows, and many classifiers not designed for multivariate data.

Statements and Declarations

The authors report funding from the UK EPSRC and declare no conflicts of interest; the study used freely available data.

  • The work was supported by UK Engineering and Physical Sciences Research Council grant EP/W030756/1.
  • The authors declare no conflicts of interest or competing interests.
  • All data used were freely available for scientific use, and ethics approval was not applicable.

Appendix A Reproducibility

The reproducibility appendix documents accessible classifier tooling, experiment interfaces, data loading, evaluation outputs, and the algorithm tables used in the redux bake off.

  • Most evaluated algorithms are available in the aeon Python toolkit, which includes a classification module and is compatible with scikit-learn.
  • The aeon interface supports fitting classifiers, predicting labels, obtaining probabilities, and calculating accuracy.The example uses ROCKET because it is fast, while the same interface applies to other aeon classifiers.
  • tsml-eval v0.2.1 runs experiments with aeon and scikit-learn estimators and supports data loaded from .ts files or pre-loaded arrays.
  • The experiment functions produce results files, with notebooks documenting the results format and multi-dataset evaluation utilities.
  • The paper makes experiment results and parameter tables available through the accompanying website and publication directory.
  • The appendix provides tables listing the original bake-off algorithms, redux algorithms, and characteristics of the 40 evaluated approaches.

Appendix C Results

The appendix identifies the result tables and figures used to evaluate classifiers on new datasets and to compare algorithm characteristics and relationships.

  • Figure C1 provides a critical-difference diagram for all algorithms evaluated on the 112 UCR datasets.
  • Figures C2 and C3 provide additional MCM diagrams for the 30 new datasets and original 142 datasets using best-in-category algorithms.
  • Figures C4 and C5 show correlation diagrams for the algorithms.
  • Table C4 reports classifier accuracy on 30 new datasets averaged over 30 resamples.
  • The work was developed from experimentation conducted in 2022 and was finalised in 2023 before online publication on 19 April 2024.
  • The review process requested major revisions and a stronger focus on target audiences, take-away messages, techniques, and metrics such as scalability.

A.1 Reviewer 1

Reviewer 1 views the paper as an expanded update of the earlier TSC bake-off, but questions its novelty, selection biases, baseline choices, and experimental scope.

  • The study extends the prior bake-off with additional datasets and methods, including newer deep learning approaches.
  • No runtime or scalability experiments are presented, limiting assessment of practical algorithm costs.
  • The review identifies limited novelty because methods, datasets, authors, and conclusions overlap substantially with earlier studies.
  • Several datasets excluded for missing values or unequal lengths could instead be included through imputation, interpolation, or resampling.
  • The selected 1-NN-DTW baseline may overlook stronger distance measures, sliding measures, kernel methods, and non-deep representation learning.
  • The separation of convolution-based methods from deep neural network methods is not clearly justified.

A.2 Reviewer 2

Reviewer 2 finds the methodology useful and comprehensive but asks for clearer audience targeting, terminology, presentation, and explanations of algorithmic behavior.

  • The paper offers a comprehensive comparison of recent TSC techniques, taxonomies, and datasets with an emphasis on reproducibility.
  • The intended audience is unclear, while uneven explanations of technical jargon may hinder both novice and expert readers.
  • Frequent acronyms and the glossary’s appendix placement create a readability challenge.
  • Additional illustrations and diagrams could clarify complex concepts and improve accessibility.
  • The review should provide a clearer, more targeted overview of the TSC state of the art for readers with basic time-series knowledge.
  • The paper would benefit from clearer English, shorter sentences, stronger structure, and deeper analysis of why algorithms differ across datasets.

Appendix B Data Mining and Knowledge Discovery

The paper moved from Machine Learning to Data Mining and Knowledge Discovery, where the revision process concluded with acceptance after one review round.

  • The authors withdrew from Machine Learning on 5/1/2024 and submitted the paper to Data Mining and Knowledge Discovery.
  • The new venue returned reviews on 25/2/24, received a revision on 15/3/24, and accepted the paper on 29/3/24.
  • The editor invited a revised manuscript accompanied by a point-by-point response.

B.1.1 Reviewer 1

Reviewer 1 considers the paper a timely and valuable update of TSC research, while raising concerns about dataset interpretation, category overlap, and generalization across benchmark sets.

  • The paper updates the earlier bake-off with post-publication algorithms, 30 new datasets, new experiments, and reproducibility materials.
  • The reviewer expects the work to benefit future TSC research because of its extensive data preparation, coding, experimentation, and comparative coverage.
  • The phrase “we contribute 30 new datasets” may misleadingly imply that the authors created all of them rather than formatting or receiving donations from other researchers.
  • Figure 9’s intuition about interval methods conflicts with Table 18, where Quant ranks worst on the spectro column.
  • MrSQM and MrSEQL may span shapelet and dictionary categories because they discretise time series similarly to dictionary methods.
  • Results across the 112, 30, and 142 datasets appear to generalize, but not strongly, with differences for MR-H and HC2.
  • The reviewer regards the refresh as timely because the data archive has expanded and the state of the art has advanced.

B.1.2 Reviewer 2

The paper revisits time series classification research through an expanded review and benchmark study, adding recent algorithms and datasets to the original bake-off framework. Review feedback largely recognizes its extensive experiments and practical value while identifying methodological and presentation concerns.

  • Evaluation strengths: Extensive resampling experiments, statistical comparisons, and efficiency analyses support broader conclusions about algorithm performance.These features were identified as strengths of the evaluation.
  • Reviewer concerns: Reviewers noted unclear terminology around dilation, convolution, and receptive fields, and ambiguity about the one- versus two-tailed Wilcoxon test.They also raised concerns about handling unequal-length datasets and other methodological inconsistencies.
  • Reviewer concerns: The review also questioned the consistency of the algorithm-selection criteria and the treatment of deep-learning evaluations.Specifically, the comments contrast criticism of subset evaluations with the paper’s own stated inclusion criterion.

B.1.3 Reviewer 3

Reviewers regarded the paper as a substantial continuation of the original bake-off, while asking for broader coverage, complexity analysis, and safeguards against biased cross-category comparisons. They also requested more discussion of dataset construction and related methods.

  • Contribution: The paper summarizes six years of TSC progress, introduces datasets, and broadens comparisons among recent methods.Reviewers described it as a thorough survey and significant contribution.
  • Coverage: Reviewers said the paper omits some eligible techniques and should connect newer methods more explicitly to earlier bake-off algorithms.They specifically requested broader hierarchical coverage and discussion of concepts such as motifs.
  • Evaluation design: Category-winner comparisons may bias results because multiple methods within one category could outperform the selected winner from another category.Reviewers also requested more complete time- and space-complexity analysis.
  • Presentation and data: Additional comments targeted confusing category terminology, missing convolution background, repeated figures, table readability, and limited discussion of bias in new datasets.These comments concern both exposition and interpretation of the experimental evidence.
  • Related work: The cited related work includes motif discovery, LSTM-FCN, wavelet-DTW attention, and other time-series classification methods.These references indicate areas reviewers considered relevant to the paper’s positioning.

B.1.4 Reviewer 4

Reviewer 4 questioned whether benchmark progress on UCR reflects real-world practice, where simple nearest-neighbor methods are often used, and raised several scope and reproducibility concerns. The review therefore asks the authors to distinguish genuine advances from benchmark-specific effects.

  • Real-world relevance: The review argues that many real TSC applications still use simple nearest-neighbor methods despite continual claims that sophisticated algorithms improve benchmark scores.It asks the authors to explain this discrepancy.
  • Interpretation: Possible explanations include overfitting to UCR, UCR’s limited realism, or genuine superiority of advanced methods over DTW nearest-neighbor classification.The reviewer presents these as competing hypotheses rather than established conclusions.
  • Reproducibility and scope: The review questions excluding SpatialBOSS because its runtime and memory requirements prevented evaluation on the full dataset collection.It argues that the technical exclusion requires stronger justification.
  • Dataset provenance: The review requests fuller acknowledgment of student work used to format the SharePriceIncrease dataset.This is a provenance concern about newly prepared data.
  • Coverage: The comments also question restricting deep-learning coverage and cite additional papers involving distance-based, transformer, industrial, and multivariate applications.These references broaden the methodological and application context considered by the reviewer.

B.2 Response to Reviewers

The responses clarify the paper’s benchmark scope, report broad performance patterns, and document added analyses and datasets. They also acknowledge limitations involving UCR generalization, missing values, excluded algorithms, category comparisons, and incomplete methodological coverage.

  • Scope and contributions: The revised study adds post-bake-off algorithms, 30 new univariate datasets, reproducible code and data, and expanded category comparisons.The authors position the work as a six-year update of the original bake-off.
  • Specific result: 82.71% average accuracy for QUANT exceeds 81.55% for HIVE-COTE 2.0 and 72.13% for MultiRocket Hydra on EthanolLevel.The authors caution that this example does not establish interval methods as best for all spectrogram problems.
  • Algorithm trends: Current methods commonly combine dilation, randomized hyperparameters, first differences, and ridge classifiers to obtain high accuracy with low training time.The associated trade-off is a larger feature-space memory footprint.
  • Comparative findings: Hybrid methods generally outperform single-domain approaches, while ridge-based pipelines can be competitive with ensembles and are generally more scalable.The authors note that these pipelines cannot produce good probability estimates.
  • Comparative findings: HC2 and MR-Hydra perform similarly on average, and pipeline methods tend toward lower average ranks even though the two top performers come from different model types.The authors added rank diagrams and factor analyses to clarify these patterns.
  • Scope limitations: Missing-value cases were removed because the evaluated implementations lacked missing-data handling, while SpatialBOSS was excluded because its cost would require omitting multiple datasets.These choices prioritize transparent full-archive comparison over broader algorithm inclusion.
  • Design factors: Dilation and differenced series are associated with significantly different rank distributions, although the ANOVA assumptions are not satisfied.No significant distribution difference was found for frequency, ensemble, or discretisation factors.
  • Scope limitations: The authors state that no algorithm can realistically claim to outperform InceptionTime and that UCR evaluations may not generalize fully to larger or different problems.They respond by adding 30 datasets, while declining extensive case studies and some additional methods.
Loading 2304.13029v3…