PaperlayerView exact revision on arXiv

Source-linked AI summary

HYDRA: Competing convolutional kernels for fast and accurate time series classification

Angus Dempster, Daniel F. Schmidt, Geoffrey I. Webb

arXiv:2203.13652v1cs.LG

TL;DR

Dictionary methods count symbolic patterns, while ROCKET transforms time series with convolutional kernels. HYDRA combines these ideas through competing grouped kernels and an adjustable balance between dictionary-like and ROCKET-like behavior. It is faster and more accurate than the leading existing dictionary methods, with a scope boundary for very long series noted by the authors.

  • Problem

    Dictionary methods count symbolic patterns while ROCKET-family methods transform time series with convolutional kernels, and the paper examines how these apparently different approaches can be connected for dictionary-like classification.

    Source: §1 · p. 2

  • Method

    HYDRA groups random convolutional kernels and counts the closest-matching kernel in each group at each timepoint, combining convolutional transforms with dictionary-style pattern counts.

    Source: §1 · p. 2

  • Results

    Hydra is faster and more accurate than the most accurate existing dictionary methods, including TDE and MrSQM, and is competitive in accuracy with several leading time-series classification methods.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Adjusting the number of kernels per group moves Hydra between models resembling dictionary methods and Rocket, providing a basis for dictionary-like classification with convolutional kernels.

    Source: §1 · pp. 2–3

    For very long time series, the logarithmically growing kernel count may make subsampling dilations, reducing groups, or downsampling desirable; the experiments did not use these restrictions.

    Source: §3.2 · p. 9

Abstract

from arXiv · show

We demonstrate a simple connection between dictionary methods for time series classification, which involve extracting and counting symbolic patterns in time series, and methods based on transforming input time series using convolutional kernels, namely ROCKET and its variants. We show that by adjusting a single hyperparameter it is possible to move by degrees between models resembling dictionary methods and models resembling ROCKET. We present HYDRA, a simple, fast, and accurate dictionary method for time series classification using competing convolutional kernels, combining key aspects of both ROCKET and conventional dictionary methods. HYDRA is faster and more accurate than the most accurate existing dictionary methods, and can be combined with ROCKET and its variants to further improve the accuracy of these methods.

1 Introduction

HYDRA connects dictionary methods, which extract and count symbolic patterns, with ROCKET-style convolutional transforms. It uses grouped, competing random kernels to perform dictionary-like classification, with grouping controlling the balance between dictionary-like and ROCKET-like behavior.

1 Introduction

  • HYDRA combines dictionary-style pattern grouping and count-based classification with random convolutional kernels, providing a basis for fast, accurate dictionary-like classification.The paper demonstrates a connection between dictionary methods and ROCKET through a single adjustable hyperparameter.Source: §1 · p. 2
  • Hydra groups random convolutional kernels and counts, at each timepoint, the kernel in each group that best matches the input, making the kernels compete to be counted.Unlike Rocket, this grouped counting treats each kernel as a dictionary pattern.Source: §1 · p. 2
  • With multiple kernels per group, Hydra resembles conventional dictionary methods more closely; as k approaches 1, it approaches Rocket, but the most Rocket-like variant is not the most accurate Hydra variant.By default, Hydra uses 64 groups with 8 kernels per group.Source: Figure 2 · p. 2, §1 · p. 3
  • Approximately 28 minutes lets Hydra train and test on 106 UCR datasets using one CPU core, versus approximately 1 hour 41 minutes for MrSQM and 22 hours for TDE; Hydra is faster and more accurate than the strongest dictionary methods.The comparison also includes cBOSS, WEASEL, and S-BOSS, which take approximately 4 hours, more than 24 hours, and more than 6 days, respectively.Source: Figure 1 · p. 1, §1 · p. 3

2 Related Work

Dictionary methods classify time series using counts of symbolic patterns, while ROCKET-family methods transform series with convolutional kernels and learned classifiers. HYDRA links these approaches; the related work also situates it among accurate but computationally demanding alternatives.

2.1 Dictionary Methods

  • Dictionary methods generally slide a window, smooth or approximate its values, quantise them into symbolic patterns, and classify using pattern counts; these methods can differ substantially because their approximation and quantisation choices differ.BOSS and related methods use Symbolic Fourier Approximation; cBOSS is faster than BOSS at approximately the same accuracy, while S-BOSS is more accurate at considerable computational expense.Source: §2.1 · p. 4

2.2 Rocket, MiniRocket, and MultiRocket

  • ROCKET transforms series with random convolutional kernels and uses the resulting features to train a linear classifier; MiniRocket removes randomness, while MultiRocket adds pooling operations and first-order-difference transforms.The paper identifies dilation and PPV pooling as two important ROCKET factors for accuracy; MiniRocket is significantly faster than comparable-accuracy methods, and MultiRocket is several orders of magnitude faster than HC2 at broadly similar accuracy.Source: §2.2 · pp. 5–6
  • HYDRA builds a connection between ROCKET and dictionary methods that supports fast, accurate dictionary-like classification using convolutional kernels.Other prominent approaches include InceptionTime, an ensemble of deep convolutional neural networks, and TS-CHIEF, an ensemble of decision trees using distance measures, intervals, and spectral splitting criteria.Source: §2.2 · p. 6, §2.3 · p. 6

2.3 Other State of the Art

  • Hydra completes training and testing on 112 UCR datasets in approximately 36 minutes, compared with approximately 3 hours for MrSQM, almost 2 days for DrCIF, more than 3 days for TDE and InceptionTime, approximately 2 weeks for HC2, and several weeks for TS-CHIEF.The paper characterizes these highly accurate methods as computationally complex.Source: §2.3 · p. 6

3 Method

HYDRA transforms time series with random convolutional kernels arranged in groups, then counts the best-matching kernels at each timepoint. This grouped competition makes the transform dictionary-like and lets its resemblance to dictionary methods or ROCKET vary with the group configuration.

3.1 Overview

  • Hydra convolves each input with random kernels organized into groups and counts, at each timepoint, the closest-matching kernel in each group; the resulting counts train a linear classifier.The kernels compete within each group, which Hydra treats as a dictionary of patterns.Source: Figure 3 · p. 7, §3.1 · p. 7
  • Hydra uses patterns that approximate the input and counts them as features, but represents those patterns with random convolutional kernels rather than the smoothing and quantisation typical of dictionary methods.The paper notes that randomisation does not necessarily make Hydra's patterns more approximate than conventional dictionary patterns.Source: §3.1 · p. 7
  • Grouping kernels lets Hydra move between dictionary-like and Rocket-like models and has a decisive influence on accuracy; with k=1, Hydra essentially becomes Rocket, with qualifications, but that variant is not the most accurate.HYDRA also departs from ROCKET by moving away from PPV features.Source: §3.1 · p. 7
  • Hyperparameters were selected using the same 40 UCR development datasets and default splits used by Dempster et al. to avoid overfitting the full archive.The overview lists kernel characteristics, counting scheme, group and kernel counts, and inclusion of first-order differences as key hyperparameters.Source: §3.1 · p. 8

3.2 Kernels

HYDRA combines random convolutional kernels with grouped, dictionary-like counting. Kernel organization and response-counting choices determine how its features behave.

3.2 Kernels

  • HYDRA uses length-9 kernels with weights drawn from N(0, 1), simplified dilations, and padding that supports variable-length series.Unlike Rocket variants, it has no fixed feature count or bias values; kernel weights are normalized to avoid spurious competition from mean or magnitude differences.Source: §3.2 · pp. 8–9
  • Because total kernels grow logarithmically with series length, very long series may warrant dilation subsampling, fewer groups, or downsampling.The experiments did not impose these restrictions.Source: §3.2 · p. 9
  • Resampling a smaller kernel set could improve speed and scalability, but initial experiments suggest accuracy decreases as unique kernels decrease.The paper also leaves open whether larger sets of nonrandom MiniRocket-like kernels could match HYDRA's random kernels, and whether fixed or stochastic resampling would work.Source: §3.2 · p. 9

3.3.1 Maximum vs Minimum Response

  • Counting both maximum and minimum responses preserves more convolution-output information without extra transform work and improves accuracy.The minimum response is an alternative closest match, equivalent to matching with the inverted kernel set −W.Source: §3.3.1 · p. 10

3.3.2 Hard (argmax) vs Soft (max) Counting

  • HYDRA defaults to soft counts for maximum responses and hard counts for minimum responses; using both is more accurate than either alone.Hard counts increment the winning kernel's count, while soft counts accumulate its response.Source: §3.3.2 · p. 10

3.3.3 Clipping

  • Clipping improves accuracy for k = 1, has little or no effect for k ≥ 2, and does not change the optimal g and k.With k = 1, clipping yields features equivalent to PPV and global average pooling.Source: §3.3.3 · p. 11

3.4 Number of Groups (g) & Kernels Per Group (k)

The number of kernels per group controls how closely HYDRA resembles dictionary methods or Rocket, both in its features and in its kernel approximation of the input.

3.4 Number of Groups (g) & Kernels Per Group (k)

  • The group count g, equivalently kernels per group k, controls both feature resemblance to dictionary methods or Rocket and the approximation level.With multiple kernels per group, HYDRA produces counts like typical dictionary methods; as k approaches one, its features change qualitatively.Source: §3.4 · p. 11, §3.4.1 · p. 11

3.4.1 Features

  • At k = 1 with clipping, hard counts yield PPV features and soft counts yield global average pooling features.These features represent average kernel responses rather than dictionary-style counts.Source: §3.4.1 · pp. 11–12
  • The most Rocket-like variant, k = 1, is not HYDRA's most accurate variant, whether or not clipping is used.PPV and global average pooling features occur only at k = 1 and only with clipped convolution outputs.Source: §3.4.1 · p. 12

3.4.2 Approximation

  • More kernels per group lower approximation: more kernels compete, increasing the chance that a counted kernel closely matches the input.Figure 4 contrasts k = 8, with high approximation, and k = 512, with low approximation.Source: §3.4.2 · pp. 12–13, Figure 4 · p. 12

3.5 First-Order Difference

HYDRA applies its transform to the original series and its first-order difference, and uses ridge or logistic regression classifiers.

3.5 First-Order Difference

  • Assigning half of the groups to the first-order difference adds it without extra features or computation and increases accuracy for every HYDRA variant.By default, 32 of 64 groups operate on the difference.Source: §3.5 · p. 13

3.6 Classifier

  • HYDRA uses ridge regression or, for datasets with more than approximately 10,000 training examples, logistic regression.The transform runs in batches by default to limit peak memory and can be integrated into minibatch updates for logistic regression trained with stochastic gradient descent or similar.Source: §3.6 · p. 13

4 Experiments

The experiments compare Hydra with leading classifiers on the UCR archive, test it on larger UCR datasets, and examine sensitivity to kernel grouping, counting, first-order differences, and clipping. Overall, Hydra is competitive with leading methods, and its strongest reported configuration uses 64 groups of 8 kernels.

4.1 UCR Archive

  • On 112 UCR datasets, Hydra outperformed TDE, MrSQM, and DrCIF, achieved broadly similar accuracy to Rocket, InceptionTime, and TS-CHIEF, while MultiRocket+Hydra was not significantly less accurate than HC2.Figure 5 reports mean rank over 30 resamples; the prose describes the average accuracy comparisons.Source: Figure 5 · p. 14, §4.1 · p. 14
  • Hydra beat TDE on 73 datasets and MrSQM on 69; its default configuration beat the single-group variant on 95 datasets.Hydra was less accurate on 37 datasets than TDE and 42 than MrSQM; the default configuration was less accurate than the single-group variant on 15.Source: Figure 7 · p. 15, §4.1 · pp. 15–16, Figure 8 · p. 15
  • HC2's accuracy advantage over Hydra came at 500× computational cost; Hydra used less than 0.2% of HC2's compute time, with accuracy differences under 5% on 90% of datasets.The authors also report that Hydra uses less than 1% of TDE's compute time.Source: §4.1 · p. 16

4.2 Larger Datasets

  • On the three largest UCR datasets, Hydra was more accurate than Rocket or MiniRocket, and MultiRocket+Hydra produced the highest accuracy on all three.Hydra was combined with the other transforms by concatenating their generated features.Source: Table 2 · p. 17, Table 1 · p. 17, §4.2 · p. 17
  • MiniRocket was fastest on the larger datasets, while MultiRocket+Hydra remained marginally faster than Rocket on all three.The authors caution that absolute timings are somewhat arbitrary because methods can use different core counts or GPUs.Source: §4.2 · p. 17

4.3.1 Number of Groups (g) & Kernels per Group (k)

  • With the tested kernel budgets, k=8/g=64 ranked highest; around k=8 was optimal except at very small budgets, while gains beyond 512 kernels were small relative to added cost.High-g/low-k variants were most accurate, favoring the combination of many approximate matches; increasing total kernels generally helped, but improvements diminished.Source: Figure 10 · p. 18, §4.3.1 · pp. 18–19

4.3.2 Method of Counting and First-Order Difference

  • At k=8/g=64 without clipping, counting both maximum and minimum responses, combining soft and hard counting, and including first-order differences each improved accuracy.These variants were compared using mean rank.Source: Figure 11 · p. 19, §4.3.2 · p. 19

4.3.3 Clipping

  • At the optimal k/g setting with first-order differences included, clipping had very little effect on accuracy.The comparison used mean rank across variants with and without clipping.Source: Figure 12 · p. 20, §4.3.3 · p. 20

5 Conclusion

The conclusion presents HYDRA as a convolutional-kernel dictionary method connecting dictionary approaches with ROCKET. It reports faster and more accurate performance than existing dictionary methods and plans further variants and applications.

5 Conclusion

  • HYDRA connects dictionary methods that extract and count symbolic patterns with convolutional-kernel approaches such as ROCKET.The paper presents HYDRA as a fast, highly accurate dictionary method incorporating aspects of both approaches.Source: §5 · p. 20
  • HYDRA is faster and more accurate than the most accurate existing dictionary methods, and consistently improves ROCKET and its variants when combined with them.Source: §5 · p. 20
  • Future work will explore deterministic HYDRA variants, more sophisticated combinations with other methods, and applications to multivariate time series.Source: §5 · p. 20

A Additional Results

The additional results examine accuracy, sparsity, and mean-rank behavior across HYDRA configurations, including changes to group and kernel settings.

A Additional Results

  • Figure 13 compares pairwise accuracy for configurations with k × g = 512 versus k × g = 256.Source: Figure 13 · p. 24
  • Figure 14 plots accuracy standard deviation against k and g, while Figure 15 plots sparsity against k with g = 1.Source: Figure 14 · p. 24, Figure 15 · p. 25
  • Figures 16–18 report mean rank across k and g for max+min and soft+hard settings, varying differencing and clipping.Figure 16 uses no differencing and no clipping; Figure 17 uses no differencing with clipping; Figure 18 uses differencing with clipping.Source: Figure 16 · p. 25, Figure 17 · p. 26, Figure 18 · p. 26

B.1 Training Time

Tables 3 and 4 present training times for HYDRA configurations on the three largest datasets.

B.1 Training Time

  • Table 3 reports training times for MiniRocket, MultiRocket, Rocket, and Hydra on the three largest datasets.Source: Table 3 · p. 27
  • Together, Tables 3 and 4 cover standalone Rocket-family methods and their Hydra combinations on the three largest datasets.Source: Table 3 · p. 27, Table 4 · p. 27
  • Table 4 reports training times for MiniRocket+Hydra, MultiRocket+Hydra, and Rocket+Hydra on the three largest datasets.Source: Table 4 · p. 27

B.2 Training Details

The training procedure uses batched transforms and shared optimization settings, with five runs per dataset and eight cores per run.

B.2 Training Details

  • Transforms use batches of 4,096 examples, divided into minibatches for training, and cached features avoid repeating transforms across epochs.Source: §B.2 · p. 27
  • Training uses Adam with a 2,048-example validation set, minibatches of 256, and an initial learning rate of 10^-4.The learning rate is halved after 50 updates without validation-loss improvement, and training stops after 100 such updates, but only after the first epoch.Source: §B.2 · p. 27
  • Experiments use five runs per dataset for each method, report mean results over those runs, and allocate eight cores per dataset per run.Source: §B.2 · p. 27

Related papers

Loading 2203.13652v1…