PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting WEASEL 2.0: Reproduction, Sensitivity, and an Adaptive Ensemble-Size Rule

Cian Higgins, Gerard Carrigan, Pinar Sungu Isiacik, Georgiana Ifrim

arXiv:2608.18021v1cs.LG

TL;DR

WEASEL 2.0’s ensemble-size and window-size heuristics lacked empirical justification. Through reproduction, sensitivity analysis, and an adaptive ensemble-size rule, the study finds robust accuracy with lower resource use, especially for long series.

  • Problem

    WEASEL 2.0 contains under-justified heuristics for maximum ensemble and window sizes, motivating empirical tests of their robustness and refinement.

  • Method

    The study independently reproduces WEASEL 2.0 on UCR datasets, perturbs four design choices, and derives an ensemble-size rule from series length and class count.

  • Results

    The reproduction matches published results; three design choices are robust, while the adaptive rule cuts peak memory and fit time with zero median accuracy change.

  • Takeaways & Limitations

    The adaptive rule offers resource savings while preserving accuracy, with benefits concentrated on long-series datasets.

  • Takeaways & Limitations

    Competitor comparisons are slightly limited by out-of-memory issues and differences in the compared classifier set.

Abstract

from arXiv · show

WEASEL 2.0 is a dictionary-based time series classifier that combines dilated sliding windows with a randomised hyperparameter ensemble and a fixed-size dense feature representation. Two of its hyperparameter choices, the maximum ensemble size and the maximum window size, are specified by simple thresholding rules whose chosen thresholds are not empirically justified in the original paper. In this work we reproduce WEASEL 2.0 on 114 UCR datasets, achieving a mean accuracy of 0.865 and median of 0.928, closely matching the published values (Wilcoxon signed-rank, p = 0.655). We then test the sensitivity of four design choices: the downstream classifier, the absence of feature weighting, the maximum window-size rule, and the maximum ensemble-size rule. The first three are robust to perturbation. The fourth is over-provisioned for long-series datasets, motivating an adaptive rule that sets the maximum ensemble size from series length and number of classes. Evaluated on fixed-length datasets, the adaptive rule reduces peak fit memory by a median of 37 MB (mean 395 MB) and fit time by a median of 0.4 s (mean 4 s), with a median accuracy change of 0% (mean -0.11%). Memory and time savings concentrate on long-series datasets where the original rule allocates the largest ensemble size.

1 Introduction

This work independently reproduces WEASEL 2.0, tests four under-justified design choices, and proposes an adaptive ensemble-size rule to reduce resource use without accuracy loss. WEASEL 2.0 addresses excessive memory consumption and sensitivity to minor subsequence changes through dilation mapping and a fixed-size dense representation.

  • Motivation: WEASEL 2.0 addresses excessive memory consumption and sensitivity to minor subsequence changes using dilation mapping and a fixed-size dense feature representation.Its representation combines a compact dictionary, variance-based Fourier coefficient selection, and randomized hyperparameter configurations.
  • Study aims: The study independently reproduces WEASEL 2.0 and examines whether four under-justified design choices are robust to perturbation.It also evaluates whether the ensemble-size heuristic can be refined to reduce resource use without accuracy loss.
  • Contributions: Three of four under-justified design choices are empirically robust in the sensitivity analysis.The analysis tests four design choices, while the contribution specifically identifies three as robust.
  • Contributions: The ensemble-size heuristic is over-provisioned for long-series datasets, motivating an adaptive rule based on series length and class count.The proposed rule reduces peak fit memory at minimal accuracy cost.

2 Related Work

Dictionary-based time-series classifiers evolved from BOSS toward scalable designs such as WEASEL 2.0, which uses dilation, fixed-size dictionaries, and randomized ensembling to control memory. Existing UCR bake-offs compare classifiers but do not test internal design sensitivity or independent reproducibility, motivating this paper’s empirical examination of WEASEL 2.0 and refinement of one heuristic.

  • Dictionary-based classifiers: WEASEL 2.0 combines dilation, a fixed-size dictionary, and randomized hyperparameter ensembling to produce a dense representation with controlled memory consumption.These design choices address the large and sometimes unpredictable memory usage exposed by earlier dictionary-based classifiers such as BOSS.
  • Related TSC designs: ROCKET, MiniRocket, MultiRocket, R-DST, and Hydra exemplify TSC research that refines established designs to improve accuracy, runtime, or scalability.The cited methods respectively use random convolutional kernels, kernel refinements, random dilation for shapelets, or hybrid kernel-dictionary approaches.
  • Evaluation protocols: UCR bake-off studies standardize cross-classifier evaluation under a unified protocol, including WEASEL 2.0 and kernel- and shapelet-based methods.However, these studies do not examine individual classifiers’ internal design sensitivity or independently verify reported numbers.
  • Paper’s approach: This paper instead treats WEASEL 2.0 as a fixed reference to assess component contributions, heuristic-threshold robustness, reproducibility, and data-aware resource refinement.The stated goal is to reduce resource use without significant accuracy loss by refining one existing heuristic rather than proposing a new method design.

3 Reproducing WEASEL 2.0

The reproduction closely matches the published WEASEL 2.0 results across 114 UCR datasets and supports its accuracy and runtime claims. WEASEL 2.0 achieves the highest median accuracy among reproduced classifiers, with moderate training time and no significant difference from several top methods.

  • Reproduction setup: The reproduction used the GitHub implementation through aeon, with RidgeClassifierCV as WEASEL 2.0’s downstream linear model, and compared seven additional classifiers.The additional classifiers were MultiRocket, MiniRocket, ROCKET, R-DST, Hydra, WEASEL 1.0, and cBOSS.
  • Reproduction fidelity: 0.865 mean accuracy and 0.928 median accuracy closely match the published 0.863 and 0.927 across the 114 UCR datasets.A Wilcoxon signed-rank test found no statistically significant difference from the published results (p = 0.655).
  • Classifier comparison: 0.928 median accuracy was highest for WEASEL 2.0, while MultiRocket had the highest mean accuracy at 0.867.WEASEL 2.0 also recorded 80 wins versus 18 losses against WEASEL 1.0 and 81 wins versus 19 losses against cBOSS.
  • Runtime comparison: 0.57 h total fit time made WEASEL 2.0 slower than MiniRocket, Hydra, and ROCKET but substantially faster than cBOSS at 7.51 h.WEASEL 2.0 occupied the desirable upper-left region of the accuracy-versus-train-time scatter, combining high accuracy with moderate runtime.
  • Statistical comparison: WEASEL 2.0 showed no significant difference from four of five top methods in pairwise Wilcoxon tests, with p-values above 0.05.The methods were R-DST, MiniRocket, Hydra, and ROCKET.
  • Limitations: The variable-length datasets PickupGestureWiimoteZ (+12.0%) and ShakeGestureWiimoteZ (+8.0%) were clear reproduction outliers, and WEASEL 1.0 required one thread on three datasets because of out-of-memory errors.The reproduced critical difference diagram also produced two overlapping top-six cliques rather than the original single clique.

4 Sensitivity Analysis of Design Choices

The sensitivity analysis tests four empirically unjustified WEASEL 2.0 design choices: the downstream classifier, feature weighting, maximum window size, and maximum ensemble size. The first three are robust to perturbation, whereas the ensemble-size rule is over-provisioned for long-series datasets and motivates an adaptive rule.

  • Design choices: The study evaluates ridge classification, absent TF-IDF weighting, the maximum window-size heuristic, and the maximum ensemble-size heuristic.These are the four original choices identified as lacking empirical justification.
  • Overall sensitivity findings: The first three design choices are robust to perturbation, but the maximum ensemble-size rule is over-provisioned for long-series datasets and motivates an adaptive rule.The adaptive rule is presented in Section 5.
  • Feature weighting: −0.008 mean accuracy difference was observed for TF-IDF weighting versus the baseline, with p = 0.015.TF-IDF also significantly increased fit time, slightly decreased predict time, and increased memory usage by a mean of 43 MB (p = 0.038).
  • Feature weighting: +0.002 mean accuracy difference was observed for sublinear TF scaling, with p = 0.794.Fit time increased significantly, while the accuracy change was not statistically significant.
  • Maximum window size: p > 0.05 was obtained for accuracy differences between the baseline and all tested window sizes except wmax = 100 and wmax = 124, which were significantly worse.The sweep tested wmax ∈ {12, 24, 44, 84, 100, 124, 200} with ensemble size fixed at 50; increasing beyond 84 provided no accuracy benefit and could be harmful.
  • Maximum ensemble size: 62 of 114 datasets (54%) use the small-dataset ensemble branch, 14 (12%) use the short-series branch, and 38 (33%) use the default branch.The rule assigns rmax values of 50, 100, or 150 according to training-set size and series length, creating sharp transitions in feature counts.

5 An Adaptive Ensemble-Size Rule

The adaptive ensemble-size rule uses series length and class count to reduce over-provisioning while preserving larger ensembles for sensitive problems. Across 112 fixed-length UCR datasets, it reduced fit time and peak memory with near-zero median accuracy change, outperforming simple halving on mean memory savings and accuracy cost.

  • Rule design: Series length and number of classes were the primary drivers of ensemble sensitivity, whereas training-set size was excluded as an unreliable predictor.The adaptive rule selectively preserves larger ensembles for sensitive problem types and reduces them elsewhere.
  • Rule design: The rule uses n > 700 as the long-series threshold and prioritizes series length when both length and class-count conditions apply.The binary-class branch c ≤2 →25 reflects baseline accuracy with smaller feature counts; ensemble values 25, 50, and 75 equal 1/2, 1, and 3/2 of the default 50.
  • Results: −395 MB mean and −37 MB median peak-memory differences accompanied −4.0 s mean and −0.4 s median fit-time differences versus WEASEL 2.0 across 112 fixed-length datasets.Predict-time differences were −3.4 s mean and −0.3 s median.
  • Results: 0.11% mean accuracy reduction and 0.00% median change indicate that more than half of datasets showed no accuracy change under the adaptive rule.Accuracy was fully preserved across small training sets, short and medium series, and binary classification tasks.
  • Comparison: 395 MB versus 306 MB mean memory savings and −0.11% versus −0.22% mean accuracy costs make the adaptive rule preferable to simple halving.Halving the original 50/100/150 tiers to 25/50/75 produced statistically significant mean accuracy reduction of 0.22% and mean memory savings of −306 MB.

6 Conclusion

The reproduction supports WEASEL 2.0’s main benchmark claims, while sensitivity analysis finds three design choices robust and identifies the ensemble-size rule as over-provisioned for long series. An adaptive rule based on series length and class count preserves accuracy while reducing memory and fit time, with data-driven and multivariate extensions remaining open.

  • Conclusion: 114 UCR datasets reproduced WEASEL 2.0’s published performance within statistical noise, supporting its benchmark, comparability, and controlled-memory claims.The reproduction supports the claims that WEASEL 2.0 is the strongest dictionary classifier, statistically indistinguishable from the best kernel and shapelet methods, and memory-controlled.
  • Conclusion: Three design choices were empirically robust: the ridge classifier, absence of feature weighting, and maximum window-size heuristic.Replacing ridge with SGD or applying TF-IDF weighting degraded accuracy, while sublinear TF scaling was accuracy-neutral but increased fit time and memory; changing window-size tiers produced no significant accuracy gain.
  • Conclusion: The original maximum ensemble-size rule is over-provisioned for long-series datasets, where extra ensemble members require substantial memory for marginal accuracy returns.This motivates adapting ensemble size to dataset properties rather than retaining the original allocation behavior.
  • Conclusion: 37 MB median peak-fit-memory reduction and 0.4 s median fit-time reduction were achieved on 112 fixed-length datasets with the adaptive rule, while median accuracy change was zero.The rule selects rmax from series length and number of classes rather than training-set size; mean reductions were 395 MB and 4 s, respectively.
  • Conclusion: Two open extensions are selecting rmax from cross-validated accuracy during fitting and evaluating the rule in multivariate settings.A data-driven rule could capture properties beyond series length and class count, while multivariate savings may increase because memory demand grows with channel count.

A Reproduction Results

The reproduction evaluates WEASEL 2.0 across 114 UCR datasets, reporting per-dataset accuracy and fit time.

  • 114 UCR datasets are included in the WEASEL 2.0 reproduction results.
  • Accuracy and fit time (s) are reported for each dataset.

B Multi-Comparison Matrix

The reproduced results are presented in a multi-comparison matrix summarizing pairwise mean differences, win/tie/loss counts, and Wilcoxon p-values. Bold entries identify comparisons with p < 0.05.

  • B Multi-Comparison Matrix: The matrix reports pairwise mean differences between the reproduced results.It is described as a multi-comparison matrix of the reproduced results.
  • B Multi-Comparison Matrix: It also summarizes win, tie, and loss counts for each comparison.
  • B Multi-Comparison Matrix: Bold entries indicate Wilcoxon p-values below 0.05.The figure includes Wilcoxon p-values, and the accompanying note defines bold entries as p < 0.05.

C Adaptive Ensemble-Size Rule Results

The adaptive ensemble-size rule is evaluated on fixed-length UCR datasets, reporting per-dataset accuracy and fit time alongside the new variant’s results.

  • The adaptive ensemble-size rule variant is evaluated per dataset on fixed-length UCR datasets.
  • Accuracy and fit time (s) are the reported evaluation metrics.
Loading 2608.18021v1…