PaperlayerView exact revision on arXiv

Source-linked AI summary

The UCR Time Series Archive

Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, Eamonn Keogh

arXiv:1810.07758v2cs.LGstat.ML

TL;DR

The paper addresses how to expand and evaluate the heavily used UCR Time Series Archive while clarifying why reported improvements over its 1-NN baseline may be misattributed. It introduces the expansion, recommends more careful evaluation practices, and uses examples showing that simpler changes can match claimed gains. The archive’s coverage remains limited for privacy-sensitive real-world problems.

  • Problem

    The archive’s widespread use and reported improvements require clearer evaluation practices because papers may misattribute gains over the 1-NN baseline.

  • Method

    The paper expands the archive, explains its baseline and design choices, and offers practical guidance for evaluating algorithms and diagnosing claimed improvements.

  • Results

    0.05 error rate is achieved by both Haar wavelets and PAA with 32 coefficients on CBF, compared with 0.148 using raw data.

  • Takeaways & Limitations

    Reported improvements should be tested with ablations because simpler modifications may produce the same gains as a claimed specialized mechanism.

  • Takeaways & Limitations

    The archive contains no data sets that motivate privacy-preserving mining, limiting representation of privacy-sensitive domains.

Abstract

from arXiv · show

The UCR Time Series Archive - introduced in 2002, has become an important resource in the time series data mining community, with at least one thousand published papers making use of at least one data set from the archive. The original incarnation of the archive had sixteen data sets but since that time, it has gone through periodic expansions. The last expansion took place in the summer of 2015 when the archive grew from 45 to 85 data sets. This paper introduces and will focus on the new data expansion from 85 to 128 data sets. Beyond expanding this valuable resource, this paper offers pragmatic advice to anyone who may wish to evaluate a new algorithm on the archive. Finally, this paper makes a novel and yet actionable claim: of the hundreds of papers that show an improvement over the standard baseline (1-nearest neighbor classification), a large fraction may be mis-attributing the reasons for their improvement. Moreover, they may have been able to achieve the same improvement with a much simpler modification, requiring just a single line of code.

I. INTRODUCTION

The UCR Archive provides reproducible baselines for time-series classification, but parameter tuning and evaluation choices can make apparent performance unreliable. Its DTW baseline depends on the warping window, whose training-based selection may not predict holdout performance.

  • The archive was created to make progress and comparisons possible after early studies largely relied on single author-created artificial data sets.
  • 1-NN baselines classify each test exemplar using its closest training match under Euclidean distance or DTW.DTW distance varies with the warping-window parameter w.
  • For most problems, a warping window above 20% is unnecessary and likely adds computational burden.
  • Euclidean distance enforces one-to-one point matching, whereas unconstrained DTW permits monotonic one-to-many matching through the distance matrix.
  • The archive reports 1-NN Euclidean distance and constrained DTW with a learned warping window among its baseline classification results.
  • The learned warping window is selected by leave-one-out cross-validation on training data, but the best training value may not be best for testing.The figure shows examples where learned constraints predict unseen-data behavior closely and others where holdout accuracy does not track the prediction.

III. CRITICISMS OF THE UCR ARCHIVE

The paper addresses criticisms of the archive’s assumptions and data preparation. It explains its original design choices while acknowledging that normalization can be inappropriate in rare cases.

  • The authors organize criticisms around assumptions about data availability, pattern alignment, equal length, class structure, and archive documentation.
  • The archive originally began as a small personal project rather than a community-scale resource, so this release documents data sets more systematically.New data sets require a detailed donor description or a citable research publication.
  • The archive z-normalizes time series to remove offset and scaling, which can reduce sensitivity to changes such as camera distance or footwear.
  • The release acknowledges that normalization is ill-advised for some rare cases and provides raw data when possible.It also identifies whether donors normalized data beforehand when original raw data are unavailable.

D. The Individual Data Sets are Too Small

The archive spans data sets of substantially different sizes but is not representative of all real-world applications. Its fixed split supports reproducibility, while small samples can magnify numerical differences.

  • StarLightCurves has 1,000 training and 8,236 test objects, whereas Beef has 30 training and 30 test objects.The two data sets cover 3 and 5 classes, respectively.
  • The archive is biased toward data sets that investigators could easily obtain or create, including many image-derived sets.
  • None of the archive’s data sets motivates privacy-preserving mining, limiting coverage of domains where privacy is important.
  • A single fixed train/test split was chosen to enable exact reproducibility of deterministic benchmark results.The split does not prevent pooling the data and using k-fold cross-validation for more rigorous evaluation.

IV. HOW BAD IS CHERRY PICKING?

The section demonstrates how reporting results on selected archive subsets can make a poor idea appear effective. A full-archive evaluation and explicit subset justification are therefore important.

  • Subset experiments are difficult to assess when authors do not explain how the data sets were selected.Without a stated selection rationale, readers cannot exclude biased selection, even when the reported results are reproducible.
  • FQT won on 19 data sets but lost or drew on 66 of the full 85 when a win required at least 1% error reduction.
  • Testing only selected data sets can make an arbitrary poor idea appear useful or even excellent.The fictional FQT example reports wins on every selected data set, although the broader evaluation reverses that impression.
  • Space limitations are not a sufficient justification because critical-difference diagrams, scatter plots, and online spreadsheets can summarize or provide full results.
  • A subset can be defensible when researchers explicitly tie it to a practical constraint, such as the computational cost of a time-consuming algorithm.

V. BEST PRACTICES FOR USING THE ARCHIVE

The paper recommends tuning simple components of ED and DTW rather than treating untuned baselines as fixed opponents. Small changes such as smoothing, parameter tuning, or using k-nearest neighbors can materially affect results.

  • Beating DTW on some data sets is necessary but insufficient evidence for a new distance measure or classifier.DTW performance can often be improved through simple modifications, narrowing the apparent gap to more complex methods.
  • DTW’s warping-window parameter should be tuned more carefully because the quick-and-dirty setting can be suboptimal.Suggested alternatives include resampling or creating synthetic examples.
  • Smoothing the data, generalizing 1-nearest neighbor to k-nearest neighbors, and testing alternative DTW step patterns can improve classification performance.The paper also notes benefits from making DTW endpoint invariant on many data sets.
  • Figure 3 shows error rates as smoothing increases for CBF and as the number of nearest neighbors increases for FordB.The leave-one-out training error approximately predicts the best parameter in these examples.
  • An hour spent optimizing these simple choices could improve ED or DTW on at least several archive data sets.

A. Mis-attribution of Improvements: a Cautionary Tale

The paper cautions that reported gains may be attributed to sophisticated mechanisms when simpler preprocessing explains them. Wavelet and FQT examples show why ablations and full evaluations are needed to identify the source of improvement.

  • A substantial fraction of papers reporting improvements over UCR baselines may misattribute the source of those gains.The authors suggest that some improvements may instead reflect simple, low-cost modifications.
  • Wavelet-classification papers commonly attribute accuracy gains to wavelets’ multi-resolution or multi-scale properties.
  • The paper hypothesizes that wavelet representations may improve accuracy simply by smoothing the data implicitly.Figure 3 provides evidence that appropriate smoothing can produce improvements comparable to those claimed for wavelet methods.
  • For power-of-two dimensions, Haar wavelet approximation is logically identical to PAA, yielding identical distances, predictions, and error rates.
  • On CBF, Haar with 32 coefficients and PAA with 32 coefficients both achieved an error rate of 0.05, versus 0.148 for raw data.
  • Applying Haar or PAA to already smoothed data slightly worsened error from 0.055 to 0.057, consistent with over-smoothing.The paper uses this observation to cast doubt on attributing the improvement specifically to wavelet multi-resolution.
  • The authors argue that claimed mechanisms require stronger ablation evidence because adopters may otherwise pursue ineffective follow-up applications.If a simple modification yields the same gain, it may be preferable to a complex method even when it is less publishable.

B. How to Compare Classifiers

Classifier comparisons require explicit performance criteria and evaluation protocols. The paper recommends using appropriate metrics, transparent train/test procedures, repeated resampling when suitable, and safeguards against test-set overfitting and numerical instability.

  • Accuracy and balanced accuracy are considered sufficient to assess predictive power across multiple diverse data sets.Accuracy is the proportion correctly classified; balanced accuracy averages accuracy across classes.
  • All classifier construction, including model selection and parameter setting, should use training data before accuracy is assessed on test data.
  • A single train/test split can invite explicit or implicit test-set parameter optimization, motivating complete code transparency.
  • Small data sets can magnify tiny numerical differences into apparent performance gaps as large as 6% on a single split.The paper attributes such discrepancies to rounding and occasional single-case classification changes.
  • Repeated resampling and averaged test accuracy can largely overcome single-split problems, but resampling should preserve split sizes and class distributions and use identical samples across algorithms.The default split should be the first resample; some data sets should not be resampled when observations must remain separated by experimental unit.
  • The archive’s deterministic single split remains a useful reproducibility sanity check before more sophisticated evaluation.

3) The choice of significance tests:

The paper combines rank-based statistical testing with a broader checklist for credible archive evaluations. It also expands the archive to address community requests while recognizing that some needs exceed its intended scope.

  • 3) The choice of significance tests:: Multiple classifiers are compared using per-data-set ranks, a Friedman test, pairwise Wilcoxon signed-rank tests, and Holm-corrected cliques.Critical difference diagrams summarize these comparisons.
  • 3) The choice of significance tests:: In Fig. 4, Rotation Forest has the best rank but is not statistically better than CVDTW and MPdist within its clique.
  • C. A Checklist: The proposed checklist asks researchers to test all data sets or explain exclusions, optimize parameters on training data, test claimed properties through ablations, and share code.
  • VI. THE NEW ARCHIVE: Community input identified variable-length, multivariate, provenance-documented, imbalanced, very small-training-set, and larger data sets as desired archive additions.
  • VI. THE NEW ARCHIVE: The authors state that the archive addresses some perceived problems and community requests while retaining a bounded scope rather than supplying every requested resource.

A. General Introduction

The Fall 2018 UCR expansion increases the archive from 85 to 128 data sets and documents the new release’s construction choices. The paper uses a Texas Sharpshooter plot to compare Euclidean distance with constrained DTW across all 128 data sets.

  • The Fall 2018 expansion increases the UCR Archive from 85 to 128 data sets.
  • The new archive includes provenance where possible and often makes test sets larger than training sets to reflect the expense of labeled training data.
  • Fig. 5 compares 1-NN Euclidean distance with 1-NN constrained DTW on 128 data sets.
  • The figure’s results are described as strongly supporting constrained DTW over Euclidean distance for most problems, although overlapping numbers are difficult to read.
  • The Texas Sharpshooter plot contrasts expected accuracy gain from training data with actual accuracy gain from test data to expose unsupported claims based on selective data sets.For this comparison, constrained DTW subsumes Euclidean distance as a special case.

B. Some Notes on the Old Archive

The old archive section documents corrections to fourteen reversed train/test splits, identifies twelve data sets solved by at least one algorithm, and revisits GunPoint through new recordings and classification tasks.

  • Archive corrections: Fourteen data sets had their train/test splits reversed in the 2015 expansion and were restored to match their original release and the UEA Archive.The corrected splits also match those used in a recent influential survey.
  • Solved data sets: Twelve old-archive data sets had achieved 100% accuracy with at least one algorithm.These data sets are listed as “solved” in Table II.
  • GunPoint data sets: The authors recreated GunPoint recordings in 2018 with the same two original actors, fifteen years after the 2003 recordings.The new release combines old and new recordings to create additional benchmark problems.
  • GunPoint data sets: The expanded GunPoint data span two actors, two behaviors, and two recording years, enabling tasks that vary invariance to actor, year, action, or actor characteristics.The original task separates Gun from Point actions, while the new tasks include age-span, old-versus-young, and male-versus-female classifications.

2) GesturePebble data sets:

The GesturePebble data sets repurpose smartwatch accelerometer recordings to test gesture classification under subject-overlap and subject-independent splits, while the expansion also includes ethanol-level spectrograph data with bottle-disjoint splits.

  • Data handling: GesturePebble includes variable-length exemplars padded with NaNs to the longest series, without prescribing how algorithms should process them.Algorithms may instead handle variable lengths directly, truncate, or renormalize the data.
  • GesturePebble construction: GesturePebble recordings use a 3-axis accelerometer on a Pebble smartwatch, with four participants repeating six gestures across two sessions.The gestures and recording setup are illustrated in the accompanying figures.
  • GesturePebble construction: Two GesturePebble data sets retain only the z-axis accelerometer channel.The source recordings provide three channels or attributes.
  • GesturePebble splits: GesturePebbleZ1 trains on the first session and tests on the second, so every subject appears in both sets.This creates a session-based split with subject overlap.
  • GesturePebble splits: GesturePebbleZ2 trains on two subjects and tests on two different subjects, making it more difficult because test subjects have unseen gait and posture characteristics.The baseline results confirm the intended difficulty difference.
  • EthanolLevel data set: The ethanol-level data set contains twenty bottle types, four alcohol classes—35%, 38%, 40%, and 45%—and 1,751 observations per spectrograph.The train/test construction keeps each bottle type out of both sets, so merging and resampling are inappropriate.

4) InternalBleeding data sets:

The InternalBleeding data sets derive three vital-sign classification problems from pig recordings before and after induced injury, while the section also describes challenging phase misalignment and polymorphic classes.

  • InternalBleeding construction: The source data record three vital signs from fifty-two pigs before and after an induced injury.The authors derive AirwayPressure, ArtPressure, and CVP from these measurements.
  • InternalBleeding construction: InternalBleeding classes correspond to individual pigs, with training examples drawn from the first 2000 points of before- and after-injury series.Testing uses the second and third 2000-point segments of both series.
  • InternalBleeding challenges: The data are challenging because they are not phase-aligned, contain many classes relative to training instances, and give each class one healthy and one injured example.These properties may favor phase-invariant or elastic distance measures.
  • Electrical Load Measurement data: The Freezer data set contrasts kitchen and garage freezer power demand, with classes that are globally similar but locally distinguishable.A subtle class-conserved difference appears shortly after the fiftieth data point, where compressor dynamics differ.
  • Electrical Load Measurement data: Freezer provides smaller and regular training-set versions with the same test set to benchmark methods designed for limited training data.The section gives synthetic-series augmentation as an example of such methods.
  • Section conclusion: The archive is released as a 128-data-set resource alongside recommended classification-evaluation practices and a warning about mis-attributing accuracy improvements.The authors encourage deeper examination of proposed distance measures and algorithms.
Loading 1810.07758v2…