Source-linked AI summary
An Empirical Evaluation of Similarity Measures for Time Series Classification
Joan Serrà, Josep Lluis Arcos
TL;DR
The paper addresses how to evaluate time-series similarity measures and uses classification accuracy with a 1NN classifier as the evaluation framework. Across the considered distances, TWED consistently outperforms the others, while several measures form an accuracy-equivalent group.
Problem
Determining similarity between time series is a core issue, and the paper identifies this evaluation problem as especially important.
Method
The evaluation uses classification accuracy from a parameter-free 1NN classifier, whose error critically depends on the similarity measure.
Results
TWED consistently outperforms all considered distances, including DTW and EDR, while several measures show no statistically significant accuracy differences.
Takeaways & Limitations
The findings support using TWED as a baseline measure and motivate more consistent evaluation criteria for time-series similarity research.
Takeaways & Limitations
A particular measure or variant could still be well-suited for a specific data set and statistically significantly outperform the group of equivalent measures.
Abstract
from arXiv · showhide
Time series are ubiquitous, and a measure to assess their similarity is a core part of many computational systems. In particular, the similarity measure is the most essential ingredient of time series clustering and classification systems. Because of this importance, countless approaches to estimate time series similarity have been proposed. However, there is a lack of comparative studies using empirical, rigorous, quantitative, and large-scale assessment strategies. In this article, we provide an extensive evaluation of similarity measures for time series classification following the aforementioned principles. We consider 7 different measures coming from alternative measure `families', and 45 publicly-available time series data sets coming from a wide variety of scientific domains. We focus on out-of-sample classification accuracy, but in-sample accuracies and parameter choices are also discussed. Our work is based on rigorous evaluation methodologies and includes the use of powerful statistical significance tests to derive meaningful conclusions. The obtained results show the equivalence, in terms of accuracy, of a number of measures, but with one single candidate outperforming the rest. Such findings, together with the followed methodology, invite researchers on the field to adopt a more consistent evaluation criteria and a more informed decision regarding the baseline measures to which new developments should be compared.
11. Introduction
Time-series similarity measures are central to classification and related tasks, yet their comparative evaluation has been limited. This study addresses gaps in broad quantitative comparison, evaluation methodology, statistical testing, and parameter assessment.
- Similarity measures are central to time-series mining, retrieval, clustering, and classification tasks.
- Few quantitative comparisons evaluate similarity-measure efficacy across multiple data sets.
- Existing studies often compare new measures against only a few alternatives and use reduced data-set corpora.
- The literature lacks agreement on evaluation methodologies, and statistical significance is often unstudied or improperly evaluated.
- Optimal parameter values are rarely discussed, leaving uncertainty about suitable baselines and parameter choices for future developments.
- The study evaluates seven measures on 45 publicly available data sets, focusing pragmatically on classification accuracy with robust cross-validation and statistical tests.
1082. Time series similarity measures
The paper surveys representative similarity-measure families and evaluates selected approaches for time-series comparison. The measures include lock-step, feature-based, model-based, and elastic approaches, with parameters affecting computation and elastic methods aligning series in time.
- The paper presents representative lock-step, feature-based, model-based, and elastic similarity-measure families.
- The evaluated pool emphasizes standard, well-performing measures while avoiding approaches with too many parameters or measures previously found less accurate than DTW or EDR.
- Euclidean distance compares samples at identical temporal locations, placing it among lock-step measures.
- Fourier-coefficient measures compute Euclidean distance on selected transformed coefficients, with θ controlling the number of coefficients and potentially filtering high frequencies.
- Auto-regressive measures compare estimated model coefficients, while η controls model order and directly affects similarity-computation speed.
- The evaluation framework formally assesses the measures, and the authors report that one measure outperforms DTW by a statistically significant margin.
- The MJC jump cost combines temporal advancement and magnitude differences, using a nonlinear penalty controlled by φ.
3453. Evaluation methodology
The evaluation compares seven similarity measures using 1NN classification across 45 UCR data sets, with repeated cross-validation, parameter optimization, and statistical tests. Data are z-normalized, and parameter grids are matched across algorithms.
- Classification scheme: The study evaluates similarity measures through a parameter-free one-nearest neighbor classifier whose error depends critically on the selected measure.The 1NN approach is also easy to implement and has theoretical links to errors from other classification schemes.
- Data sets: 45 publicly available UCR data sets span 2–50 classes, 56–9,236 series per set, and lengths of 24–1,882 samples.The collection includes synthetic, real-world, and shape-derived time series and represented the complete repository as of March 2013.
- Validation: Three-fold cross-validation is repeated 20 times on balanced data sets, with average error ratios reported for out-of-sample assessment.Balancing equalizes class representation in error estimation, while repeated validation supports statistical assessment beyond a single split.
- Preprocessing and parameter selection: All series are z-normalized, and each measure’s parameters are optimized during training by a grid search with the same number of combinations per algorithm.The grids follow common practice and the original measure specifications; Table 1 includes the tested ranges and spacing choices.
4404. Results
Across the 45 data sets, no measure is best everywhere, but TWED has the strongest overall performance and is statistically superior to the other measures. Training-stage rankings generally agree with testing-stage rankings, while parameter choices reveal measure- and data-set-specific alignment patterns.
- Classification performance: test: All considered measures outperform the random baseline for practically all 45 data sets, although specific measure choice can still matter on individual data sets.Some data sets show near-perfect accuracies, but the strongest measure varies across data sets.
- Classification performance: test: TWED is best on 20 data sets, compared with 2 for Euclidean, 4 for FC, 1 for AR, 6 for DTW, 7 for EDR, and 5 for MJC.No single measure achieves the best performance on every data set.
- Classification performance: test: Warping-based measures show an order-of-magnitude advantage over Euclidean distance on Trace and the three Face data sets, unlike DiatomSizeReduction and NonInvasiveFetalECG2.These contrasts provide an indication of where alignments, warping, or sample correspondences may matter for classification.
- Classification performance: test: TWED has average rank 2.29 and is statistically significantly superior overall, while DTW, MJC, and EDR form an equivalent group.The global analysis separates the remaining measures from this group and from one another.
- Classification performance: test vs. train: The best-performing measure during training is generally also best during testing, and relative rankings of the other measures mostly agree across the two stages.Only a few exceptions are reported.
- Parameter assessment: TWED and MJC show spread parameter distributions alongside fairly good accuracies, indicating robustness to specific parameter choices.The selected parameters generally avoid grid boundaries, except that TWED may benefit from reconsidering its range.
- Parameter assessment: Large DTW windows are selected for Synthetic control, Trace, Two Patterns, and TwoLeadECG, indicating that tracking warping paths beyond the main diagonal may help there.TWED’s corresponding stiffness parameter takes relatively small values, reinforcing the alignment-related interpretation.
5385. Conclusion
The evaluation finds one similarity measure consistently outperforming the alternatives, while several measures form an accuracy-equivalent group. The authors argue that broad empirical comparison supports more unified validation and baseline choices, while acknowledging dataset-specific exceptions.
- Several measures form a statistically equivalent group, with no significant differences among them.
- TWED consistently outperforms all considered distances, including DTW, EDR, and MJC.
- MJC should serve as a baseline in future evaluations because it performs competitively and has additional attractive properties.
- Euclidean distance is competitive but performs significantly worse than TWED, DTW, MJC, and EDR, especially on large data sets.
- Training errors generally provide a good estimate of test errors, although the authors identify notable exceptions and assess them with Texas sharpshooter plots.
- The study’s large-scale comparison is presented as a step toward unified validation procedures, while particular measures may still outperform others on specific data sets.
Organization and Algorithms, pp. 69–84. 615
This section presents references covering time-series similarity, matching, classification, clustering, and statistical evaluation methods.
- The references include foundational and applied work on dynamic time warping and related time-series matching methods.
- The section also cites statistical methods for comparing classifiers and controlling multiple hypothesis tests.
- The cited literature covers time-series classification, clustering, and similarity evaluation across multiple methodological perspectives.
- Additional references address dimensionality reduction, representation methods, indexing, and retrieval for time-series data.