Source-linked AI summary
Precision and Recall for Time Series
Nesime Tatbul, Tae Jun Lee, Stan Zdonik, Mejbah Alam, Justin Gottschlich
TL;DR
Classical Precision and Recall do not adequately evaluate time-series anomalies that span ranges or reflect domain-specific preferences. The paper extends these metrics with a customizable range-based model, whose configured behavior subsumes the classical model and captures overlap and positional effects. Experiments compare the scoring models across datasets and detectors, while noting that parameter settings must suit each application.
Problem
Classical Precision and Recall are limited for range-based time-series anomalies and can misrepresent anomaly-detection accuracy.
Method
The paper extends Precision and Recall to ranges using tunable overlap, cardinality, and positional-bias functions.
Results
The configured range-based model subsumes the classical model, while experiments illustrate sensitivity to positional bias and range structure.
Takeaways & Limitations
The model provides an expressive, flexible, and extensible framework for evaluating and comparing time-series anomaly detectors.
Takeaways & Limitations
Its tunable parameters must be properly set for each application domain, and the evaluation experiment targets scoring-model behavior rather than detector quality.
Abstract
from arXiv · showhide
Classical anomaly detection is principally concerned with point-based anomalies, those anomalies that occur at a single point in time. Yet, many real-world anomalies are range-based, meaning they occur over a period of time. Motivated by this observation, we present a new mathematical model to evaluate the accuracy of time series classification algorithms. Our model expands the well-known Precision and Recall metrics to measure ranges, while simultaneously enabling customization support for domain-specific preferences.
1 Introduction
Time-series anomalies often occur over ranges rather than individual points, limiting classical point-based evaluation. The paper introduces a customizable mathematical model that extends Precision and Recall to range-based anomalies.
- Many real-world time-series anomalies occur over periods, so anomaly-detection systems must reason about events as ranges.The paper identifies range-based anomalies as events occurring over a period of time.
- Classical Precision and Recall are useful for point anomalies but cannot represent domain-specific characteristics of time-series anomalies.The paper links this limitation to misrepresented accuracy for some time-series anomaly-detection systems.
- The proposed model redefines Precision and Recall to encompass range-based anomalies and subsumes the classical point-based model.The extension is intended to preserve the classical formulation as a special case.
- Specialization functions let users control domain bias across multiple dimensions when evaluating and comparing anomaly-detection results.The paper presents customization as a way to accommodate domain-specific preferences.
2 Problem motivation and design goals
Classical point-based metrics fall short when time-series anomalies are intervals with partial overlaps. The paper therefore designs a range-based model that is expressive, flexible, and extensible.
- Classical metrics treat anomalies as independent points, whereas time-series detectors operate on intervals where partial overlaps matter.A predicted range may overlap a real range without matching it exactly.
- The classical model suffices for point anomalies or cases without partial overlap, but falls short for general range-based anomalies.This limitation motivates the paper’s new evaluation model.
- The model is expressive because it captures range-specific criteria such as overlap position and cardinality.
- The model is flexible and extensible through adjustable criterion weights and additional domain-specific criteria.These properties support customization beyond criteria known in advance.
3 Related work
Prior work developed time-series anomaly detectors and related evaluation measures, but efficacy measures for time-series anomaly detection remained underdeveloped. Existing alternatives generally lacked the positional bias and tunability of this model.
- Time-series classification and anomaly-detection research spans applications from space shuttles to web services, but evaluation techniques remained underdeveloped.
- Related evaluation work addressed segmented activity events, fragmentation, merging, and timing offsets.
- Those related approaches did not account for positional bias or provide a tunable model like the paper’s approach.The paper characterizes them as complementary rather than equivalent alternatives.
- The NAB scoring model remained point-based with a fixed early-detection bias, making it less generalizable than the proposed model.
4 Precision and recall for ranges
The paper defines range-based precision and recall by combining anomaly existence, overlap size, positional bias, and prediction cardinality, with tunable functions for domain-specific preferences. The formulations average scores across real or predicted anomaly ranges and reduce to classical metrics under specified conditions.
- Range-based recall: Range-based recall averages per-range scores across all real anomaly ranges to evaluate how successfully anomalies are detected.Each real range contributes a recall score before division by the number of real anomalies.
- Range-based recall: Recall combines an existence reward with an overlap reward weighted by α and 1 − α, respectively.The overlap reward captures size, position, and cardinality.
- Range-based recall: Overlap reward uses cardinality, size, and position functions to score partial overlaps between real and predicted anomaly ranges.The cardinality term scales rewards from overlap size and position.
- Customization guidelines and examples: The tunable weight and specialization functions can differ by application and between precision and recall.The model subsumes classical precision and recall when ranges are unit-sized and specified parameter settings provide flat positional treatment.
- Range-based precision: Range-based precision averages scores across predicted ranges, omits the existence reward, and retains overlap-based size, position, and cardinality evaluation.Precision emphasizes prediction quality rather than merely detecting an anomaly's existence.
- Customization guidelines and examples: The model makes positional bias domain-specific: flat, front-end, back-end, and middle functions prioritize different parts of an anomaly range.Front-end bias suits early-response settings, whereas back-end bias suits settings requiring greater certainty before action.
- Customization guidelines and examples: Cardinality functions penalize fragmented detection by decreasing the reward as the number of distinct overlapping prediction ranges increases.The paper describes reciprocal forms such as γ() = 1/x.
5 Experimental study
The experimental study evaluates the range-based model against classical and Numenta scoring across diverse datasets, examining expressiveness, positional flexibility, detector comparisons, and computational cost. Results show that the model subsumes classical metrics, captures range-specific distinctions, supports scenario-dependent scoring, and can be efficiently optimized.
- Experimental study: The study evaluates classical, Numenta, and range-based precision/recall on diverse real and synthetic time series datasets.Experiments use known anomalous points or ranges and compare detector outputs under different scoring models.
- Comparison to the classical point-based model: The first two bars are equal across all datasets, demonstrating that the range-based model subsumes the classical model when properly configured.The comparison covers Recall, Precision, and F1-Score values for LSTM-AD testing.
- Comparison to the classical point-based model: RecallT varies with positional bias, while classical Recall cannot capture the positions of correctly predicted portions within anomaly ranges.Time-Guided illustrates this pattern because its back-ended half-predictions make Recall_T_Back larger than other biases.
- Comparison to the classical point-based model: PrecisionT is typically smaller than Precision and is more range-aware, judging the exactness of range-based predictions more accurately.Narrow predictions against wide NYC-Taxi anomalies explain a case where PrecisionT is slightly larger than Precision.
- Comparison to the Numenta scoring model: The model distinguishes scenarios that NAB does not separate when its positional bias is set appropriately, demonstrating greater flexibility and generality.NAB favors early detections, whereas the proposed model can use different positional biases.
- Cost analysis: Optimized implementations compute both classical and range-based metrics in almost three orders of magnitude less time than corresponding naive baselines, with a remaining factor-of-2–3 difference.The range-based metrics are substantially more costly naively, but optimization brings their values notably closer to classical costs at the tested scale.
6 Conclusions and future directions
Classical precision and recall do not capture range-based anomalies in time series, which occur over intervals rather than isolated points. The paper introduces a customizable formal model that accounts for range-specific issues and illustrates its expressiveness, flexibility, and extensibility through comparisons with two other models.
- Classical precision and recall were designed for point-based data, whereas time-series anomalies occur as ranges.
- The proposed model accounts for partial overlaps and the relative positions of real and predicted anomaly ranges.
- Users can supply customizable bias functions to weigh multiple criteria differently for domain-specific evaluation.
- Experiments comparing the model with two others illustrated that it is expressive, flexible, and extensible.
- The model has tunable parameters that must be properly set for each application domain.