PaperlayerView exact revision on arXiv

Source-linked AI summary

Local Evaluation of Time Series Anomaly Detection Algorithms

Alexis Huet, Jose Manuel Navarro, Dario Rossi

arXiv:2206.13167v1cs.LG

TL;DR

Time series anomaly detection metrics struggle with temporal proximity, event-based interpretability, parameter selection, and adversarial predictions. The paper introduces affiliation metrics that use local time distances and random sampling to produce normalized precision/recall, and reports theoretical and practical evidence for their robustness and interpretability. The approach remains subject to ground-truth quality and the need for complementary deployment measures.

  • Problem

    Existing anomaly-detection metrics inadequately handle temporal structure, may require parameters, obscure interpretation, and can be vulnerable to adversarial or random predictions.

  • Method

    The paper introduces parameter-free affiliation metrics that partition time around ground-truth events, measure directed temporal distances locally, and normalize them against random sampling.

  • Results

    The affiliation metrics provide local, theoretically grounded precision/recall whose values are interpretable relative to random prediction, with scores around 0.5 matching that baseline.

  • Takeaways & Limitations

    The construction supports event-level visualization and comparison while avoiding additional parameters and reducing sensitivity to adversary predictions.

  • Takeaways & Limitations

    Deployment still requires complementary measures such as predicted-event counts and prediction direction, with trade-offs left to field experts.

Abstract

from arXiv · show

In recent years, specific evaluation metrics for time series anomaly detection algorithms have been developed to handle the limitations of the classical precision and recall. However, such metrics are heuristically built as an aggregate of multiple desirable aspects, introduce parameters and wipe out the interpretability of the output. In this article, we first highlight the limitations of the classical precision/recall, as well as the main issues of the recent event-based metrics -- for instance, we show that an adversary algorithm can reach high precision and recall on almost any dataset under weak assumption. To cope with the above problems, we propose a theoretically grounded, robust, parameter-free and interpretable extension to precision/recall metrics, based on the concept of ``affiliation'' between the ground truth and the prediction sets. Our metrics leverage measures of duration between ground truth and predictions, and have thus an intuitive interpretation. By further comparison against random sampling, we obtain a normalized precision/recall, quantifying how much a given set of results is better than a random baseline prediction. By construction, our approach keeps the evaluation local regarding ground truth events, enabling fine-grained visualization and interpretation of algorithmic results. We compare our proposal against various public time series anomaly detection datasets, algorithms and metrics. We further derive theoretical properties of the affiliation metrics that give explicit expectations about their behavior and ensure robustness against adversary strategies.

1 INTRODUCTION

Time series anomaly detection evaluates unusual temporal behavior against ground truth labels, but classical and newer metrics have important limitations. The paper introduces affiliation metrics designed to provide parameter-free, locally interpretable, theoretically grounded evaluation.

  • 1 INTRODUCTION: Time series anomaly detection identifies elements behaving differently from the rest of a series and is used across monitoring, security, industry, and science.Anomalies may be isolated points or consecutive ranges, and performance is assessed against collected ground truth labels.
  • 1 INTRODUCTION: The paper surveys limitations of classical metrics and prior directions for addressing them.
  • 1 INTRODUCTION: Affiliation metrics extend precision and recall with theoretical grounding, no parameters, physical time-based meaning, robustness, and local interpretability.
  • 1 INTRODUCTION: The paper evaluates the metrics through theoretical properties, real datasets, algorithm comparisons, and event-level visual interpretation.

2 BACKGROUND AND MOTIVATION

Classical precision and recall fail to represent temporal proximity and event-level behavior, while newer metrics address these issues unevenly and often introduce parameters or interpretability problems. The paper motivates a parameter-free, expressive, locally interpretable metric with statistical bounds.

  • Limitations of classical metrics: Classical metrics penalize near detections as false positives and false negatives, ignoring temporal adjacency and ambiguous samples.
  • Recent evaluation metrics: Recent metrics include distance-based, window-based, and range-specific approaches, but most do not handle both temporal limitations (A) and (B).Only TaP/TaR is identified as handling both, while other categories remain limited to particular anomaly types or failure modes.
  • Limitations of classical metrics: Figure 1 reports precision/recall values of 0.50/0.09 and 1.00/0.91 for cases where scores respectively understate broad detection and overstate limited detection.
  • Design targets: Many existing metrics introduce parameters that can reduce generality when tuned or overspecialized for particular scoring aspects.
  • Interpretability and robustness: Event-based metrics can combine overlap, position, and ambiguous-sample information into difficult-to-interpret quantities while allowing prediction clusters to distort global scores.
  • Design targets: The paper targets parameter-free, expressive, locally interpretable scores with statistical bounds and robustness against adversarial or random predictions.

3 LOCAL EVALUATION BASED ON AFFILIATION

Affiliation evaluation partitions time around ground-truth events, measures directed temporal distances locally, and converts them into normalized probabilities relative to random sampling. This yields interpretable precision/recall values with a random baseline.

  • 3 LOCAL EVALUATION BASED ON AFFILIATION: The method computes directed average distances between prediction and ground-truth sets, affiliates predictions to the closest ground-truth event, and averages locally derived probabilities.
  • 3.1 Average distance between sets: Anomalous range events are represented as disjoint continuous intervals for predictions and ground truth.
  • 3.1 Average distance between sets: The directed set distance averages each sample’s distance to the other set, preserving a time-valued interpretation while remaining smooth rather than thresholded.
  • 3.1 Average distance between sets: Prediction-to-ground-truth distance is interpreted as precision, whereas ground-truth-to-prediction distance is interpreted as recall.In the example, the corresponding distances are 18s for precision and 76.5s for recall.
  • 3.2 Local affiliation to the closest ground truth: The timeline is partitioned into zones affiliated with the closest ground-truth events, and precision/recall distances are computed separately within each zone.The example produces zones with individual distances of 18s / 76.5s, 11min30s / 2min30s, and 31min15s / 2min30s.
  • 3.3 Comparison against random sampling: Observed distances are converted to probabilities by comparison with uniformly random predictions, then averaged across ground-truth events.Predicted samples inside a ground-truth event receive precision value 1, while values decrease with distance; the example reports an individual recall probability of 0.944.
  • 3.3 Comparison against random sampling: A precision or recall below 0.5 indicates performance worse than random prediction, while a value around 0.5 indicates performance comparable to random prediction.

4 EVALUATION AND PROPERTIES

The evaluation compares affiliation metrics with classical and range-based metrics across datasets and algorithms, focusing on adversarial robustness, local interpretability, and theoretical behavior. Affiliation metrics resist adversarial inflation, support event-level analysis, and distinguish predictions relative to random baselines.

  • Evaluation setup: The evaluation contrasts affiliation metrics with classical sample-based metrics and RP/RR and TaP/TaR range-based metrics across selected datasets and algorithms.The benchmark includes Machine-Temp, NYC-Taxi, Twitter-AAPL, and SWaT, using previously selected algorithm predictions.
  • Adversary predictions: Range-based RP/RR and TaP/TaR metrics are easily gamed: the adversary algorithm beats other tested algorithms while labeling almost all samples positively.This produces high scores without informative anomaly localization, making the results unreliable in a conceivable detection setting.
  • Adversary predictions: For a dataset with n ground-truth events and a trivial event cut into k pieces, adversarial precision and recall approach 1 as k and n become sufficiently large under classical and range-based metrics.The theoretical recall bounds are approximately 1 − 1/(2n) for RP/RR and at least 1 − 1/(4n) for TaP/TaR, while precision is at least k/(k+2).
  • Adversary predictions: Affiliation metrics reduce the adversary's precision to 0.50, indicating predictions no better than a random prediction, while retaining event-level recall behavior.For the SWaT example, the trivial algorithm has precision one and recall 1/35 ≈ 0.03 because recall counts correctly identified events.
  • Event-level comparison: Affiliation metrics provide local precision and recall for individual anomalous events, enabling quantitative comparisons that expose differences hidden by global scores.On SWaT, seq2seq performs better than iForest for 21 events, with better results for both precision and recall on 13 events.
  • Theoretical properties: In theoretical settings, affiliation scores around 1/2 correspond to random detection, while overlapping predictions can retain perfect precision even as recall decreases for larger events.When p is much smaller than 1, affiliation precision is close to 1/2; for overlapping predictions, recall decreases as p increases because one prediction cannot fully recall a large event.

5 DISCUSSION

The discussion examines how affiliation metrics behave under labeling uncertainty, parameter choices, theoretical summarization, and practical deployment. It emphasizes reduced sensitivity to imprecise labels and the need for expert judgment and complementary measures.

  • Figure 4: Figure 4 evaluates affiliated metrics for a single-point prediction at four positions relative to a ground-truth event and its affiliation zone.The positions are the zone border, halfway between border and event, the event’s first element, and its center.
  • Shape of the ground truth labels: Affiliation metrics are less sensitive to precise ground-truth labeling than previous range- or window-based metrics.The amplitude and end time of anomaly labels can remain subjective, especially when the system’s return to steady state is unclear.
  • Expressiveness of the metrics: The metrics support fair algorithmic comparison without additional parameters, while customizable survival functions could make evaluation arbitrary in small-event settings.Possible modifications include focusing only on overlaps or weighting event beginnings, but selecting such parameters adds complexity.
  • Theoretical properties: Affiliation metrics focus on proximity between predicted and ground-truth labels, while variance bounds for summarized scores remain an open research need.F-scores and Average Precision can be derived straightforwardly, but their variance is not yet theoretically characterized.
  • Practical deployment: Practical deployment requires complementary measures such as predicted-event counts and prediction direction, with trade-offs left to field experts.The paper identifies interactive visualization as beneficial for supporting algorithm selection and score thresholding.

6 CONCLUSION

The paper proposes a generic, local precision/recall pair for time series anomaly detection. Its construction is intended to provide interpretable, theoretically principled, and robust evaluation, including against adversary predictions.

  • 6 CONCLUSION: The proposed affiliation precision/recall pair is parameter-free, generic across datasets, and evaluates each ground-truth event separately.Locality supports interpretable and visualizable score components while reducing sensitivity to adversary predictions.

A CLOSED-FORM OF THE AFFILIATION METRICS

The appendix derives closed-form affiliation metrics by modeling distances within affiliation zones and integrating piecewise survival functions. It covers precision, recall, interval predictions, and point-anomaly limits.

  • A CLOSED-FORM OF THE AFFILIATION METRICS: For a ground-truth event [a,b) inside affiliation zone [A,B), m and M are the shortest and largest distances from the event to the zone borders.They are defined as min(a−A, B−b) and max(a−A, B−b).
  • A.1 Survival functions: The precision survival function starts at one and is defined piecewise over distances up to M.Its distance-dependent decrease changes at m.
  • A.1 Survival functions: Uniform sampling within the affiliation zone yields a zero-distance probability of |gt_j|/|I_j| and different decline rates before and after m.Two elements exist at each distance before m, but only one exists after m.
  • A.1 Survival functions: For each ground-truth sample y, m_y and M_y measure its closest and farthest distances to the affiliation-zone borders.These quantities support the recall survival function defined over distances up to M_y.
  • A.2 Closed-form of the integrals: The closed-form integral expressions contain many min/max cases, all of which are implemented in Python.The underlying integrals are piecewise linear.
  • A.2.1 Integral over the samples for the precision.: For precision, each predicted interval is divided into portions before, within, and after the ground-truth event, with cases determined by its distances to the event.The resulting integral uses the interval’s center distance and applies to the considered distance configurations.
  • A.2.2 Integral over the samples for the recall.: For recall, the ground-truth interval is partitioned so each piece is covered or has a unique closest prediction or border, enabling piecewise integration.The derivation defines the piece center, pivot distance, and center-to-border distance before combining the linear terms.

B PARTICULARIZATION TO POINT ANOMALIES

Point anomalies are obtained as the limit of increasingly short events. The affiliation formulas therefore specialize directly to point predictions, with the survival-function forms unchanged apart from the ground-truth-length term.

  • B PARTICULARIZATION TO POINT ANOMALIES: A point anomaly at time t is represented as the limit of the event [t,t+ε) as ε→0.This lets the event-based affiliation expressions be restated for point predictions inside the affiliation zone.
  • B PARTICULARIZATION TO POINT ANOMALIES: The survival-function forms remain unchanged, except that |gt_j| is replaced by 0 for point anomalies.The ground-truth event is a single point in this specialization.

C PROOF OF THE PROPERTIES

The proof establishes affiliation-metric properties by partitioning intervals into cuts satisfying the appendix conditions and applying the relevant equations to each part.

  • The generic proof method finds an interval cut satisfying Appendix A's conditions, then applies Equations 13 and 18.
  • The predicted interval is partitioned into four subintervals before Equation 13 is applied to each part.
  • The resulting four areas yield the expression corresponding to Equation 7.

C.2 Expected precision and recall given a single random prediction

For a single random prediction, integrating the affiliation contribution over the ground-truth interval produces an expected recall of one-half, with precision computed analogously.

  • The expected recall is formed by averaging the recall expression over every t in the ground-truth interval.
  • Fubini's theorem simplifies the calculation by evaluating the inner integral for each fixed y.
  • The distance integral is evaluated piecewise because |t−y| spans different ranges around and beyond the affiliation radius.
  • The resulting quantity is independent of y and equals |I_j|/2.
  • E[P_recall] = 1/2, while the expected precision follows from identical computations.

C.3 Single prediction at a defined position

For a single prediction at a fixed position, the proof centers the ground-truth interval, analyzes separate cases for its size, and combines them into position-specific curves and formulas.

  • The illustrative setup normalizes the affiliation interval to [0,1) and centers a ground-truth region of proportion p at 1/2.
  • The analysis focuses on case (d), where the prediction is located at position 1/2.
  • For p≤1/2, splitting the ground-truth region at 1/2 gives P_recall = 1 − p/2.
  • For p>1/2, partitioning the ground-truth region into four areas gives P_recall = 1/2 + 1/(8p).
  • Combining the cases produces the position-specific expression P_precision = 1 and P_recall = 1 − p/2 + 1/(2p).

D REPRODUCIBILITY

The affiliation metrics are implemented in standard Python 3, tested for reliability, and accompanied by reproducible numerical computations.

  • The implementation converts binary inputs into events, separates affiliation zones, and computes segment-level precision, recall, distances, and probabilities.
  • Unit tests check code reliability, and the numerical results from Section 4 can be reproduced with `python -m unittest discover`.
  • The complete tests took 8 seconds on the specified Windows 10 machine, including computation for the SWaT dataset with 449919 samples and 35 ground-truth events.
Loading 2206.13167v1…