PaperlayerView exact revision on arXiv

Source-linked AI summary

Detect, Explain, Interpret: An End-to-End Benchmark for Time Series Anomaly Detection, Explainability and Interpretability

Roberto Stanzione, Jules Barbe, Magali Parrino, Jérémie Fourmann, Paul Boniol

arXiv:2610.01168v1cs.LGcs.AIcs.DB

TL;DR

Existing TSAD benchmarks emphasize detection accuracy and provide limited fine-grained semantic context for assessing explanations and interpretations. The paper introduces SHAD, a real-world annotated benchmark, and evaluates detection, dimension attribution, and frozen LLM interpretation. The evaluation finds that detection is accurate but imperfect, existing methods do not natively solve explainability, and frozen LLMs do not correctly interpret anomalies.

  • Problem

    Existing benchmarks focus mainly on detection accuracy and lack fine-grained labels and rich semantic context for evaluating anomaly explanation and interpretation.

    Source: §1 · pp. 1–2

  • Method

    The authors construct SHAD from real-world distributed storage time series and evaluate detectors, dimension contributions for explainability, and frozen LLM baselines for interpretation.

    Source: §1 · p. 2, §4 · p. 7

  • Results

    The evaluation concludes that detection approaches are accurate but imperfect, existing TSAD methods do not natively solve explainability on SHAD, and frozen LLMs do not provide correct anomaly interpretations.

    Source: §5 · p. 10

  • Takeaways & Limitations

    SHAD supports end-to-end evaluation of detection, explanation, and interpretation using per-dimension labels and rich textual annotations.

    Source: §1 · p. 2, §5 · p. 10

    ChatTS was considered for descriptions but supported only up to 30 time series at the time of writing.

    Source: §4.1 · p. 7

Abstract

from arXiv · show

Time Series Anomaly Detection has received increasing attention, driven by the growing availability of complex time series data. This surge has led to the development of numerous detection methods, as well as a variety of benchmarks aimed at thoroughly evaluating their performance. However, most existing detectors remain largely agnostic to domain context, overlooking explainability and interpretability. One of the main reasons for this gap is that current benchmarks primarily focus on detection accuracy, and only few of them evaluate spatial explainability. Moreover, no benchmark currently provides sufficiently rich semantic annotations to support the generation of human-understandable interpretations of anomalies. To address these limitations, we introduce SHAD (Scality High-dimensional Anomaly Detection benchmark), a fully annotated benchmark composed of 215 multivariate, high-dimensional time series collected from real-world distributed cloud storage systems operated by Scality. The proposed dataset includes rich contextual information, covering three families of anomalies with varying degrees of severity. As further contribution, we provide a foundation for future work by evaluating baseline methods for Detection, Explainability, and Interpretability, covering all stages of a TSAD pipeline. For Detection, we benchmark a wide range of existing anomaly detectors, testing their effectiveness on the proposed real-world dataset. Then, we consider explainability by evaluating whether measuring the contribution of each dimension in the generated anomaly score can provide accurate anomaly attributions. Finally, for interpretability, we investigate the effectiveness of frozen LLM baselines in localizing and interpreting anomalies.

1 Introduction

Existing TSAD benchmarks emphasize detection accuracy, leaving explainability, interpretability, and rich semantic context under-evaluated. SHAD addresses this gap with real-world high-dimensional series annotated with anomaly and system context.

1 Introduction

  • Most TSAD benchmarks focus on detection accuracy and lack rich semantic annotations for evaluating human-understandable anomaly explanations and interpretations.Exathlon provides root-cause and effect intervals but lacks fine-grained per-dimension annotations mapped to system components.Source: §1 · pp. 1–2
  • SHAD contains 215 high-dimensional time series from Scality RING, with per-dimension labels and documentation of anomaly type, affected components, and severity.Its documentation covers all 171 dimensions and maps each sensor to system components and roles.Source: §1 · p. 2
  • The benchmark evaluates 18 anomaly detectors, dimension-level contributions for spatial explainability, and zero-shot LLMs for anomaly localization and explanation.Its annotations support evaluation across detection, explanation, and interpretation.Source: §1 · p. 2

2 Background and Related Work

The background distinguishes anomaly detection, spatial explanation, and contextual interpretation, then reviews existing methods and benchmarks. It identifies gaps in end-to-end evaluation, fine-grained labels, and semantic context that motivate SHAD and the paper’s research questions.

Time Series Anomaly Detection

  • Detection locates anomalies in time without identifying the responsible dimension; distance-, density-, and prediction-based methods use different anomaly signals.Prediction-based methods use forecasting or reconstruction error, while most detectors remain domain-agnostic.Source: §2 · pp. 2–3

Interpretable and Explainable Anomaly Detection

  • The paper distinguishes explainability as spatial attribution of contributing input components from interpretability as a human-understandable account of anomaly nature and context.Existing explainability approaches include intrinsically explainable models, model-agnostic perturbations and feature importance, and global approximations or rules.Source: §2 · p. 3
  • Prior interpretability work includes attribute-based and biomedical methods, while LLM research includes multimodal time-series analysis and text-context integration.ChatTS is described as a multimodal LLM for reasoning over multivariate series and interpreting patterns, relationships, and anomalies.Source: §2 · pp. 3–4

Existing Benchmarks

  • Existing benchmarks span streaming, multivariate, univariate, real and synthetic datasets, and comparisons of anomaly detection methods, but are not one-size-fits-all.Examples include NAB for streaming detection, Exathlon for explainable high-dimensional detection, and TAB’s comparison of 44 approaches.Source: §2 · p. 4

Limitations of Existing Benchmarks

  • The review identifies three gaps: benchmarks overlook explanation and interpretation, provide only coarse anomaly labels, and lack contextual annotations for human-readable interpretations.Existing labels do not specify affected dimensions or anomaly nature.Source: §2 · p. 4

Objectives and Research Questions

  • The paper proposes a real-world curated dataset with rich descriptions and fine-grained documentation, and asks research questions about detection, dimension-level attribution, and frozen-LLM interpretation.The questions test detector effectiveness, whether dimension-level attribution can provide explainability, and whether frozen LLMs can interpret anomalies accurately.Source: §2 · p. 4

3 SHAD: Our Proposed Benchmark

SHAD is constructed through controlled experiments on Scality Ring, pairing telemetry with three injected anomaly families and expert-curated labels. Its design combines broad failure configurations with global and dimension-level semantic annotations.

System Overview

  • SHAD telemetry comes from Scality Ring deployments with three storage servers, six software nodes per server, and 171 metrics.A supervisor server manages, monitors, and orchestrates the experimental framework.Source: §3 · pp. 4–5

Experiment Design

  • Experiments run an 18-hour GET-heavy S3 workload while controlled injections cover server, disk, and storage-node failures.The workload uses a one-minute telemetry interval and yields 1000–1050 points per experiment.Source: §3 · p. 5

Anomaly Simulation

  • Injected failures span disk disconnections, network partitions or degradation, and killed storage-node processes, with onset, duration, and target configurations varied broadly.Onsets range from 30 minutes to 16.5 hours into the experiment, and durations range from 1 minute to over 9 hours; storage-node failures typically have the smallest impact because the DHT redistributes work across live nodes.Source: §3 · pp. 5–6

Dataset Characteristics

  • 215 multivariate series (∼1021 one-minute samples across 171 features) carry expert-verified global and root-cause dimension labels plus structured XML explanations.Annotations include anomaly criticality, category, and textual explanations; dimension-level records include failure-specific parameters and links to other dimensions involved in the same event.Source: §3 · p. 6

4 Experimental Evaluation on SHAD

The evaluation tests detection, dimension attribution, and frozen LLM interpretation on SHAD. It finds strong but uneven detector performance, while simple attribution and zero-shot interpretation remain inadequate.

4.1 RQ1: Time Series Anomaly Detection

  • Semi-supervised detectors generally outperform unsupervised methods on SHAD, but KMAD is statistically competitive with the top five; no method is accurate across every anomaly type.KMAD joins the top methods in the critical-difference diagram, while its strengths differ by event: semi-supervised methods detect subtle failures better, and KMAD is more robust to system-wide network events.Source: §4.1 · pp. 7–8, Figure 7 · p. 8

4.2 RQ2: Time Series Anomaly Explanation

  • SHAD’s anomaly diversity and complexity cannot be handled by simple dimension-contribution analysis alone.The explainability experiment tests dimension contributions to the anomaly score; the authors leave refined strategies for future work.Source: Figure 8 · p. 8, §4.2 · p. 9

4.3 RQ3: Time Series Anomaly Interpretation

  • Zero-shot LLMs struggle across interpretation tasks: dimension localization is near-zero, and anomaly type and criticality scores are also low.Images are the best representation for three tasks and second-best for the others; among models, Claude leads on type, criticality, and time localization, while Grok leads on dimension localization at only 4% accuracy.Source: §4.3 · p. 9, Figure 9 · p. 9
  • Providing anomaly presence improves dimension and time localization by 7.2% and 7.9% on average; adding affected components further improves time localization by 14.4% and anomaly type by 4.7%.Criticality changes negligibly, and the authors conclude that zero-shot LLMs still cannot reliably characterize anomalies.Source: §4.3 · pp. 9–10
  • Mistral Small generally performs worse, while Mistral Medium’s image results remain stable and improve time localization to 14% from 7%.With Mistral Medium, Catch22 performance drops significantly; image descriptions produce results nearly identical to Mistral Large.Source: §4.3 · p. 10, Figure 10 · p. 10

5 Conclusion and Future directions

The conclusion presents SHAD as a real-world, high-dimensional benchmark spanning detection, explanation, and interpretation. Its evaluation finds that existing approaches do not fully solve all three tasks, and it outlines research directions grounded in the benchmark’s annotations.

5 Conclusion and Future directions

  • SHAD contains normal and anomalous real-world high-dimensional series, with component- and severity-varying anomalies, per-dimension labels, and rich textual annotations.The benchmark covers all three TSAD pipeline stages: detection, explanation, and interpretation.Source: §5 · p. 10
  • The evaluation finds that state-of-the-art detectors are not accurate on every SHAD anomaly, existing TSAD methods do not natively solve explainability, and frozen LLMs do not correctly interpret anomalies.Source: §5 · p. 10
  • The authors propose studying detector robustness, developing explanation methods trained on SHAD annotations, and giving LLMs reliable dimension-level explanations as semantically enriched context.Source: §5 · p. 10

A.1 Existing benchmarks

Existing TSAD benchmarks cover diverse datasets and methods, but evaluation remains centered on detection accuracy. Interpretability and explainability work includes feature attribution and emerging LLM approaches, while available datasets are cataloged by size, length, dimensionality, anomaly type, and domain.

A.1 Existing benchmarks

  • Existing benchmarks span heterogeneous datasets and detection methods, but generally emphasize accuracy; Exathlon targets explainable high-dimensional detection and TAB compares statistical, neural, and foundation-model approaches.The cited review also notes that Exathlon often has one anomaly per series, often artificially generated.Source: §A.1 · p. 19

A.2 TSAD Datasets

  • Table 2 organizes 30 TSAD datasets by series count, average length and dimensionality, anomaly type, and domain.The passage specifies the characteristics reported for each dataset.Source: §A.2 · p. 19, Table 2 · p. 20

A.3 Interpretability and Explainability

  • Explainability approaches include local surrogates, rules, and feature-importance scores, while anomaly interpretation remains limited and includes LLM-based analysis such as ChatTS.ChatTS accepts multivariate time series and supports natural-language interpretation of patterns, relationships, and anomalies.Source: §A.3 · pp. 19–20

B Metric Catalog

The metric catalog groups 171 dimensions across telemetry hierarchy levels and documents naming conventions. The experiment registry groups runs by anomaly type and records configurations, artifacts, and compact notation for fault timing and patterns.

B Metric Catalog

  • The 171 dimensions comprise 17 platform-level, 13 S3-level, 57 server-level, and 84 device-level metrics.Server metrics replicate 19 unique metrics across three servers; device metrics cover four metrics across seven devices per server.Source: B Metric Catalog · p. 21, Table 3 · p. 21, Table 5 · p. 21
  • Server- and device-level columns encode their hierarchy in names, using {server}-{metric} and {server}-{device}-{metric} patterns.The catalog gives examples such as store1-cpu_usage_ratio and store1-g1disk01-disk_read_bytes.Source: B Metric Catalog · p. 21

C Experiment Registry

  • The registry groups 215 experiments by anomaly type; details summarize configuration and notes record artifacts such as unrecovered disks, telemetry loss, crashes, and reruns.Permanent disk loss means the target stayed offline, while some reruns produce very different time series.Source: C Experiment Registry · pp. 23–24
  • Registry entries use compact notation for disk targets, fault onset and duration, asynchronous failure patterns, degradation stages, and software-node counts.Cascading, rolling, and independent denote overlapping, partially overlapping, and non-overlapping asynchronous failures, respectively.Source: C Experiment Registry · p. 24

C.1 Baseline Experiments

SHAD experiments use a common 18-hour workload and inject disk, server, and storage-node failures into a distributed storage cluster. The experimental descriptions document failure variants, anomalies in execution, and the cluster mechanisms that shape telemetry.

C.1 Baseline Experiments

  • 83 nominal runs use the shared 18-hour GET-heavy workload, with 16 workers, 15,000 initial objects, and fixed S3 operation proportions.The workload uses GET 95%, PUT 0.8%, STAT 4%, and DELETE 0.4%.Source: §C.1 · p. 24, Table 11 · p. 24

C.2 Disk Failures

  • Disk experiments cover single, simultaneous, and asynchronous failures; some simultaneous runs left disks offline, producing longer local anomalies despite accurate labels.Later same-server disk recoveries used a 10-minute staggered strategy.Source: §C.2 · p. 25, Table 12 · p. 26

C.3 Server Failures

  • Server experiments include instant loss and progressive network degradation; two stress-related unplanned crashes required manual resets and were rerun at shorter controlled durations.The original and rerun telemetry profiles differ substantially.Source: §C.3 · pp. 26–27, Table 13 · p. 27

C.4 Storage Node Failures

  • Storage-node runs kill configured nodes on one or more servers; a telemetry-agent bug affected 16 of 46 runs and produced distinct profiles in affected runs and reruns.The node failures are grouped as single, simultaneous, or asynchronous.Source: §C.4 · p. 28, Table 14 · p. 28

D.1 System Details

  • The shared cluster has three storage servers and 18 software storage nodes; Chord redistributes key ranges, while rebuild, repair, and rebalance tasks produce metric signatures after failures.The workload is distributed across servers, and background tasks intensify after failure events.Source: §D.1 · p. 29, Figure 12 · p. 29

D.2 Implementation Details

Implementation details describe how the experiments generate workload and inject failures, while the evaluation lists anomaly detector families and the prompt components used for the interpretability study.

D.2 Implementation Details

  • Warp generates round-robin S3 traffic, so an unreachable server causes approximately one-third of client requests to be dropped and reduces cluster throughput proportionally.Experiments are defined as timed fault-injection events over the baseline workload, which runs for 18 hours.Source: §D.2 · pp. 29–30
  • Disk, server, and storage-node failures are injected through device detachment, network impairment, or process termination, respectively, with recovery procedures documented for each.The server experiments preserve SSH and ARP traffic during network impairment; disk replacement can trigger background rebuilds.Source: §D.2 · p. 30

E.1 Anomaly Detection

  • The detector pool is heterogeneous, covering semi-supervised and unsupervised methods as well as prediction-, density-, and distance-based approaches.The pool is the one used for the experimental evaluation.Source: Table 17 · p. 32, §E.1 · p. 33

E.2 Prompt Template example

  • The interpretability prompt combines task context, a provided representation, and a questionnaire, and asks the LLM to motivate its answers.The figure shows the Catch22 context example.Source: §E.2 · p. 33, Figure 13 · p. 33

E.3.1 Anomaly Detection

The appendix compares anomaly-detection efficiency and reports stratified explainability results, model rankings, and gains by supervision level. Claude ranks best overall across interpretability tasks, while dimension and temporal localization show the strongest supervision gains.

E.3.1 Anomaly Detection

  • Unsupervised methods are generally faster because they require no training, and KMAD offers the best performance–efficiency trade-off.Figure 14 reports average time to obtain an anomaly score on a SHAD time series; semi-supervised methods include training and inference times.Source: §E.3.1 · pp. 33–34, Figure 14 · p. 34
  • FITS is the slowest method, taking a day to run on the full dataset; nondeterministic methods' VUS-PR results average five runs.The runtime comparison concerns inference and training time per time series, with training times also reported for semi-supervised methods.Source: §E.3.1 · p. 34, Figure 14 · p. 34

E.4 Complete Reults tables

  • NDCG values for the top detectors are close to random prediction, and localization difficulty varies with the number of affected dimensions.The pattern is evident for highly localized anomalies such as Single Disk.Source: §E.4 · p. 34, Table 19 · p. 35
  • Dimension localization and temporal localization show the strongest average gains across supervision levels, models, and representations.The reported gains are averaged across all models and representations.Source: §E.4 · p. 34
  • Claude leads three interpretability tasks and ranks second for anomaly presence, averaged across Catch 22, Images, and Descriptions.The model ranking covers anomaly criticality, temporal localization, anomaly type, and anomaly presence.Source: §E.4 · p. 34, Table 20 · p. 35

F Limitations

SHAD's scope is limited to single-anomaly series from one industrial distributed storage system, while its explainability and interpretability evaluations use simple baselines. The annotations may support debugging research, but exclusive use of SHAD may limit generalization to substantially different domains.

F Limitations

  • Each series contains one anomaly type, and SHAD covers one industrial storage system, prioritizing annotation quality and reliability over system heterogeneity.The authors identify mixed failures and additional systems or domains as future extensions requiring real infrastructures and substantial annotation effort.Source: F Limitations · p. 35
  • Explainability tests simple dimension-score attribution, while interpretability uses a single-call LLM baseline; refined strategies are left for future work.Examples of possible refinements include per-dimension analysis, advanced prompting, and multi-stage reasoning.Source: F Limitations · p. 36

G Broader Impacts

  • Per-dimension labels and rich textual annotations may support interpretable-machine-learning research and help practitioners understand failures and conduct root-cause analysis.The authors describe these as potential broader impacts of SHAD.Source: G Broader Impacts · p. 36
  • Models developed and evaluated exclusively on SHAD may generalize poorly to substantially different domains, and deployed systems may still produce false positives or missed detections.The authors also flag possible misuse of observability advances to infer sensitive operational patterns or evade detection.Source: G Broader Impacts · p. 36

Related papers

Loading 2610.01168v1…