PaperlayerView exact revision on arXiv

Source-linked AI summary

Exathlon: A Benchmark for Explainable Anomaly Detection over Time Series

Vincent Jacob, Fei Song, Arnaud Stiegler, Bijan Rad, Yanlei Diao, Nesime Tatbul

arXiv:2010.05073v3cs.LGcs.DB

TL;DR

Public benchmarks and repositories for explainable anomaly detection over high-dimensional time series are limited. Exathlon constructs a Spark-trace benchmark with labeled anomaly intervals and an evaluation framework, and experiments show useful AD signals while revealing challenges for tested methods.

  • Problem

    Public data repositories and benchmarks for explainable anomaly detection over high-dimensional time series are limited, constraining a common ground for reproducible research.

    Source: §1 · p. 1

  • Method

    Exathlon builds a benchmark from repeated Spark job executions with six injected anomaly types, root-cause and effect-interval labels, and evaluation tools for AD and ED.

    Source: §1 · p. 1, §3.1 · p. 3, §3.2 · p. 4

  • Results

    Experiments indicate that Exathlon’s dataset carries useful signals for AD, while the benchmark exposes limitations of semi-supervised methods under noisy training data and mixed anomaly types.

    Source: §6.2 · p. 9, §7 · p. 12

  • Takeaways & Limitations

    Exathlon provides a common framework for analyzing the strengths and limitations of diverse ED methods across conciseness, consistency, and accuracy.

    Source: §7 · p. 12

    Normal traces include occasional metric variations from Spark’s inherent mechanisms, which the authors treat as normal system behavior.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Access to high-quality data repositories and benchmarks have been instrumental in advancing the state of the art in many experimental research domains. While advanced analytics tasks over time series data have been gaining lots of attention, lack of such community resources severely limits scientific progress. In this paper, we present Exathlon, the first comprehensive public benchmark for explainable anomaly detection over high-dimensional time series data. Exathlon has been systematically constructed based on real data traces from repeated executions of large-scale stream processing jobs on an Apache Spark cluster. Some of these executions were intentionally disturbed by introducing instances of six different types of anomalous events (e.g., misbehaving inputs, resource contention, process failures). For each of the anomaly instances, ground truth labels for the root cause interval as well as those for the extended effect interval are provided, supporting the development and evaluation of a wide range of anomaly detection (AD) and explanation discovery (ED) tasks. We demonstrate the practical utility of Exathlon's dataset, evaluation methodology, and end-to-end data science pipeline design through an experimental study with three state-of-the-art AD and ED techniques.

1 INTRODUCTION

Exathlon addresses the shortage of public benchmarks for explainable anomaly detection (AD) over high-dimensional time series. It combines a curated dataset, evaluation methodology, and extensible pipeline, then uses experiments to show that its traces contain useful signals and expose challenges for tested algorithms.

1 INTRODUCTION

  • Exathlon is presented as the first comprehensive public benchmark for explainable AD over high-dimensional time series, addressing a shortage of shared resources for reproducible research.The paper identifies the lack of open repositories and benchmarks as a critical roadblock to progress.Source: §1 · p. 1
  • The dataset contains real Spark traces from about 100 executions of 10 streaming jobs, with nearly 100 instances across six anomaly classes and labels for root-cause and effect intervals.The traces include normal and disturbed executions, with some noise from Spark’s inherent behavior.Source: §1 · p. 1
  • Exathlon evaluates AD and explanation discovery (ED) functionality and computational performance, using range-based criteria for AD and conciseness, consistency, and accuracy for ED.Its scope includes semi-supervised AD for range-based anomalies and both model-free and model-dependent ED techniques.Source: §1 · p. 1
  • The modular pipeline turns raw datasets into AD and ED results and benchmark scores, and supports implementing and comparing variants of pipeline steps.The paper gives training-data preparation and AD-result scoring as configurable examples.Source: §1 · pp. 1–2
  • Experiments with three state-of-the-art AD and ED techniques find useful signals in the dataset and show that the benchmark exposes increasing levels of challenge.The authors describe this as a systematic stress test of the algorithms.Source: §1 · p. 2
  • Exathlon focuses on one challenging application domain end to end, while the authors note that public time-series ED benchmarks with ground-truth labels are largely lacking.The paper presents the benchmark as an opportunity for more in-depth evaluation of AD and ED methods.Source: §1 · p. 2

2 RELATED WORK

Related work spans benchmarks for system performance and analytics, time-series AD datasets and methods, interpretability approaches, and outlier explanation. The paper positions Exathlon against gaps in public anomaly data and in ground-truth-based evaluation of ED methods.

Datasets and Benchmarks

  • Existing database benchmarks mainly target computational performance, while newer ML/DL analytics benchmarks emphasize end-to-end pipelines and metrics such as time to accuracy.General time-series archives support tasks such as classification and clustering; NAB is identified as a public time-series AD benchmark.Source: §2 · p. 2
  • NAB provides more than 50 real and artificial datasets for streaming AD, but the paper describes its datasets as smaller and less dimensional than Exathlon’s and lacking information for ED.The authors also cite ambiguities in NAB’s scoring function and missing values as technical weaknesses.Source: §2 · p. 2

Anomaly Detection (AD)

  • AD research ranges from statistical, distance- and density-based, and isolation-forest methods to deep learning; the experiments focus on three recent deep-learning methods.The paper notes that deep-learning methods may handle varied anomaly patterns and avoid known dimension sensitivity of distance- and density-based methods.Source: §2 · p. 2

Interpretable Machine Learning

  • Interpretability methods include human-readable models and model-agnostic explanations; the paper evaluates LIME for local explanations and also discusses global explanation approaches.The related work says automated comparison of ED methods is difficult without ground-truth explanations and evaluation procedures.Source: §2 · pp. 2–3

Explaining Outliers in Data Streams and SQL Query Results

  • Stream-outlier methods such as EXstream and MacroBase provide explanations, but database-query explanation approaches target traditional DB workloads and are not applicable to this problem.Industrial time-series explanation systems may require user inputs such as causal hypotheses or root-cause labels, and are largely based on inaccessible proprietary code and datasets.Source: §2 · p. 3

3 DATASET

Exathlon is built from Spark application traces, with normal executions and injected anomalous events represented in a dataset of multivariate metrics. Its construction includes real-system variation in normal traces and ground-truth intervals for anomalous traces.

3.1 Use Case: Spark Application Monitoring

  • The dataset section introduces a Spark monitoring use case in which execution anomalies can disrupt deadline-bound analytics jobs and critical operations.The benchmark models this use case using stream-processing applications that analyze WorldCup 1998 website click streams.Source: §3.1 · p. 3
  • The workload runs 10 Spark streaming applications on a four-node cluster, with five or ten randomly selected applications running concurrently.YARN allocates resources and places driver and executor processes across cluster nodes.Source: §3.1 · p. 3
  • The authors retained 93 traces from runs collected over 2.5 months after manually pruning traces affected by cluster downtime or poorly reflected injected anomalies.Each trace averages seven hours and records 2,283 metrics per second from Spark and the operating system.Source: §3.1 · p. 3

3.2 Undisturbed vs. Disturbed Traces

  • Undisturbed executions produced 59 normal traces, but these include occasional metric variations from Spark mechanisms that the authors treat as normal behavior.The authors characterize this as noise in normal data traces.Source: §3.2 · pp. 3–4
  • The authors injected six anomaly types, including bursty or stalled input, CPU contention, and driver or executor failures, into disturbed traces.The events were designed to have visible trace effects, avoid immediate application crashes, and remain traceable to root causes.Source: §3.2 · p. 4
  • Ground truth covers 97 anomaly instances across 34 disturbed traces, labeling root-cause intervals and extended effect intervals through recovery or application crash.Extended effect intervals were manually determined using domain knowledge.Source: §3.2 · p. 4

4 BENCHMARK DESIGN

Exathlon benchmarks semi-supervised anomaly detection on high-dimensional, range-labeled time series and evaluates explanation discovery alongside computational performance. Its criteria span increasingly demanding detection requirements, local and global explanations, and broader benchmark applicability.

4.1 Anomaly Detection (AD) Functionality

  • AD1–AD4 progress from flagging an anomaly to range localization, early detection, and exactly-once reporting, with scores designed to decrease as requirements rise.The benchmark also reports recall by anomaly type when useful, although type-wise precision is not meaningful for typeless false positives.Source: §4.1 · pp. 5–6

4.2 Explanation Discovery (ED) Functionality

  • Exathlon evaluates local explanations for individual anomalies and global explanations for same-type anomalies across applications or runs, using conciseness, consistency, and predictive accuracy.Accuracy is limited to explanations that act as point-based predictive models; feature-importance scores cannot be evaluated this way, and temporal-pattern explanations are largely absent from the literature.Source: §4.2 · pp. 6–7
  • ED consistency measures distinguish stability under subsampling for local explanations from concordance across same-type anomalies for global explanations.Normalized consistency factors out the effect of explanation size; accuracy tests an explanation on similar data not used to generate it.Source: §4.2 · pp. 6–7

4.3 Computational Performance

  • The benchmark measures AD training and inference time and ED explanation time, while scalability tests vary feature dimensionality and data cardinality.Users may reduce cardinality through resampling; the benchmark reports the resulting dimensionality used in AD and ED tasks.Source: §4.3 · p. 7

4.4 Broader Applicability

  • Exathlon’s AD metrics can extend to labeled time-series datasets with either range labels or point labels, while its automated ED metrics cover several prior-work measures but exclude model-specific or expert-dependent metrics.The paper identifies additional applications of its ED metrics to datasets used for classification explanations and, with preprocessing, text or image data.Source: §4.4 · p. 7

5 A FULL PIPELINE FOR EXPLAINABLE AD

Exathlon provides an open, modular pipeline that takes raw traces through data processing, anomaly detection, explanation discovery, and benchmark scoring. Its components allow methods and processing choices to be added or combined.

5 A FULL PIPELINE FOR EXPLAINABLE AD

  • The pipeline covers data partitioning, feature engineering, dimensionality reduction, AD, and ED, with modular components for implementing and evaluating methods.Its open architecture allows different methods to be added and combined.Source: §5 · pp. 7–8
  • By default, the pipeline trains on undisturbed traces and tests on disturbed traces, with optional resampling, PCA-based reduction, curated features, and context-aware rescaling.Rescaling can be performed per trace or dynamically as the AD model processes test data.Source: §5 · p. 8
  • AD models learn normal behavior from sliding windows, derive outlier scores from forecast errors or reconstruction errors, and use unsupervised threshold selection to produce binary predictions.Threshold selection is fit on training data because Exathlon does not provide labeled data for this step.Source: §5 · p. 8
  • The pipeline evaluates AD separation and range-based detection, then generates explanations using either the anomalous instance and reference data or the AD model.It supports model-free and model-dependent explanation methods and evaluates explanations for conciseness, consistency, accuracy, and time.Source: §5 · pp. 8–9

6 EXPERIMENTAL STUDY

The study evaluates selected AD and ED methods on Exathlon using a configured pipeline and benchmark settings. Results expose trade-offs across detection performance, explanation quality, and runtime.

6.1 Experimental Setup

  • The experiments integrated LSTM, AE, and BiGAN for AD and EXstream, MacroBase, and LIME for ED, using reduced data and 19-feature sets.Each deep-learning algorithm had 1.5 days, including hyperparameter tuning, and AD2 was the default evaluation setting.Source: §6.1 · p. 9

6.2 AD Evaluation Results and Discussion

  • Exathlon’s traces varied across contexts and contained some noise, challenging the tested deep-learning AD methods.The study attributes weaker separation across broader contexts partly to normal records overlapping anomaly scores.Source: §6.2 · p. 9
  • Across global separation, T1, T3, and T5 were best separated by LSTM and AE, while BiGAN’s best-separated types were T1, T4, and T5.T6 was poorly separated by most methods because executor failure had limited visibility in the selected features.Source: §6.2 · pp. 9–10, Table 3 · p. 9
  • 0.52 median F1-score for AE was the best among the three methods under FScustom and AD2, but remained limited by unsupervised threshold selection.AE’s strongest separation ability did not eliminate the difficulty of choosing one threshold across traces and anomaly types.Source: Table 4 · p. 10, §6.2 · p. 10
  • AE improved quickly with the first training traces and then plateaued, while LSTM showed a roughly linear trend and BiGAN was less stable.The authors suggest LSTM and BiGAN could benefit from more data and tuning; AE’s limits also reflect benchmark challenges beyond data quantity.Source: §6.2 · p. 11, Figure 5 · p. 11

6.3 ED Evaluation Results and Discussion

  • Feature correlation can make distinct explanations plausible: EXstream selected one feature among correlated ones, whereas MacroBase returned multiple important features.The examples also illustrated LIME’s known inconsistency issues.Source: §6.3 · p. 11, Figure 6 · p. 11
  • EXstream produced the most concise explanations and ran in ~0.01 sec, while MacroBase was more accurate locally and LIME took > 4 min.MacroBase took 0.2-9 sec; LIME’s high latency was described as unsuitable for stream-processing use cases.Source: Table 5 · p. 12, §6.3 · p. 12

7 CONCLUSIONS AND FUTURE DIRECTIONS

Exathlon provides a public benchmark for explainable anomaly detection and uses experiments with selected AD and ED algorithms to demonstrate its utility. The authors identify limits in semi-supervised AD under noisy training data and mixed anomaly types, and call for further AD and ED research and integrated solutions.

7 CONCLUSIONS AND FUTURE DIRECTIONS

  • The experiments reveal limitations of semi-supervised AD methods under noisy training data and mixed anomaly types.The authors call for new AD and ED research and integrated anomaly-and-explanation solutions.Source: §7 · p. 12
  • Exathlon provides a common framework to analyze ED methods’ conciseness, consistency, accuracy, and efficiency.The authors note that comparative analysis tools and studies for ED were lacking.Source: §7 · p. 12
  • Integrated anomaly and explanation discovery requires ED methods capable of discovering range-based explanations, a step toward automated root cause analysis.Source: §7 · p. 12
  • The authors envision Exathlon developing into a collaborative community platform for reproducible research and experimentation.They intend to maintain and extend the platform and welcome community feedback and contributions.Source: §7 · p. 12

B DETAILS ON DATA COLLECTION

This section describes Exathlon’s Spark workload, trace collection and anomaly design, and the labeling of extended effects. Its workload spans varied streaming primitives, while the traces include normal variability and injected anomalies.

B.1 Spark Workload

  • The workload comprises 10 Spark stream-processing applications over WorldCup 1998 click-stream records, with group-by aggregations, varied windows and filters, two joins, and one user-defined function.Applications ran with 5-second batch intervals; outputs were fully saved for four applications and only for the first batch in the others.Source: §B.1 · p. 14

B.2 Undisturbed and Disturbed Traces

  • Undisturbed traces were collected over a month, and 59 were retained; normal traces still show metric variability, including scheduling-delay and processing-time spikes.The authors treat such recurring variations as normal, noting they may arise from Spark checkpointing or HDFS DataNode CPU use.Source: §B.2 · p. 14
  • Injected disturbances cover bursty and stalled input, CPU contention, and driver or executor failures, with effects that can include rising delays, process restarts, or application crashes.The anomaly designs were intended to produce visible effects without an instant application crash and to be traceable to root causes.Source: §B.2 · pp. 14–16

B.3 Extended Effect Intervals

  • Extended effect intervals begin after each root cause interval and end when anomaly detection is no longer deemed helpful, giving the benchmark a combined anomaly interval.The endpoint is set using domain knowledge, such as application restart or metrics returning to normal.Source: §B.3 · p. 16

C DETAILS ON BENCHMARK DESIGN

Exathlon’s AD design evaluates detection functionality through range-based precision and recall and varies learning conditions to test generalization across workload settings.

C.1 Anomaly Detection (AD) Functionality

  • Range-based precision and recall parameters encode anomaly-existence rewards, detection-position bias, and penalties for fragmented detections across AD requirements.For AD3–4 recall, positional bias favors early detection; for AD4, fragmentation is strictly undesirable.Source: §C.1 · p. 16
  • By default, AD models train on undisturbed traces only, treating training data as mostly normal but potentially noisy.The benchmark tests generalization across application, input-rate, and concurrency characteristics.Source: §C.1 · p. 16
  • The learning settings vary whether models train on one or multiple applications and whether they receive many or few examples of workload characteristics.Many-example training can use an earlier normal segment from each disturbed trace, while few-example training uses only the undisturbed traces and reserves disturbed traces for testing.Source: §C.1 · pp. 16–17

C.2 Broader Applicability of AD Evaluation

Exathlon’s AD evaluation can accommodate techniques that score test records from models learned on mostly normal data, and its metrics can extend to other labeled time-series datasets. The authors note that one sliding-window evaluation setting was not included in this paper.

C.2 Broader Applicability of AD Evaluation

  • Any technique that assigns each test record an outlier score from a model learned on mostly normal data can be evaluated, spanning deep-learning, statistical, subsequence, and data-drift methods.Source: §C.2 · p. 17
  • Exathlon’s AD metrics can support labeled datasets with range-based or point-based anomaly labels, and multiple anomaly types can be reported type-wise.Point labels use classical precision and recall; range labels can use the benchmark’s four AD levels.Source: §C.2 · p. 17
  • Trace- and application-level data separation may not fit every use case, but the AD evaluation metrics and methodology also apply when granularity differs or only one level exists.Source: §C.2 · p. 17
  • The authors had not applied their AD metrics to datasets beyond Spark-based traces, although the underlying range-based framework had been tested on other real and artificial datasets.Source: §C.2 · p. 17

C.3 Broader Applicability of ED Evaluation

Exathlon’s ED evaluation uses automated metrics intended to apply across explanation methods and datasets, while its pipeline supports multiple learning settings and final explanation evaluation.

C.3 Broader Applicability of ED Evaluation

  • Conciseness, accuracy, and consistency are the three general ED metrics, selected to reflect explanation size, accuracy where meaningful, and consistency across instances.Consistency also addresses explanations that rely on context-dependent constants, which may perform poorly when applied to other instances.Source: §C.3 · p. 17
  • Consistency is calculated differently according to method inputs: dataset-based explanations use one implementation, while other methods can use window-based evaluation.The pipeline’s LIME implementation uses the window-based approach, and users can add other implementations.Source: §C.3 · p. 18
  • Exathlon’s ED metrics apply directly to some time-series datasets and can extend to classification, text, or image datasets with stated adjustments.Text and image datasets require additional preprocessing; datasets without anomaly-type labels can treat all anomalies as one type.Source: §C.3 · p. 18

D DETAILS ON PIPELINE DESIGN

  • The pipeline partitions traces according to the selected learning setting, separating undisturbed traces for training from disturbed traces for testing.In 1-App settings, partitioning is performed separately for each Spark application; in N-App settings, all 93 traces are considered together.Source: D DETAILS ON PIPELINE DESIGN · p. 18
  • Final ED evaluation collects explanations across test traces and computes metrics using method-specific feature extraction and anomaly sampling procedures.For arbitrary-duration anomalies, the sampling procedure uses 80% to construct an explanation and the remaining 20% with neighboring normal data for ED1 accuracy.Source: D DETAILS ON PIPELINE DESIGN · p. 18

E DETAILS OF OUR EXPERIMENTAL STUDY

The experimental feature set was manually customized using domain knowledge and visualizations, covering Spark driver, executor, and operating-system metrics.

E.1 Customized Feature Set

  • The customized feature set contains 19 features selected using domain knowledge and visualizations: nine driver, six executor, and four OS features.The features characterize different aspects of a running Spark application.Source: §E.1 · p. 18
  • Driver features include processing, scheduling, and total batch delays, plus changes in batch, record, and memory measures.The listed changes include completed batches, received and processed records, the last received batch, total memory, and JVM heap memory.Source: §E.1 · p. 18
  • Executor features include changes in HDFS writes, CPU time, runtime, shuffle records, and JVM heap memory; OS features track idle CPU time across four nodes.Executor measurements are averaged across active executors before differencing; the OS feature is measured for each minicluster node.Source: §E.1 · p. 18

E.2 Details of AD Methods

The AD methods derive outlier scores through forecasting or reconstruction, then select thresholds from training-score distributions using descriptive statistics and a thresholding factor.

E.2 Details of AD Methods

  • LSTM assigns each test record an outlier score based on its relative forecasting error, without averaging the resulting scores.LSTM is a forecasting-based sequential model in this AD setup.Source: §E.2 · p. 19
  • The Autoencoder scores a test window by reconstruction mean squared error and a test record by averaging its enclosing window scores.Its dense architecture learns to reconstruct the input through a lower-dimensional latent representation.Source: §E.2 · p. 19
  • BiGAN scores each test window using the average of encoder-generator reconstruction MSE and discriminator feature loss, then averages enclosing-window scores for each record.The method uses an encoder alongside generator and discriminator networks.Source: §E.2 · p. 19
  • Thresholds are derived from the distribution of AD scores on D2_train using a statistic S1 plus a thresholding factor times statistic S2.The procedure may repeat after removing scores above the first threshold to exclude obvious outliers from the second calculation.Source: §E.2 · p. 19
  • 24 thresholding parameter combinations were evaluated for each AD method, with median final detection performances reported in Table 4.The combinations use STD, MAD, or IQR statistics with thresholding factors in {1.5, 2, 2.5, 3}.Source: §E.2 · p. 19

E.3 Details of ED Methods

The pipeline integrates three explanation-discovery methods that differ in how they identify features and construct explanations.

E.3 Details of ED Methods

  • EXstream ranks features using entropy-based rewards for separating anomalous from normal data, then applies heuristics to select a small feature set.Its false-positive filtering step was omitted because it required extra labeled data based on unavailable user annotations.Source: §E.3 · p. 19
  • MacroBase calculates risk ratios for individual features and feature combinations using frequent itemset mining, with equal-width binning added for numerical inputs.The original method was designed for categorical features.Source: §E.3 · p. 19
  • LIME uses Lasso to identify five important features and learns a local linear model that approximates the original model’s output.The implementation uses LIME’s RecurrentTabularExplainer interface.Source: §E.3 · p. 19

E.4 Additional AD Evaluation Results

The additional AD experiments show that feature extraction and learning settings affect separation ability, while AE and LSTM produce different score patterns. The results also illustrate how these experiments can probe models’ data needs and feature-space effects.

E.4 Additional AD Evaluation Results

  • AE typically produces smooth record-wise outlier scores through overlapping-window averaging, whereas LSTM scores often show discontinuous spikes.The last two LSTM scores were removed from the figure because their high values visually compressed the others.Source: §E.4 · p. 20, Figure 8 · p. 20

Experiment 5 (AD2)

  • Using FSpca significantly reduced global separation ability for all three methods, especially LSTM, because it failed to preserve signals for many anomalies.The comparison is between custom feature selection (FScustom) and PCA applied directly to raw input features (FSpca).Source: §E.4 · p. 20
  • With FSpca, AE and BiGAN maintained or increased separation ability for T6 and T4, possibly because PCA underrepresents low-variance features such as scheduling delay and input rate.FSpca no longer averages across active executor features, while favoring naturally active CPU- and memory-related metrics.Source: §E.4 · p. 20

Experiment 6 (FScustom, AD2)

  • AE and BiGAN’s average separation ability decreased as data variety increased and peeking at test-trace beginnings was disabled, while LSTM performed better in LS2 and LS4.The LSTM pattern was consistent with Experiment 3, where more training data increased its performance even as data variety increased.Source: §E.4 · p. 20
  • Although changes were minor in these examples, varying learning settings can help assess models’ data needs and generalization ability.The authors also describe the benchmark as useful for analyzing how feature-space construction affects overall detection and specific anomaly types.Source: §E.4 · p. 20

Related papers

Loading 2010.05073v3…