PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Real-time Anomaly Detection Algorithms - the Numenta Anomaly Benchmark

Alexander Lavin, Subutai Ahmad

arXiv:1510.03336v4cs.AIcs.LG

TL;DR

Streaming anomaly detection requires online processing, adaptation, and early warnings, but existing benchmarks do not adequately capture these requirements. NAB combines labeled real-world time series with time-aware, application-specific scoring and evaluates several detectors. The paper reports HTM outperforming the other tested detectors while also finding room for improvement.

  • Problem

    Real-time anomaly detection lacks benchmarks that adequately evaluate detectors processing streaming data while learning and making predictions.

  • Method

    NAB combines labeled real-world time-series data, anomaly-window scoring that rewards early detection, application profiles, and an open-source code library.

  • Results

    HTM outperformed the other tested anomaly detectors, while the analyses identified strengths and shortcomings across algorithms.

  • Takeaways & Limitations

    NAB provides a controlled, repeatable framework for comparing real-time anomaly detection algorithms across application requirements.

Abstract

from arXiv · show

Much of the world's data is streaming, time-series data, where anomalies give significant information in critical situations; examples abound in domains such as finance, IT, security, medical, and energy. Yet detecting anomalies in streaming data is a difficult task, requiring detectors to process data in real-time, not batches, and learn while simultaneously making predictions. There are no benchmarks to adequately test and score the efficacy of real-time anomaly detectors. Here we propose the Numenta Anomaly Benchmark (NAB), which attempts to provide a controlled and repeatable environment of open-source tools to test and measure anomaly detection algorithms on streaming data. The perfect detector would detect all anomalies as soon as possible, trigger no false alarms, work with real-world time-series data across a variety of domains, and automatically adapt to changing statistics. Rewarding these characteristics is formalized in NAB, using a scoring algorithm designed for streaming data. NAB evaluates detectors on a benchmark dataset with labeled, real-world time-series data. We present these components, and give results and analyses for several open source, commercially-used algorithms. The goal for NAB is to provide a standard, open source framework with which the research community can compare and evaluate different algorithms for detecting anomalies in streaming data.

I. INTRODUCTION

Real-time anomaly detection requires streaming, unsupervised systems that adapt while predicting, but existing benchmarks do not adequately measure these requirements. NAB addresses this gap with a controlled evaluation framework for streaming detectors.

  • Streaming detectors must process data in real time, operate with limited human intervention, and learn changing statistics while predicting.
  • Static-data benchmarks and precision-recall metrics fail to capture early detection, continuous learning, and other real-time requirements.
  • NAB provides a controlled, repeatable environment for testing and measuring anomaly detection algorithms on streaming data.
  • The paper introduces NAB’s scoring system and dataset, then evaluates and analyzes four real-time algorithms.

II. NUMENTA ANOMALY BENCHMARK

NAB defines streaming anomalies as patterns that depart from a stream’s past behavior and aims to represent their variety and detection challenges. The benchmark’s dataset and scoring system are presented as its two main components.

  • NAB defines an anomaly as a streaming pattern that does not conform to the stream’s past behavior.
  • This definition includes both point anomalies and temporal anomalies, covering varied anomalous behavior in real-world streams.
  • NAB organizes its benchmark around a dataset and a scoring system designed to evaluate real-world streaming anomaly detectors.

A. Benchmark Dataset

The NAB dataset uses labeled, real-world time series spanning domains and anomaly types, with requirements that reflect streaming challenges. Its hand-labeling process and known anomaly causes support its intended role as a standard benchmark.

  • NAB’s current data consists of time series with timestamps and a single scalar value per row, covering varied anomaly types, metrics, noise, and new normal patterns.
  • The corpus spans domains including IT, industrial sensors, and social media because anomaly significance differs across applications.
  • NAB labels anomalies by hand using documented rules and combines labeler agreement into ground-truth labels.
  • The dataset includes real-world anomalies with known causes and is proposed as a standard benchmark for streaming applications.
  • Figure 1 illustrates subtle, persistent, spiking, and prolonged anomalies in representative NAB data streams.

B. Scoring Real-Time Anomaly Detectors

NAB scores detectors against real-time requirements by combining anomaly windows, temporally sensitive scoring, and application profiles. The system rewards early detections, penalizes false alarms, and adapts evaluation to different application priorities.

  • The ideal detector finds all anomalies early, avoids false alarms, operates without look-ahead, and adapts automatically across datasets.
  • NAB uses anomaly windows and a scoring function to assign credit for detections and penalties for false positives and false negatives.
  • Standard precision and recall do not incorporate time or reward early detection, so they are insufficient for real-time anomaly evaluation.
  • The first 15% of each data file is a probationary period during which detectors may learn patterns without being tested.
  • Windows are designed to reward early and near-after-anomaly detections while avoiding excessive reinforcement of random or unreliable detections.
  • NAB provides standard, low-false-positive, and low-false-negative application profiles with adjustable relative weights.
  • Figure 3 counts only the earliest detection within an anomaly window and makes later distant false positives more detrimental.
  • The scoring system evaluates real-time performance, prefers earlier detection, penalizes spam, and assigns realistic costs to classification outcomes.

C. Computing NAB Score: Details

NAB computes scores by weighting detections according to their timing within anomaly windows, aggregating true and false detections across files, and normalizing against perfect and null detectors.

  • NAB assigns each detection a scaled sigmoidal weight based on its relative position within an anomaly window.Detections near the window’s end receive smaller rewards, while detections outside the window are penalized relative to the preceding window.
  • Every detection outside an anomaly window counts as a false positive, while a completely missed window counts as a false negative.False negatives receive the profile’s false-negative weight.
  • The raw score for each data file sums weighted true and false detections and subtracts weighted penalties for missed anomaly windows.
  • The benchmark score for an algorithm sums raw scores across all data files before normalization.
  • The normalized NAB score is scaled using perfect- and null-detector raw scores, with a maximum of 100 and a null-detector score of 0.

D. Other NAB Details

NAB is designed as an open-source community benchmark whose datasets, algorithms, versioning, and issue tracking can evolve through public contributions.

  • NAB supports academic and industry researchers as a community tool for contributing data and online anomaly detection algorithms.
  • The benchmark is released under the permissive MIT License with versioning and public issue tracking for transparent changes and dataset additions.

III. ALGORITHMS TESTED

NAB tests four primary real-time anomaly detectors alongside control detectors, using threshold optimization and algorithm-specific streaming approaches.

  • Algorithms and controls: The initial evaluation includes HTM, Etsy Skyline, two Twitter algorithms, and null, perfect, and random control detectors.The Twitter methods are AnomalyDetectionTs and AnomalyDetectionVec.
  • Numenta HTM: HTM models temporal sequences, compares predictions with subsequent values, and produces an instantaneous anomaly score.
  • Numenta HTM: HTM continuously learns, handles predictable and highly unpredictable streams, and adapts automatically to changing data statistics without retraining.
  • Etsy Skyline: Skyline combines simple detectors, including moving-average, least-squares, and histogram deviations, through a voting scheme.
  • Twitter algorithms: Twitter’s methods combine Generalized ESD, robust statistical metrics, and piecewise approximation for trend detection; AnomalyDetectionVec requires manually tuned periodicity.
  • Control detectors: The null detector outputs 0.5 constantly, the perfect detector outputs only earliest true positives, and the random detector supplies chance-level intuition.
  • Thresholding: NAB thresholds each algorithm’s anomaly scores using an automated hill-climbing search that maximizes the NAB scoring function.The search optimizes thresholds over the full NAB dataset.

A. Overall NAB Scores

Across three application profiles, HTM achieves the strongest NAB scores, while Etsy and Twitter also outperform chance consistently.

  • HTM achieves the best overall scores across the three application profiles, followed by Etsy and Twitter.
  • Etsy and Twitter perform significantly better than chance across all three application profiles despite trailing HTM.
  • Each algorithm’s parameters were optimized to yield the best possible NAB scores.
  • Random detections do not produce scores approximately equal to zero because of the NAB score optimization step.

B. Results Analysis

The results analysis shows that detectors differ in false-positive behavior, adaptation to changing patterns, and ability to detect temporal anomalies early. HTM and Skyline often identify changes sooner or adapt to new normal behavior, while ADVec can continue flagging anomalies after a shift.

  • Error analysis: 1161 Skyline detections versus 387 for HTM and 612 for ADVec show that detection volume affects false-positive exposure.The analysis also reports that Skyline increased the most under the Reward low FN profile.
  • Production-server CPU usage: Skyline and HTM detect a sustained CPU-usage shift and adapt to the new normal, whereas ADVec continues generating anomalies for several days.All algorithms detect the initial simple spike.
  • Temporal anomalies: Only HTM detects the first machine-temperature anomaly, a purely temporal anomaly preceding a catastrophic failure.All detectors identify the second anomaly, with HTM and Skyline detecting it hours before ADVec; HTM and Skyline each produce a false positive.
  • Early detection: Three hours earlier, HTM detects an anomaly than Skyline and ADVec because of a subtle shift in metric dynamics.All three detectors ultimately identify the anomaly.
  • Early detection: Temporal and sequence-based techniques can detect streaming anomalies before a large, easily visible shift occurs.The paper presents this pattern as a potential source of early warning in production applications.

V. CONCLUSION AND FUTURE WORK

NAB provides a controlled framework built from real-world labeled time-series data, application-oriented scoring, and an open-source code library. Evaluations show HTM outperforming the other tested detectors, while also revealing substantial room for improvement.

  • Benchmark dataset: NAB offers a benchmark dataset of real-world time-series files from varied domains, labeled with anomalies.The paper emphasizes the value of accessible real data for streaming applications.
  • Performance evaluation: NAB’s scoring philosophy incorporates time and explicitly rewards earlier anomaly detection for real-time applications.Application profiles let developers test algorithms against specific requirements.
  • Code library: The open-source NAB repository includes data, algorithms, and documentation for reproducible evaluation.
  • Evaluation results: HTM outperforms the other tested anomaly detectors, but the results still show room for improvement.Analyzing the results identifies strengths and shortcomings across the evaluated algorithms.
  • Future work: Future NAB development will add more real-world files and incorporate multivariate anomaly detection and categorical data.The stated longer-term aim is broader testing and development of anomaly detectors for real-time streaming applications.
Loading 1510.03336v4…