PaperlayerView exact revision on arXiv

Source-linked AI summary

Time Series Anomaly Detection; Detection of anomalous drops with limited features and sparse examples in noisy highly periodic data

Dominique T. Shipmon, Jason M. Gurevitch, Paolo M. Piselli, Stephen T. Edwards

arXiv:1708.03665v1stat.MLcs.LG

TL;DR

Detecting anomalous drops in traffic streams is challenging when streams are uncorrelated and have limited features. The paper compares model predictions with observed values using two anomaly rules, and reports that combining the rules was more effective than using either alone. The authors conclude that the rules worked across most models, while model choice mattered less for this problem.

  • Problem

    The streams were largely uncorrelated, and the authors had to detect anomalies using models trained separately for each stream.

    Source: B. About the Problem · p. 1, §III · p. 2

  • Method

    The authors trained models to predict each stream, compared predictions with observed values, and combined accumulator and Gaussian tail probability rules to detect anomalies.

    Source: B. About the Problem · p. 1, §IV · p. 3

  • Results

    The intersection of the accumulator and Gaussian tail probability rules was more effective at identifying anomalous points than either rule alone.

    Source: §IV · p. 3

  • Takeaways & Limitations

    For this problem, anomaly-detection rules mattered more than the particular model, and the rules were effective on most models.

    Source: B. About the Problem · p. 2

    With limited features, deep learning provided no significant advantage over simpler periodic models, and the methods focused only on drops below predictions.

    Source: §VIII · p. 7

Abstract

from arXiv · show

Google uses continuous streams of data from industry partners in order to deliver accurate results to users. Unexpected drops in traffic can be an indication of an underlying issue and may be an early warning that remedial action may be necessary. Detecting such drops is non-trivial because streams are variable and noisy, with roughly regular spikes (in many different shapes) in traffic data. We investigated the question of whether or not we can predict anomalies in these data streams. Our goal is to utilize Machine Learning and statistical approaches to classify anomalous drops in periodic, but noisy, traffic patterns. Since we do not have a large body of labeled examples to directly apply supervised learning for anomaly classification, we approached the problem in two parts. First we used TensorFlow to train our various models including DNNs, RNNs, and LSTMs to perform regression and predict the expected value in the time series. Secondly we created anomaly detection rules that compared the actual values to predicted values. Since the problem requires finding sustained anomalies, rather than just short delays or momentary inactivity in the data, our two detection methods focused on continuous sections of activity rather than just single points. We tried multiple combinations of our models and rules and found that using the intersection of our two anomaly detection methods proved to be an effective method of detecting anomalies on almost all of our models. In the process we also found that not all data fell within our experimental assumptions, as one data stream had no periodicity, and therefore no time based model could predict it.

I. INTRODUCTION

The paper studies detecting sustained drops in noisy traffic streams with limited features and no accurate anomaly labels. It compares prediction models with rules that identify anomalies from differences between predictions and observed values.

A. About the Data

  • The dataset contained 14 streams sampled every five minutes, with only timestamps and bytes per second and no accurate anomaly labels.The goal was to find missing daily traffic bursts or unexpected throughput decreases without flagging regular low or zero values.Source: A. About the Data · p. 1

B. About the Problem

  • Classification approaches often need many labeled anomalies and features, which are unavailable for this time-series problem.The paper therefore focuses on regression-based approaches for periodic data with limited features and few labeled anomalies.Source: B. About the Problem · p. 1
  • The authors trained a separate model for each stream, compared its predictions with observed values, and applied anomaly rules to the differences.They note that the offline approaches could be adapted to online training, prediction, and detection because they require past or current points.Source: B. About the Problem · p. 1
  • A simple distance threshold can generate false positives on noisy data, motivating rules that assess sequences of activity rather than isolated points.The two rules use an accumulator for continuous outages and a probabilistic approach based on Ahmad and Purdy.Source: B. About the Problem · p. 2
  • The anomaly rules were effective on most models, and the detection method mattered more than the model itself.The authors report that the rules worked across most models, which were able to fit the test and validation data well.Source: B. About the Problem · p. 2

II. RELATED WORK

Prior anomaly-detection approaches include classifiers and statistical methods, but their data requirements can limit their use. This paper examines regression-based machine learning for periodic time series with limited features and few labeled anomalies.

II. RELATED WORK

  • Classification is widely used for anomaly detection, but sufficient and diverse labeled anomalies may not be available.Some alternatives, including clustering and isolation forests, also rely on labels and work best with many features.Source: §II · p. 2
  • The paper focuses on regression-based machine learning for periodic time series with limited features and few labeled anomaly examples.The authors consider DNNs, RNNs, LSTMs, and other model types.Source: §II · p. 2

III. DATA PREPROCESSING AND INITIAL ANALYTICS

The data offered only timestamps as input features, and streams were largely uncorrelated, so the authors engineered features within each stream. They transformed time into calendar features and tested derivatives of past values.

III. DATA PREPROCESSING AND INITIAL ANALYTICS

  • The only original input feature was the Unix timestamp; bytes received served as the label, leaving one input feature without feature engineering.Because the label could not also be used as an input, the authors sought features not linearly correlated with the timestamp.Source: §III · p. 2
  • Almost none of the streams were linearly correlated, so a composite based on principal components would not add information.The authors considered averages and principal components as possible composite features.Source: §III · p. 2
  • For each stream, the authors encoded weekday and hour as one-hot features, added linear minutes, and tested past-label derivatives as features.The derivative was chosen partly because it could be used in online monitoring; simulated continuous anomalies still produced predicted spikes, suggesting the models were not overfitting on it.Source: §III · p. 2
  • The authors normalized bytes per second to 0–1 for all models, which made neural-network inputs more suitable and sped up training.The bounded values also allowed experiments with activation functions such as tanh and sigmoid.Source: §III · pp. 2–3

IV. DETECTION RULES

The authors combined an accumulator rule with a Gaussian tail probability rule, finding their intersection more effective than either alone. The accumulator was designed to require repeated local anomalies and was tuned to reduce false positives after peaks.

IV. DETECTION RULES

  • The intersection of the accumulator and Gaussian tail probability rules was more effective at identifying anomalous points than either rule alone.Different streams favored different rules, motivating the hybrid combination.Source: §IV · p. 3

A. Accumulator

  • The accumulator grows on local anomalies and shrinks on normal values, requiring repeated anomalies within a short period before signaling.It grows by one for an anomaly and shrinks by two for a non-anomalous value; its cap is set relative to the signaling threshold.Source: A. Accumulator · p. 3
  • A local anomaly is a value sufficiently below its prediction while remaining above a hardcoded threshold.The rule is parametric, and the chosen values may not be optimal for other datasets.Source: A. Accumulator · p. 3
  • A rolling-variance rule treated values outside 20 times the rolling variance from prediction as outages, but it was slightly less effective than the simpler alternative.The variance-based threshold was intended to allow more noise in noisier regions.Source: A. Accumulator · p. 3
  • The authors added a peak-value parameter to reduce false positives after peaks, decrementing the accumulator and allowing it to fall below zero.The accumulator then decays back to zero if further non-anomalous points occur.Source: A. Accumulator · p. 3
  • The general accumulator settings used a non-anomalous threshold of 0.3, a peak value of 0.35, an accumulator threshold of 15, and a local-outage delta of 0.1.These settings were used across the models.Source: A. Accumulator · p. 3

B. Gaussian Tail Probability

The Gaussian tail probability rule scores low-activity prediction errors, then uses rolling statistics to produce an anomaly likelihood.

B. Gaussian Tail Probability

V. PREDICTION MODELS

The authors tested prediction models on artificial periodic series and real monthly traffic streams, evaluating them with anomaly-detection rules.

V. PREDICTION MODELS

  • Two artificial tests used a 288-point sine wave and a stepwise sine-based series to assess prediction of periodic patterns and regular rule changes.Source: §V · p. 3
  • Models trained on April data and tested on May data; 13 of 14 traffic datasets were periodic, with variation in spike timing, values, and noise.Source: §V · p. 4
  • Each model was evaluated with the accumulator rule, Numenta’s tail probability rule, and their intersection; the Fourier model used Adagrad while the others used Adam.Source: §V · p. 4

A. Hyperparameters

  • Hyperparameters were chosen experimentally because optimization based on mean squared error did not produce useful models.Source: A. Hyperparameters · p. 4

B. Baseline (Threshold model)

The section compares a fixed-threshold baseline with a Fourier-series baseline for periodic data, reporting rule behavior and model settings.

B. Baseline (Threshold model)

  • The threshold baseline hardcodes 0.065 as a floor that data should not fall below, rather than fitting distance to all points.Source: B. Baseline (Threshold model) · p. 4
  • The accumulator could be tuned to detect abnormally low values, while sliding windows were less effective; the accumulator delta was reduced to 0.05 for a local outage.Source: B. Baseline (Threshold model) · p. 4

C. Fourier Series

  • The Fourier baseline assumes periodicity and uses sine terms with phase added to the harmonic argument; amplitude, phase, and harmonic count are model parameters.Source: C. Fourier Series · p. 4
  • The Fourier model used a weekly period, 448 harmonics, MSE loss, Adagrad, learning rate 0.5, and 3000 training steps.Daily and weekly periods were similarly effective, so the longer period was selected; the reported loss was fairly low.Source: C. Fourier Series · p. 4

D. Deep Neural Network (DNN)

The DNN was described as effective and stable, while the tested RNN did not significantly outperform it despite its temporal-state design.

D. Deep Neural Network (DNN)

  • The DNN used a fully connected TFLearn DNNRegressor with ten 200-neuron layers, ReLU6 activations, and Adam training for 1200 steps.Its batch size was 200 and learning rate was 0.0001.Source: D. Deep Neural Network (DNN) · p. 5
  • The DNN model was reported as quite effective and stable, possibly in part because of optimizations in the TFLearn implementation.Source: D. Deep Neural Network (DNN) · p. 5

E. Recurrent Neural Network (RNN)

F. Long Short-Term Memory (LSTM)

The LSTM added a more complex memory cell, but did not perform significantly better than the DNN. Evaluation used both simulated noise anomalies and an observed missing-peak anomaly.

F. Long Short-Term Memory (LSTM)

  • The LSTM used 70 hidden units, 10 layers, ELU activations, a 0.001 learning rate, and 200-example batches for 2,500 steps, but did not significantly outperform the DNN.It used the standard non-peephole LSTM cell and a single linear output layer.Source: F. Long Short-Term Memory (LSTM) · p. 5

A. Anomaly Simulation

  • The evaluation included a simulated anomaly created by adding random noise around a specific value for the anomaly’s duration.Source: A. Anomaly Simulation · p. 6
  • An observed late-May anomaly consisted of a missing peak amid normal data, so its start and end were estimated; the models detected both this and the simulated anomaly.Source: A. Anomaly Simulation · p. 6

B. Test/Train Split

The authors held out a different month to test predictions on realistic contiguous sequences. Their findings highlight a trade-off between false positives and true positives, similar performance across neural models, and the influence of visual patterns and labeling choices.

B. Test/Train Split

  • Testing on a different month rather than randomly selected points provided a realistic setting for rules needing contiguous prior data and models with temporal awareness.Source: B. Test/Train Split · p. 6

VII. FINDINGS

  • The intersection of the accumulator and tail-probability rules reduced false positives in every instance, but also decreased the true-positive rate; neural-network detection matrices differed little.Source: §VII · p. 6
  • The observed anomaly’s boundaries were estimated, and the authors note that measured performance depends on ground-truth labeling guidelines.Source: §VII · p. 6
  • Training and evaluation MSE scores were not significantly different, which the authors interpret as evidence against overfitting and for reasonable generalization.Source: §VII · p. 6
  • Graphical review found four Fourier false positives outside the two anomalies; DNN and LSTM errors clustered after peaks, a pattern the accumulator rule could be modified to resist.Source: §VII · p. 6
  • The methods detected anomalies in 13 of 14 streams; the remaining stream was nonperiodic, outside the models’ time-based assumptions.Source: §VII · p. 6

VIII. CONCLUSION

The authors found their models effective with the anomaly rules, with Fourier slightly ahead, though stream characteristics may explain that edge. They conclude that limited features constrain deep learning’s advantage over simpler periodic models, while combining distinct detection rules proved robust.

VIII. CONCLUSION

  • All models were effective with the anomaly rules, while Fourier was slightly more effective, possibly because of differences among data streams.Source: §VIII · p. 6
  • With limited features, deep learning showed no significant advantage over simpler periodic models; more features could yield more accurate deep-learning results.Source: §VIII · p. 7
  • The intersection of statistically backed and user-adjustable anomaly rules provided a robust detection method, though the rules were tailored to drops below predictions.Source: §VIII · p. 7

Related papers

Loading 1708.03665v1…