PaperlayerView exact revision on arXiv

Source-linked AI summary

GEE: A Gradient-based Explainable Variational Autoencoder for Network Anomaly Detection

Quoc Phong Nguyen, Kar Wai Lim, Dinil Mon Divakaran, Kian Hsiang Low, Mun Choon Chan

arXiv:1903.06661v1cs.LGstat.ML

TL;DR

Network anomaly detection on evolving NetFlow data needs methods that work without extensive labels and provide interpretable alerts. GEE combines an unsupervised VAE with gradient-based fingerprinting, and evaluation on UGR found robust detection across multiple attack types with useful attack representations.

  • Problem

    Network anomaly detection requires handling evolving, high-volume NetFlow data, while supervised methods need large labeled datasets, may miss zero-day attacks, and provide limited explanations.

  • Method

    GEE combines a VAE for unsupervised anomaly detection with gradient-based fingerprinting to explain anomalous traffic and identify relevant features.

  • Results

    Evaluation on the UGR dataset found GEE effective for detecting Spam, Botnet, low-rate DoS, and port-scan attacks, with gradient fingerprints representing attacks.

  • Takeaways & Limitations

    GEE provides both anomaly detection and feature-level explanations, while its gradient fingerprints can also be used directly for anomaly detection.

Abstract

from arXiv · show

This paper looks into the problem of detecting network anomalies by analyzing NetFlow records. While many previous works have used statistical models and machine learning techniques in a supervised way, such solutions have the limitations that they require large amount of labeled data for training and are unlikely to detect zero-day attacks. Existing anomaly detection solutions also do not provide an easy way to explain or identify attacks in the anomalous traffic. To address these limitations, we develop and present GEE, a framework for detecting and explaining anomalies in network traffic. GEE comprises of two components: (i) Variational Autoencoder (VAE) - an unsupervised deep-learning technique for detecting anomalies, and (ii) a gradient-based fingerprinting technique for explaining anomalies. Evaluation of GEE on the recent UGR dataset demonstrates that our approach is effective in detecting different anomalies as well as identifying fingerprints that are good representations of these various attacks.

I. INTRODUCTION

Network anomaly detection must handle evolving, high-volume NetFlow data while giving analysts actionable explanations. GEE addresses these challenges with unsupervised VAE detection and gradient-based anomaly fingerprints.

  • Network behavior evolves across users, applications, locations, and time, limiting the utility of rule-based detection systems.
  • NetFlow reduces traffic volume compared with raw packet capture, enabling analysis of metadata from high-capacity networks.
  • SOC analysts need anomaly detectors to report timing, type, and suspicious flows rather than merely flagging anomalies.
  • Existing supervised approaches require costly labeled data and are unlikely to detect unknown or zero-day attacks.
  • GEE combines an unsupervised VAE for anomaly detection with gradient-based fingerprinting to explain detected threats and attacks.
  • The framework is presented as scalable in data size and feature dimension, and evaluation identifies Spam, Botnet, low-rate DoS, and port-scan attacks.

II. RELATED WORK

Prior network anomaly-detection research spans statistical, dimensionality-reduction, supervised, and unsupervised learning methods. GEE differs by using unlabeled deep learning for detection and gradients to explain relevant features.

  • Prior work includes rule-based, information-theoretic, signal-analysis, statistical, data-mining, and machine-learning approaches for network anomaly detection.
  • PCA separates traffic into normal and anomalous subspaces, but its reduced components do not directly interpret original features.
  • Statistical decision frameworks model normal traffic and test deviations, but parameter learning requires continuously available labeled data.
  • Supervised Random Forest detection of C&C servers relies on reputation systems, which can reduce accuracy for recent or evasive botnets.
  • Some prior work uses autoencoders to extract compressed features before training supervised classifiers, while other deep models target detection directly.
  • GEE uses unsupervised deep learning without labeled information and adds gradient-based explanations of detected anomalies and relevant features.

III. UNSUPERVISED DEEP LEARNING MODELS

The paper uses deep models to represent network behavior and focuses on VAE-based anomaly detection. Its autoencoder architecture maps features through latent representations, while gradients support interpretation.

  • Deep learning models can handle large datasets and streaming training, while VAEs are described as robust to noisy network-flow data.
  • The VAE is a probabilistic generalization of the AE and is selected for flexible, robust anomaly detection and gradient-based interpretation.
  • An autoencoder contains input, latent-representation, and output layers; its encoder maps features to latent space and its decoder reconstructs them.
  • The illustrated architecture uses a 100-unit latent layer and encoder and decoder hidden layers sized 512, 512, and 1,024.
  • Each hidden layer computes a nonlinear transformation from the previous layer using learned weights, biases, and an activation function.
  • Unlike the deterministic AE, the VAE is a generative model capable of generating new data based on the training-data distribution.

B. Variational Autoencoder (VAE)

The VAE treats latent representations probabilistically, learns their conditional distributions, and trains by optimizing a variational lower bound with stochastic sampling.

  • Unlike an autoencoder, the VAE models the latent representation as random variables conditional on the inputs.The encoder computes parameters for conditional latent distributions, which are then sampled for decoding.
  • The conditional latent distributions are generally Gaussian, with encoder outputs providing their means and variances.The latent representation is sampled from these distributions before decoding.
  • The VAE uses the same hidden-layer computation as an autoencoder, with ReLU activations in intermediate layers and a linear output activation.
  • VAE training optimizes a variational lower bound rather than a standard objective such as mean square error.Stochastic sampling is used to approximate the optimization process.

IV. GEE: ANOMALY DETECTION FRAMEWORK

GEE extracts aggregated NetFlow features for source IP addresses in three-minute windows, feeds them to a VAE, and uses reconstruction errors to detect anomalous behavior.

  • GEE extracts information such as average packet size and destination-port entropy before feeding the resulting features into a VAE.The VAE learns the network’s normal behavior from these features.
  • NetFlow records are grouped into three-minute sliding windows by source IP address to form aggregated data points.This representation supports identifying both the offending IP address and the anomaly’s time window.
  • The framework extracts 53 aggregated features, including distribution statistics, protocol and address entropies, and proportions of ports used by common applications.
  • Data points containing fewer than 10 flows are removed, and the retained features are scaled or normalized before VAE input.This preprocessing is intended to reduce training-data noise and ensure meaningful statistics.

B. Unsupervised Learning of VAE

GEE trains the VAE with automatic differentiation and uses reconstruction error for detection, while gradients provide explanations of which features contribute to anomalous behavior.

  • B. Unsupervised Learning of VAE: VAE training samples latent representations to compute a variational lower bound containing KL-divergence and expectation terms.The objective is optimized with stochastic techniques because gradients are available through automatic differentiation.
  • B. Unsupervised Learning of VAE: The framework does not use label information during VAE training.
  • C. Anomaly Detection: An IP address and time window are flagged as abnormal when the input features have high reconstruction error.The reconstruction error is the mean square difference between observed features and their expected reconstruction.
  • D. Gradient-based Explanation for Anomalies: The gradient analysis provides an explanation for why a data point is flagged rather than evaluating detection accuracy alone.
  • D. Gradient-based Explanation for Anomalies: Gradients of the variational lower bound with respect to each feature indicate which feature values are significantly abnormal.Gradient patterns can cluster flagged anomalies by similar behavior and can support fingerprinting when attack labels are available.

V. DATASET AND EVALUATION

GEE is evaluated on UGR16, a large-scale Tier 3 ISP NetFlow dataset containing background traffic and multiple attack categories. Reconstruction-error distributions overlap for several attacks, while blacklist behavior is indistinguishable from background traffic.

  • UGR16 contains real NetFlow traffic from a Tier 3 ISP, including broad user activity and attack traffic from generated, labeled real, and controlled botnet sources.
  • The evaluation includes low-rate DoS, port scanning, botnet, spam, and blacklist categories.The blacklist category contains flows involving IP addresses listed in public blacklists.
  • Five UGR days are selected, with two Saturdays for training and three Friday or Sunday days for testing.Testing evaluates generalization to a new day using statistics learned from another training day.
  • 1,957,711 data points were processed on March 18, 2,954,983 on March 20, and 2,878,422 on July 31.A point is assigned an attack type when more than half of its flows carry that label within the three-minute aggregation.
  • Blacklist behavior is indistinguishable from background traffic, while spam, botnet, DoS, and scanning show some overlap but permit a rough cutoff.

A. Baseline

GEE is compared with an autoencoder and Gaussian-based thresholding using shared extracted features, with reconstruction error used to flag anomalies in the autoencoder baseline.

  • A. Baseline: The baseline compares GEE’s VAE with an AE sharing its architecture and with a Gaussian Based Thresholding approach.Both AE and GBT are trained on the same extracted features.
  • A. Baseline: AE flags data points with large reconstruction error as anomalies.The AE minimizes mean square reconstruction error during training.
  • A. Baseline: GBT models feature behavior with independent Gaussian distributions and uses a thresholded product of average, standard deviation, and maximum feature Z-scores.A data point exceeding the selected score threshold is considered anomalous.

B. ROC Performance on Anomaly Detection

The evaluation compares anomaly-detection ROC performance across models and uses VAE gradients as attack fingerprints. VAE is reported as the most robust detector, while normalized gradient fingerprints represent attacks and can improve detection over reconstruction error.

  • ROC evaluation: ROC evaluation varies the anomaly-detection threshold and compares true-positive rates against false-positive rates on training and testing data.Training results assess finding anomalies from same-day statistics, whereas testing results assess generalization to a new day.
  • ROC evaluation: VAE has the best overall AUC and is the most robust model across attack types, whereas GBT and AE perform unevenly across attacks.GBT works well for high-volume spam and port scans but less well for botnet and low-rate DoS; AE performs poorly on spam and low-rate DoS.
  • Gradient fingerprints: The gradient analysis identifies a small subset of features with large gradients that explain why flows are detected as anomalous.For spam, five features have more positive gradients and four have much negative gradients than the learned normal.
  • Gradient fingerprints: Gradient-feature combinations form fingerprints that can identify or cluster similar attacks, including the similar fingerprints observed for Scan11 and Scan44.The paper reports that an unseen gradient fingerprint may indicate a new attack.
  • Fingerprint-based detection: Normalized gradient fingerprints are good attack representations and improve anomaly detection over reconstruction error, although results are slightly worse for botnet.The paper attributes the botnet result potentially to multiple underlying anomaly patterns that averaging may not capture well.

D. Clustering of Anomalies

The paper clusters VAE-generated gradients to group attacks, finding that attacks generally concentrate in a small number of behavioral clusters that analysts can prioritize.

  • D. Clustering of Anomalies: The clustering approach groups attacks using VAE-generated gradients, with effective clustering expected to confine attacks to relatively few clusters.The experiment uses k-means clustering with k = 100 on the training dataset.
  • D. Clustering of Anomalies: 92.4% of DoS attacks appear in two clusters, while the remaining 7.6% appear in four additional clusters.The main DoS clusters are c82 and c84.
  • D. Clustering of Anomalies: 74.3% of spam attacks appear in two clusters, while 25.7% appear across another 11 clusters.The main spam clusters are c11 and c15.
  • D. Clustering of Anomalies: The concentration of attacks in a small number of clusters lets analysts focus on a limited subset when studying particular attack types.

E. Processing Overhead of VAE

GEE’s VAE can support real-time anomaly detection, with most processing cost concentrated in NetFlow feature extraction rather than model training or testing. The framework also uses gradients to explain flagged anomalies, while extending detection to multiple data sources remains future work.

  • Processing overhead: 25 seconds per 3-minute window was required for feature extraction on UGR data, making it the dominant processing cost.The reported setup processed about 200k records per window.
  • Processing overhead: 10ms per minibatch was sufficient for VAE training, while testing abnormality took 20ms for the same 300 aggregated data points.These timings support real-time anomaly detection with the described setup.
  • Explainability: GEE uses VAE gradient information to provide explanations for flagged anomalies and to form gradient-based fingerprints.The fingerprints were also evaluated directly for anomaly detection and achieved better overall performance.
  • Future research: Using a conditional VAE to incorporate auxiliary labels from multiple data sources with different behaviors is identified as future research.This extension would address settings involving heterogeneous data sources.
Loading 1903.06661v1…