PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking datasets for Anomaly-based Network Intrusion Detection: KDD CUP 99 alternatives

Abhishek Divekar, Meet Parekh, Vaibhav Savla, Rudra Mishra, Mahesh Shirole

arXiv:1811.05372v1cs.LGcs.AIcs.CRstat.ML

TL;DR

KDD-99 and NSL-KDD provide imperfect benchmarks because skewed classes obscure poor minority detection, while NSL-KDD also misses low-footprint attacks. The paper benchmarks standard classifiers across these datasets and balanced variants, finding improved minority performance after resampling and strong binary performance from UNSW-NB15. It concludes that UNSW-NB15 can substitute for older benchmarks, while further pipeline optimization remains possible.

  • Problem

    KDD-99 and NSL-KDD have skewed classes that obscure poor detection of minority attacks, creating a benchmarking concern for anomaly-based NIDS.

  • Method

    The paper benchmarks common classifiers on KDD-99 alternatives, applying SMOTE and random undersampling to construct balanced NSL-SMOTE and NB15-SMOTE datasets.

  • Results

    UNSW-NB15 matches or exceeds NSL-KDD in binary Weighted F1-Score, while NSL-SMOTE improves mean minority F1-Score over NSL-KDD.

  • Takeaways & Limitations

    UNSW-NB15 can satisfactorily substitute KDD-99 and NSL-KDD for machine-learning anomaly-based NIDS research.

  • Takeaways & Limitations

    Further performance optimization remains possible through alternative techniques across the machine-learning pipeline.

Abstract

from arXiv · show

Machine Learning has been steadily gaining traction for its use in Anomaly-based Network Intrusion Detection Systems (A-NIDS). Research into this domain is frequently performed using the KDD~CUP~99 dataset as a benchmark. Several studies question its usability while constructing a contemporary NIDS, due to the skewed response distribution, non-stationarity, and failure to incorporate modern attacks. In this paper, we compare the performance for KDD-99 alternatives when trained using classification models commonly found in literature: Neural Network, Support Vector Machine, Decision Tree, Random Forest, Naive Bayes and K-Means. Applying the SMOTE oversampling technique and random undersampling, we create a balanced version of NSL-KDD and prove that skewed target classes in KDD-99 and NSL-KDD hamper the efficacy of classifiers on minority classes (U2R and R2L), leading to possible security risks. We explore UNSW-NB15, a modern substitute to KDD-99 with greater uniformity of pattern distribution. We benchmark this dataset before and after SMOTE oversampling to observe the effect on minority performance. Our results indicate that classifiers trained on UNSW-NB15 match or better the Weighted F1-Score of those trained on NSL-KDD and KDD-99 in the binary case, thus advocating UNSW-NB15 as a modern substitute to these datasets.

I. INTRODUCTION

A-NIDS research uses machine learning to distinguish benign from suspicious traffic, but benchmark selection is critical because KDD-99 has age, imbalance, non-stationarity, redundancy, and irrelevant-feature problems. The paper evaluates NSL-KDD variants and UNSW-NB15 as alternatives using standard classifiers and resampling methods.

  • A-NIDS uses learned traffic patterns to flag suspicious network activity, complementing signature-based misuse detection.
  • KDD-99 remains a common benchmark despite age, skewed targets, train-test non-stationarity, redundancy, and irrelevant features.
  • Minority-class performance is poor in KDD-99, and NSL-KDD does not eliminate this obstacle.
  • NSL-SMOTE combines oversampling and undersampling to balance NSL-KDD and improve performance on the same minority classes.
  • The paper benchmarks UNSW-NB15 and its oversampled variant against established classifiers and binary versions of alternative datasets.

1) Genesis:

KDD-99 derives from simulated network traffic collected at MIT Lincoln Laboratory in 1998 and 1999, then filtered for the KDD competition. Its five attack-related target categories and substantial redundancy shape the benchmark used here.

  • 1) Genesis:: The dataset contains Normal, DoS, U2R, R2L, and Probe pattern classes.
  • 1) Genesis:: 78% training redundancy leaves 1,074,992 unique points from 4,898,431 records, while 89.5% test redundancy leaves 311,029 from 2,984,154.

5) Skewedness:

KDD-99 is heavily skewed and non-stationary, while NSL-KDD addresses some dataset issues but still has representational limitations. The benchmark therefore compares class distributions and dataset characteristics before evaluating classifiers.

  • 5) Skewedness:: 98.61% of KDD-99 data belongs to Normal or DoS, leaving minority classes underrepresented.
  • 5) Skewedness:: KDD-99 train and test distributions diverge sharply: DoS rises from 23% to 73.9%, while Normal falls from 75.61% to 19.48%.
  • 5) Skewedness:: NSL-KDD was designed to rectify KDD-99 but does not represent low-footprint attacks.
  • 5) Skewedness:: NSL-KDD uses fewer unique data points and undersamples Normal, DoS, and Probe to mitigate some KDD-99 skewness.

1) Size and redundancy:

The datasets differ in feature reduction, class resampling, and construction context. UNSW-NB15 is presented as a modern alternative to KDD-99.

  • 1) Size and redundancy:: KDD-99 and NSL-KDD use reduced feature sets, with NSL-KDD reduced to 20 features.KDD-99 uses a reduced set of 20 features; NSL-KDD’s reduction is described separately as 20 features.
  • 1) Size and redundancy:: NSL-KDD undersamples Normal, DoS, and Probe classes to mitigate KDD-99’s skewness.The paper compares the resulting classifier-performance improvement in a later section.
  • 1) Size and redundancy:: UNSW-NB15 is described as a modern KDD-99 alternative that mitigates several deficiencies of the older dataset.Its generation involved the IXIA PerfectStorm tool over two sessions at the Australian Center of Cyber Security.

1) Genesis:

UNSW-NB15 was generated from contemporary attack traffic and provides multiple target classes and reduced features for intrusion-detection benchmarking.

  • 1) Genesis:: UNSW-NB15 contains one Normal class and nine anomalous target classes.The anomalous classes are Fuzzers, Analysis, Backdoors, DoS, Exploits, Generic, Reconnaissance, Shell Code, and Worms.
  • 1) Genesis:: Its training and test sets contain 175,341 and 82,332 data points, respectively, with no redundant data points.These sizes were considered sufficient for training high-variance intrusion-detection classifiers.
  • 1) Genesis:: UNSW-NB15 extracts 49 features across five groups and reduces them to 30 using Mean Decrease Impurity.The groups are Flow, Basic, Content, Time, and Additionally Generated.
  • 1) Genesis:: UNSW-NB15 has lower target skewness than KDD-99 and is described as more uniform than the traditional datasets.The paper uses the largest-to-smallest target ratio to compare uniformity.

6) Non-Stationarity:

The pipeline preprocesses and balances datasets before training and prediction, while the study compares class distributions and feature-selection choices across datasets.

  • 6) Non-Stationarity:: Training and test distributions in UNSW-NB15 are similar, indicating maintained data stationarity.This comparison is reported in Table III.
  • 6) Non-Stationarity:: The pipeline applies preprocessing, feature selection, training, and prediction to five datasets.The datasets are KDD-99, NSL-KDD, NSL-SMOTE, UNSW-NB15, and NB15-SMOTE.
  • 6) Non-Stationarity:: Feature selection removes predictors with zero Mean Decrease Impurity and reduces KDD-99, NSL-KDD, and UNSW-NB15 to 24, 20, and 30 features.Mean standardization is then used to place predictor values on a similar scale.
  • 6) Non-Stationarity:: SMOTE oversampling and random undersampling are used to construct NSL-SMOTE and NB15-SMOTE.The test sets remain unchanged for prediction.
  • 6) Non-Stationarity:: SMOTE sets NSL-KDD’s U2R-to-Normal ratio to 0.015 and oversamples selected UNSW-NB15 minority classes.NSL-SMOTE is then undersampled to 995 examples per class, whereas NB15-SMOTE is not randomly undersampled.

C. Training

The study benchmarks six classifiers using cross-validation and evaluates them with class-wise and weighted F1-based metrics rather than relying on accuracy alone.

  • C. Training: Six machine-learning models are benchmarked, with hyperparameters tuned by five-fold cross-validation and randomized grid search.The models include Naive Bayes, Support Vector Machine, Decision Tree, Random Forest, Neural Network, and K-Means.
  • C. Training: Classifier performance is assessed using Precision, Recall, and F1-Score computed from prediction confusion matrices.F1-Score is the harmonic mean of Precision and Recall.
  • C. Training: Null Error Rate measures the test-set error from assigning every pattern to the majority class.Its calculation depends on the number of examples in the majority class.
  • C. Training: Accuracy is discounted because imbalanced datasets can yield high accuracy while minority classes are misclassified.The paper therefore presents F1-Scores on a class-wise basis, especially where false negatives can be consequential.
  • C. Training: Weighted F1-Score averages class-wise F1-Scores with each class weighted by its Support.The paper uses it as a single overall performance variable.

A. KDD-99 vs. NSL-KDD

The study finds that class imbalance weakens minority-class detection in KDD-99 and NSL-KDD, while SMOTE improves minority performance. UNSW-NB15 performs comparably or better than NSL-KDD on nearly all binary learning models, supporting its use as a contemporary alternative.

  • KDD-99 vs. NSL-KDD: SMOTE substantially improves R2L F1-Scores compared with KDD-99 and NSL-KDD across the reported classifiers.Compared with KDD-99, scores rise from 18 to 28, 6 to 24, 8 to 23, 8 to 31, and 4 to 23; compared with NSL-KDD, Decision Tree rises from 1 to 37 and Neural Network from 7 to 32.
  • KDD-99 vs. NSL-KDD: NSL-SMOTE reduces DoS and Probe F1-Scores relative to KDD-99, because undersampling forces classifiers to attend more equally to all classes.NSL-KDD has DoS F1-Scores similar to NSL-SMOTE.
  • UNSW-NB15: UNSW-NB15 achieves high performance on Exploits, Generic, and Normal, which together comprise over 73% of its training data.The least represented classes—Analysis, Backdoor, Shell Code, and Worms—comprise only 2.857% and generally receive weaker performance.
  • UNSW-NB15: SMOTE raises the mean F1-Score for Shell Code from 16.8 to 22.3 and produces a total mean increase of 17.5 points across four minority classes.The largest reported gain is for Random Forest on Worms, increasing from 25 to 65.
  • Binary comparison: UNSW-NB15 equals or betters NSL-KDD on almost all implemented learning models when the datasets are evaluated in binary form.Binarization can conceal weak minority-class F1-Scores in KDD-99.

V. DISCUSSION

The discussion attributes poor minority-class detection to skewed target distributions and evaluates balancing strategies for NSL-KDD and UNSW-NB15. It concludes that UNSW-NB15 can substitute for KDD-99 and NSL-KDD, while noting remaining imbalance and limited diversity.

  • Discussion: KDD-99 and NSL-KDD have unsatisfactory minority F1-Scores despite high DoS performance, and aggregate metrics can obscure this discrepancy.Best R2L scores were 18 and 31, U2R scores were 4 and 12, while DoS approached 98 and 85.
  • Discussion: NSL-SMOTE increases mean F1-Score by factors of 2.35 for U2R and 2.1 for R2L compared with NSL-KDD.Absolute scores remain marginal, which the paper associates with oversampling noisy and borderline examples from only 52 U2R instances.
  • Discussion: UNSW-NB15 is less skewed than KDD-99 and NSL-KDD but remains imbalanced, with a largest-to-smallest class ratio of 430.Minority F1-Scores suffer, although SMOTE especially improves Analysis and Backdoor.
  • Discussion: Random Forest is the best-performing model, and oversampling gives its trees more minority-class examples to contribute to the majority vote.The discussion suggests that bagging and boosting methods may also improve minority performance.
  • Discussion: Binary UNSW-NB15 matches or exceeds NSL-KDD in Weighted F1-Score, with maximum scores of 88.5 and 83 respectively.The paper reasons that UNSW-NB15 can train an adequate binary anomaly detector for preliminary filtering in a multistage NIDS.
  • Conclusion: The paper concludes that UNSW-NB15 can satisfactorily substitute for KDD-99 and NSL-KDD in machine-learning anomaly-based NIDS research.This conclusion is framed as an encouragement for forthcoming research.

VI. CONCLUSION

The paper benchmarks UNSW-NB15 as a modern alternative to KDD-99 and shows that oversampling improves classifier performance on imbalanced intrusion-detection datasets. Further optimization remains possible, while most of the complete UNSW-NB15 source data remains unexplored.

  • UNSW-NB15 contains 10 modern attack classes and a less skewed target distribution, and its F1 performance was benchmarked against models commonly used with KDD-99 and NSL-KDD.The evaluation was intended to facilitate UNSW-NB15 adoption in future research.
  • Oversampling improves the previously weak classifier performance on NSL-KDD and UNSW-NB15.
  • Alternative ensemble, clustering, unsupervised, and hybrid methods remain potential ways to improve minority-class F1 performance across the machine-learning pipeline.
  • Random undersampling discards several majority examples, motivating alternatives such as EasyEnsemble and BalanceCascade.
  • Only a small portion of the ACCS labs data was used to configure UNSW-NB15, leaving more than 2.5 million records largely unexplored.
Loading 1811.05372v1…