PaperlayerView exact revision on arXiv

Source-linked AI summary

NetFlow Datasets for Machine Learning-based Network Intrusion Detection Systems

Mohanad Sarhan, Siamak Layeghy, Nour Moustafa, Marius Portmann

arXiv:2011.09144v1cs.NI

TL;DR

ML-based NIDS evaluation is hindered by incompatible feature sets and limited real-world labelled traffic. The paper converts four benchmark datasets’ packet captures into labelled NetFlow datasets for binary- and multiclass learning. NetFlow features achieve similar binary-class detection performance to original features, while NF-ToN-IoT and NF-CSE-CIC-IDS2018 are inefficient in multi-class experiments.

  • Problem

    Different NIDS datasets use distinct feature sets, limiting evaluation of ML model performance and generalisation across datasets using common features.

  • Method

    The paper converts publicly available packet captures from four benchmark NIDS datasets into labelled NetFlow version 9 datasets using selected NetFlow features.

  • Results

    The NetFlow datasets show similar binary-class detection performance to the original datasets, while NF-ToN-IoT and NF-CSE-CIC-IDS2018 are inefficient for multi-class detection.

  • Takeaways & Limitations

    NetFlow datasets support cross-dataset ML evaluation with a common feature set and require lower collection and storage complexity than the original features.

  • Takeaways & Limitations

    Further feature analysis and experiments are required, particularly to improve missed attack-type detection in multi-class settings.

Abstract

from arXiv · show

Machine Learning (ML)-based Network Intrusion Detection Systems (NIDSs) have proven to become a reliable intelligence tool to protect networks against cyberattacks. Network data features has a great impact on the performances of ML-based NIDSs. However, evaluating ML models often are not reliable, as each ML-enabled NIDS is trained and validated using different data features that may do not contain security events. Therefore, a common ground feature set from multiple datasets is required to evaluate an ML model's detection accuracy and its ability to generalise across datasets. This paper presents NetFlow features from four benchmark NIDS datasets known as UNSW-NB15, BoT-IoT, ToN-IoT, and CSE-CIC-IDS2018 using their publicly available packet capture files. In a real-world scenario, NetFlow features are relatively easier to extract from network traffic compared to the complex features used in the original datasets, as they are usually extracted from packet headers. The generated Netflow datasets have been labelled for solving binary- and multiclass-based learning challenges. Preliminary results indicate that NetFlow features lead to similar binary-class results and lower multi-class classification results amongst the four datasets compared to their respective original features datasets. The NetFlow datasets are named NF-UNSW-NB15, NF-BoT-IoT, NF-ToN-IoT, NF-CSE-CIC-IDS2018 and NF-UQ-NIDS are published at http://staff.itee.uq.edu.au/marius/NIDS_datasets/ for research purposes.

1 Introduction

Anomaly-based NIDSs learn network traffic behaviours to classify incoming traffic as attacks or benign activity. Because real labelled traffic is difficult to obtain, researchers rely on synthetic benchmark datasets whose author-selected features shape ML training and evaluation.

  • Anomaly-based NIDSs classify incoming traffic as attack or benign by learning complex network-data behaviours.Attack patterns can involve packet counts, sizes, protocols, services, and flags.
  • Network attack patterns must be learned from traffic data, making data collection important for ML training and evaluation.Real network data is difficult to obtain because of security and privacy issues.
  • Production networks generally lack labelled flows, so researchers create publicly available synthetic datasets through virtual test beds.These datasets support supervised ML research by combining labelled network flows with extracted features.
  • Dataset authors pre-determine network features using domain knowledge and the tools used for feature extraction.Consequently, feature choices become part of how ML models are trained and evaluated.
  • NetFlow conversion enables ML models to be evaluated across datasets using the same features while testing detection of different attack types.NetFlow is described as a widely deployed network-traffic collection protocol.

2 Limitations of Existing Datasets

Existing NIDS datasets provide labelled benchmark flows but use distinct, often complex feature sets. This limits common-feature evaluation across datasets and introduces practical and distributional differences that affect comparisons.

  • Benchmark NIDS datasets provide labelled network flows for ML training and testing because real-world labelled traffic is difficult to obtain.They combine benign behaviour with synthetic attack scenarios captured in packet captures.
  • There is no standard benchmark feature set, so dataset authors select features using domain knowledge and extraction tools.The resulting datasets therefore contain different network-feature sets.
  • Different feature sets create dimensional overload and prevent evaluating model generalisation across datasets with a targeted feature set.The paper connects these limitations to a gap between academic research and real-world deployment, using qualified language.
  • Each dataset uses its own feature design, leaving only a small set of shared features across the four datasets.This makes it difficult to measure model performance using identical features.
  • The datasets also differ substantially in benign-versus-attack balance, with UNSW-NB15 and CSE-CIC-IDS2018 benign-heavy and ToN-IoT and BoT-IoT attack-heavy.The cited dataset descriptions report 87.35% benign flows for UNSW-NB15, 99.99% attacks for BoT-IoT, and 96.44% attacks for ToN-IoT.

3 NetFlow Datasets

This section describes NetFlow as a scalable common representation for network traffic and details the conversion and labelling of four benchmark NIDS datasets. The resulting datasets provide shared features and include dataset-specific flow distributions, with NF-UQ-NIDS merging the four sources.

  • 3.1 NetFlow: NetFlow summarizes packets into flows, improving scalability and reducing storage demands compared with capturing complete network traffic.A flow groups packets between endpoints using shared attributes such as the five-tuple.
  • 3.1 NetFlow: The four NIDS datasets share only a few features, motivating conversion to a common NetFlow feature set for cross-dataset evaluation.NetFlow is presented as a format that can convert available datasets into comparable features.
  • 3.2 Conversion: The conversion used publicly available packet captures, nProbe, NetFlow version 9, and 12 selected features exported as comma-separated text flows.The extraction command specifies NetFlow version 9 and CSV-compatible output.
  • 3.2 Conversion: The four converted datasets are NF-UNSW-NB15, NF-BoT-IoT, NF-ToN-IoT, and NF-CSE-CIC-IDS2018, each labelled with benign and attack flows.Their reported flow counts and class ratios differ substantially across datasets.
  • 3.2 Conversion: NF-UQ-NIDS merges the four NetFlow datasets and adds an original-dataset label for comparing flows across test-bed networks.It contains 11,994,893 records: 9,208,048 benign flows and 2,786,845 attack flows.

4 Evaluation

The evaluation compares NetFlow and original datasets on binary- and multiclass intrusion detection using standardized preprocessing, class weighting, and five-fold cross-validation. NetFlow features generally preserve binary detection performance but produce mixed multiclass results across datasets.

  • Evaluation setup: The evaluation removes identifiers and time-related fields, additionally excluding TTL-correlated features from UNSW-NB15 to reduce label bias.The preprocessing drops flow IDs, addresses, ports, timestamps, start/end times, and selected UNSW-NB15 TTL features.
  • Evaluation setup: Five cross-validation splits average accuracy, AUC, F1 Score, Detection Rate, False Alarm Rate, and single-sample prediction time.Custom class weights address extreme binary- and multiclass-label imbalance.
  • Binary-class classification: 98.62% accuracy for NF-UNSW-NB15 versus 99.25% for UNSW-NB15, while NF-ToN-IoT reaches 99.66% versus 97.86% for ToN-IoT.NF-ToN-IoT also achieves 99.67% DR and 0.37% FAR, whereas NF-UNSW-NB15 has nearly the same DR but higher FAR and shorter prediction time.
  • Binary-class classification: The shared eight-feature NetFlow representation enables a fair AUC comparison across all four datasets.The comparison uses the same feature set, which was not possible with the datasets’ unique original features.
  • Multi-class classification: 97.62% multiclass accuracy for NF-UNSW-NB15 is slightly below 98.19% for UNSW-NB15, with lower prediction time and improved detection for several attack types.Detection increases for analysis, backdoor, DoS, fuzzers, shellcode, and worms but is inefficient for generic attacks.
  • Multi-class classification: 100% accuracy and 1 F1 Score for BoT-IoT are described as potentially unreliable because of extreme class imbalance.NF-BoT-IoT instead achieves 73.58% DR and 0.77 F1 Score, with lower prediction time and 90% DR for reconnaissance and theft attacks.
  • Multi-class classification: NF-ToN-IoT reaches 56.34% multiclass accuracy and NF-CSE-CIC-IDS2018 reaches 71.92%, with both showing difficulty detecting some individual attack patterns.NF-ToN-IoT improves DoS detection but lowers detection for several other categories; NF-CSE-CIC-IDS2018 better detects FTP-bruteforce and infiltration but misses several attack types, including SlowHTTPTest.
  • Multi-class classification: NF-UQ-NIDS achieves 70.81% accuracy with 14.74 µs prediction time, reliably detecting some categories but not analysis, DoS, fuzzers, generic, infiltration, worms, injection, MITM, password, scanning, or XSS.The merged dataset is efficient for backdoor, bot, bruteforce, exploits, shellcode, DDoS, and ransomware detection.

5 Conclusion

The paper publishes four NetFlow-based NIDS datasets for machine-learning training and evaluation. They show similar binary-class performance to the original datasets, while NF-ToN-IoT and NF-CSE-CIC-IDS2018 are inefficient for multiclass detection.

  • Contribution: Four new NIDS datasets using NetFlow features are provided for ML-based NIDS training and evaluation.The datasets are intended to support research using a shared feature representation.
  • Conclusion: NetFlow datasets achieve similar binary-class detection performance to the complete original feature sets.This supports their use for binary-class evaluation across the benchmark datasets.
  • Limitations: NF-ToN-IoT and NF-CSE-CIC-IDS2018 are inefficient in multiclass detection experiments.The conclusion identifies further feature analysis as necessary to improve detection of missed attack types.
  • Implications: NetFlow features require lower collection and storage complexity and allow ML models to be evaluated with the same features across datasets and attack types.Future work includes adding NetFlow and critical original-dataset features to improve binary- and multiclass classification.
Loading 2011.09144v1…