Source-linked AI summary
Towards a Standard Feature Set for Network Intrusion Detection System Datasets
Mohanad Sarhan, Siamak Layeghy, Marius Portmann
TL;DR
Existing NIDS datasets use distinct feature sets, limiting reliable comparison of ML-based traffic classifiers across datasets and network scenarios. This paper proposes and evaluates NetFlow-based standard feature sets, including a 43-feature version, which achieves higher F1-Score than proprietary feature sets across the considered benchmark datasets and classification scenarios.
Problem
Distinct feature sets across NIDS datasets make reliable comparison and evaluation of ML-based traffic classifiers across network environments and attack scenarios difficult.
Method
The paper proposes NetFlow-based standard feature sets and evaluates them by extracting and labelling features from four well-known NIDS datasets.
Results
The 43-feature NetFlow-based feature set achieves a higher F1-Score than proprietary feature sets across all considered benchmark datasets in binary and multi-class classification scenarios.
Takeaways & Limitations
A standard feature set supports more reliable evaluation across datasets and enables multiple labelled NIDS datasets to be merged.
Takeaways & Limitations
The four datasets share only 3 features, underscoring the limited common feature coverage among existing datasets.
Abstract
from arXiv · showhide
Network Intrusion Detection Systems (NIDSs) are important tools for the protection of computer networks against increasingly frequent and sophisticated cyber attacks. Recently, a lot of research effort has been dedicated to the development of Machine Learning (ML) based NIDSs. As in any ML-based application, the availability of high-quality datasets is critical for the training and evaluation of ML-based NIDS. One of the key problems with the currently available datasets is the lack of a standard feature set. The use of a unique and proprietary set of features for each of the publicly available datasets makes it virtually impossible to compare the performance of ML-based traffic classifiers on different datasets, and hence to evaluate the ability of these systems to generalise across different network scenarios. To address that limitation, this paper proposes and evaluates standard NIDS feature sets based on the NetFlow network meta-data collection protocol and system. We evaluate and compare two NetFlow-based feature set variants, a version with 12 features, and another one with 43 features.
1 Introduction
ML-based NIDS research depends on high-quality datasets, but existing datasets use distinct feature sets that hinder reliable comparison and generalisation. The paper therefore evaluates standard NetFlow-based feature sets to address this problem.
- Real-world labelled network-flow datasets are difficult to obtain because of security and privacy concerns.
- Distinct feature sets across NIDS datasets make ML-based evaluations unreliable and impede comparison across network scenarios.Feature selection affects ML-model performance, while some features require computationally impractical deep packet inspection.
- A standard feature set would support fairer evaluation across datasets, network environments, and attack scenarios while enabling assessment of model generalisability.It would also keep the represented security events and network information consistent and controlled.
- NetFlow is proposed as the basis for standard features because it is industry-standard, scalable, and captures security-relevant flow information.
- Four widely used NIDS datasets were converted to a common basic NetFlow feature set, but insufficient security information limited multi-class detection accuracy.The paper consequently proposes an extended NetFlow feature set and releases five converted datasets.
- The evaluation compares two NetFlow-based variants with proprietary feature sets using an Extra Tree classifier.
2 Limitations of Existing Datasets
NIDS benchmark datasets are commonly engineered from controlled testbeds because realistic labelled traffic is difficult to obtain. Their feature sets differ substantially, creating extraction, comparability, and generalisation problems for ML evaluation.
- Engineered benchmark datasets use controlled testbeds to generate and label benign and malicious network flows when realistic labelled traffic is unavailable.Packets are captured as pcap files, then features are extracted to form labelled flows for ML-based NIDS development and evaluation.
- Dataset features must represent security events adequately while remaining feasible in number and extraction complexity for scalable deployments.
- Without a standard feature set, datasets use nearly exclusive feature collections chosen by their authors’ domain experience.Only a small number of features are shared across current datasets, complicating cross-dataset evaluation.
- The lack of standardisation creates concerns about evaluation reliability through complex extraction, limited cross-dataset generalisation assessment, and missing universal multi-environment data.
- Figure 1 shows that all four datasets share 3 features, while pairwise overlaps range from 1 to 5 features.Most features are exclusive to individual datasets, making targeted cross-dataset evaluation challenging.
- The datasets also have sharply different benign-to-attack ratios, with some dominated by benign flows and others mainly by attack samples.
3 Benchmarking a Standard Feature Set
The paper proposes a standard NetFlow-based feature set and generates common datasets from four existing NIDS benchmarks. The resulting datasets use 43 flow-based features and include a merged collection spanning multiple network setups and attack settings.
- 3 Benchmarking a Standard Feature Set: The standard feature set is intended to support reliable ML-based NIDS evaluation across datasets, network environments, and attack scenarios.Its design also targets efficient feature extraction and benchmarking for future NIDS dataset releases.
- 3.1 NetFlow: 43 numerical NetFlow v9 features capture general flow statistics and protocol-specific information without using encrypted payloads.The features are extracted from packet headers, supporting efficient ML experiments and practical deployment.
- 3.1 NetFlow: The extraction pipeline uses nProbe to convert public pcap files into CSV flows and labels them by matching five flow identifiers to published ground-truth events.Flows receive binary benign/attack labels and attack-type labels; benign flows are assigned class 0 and attacks class 1.
- 3.2 Datasets: The proposed feature set is extracted from UNSW-NB15, BoT-IoT, ToN-IoT, and CSE-CIC-IDS2018 to create corresponding NetFlow datasets.The datasets are compared with original and basic NetFlow versions using properties including extraction tool, feature count, file size, and benign-to-attack ratio.
- 3.2 Datasets: The extended datasets preserve dataset-specific attack categories while expanding the basic NetFlow format with additional features.NF-UNSW-NB15-v2 contains nine attack subcategories, NF-BoT-IoT-v2 contains four attack categories, and the other datasets retain their respective attack distributions.
- 3.2 Datasets: NF-UQ-NIDS-v2 merges the four datasets and adds an original-dataset label, enabling comparisons across different testbed networks and attack settings.Its attack categories are consolidated into parent categories such as DoS, DDoS, brute force, and injection.
4 Evaluation
The evaluation compares the proposed extended NetFlow feature set with basic NetFlow and original features across five NIDS datasets using Extra Trees. Overall, the extended set improves or matches binary and multi-class detection performance, though some attack types remain unreliable and prediction time can increase.
- Evaluation setup: Extra Trees was evaluated on five NIDS datasets using extended NetFlow, basic NetFlow, and original feature sets.Metrics included accuracy, AUC, F1 score, detection rate, false alarm rate, and single-sample prediction time.
- Binary-class Classification: 0.9845 AUC and 0.97 F1 score were achieved on NF-UNSW-NB15-v2, with the highest F1 score obtained in the shortest prediction time.The AUC exceeded the corresponding NF-UNSW-NB15 and UNSW-NB15 values of 0.9485 and 0.9545.
- Binary-class Classification: 99.64% accuracy on NF-ToN-IoT-v2 exceeded ToN-IoT's 97.86% and was similar to NF-ToN-IoT's 99.66%.NF-CSE-CIC-IDS2018-v2 achieved 96.89% detection rate, 0.31% false alarm rate, and 21.75 µs prediction time.
- Binary-class Classification: The proposed feature set outperformed original and basic features overall in binary attack detection, while matching or exceeding each dataset's F1 score.The comparison used the same classifier across the three feature sets and five datasets.
- Multi-class Classification: Some attack types remained unreliable despite improved aggregate results, including analysis, backdoor, DoS, Brute Force, DDoS attack-HOIC, infiltration, and SQL injection.The paper attributes several lower scores to missing key features or high false-positive counts and calls for further feature analysis.
- Multi-class Classification: 0.98 weighted F1 on NF-ToN-IoT-v2 exceeded 0.87 for ToN-IoT and 0.60 for NF-ToN-IoT in multi-class classification.The extended set increased detection rates for most attack types, but DoS, MITM, and XSS remained exceptions.
- Multi-class Classification: 96.96% accuracy and 0.97 F1 score on NF-UQ-NIDS-v2 improved from 70.81% and 0.79, while prediction time increased from 14.74 µs to 25.67 µs.Infiltration and MITM attacks were detected less accurately with the extended feature set.
- Multi-class Classification: The extended NetFlow feature set was often comparable to original features and markedly superior to basic NetFlow features in multi-class classification.The authors report enhanced detection performance while noting that further feature selection is needed to identify key features and improve extraction.
5 Conclusion
The paper proposes a NetFlow-based standard feature set to support reliable evaluation across datasets, environments, and attack scenarios. Its 43-feature variant achieves higher F1-Score performance than proprietary feature sets in the Extra Tree evaluation.
- The proposed NetFlow-based standard feature set is intended to enable reliable evaluation of ML-based NIDS across multiple datasets, environments, and attack scenarios.
- Multiple NIDS datasets can be merged using the common feature set, producing a larger variety of labelled datasets.
- Five new NIDS dataset variants were generated from existing benchmark datasets and made publicly available.
- 43 features achieve higher classification performance (F1-Score) than the proprietary feature sets in the Extra Tree evaluation.
- The NetFlow-based feature sets are practical and scalable because efficient NetFlow exporters and collection systems are widely available.