Source-linked AI summary
Evaluating Standard Feature Sets Towards Increased Generalisability and Explainability of ML-based Network Intrusion Detection
Mohanad Sarhan, Siamak Layeghy, Marius Portmann
TL;DR
ML-based NIDS research has limited comprehensive cross-dataset evaluation and understanding of model decisions, factors the paper identifies as part of the research-to-deployment gap. The authors compare NetFlow and CICFlowMeter across three datasets and use SHAP to examine feature contributions. They report higher detection accuracy with NetFlow across the datasets.
Problem
Method
The paper evaluates NetFlow and CICFlowMeter feature sets across three datasets with RF and DFF classifiers, and analyzes feature contributions using SHAP.
Source: §4.1 · p. 6, §5.1 · p. 8, §6 · p. 10Results
NetFlow features achieve consistently higher detection accuracies than CICFlowMeter features across the three datasets.
Source: §4.2 · p. 7Takeaways & Limitations
SHAP analyses identify influential features, including 13 common top-20 features for RF and DFF on NF-CSE-CIC-IDS2018-v2.
Source: §5.2 · p. 9
Abstract
from arXiv · showhide
Machine Learning (ML)-based network intrusion detection systems bring many benefits for enhancing the cybersecurity posture of an organisation. Many systems have been designed and developed in the research community, often achieving a close to perfect detection rate when evaluated using synthetic datasets. However, the high number of academic research has not often translated into practical deployments. There are several causes contributing towards the wide gap between research and production, such as the limited ability of comprehensive evaluation of ML models and lack of understanding of internal ML operations. This paper tightens the gap by evaluating the generalisability of a common feature set to different network environments and attack scenarios. Therefore, two feature sets (NetFlow and CICFlowMeter) have been evaluated in terms of detection accuracy across three key datasets, i.e., CSE-CIC-IDS2018, BoT-IoT, and ToN-IoT. The results show the superiority of the NetFlow feature set in enhancing the ML models detection accuracy of various network attacks. In addition, due to the complexity of the learning models, SHapley Additive exPlanations (SHAP), an explainable AI methodology, has been adopted to explain and interpret the classification decisions of ML models. The Shapley values of two common feature sets have been analysed across multiple datasets to determine the influence contributed by each feature towards the final ML prediction.
1 Introduction
ML-based intrusion detection research has not translated widely into deployment, with limited cross-dataset evaluation and difficulty understanding model decisions among the stated gaps. This paper compares NetFlow and CICFlowMeter features across datasets and uses SHAP to interpret model predictions.
1 Introduction
- Despite high detection accuracy on selected datasets, few ML-based NIDSs are deployed, amid limited comprehensive evaluation and understanding of model operations.The introduction identifies these as two leading reasons for the research-to-production gap.Source: §1 · p. 1
- Prior evaluations often use dataset-specific features, limiting assessment of a feature set across datasets and leaving NetFlow’s status versus CICFlowMeter unvalidated.The paper compares the two feature sets on CSE-CIC-IDS-2018, ToN-IoT, and BoT-IoT.Source: §1 · p. 1
- The study generates and publicly releases CICFlowMeter-format ToN-IoT and BoT-IoT datasets to support comparisons alongside the existing formats.The datasets are labelled for binary- and multi-class tasks.Source: §1 · p. 2
- SHAP feature values are used to explain each feature’s contribution to model decisions and identify features influential in predictions.The paper presents this analysis as a way to interpret classification decisions.Source: §1 · p. 2
- The paper evaluates NetFlow and CICFlowMeter across three NIDS datasets, releases two datasets, and analyses feature importance with SHAP.The introduction frames these contributions as addressing evaluation and interpretability limitations.Source: §1 · p. 2
2 Related Work
Related work examines both cross-dataset feature evaluation and explanations of ML-based NIDS decisions. The reviewed studies include shared NetFlow datasets and SHAP- or other XAI-based analyses, alongside limitations in evaluation scope.
2 Related Work
- Moustafa et al. compared KDD-99 and UNSW-NB15 features, but did not replicate KDD-99 features on UNSW-NB15 for a reliable comparison.NB and EM accuracy was lower with original KDD-99 features than with UNSW-NB15 features.Source: §2 · p. 2
- Sarhan et al. [19] created four datasets sharing 12 NetFlow features; the related evaluation found inferior multiclass results despite reliable binary-class performance.The features are described as easier to extract from packet headers than features requiring deep packet inspection.Source: §2 · pp. 2–3
- A later standard set of 43 NetFlow features [6] reportedly outperformed earlier feature sets across datasets, with added features capturing further security events.The work extended the feature set proposed in [19].Source: §2 · p. 3
- SHAP-based studies, [22] interpreted NIDS decisions, while [22] also explored LIME, BRCG, and CEM using NSL-KDD.The reviewed studies used a single dataset; the paper notes that NSL-KDD does not reflect current network threats and attacks.Source: §2 · p. 3
3 Common NIDS Feature Set
Dataset-specific feature sets restrict cross-dataset evaluation, while common features support more reliable assessment across environments and attack types. The paper uses NetFlow and CICFlowMeter formats across six datasets, including two newly generated CICFlowMeter datasets.
3 Common NIDS Feature Set
- Unique, incompatible dataset features restrict cross-dataset testing; a shared feature set is presented as important for reliable evaluation and potential deployment.The paper notes that extracting every feature may be infeasible in practical networks.Source: §3 · p. 4
3.1 Current Datasets
- CSE-CIC-IDS2018, ToN-IoT, and BoT-IoT represent different environments and attack scenarios, but their original feature sets differ, limiting generalisability assessment.The datasets include organizational, IoT, and botnet-network settings, respectively.Source: §3.1 · p. 4
3.2 Generated Datasets
- To create CICFlowMeter versions of ToN-IoT and BoT-IoT, the authors processed pcaps into flows and assigned binary and multiclass labels using ground-truth events.The resulting datasets are named CIC-ToN-IoT and CIC-BoT-IoT and are publicly available.Source: §3.2 · p. 5
- CIC-ToN-IoT and CIC-BoT-IoT each use 83 CICFlowMeter features, with 5,351,760 and 13,428,602 records, respectively.Both datasets were generated as part of the paper and published for research use.Source: §3.2 · p. 5
- The six datasets—three NetFlow and three CICFlowMeter—enable comparison of two common feature sets across multiple network environments.The authors present this shared-format evaluation as a resource for assessing ML-based intrusion detection.Source: §3.2 · p. 5
4 Evaluation
The evaluation compares NetFlow and CICFlowMeter feature sets across six datasets, using DFF and RF classifiers. NetFlow achieves higher F1 across the three datasets, while using fewer features than CICFlowMeter.
4 Evaluation
- The comparison covers three CICFlowMeter datasets and their three NetFlow counterparts, spanning different network environments and attack scenarios.Source: §4 · pp. 5–6
4.1 Methodology
- DFF and RF classify flows; the evaluation uses 70%-30% train-test splits, five cross-validation splits, and reports the mean across them.DFF has three 10-node ReLU hidden layers; RF uses 50 randomized decision trees.Source: §4.1 · p. 6
4.2 Results
- RF and DFF F1 rise from 0.93 to 0.98 and 0.90 to 0.97, respectively, on NF-CSE-CIC-IDS2018-v2 versus CSE-CIC-IDS2018.Source: §4.2 · p. 6
- RF F1 rises from 0.99 to 1.00 on both ToN-IoT and BoT-IoT, while FAR falls from 1.22% to 0.58% and 1.53% to 0.25%, respectively.Source: §4.2 · p. 6
- 95.37% versus 92.29% is the DFF detection rate on NF-ToN-IoT-v2 versus CIC-ToN-IoT, with F1 rising from 0.94 to 0.96.Source: §4.2 · p. 6
- NetFlow consistently achieves higher F1 than CICFlowMeter and the original feature sets across the three datasets.Figure 2 presents the comparison across datasets and classifiers.Source: Figure 2 · p. 7, §4.2 · p. 7
- NetFlow uses 43 features versus CICFlowMeter's 83, and its packet-header features do not require the additional calculations described for CICFlowMeter.The paper notes fewer features can aid feature extraction, analysis, and storage in operational networks.Source: §4.2 · p. 7
5 Explainable ML-based NIDS
SHAP is used to make ML-based intrusion-detection decisions more interpretable by identifying influential network features. Across datasets, feature influence varies, though some patterns recur across models and feature sets.
5 Explainable ML-based NIDS
- Explaining model decisions matters because security teams may be reluctant to trust opaque NIDS predictions and feature-level explanations can help justify classifications and troubleshoot misclassifications.The paper also notes that feature analysis can identify security-relevant features and features with limited security events.Source: §5 · pp. 7–8
5.1 Shapley Analysis
- SHAP feature importance is calculated from Shapley values, which represent each feature value's contribution to a sample's prediction relative to the average prediction.Mean Shapley values average feature contributions across test samples; larger values indicate stronger influence on the model's final prediction.Source: §5.1 · p. 8
5.2 Results
- On CSE-CIC-IDS2018, forward-flow features dominate the RF rankings; on CIC-ToN-IoT, idle-time features influence over 50% of RF predictions, unlike the DFF rankings.The RF's 'Fwd Seg Size Min' feature has almost twice the influence of any other feature on CSE-CIC-IDS2018; further analysis of idle-based features is needed.Source: §5.2 · p. 9
- Five top features are shared between deep and shallow models for NetFlow, compared with two for CICFlowMeter, while mean SHAP values vary substantially across datasets.The figures rank features by mean Shapley value across test samples; their error bars span the minimum and maximum SHAP values across the three datasets.Source: §5.2 · p. 9, Figure 3 · p. 9, Figure 4 · p. 10
- Feature contributions differ across datasets, but RF is heavily influenced by forward-direction features in the CICFlowMeter format.The paper relates variation to differences in attack scenarios and benign application traffic.Source: §5.2 · p. 10
6 Conclusion
The paper compares NetFlow and CICFlowMeter features across three datasets and uses SHAP to explain model predictions. NetFlow achieves higher attack-detection accuracy and lower prediction time across the datasets.
6 Conclusion
- Across CSE-CIC-IDS2018, ToN-IoT, and BoT-IoT, NetFlow features consistently achieve higher attack-detection accuracy and lower prediction time than CICFlowMeter features.The experiments use RF and DFF classifiers and include two generated datasets, CIC-BoT-IoT and CIC-ToN-IoT.Source: §6 · p. 10
- The paper evaluates NetFlow and CICFlowMeter feature sets across three datasets and publishes CIC-BoT-IoT and CIC-ToN-IoT for the experiments.The comparison examines generalisability across network environments and attack scenarios.Source: §6 · p. 10
- SHAP feature-importance analysis identifies the network features influencing model predictions for each dataset.The conclusion presents this analysis alongside the feature-set comparison as part of its evaluation.Source: §6 · p. 10