Source-linked AI summary
Robustness Evaluation and Detection of Transferable Adversarial Attacks in ML-Based NIDS
Huda Ali Alatawi
TL;DR
Comparative evidence on multiple attacks, detection models, and defenses in ML-based NIDS remains limited. This study evaluates them under a transfer-based setting and introduces the Robustness Index and AR-NIDS. The adversarially trained ensemble reduces average ASR from 0.41 to 0.03 and achieves an RI of 0.98.
Problem
Prior studies often examine a limited set of attacks, models, or defenses, leaving their joint comparative evaluation limited.
Source: §1 · p. 1Method
Results
Average attack success rate fell from 0.41 to 0.03 and the adversarially trained ensemble achieved an RI of 0.98.
Source: §9 · p. 13Takeaways & Limitations
Under the evaluated transfer-based setting, combining adversarial training with ensemble modeling supports stronger empirical robustness than relying on either strategy alone.
Source: §8 · p. 12The findings are bounded by the evaluated transfer-based setting and NF-UQ-NIDS benchmark; validation on additional datasets and feature representations would strengthen generalizability.
Source: §8 · p. 12
Abstract
from arXiv · showhide
Machine learning-based network intrusion detection systems (ML-based NIDS) are vulnerable to adversarial evasion, where malicious samples are perturbed to evade detection and be misclassified as benign. Despite growing research on adversarial attacks and defenses for ML-based NIDS, comparative evaluations of multiple attack types, detection models, and defense strategies under a common setting remain limited. In this paper, we evaluate eight adversarial evasion attacks, fifteen detection models, and three representative defense strategies using the NF-UQ-NIDS dataset, which includes recent traditional and IoT network traffic with twenty distinct attack categories. The evaluation compares model performance on clean test data and on robustness evaluation sets that include adversarial samples, analyzes attack success consistency across models, and examines the effect of defense strategies on adversarial robustness. To support model comparison, we introduce the Robustness Index (RI), a compact comparative metric that rewards high balanced accuracy and macro-F1 score computed on the robustness evaluation set while penalizing high attack success rate (ASR). We further present AR-NIDS, a two-stage framework that uses an adversarially trained ensemble to distinguish normal, attack, and adversarial samples, followed by an adversarial attack classifier to identify the attack type. Under the evaluated transfer-based setting, the proposed adversarially trained ensemble achieves the strongest overall trade-off between classification performance on the robustness evaluation set and evasion resistance, reducing the average ASR from 0.41 to 0.03 and achieving an RI of 0.98.
1. Introduction
Comparative evaluations of adversarial attacks, NIDS models, and defenses remain limited, while robustness metrics can obscure trade-offs between classification performance and evasion resistance. This study evaluates these dimensions under a shared transfer-based setting and introduces a comparative metric and a two-stage framework.
1. Introduction
- Defense effectiveness can vary by attack, model architecture, and evaluation scenario, so ensemble learning alone does not guarantee adversarial robustness.Adversarial training may also be sensitive to the adversarial examples used during training.Source: §1 · pp. 1–2
- The study compares eight evasion attacks, fifteen detection models, and three defense strategies on NF-UQ-NIDS, which covers recent traditional and IoT traffic across twenty attack categories.Adversarial examples are generated against a feed-forward DNN surrogate and evaluated on other NIDS models.Source: §1 · p. 2
- The Robustness Index combines balanced accuracy and macro-F1 on the robustness evaluation set with attack success rate to compare classification performance and evasion resistance.The metric addresses the limits of reporting detection performance and ASR separately.Source: §1 · pp. 1–2
- The proposed AR-NIDS uses an adversarially trained ensemble to distinguish normal, attack, and adversarial samples, then classifies the adversarial attack type.The ensemble is reported to improve empirical robustness over the evaluated baseline and defense models under the transfer-based setting.Source: §1 · p. 2
2. Background
The background introduces adversarial evasion attacks and the classification metrics used to assess NIDS performance and robustness. It distinguishes attack strategies and describes measures spanning class detection, evasion success, and comparative robustness.
2.1. Adversarial Attacks
- Adversarial examples perturb an input while keeping it close to the original, with perturbations constrained by a selected distance or perturbation measure.The perturbation is intended to change the model's prediction toward an adversarial objective.Source: §2.1 · p. 2
- Evasion attacks modify test samples to make malicious traffic appear benign; the study considers gradient-based, optimization-based, and query-based methods, including FGSM, PGD, and HSJ.Other evaluated methods described include JSMA, CW, DeepFool, and ZOO.Source: §2.1 · p. 2
2.2. Evaluation Metrics for ML-based NIDS
- For binary NIDS classification, attack traffic is positive and normal traffic negative; the confusion matrix tracks correct and incorrect predictions for both classes.Accuracy, precision, recall, and F1-score describe overall correctness, false alarms, missed attacks, and the balance of precision and recall.Source: §2.2 · pp. 2–3
- Balanced accuracy weights attack and normal classes equally, while attack success rate measures adversarial samples misclassified as normal.These measures, along with accuracy, precision, recall, and F1-score, underpin the robustness evaluation and Robustness Index.Source: §2.2 · p. 3
3. Related Work
The related-work review covers adversarial training and ensemble learning, then identifies gaps in common-setting comparisons, dataset coverage, metrics, and defense evaluation. The study responds with a comparative evaluation, the Robustness Index, and AR-NIDS.
3.1. Adversarial Training (AT)
- Prior adversarial-training studies report improvements in some settings, but effectiveness varies with attack type, model architecture, training data, and evaluation setting.The review notes that robustness against diverse or transferable attacks remains an evaluation question.Source: §3.1 · p. 3
3.2. Ensemble Learning (EL)
- Prior ensemble-learning results are mixed, with studies reporting both weaker and improved robustness under adversarial conditions.Effectiveness depends on ensemble diversity, base-model selection, attack type, and evaluation protocol; Table 1 compares attacks and defenses studied in prior NIDS work.Source: §3.2 · p. 4, Table 1 · p. 4
3.3. Research Gaps and Study Positioning
- Many prior studies examine few attacks, models, or defenses, leaving the consistency of attack success across heterogeneous detectors insufficiently explored.Some evaluations also rely on older or single-source datasets, limiting the relevance of findings to recent and diverse network environments.Source: §3.3 · p. 4
- Isolated metrics and single-defense evaluations leave trade-offs between detection performance and evasion resistance difficult to compare across strategies.Adversarial training, ensemble learning, and adversarial detection may behave differently by attack type and model architecture.Source: §3.3 · p. 4
- The study addresses these gaps with eight attacks, fifteen models, and three defenses on NF-UQ-NIDS, adds the Robustness Index, and presents two-stage AR-NIDS.The evaluation includes white-box generation and black-box transfer settings; AR-NIDS uses an adversarially trained ensemble followed by an attack classifier.Source: §3.3 · p. 4
4. Framework and Index for ML-based NIDS Robustness
The section introduces the Robustness Index (RI) to compare classification performance and evasion resistance, and presents AR-NIDS as a two-stage framework for detecting adversarial samples and identifying their attack method.
4.1. Robustness Index
- Balanced accuracy gives evaluated classes equal importance by averaging class-level recall, with two classes in the baseline and three in AR-NIDS.The evaluation uses Normal and Attack in the binary baseline, and adds Adversarial as a third class for AR-NIDS.Source: §4.1 · p. 5
- RI combines robustness-set balanced accuracy and macro-F1 with attack success rate (ASR), using an equal-weight formula normalized to [0, 1].The index supports comparison of trade-offs; the paper reports it alongside its underlying metrics rather than as standalone proof of robustness.Source: §4.1 · p. 5
4.2. Adversarial-Robust Network Intrusion Detection System (AR-NIDS)
- AR-NIDS’s first-stage Adversarially Trained Ensemble combines adversarial training and ensemble learning to distinguish normal, attack, and adversarial samples.Its LGBM-based multiclass design treats ordinary attacks and adversarially perturbed attacks as separate categories.Source: §4.2 · pp. 5–6
- When the first stage identifies a sample as adversarial, the second-stage classifier identifies its adversarial attack type.The classifier is implemented using LGBM and covers the evaluated attack methods.Source: §4.2 · p. 6
- AR-NIDS is evaluated in a transfer-based setting, so its empirical robustness does not establish a universal robustness guarantee.The evaluation generates adversarial samples against a surrogate model and assesses them against the evaluated detection models.Source: §4.2 · p. 6
5. Experimental Setup
The study evaluates ML-based NIDS robustness under transferable adversarial attacks using a unified NetFlow dataset, diverse detection models, and a common evaluation setup. It compares detection performance and evasion resistance using multiple metrics, including the Robustness Index.
5.1. Adversary Model
- Adversarial examples are generated against an FF-DNN surrogate and tested on other NIDS models, making generation white-box and target evaluation black-box.The results represent feature-level robustness in normalized NetFlow space, not packet-level attack realizability.Source: §5.1 · p. 6
5.2. Dataset
- The NF-UQ-NIDS subset contains 6,338,509 records, with 81.8% normal and 18.2% attack samples across a dataset containing twenty attack categories.The benchmark combines NetFlow-based data from multiple NIDS datasets with a consistent feature representation.Source: §5.2 · p. 7
5.3. Dataset Preprocessing
- Min-Max normalization rescales each feature to [0, 1], using its minimum and maximum values.The common feature scale prevents larger numeric ranges from dominating training or adversarial perturbation generation.Source: §5.3 · p. 7
5.4. Adversarial Attacks and Models
- Eight evasion attacks were generated from malicious records using FF-DNN; the study evaluates fifteen detection models spanning neural, tree-based, ensemble, linear, and probabilistic families.The attacks include gradient-based, optimization-based, and query-based methods, and a consistent ART configuration supports comparison.Source: §5.4 · p. 7
5.5. Models’ Performance Evaluation Metrics
- Evaluation reports balanced accuracy, precision, recall, macro F1-score, attack success rate, and RI to compare classification performance and evasion resistance.RI combines balanced accuracy and macro-F1 on the robustness evaluation set with ASR; the metrics are reported together rather than relying on RI alone.Source: §5.5 · pp. 7–8
6. Robustness Evaluation of ML-based NIDS to Adversarial Attacks
The evaluation finds that adversarial examples degrade baseline detection performance and that transfer success varies by attack and target model. LGBM has the strongest baseline RI, while PGD, FGSM, and DeepFool are the most effective transferable attacks in this setting.
6.1. Model Performance on Clean Test Data
- On clean test data, CatBoost, DT, ET, RF, and XGBoost each achieved balanced accuracy of 0.98, while QDA had the lowest macro F1-score at 0.46.The results also state that high clean-data performance does not necessarily imply adversarial robustness.Source: §6.1 · p. 8
6.2. Performance Degradation under Adversarial Samples
- Average balanced accuracy fell from 0.89 to 0.79; average attack recall fell from 0.87 to 0.67 and normal-class precision from 0.97 to 0.86.QDA, FF-DNN, and RF had the largest attack-recall reductions, ranging from 0.32 to 0.36.Source: §6.2 · pp. 8–9
6.3. Model Robustness Based on RI
- LGBM achieved the highest baseline RI, while MLP and GBDT also had relatively strong RI values; NB, LDA, and QDA had the weakest.CatBoost, DT, ET, RF, XGB, AdaBoost, LR, and SGD formed a mid-range group with higher attack success or greater performance degradation than the strongest models.Source: §6.3 · p. 9
6.4. Attack Transferability and Success Consistency
- Median success rates of 0.82 to 0.88 made PGD, FGSM, and DeepFool the strongest transferable attacks across baseline models.Their perturbations generated against FF-DNN can exploit vulnerabilities shared by multiple target models.Source: §6.4 · p. 9
- HSJ and CW2 had average transfer success rates of 0.53 and 0.47, respectively, while JSMA and CW∞ had the lowest median success rates.HSJ and CW2 effectiveness varied more across targets, and ZOO had lower average transfer success.Source: §6.4 · p. 10
7. Evaluation of Adversarial-Robust NIDS Framework
The evaluation compares baseline ensembles, adversarial training, an adversarially trained ensemble, and an attack classifier under transferable attacks. Across these defenses, robustness varies by attack, while the ATE provides the strongest and most consistent results in the evaluated setting.
7.1. Baseline Ensemble (BE)
- Baseline ensembles average 0.87 balanced accuracy on the robustness evaluation set and 0.74 RI, below the strongest individual baseline models; meta-learning has the lowest ensemble ASR at 0.52.Soft voting has ASR 0.55 and hard voting 0.60; ensemble learning alone does not consistently improve robustness when base models share vulnerabilities.Source: §7.1 · pp. 10–11
7.2. Adversarially Trained Models (ATM)
- On the robustness evaluation set, adversarial training raises average RI from 0.68 to 0.80 and lowers average ASR from 0.56 to 0.25 versus baseline models.The adversarially trained models also averaged 0.82 balanced accuracy and 0.82 macro F1, versus 0.79 and 0.80 for baseline models; improvement varied by attack type.Source: §7.2 · p. 11
7.3. Adversarially Trained Ensemble (ATE)
- ATE reduces ASR to 0.03 and achieves balanced accuracy, macro F1-score, and RI values of 0.98, the strongest trade-off among evaluated defenses.The average RI is 0.98 for ATE, compared with 0.68 for baseline models, 0.74 for baseline ensembles, and 0.80 for adversarially trained models.Source: §7.3 · p. 11, Figure 7 · p. 11
7.4. Performance of the Adversarial Attack Classifier
- The LGBM adversarial attack classifier achieves 0.97 overall accuracy and 0.97 macro-average precision, recall, and F1-score on the evaluated attack types.F1-score is 1.00 for DeepFool, FGSM, PGD, and ZOO, and ranges from 0.93 to 0.95 for CW2, CW∞, HSJ, and JSMA; evaluation is closed-set, not open-set detection of unseen methods.Source: §7.4 · p. 11, Table 6 · p. 11
7.5. Impact of Defense Strategies on Adversarial Attack Success
- ATE reduces ASR to zero for all evaluated attacks except ZOO, which reaches 0.27; defense effectiveness varies across attack types.Adversarial training reduces ASR for PGD and FGSM to 0.05 and 0.03, while baseline ensembles reach 0.98 and 0.95; for JSMA and CW∞, baseline ensembles reach 0.13 and 0.09, while adversarial training reaches 0.44 for ZOO and 0.39 for CW∞.Source: §7.5 · pp. 11–12, Figure 8 · p. 12
8. Discussion
The discussion finds that clean-data performance, attack transferability, and defense robustness do not align uniformly. It identifies the ATE as strongest among evaluated defenses while bounding that conclusion to the tested setting.
8. Discussion
- Strong clean-data performance does not necessarily imply adversarial robustness, as several models degrade when adversarial samples are introduced, particularly in attack recall.Clean-data accuracy or F1-score alone can overestimate NIDS reliability under adversarial conditions.Source: §8 · p. 12
- PGD, FGSM, and DeepFool transfer most effectively from the FF-DNN surrogate, while ZOO, JSMA, and CW∞ show lower or more variable transfer success.The findings support evaluating robustness with multiple attack families rather than relying on one method.Source: §8 · p. 12
- ATE provides the strongest empirical robustness among evaluated defenses, but defense effectiveness varies by attack and the result is limited to the evaluated transfer-based setting and attack set.Baseline ensembles do not consistently reduce transferred attack success, while adversarial training improves robustness overall but remains attack-dependent.Source: §8 · p. 12
- The adversarial attack classifier identifies attack types associated with detected adversarial samples, but its evaluation covers the attack families used in the study.The authors identify testing against unseen adversarial methods as future work.Source: §8 · p. 12
- Generalizability would be strengthened by validation on additional datasets and feature representations beyond the NF-UQ-NIDS NetFlow benchmark.The study generates adversarial examples over feature representations and proposes future protocol-constrained generation and broader classifier validation.Source: §8 · p. 12
9. Conclusion
The conclusion reports that transferable attack effectiveness varies across models and that clean-data performance alone is insufficient to characterize robustness. It presents RI and AR-NIDS results within the evaluated NF-UQ-NIDS setting.
9. Conclusion
- PGD, FGSM, and DeepFool are the most effective transferable attacks in the analysis, while other attacks show more variable success across models.Source: §9 · p. 12
- RI combines balanced accuracy and macro-F1 on the robustness evaluation set with ASR; LGBM, MLP, and GBDT achieve stronger baseline robustness.Source: §9 · pp. 12–13
- The adversarially trained ensemble reduces average ASR from 0.41 to 0.03 and achieves RI 0.98; the attack classifier achieves 0.97 accuracy on evaluated attack types.AR-NIDS combines the adversarially trained ensemble with an adversarial attack classifier.Source: §9 · p. 13