Source-linked AI summary
Adversarial Machine Learning in Network Intrusion Detection Systems
Elie Alhajjar, Paul Maxwell, Nathaniel D. Bastian
TL;DR
Machine-learning NIDS may be vulnerable to adversarial perturbations that make malicious traffic appear normal. The paper evaluates PSO, GA, GAN, and Monte Carlo perturbations on NSL-KDD and UNSW-NB15, finding high misclassification rates across many classifiers and highlighting this vulnerability.
Problem
The paper examines the vulnerability of machine-learning NIDS to adversarial examples that evade detection.
Method
The study generates adversarial examples with PSO, GA, GAN, and Monte Carlo methods and evaluates them on NSL-KDD and UNSW-NB15.
Results
GA achieved 73.71% versus 5.00% accuracy for adversarial vectors compared with original vectors on UNSW-NB15.
Takeaways & Limitations
The results highlight the vulnerability of machine-learning NIDS to adversarial perturbation, with SVM and DT exceeding 90% evasion rates.
Takeaways & Limitations
The datasets cannot be directly compared because their attack types, creation dates, training-test composition, and features differ.
Abstract
from arXiv · showhide
Adversarial examples are inputs to a machine learning system intentionally crafted by an attacker to fool the model into producing an incorrect output. These examples have achieved a great deal of success in several domains such as image recognition, speech recognition and spam detection. In this paper, we study the nature of the adversarial problem in Network Intrusion Detection Systems (NIDS). We focus on the attack perspective, which includes techniques to generate adversarial examples capable of evading a variety of machine learning models. More specifically, we explore the use of evolutionary computation (particle swarm optimization and genetic algorithm) and deep learning (generative adversarial networks) as tools for adversarial example generation. To assess the performance of these algorithms in evading a NIDS, we apply them to two publicly available data sets, namely the NSL-KDD and UNSW-NB15, and we contrast them to a baseline perturbation method: Monte Carlo simulation. The results show that our adversarial example generation techniques cause high misclassification rates in eleven different machine learning models, along with a voting classifier. Our work highlights the vulnerability of machine learning based NIDS in the face of adversarial perturbation.
1 Introduction
Machine-learning systems face robustness weaknesses and intentional adversarial attacks, creating a security concern for NIDS. This study tests evolutionary, deep-learning, and Monte Carlo perturbations across two datasets and multiple classifiers.
- Machine-learning robustness requires maintaining performance on data from distributions different from training data.
- Adversarial examples are intentionally designed inputs that cause machine-learning models to make mistakes, including in black-box settings.
- NIDS use machine-learning models to detect abnormal network traffic and novel intrusions, reducing reliance on manually codified rules.
- The study generates adversarial examples with particle swarm optimization, genetic algorithms, and generative adversarial networks.
- The methods are evaluated on NSL-KDD and UNSW-NB15 against eleven machine-learning models, an ensemble model, and Monte Carlo perturbations.
2 Prerequisites
The prerequisites introduce supervised, unsupervised, reinforcement, and adversarial machine learning, together with their data representations, objectives, and attack distinctions.
- Supervised Learning: Supervised learning maps labeled inputs to outputs by minimizing a cost function over observed data.
- Supervised Learning: Classification uses finite labels, with 0 representing benign network traffic and 1 representing malicious traffic.
- Unsupervised Learning: Unsupervised learning analyzes feature-only data to identify structure, including through clustering, principal component analysis, and matrix completion.
- Reinforcement Learning: Reinforcement learning solves tasks through interaction and feedback, with Markov decision processes providing its mathematical foundation.
- Adversarial Machine Learning: Adversarial attacks differ by timing and attacker information, including evasion versus poisoning and white-box versus black-box settings.
3 Related Work
Related work surveys adversarial machine learning across unconstrained and constrained domains, motivating further evidence about adversarial-example generation for network data.
- Prior work developed adversarial attacks for spam classifiers, classifier reverse engineering, and neural-network image recognition.
- Surveys organize adversarial machine-learning attacks and defenses by training or inference phase and discuss transferability.
- Common unconstrained-domain methods include FGSM and JSMA, which generate perturbations using gradients or feature saliency.
- Constrained domains contain binary, continuous, or categorical features, and some features may be fixed or uncontrollable by an adversary.
- Prior constrained-domain experiments achieved misclassification rates greater than 95% with perturbations obeying domain constraints.
- This work adapts GA, PSO, and GAN methods for adversarial examples in unconstrained domains using NSL-KDD and UNSW-NB15, with Monte Carlo as a baseline.
4 Methodology
The methodology section describes the datasets, adversarial-example generation techniques, and computational setting used in the experiments.
- The methodology first describes dataset features, then explains the adversarial-example generation techniques and computational setting.
4.1 Data sets
The study evaluates adversarial perturbations on NSL-KDD and UNSW-NB15, publicly available NIDS datasets with differing traffic composition, feature structure, and quality limitations. Because some network fields are immutable or functionally constrained, perturbations are restricted to mutable fields that preserve traffic functionality.
- Dataset scope: Both datasets mix benign and malicious traffic but have quality limitations involving generation methods, attack prevalence, and dataset size.Despite these limitations, both are commonly used to evaluate machine-learning-based NIDS.
- Feature constraints: Perturbations must preserve traffic functionality because changing fields such as protocol type can cause transmission failure.The study therefore treats some fields as immutable and constrains changes to mutable fields.
- NSL-KDD: NSL-KDD contains over 125,000 training samples, more than 22,000 test samples, 41 original features, and four attack types.After one-hot encoding and Min-Max Scaling, it has 121 features; protocol-type, service, and flag are immutable.
- UNSW-NB15: UNSW-NB15 contains over 175,000 training samples, more than 82,000 test samples, nine modern attack types, and benign traffic more reflective of current network traffic.Its 49 fields become 196 engineered features, of which 23 are mutable.
4.2 Adversarial Example Generation
The paper generates adversarial NIDS inputs using GA, PSO, GAN, and Monte Carlo-inspired perturbations while respecting immutable and mutable feature constraints. GA and PSO search for classifier outputs favoring benign labels, whereas GAN training alternates generator and discriminator updates.
- Overview: The study introduces PSO and GA generators, uses a constrained-feature GAN, and implements Monte Carlo simulation as a baseline.The methods target adversarial examples for evading machine-learning systems while accounting for constrained data fields.
- Genetic Algorithm: GA chromosomes encode dataset features, separating mutable and immutable elements before crossover and mutation operations.Each row creates a chromosome population seeded from the original values and randomly generated variants.
- Genetic Algorithm: GA crossover swaps chromosome portions between randomly selected parents, while mutation changes randomly selected cells according to a mutation rate.Fitness is evaluated after these operations, and the best chromosomes are retained across generations.
- Particle Swarm Optimization: PSO initializes particles from a seed row, assigns velocities, evaluates fitness, and updates particle and global best locations during iterative search.Particle velocities and locations are adjusted using distances to each particle’s best location and the swarm’s global best location.
- Generative Adversarial Network: GAN input divides mutable and immutable features, adds noise to mutable values, and trains a generator to produce adversarial mutable vectors.The generated mutable output is recombined with the immutable input portion before discriminator evaluation.
- Generative Adversarial Network: The constrained GAN supports non-binary features and avoids requiring minimum perturbations because large data changes are not readily recognized by human observers in this domain.The discriminator and generator are alternately trained using adversarial and benign vectors.
4.3 Computational Experiment
The computational experiment runs the perturbation methods on malicious test vectors from NSL-KDD and UNSW-NB15 and evaluates the resulting vectors across eleven classification models. GA and PSO use a soft-voting ensemble as their fitness function, while Monte Carlo provides the baseline.
- Evaluation data: The methods are evaluated on 12,828 malicious NSL-KDD test vectors and 45,328 malicious UNSW-NB15 test vectors.Only malicious vectors are used because the evaluation targets classifiers’ deception by modified malicious inputs.
- Evaluation design: The GA and PSO fitness function is a soft-voting ensemble classifier, and baseline classification is established on the unmodified test set.Monte Carlo simulation supplies the baseline adversarial-example generation method.
- Parameters: GA uses 100 chromosomes, up to 1000 generations, a 0.25 crossover rate, and a 0.2 mutation rate.Its early termination criterion is less than 0.01% improvement over 5 generations.
- Parameters: PSO uses 200 particles, c1 = 0.5, c2 = 0.4, w = 0.7, and up to 100 iterations.The coefficients weight particle-local and swarm-global best solutions, while w weights the last velocity.
- Classifiers: Each perturbation method is tested with eleven classifiers, including Random Forest, MLP, Gradient Boosting, Logistic Regression, LDA, QDA, and Bagging.Four additional evaluation classifiers are the sub-models used in the GA/PSO voting ensemble.
5 Results and Discussion
Across NSL-KDD and UNSW-NB15, adversarial perturbations substantially increased NIDS evasion, but effectiveness varied by classifier, generation method, and dataset. The datasets are not directly comparable because their attacks, features, and train/test compositions differ.
- Dataset Limitations: The two datasets overlap in only five collected fields, making their malicious-traffic features mostly incomparable and preventing comparison based on commonalities.Differences in attack types and NSL-KDD train/test composition further constrain cross-dataset interpretation.
- NSL-KDD Results: PSO, GA, and GAN outperformed MC for SVM, DT, MLP, and BAG on NSL-KDD, while results varied across the remaining classifiers.Against original vectors, evasion was higher for all classifiers except RF, GB, and LR.
- NSL-KDD Results: PSO and GA exceeded GAN for SVM, NB, KNN, and LDA on NSL-KDD, whereas GAN exceeded both evolutionary methods for DT, RF, MLP, GB, LR, and BAG.The paper partially attributes these patterns to the models used in heuristic evaluation and GAN discriminator training.
- NSL-KDD Results: Above 90% evasion: DT showed low variability across NSL-KDD perturbation methods and more than double the original-vector performance.DT was also reported as highly vulnerable on UNSW-NB15.
- UNSW-NB15 Results: 73.71% compared to 5.00% accuracy: on UNSW-NB15, GA achieved the highest average evasion against PSO, GAN, and MC.The comparison is between adversarially perturbed and original vectors.
- UNSW-NB15 Results: Above 95% evasion: BAG, DT, and GB showed low variability across UNSW-NB15 perturbation methods.The results indicate similarly high evasion performance across methods for these tree-based classifiers.
6 Conclusion
The paper finds constrained network intrusion detection systems vulnerable to adversarial perturbations generated by evolutionary computation and deep learning. It reports high evasion against multiple classifiers while leaving differences in model robustness unexplained.
- The study tests whether constrained network domains are vulnerable to adversarial attacks.
- PSO, GA, and GAN perturbations are used to modify malicious traffic so NIDS classify it as normal.
- The PSO, GA, and GAN methods achieve high misclassification rates against multiple machine learning classifiers.
- The paper identifies unexplained differences in classifier robustness and proposes analyzing model internals in future work.
Disclaimer
The authors state that the views expressed are their own and do not represent official policy or positions of the named institutions.
- The views expressed are those of the authors.
- The paper does not represent the official policy or position of the United States Military Academy or United States Army.
- The disclaimer also excludes the Department of Defense and United States Government.