Source-linked AI summary
From Zero-Shot Machine Learning to Zero-Day Attack Detection
Mohanad Sarhan, Siamak Layeghy, Marcus Gallagher, Marius Portmann
TL;DR
ML-based NIDSs are commonly evaluated on known attack classes, leaving limited evaluation of their ability to detect new classes that emerge after training. This paper applies a zero-shot learning methodology and a new detection metric to that scenario; most attack classes were detected, but some showed lower rates alongside distinct feature distributions.
Problem
ML-based NIDS evaluation has focused largely on known attacks, although deployed systems may encounter attack classes absent from training.
Source: §1 · pp. 2–3Method
Results
Most attack classes were reliably detected, but Fuzzers, Exploits, and Analysis had lower zero-day detection rates; average Z-DR was 92.45% for MLP and 87.37% for RF.
Source: §5.1 · p. 11, §6 · p. 14Takeaways & Limitations
Low zero-day detection rates for some sophisticated attacks coincided with feature distributions distinct from those of attacks seen during training.
Source: §6 · p. 14The Wasserstein Distance analysis is not perfect, and further studies are required to improve detection of unique behavior in sophisticated attacks.
Source: §5.2 · p. 14
Abstract
from arXiv · showhide
The standard ML methodology assumes that the test samples are derived from a set of pre-observed classes used in the training phase. Where the model extracts and learns useful patterns to detect new data samples belonging to the same data classes. However, in certain applications such as Network Intrusion Detection Systems, it is challenging to obtain data samples for all attack classes that the model will most likely observe in production. ML-based NIDSs face new attack traffic known as zero-day attacks, that are not used in the training of the learning models due to their non-existence at the time. In this paper, a zero-shot learning methodology has been proposed to evaluate the ML model performance in the detection of zero-day attack scenarios. In the attribute learning stage, the ML models map the network data features to distinguish semantic attributes from known attack (seen) classes. In the inference stage, the models are evaluated in the detection of zero-day attack (unseen) classes by constructing the relationships between known attacks and zero-day attacks. A new metric is defined as Zero-day Detection Rate, which measures the effectiveness of the learning model in the inference stage. The results demonstrate that while the majority of the attack classes do not represent significant risks to organisations adopting an ML-based NIDS in a zero-day attack scenario. However, for certain attack groups identified in this paper, such systems are not effective in applying the learnt attributes of attack behaviour to detect them as malicious. Further Analysis was conducted using the Wasserstein Distance technique to measure how different such attacks are from other attack types used in the training of the ML model. The results demonstrate that sophisticated attacks with a low zero-day detection rate have a significantly distinct feature distribution compared to the other attack classes.
1 Introduction
ML-based NIDSs use learned traffic behavior rather than signatures to detect attacks, but their ability to detect attack classes absent from training has received limited evaluation. This paper proposes a zero-shot learning evaluation that tests this post-deployment scenario using semantic attributes learned from known attacks.
1 Introduction
- ML models learn patterns from data, including complex patterns that may be difficult for domain experts to realize.The paper situates this capability as motivation for applying ML in cybersecurity.Source: §1 · pp. 1–2
- Signature-based NIDSs can miss zero-day attacks because the complete set of attack indicators may not yet be discovered or registered.The passage reports an average of 312 days to obtain the full set of attack IOCs.Source: §1 · p. 2
- ML-based NIDSs learn traffic behavior, but much prior evaluation uses known attack groups and does not test the likely appearance of a new class after deployment.The paper identifies limited research measuring zero-day detection benefits over signature-based NIDSs.Source: §1 · p. 2
- Zero-shot learning evaluates recognition of unseen classes by learning distinguishing semantics and applying them during inference.The approach addresses the challenge of obtaining training samples for every class a deployed model may encounter.Source: §1 · pp. 2–3
- The paper proposes a zero-shot evaluation of ML-based NIDSs that maps known-attack features to semantic attributes and uses relationships between known and unseen attacks to assess zero-day detection.It introduces Zero-day Detection Rate (Z-DR) and analyzes results using Wasserstein Distance.Source: §1 · p. 3
2 Related Works
Prior work on zero-day NIDS detection includes signature-based, anomaly-detection, and zero-shot approaches, but the review identifies few ZSL evaluations using modern network datasets. It argues that reliance on older attack scenarios limits the reliability of evaluations for contemporary threats.
2 Related Works
- Snort detected 17% of 183 unknown attacks among 356 tested attacks, which the cited study argued was above zero but still unreliable.The study proposed the result as a baseline and recommended complementary mechanisms for unregistered attacks.Source: §2 · pp. 3–4
- An autoencoder study reported zero-day detection accuracy of 89–99% on NSL-KDD and 75–98% on CICIDS2017, but did not measure undetected attacks or false alarms.Its models trained on benign samples and did not account for attack behavior.Source: §2 · p. 4
- A sparse-autoencoder ZSL study using NSL-KDD reported 88.3% average accuracy across its four available attack scenarios.The dataset was released in 1998.Source: §2 · p. 4
- A Random Forest feature-selection and spatial-clustering approach detected DoS (apache2) and Probe (saint) attacks at 34.71% overall accuracy, versus 13.59% for a decision tree.The evaluation used NSL-KDD.Source: §2 · p. 4
- Only a few reviewed studies use zero-shot learning for zero-day detection, and the review finds little experimental work with modern datasets and attack types.It notes that NSL-KDD is more than 20 years old and its attack scenarios do not represent modern network traffic and threats.Source: §2 · pp. 3–4
3 Proposed Methodology
The proposed ZSL-based methodology evaluates whether an ML-based NIDS can detect attack classes withheld from training. It learns semantic attributes from known attacks and benign traffic, then tests whether those attributes help identify the unseen class as malicious.
3 Proposed Methodology
- Traditional ML evaluation trains and tests on the same classes, whereas zero-shot learning addresses recognition of classes without training samples by capturing distinguishing knowledge and applying it at inference.The traditional setup assumes the training data covers the classes the model will encounter after deployment.Source: §3 · pp. 4–5
- The methodology maps network features from known attacks and benign traffic to semantic attributes, then uses relationships between known and unseen attacks to classify the unseen group as malicious.The attributes distinguish attack from benign traffic and represent behavior learned from known attacks.Source: §3 · p. 6
- For each attack class z, the training set excludes that class while the test set retains samples from all classes, simulating an attack that emerges after training.The setup evaluates whether a model trained on the other attack classes detects the withheld class.Source: §3 · p. 6
- The inference phase adds the withheld attack class to testing and measures zero-day detection in a generalized setting that includes both seen and unseen samples.The paper presents this as more practical than testing only unseen samples because a network test set may include known attacks and benign traffic too.Source: §3 · p. 6
- The evaluation is intended to assess post-deployment generalization and inform model hyperparameter, architecture, and feature-selection experiments.The paper connects reliable zero-day detection to the inability of signature-based systems to identify all indicators of a future attack class.Source: §3 · pp. 6–7
4 Experimental Setup
The setup evaluates Random Forest and Multi-Layer Perceptron models on two UNSW-NB15 datasets, using standard classification metrics and a class-specific zero-day detection rate.
4 Experimental Setup
- The experiments use Random Forest with 50 decision trees and an MLP with two 100-neuron hidden layers, evaluated using five-fold cross-validation.The MLP uses ReLU activation and stochastic gradient-based optimization.Source: §4 · p. 7
4.1 Datasets
- Two synthetic NIDS datasets, UNSW-NB15 and NF-UNSW-NB15-v2, provide modern attack scenarios and benign traffic for evaluation.UNSW-NB15 contains 2,540,044 samples with 49 features; NF-UNSW-NB15-v2 contains 2,390,275 flows with 43 NetFlow features.Source: §4.1 · pp. 7–8
4.2 Zero-day Detection Rate
- Zero-Day Detection Rate (Z-DRz) measures detection of an attack class excluded from training, using information learned from other attack classes.The evaluation treats classification as binary: benign versus attack traffic.Source: §4.2 · p. 9
5 Evaluation
The evaluation tests whether models can detect attack classes withheld from training, then examines how feature-distribution differences relate to zero-day detection. Most attacks were detected well, but results varied by attack group and dataset; distinctive distributions aligned with lower detection rates.
5 Evaluation
- The experiments used MLP and RF on UNSW-NB15 and NF-UNSW-NB15-v2, treating each available attack as a simulated zero-day and evaluating Z-DR alongside whole-test-set accuracy.The test set included known attacks, the withheld attack, and benign samples.Source: §5 · p. 9, §5.1 · p. 9
5.1 Results
- MLP and RF average Z-DR rose from 85.5% and 80.67% on UNSW-NB15 to 92.45% and 87.37% on NF-UNSW-NB15-v2, around 7% higher with NetFlow features.The datasets contain the same attack groups but differ in feature sets.Source: Table 2 · p. 9, Table 3 · p. 10, §5.1 · pp. 10–11
- On NF-UNSW-NB15-v2, Exploits and Fuzzers were harder to detect, while Generic, Reconnaissance, Backdoor, and Shellcode were detected close to 100%.MLP outperformed RF on Exploits and Fuzzers, with rates of 82% versus 59% and 76% versus 51%, respectively.Source: §5.1 · p. 11, Table 4 · p. 11, Table 5 · p. 11
5.2 Analysis
- The analysis compared feature distributions between known-attack training data and test data containing a zero-day attack, using per-feature Wasserstein distances and averages across features.Higher Wasserstein distance indicates a more distinctive distribution between the training and test sets.Source: §5.2 · pp. 12–13
- Most attack groups had average Wasserstein distance around 0.2 and high zero-day detection, with only minor degradation from known-attack detection.This pattern included Analysis, Backdoor, DoS, Reconnaissance, Shellcode, and Worms; similar feature distributions were associated with useful learned patterns for unseen attacks.Source: §5.2 · p. 13
- Higher-distance exceptions—UNSW Fuzzers and Exploits (0.5), Generic (1.0), and NF Fuzzers (0.9) and Exploits (1.0)—had lower Z-DR; the authors describe the analysis as not perfect.The paper says further study is needed to improve detection of unique attack behaviour related to sophisticated attacks.Source: §5.2 · pp. 13–14
6 Conclusion
The proposed zero-shot evaluation uses attributes learned from known attacks to assess unseen-attack detection, finding significant detection for most attacks but lower rates for Fuzzers, Exploits, and Analysis. Those groups’ feature distributions differ from training attacks and correlate with their lower detection rates.
6 Conclusion
- The evaluation maps network features to semantic attributes of known attacks, then associates those attributes with unseen attack behavior to assess maliciousness.Two ML models were evaluated on attacks in the UNSW-NB15 and NF-UNSW-NB15-v2 datasets as zero-day attacks.Source: §6 · p. 14
- The ML-based NIDS detected most zero-day attacks to a significant extent, but Fuzzers, Exploits, and Analysis had much lower Zero-day detection rates.A model trained on all other attack types was unable to generalise to these groups.Source: §6 · p. 14
- Wasserstein Distance between feature distributions correlated strongly with Zero-day detection rates; the low-rate groups’ distributions differed from those of training attacks.The authors report that these differences prevented the model from generalising to and detecting those groups in the test phase.Source: §6 · p. 14