Source-linked AI summary
Evaluating ML-based Intrusion Detection Systems: The Illusion of Model Efficacy
Achilleas Spanos, Ioanna Kantzavelou
TL;DR
ML-based NIDS can appear highly effective when evaluated on known, in-distribution attacks, while their ability to generalize to unseen attacks remains insufficiently tested. The paper evaluates classifier generalization through controlled intra-dataset experiments that vary feature dimensionality, finds generalization weaknesses, and proposes seven evaluation criteria.
Problem
High in-distribution accuracy does not establish strong detection performance, while limited research tests ML-based NIDS against unseen attacks and operational generalization.
Method
The paper tests classifier generalization with two experimental methodologies and feature-selection approaches, including full features, local SHAP, and global feature rankings.
Results
The experiments reveal generalizability challenges, including a cross-day performance reduction interpreted as a generalization gap under consistent labeling and preprocessing.
Takeaways & Limitations
Seven evaluation criteria shift ML-based NIDS assessment toward generalization, novelty handling, robustness, transparency, and continued research.
Takeaways & Limitations
Static top-N feature selection can produce models tied to known attacks and constrain generalization to emerging threats.
Abstract
from arXiv · showhide
Intrusion Detection has been revolutionized due to the integration of Machine Learning. Improved detection rates, reduced false alarms, and optimized algorithms contribute to the perception of improved systems with optimal accuracy and near-perfect performance, the illusion of model efficacy. However, the value of this effectiveness diminishes when confronted with unseen attacks. In this paper, we go beyond solely algorithmic enhancements and metric adjustments in ML-based Network Intrusion Detection Systems. We design an experiment to test the generalization capabilities of certain classifiers on unseen attacks. Our approach examines the dimensionality parameter's impact through two experimental methodologies, which are applied in two distinct settings. The experimental findings reveal how effectively the models could identify even a fraction of unseen attacks and underscore structural weaknesses in ML-based IDS research and evaluation techniques. Finally, seven evaluation criteria are outlined to address these challenges.
1 Introduction
The paper argues that strong in-distribution metrics can conceal weak detection of unseen attacks. It tests classifier generalization and reframes ML-based NIDS evaluation around security needs rather than benchmark metrics.
- High accuracy on attacks within the training distribution does not indicate strong detection performance.
- Current studies increasingly evaluate known attacks and in-training-distribution data while pursuing incremental algorithmic and metric improvements.
- The paper designs an experiment to test classifier generalization and assess underlying and hidden risks in ML-based NIDS.
- The paper prioritizes zero-day robustness and consolidates evaluation criteria that reframe the research gap around security necessities rather than benchmark metrics.
- The experiments examine dimensionality through three feature-selection approaches, two methodologies, and two experimental settings.
2 Background and Related Work
Prior work documents poor transfer beyond training data, but often confounds unseen attacks with domain shifts or lacks criteria derived from empirical failures. This paper keeps the network environment intact and links generalization findings to reporting criteria.
- ML-based NIDS studies frequently report optimal benchmark detection while limited work examines whether scores reflect operational capabilities.
- Cross-dataset studies found supervised and unsupervised classifiers unable to detect unseen traffic, including evaluations restricted to the same attack classes.
- Near-perfect within-dataset scores do not generalize across datasets, where attack–dataset pairs are not isolated.
- Most prior studies vary network environments between training and testing, making domain shift and attack novelty contributors to observed performance drops.
- The paper uses an intact network environment for cross-attack intra-dataset evaluation and reports that its criteria are derived from empirical generalization failure.
3 Experimental Methodology
The methodology evaluates classifiers trained on one daily file and tested on other days to isolate unseen-attack generalization without domain shifts. It compares full, local, and global feature-selection settings under controlled configurations.
- The core test uses cross-attack intra-dataset evaluation to assess classifier generalization to unseen attacks in a controlled setting.
- 3.1 Dataset Selection: CICIDS2017 is selected for its balance between recency and broad adoption, with daily capture files providing the experimental structure.
- 3.1 Dataset Selection: The Monday file is excluded because it contains only benign data, while remaining daily files serve as independent datasets.
- Training on each daily file and testing on the others preserves the network environment while introducing temporal variation and unseen attacks.
- Experimental Study I: Full Featured Set: Study I retains all 77 features, whereas Study II compares local and greedy global feature selection to examine dimensionality effects.
- The experiments use binary classification, default classifier settings, fixed random state zero, and test-set random undersampling to support consistent comparisons.
- Experimental Study II: Feature Selection Set: Study I evaluates Random Forest and XGBoost with SMOTE and Random Under-Sampling, while Study II estimates feature importance using tree splits, permutation, and SHAP methods.
- Experimental Study II: Feature Selection Set: Local selection chooses the top 20 SHAP features per training dataset, while global greedy selection uses an aggregated ranking to select the same 20 features across test sets.
4 Results
The experiments evaluate classifier generalization across train-test days and examine how feature-selection strategies affect cross-day detection. Within-day accuracy remains strong, but unseen-attack generalization declines, with global feature selection outperforming local selection and the full feature set.
- Cross-day evaluation: Each model was trained on one day-dataset and tested on others, exposing it to attack classes absent from training.Unseen attacks are those not present in the training distribution.
- Experimental Study I: XGBoost with Z-normalization and Random Under-Sampling using all features showed the strongest generalization among tested configurations.SMOTE with XGBoost exhibited comparable prediction patterns.
- Experimental Study I: Off-diagonal detection rates declined relative to diagonal intra-day results, especially for early-day datasets.Diagonal dominance is expected because train and test data come from the same day.
- Experimental Study II: Spearman correlations between local and global SHAP feature rankings varied substantially across days.Higher correlation indicates closer agreement between local and globally aggregated feature ordering.
- Experimental Study II: Same-day accuracy remained optimal across feature-selection approaches, while cross-day generalization was especially difficult for Local feature selection.The comparison covers the full feature set, Local SHAP, and Global feature selection.
- Experimental Study II: Global feature-importance rankings consistently outperformed Local SHAP and the full feature set for cross-day generalization.The results indicate that globally aggregated importance better represents feature generalization across multiple attack vectors.
5 Results Interpretation
The experiments expose a gap between strong in-distribution performance and inconsistent detection of unseen attacks. Cross-day and cross-attack results show that generalization is limited, asymmetric, and sensitive to feature-selection choices.
- The Illusion of Model Efficacy: Near-zero unseen-attack detection produced many false negatives and reduced overall accuracy below 50%.The objective was to detect even a fraction of novel attacks rather than optimize same-distribution detection.
- The Illusion of Model Efficacy: 99.92% accuracy was achieved in ten-fold cross-validation, with SD = 0.01%, yet this holistic score did not reflect cross-day generalization.The reported near-perfect result arose from a pooled daily-dataset setup with a 75-25 train-test split.
- Cross-Attack Generalization: 93.6% accuracy was obtained when a DoS-trained classifier was tested on Web Attack traffic, while other cross-attack results ranged from 60.21% to 91.93%.These results were associated with structural or behavioral similarities between attack types.
- Cross-Attack Generalization: Cross-attack generalization was inconsistent and asymmetric, with classifiers often transferring in one direction but not the reverse.The findings suggest reliance on attack-specific features rather than universal behavioral patterns.
- Feature Selection Impact: Feature selection may improve efficiency while constraining adaptability, and static top-N selection can overfit known attacks.The paper therefore calls for feature-selection mechanisms that promote generalization while maintaining interpretability.
- Cross-Day Generalization: High same-day accuracy can coexist with substantial cross-day performance drops, indicating distributional fragility even under consistent dataset structure.The authors interpret the reduction as a generalization gap under consistent labeling and preprocessing assumptions.
6 Evaluation Criteria for ML-based Intrusion Detection Research
The paper translates structural weaknesses in ML-based IDS evaluation into seven criteria covering generalization, novelty detection, interpretability, reproducibility, robustness, feature selection, and multimodal inputs. These criteria connect reporting requirements and testing protocols to evidence from the experiments and prior literature gaps.
- Motivation: The evaluation framework addresses failures in recognizing different traffic or attack families despite high accuracy on pooled or in-distribution datasets.The criteria aim to connect reported research performance with real-world effectiveness.
- Criterion 1: Cross-dataset Generalization Testing: Cross-dataset generalization testing should evaluate IDS performance on unseen traffic distributions and report a matrix spanning network configurations and attacks.Changing domains shifts the benign baseline, topology, and attack perception together.
- Criterion 2: Novel and Zero-day Attack Detection: Novel-attack evaluation should use Leave-One-Attack-Out testing and report recall for each excluded attack family.A model’s ability to detect novel attacks cannot be assumed from surface-level accuracy, including 99.92% accuracy.
- Criterion 3: Explainability and Interpretability: Interpretability is presented as operationally useful because feature-level categories can become human-readable signatures for Snort, Suricata, or Zeek.The criterion extends beyond treating IDS models as black boxes.
- Criterion 4: Reproducibility and Fair Comparison: Reproducibility requires disclosure of data sources, feature-reduction settings, hyperparameters, random seeds, dataset versions, and data-integrity evaluation.Evaluation on a single or outdated dataset without justification is considered insufficient.
- Criterion 5: Adversarial Robustness: Robustness evaluation should include evasion and poisoning techniques under adversary assumptions that reflect real-world network constraints.The criterion challenges evaluations that assume complete white-box knowledge of the detection pipeline.
- Criterion 6: Feature Selection Beyond Metric Improvements: Feature-selection evaluation should examine generalization beyond metric improvements because static top-N feature sets can overfit known attacks.The paper recommends attack-agnostic feature-selection approaches.
- Criterion 7: Multimodal Detection Techniques: Multimodal evaluation should consider both flow-level features and raw packet payloads because reliance on one format limits coverage.Flow features are susceptible to manipulation, whereas payload inspection captures semantic attack contents.
7 Conclusions
The paper concludes that ML-based IDS research should move from isolated metric improvements toward generalized behavioral attack profiles. Applying the proposed criteria is intended to narrow the gap between reported performance and real-world effectiveness.
- Conclusion: ML-based IDS literature could be reframed from slight metric improvements in a single distribution domain toward a generalized behavioral attack profile.The proposed criteria collectively address generalization, novelty handling, robustness, transparency, and research continuation.
- Conclusion: The absence of these evaluation criteria can let a detector appear effective while concealing its inability to perform its intended task.The paper presents adoption of the recommended tests and methodologies as a way to narrow the gap between reported and real-world effectiveness.