PaperlayerView exact revision on arXiv

Source-linked AI summary

A Three-Axis Stress Test of LLM vs Classical ML for Network Intrusion Detection under Distribution Shift and Adversarial Evasion

Muhammad Ebad Atif, Muhammad Haider Ali

arXiv:2609.13511v1cs.LGcs.AIcs.CR

TL;DR

Same-dataset evaluation leaves open how NIDS models perform across independently collected networks and against adversarially crafted traffic. The paper compares XGBoost and RoBERTa-LoRA across three axes, finding that their ranking reverses by axis rather than producing a universal winner.

  • Problem

    NIDS comparisons commonly use same-dataset splits, which do not test deployment on a different network or against deliberately evasive traffic.

    Source: §1 · p. 1

  • Method

    The paper evaluates XGBoost and RoBERTa-LoRA on two independently collected NetFlow v2 networks across same-dataset performance, cross-dataset transfer, and adversarial evasion.

    Source: §1 · p. 1

  • Results

    Across the three axes there is no universal winner; XGBoost leads by roughly 15 F1 points in UNSW→CIC transfer, while RoBERTa-LoRA leads under adversarial evasion.

    Source: §5 · p. 8, §7 · p. 10

  • Takeaways & Limitations

    Model selection for NIDS deployment should follow the anticipated threat model rather than a single benchmark score.

    Source: §5 · p. 8, §7 · p. 10

    Findings come from exactly two networks, which demonstrates directional, asymmetric transfer but cannot establish how that asymmetry generalizes more broadly.

    Source: §6 · p. 10

Abstract

from arXiv · show

Large language models are increasingly benchmarked against classical machine learning for network intrusion detection (NIDS), almost always using same-dataset evaluation, and that protocol turns out to be incomplete. Evaluating XGBoost and RoBERTa-LoRA on two independently collected NetFlow v2 networks across three axes (same-dataset performance, cross-dataset transfer, and adversarial evasion) reveals no universal winner. The two models are statistically tied same-dataset. XGBoost wins decisively under cross-dataset distribution shift, by 15 points of F1 and 25 points of balanced accuracy; on the target network RoBERTa-LoRA's false positive rate reaches 0.78, leaving it barely above chance despite a superficially moderate F1. RoBERTa-LoRA wins decisively under adversarial evasion, by roughly 17 points of F1 at a representative mid-range perturbation strength, while both models hold false positive rates below 0.01 throughout. The model an evaluator would recommend therefore depends entirely on which axis is tested, not on same-dataset accuracy alone. A staged feature-leakage ablation improves cross-dataset transfer non-monotonically, indicating the leakage signal is distributed across the feature representation rather than confined to a few columns, and cross-dataset transfer between our two networks is strongly directional. These results argue for evaluating NIDS models along multiple independent robustness axes, and with more than one metric per axis.

1. Introduction

The paper evaluates XGBoost and RoBERTa-LoRA across same-dataset performance, cross-dataset transfer, and adversarial evasion on two independently collected NetFlow v2 networks. The models have no universal ranking, and the preferred model depends on the evaluation axis.

1. Introduction

  • Same-dataset benchmarks test neither deployment on a different network nor traffic deliberately crafted to evade detection.The paper connects this concern to Sommer and Paxson ’s critique of closed-world evaluation and surveys of inflated NIDS results [2].Source: §1 · p. 1
  • Earlier comparisons examined individual evaluation axes, leaving cross-dataset transfer between independently collected networks and adversarial evasion unevaluated together.The paper argues that these axes can yield opposite conclusions about which model is preferable.Source: §1 · p. 1
  • The models are statistically tied same-dataset, XGBoost leads under cross-dataset shift, and RoBERTa-LoRA leads under adversarial evasion by roughly 17 F1 points.Cross-dataset transfer between the two networks is also strongly directional; its proposed link to differing attack character was not directly verified.Source: §1 · pp. 1–2
  • A staged feature-ablation study finds that leakage removal does not improve cross-dataset transfer monotonically, suggesting leakage is distributed across features rather than confined to a few columns.Transfer between the two networks is strongly directional, with differing attack character proposed as a link but not directly verified.Source: §1 · pp. 1–2

2. Related Work

Prior LLM-versus-classical NIDS studies examined same-dataset comparisons or within-network distribution shift, but not cross-dataset transfer and adversarial evasion together. This paper places its evaluation alongside those studies and related evidence on transfer failure.

2. Related Work

  • Mehavilla et al. found XGBoost narrowly ahead of fine-tuned LLMs on same-dataset accuracy, while pre-trained LLMs were more data-efficient.Their evaluation used CIC-IoT-2023 and did not test transfer or adversarial robustness.Source: §2 · p. 2
  • Bui/Boffa et al. found fine-tuned BERT ahead in-distribution but sharply degraded under a time-based zero-day split.Their study did not test adversarial evasion or transfer between independently collected networks.Source: §2 · p. 2
  • Neither directly comparable study evaluated cross-dataset transfer between independently collected networks and adversarial evasion on the same matched models.The present study addresses this gap with XGBoost and RoBERTa-LoRA under a shared harness across same-dataset performance, bidirectional transfer, and evasion.Source: §2 · p. 2
  • Cantone et al. [5] independently reported near-perfect same-dataset performance collapsing under distribution shift across four classifiers and four NIDS datasets.Other cited work also reports generalization failures in adjacent settings, while a tabular-representation comparison finds no single approach consistently dominates across scenarios [7].Source: §2 · p. 2

3. Methodology

The methodology combines standardized, independently collected NetFlow v2 datasets with leakage auditing and a shared harness for same-dataset, cross-dataset, and adversarial-evasion evaluation. Its staged ablation shows that cross-dataset transfer responds non-monotonically to feature removal, even as same-dataset F1 remains nearly unchanged.

3.1. Datasets

  • Two independently collected NetFlow v2 datasets share 43 features, but their disjoint attack taxonomies require binary cross-dataset classification.Their different collection origins and shared schema support comparing transfer between networks rather than between incompatible feature definitions.Source: §3.1 · p. 3

3.2. Preprocessing

  • Balanced 50,000-per-class subsets control attack prevalence; session identifiers and subtype labels are excluded, leaving benign-versus-attack prediction.This isolates distribution shift from prevalence differences, although the balanced protocol makes absolute precision unlike an operational estimate.Source: §3.2 · p. 3

3.3. Feature Ablation Study

  • A held-out one-feature stump flags solo F1 above 0.90 for scrutiny, identifying TTL, packet-length, and TCP-statistic suspect groups.High solo F1 triggers investigation, not automatic removal; cross-dataset transfer is used to assess suspected leakage.Source: §3.3 · p. 4
  • Cross-dataset F1 fell from 0.0488 in Set A to 0.0282 in Set B, then recovered to 0.8116 in Set C, while same-dataset F1 stayed near 0.997.The authors interpret this non-monotonic pattern as evidence that leakage signal is distributed across the feature representation.Source: §3.3 · p. 4

3.4. Models

  • XGBoost uses four seeds while RoBERTa-LoRA uses one; serialized feature names also give RoBERTa structural input unavailable to numeric tree models.RoBERTa uses LoRA adapters, with approximately 0.82% of its parameters trainable.Source: §3.4 · pp. 4–5

3.5. Adversarial Evasion Attack

  • The evasion attack interpolates attack flows toward the training benign centroid, then projects them to constraints; 23 of 29 features vary, but realizability is not guaranteed.Projection enforces non-negativity, integer packet and byte counts, retransmitted-byte bounds, and fixed categorical features.Source: §3.5 · p. 5

3.6. Evaluation Protocol

  • A shared harness scores three conditions with F1, precision, recall, balanced accuracy, FPR, FNR, and throughput; evasion sets retain benign flows to measure false alarms.The conditions are same-dataset, cross-dataset transfer in both directions, and adversarial evasion.Source: §3.5 · p. 5, §3.6 · pp. 5–6

3.5. Adversarial Evasion Attack

  • At ε=1, perturbations collapse to one distinct flow, so that condition is excluded and ε≤0.7 is treated as valid; evasion is tested only on UNSW.The evasion and cross-dataset stress axes are evaluated independently.Source: §3.5 · p. 5

4. Results

Across three evaluation axes, the models have no universal winner: they tie same-dataset, XGBoost leads cross-dataset, and RoBERTa-LoRA leads under evasion.

4.1. Same-Dataset Performance

  • Same-dataset F1 gaps never exceed 0.004, while XGBoost's complete inference pipeline processes about 5,806× more flows per second.RoBERTa-LoRA throughput includes per-flow text serialization and tokenization; XGBoost throughput measures native numeric prediction on vectorized input.Source: §4.1 · p. 6, Figure 1 · p. 6

4.2.1. Feature ablation is non-monotonic.

  • Cross-dataset F1 worsens from Set A to B (0.0488 to 0.0282), then recovers at Set C (0.8116), while same-dataset F1 stays 0.9966–0.9972.The non-monotonic pattern indicates leakage signal is distributed across the feature representation, rather than concentrated in TTL and packet-length features.Source: Figure 2 · p. 6, Table 2 · p. 6, §4.2.1 · pp. 6–7

4.2.2. Transfer is asymmetric under the locked feature set.

  • UNSW→CIC favors XGBoost by 15.3 F1 points (0.8116 vs. 0.6582); RoBERTa-LoRA's FPR is 0.7756 and balanced accuracy is 0.5477.In the reverse CIC→UNSW direction, every model collapses to 0.0201–0.0730 F1; XGBoost has zero seed variance, while Random Forest varies due to bagging randomness.Source: §4.2.2 · p. 7

4.3. Adversarial Evasion

  • At ε = 0.5, RoBERTa-LoRA leads by 17.3 F1 points (0.9448 vs. 0.7720); both models' FPRs stay below 0.01 throughout.FPR is 0.0057 for XGBoost and 0.0081 for RoBERTa-LoRA at every ε, because benign traffic is not perturbed.Source: §4.3 · pp. 7–8, Table 6 · p. 7

4.4. Summary Across Axes

  • Across axes, neither model wins universally: same-dataset results tie, XGBoost leads UNSW→CIC transfer, and RoBERTa-LoRA leads under evasion.Table 7 collects all three axes; no single benchmark number characterizes deployment-relevant performance on its own.Source: Figure 5 · p. 8, Table 7 · p. 8, §4.4 · p. 8

5. Discussion

XGBoost and RoBERTa-LoRA have no universal ranking: the preferred model changes across same-dataset, cross-dataset, and adversarial evaluations. The discussion also shows why F1 alone can obscure failure modes and treats proposed explanations for ablation, transfer, and evasion patterns as unverified hypotheses.

5. Discussion

  • XGBoost and RoBERTa-LoRA are within 0.004 F1 same-dataset, while XGBoost leads transfer and RoBERTa-LoRA leads evasion by 17–35 F1 points.The authors recommend choosing according to the anticipated threat model; cross-network deployment favors XGBoost, while adaptive feature manipulation favors RoBERTa-LoRA if inference budget permits.Source: §5 · p. 8
  • RoBERTa-LoRA’s cross-dataset F1 of 0.6582 masks a near-chance classifier: balanced accuracy is 0.5477 and FPR is 0.7756.On the adversarial axis, retained benign flows make FPR distinguish sustained detection (0.0081) from the cross-dataset failure mode; attacks-only evaluation cannot make that distinction.Source: §5 · pp. 8–9
  • Removing more candidate leakage features in Set B worsens cross-dataset transfer relative to Set A, despite unchanged same-dataset F1.The proposed competing-signal account is that packet-length features carry both environment-correlated leakage and transferable behavioral information; this interpretation is not established by the ablation alone.Source: §5 · p. 9
  • UNSW→CIC transfer is workable for XGBoost (0.8116 F1; 0.7979 balanced accuracy), but CIC→UNSW collapses to chance for every model.The proposed attack-character account is consistent with the observed directionality, but subtype recall is needed to resolve it.Source: §5 · p. 9
  • XGBoost’s F1 varies by roughly 38 points across valid perturbations, versus about 10 for RoBERTa-LoRA.Hard, axis-aligned tree splits and structural patterns in serialized FEATURE=value text are proposed explanations, not verified mechanisms; the paper calls for feature attribution and serialization diagnostics.Source: §5 · p. 9

6. Limitations

The reported comparisons have limits in run replication, checkpoint selection, attack feasibility and design, network coverage, model scope, and verification of residual leakage. These boundaries qualify the strength and generality of the paper’s robustness findings.

6. Limitations

  • Every RoBERTa-LoRA result comes from one training run, unlike the classical arm’s four seeds, so comparisons involving the LLM are point estimates without equivalent variance estimates.The authors identify multi-seed LLM evaluation with proper significance testing as the most direct next step.Source: §6 · p. 9
  • About 13% of each reported same-dataset evaluation set participated in RoBERTa checkpoint selection, potentially making its same-dataset and adversarial figures marginally optimistic.Cross-dataset figures are unaffected because the target network is not used in training or selection.Source: §6 · p. 9
  • The feature-space attack does not establish that perturbed flows are realizable as raw traffic, so adversarial results are an upper bound on robustness.The study does not implement an explicit problem-space attack construction.Source: §6 · pp. 9–10
  • The evasion attack moves every attack flow toward the global benign centroid and degenerates at ε = 1, so its robustness gap may not generalize to per-flow or cluster-targeted attacks.The paper notes that a per-flow or cluster-centroid target could preserve more diversity.Source: §6 · p. 10
  • Findings from exactly two networks cannot establish how directional transfer asymmetry generalizes more broadly.Binary labels also prevent subtype-level transfer analysis, which the authors propose as a low-cost next step.Source: §6 · p. 10
  • The LLM comparison covers only RoBERTa-base with LoRA, leaving other architectures, model sizes, and fine-tuning strategies unevaluated.The authors state that the qualitative findings have not been verified for these alternatives and that a larger or decoder-only model could behave differently, particularly under evasion.Source: §6 · p. 10
  • Residual environment-correlated signal may remain in Set C, and the ablation’s leakage interpretation was not independently checked with feature attribution.Near-zero same-dataset F1 movement across the ablation sets is consistent with this possibility.Source: §6 · p. 10

7. Conclusion

Across the three evaluated axes, neither model wins universally, and a single headline metric can conceal important performance differences. The authors conclude that NIDS evaluation should assess multiple independent robustness axes with more than one metric each.

7. Conclusion

  • The two models tie same-dataset, XGBoost leads under distribution shift, and RoBERTa-LoRA leads under adversarial evasion at comparable false positive rates.A single false-positive diagnostic confirms RoBERTa-LoRA’s adversarial robustness while exposing its cross-dataset F1 as concealing a near-chance classifier; transfer directionality and distributed leakage remain qualified findings.Source: §7 · p. 10

LLM Usage Statement

The authors used Claude to assist with manuscript preparation, pipeline implementation, and experimental-design discussion. They report that all experiments and results were author-validated and take responsibility for the paper.

LLM Usage Statement

  • Claude assisted with drafting and editing, implementing and debugging the pipeline, and reviewing experimental design and result interpretation.The authors say they verified and corrected issues identified during that review, including the attacks-only adversarial set and maximal-perturbation degeneracy.Source: LLM Usage Statement · p. 11
  • The authors state that they designed, executed, and validated all experiments and independently checked reported numbers against recorded outputs.They also state that cited works were verified and that they take full responsibility for the paper.Source: LLM Usage Statement · p. 11
Loading 2609.13511v1…