PaperlayerView exact revision on arXiv

Source-linked AI summary

Navigating the Latent Manifold: Proactive Concept Drift Adaptation for Resilient NIDS

Chao Zha, Zifeng Kang, Tian Liu, Dakun Shen, Ruyun Zhang

arXiv:2609.08623v1cs.CR

TL;DR

AI-based NIDS often assume static traffic distributions, leaving a gap in handling concept drift. DriftXpert combines unsupervised drift detection with adaptation designed to preserve historical knowledge without replay. The paper reports drift detection across four scenarios and evaluates its adaptation on public benchmarks and enterprise traffic.

  • Problem

    AI-based NIDS often assume static traffic distributions even though real network traffic evolves, degrading detection and increasing false positives and false negatives.

    Source: §1 · p. 1

  • Method

    DriftXpert uses clustering-based, label-free drift indicators to trigger model updates, with representation consistency, weight aggregation, and selective freezing to preserve knowledge without historical traffic replay.

    Source: §1 · p. 2

  • Results

    Across four scenarios, DriftXpert's deep-clustering method identified concept drift and empirically served as a high-fidelity trigger for adaptation.

    Source: §5.2 · p. 7

  • Takeaways & Limitations

    Experiments on public benchmarks and real-world enterprise traffic support DriftXpert's effectiveness for long-term adaptive NIDS deployment.

    Source: §1 · p. 2

    The threat model assumes naturally occurring drift and excludes adversarial drift.

    Source: §2.3 · p. 3

Abstract

from arXiv · show

Network intrusion detection systems (NIDS) are critical for cybersecurity, safeguarding services and data from potential attacks. However, existing AI-based NIDS often assume static data distributions and fail to handle concept drift, leading to degraded performance and increased false positives in dynamic network environments. To address this issue, we propose DriftXpert, a novel NIDS for drift-adaptive detection. Specifically, we propose a decoupled two-stage offline adaptive framework. In Phase 1, we introduce an unsupervised anomaly metric based on latent manifold deviation. By performing outlier analysis within the latent space, the framework achieves high-sensitivity detection of network traffic concept drift. In Phase 2, to mitigate catastrophic forgetting under non-stationary distributions, we design a representation consistency alignment strategy. This strategy constrains the feature mapping between the legacy model and the drifted distribution, ensuring the model captures emerging attack characteristics while retaining discriminative power over known patterns. Furthermore, we incorporate cross-epoch neuron weight aggregation and selective freezing mechanisms to enable fine-grained knowledge transfer in the parameter space, effectively balancing model plasticity and stability. Extensive experiments on public datasets demonstrate that DriftXpert effectively adapts to drifted data without catastrophic forgetting. Furthermore, real-world evaluations on enterprise network further confirm its robustness and practical applicability, contributing to improved security protection for millions of users.

1 Introduction

AI-based NIDS often assume static traffic distributions, although real network traffic evolves and can degrade detection. DriftXpert integrates unsupervised drift detection with model adaptation designed to preserve historical knowledge without replaying historical traffic.

1 Introduction

  • AI-based NIDS often assume static traffic, but evolving distributions can degrade detection and increase false positives and false negatives.The paper identifies static-traffic assumptions as a gap between controlled experiments and practical network scenarios.Source: §1 · p. 1
  • Practical adaptation must detect drift before updating, preserve historical detection without historical data, and avoid costly manual labeling.Historical traffic may be impractical to retain or reuse because of resource constraints and privacy risks.Source: §1 · pp. 1–2
  • Existing drift detectors often require labels or historical data and threshold tuning, while model-update alternatives can train unstably or introduce security risks.The paper also notes that some adaptation methods rely heavily on historical data.Source: §1 · p. 2
  • DriftXpert combines label-free clustering indicators with drift-triggered adaptation using representation consistency, weight aggregation, and selective freezing.The framework is designed to balance adaptation and knowledge preservation without historical traffic replay or storage.Source: §1 · p. 2

2 Overview

The overview defines network concept drift and frames NIDS adaptation as a balance between responding to new traffic and retaining historical detection. It also specifies the dynamic-network setting and associated risks.

2.1 Background

  • Concept drift is a change in the joint distribution P(x,y); local drift affects a feature subspace, while global drift spans the feature space.The overview distinguishes global concept drift, involving changed conditional label distributions, from global covariate shift, involving changed feature distributions.Source: §2.1 · p. 2

2.2 Challenges

  • Unsupervised drift detection is presented as more practical than supervised monitoring, which requires costly manual inspection of predictions.Detection can use periodic monitoring rather than real-time analysis.Source: §2.2.1 · p. 3
  • Historical data may be unavailable for updates, and fine-tuning can struggle under global drift and degrade performance.The overview identifies data storage or access as impractical in resource-constrained or privacy-sensitive settings.Source: §2.2.2 · p. 3
  • Updates can improve recent-data detection while reducing historical-data detection, so adaptation must balance both without historical datasets.The paper calls this loss of historical detection capability catastrophic forgetting.Source: §2.2.3 · p. 3

2.3 Threat Model

  • The threat model assumes naturally occurring, non-adversarial drift as incoming traffic departs from the historical training distribution.The model may otherwise produce incorrect predictions, including false positives or missed detections.Source: §2.3 · p. 3
  • The threat model identifies undetected drift and catastrophic forgetting as distinct risks that together can eliminate both adaptability and stability.In-scope threats include evasion through distribution shifts, amplified false positives, and vulnerabilities from forgetting known attack patterns.Source: §2.3 · p. 3, §2.3.1 · p. 3

3 Design

DriftXpert links drift detection to model adaptation: it monitors latent-space shifts and updates the NIDS to learn new patterns while retaining historical knowledge. Its design combines latent-space drift sensing with representation alignment, weight aggregation, and selective freezing.

3.1 Design Motivation

  • The motivation is to infer latent-variable changes from prediction shifts, using their approximately linear relationship to simplify attribution.The design argues that aligning historical and drifted latent distributions can support generalization across temporal domains.Source: §3.1.1 · p. 4, §3.1.2 · p. 4
  • Latent-variable shifts approximately track predicted-probability changes, motivating drift attribution through latent representations despite deep models’ nonlinearity and opacity.The encoder’s latent variables and predicted probabilities are linked through a single fully connected layer.Source: §3.1.1 · p. 4

3.2 System Architecture

  • DriftXpert uses a lightweight encoder-classifier for real-time detection, then runs drift sensing and model adaptation offline when a shift is detected.The updated model replaces the historical one, with the design aiming to preserve robustness to historical data.Source: §3.2 · p. 4

3.3 Phase 1: Proactive Sensing

  • Drift sensing reuses the historical encoder and applies unsupervised outlier analysis to latent variables.K-Means clustering defines latent-space centers; Confidence Interval marks reliable distances versus outliers, and Outlier Ratio compares outlier rates in recent and historical windows.Source: §3.3 · pp. 4–5

3.4 Phase 2: Evolutionary Alignment

  • For adaptation, contrastive learning aligns current latent variables with historical ones while distancing them from the previous epoch’s representations.The alignment objective is paired with cross-entropy classification on drifted data; historical and current-model weights are aggregated, while initial encoder layers are frozen and higher layers remain trainable.Source: §3.4 · pp. 5–6

4 Experimental Setup

The evaluation uses CICIDS2017 and CICIDS-2018, alongside enterprise network traffic collected over consecutive months. It examines local and global drift using distribution-based partitions and compares DriftXpert with five baselines.

4.1.2 Public Datasets

  • The public-dataset evaluation uses CICIDS2017 and CICIDS-2018, extracting 80 flow features and retaining 50 after removing features for stated data-quality or performance reasons.The evaluation also addresses duplicate packets and labeling errors.Source: §4.1.2 · p. 6
  • The evaluation covers local drift in selected data regions and global drift affecting the overall distribution, using distribution-based rather than time-based partitions.Local-drift data are split by label through distribution-based sampling; global-drift data include attack traffic differing from historical data.Source: §4.1.2 · p. 6
  • Historical and current data are divided into training and test sets, and experiments use only training data to keep test data unseen.Source: §4.1.2 · p. 6

4.1.3 Real-world Datasets

  • Enterprise traffic collected over consecutive months includes verified labels, diverse attacks, benign-distribution shifts, and previously unseen attack patterns.Earlier monthly traffic serves as historical data and later traffic as newly observed data; the enterprise serves millions of users.Source: §4.1.3 · p. 6

4.1.4 Baselines

  • Five comparison methods include two fine-tuning variants and OWAD, A-NIDS, and SSF.The variants differ in whether the first encoder layer is frozen; the other baselines represent parameter weighting, historical-data replay, and sample selection, respectively.Source: §4.1.4 · p. 6

4.1.5 Metrics

  • Evaluation metrics are Accuracy, Precision, Recall, F1-Score, and True Positive Rate.Source: §4.1.5 · p. 7

5 Benchmark Evaluation on Public Datasets

The public-dataset evaluation tests DriftXpert’s drift sensing and adaptation, then examines component contributions, latent alignment, and hyperparameter sensitivity. Across the evaluated drift scenarios, it reports a balance between adapting to new traffic and retaining performance on historical traffic.

5.2 RQ1: Proactive Sensing Results by DriftXpert

  • Across four benchmark scenarios, clustering-based outlier analysis identifies concept drift and serves as a high-fidelity trigger for phase-two adaptation.Outlier divergence is more pronounced in isolated clusters; the analysis uses cluster count K and threshold coefficient α.Source: §5.2 · p. 7

5.3 RQ2: Performance of Evolutionary Alignment vs. Baselines

  • Across CICIDS-2017 and CICIDS-2018 local and global drift scenarios, DriftXpert outperforms all baselines while balancing adaptability and stability.The comparisons cover benign and malicious performance in both local and global drift settings.Source: §5.3 · p. 7, Table 3 · p. 8, Table 4 · p. 8
  • DriftXpert achieves the smallest adaptation–retention gap across all four settings, maintaining balanced Macro-F1 on historical and drifted traffic.The study reports Macro-F1 to address class imbalance; conventional fine-tuning adapts poorly, while SSF’s stronger adaptation comes with historical-data degradation.Source: §5.3 · p. 7, Figure 4 · p. 9

5.4 RQ3: Ablation Study

  • Ablations indicate that first-layer freezing supports feature retention and aggregation supports drifted-traffic detection; DriftXpert maintains about 98% precision and recall on drifted data.On CICIDS-2017, it also retains high historical recall while mitigating catastrophic forgetting.Source: §5.4 · p. 9

5.5 RQ4: Alignment Analysis of Latent Space

  • After adaptation, historical and drifted traffic show more consistent latent alignment, although full distribution overlap is not required.Alignment within certain subspaces is described as sufficient to support adaptation.Source: §5.5 · p. 9, Figure 5 · p. 10, Figure 6 · p. 10

5.6 RQ5: Sensitivity Analysis of Hyperparameter

  • The outlier ratio first rises and then falls as α increases; performance remains relatively stable around β = 0.5 and γ = 1.0 across drift scenarios.For α, the authors identify the maximum as an appropriate choice; K is estimated from historical clustering profiles rather than separately sensitivity-tested.Source: Figure 7 · p. 10, §5.6 · p. 10, §5.6.2 · p. 10

6 Real-world Test on Enterprise Network

Enterprise tests evaluate proactive drift sensing, adaptation with forgetting control, and encrypted-traffic detection. DriftXpert reports strong results on drifted data while retaining historical recall and outperforming baselines under encryption.

6.2 RQ1: Proactive Sensing in the Real-world Environment

  • At α≈3.5, the most-drifted cluster had 1,926/10,326 drifted versus 39/4,265 historical outliers, yielding a localized OR of 20.4.The α choice is identified deterministically at the historical-telemetry inflection point where the outlier ratio peaks.Source: Figure 10 · p. 11, §6.2 · p. 11

6.3 RQ2: Evolutionary Alignment in the Real-world Environment vs. Baselines

  • DriftXpert records F1-Scores of 99.43% and 97.10% on drifted data while retaining 96.12% historical recall; OWAD falls to 15.22% F1-Score.The enterprise benchmark compares DriftXpert with OWAD, A-NIDS, and SSF.Source: §6.3 · p. 11, Table 11 · p. 12

6.4 RQ3: Evaluation on Encrypted Environment vs. Baselines

  • Under encryption, DriftXpert achieves 0.88 ACC and 0.94 TPR on baseline distributions, then 0.85 ACC post-drift; A-NIDS falls to 0.71 accuracy after drift.OWAD and SSF do not provide reliable detection in the encrypted post-drift setting.Source: §6.4 · p. 11, Table 12 · p. 12

7 Discussion

The discussion reports DriftXpert's drift-detection and annotation-efficiency results, and contrasts its lightweight role with foundation models. Drift detection can limit manual labeling to detected drift samples.

7.1 Foundation Models and Drift Adaptation

  • DriftXpert's lightweight architecture is presented as suitable for sub-millisecond edge sensing, while LLM overhead and hardware redundancy are described as impractical for many production environments.The authors envision LLMs as strategic commanders and DriftXpert as a front-line network-edge scout.Source: §7.1 · p. 12

7.2 Drift Detection and Annotation Efficiency

  • DriftXpert's Macro-Recall is 84.29% for local drift and 81.77% for global drift, while local-drift Macro-Precision is 50.07%.The text attributes the lower local-drift precision mainly to class imbalance, since only a small fraction of samples shift.Source: Table 13 · p. 12, §7.2 · p. 12
  • Only 1,357 of 42,875 sampled flows in the enterprise deployment's second month exhibited distribution shifts, supporting labeling detected drift samples rather than all traffic.The authors note that drifted samples typically form a small fraction unless major business changes or system migrations occur.Source: §7.2 · pp. 12–13

8 Related Work

The related-work discussion describes limitations in drift detection and adaptation methods for NIDS, then positions DriftXpert as a unified, label-free framework that does not require historical traffic access.

Detection Methods of Concept Drift

  • In NIDS, historical data is not retained, so direct comparison of historical and drifted data is inapplicable; distance-based methods require specific hyperparameters, and choosing them is a key challenge.Source: §8 · p. 13

ML-Based Intrusion Detection Methods

  • Machine-learning algorithms have been extensively applied to intrusion detection; reviewed examples span conditional-probability models, clustering, ensemble trees, autoencoder–SVM systems, point-cloud analysis, temporal graphs, and generative adversarial networks.One unsupervised point-cloud method uses voxel density to reduce false positives.Source: §8 · p. 13

Adaptation Methods for Concept Drift

  • Prior adaptation methods include ensembles and incremental learning, but the review identifies limits such as continuous labeling, noisy pseudo-labels, historical-data reliance, and possible information loss.It also describes parameter-importance approaches and sample selection with forgetting.Source: §8 · p. 13
  • DriftXpert combines clustering-based CI and OR indicators with contrastive representation alignment, weight aggregation, and first-layer freezing, without historical traffic access or labeled drift detection.Detected drift triggers model updates, while theoretical analysis guides indicator hyperparameter selection.Source: §8 · p. 13

9 Conclusion

The conclusion presents DriftXpert as a lightweight NIDS combining drift detection with latent-space alignment, and reports validation across public datasets and resource-constrained deployment tests. The appendices describe artifact availability, ethics, AI-tool use, and a Gaussian prediction-distribution analysis.

9 Conclusion

  • DriftXpert combines deep-clustering drift detection with contrastive latent-space alignment to mitigate catastrophic forgetting during updates.The authors also report theoretical analysis of detection hyperparameters and validation on two public datasets, with latency and throughput tests on a low-end PC.Source: §9 · p. 14

A Open Science

  • The authors provide the framework implementation, hyperparameter configurations, and reproduction scripts in a GitHub repository.The public benchmarks are available through their respective official websites.Source: A Open Science · pp. 15–16

B Ethical Considerations

  • The study used public benchmarks and an industrial-partner dataset with authorization and anonymization, and involved no human or animal subjects.The authors state that no personally identifiable information or sensitive network configurations are included and that IRB approval was not required.Source: B Ethical Considerations · p. 16

C Generative AI Usage

  • Generative AI tools were used only to polish language and readability, not for data analysis, code implementation, or research ideas.The authors state that they reviewed and verified the manuscript content.Source: C Generative AI Usage · p. 16

D Theoretical Analysis of Predict Distribution

  • The appendix derives the predicted distribution p(X) as Gaussian under a maximum-entropy analysis, with μ as mean and σ² as variance.The analysis frames p(X) as the model's predicted distribution.Source: D Theoretical Analysis of Predict… · p. 16

E Theoretical Analysis of Hyperparameter 𝛼for Drift Detection

The analysis relates the drift threshold α to latent-space outlier counts and outlier ratios, then shows that parameter transformation preserves a local minimum under the stated conditions. It also specifies feature selection and drift-scenario settings for the experiments.

E Theoretical Analysis of Hyperparameter 𝛼for Drift Detection

  • As α changes, the confidence interval and outlier counts change, affecting the outlier ratio across two regimes; Figure 11 simulates the relationships among outliers, OR, and α.In the first regime, drifted data are entirely outliers while historical outliers decrease; in the second, historical outliers are absent and drifted outliers decrease.Source: E Theoretical Analysis of… · pp. 16–17, Figure 11 · p. 17
  • Under the Gaussian-cluster assumption, samples beyond threshold V_i (V_i > μ_i) are counted as outliers using cluster sizes and constants A and B.The analysis assumes each cluster distribution is Gaussian.Source: E Theoretical Analysis of… · p. 16

F Theoretical Analysis of Contrastive Loss on Convergence

  • If drifted data converge on the classification model, transforming parameters within the non-Euclidean geometry preserves a local minimum for the loss.The theorem states that η* = g^-1(θ*) is also a local minimum; the proof defines the transformed loss as L composed with g.Source: F Theoretical Analysis of Contrastive… · p. 17

G.1 Features Used in Our Experiments

  • The experiments use 50 selected features from approximately 80 CICFlowMeter features, excluding identifying fields and features with many outliers or no experimental impact.Source and destination IP addresses and ports were removed to avoid “cheating effects.”Source: §G.1 · p. 17

G.2 Settings of Drift Scenario

  • The experimental settings specify two drift scenarios for the CICDS-2017 and CICIDS-2018 datasets.The scenario details are presented in Table 15.Source: §G.2 · p. 17, Table 15 · p. 18

G.3 Details of Model and Training

The appendix reports model and training configuration, examines changing feature importance under drift, and describes how contrastive loss behaves during adaptation.

G.3 Details of Model and Training

  • Table 16 gives model structure and parameter counts, Table 17 lists experimental hyperparameters, and the model has 11,550 total parameters.The tables contain the architecture and training configuration details.Source: §G.3 · p. 18, Table 16 · p. 18, Table 17 · p. 18

H.1 Feature Importance Analysis under Concept Drift

  • SHAP analysis finds that key features differ between historical and drifted models, motivating first-layer encoder freezing while later fully connected layers adapt.The stated rationale is to preserve initial feature influence while allowing subsequent layers to respond to evolving feature importance.Source: §H.1 · pp. 18–19, Figure 12 · p. 19

H.2 Analysis of Contrastive Loss

  • Contrastive loss initially rises as training on drifted data shifts the latent distribution, then is expected to decrease toward—but not below—0.693 as alignment progresses.The authors identify 0.693 as the optimal alignment value when both specified similarities equal 1.0.Source: §H.2 · p. 19

H.3 Sensitivity Analysis of Hyperparameter on the CICIDS-2018 Dataset

Sensitivity analyses report stable performance across tested hyperparameter settings on CICIDS-2018 and enterprise traffic. The appendix also reports memory, latency, and throughput measurements for DriftXpert.

H.3 Sensitivity Analysis of Hyperparameter on the CICIDS-2018 Dataset

  • Across local and global CICIDS-2018 drift, performance remains stable as β and γ vary, with marginal β variation and no significant γ degradation.The authors characterize DriftXpert as insensitive to both parameter choices in these scenarios.Source: §H.3.1 · pp. 19–20, Figure 14 · p. 19, Figure 15 · p. 19

H.4 Sensitivity Analysis of Hyperparameter on the Real-World Dataset

  • On enterprise traffic, performance remains stable across tested β and γ values, with only minor β fluctuations and no noticeable γ-related degradation.The authors describe the results as robust to β variation and not highly dependent on the exact γ choice.Source: §H.4 · p. 20, Figure 16 · p. 20, Figure 17 · p. 20

H.5.1 Memory Usage.

  • DriftXpert’s training memory footprint is approximately 230 KB, including model, optimizer, previous-model, and aggregated-model storage.The model has 11,550 parameters, requiring approximately 46 KB; optimizer and contrastive-learning storage each add an estimated 92 KB.Source: §H.5.1 · p. 20

H.5.2 Latency.

  • Encoder latency averages 0.7 µs and classifier latency averages 0.1 µs per sample, with overall latency no higher than 2.3 µs.The reported ranges are 0.1–2.1 µs for the Encoder and 0.0–0.24 µs for the Classifier.Source: §H.5.2 · p. 20, Figure 18 · p. 20

H.5.3 Throughput.

  • DriftXpert’s average throughput is 1.6 MQPS; the Encoder averages 1.6 MQPS and the Classifier averages 16 MQPS.The reported ranges are 0.4–2.8 MQPS for the Encoder, while the Classifier reaches up to 40 MQPS.Source: §H.5.3 · p. 20, Figure 18 · p. 20

Related papers

Loading 2609.08623v1…