PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Domain Generalization Failure in Lightweight Intrusion Detection Models for IIoT Networks

MD Azizul Hakim, Md Shihab Uddin, Talha Ibne Anis

arXiv:2607.00553v1cs.CRcs.AI

TL;DR

Lightweight IIoT intrusion detection models are rarely tested beyond their training network, leaving their unseen-network behavior uncertain. This paper evaluates four architectures across three datasets without retraining and finds substantial, consistent cross-network performance collapse, driven by reliance on coarse port categories.

  • Problem

    Whether lightweight IIoT intrusion detection models retain within-network performance on unseen networks remains insufficiently evaluated.

  • Method

    Four lightweight architectures are trained on one IIoT dataset and evaluated without retraining on two structurally distinct datasets using shared features and explainability analysis.

  • Results

    F1 falls from approximately 0.97 in-domain to 0.18–0.28 on Gotham and 0.09–0.13 on WUSTL-IIoT-2021 under natural class distributions.

  • Takeaways & Limitations

    Within-domain performance does not establish cross-network performance, and adaptation effectiveness varies substantially across lightweight architectures.

  • Takeaways & Limitations

    Findings are bounded by a shared feature schema that excludes TCP flags from WUSTL-IIoT-2021 and by limited protocol diversity in the source training traffic.

Abstract

from arXiv · show

Lightweight machine learning models are increasingly proposed for intrusion detection in Industrial Internet of Things (IIoT) networks due to their suitability for resource-constrained edge deployment. Most reported results evaluate these models only within their training network, leaving behavior on unseen networks unverified. This study trains four lightweight architectures on one IIoT dataset and evaluates them, without retraining, on two structurally distinct IIoT datasets using a feature representation restricted to attributes available across all three sources. Explainability analysis across two top-performing models shows both rely overwhelmingly on coarse port-category features; the most influential category occurs in source-domain attack traffic at 96 to 435 times the rate in the two target domains, indicating that coarsening port resolution relocates rather than removes a documented shortcut. Evaluation under naturally imbalanced class distributions reveals a further effect: the evaluation protocol used can reverse which target network appears to pose the greater generalization challenge. Adversarial robustness and recovery through limited target-domain exposure are also assessed; robustness to adversarial perturbation is unrelated to cross-network generalization, and recovery through adaptation varies considerably by architecture. These findings suggest deployment readiness should be assessed using cross-network evaluation under realistic class distributions, rather than within-domain accuracy alone.

1 Introduction

Lightweight IIoT intrusion detectors often look highly accurate because they are tested within the network used for training. This study asks whether that performance transfers across networks and whether failures can be diagnosed or cheaply adapted.

  • The evaluation gap: Same-network evaluation rarely tests whether lightweight intrusion detectors work on unseen industrial networks.Different devices, traffic patterns, and attacker behavior can undermine deployment performance.
  • Study design: The study trains four lightweight architectures on one IIoT dataset and tests them without retraining on two independent datasets.It combines explainability, feature ablation, class-distribution comparisons, adversarial testing, and few-shot adaptation.
  • Contributions: The contribution includes evidence that strong within-domain performance does not imply cross-network performance.It also examines port-category shortcuts, realistic class imbalance, adversarial robustness, and architecture-dependent fine-tuning recovery.
  • Practical implication: Deployment readiness should be judged with cross-network evaluation under realistic class distributions, not within-domain accuracy alone.The paper treats cross-network testing as a standard requirement rather than an optional extension.

2 Related Work

Prior intrusion-detection research has explored lightweight models, cross-dataset failure, shortcut features, robustness, and adaptation separately. This study combines these dimensions in a focused IIoT evaluation.

  • Existing IIoT evidence: Earlier IIoT benchmark studies generally reported high performance only within each dataset’s own train/test split.The originating publications did not report testing models trained on one dataset against another.
  • Cross-dataset generalization: Cross-dataset studies in broader intrusion detection consistently report major performance losses when models encounter unseen networks.Reported failures span supervised classifiers, selected attack categories, and unsupervised models.
  • Shortcut features: Prior work links inflated detection results to origin-identifying fields such as destination port rather than malicious behavior.This motivates directly testing whether port reliance contributes to transfer failure.
  • Open mechanism: IIoT cross-dataset work has also reported macro-F1 losses of up to thirty percentage points while excluding port identifiers entirely.That design leaves the contribution of port reliance to transfer loss unresolved.
  • Study gap: No prior study combines lightweight IIoT architectures, multiple target datasets, port-shortcut testing, class-distribution comparison, and matched few-shot adaptation.The present work is designed to close this combined gap.

3 Datasets and Feature Engineering

The study uses one training dataset and two structurally different IIoT targets, then restricts models to a minimal feature schema shared across all sources. This makes cross-network transfer testable while exposing the limits of coarse features.

  • Datasets: Edge-IIoTset is the training domain, while Gotham 2025 and WUSTL-IIoT-2021 serve as independent cross-domain targets.The datasets differ in collection tools, network settings, and traffic formats.
  • Feature engineering: The common schema uses source and destination ports, transport protocol, and four TCP flag bits.Raw port numbers are excluded from the representation.
  • Label harmonization: Labels are harmonized to binary attack-versus-benign classes across the three datasets.This discards native multiclass distinctions for cross-domain experiments.
  • Dataset summary: Table 2 summarizes each dataset’s source or target role, capped size, and class balance.These fields frame the distribution shift examined in the experiments.
  • Class distributions: Each target dataset is evaluated with both balanced sampling and its natural, heavily benign-skewed distribution.This tests whether the common balanced convention reflects deployment conditions.

4 Methodology

The methodology compares four compact classifiers under controlled in-domain and cross-domain conditions, then measures deployment cost, feature reliance, adversarial degradation, statistical stability, and few-shot recovery.

  • Model architectures: The four models are DecisionTree, SmallMLP, Small1DCNN, and SmallLSTM, all trained on the 16-dimensional common schema.The neural architectures are compact designs intended for lightweight deployment.
  • Evaluation protocol: Models are trained on Edge-IIoTset and evaluated without parameter updates on Gotham and WUSTL-IIoT-2021 under balanced and natural distributions.A single StandardScaler is fit on the Edge-IIoTset training split and reused everywhere.
  • Metrics: Attack-class F1 is used because it avoids rewarding majority-class predictions under natural imbalance.The protocol reports F1 rather than relying on accuracy alone.
  • Deployment cost: Deployment measurements include serialized size, inference latency percentiles, peak memory, parameter or node count, and training time.Latency is measured over 1,000 repeated single-sample calls.
  • Explainability: SHAP values and decision-tree feature importances quantify which features drive attack-class predictions.SHAP rankings use a fixed sample of 200 in-domain test instances.
  • Adversarial robustness: HopSkipJump evaluates adversarial robustness by perturbing 100 in-domain test instances and measuring the resulting accuracy drop.The same black-box attack is applied to all four models.
  • Statistical validation: Five random seeds and pairwise McNemar tests assess variation and prediction-level significance.McNemar comparisons use identical test instances and report significance at p < 0.05.
  • Few-shot adaptation: Few-shot adaptation uses fractions of labeled Gotham data from 0 to 0.25 of a fixed adaptation pool.The tree is refit, while neural models receive five additional fine-tuning epochs.

5 Results

All four lightweight models fit the source network similarly, yet their performance collapses on unseen IIoT networks. The results also expose port-bucket shortcut reliance, protocol-dependent rankings, and architecture-dependent adaptation.

  • Cross-domain performance: F1 falls from approximately 0.97 in-domain to 0.18–0.28 on Gotham and 0.09–0.13 on WUSTL-IIoT-2021 under natural class distributions.No model retains more than 29% of its in-domain F1 outside the training network.
  • In-domain baseline: F1 ranges from 0.971 to 0.972 in-domain, leaving only a 0.001 gap between the strongest and weakest model.Precision exceeds 0.99 and recall exceeds 0.95 for all four architectures.
  • Evaluation protocol: Natural-distribution evaluation can reverse the apparent difficulty of the target networks, because balanced evaluation ranks WUSTL-IIoT-2021 easier than Gotham for every model.Natural evaluation instead produces the larger decline on WUSTL-IIoT-2021.
  • Architecture ranking: SmallLSTM reaches 0.275 natural F1 on Gotham but 0.093 on WUSTL-IIoT-2021, while DecisionTree shows the reverse pattern at 0.180 and 0.134.No architecture dominates across both target datasets.
  • Feature reliance: DecisionTree’s three strongest Gini features account for 93% of total split importance, while port-bucket indicators dominate and protocol features contribute little.The top features are src_port_wellknown (0.366), dst_port_wellknown (0.298), and dst_port_none (0.270).
  • Feature reliance: dst_port_none occurs in 40.5% of source-domain attack traffic but only 0.42% of Gotham and 0.09% of WUSTL-IIoT-2021 attack traffic.This corresponds to 96-fold and 435-fold differences, respectively, showing that coarse bucketing relocates rather than removes shortcut reliance.
  • Adversarial robustness: Adversarial robustness and cross-network generalization are distinct properties: SmallMLP and SmallLSTM generalize relatively better but are less robust to perturbation.The adversarial evaluation used a single 100-sample run without seed repetition or comparable confidence intervals.
  • Few-shot recovery: DecisionTree’s Gotham F1 rises from 0.170 to 0.638 at 25% adaptation data, whereas Small1DCNN does not improve at any tested fraction.SmallLSTM peaks at 0.585 with 5% adaptation, and SmallMLP reaches 0.289 at 25%.

6 Discussion

The discussion shows that cross-network deployment exposes shortcut reliance, protocol-sensitive evaluation effects, and architecture-specific adaptation behavior. These results favor realistic cross-domain testing over any single within-domain or efficiency metric.

  • Shortcut reliance: Coarsening port features removes fine-grained port memorization but leaves models dependent on broad port-bucket categories.DecisionTree assigns 93% of split importance to three categories, while SmallMLP independently shows the same category-level dependence.
  • Shortcut reliance: 96 and 435 times: the most influential port bucket appears at those source-domain attack-traffic rates relative to Gotham and WUSTL-IIoT-2021.The disparity links category-level reliance to the observed cross-domain collapse.
  • Evaluation protocol: Balanced evaluation can reverse which target dataset appears harder, making comparative deployment claims protocol-dependent.Artificial class balance substantially overstates operational performance and can change the relative difficulty ranking of WUSTL-IIoT-2021 and Gotham.
  • Adaptation: Few-shot recovery varies by architecture rather than following a single data-volume rule.DecisionTree recovers substantially after a larger sample, SmallLSTM recovers with less data but later declines, SmallMLP improves modestly, and Small1DCNN does not improve.
  • Deployment trade-offs: Cross-domain generalization, adversarial robustness, and computational efficiency do not reliably track one another.DecisionTree is cheap and adapts well, whereas SmallLSTM is costlier and transfers comparably or better while remaining among the least adversarially robust.
  • Deployment trade-offs: No architecture is best on every axis, so model selection must match whether deployment prioritizes zero-shot transfer, adaptation, cost, or robustness.The recommendations apply specifically to the two tested target networks and should be revisited for structurally different deployment targets.

7 Conclusion

Cross-network testing shows that strong within-domain results do not establish deployment readiness for lightweight IIoT intrusion detectors. The study instead finds persistent shortcut dependence, protocol-sensitive difficulty, and architecture-specific adaptation.

  • Cross-network generalization: Detection performance collapses substantially and consistently when four lightweight models are transferred without retraining to two independent target datasets.The result remains stable across random seeds and statistically significant.
  • Shortcut dependence: Feature-resolution ablation does not eliminate the shortcut: reducing granularity relocates dependence from raw port memorization to coarser feature resolutions.Reducing port resolution alone therefore does not resolve the underlying generalization problem.
  • Evaluation protocol: Natural imbalanced evaluation can reverse which target network appears more difficult, while balanced evaluation overstates performance.The evaluation protocol therefore changes conclusions about deployment difficulty, not merely reported scores.
  • Adaptation: Small-scale target-domain fine-tuning restores performance for some architectures but not others.Adaptability is consequently a property of model design rather than lightweight models as a class.
Loading 2607.00553v1…