PaperlayerView exact revision on arXiv

Source-linked AI summary

Unmasking Shortcut Learning in IoT Intrusion Detection: A Forensic, Multi-Paradigm Evaluation of Feature Dependence and Data Leakage

Uday Shankar Roy, Mahbuba Jahan Minu

arXiv:2609.28725v1cs.CRcs.LGcs.NI

TL;DR

The paper examines whether IoT intrusion detectors learn generalizable attack behavior or exploit dataset shortcuts, a concern for interpreting benchmark performance. It audits CyberFlowIoT-GICAP using multiple model paradigms and feature configurations with PCAP-disjoint evaluation, then finds performance and detectability depend strongly on feature representation and attack type, while random-flow splitting inflates recall.

  • Problem

    Whether IoT NIDS benchmark performance reflects generalizable attack behavior or shortcuts tied to identities, timestamps, and session structure remains an open evaluation question.

    Source: §I · p. 1, §II · p. 2

  • Method

    The study combines forensic auditing and feature ablations across four learning paradigms on CyberFlowIoT-GICAP, using PCAP-disjoint evaluation and a random-flow baseline.

    Source: §I · p. 2, C. Data Splitting Protocols · p. 4

  • Results

    Behavioral-feature Macro-F1 converges across three nonlinear paradigms, while timestamped tree models outperform the linear model and random-flow splitting inflates attack recall by up to +14.00%.

    Source: A. RQ1: Feature Shortcuts & Temporal… · p. 4, C. RQ3: Evaluation Bounds &… · p. 5

  • Takeaways & Limitations

    The study derives a four-point protocol checklist for more realistic IoT NIDS evaluation.

    Source: §I · p. 2

    Spatial generalization remains untested beyond the predominantly single-IP non-MITM attacks, and rare covert categories were too sparse for stateful sequence-model evaluation.

    Source: D. Practical Implications and… · p. 6

Abstract

from arXiv · show

Machine learning-based Network Intrusion Detection Systems often report near-perfect performance on IoT benchmarks. However, whether these models learn generalizable attack behavior or exploit spurious dataset shortcuts- such as static testbed IP/MAC addresses and chronological recording artifacts-remains an important question. We evaluate the CyberFlowIoT-GICAP benchmark, containing 3,617,388 flow records across 126 PCAP sessions with 849,395 benign flows. Four learning paradigms are evaluated across four feature configurations using PCAP-disjoint splits; LightGBM is additionally evaluated using conventional random-flow splitting. When only statistical flow behavior is used (Fbehav), LightGBM (92.58% +/- 8.18%), Random Forest (92.59% +/- 8.18%), and Deep MLP (92.55% +/- 8.18%) achieve nearly identical Macro-F1, indicating that performance is constrained by feature representation rather than model complexity. With raw timestamps (Ftstamp), tree-based models reach 99.28% Macro-F1, while the linear model remains at 90.62%, showing that nonlinear models can exploit dataset-specific temporal structure. Attack detectability is highly asymmetric: high-rate and active attacks maintain >99.8% recall from flow behavior alone in nonlinear models, whereas the DNS Beaconing drops from 27.78% to 0.00% recall when contextual features are removed. Conventional random-flow splitting increases attack recall by up to 14.00%, highlighting the effect of placing flows from the same sessions in both training and test sets. We conclude with a 4-point protocol checklist for realistic IoT NIDS evaluation.

I. INTRODUCTION

Near-perfect benchmark performance may rely on shortcuts rather than generalizable attack behavior, especially when testbed identities, recording times, or session overlap carry predictive signals. The paper frames questions around feature dependence, attack-specific detectability, and leakage under random-flow versus PCAP-disjoint evaluation.

I. INTRODUCTION

  • Benchmark performance may not translate to operational environments when models exploit predictive correlations unrelated to attack behavior.Such shortcuts may generalize poorly when network environments, hosts, or recording conditions change.Source: §I · p. 1
  • Static IP/MAC identities, isolated recording intervals, and shared capture sessions can each create shortcuts in flow datasets.Random flow splitting can place flows from the same capture session in both training and test partitions.Source: §I · p. 1
  • The research questions examine feature configurations and model paradigms, attack-specific detectability, and random-flow versus PCAP-disjoint evaluation.The evaluation asks whether performance reflects model complexity or feature representation and how intra-session sharing relates to split differences.Source: §I · pp. 1–2
  • Across 3.61M flows and 126 PCAP sessions, four learning paradigms show behavioral-feature parity, timestamp-sensitive tree performance, asymmetric attack detection, and up to +14.00% recall inflation under random splitting.The study derives a four-point protocol checklist for realistic IoT NIDS evaluation.Source: §I · p. 2

II. RELATED WORK

Prior work documents shortcut learning, benchmark artifacts, and methodological pitfalls, while IoT reviews have raised concerns about evaluation bias. The paper positions its contribution as a controlled study combining forensic auditing, feature ablation, and session-level holdout on a modern IoT benchmark.

II. RELATED WORK

  • Meta-analyses identify sampling bias, data snooping, and spurious correlations as drivers of non-replicable security-ML research.Studies of tabular intrusion datasets also report decision trees overfitting to ephemeral metadata rather than intrinsic flow dynamics.Source: §II · p. 2
  • Forensic audits have documented benchmark construction and labeling artifacts, including dependence on attacker IP addresses and recording intervals.The paper reports similar IP- and timestamp-dependent artifacts in a modern IoT benchmark.Source: §II · p. 2
  • An IoT/IIoT survey of 91 studies offered a qualitative taxonomy but lacked empirical ablations, multi-paradigm comparisons, and forensic ground-truth auditing.The paper identifies a gap for controlled multi-paradigm study combining forensic auditing, systematic feature ablation, and session-level PCAP holdout.Source: §II · p. 2

III. DATASET AUDIT

The audited CyberFlowIoT-GICAP benchmark contains millions of flows across 126 PCAP sessions, with dedicated benign traffic and heterogeneous, imbalanced attack categories. Table I is identified as a forensic inventory of those captures.

III. DATASET AUDIT

  • 3,617,388 bidirectional flow records comprise 849,395 dedicated benign flows and 2,767,993 flows from 126 PCAP capture sessions.The benchmark contains seven attack categories with severe class imbalance and heterogeneous capture compositions.Source: A. Benchmark Composition · p. 2
  • Table I presents a complete forensic inventory across 126 PCAP captures.The caption supplies the inventory's scope, not specific cell values.Source: Table I · p. 2
  • Capture counts distinguish mixed PCAPs containing benign and attack flows from pure, fully benign captures.Parentheses denote the mixed/pure PCAP counts.Source: A. Benchmark Composition · p. 2

B. IP Address Distribution

The audit identifies a strong concentration of non-MITM attack traffic at one source IP, while attack categories were recorded in separate time windows. The methodology workflow proceeds from forensic auditing through feature construction and splitting to model evaluation.

B. IP Address Distribution

  • 99.50% of non-MITM attack flows originated from one machine, creating an opportunity to exploit source identity rather than intrinsic behavior.MITM traffic instead involved multiple endpoints, including the hub and poisoned victim devices.Source: B. IP Address Distribution · p. 2

C. Timestamp and Recording Artifacts

  • Attack experiments occupied separate, non-overlapping calendar windows, making raw timestamps a shortcut for separating traffic by time.Dedicated benign traffic was recorded first, followed on subsequent days by the attack categories.Source: C. Timestamp and Recording Artifacts · p. 3

IV. METHODOLOGY

  • The workflow spans dataset acquisition and forensic auditing, feature construction, dual splitting protocols, and multi-paradigm evaluation.Fig. 1 summarizes this overall experimental sequence.Source: §IV · p. 3, Figure 1 · p. 3

A. Feature Taxonomy

The study excludes administrative and ground-truth annotations, groups the remaining telemetry into feature families, and defines four feature configurations. It compares four model families to examine how architectures respond to shortcut features.

A. Feature Taxonomy

  • The 81 retained telemetry features comprise six identifiers, two ports, 12 application/protocol fields, 55 behavioral fields, and six timestamps.Seventeen administrative metadata and ground-truth boundary annotations are excluded.Source: A. Feature Taxonomy · p. 3
  • The four configurations are the 75-feature identity-inclusive baseline (Fall), 69-feature identity-scrubbed set (F−ident), 55-feature behavioral set (Fbehav), and 61-feature temporal set (Ftstamp).Ftstamp combines behavioral dynamics with timestamps.Source: A. Feature Taxonomy · p. 3

B. Multi-Paradigm Models

  • The four model families are LightGBM, Random Forest, Deep MLP, and Linear SGD, representing boosted trees, bagged trees, neural networks, and a linear baseline.The study uses these families to test shortcut exploitation and responses to timestamp artifacts.Source: B. Multi-Paradigm Models · p. 3, Table III · p. 3

C. Data Splitting Protocols

The evaluation contrasts PCAP-disjoint and random-flow splits to assess generalization and intra-session leakage. Macro-F1 and per-attack recall are reported across three seeds to address class imbalance and detection blind spots.

C. Data Splitting Protocols

  • PCAP-disjoint splitting assigns captures to train or test without overlap, while random-flow splitting allows flows from the same capture in both partitions.The primary protocol uses 80/20 PCAP partitions across three seeds; the conventional LightGBM baseline uses an 80/20 stratified flow split.Source: C. Data Splitting Protocols · p. 4

D. Evaluation Metrics

  • Macro-F1 is the primary metric because 79.49% of traffic is benign and MQTT and DNS Beaconing each represent under 0.15% of flows.Per-attack recall is also measured across all seven attack categories to identify detection blind spots.Source: D. Evaluation Metrics · p. 4
  • All experiments report mean ± standard deviation across three random seeds.The PCAP-disjoint protocol governs 48 primary trials, while the LightGBM random-flow audit comprises 12 trials.Source: D. Evaluation Metrics · p. 4, C. Data Splitting Protocols · p. 4

E. Implementation Details

The study reports 60 trials across four architectures and four feature sets, with PCAP-disjoint evaluation as its primary protocol. Results show that feature representation and model-specific use of timestamps shape performance.

E. Implementation Details

  • 60 experimental trials cover four architectures and four feature configurations, with 48 PCAP-disjoint trials and 12 LightGBM random-flow trials.All trials use three fixed seeds, and reported metrics are mean ± standard deviation across runs.Source: E. Implementation Details · p. 4

A. RQ1: Feature Shortcuts & Temporal Inductive Bias

  • 92.58% vs. 88.36% Macro-F1 for LightGBM with Fbehav versus Fall; LightGBM, Random Forest, and Deep MLP converge near 92.6% on Fbehav.Removing host identifiers alone changes LightGBM from 88.36% to 88.07%; Fbehav also removes transport ports and application-layer metadata.Source: Table IV · p. 4, A. RQ1: Feature Shortcuts & Temporal… · p. 4
  • 99.28% and 97.37% Macro-F1: LightGBM and Random Forest, respectively, with timestamps added to behavioral features.Tree splits can gate on calendar windows, excluding dedicated benign traffic outside attack intervals before separating interleaved flows by behavior.Source: A. RQ1: Feature Shortcuts & Temporal… · p. 4
  • 90.62% ± 7.00% Macro-F1: Linear SGD performs identically with Fbehav and Ftstamp.Its global hyperplane cannot partition scattered, noncontiguous campaign intervals, and timestamp weights yield no measurable improvement.Source: A. RQ1: Feature Shortcuts & Temporal… · p. 4
  • 84.07% ± 13.03% Macro-F1 for Deep MLP with Ftstamp, down from 92.55% ± 8.18% with Fbehav.DoS recall falls from 99.89% ± 0.04% under Fbehav to 72.52% ± 47.57% under Ftstamp.Source: A. RQ1: Feature Shortcuts & Temporal… · p. 5

B. RQ2: Asymmetric Attack Vulnerability & DNS Collapse

Attack detectability on pure flow behavior varies sharply by category: active and high-rate attacks retain strong recall, while DNS Beaconing collapses without identity features. Timestamp effects on MITM detection also differ substantially across model architectures.

Behavior-only attack detectability

  • DoS and Nmap Recon each exceed 99.8% recall, and SQL Injection exceeds 99.9%, across non-linear models using pure behavior (Fbehav).Even the convex linear classifier maintains over 91% recall on these categories.Source: Table V · p. 5, B. RQ2: Asymmetric Attack… · p. 5

MITM and timestamp sensitivity

  • ARP Spoofing MITM recall is approximately 35.6% across all four architectures on Fbehav, but timestamp augmentation raises LightGBM recall to 99.50%.With timestamps, Deep MLP reaches 65.72%, Random Forest 47.59%, and Linear SGD remains at 35.48%.Source: Table VI · p. 5, B. RQ2: Asymmetric Attack… · p. 5

DNS Beaconing collapse

  • DNS Beaconing recall falls from 27.78% ± 4.81% under the 75-feature baseline (Fall) to 0.00% ± 0.00% on Fbehav across all four paradigms.The 1,611 beacon flows are diluted among 1,070,736 benign background flows; the paper notes that isolated flow statistics cannot distinguish them from legitimate DNS lookups without static IP addresses.Source: B. RQ2: Asymmetric Attack… · p. 5

C. RQ3: Evaluation Bounds & Intra-Session Data Leakage

On LightGBM, conventional random-flow splitting raises attack recall relative to PCAP-disjoint evaluation, but also increases false alarms on pure behavioral features. Timestamp-based shortcuts make the random-split recall difference negligible for the timestamp configuration.

Split-protocol comparison

  • Random-flow splitting inflates attack recall by up to +14.00% on F−ident and +12.89% on Fall compared with PCAP-disjoint evaluation.The paper attributes this to flows from the same capture session appearing in both training and test sets; 108 of 126 attack PCAPs contain interleaved benign traffic.Source: Table VII · p. 5, C. RQ3: Evaluation Bounds &… · p. 5

Trade-offs and timestamp condition

  • On Fbehav, attack precision drops from 86.78% under PCAP holdout to 73.44% under random splitting, while overall Macro-F1 falls by 3.93%.Under Ftstamp, the leakage delta is negligible (∆Rec = −0.45%) because calendar timestamps provide a global shortcut independent of session boundaries.Source: C. RQ3: Evaluation Bounds &… · p. 5

D. Practical Implications and Evaluation Guidelines

The paper proposes a four-part evaluation protocol to reduce shortcut learning and non-replicable benchmark claims. It also identifies boundaries on the study’s generalizability and leaves stateful detection of covert threats for future work.

Future work

  • The study did not implement or evaluate stateful, multi-flow temporal aggregation, which it identifies as a likely direction for addressing covert-attack detection.Examples proposed include session-level query-entropy or host-interaction graph modeling.Source: D. Practical Implications and… · pp. 5–6

Study limitations

  • Spatial generalization across multi-subnet deployments remains untested because 99.50% of non-MITM attacks originated from a single testbed IP.Covert categories comprised only 0.12%–0.15% of their capture files, too few instances for stateful sequence-model training or evaluation in this study.Source: D. Practical Implications and… · p. 6

Evaluation guidelines

  • The proposed protocol calls for capture-level holdout, removal of host identifiers and timestamps, multi-seed and multi-paradigm validation, and per-attack recall reporting.The checklist aims to prevent intra-session leakage, reduce reliance on testbed artifacts, expose inductive-bias vulnerabilities, and detect asymmetric attack failures.Source: D. Practical Implications and… · p. 6

VI. CONCLUSION

The conclusion finds that behavioral-feature performance converges across three nonlinear and tree-based models, while timestamps benefit tree ensembles but not the linear classifier. It also reports complete DNS Beaconing recall collapse after identity shortcuts are removed and higher attack recall under random-flow splitting.

Conclusion

  • On pure flow behavior, LightGBM, Random Forest, and Deep MLP achieve 92.55%–92.59% Macro-F1, within 0.04 percentage points of one another.The paper concludes that detection capacity is bounded primarily by feature representation rather than model complexity.Source: §VI · p. 6
  • With raw timestamps, tree ensembles reach up to 99.28% Macro-F1, while the linear classifier remains at 90.62%, unchanged from pure behavior.Source: §VI · p. 6
  • DNS Beaconing recall is 0.00% across all four paradigms after identity shortcuts are removed, while conventional random-flow splitting inflates attack recall by up to +14.00%.Source: §VI · p. 6
  • The findings motivate a four-point evaluation protocol and a focus on session-level analysis for low-rate covert threats that remain difficult to identify from individual flows.Source: §VI · p. 6

Related papers

Loading 2609.28725v1…