PaperlayerView exact revision on arXiv

Source-linked AI summary

Provenance, Not Behaviour: A Serialisation Artifact in Edge-IIoTset and a Leakage-Free Benchmark for Precision-Agriculture Intrusion Detection

Mostafa M. Galal

arXiv:2608.15761v1cs.CRcs.LG

TL;DR

Edge-IIoTset’s extraordinary intrusion-detection results lack evidence that they reflect network behaviour rather than dataset provenance. We audit the serialization pipeline and rebuild the benchmark, finding exact label recovery under the distributed recipe and a deployment-relevant generalisation boundary on the corrected data.

  • Problem

    Whether Edge-IIoTset’s extraordinary reported performance reflects intrusion detection rather than a serialization artifact remains insufficiently established.

  • Method

    We audit token purity and file provenance, then rebuild the benchmark from raw captures using uniform parsing and evaluate held-out-device generalisation.

  • Results

    The distributed recipe yields exact label recovery, while corrected evaluation exposes cross-layer generalisation failure, with random forest falling to 0.5083 balanced accuracy when the Modbus gateway is withheld.

  • Takeaways & Limitations

    Reliable precision-agriculture intrusion detection requires provenance-safe preprocessing, held-out-domain evaluation, and deployment-cost reporting rather than random-split accuracy alone.

  • Takeaways & Limitations

    The inferred perception/actuation boundary rests on exactly one actuation-layer device and requires confirmation with several distinct fieldbus devices.

Abstract

from arXiv · show

Edge-IIoTset is the reference benchmark for machine-learning intrusion detection in the industrial Internet of Things, and results reported on it cluster above 99%. We show that much of that performance is not intrusion detection. The preprocessing recipe distributed with the dataset instructs researchers to one-hot encode seven categorical columns. Four of them separate attack from normal traffic with an accuracy of 1.0000 on their own, through the spelling of the placeholder written for an absent protocol field: the string "0" in the normal-traffic branch of the dataset build against "0.0" in the attack branch. The label is recoverable from a serialisation artifact encoding file provenance, with no network behaviour modelled, and separates every row of both curated subsets. Under 5-fold x 3-repeat cross-validation, five of six standard classifiers attain exactly 1.0000 +/- 0.0000 accuracy and the sixth attains 0.99998. Under a corrected protocol, naive Bayes falls by 0.3005 macro-F1 and the strongest model settles at 0.9503 +/- 0.0011. Label, ordinal and frequency encoding leak identically. Because the curated subsets also lack Modbus and per-device identity, we rebuild the benchmark from the raw captures under uniform parsing, producing AgriEdge: 1,276,122 rows, five devices with full attribution, and no column separating the classes above 0.0288. A leave-one-device-out sweep locates the generalisation boundary at the perception/actuation layer, where random forest falls from 0.9988 to 0.5083 balanced accuracy. Non-IID federated partitioning costs at most 0.0037 macro-F1, but a 20-round LoRaWAN training run costs 4.6 hours of uplink.

1. Introduction

This paper argues that extraordinary Edge-IIoTset intrusion-detection results are substantially driven by a serialisation artifact rather than learned network behaviour. It introduces leakage diagnosis and corrected evaluation as the basis for a more credible precision-agriculture benchmark.

  • Motivation: Precision agriculture links IIoT telemetry to irrigation actuation, making compromised readings capable of causing immediate physical consequences.The setting includes soil-moisture, pH, water-level and climate sensors, edge gateways, MQTT, and Modbus TCP.
  • Motivation: 99.27% accuracy with 99.21% F1 and 99.94% accuracy on Edge-IIoTset have been interpreted as evidence that intrusion detection is nearly solved.Edge-IIoTset is a physical seven-layer testbed spanning fourteen attack classes and supporting centralised and federated learning.
  • Leakage mechanism: The standard preprocessing recipe creates a near-perfect label proxy because separately parsed normal and attack branches serialize an absent protocol field as 0 versus 0.0.One-hot encoding seven columns treats these placeholders as distinct tokens, revealing file provenance rather than network behaviour.
  • Leakage mechanism: 1.0000 accuracy is achieved in isolation by four of the seven affected columns, so models using them are reading a build artifact instead of detecting intrusions.The leakage mechanism is quantified through token purity, single-column held-out accuracy, and normalised mutual information.
  • Study scope: Under 5-fold × 3-repeat cross-validation, five of six classifiers reach exactly 1.0000 ± 0.0000 accuracy and the sixth reaches 0.99998; corrected evaluation spans 0.6995 to 0.9503 macro-F1.The paper also tests device-transfer generalisation and federated-learning costs after correcting the benchmark protocol.

2. Background

Precision-agriculture deployments span perception, network, and edge-system layers with distinct security and resource constraints. Edge-IIoTset provides raw captures and curated subsets widely used under a preprocessing recipe now targeted for audit, amid growing concerns about benchmark leakage and generalisation.

  • Deployment layers: Precision-agriculture deployments decompose into perception, network, and edge-system layers whose security properties differ sharply.These layers respectively observe the field, carry telemetry, and perform local processing.
  • Deployment layers: Perception-layer sensors emit low-rate, periodic, semantically narrow measurements, making anomaly detection tractable but enabling altered readings to control downstream actuation.Examples include soil moisture, pH, water level, temperature, and humidity sensors.
  • Deployment layers: MQTT dominates sensor reporting while Modbus TCP supports actuation, and typical farm deployments provide neither protocol authentication nor encryption by default.A man-in-the-middle can rewrite moisture readings, while broker denial-of-service prevents telemetry delivery.
  • Deployment layers: Severely constrained cellular or LoRaWAN uplinks require agricultural detection to run on gateways, binding detection quality to resource budgets.This trade-off is concentrated in the edge system layer.
  • Dataset and literature context: Edge-IIoTset combines raw per-device normal and attack captures with 157,800-row and 2,219,201-row curated subsets containing 63 columns.The curated files target traditional machine learning and deep learning, and include protocol features, Attack_label, and Attack_type.
  • Dataset and literature context: Most published work uses the curated subsets and follows a Readme.txt recipe that drops identifiers, payloads, nulls, and duplicates before dummy-encoding seven categorical columns.This preprocessing recipe is the object of the paper’s audit.

3. Threat Model

The threat model gives an attacker network-level control while treating sensors’ physical measurements and the edge gateway as trusted. It maps dataset attack classes to agricultural consequences and preserves severe natural rarity in MITM and OS fingerprinting.

  • Adversary capabilities and trust assumptions: The adversary accesses the farm IIoT segment through a compromised gateway, exposed cellular modem, or unsecured wireless link, and can observe, inject, or modify traffic.Sensors’ physical measurements remain uncompromised; detection runs on a trusted edge gateway, while compromised-gateway detection is out of scope.
  • Agricultural consequences of attack classes: Man-in-the-middle attacks alter telemetry, denial-of-service attacks sever irrigation triggers, and injection or application attacks target management interfaces.Reconnaissance precedes targeted attacks, while backdoor and ransomware establish persistence; ransomware against irrigation controllers compresses decisions under crop water stress.
  • Class representation: 1,229 rows represent MITM and 1,001 represent OS fingerprinting, making both classes severely under-represented despite MITM’s importance to falsified telemetry.The study preserves this natural rarity rather than resampling it away and reports per-class recall accordingly.

4. Methodology

The methodology audits whether categorical columns alone encode the label, then compares distributed and corrected preprocessing protocols. It rebuilds the benchmark from uniformly parsed, device-attributed raw captures and evaluates models under random and device-held-out splits, including federated and edge-cost measurements.

  • Column audit: The audit measures each categorical column using token purity, single-column held-out accuracy, and normalised mutual information.A separation rate of 1.0 means pure tokens relabel the target; held-out accuracy uses a decision tree on one-hot features with a stratified 80/20 split.
  • Column audit: A mechanism-specific probe tests whether separate 0 and 0.0 spellings occur under disjoint labels, revealing provenance encoding for absent values.All columns are read as strings because type inference can normalise 0 and 0.0 and erase the artifact.
  • Preprocessing protocols: Protocol A reproduces the distributed preprocessing, while Protocol B canonicalises absence placeholders, deduplicates before splitting, and uses structural rather than identity encoding.Protocol A drops listed columns, null rows, and duplicates, dummy-encodes seven columns, and coerces the remainder to numeric.
  • AgriEdge construction: The rebuilt dataset enforces identical parsing and placeholder canonicalisation across captures while preserving device attribution for non-IID farm partitioning.Raw captures share a byte-identical 63-column header, and each row retains its originating device or attack capture.
  • Evaluation and deployment: Evaluation combines random stratified 80/20 splitting with leave-one-device-out testing, using six seeded estimators and federated FedAvg client constructions.LODO withholds all normal traffic from one sensor type and tests on it with 20% of attack traffic; edge cost includes model size, parameters, latency, and backhaul communication.

5. Results · 5.1. Four columns recover the label perfectly · 5.2. What the artifact is worth

The distributed Edge-IIoTset preprocessing recipe exposes a serialisation artifact that perfectly encodes labels across both curated subsets, inflating classical and deep-model results. Correcting the artifact sharply reduces performance, while the leak persists across common categorical encodings.

  • 5.1. Four columns recover the label perfectly: Four columns separate every class because absent-protocol placeholders appear as ’0’ in normal rows and ’0.0’ in attack rows.The distinction records the parsing branch, not network behaviour; three HTTP columns show the reversed artifact and separate 19–40% of data.
  • 5.1. Four columns recover the label perfectly: 2,219,201 rows are separated independently by each of four columns, exposing the same artifact in both ML and DNN branches despite inverted class balance.The larger DNN subset contains 1,615,643 normal and 603,558 attack rows.
  • 5.2. What the artifact is worth: 1.0000 ± 0.0000 is achieved by five of six models across all fifteen stratified folds, while the MLP averages 0.99998.Evaluation uses 5-fold × 3-repeat cross-validation.
  • 5.2. What the artifact is worth: 0.3005 macro-F1 is lost by Gaussian naive Bayes under correction, revealing that the recipe encoded the label directly.Its corrected interval widens to ±0.0089 versus ±0.0009 for the trees.
  • 5.2. What the artifact is worth: 0.9503±0.0011 macro-F1 is the corrected best-model result, roughly four points below the literature’s above-99% range.The corrected score is described as the honest ceiling for the ML subset.
  • 5.2.1. Deep models are equally deceived: 1.0000 accuracy is reached by both deep baselines under the distributed recipe, but each loses more than 0.21 macro-F1 under correction.The tested models are a 61,378-parameter deep MLP and a 6,658-parameter 1D-CNN.
  • 5.2.2. The effect holds at scale on the deep-learning subset: 1.9 million rows reproduce the artifact: every model reaches 1.0000 accuracy under the distributed recipe, while corrected naive Bayes falls to 0.5190 macro-F1.After cleaning, 1,909,671 rows remain; the corrected DNN subset’s best model reaches 0.9612 macro-F1.
  • 5.2.3. The leak is encoding-agnostic: 1.0000 is the minimum single-column accuracy across four leaking columns and four encodings, showing that the artifact is encoding-agnostic.Any categorical encoding preserving the distinction between ’0’ and ’0.0’ transmits the leak, including label encoding.

5.3. The curated subsets cannot support agricultural research · 5.4. AgriEdge: construction and validation · 5.5. Detection fails across layers, not across devices

The curated subsets omit Modbus and device attribution, preventing agricultural filtering, while raw-capture reconstruction yields AgriEdge without provenance leakage. AgriEdge shows generalisation fails at the perception/actuation boundary, not across similarly profiled devices.

  • 5.3. The curated subsets cannot support agricultural research: The curated subsets cannot support agricultural research because Modbus is absent and device identity is stripped, preventing observation of irrigation actuation or agricultural-sensor filtering.Across 157,800 ML rows, mbtcp.len, mbtcp.trans_id and mbtcp.unit_id are uniformly zero; only Temperature_and_Humidity MQTT topics survive.
  • 5.3. The curated subsets cannot support agricultural research: Raw captures retain millions of attributed sensor rows and 159,502 Modbus rows, enabling reconstruction of the agricultural benchmark.Retained counts include 1,192,777 soil-moisture, 2,295,288 water-level, 1,615,722 temperature-humidity, and 746,908 pH rows.
  • 5.4. AgriEdge: construction and validation: 1,276,122 rows and 49 features define AgriEdge, including 751,559 normal and 524,563 attack rows at attack rate 0.4111.Attack classes preserve natural rarity, ranging from 60,069 DDoS-HTTP to 1,229 MITM and 1,001 fingerprinting.
  • 5.4. AgriEdge: construction and validation: 0.0288 maximum separation rate confirms that no AgriEdge column separates classes after uniform parsing eliminates provenance markers.Maximum single-column accuracy falls to 0.6179 and maximum NMI to 0.0643, from 1.0000 and 0.9859 respectively.
  • 5.5. Detection fails across layers, not across devices: 0.9880–0.9999 balanced accuracy persists when withholding soil moisture, water level or pH, while temperature and humidity declines to 0.9106–0.9385.Tree-based models remain at 0.9995 or better on the mutually substitutable perception sensors.
  • 5.5. Detection fails across layers, not across devices: 0.5083 random-forest balanced accuracy on Modbus marks collapse across the perception/actuation boundary, alongside 0.4877 for logistic regression.Modbus is the sole actuation-layer device and speaks a different protocol; the trivial baseline is 0.5.
  • 5.5. Detection fails across layers, not across devices: 0.8164 makes the MLP best on the hard Modbus case, despite random forest reaching 0.9997 on perception holdouts; model rankings invert under layer shift.Gaussian naive Bayes is worst on perception holdouts at 0.6022–0.6248 but second best on Modbus at 0.8068.
  • 5.5. Detection fails across layers, not across devices: 0.9988 random-split balanced accuracy is optimistic, exceeding worst-case leave-one-device-out performance by 0.18 to 0.49 and serving as an upper bound.The optimism reflects packets from the same capture session and attack burst appearing on both sides of the split, not audited column leakage.

5.6. Federated learning: heterogeneity is cheap, the uplink is not · 5.7. Edge deployment cost

Federated heterogeneity reduces macro-F1 by only 0.0037 relative to IID, whereas a 20-round LoRaWAN run costs 4.6 hours of uplink. At the gateway, decision trees match random forests at 0.9988 balanced accuracy under random splitting while requiring far less latency and model size.

  • 5.6. Federated learning: heterogeneity is cheap, the uplink is not: 0.0037 macro-F1 is the cost of the maximally heterogeneous per-sensor client construction relative to IID.Per-farm partitioning is indistinguishable from IID.
  • 5.6. Federated learning: heterogeneity is cheap, the uplink is not: 0.9932 macro-F1 is reached only at round 19 by the per-device split, while the per-farm split passes 0.9950 by round 8 and plateaus at round 12.The conclusion is measured under random splitting of the test set.
  • 5.6. Federated learning: heterogeneity is cheap, the uplink is not: 4.6 hours of continuous transmission are required for a 20-round LoRaWAN training run, before duty-cycle regulations are accounted for.A single round costs nearly fourteen minutes of pure uplink, and five clients upload dense parameter vectors.
  • 5.6. Federated learning: heterogeneity is cheap, the uplink is not: 0.0037 macro-F1 is tolerable to a farm gateway, but 4.6 hours of uplink is not, making compression, sparsification, or fewer rounds more valuable for deployability than heterogeneity-robust aggregation.The design problem is framed as an uplink constraint rather than a statistical one.
  • 5.7. Edge deployment cost: 0.9988 balanced accuracy is achieved by both random forest and decision tree under random splitting, despite severe differences in deployment footprint.The comparison is based on the five models considered plausible on a gateway.
  • 5.7. Edge deployment cost: 342× slower and 171× larger is random forest than decision tree, making random forest unsuitable for inline operation.Random forest has 13.7 ms median inference and 73 samples/second throughput, below the aggregate packet rate of a modest farm deployment.
  • 5.7. Edge deployment cost: 0.9986 macro-F1 is random forest’s apparent score under random splitting, described as the least informative number in the study.The practical conclusion is based on reading this result with Table 10.
  • 5.7. Edge deployment cost: 1/171 the size and 1/342 the latency make the decision tree strictly preferable to random forest on both deployment axes.This conclusion follows the reported gateway deployment comparison.

6. Discussion

The distributed preprocessing recipe propagated a serialisation artifact that survives multiple encoding choices and can produce apparently perfect published results, so affected scores cannot be interpreted as detection performance. Leakage-free evaluation must test cross-layer generalisation, prefer worst-case performance, and treat the mechanism as a broader security-dataset risk.

  • Recipe propagation: The affected recipe is reproduced verbatim in public Edge-IIoTset code, including its exact drop list and seven dummy-encoded columns.The helper encode_text_dummy appears in implementations, including a CNN-LSTM system, indicating researchers copied the documented recipe.
  • Recipe propagation: Label, ordinal and frequency encoding preserve the leak unchanged, so avoiding get_dummies does not remove the artifact.A report that categorical variables used label encoding therefore does not establish that the leakage was avoided.
  • Published signatures: 4,820 true positives, 25,620 true negatives, zero false positives and zero false negatives accompany Hasan et al.’s 99.94% accuracy.The paper lacks their exact column list but notes that their encoding preserves the artifact and that correlation pruning could retain one sufficient leaking column.
  • Cross-layer evaluation: A detector validated on sensors does not transfer to actuators, where a third of the model suite performs at chance.The discussion recommends holding out a device from a different architectural layer because average LODO performance is dominated by easier perception cases.
  • Cross-layer evaluation: 0.3098 macro-F1 versus 0.6066 shows the random forest degrades more across the layer boundary than the decision tree.The decision tree is 342× faster and 171× smaller; the MLP has the best mean (0.9455) and worst case (0.8164) and is recommended as default.
  • Broader implications: The placeholder-and-encoding mechanism can arise whenever separately parsed, label-correlated sources are concatenated, making similar artifacts plausible across security datasets.The authors release the audit as a general tool and note that benign and malicious traffic are often captured in separate sessions.

7. Threats to Validity

The study addresses construct and internal validity through a provenance-specific leakage test and ablations, while its generalisation claims remain limited to within-testbed device and architectural-layer shifts. Cross-network transfer is deliberately out of scope, and several reported experiments lack directly quantified variance.

  • Construct validity: Four flagged columns pass a stricter provenance test because their absence-marker spelling carries no network semantics.The separation-rate measure alone could flag legitimately predictive pure features; the provenance probe tests whether placeholder spelling carries the signal.
  • Internal validity: 0.52% duplicate rate and strict-threshold ablation bound the effects of deduplication and structural high-cardinality encoding, leaving canonicalisation dominant.Protocols A and B differ in several preprocessing choices, and a full factorial ablation was not run.
  • External validity: LODO covers all five devices, but the perception/actuation boundary is inferred from exactly one actuation-layer device.A testbed with several distinct fieldbus devices would be needed to confirm the boundary.
  • Cross-dataset generalisation: Cross-network validation is not established because Edge-IIoTset exposes protocol-field names while comparable datasets expose flow-statistical features, yielding a near-empty direct intersection.The paper presents LODO as within-testbed distribution-shift evidence, not a substitute for cross-network validation.
  • Statistical reporting: Single seeded runs cover the deep baselines, LODO sweep, federated runs, and edge-cost measurements, whose variance is not quantified directly.The headline protocol comparison uses 5-fold × 3-repeat stratified cross-validation with 95% intervals, but fold intervals are not strictly independent within repeats.

8. Reproducibility

The authors release the complete AgriEdge implementation and experiment scripts under the MIT licence, with seeded experiments runnable on modest hardware. They document the audited dataset copy’s provenance and provide a dependency-free verification path for readers’ own pipelines.

  • Implementation release: All code is released under the MIT licence as the agriedge Python package, covering leakage audits, canonicalisation, recipes, benchmark construction, federated learning, evaluation, and models.Nine experiment scripts each write their own result tables.
  • Implementation release: All randomness is seeded, and every experiment ran on a 12-core Apple M4 Pro with 25 GB of memory.Deep baselines used PyTorch’s Metal backend and trained in under a minute each; no experiment required datacentre hardware.
  • Dataset provenance: The audited copy is identified through its IEEE DataPort publication, Kaggle retrieval path, author account, and dated dataset files.The copy’s Readme.txt, documentation PDF, and CSV files carry 18 March 2022 timestamps, while the DataPort submission dates to 18 January 2022.
  • Verification: Readers can verify the placeholder split with a dependency-free script that reads every field as text, preventing type inference from masking the difference.The package also exposes audit_columns and summarize for a single-call audit.

9. Conclusion

Edge-IIoTset’s documented preprocessing exposes a serialisation artifact that enables perfect or near-perfect classification without modelling network behaviour. Corrected evaluation and AgriEdge reveal substantially weaker cross-device generalisation, while federated deployment is constrained more by LoRaWAN bandwidth than non-IID partitioning.

  • Conclusion: Four prescribed categorical columns encode a placeholder-spelling difference that separates every row of both curated subsets by label.The artifact reflects file provenance rather than network behaviour.
  • Conclusion: Published results cannot individually be deemed artifacts, but the prescribed and alternative encodings produce perfect or near-perfect classification across tested models.The paper distinguishes this documented preprocessing failure from claims about particular published numbers.
  • Conclusion: 0.0288 is the highest class-separation accuracy for any column in the rebuilt benchmark after uniform parsing restores Modbus traffic and per-device attribution.The rebuild eliminates the serialisation artifact by construction.
  • Conclusion: 0.9988 to 0.5083 balanced accuracy is random forest’s drop when the Modbus gateway is withheld across the perception/actuation layer.Logistic regression falls to 0.4877, near the trivial baseline, and model rankings invert under this shift.
  • Conclusion: 0.0037 macro-F1 is the maximum cost of non-IID partitioning, whereas LoRaWAN training requires 4.6 hours of continuous uplink.The transmission figure applies to training a 28,450-parameter model.
  • Conclusion: Future work should test cross-network transfer, determine whether the generalisation boundary spans actuation protocols, and audit other concatenated intrusion-detection datasets.The current boundary finding rests on a single fieldbus device, and the authors suspect the failure mode is widespread.

Data availability

Edge-IIoTset is publicly available for academic use, while the AgriEdge package, construction pipeline, experiment scripts, and result tables are released openly. The constructed 38 MB benchmark is published separately rather than included in the repository.

  • Edge-IIoTset is publicly available and free for academic use under its own licence.
  • The agriedge package, AgriEdge construction pipeline, experiment scripts, and result tables are released under the MIT licence on GitHub and Zenodo.Repository: https://github.com/MostafaGalal1/agriedge; persistent identifier: https://doi.org/10.5281/zenodo.21941210.
  • 38 MB: the constructed benchmark is not included in the repository and is published separately.
Loading 2608.15761v1…