PaperlayerView exact revision on arXiv

Source-linked AI summary

Robustness of Anomaly Detection Models for Industrial Control Systems under Training-Time Data Contamination

Mustafa Umut Ozbek, Taiwo Ojo, Pooria Madani, Khalil El-Khatib, Li Yang

arXiv:2608.23547v1cs.CRcs.LG

TL;DR

Offline ICS anomaly detectors are evaluated because prior work largely assumes trustworthy training data and leaves comparative robustness under contamination insufficiently characterized. The paper benchmarks 11 heterogeneous detectors on SWaT using three contamination strategies and common offline evaluation rules. Robustness is strongly model-dependent: injection is most damaging, while PCA, SVM, HBOS, and IForest remain comparatively stable despite strong clean performance not guaranteeing robustness.

  • Problem

    Comparative robustness of heterogeneous ICS anomaly detectors under unified offline training-time contamination remains insufficiently characterized despite assumed trustworthy training data.

  • Method

    The study evaluates 11 heterogeneous detectors on SWaT under random injection, similarity-targeted injection, and feature-noise contamination across four budgets.

  • Results

    Injection causes the greatest degradation, while PCA, SVM, HBOS, and IForest remain comparatively stable and AE and LSTM-AE show intermediate robustness.

  • Takeaways & Limitations

    Clean-data performance alone is not a reliable indicator of contamination robustness under the evaluated offline protocols.

  • Takeaways & Limitations

    The benchmark uses a single ICS dataset, contamination-style rather than worst-case attacks, clean validation and test splits, and uneven hyperparameter optimization.

Abstract

from arXiv · show

Machine-learning-based anomaly detection is increasingly used in industrial control systems (ICS), yet most studies assume that detector training data is trustworthy. In practice, training data may be corrupted through compromised logs, labeling errors, manipulated historian records, or unsafe retraining processes. This paper evaluates the robustness of offline ICS anomaly-detection pipelines on the Secure Water Treatment (SWaT) benchmark under training-time contamination. We assess 11 heterogeneous anomaly detectors under three contamination strategies: random injection, similarity-targeted injection, and feature-noise injection. The first two insert attack samples into the nominal training pool, while the third adds bounded Gaussian noise to selected normal training samples. These attacks are contamination-based rather than gradient-driven poisoning methods. Contamination budgets from 1% to 10% are evaluated using clean validation and test sets under a unified offline protocol. The results show that robustness is strongly model-dependent and cannot be predicted from clean-data performance alone. Injection-based contamination causes the greatest degradation, particularly for local-density and distance-based detectors, whereas feature-noise contamination has a comparatively limited effect. PCA, SVM, HBOS, and IForest remain relatively stable, while the tuned neural detectors demonstrate intermediate robustness. Overall, the findings highlight the importance of training-data integrity in ML-enabled ICS monitoring, subject to the evaluated dataset, models, and threat assumptions.

I. INTRODUCTION

The introduction identifies a gap in understanding how offline ICS anomaly detectors withstand training-time contamination. It presents a unified SWaT benchmark spanning heterogeneous detectors, contamination strategies, budgets, and evaluation criteria.

  • Motivation: ICS anomaly detectors often rely on normal-only training data whose integrity may be compromised before deployment.Contamination can distort the learned representation, support region, density structure, geometry, or reconstruction pattern of normal behavior.
  • Research gap: Prior adversarial ICS research has focused mainly on inference-time evasion or poisoning of online-adaptive neural detectors.
  • Benchmark: Eleven normal-only detectors are evaluated on SWaT under random injection, similarity-targeted injection, and feature-noise injection.Contamination budgets are 1%, 3%, 5%, and 10%.
  • Evaluation protocol: The benchmark uses clean validation for threshold calibration and a clean held-out test set for final evaluation.
  • Evaluation scope: The study jointly examines F1-score degradation, false-negative rate, training cost, and configuration-level differences rather than clean F1-score alone.

A. ICS Anomaly Detection

ICS anomaly detection is motivated by structured, repetitive process behavior and limited attack labels, with SWaT serving as a reproducible benchmark. Prior work nevertheless highlights dataset dependence, evaluation inconsistencies, and limited coverage of offline training-time contamination.

  • ICS anomaly detection: Anomaly-based monitoring fits ICS because normal process behavior is structured and usually easier to collect than diverse attack traces.
  • Benchmarks: SWaT is widely used because it provides a realistic and reproducible platform for evaluating process-aware attack-detection methods.
  • Evaluation dependence: No detector family is universally dominant, and reported rankings can shift with metrics, hyperparameter treatment, and evaluation protocols.
  • Adversarial settings: Most adversarial ICS studies emphasize inference-time evasion rather than offline training-time corruption.
  • Broader context: Prior work outside ICS shows that training-time corruption can distort learned normality in anomaly detectors and degrade related classifiers.

D. Research Gap

The paper targets the limited characterization of heterogeneous ICS detector robustness under common offline training-time contamination protocols. It defines a gray-box, contamination-style evaluation that isolates training-data effects while acknowledging important scope boundaries.

  • D. Research Gap: Existing literature lacks consistent comparisons of heterogeneous anomaly detectors under shared contamination definitions, budgets, calibration, and held-out evaluation.
  • D. Research Gap: The benchmark asks whether strong clean-data detectors retain their advantage after the normal training pool is contaminated.
  • Method: Each detector models nominal behavior from clean normal data, with labeled validation for selection and calibration and held-out testing after contamination.
  • Threat model: The gray-box adversary has training-time access, knows the representation and general pipeline, and either injects attack samples or perturbs normal observations.
  • Limitations: Clean validation and test splits isolate training-time contamination effects but may be optimistic when data-governance problems propagate across splits.
  • Threat model: The evaluated mechanisms are repeatable contamination-style stress tests rather than gradient-driven worst-case poisoning attacks.

B. Contamination Strategies

The benchmark applies a common clean-pool-relative budget across injection and feature-noise strategies, while implementation choices create controlled but potentially optimistic comparisons.

  • Budget definition: For budget p, the adversary manipulates k = floor(pN) samples from a clean pool of size N.
  • Budget definition: Injection appends k samples, producing a final fraction k/(N+k), whereas feature noise modifies k of the original N samples.At p = 10%, the injected fraction is approximately 9.09%.
  • Implementation: The contamination resource is disjoint from validation and test observations for pointwise detectors and AE, while LSTM-AE uses it to contaminate a separate contiguous normal block.
  • Caveat: Clean validation and test assumptions prevent threshold corruption from being conflated with model corruption but may underestimate failures across contaminated data splits.

1) Random Injection:

Random injection models unsafe inclusion of attack traces by adding attack samples to the nominal training pool. It provides a simple, model-agnostic baseline for testing whether learned normality expands around malicious states.

  • Random injection appends attack samples to the clean training pool, modeling unsafe curation or accidental inclusion of attack traces treated as normal.
  • Its strength comes from implicitly relabeling attack behavior as normal during fitting rather than exploiting detector gradients or parameters.
  • The benchmark interprets random injection as a reproducible contamination baseline, not a gradient-driven or worst-case attack.
  • The offline workflow confines contamination to the normal training pool while keeping validation and test data uncontaminated.
  • The strategy tests whether a detector’s learned support expands around malicious process states.

3) Feature-Noise Injection:

Feature-noise injection corrupts selected normal training observations with bounded, scaled Gaussian noise while preserving the training-set size. Its reported near-null effect is limited to the tested noise magnitude and contamination configuration.

  • Feature-noise injection replaces selected normal samples with clipped, feature-scaled Gaussian perturbations after MinMax scaling.
  • The method preserves training-set size while modeling corruption of stored process values.
  • Clipping maintains the normalized feature range, while per-feature scaling prevents low-variance variables from being dominated by a common perturbation magnitude.
  • The Gaussian magnitude is fixed at 0.15, with only the fraction of modified samples varied across experiments.
  • A near-null result establishes limited sensitivity to this tested configuration, not general robustness to feature corruption.
  • The SWaT preprocessing yields a pointwise pool of 449,919 samples and 44 features, with MinMax parameters fitted on training data only.

B. Detectors, Thresholding, and Reproducibility

The benchmark compares heterogeneous classical and neural detectors using explicit configurations, separate clean-validation thresholding, and reproducible multi-seed execution. Its ordering is configuration-level evidence rather than a definitive comparison of optimally tuned model families.

  • Detectors: The benchmark spans IForest, One-Class SVM, LOF, CBLOF, KNN, HBOS, PCA, MCD, ABOD, AE, and LSTM-AE.
  • Detectors: Classical detectors use heterogeneous scoring mechanisms and training inputs, enabling contamination sensitivity comparisons across incompatible definitions of normality.
  • Tuning: Only PCA and SVM receive promoted classical tuning; other classical detectors retain default or near-default configurations.
  • Tuning: AE and LSTM-AE undergo staged searches over architecture, optimization, regularization, sequence, and training choices.
  • Reproducibility: Experiments use fixed seeds 42, 123, and 456 across splitting, contamination, initialization, and training.
  • Scope: SOD is excluded because its evaluated configuration had inverted clean ROC-AUC ordering and severe baseline precision collapse, leaving 429 retained runs.
  • Thresholding: Each model’s threshold is calibrated on the same clean labeled validation split and then frozen for held-out test evaluation.

A. Clean Performance and Contamination Robustness

Clean-data performance does not reliably predict contamination robustness: several high-performing detectors degrade sharply under injection, while PCA, SVM, HBOS, and IForest remain comparatively stable. Feature noise has limited effect at the tested magnitude.

  • Clean performance: LSTM-AE has the highest clean F1-score at 0.8813, but clean rankings change substantially under contamination.LOF, ABOD, and AE also rank among the strongest clean configurations.
  • Contamination robustness: LOF, KNN, and ABOD each lose more than 0.64 F1 in their worst condition, whereas IForest, HBOS, PCA, and SVM remain within 0.068 of clean baselines.AE and LSTM-AE show intermediate robustness.
  • Evaluation stability: Most detectors vary by less than 0.01 F1 across three seeds, but LOF and MCD vary more because their fitting pools are subsampled.Degradation is computed from seed-averaged contaminated and clean scores rather than a single favorable run.
  • Injection effects: At 1% random injection, LOF, KNN, ABOD, and LSTM-AE fall from 0.876, 0.848, 0.874, and 0.881 to approximately 0.398, 0.335, 0.305, and 0.440, respectively.Feature noise at σ = 0.15 changes little, while similarity-targeted injection becomes severe for selected detectors at 10%.
  • Neural detectors: AE deteriorates gradually to approximately 0.566 F1 at 10% random injection, while LSTM-AE drops to approximately 0.440 at 1% random injection.LSTM-AE responds much more mildly to similarity-targeted injection through 5%.
  • Model dependence: The heatmap shows IForest, HBOS, PCA, and SVM near clean values, while injection sharply harms LOF, KNN, ABOD, and LSTM-AE.Similarity-targeted injection produces delayed failures for LOF, KNN, and ABOD.

B. Attack-Type Comparison

The contamination strategies produce distinct degradation patterns. Random injection causes the earliest and largest overall drops, similarity-targeted injection produces detector-dependent delayed failures, and bounded feature noise remains close to baseline under the tested configuration.

  • Random injection: Random injection is the dominant attack family overall, causing the earliest large drops across pointwise neighborhood methods and the sequence model.Injected samples span the available attack pool, introducing diverse malicious states into nominal reference data.
  • Evaluation basis: The benchmark compares clean performance, mean training time, and worst-case F1 drop across twelve attack-budget combinations using seed-averaged F1 values.The table’s clean performance is reported as mean ± SD over three seeds.
  • Similarity-targeted injection: LOF, KNN, and ABOD show delayed failures under similarity-targeted injection as the contamination budget increases.Global proximity to the average clean state does not guarantee local proximity in every detector representation.
  • Feature-noise injection: Feature-noise injection remains close to the clean baseline across the evaluated grid.The tested configuration uses σ = 0.15, clips perturbations to the normalized range, and modifies only selected normal observations.
  • Missed detections: ABOD, KNN, and LOF exceed approximately 0.70 FNR in at least one contamination setting, while LSTM-AE reaches a lower but concerning peak under random injection.PCA, SVM, HBOS, AE, and MCD remain in the lower-FNR group, and IForest stays comparatively stable despite its lower clean baseline.
  • Metric interpretation: FNR is most informative alongside F1-score because lower missed detections can reflect overly permissive decisions that reduce precision.The paper therefore does not treat FNR as a standalone safety measure.

D. Overall Trade-offs Across Performance, Robustness, and Cost

Clean-data performance, contamination robustness, and training cost produce different detector orderings. Under the evaluated SWaT benchmark, PCA and SVM offer a favorable observed balance, while several high-performing clean-data models are fragile under contamination.

  • LSTM-AE, LOF, ABOD, and AE form the strongest clean-data group, but several remain fragile after normal-training-pool contamination.LOF, KNN, and ABOD degrade particularly severely; AE and LSTM-AE show lesser but meaningful degradation.
  • PCA and SVM combine moderate clean performance with small worst-case F1 loss and substantially lower training cost.Neither leads the clean baseline table, but both provide a favorable observed balance under the evaluated setup.
  • IForest remains stable across the contamination grid, although its clean baseline is lower than PCA and SVM.
  • PCA and SVM provide the most defensible observed balance among clean performance, robustness, safety-relevant FNR behavior, and computational cost.This is configuration-specific rather than a formal multi-objective ranking because hardware and training scales differ.
  • The benchmark’s conclusions are bounded by one dataset, contamination-style rather than optimization-based attacks, clean validation and test splits, unequal tuning depth, and differing split structures.The limited feature-noise effect applies only at σ = 0.15, and the authors recommend interpreting comparisons as configuration-level evidence.
Loading 2608.23547v1…