Source-linked AI summary
Anomaly Detection for a Water Treatment System Using Unsupervised Machine Learning
Jun Inoue, Yoriyuki Yamagata, Yuqi Chen, Christopher M. Poskitt, Jun Sun
TL;DR
CPS anomaly detection needs approaches that avoid requiring detailed models of complex physical and computational systems. This paper compares an LSTM-based DNN and a one-class SVM trained on normal SWaT logs and evaluated against 36 attack scenarios. The DNN achieves slightly better overall F measure through higher precision, while the SVM has slightly better recall and more false positives.
Problem
Accurate CPS models are difficult to construct, motivating unsupervised anomaly detection that can learn from available logs without prior system knowledge.
Method
The paper compares an LSTM-based DNN with feedforward layers and a one-class SVM, training on seven days of normal SWaT data and evaluating four days of attack data.
Results
The DNN has slightly better F measure and precision, while the SVM has slightly better recall and reports more false positives.
Takeaways & Limitations
Unsupervised DNN and SVM detectors can be compared on full SWaT attack logs, but their anomaly sensitivity differs by behavior type.
Takeaways & Limitations
Results may not generalize beyond the single SWaT dataset, and both methods struggle with gradual sensor changes and anomalous actuator behavior.
Abstract
from arXiv · showhide
In this paper, we propose and evaluate the application of unsupervised machine learning to anomaly detection for a Cyber-Physical System (CPS). We compare two methods: Deep Neural Networks (DNN) adapted to time series data generated by a CPS, and one-class Support Vector Machines (SVM). These methods are evaluated against data from the Secure Water Treatment (SWaT) testbed, a scaled-down but fully operational raw water purification plant. For both methods, we first train detectors using a log generated by SWaT operating under normal conditions. Then, we evaluate the performance of both methods using a log generated by SWaT operating under 36 different attack scenarios. We find that our DNN generates fewer false positives than our one-class SVM while our SVM detects slightly more anomalies. Overall, our DNN has a slightly better F measure than our SVM. We discuss the characteristics of the DNN and one-class SVM used in this experiment, and compare the advantages and disadvantages of the two methods.
I. INTRODUCTION
The paper frames CPS anomaly detection as important but difficult because accurate system models require extensive knowledge of complex physical and computational processes. It therefore compares two unsupervised, data-driven approaches on SWaT: an LSTM-based DNN and a one-class SVM.
- CPS anomaly detection identifies behavior outside normal operation, including attacks, faults, operator errors, bugs, and misconfigurations.Detection provides a defensive mechanism and supports CPS development, maintenance, and repair.
- Model-based detection is difficult because constructing accurate CPS models requires representing tightly integrated algorithmic control and complex physical processes.
- Unsupervised learning builds CPS models from routinely available historian logs without requiring prior understanding of the target system’s complexities.
- The study compares an LSTM-based DNN with feedforward layers against a one-class SVM for anomaly detection.The DNN is selected because it can learn nonlinear relations, while the SVM searches for unusual time-series shapes.
- The methods are trained on seven days of normal SWaT operation and evaluated on four days containing 36 network attack scenarios.
- The DNN has higher precision and slightly better overall F measure, whereas the SVM has slightly better recall but reports more false positives.Both methods struggle with gradual anomalous changes and anomalous actuator movements; the DNN also has difficulty with out-of-bound values.
II. RELATED WORK
Related work spans model-based, supervised, rule-based, and unsupervised anomaly detection, but existing approaches impose knowledge, dimensionality, scope, or data-type constraints. The paper distinguishes its full-testbed probabilistic evaluation from prior SWaT studies.
- Simulation and model-based CPS anomaly detection requires prior knowledge of system configuration in addition to operation logs.
- Supervised machine-learning detection requires control-element source code and labeled correct and incorrect behaviors generated through injected faults.The cited approach had only received preliminary investigation.
- The paper argues that short Signal Temporal Logic descriptions are unlikely for SWaT because its behavior is dynamic, nonlinear, stochastic, and high-dimensional.
- LOF is unsuitable when normal behaviors are known, whereas the cited aquarium study used it without prior knowledge of normal behavior.
- The paper’s DNN directly models next-status probability distributions, handling mixed discrete and real-valued data without relying on prediction-error thresholds.
- Compared with prior SWaT work covering one subsystem and ten attacks, this study evaluates all six subsystems across 36 attacks using probabilistic density estimation.Its precision and recall are calculated from detected log entries rather than detected attacks.
III. SECURE WATER TREATMENT (SWAT) TESTBED
SWaT is a scaled-down but fully operational six-stage water purification CPS combining physical treatment processes with networked control infrastructure. Its historian dataset records normal operation and attack scenarios across the complete plant.
- SWaT is a scaled-down, fully operational raw-water purification plant capable of producing five gallons of safe drinking water per minute.
- The plant uses six treatment stages, including ultrafiltration, de-chlorination, and reverse osmosis, with water and chemical flows shaping its dynamic behavior.
- SWaT’s cyber infrastructure includes PLCs, a layered communications network, HMIs, a SCADA workstation, and a historian.
- Each stage has a dedicated PLC that reads sensors, computes actuator signals through ladder logic, and communicates over a ring network.Sensors monitor variables such as tank level and pipe flow, while actuators operate equipment such as motorized valves.
- Compromised networks or PLC programs could drive physical states such as overflowing tanks, pumping empty tanks, or unsafe chemical mixing.
- The historian dataset contains network, sensor, and actuator data collected over 11 days: seven normal days and four days with 36 attack scenarios.
IV. ANOMALY DETECTION METHODS
The paper presents a DNN-based probabilistic outlier detector that models normal CPS log data and compares it with a one-class SVM. The DNN decomposes mixed discrete and continuous outputs and uses outlier factors to identify low-probability behavior.
- A. Deep Neural Network: The DNN implements probabilistic outlier detection by judging low-probability data points as outliers.The probability distribution is learned from normal CPS log data.
- A. Deep Neural Network: The DNN computes the outlier factor −log q(l_i | l_i−1) from the probability assigned to each log entry given its history.The direct probability calculation is decomposed because actuator positions have several combinations and sensor values are continuous.
- A. Deep Neural Network: Discrete actuator distributions and continuous sensor distributions are represented by one neural network with multiple inputs and outputs.The actuator distributions q_j are discrete, while the sensor distributions r_k are approximated as Gaussian.
- A. Deep Neural Network: An LSTM processes preceding log entries, after which the network predicts actuator positions and subsequent sensor-value distributions.The architecture combines recurrent history processing with multiple prediction outputs.
- A. Deep Neural Network: The network is trained by minimizing the sum of outlier factors over the normal training data.Training uses cross entropy between real and predicted probability distributions; the theoretical threshold for declaring outliers is selected experimentally.
B. One-Class SVM
The one-class SVM detects unusual shapes in CPS time series by classifying sliding windows as normal or abnormal. Its evaluation uses window-level labels, while larger windows reduce verdict resolution and normalization strongly affects performance.
- B. One-Class SVM: The one-class SVM uses an RBF kernel to learn a non-linear boundary around normal data.The method is applied directly to anomaly detection as a comparison with the DNN.
- B. One-Class SVM: Sliding windows convert the log time series into feature vectors, with each window classified as normal or abnormal.A log containing k entries yields k − w + 1 windows of size w.
- B. One-Class SVM: A window is labeled abnormal if at least one of its constituent log entries is abnormal.The classifier verdict is compared with this derived window label during evaluation.
- B. One-Class SVM: Increasing the window size can shift the plotted first anomaly by up to w − 1 entries and degrade verdict resolution.The authors report that this resolution effect was too fine to affect their conclusions in the cases tried.
- B. One-Class SVM: SVM F measures are around 20–30% with training-set normalization but can reach almost 80% when testing data use their own mean and variance.The preliminary grid search found normalization to have a large effect on SVM performance.
V. IMPLEMENTATION AND EXPERIMENTAL SETUP
The DNN and SVM use established machine-learning software and substantially different computing setups. DNN training can require extended GPU-based computation, while SVM runtime varies widely with parameter settings.
- V. IMPLEMENTATION AND EXPERIMENTAL SETUP: The DNN is implemented with Chainer, while the SVM uses scikit-learn with libsvm as its backend.These implementations support the paper’s two evaluated anomaly-detection methods.
- V. IMPLEMENTATION AND EXPERIMENTAL SETUP: Training a 100-dimensional-hidden-layer DNN takes about two weeks over 58 epochs on a CPU-and-GPU machine.Larger DNNs use cluster machines equipped with multiple GPUs.
- V. IMPLEMENTATION AND EXPERIMENTAL SETUP: One-class SVM training ranges from 11 seconds to 26 hours, and evaluation takes up to 7 hours depending on parameters.The best-performing parameter combination trains in 30 minutes and evaluates in 10 minutes.
VI. HYPER-PARAMETER TUNING
Both anomaly-detection methods require hyper-parameter tuning before learning. The supplied passage introduces this tuning stage but does not specify its procedures or selected values.
- VI. HYPER-PARAMETER TUNING: Both methods require tuning hyper-parameters whose values are set before learning.The paper states that the tuning procedure is explained afterward.
A. Deep Neural Network
The DNN study tunes intermediate-layer dimensions and an outlier-factor threshold, then examines training error and F measure across epochs. Although held-out normal data suggests early stopping near 13 epochs, that model performs substantially worse on attack data than a better-trained model.
- A. Deep Neural Network: The DNN tuning varies intermediate-layer dimensions and the outlier-factor threshold.The authors also compare training behavior across intermediate layers of 100–500 dimensions.
- A. Deep Neural Network: Training error steadily decreases as the number of training epochs increases, regardless of intermediate-layer dimension.
- A. Deep Neural Network: The first epoch nearly achieves the best F measure, while epochs 10–30 show the worst performance; AUC follows a similar tendency.F measure is maximized using a different threshold for each epoch and DNN.
- A. Deep Neural Network: 13-epoch early stopping suggested by held-out normal data underperforms better-trained models by a wide margin on attack data.The evaluation therefore uses a 58-epoch DNN with 100-dimensional intermediate layers.
B. One-Class SVM
The one-class SVM uses sliding-window features and is tuned over window size, ν, and γ. Random search around the best grid-search region improves the F measure to 0.79628 for the selected configuration.
- B. One-Class SVM: One-class SVM parameters are window size w, trade-off weight ν, and kernel coefficient γ.With one feature vector per window, the setup uses n = 52w dimensions.
- B. One-Class SVM: 0.79628 F measure is achieved after testing 4204 random instances around w = 4 and γ = ν = 10^-3.The selected instance uses w = 4, γ = 0.0008181483058667633, and ν = 0.004584962079820046.
VII. EVALUATION
On the SWaT attack log, DNN and one-class SVM achieve comparable anomaly-detection rates, but the SVM produces more false alarms. The evaluation also identifies differences in which attack patterns each detector recognizes and notes two threats to validity.
- DNN achieves better precision, SVM achieves slightly better recall, and DNN has the slightly better overall F measure.
- Both detectors catch anomalies at comparable rates, but SVM is more prone to false alarms.
- 0.8% of non-attack windows receive false-alarm verdicts from the SVM.
- The SVM’s false alarms are associated with abrupt sensor-value changes because its moving windows exclude longer-term trends.
- DNN usually detects attacks producing constant anomalous sensor values, but misses attacks involving anomalous actuator movements or gradual sensor changes.
- The evaluation is limited to SWaT and deliberately injected anomalies, so results may not generalize to other CPSs or real attacks.
VIII. CONCLUSION AND FUTURE WORK
The study compares a time-series DNN and one-class SVM for anomaly detection on SWaT, finding slightly better F measure for the DNN. Both methods remain limited in detecting gradual sensor changes and anomalous actuator behavior, while generalizability is constrained by the single attack-focused dataset.
- Conclusion: The DNN achieved a slightly better F measure than the one-class SVM on SWaT attack data.The DNN had higher precision, while the SVM had slightly better recall.
- Limitations: Training took about two weeks for the DNN versus about 30 minutes for the best-performing SVM.Detection on four days of attack data took 8 hours for the DNN and about 10 minutes for the SVM.
- Limitations: Both methods had difficulty detecting gradual sensor-value changes and anomalous actuator behavior.Addressing actuator anomalies may require incorporating controller logic.
- Threats to Validity: The results may not generalize to other CPSs because the study used a single SWaT dataset.The dataset contained anomalies from deliberately injected attacks, which may differ from other anomaly types.
- Future Work: Future work targets improved performance, long-term trend features, higher recall, sensor- and actuator-level outlier factors, and broader testing.The authors also plan to improve the neural architecture and apply feature engineering.