Source-linked AI summary
SCADA System Testbed for Cybersecurity Research Using Machine Learning Approach
Marcio Andrey Teixeira, Tara Salman, Maede Zolanvari, Raj Jain, Nader Meskin, Mohammed Samaka
TL;DR
SCADA systems connected through open IP networks remain vulnerable, while public datasets may not represent evolving attacks. This paper builds a water-storage-tank testbed and process-control dataset, trains and deploys five machine-learning models, and finds that detection is feasible in real time, with Random Forest and Decision Tree performing best across offline and online evaluations. The study is centered on reconnaissance attacks and a Modbus-based testbed setting.
Problem
Open IP connectivity exposes SCADA systems to cyber attacks, while public datasets may not capture evolving attack types or process-control conditions.
Method
The paper builds a water-storage-tank SCADA testbed, captures and labels network traffic, extracts features, trains five machine-learning algorithms, and evaluates them offline and online.
Results
Random Forest and Decision Tree performed better than the other models across offline and online evaluations using accuracy, FAR, and UND; KNN was the exception in online performance.
Takeaways & Limitations
The testbed and dataset support research on reconnaissance-attack detection and comparison of machine-learning models in SCADA environments.
Takeaways & Limitations
The testbed uses Modbus although other ICS protocols exist, so the reported setting is bounded by protocol support and deployment conditions.
Abstract
from arXiv · showhide
This paper presents the development of a Supervisory Control and Data Acquisition (SCADA) system testbed used for cybersecurity research. The testbed consists of a water storage tank's control system, which is a stage in the process of water treatment and distribution. Sophisticated cyber-attacks were conducted against the testbed. During the attacks, the network traffic was captured, and features were extracted from the traffic to build a dataset for training and testing different machine learning algorithms. Five traditional machine learning algorithms were trained to detect the attacks: Random Forest, Decision Tree, Logistic Regression, Naive Bayes and KNN. Then, the trained machine learning models were built and deployed in the network, where new tests were made using online network traffic. The performance obtained during the training and testing of the machine learning models was compared to the performance obtained during the online deployment of these models in the network. The results show the efficiency of the machine learning models in detecting the attacks in real time. The testbed provides a good understanding of the effects and consequences of attacks on real SCADA environments
1. Introduction
SCADA systems increasingly use open IP networks, exposing industrial processes to Internet-based attacks. The paper addresses limitations of public attack datasets by building a real-world testbed and dataset for evaluating machine-learning attack detection.
- SCADA systems monitor and control processes including water distribution, power grids, and oil and gas pipelines.
- Remote management has moved SCADA communications onto open IP networks, exposing these systems to cyber attacks.
- Public network-trace datasets may fail to represent new attacks as malware and attack strategies evolve.
- The paper develops a SCADA testbed, captures normal and abnormal traffic, extracts features, and builds a dataset for machine-learning detection.
- Five traditional machine-learning algorithms are trained and deployed to investigate real-time cyber-attack detection.
2. Background
The background introduces ICS-SCADA architectures and the Modbus protocol used for device communication. It outlines the layered system model, core SCADA components, and Modbus data-reference types.
- ICS-SCADA reference model: ICS encompasses control systems such as SCADA and combines control components to achieve industrial objectives.Examples include electrical, mechanical, hydraulic, and pneumatic components used in manufacturing or transportation.
- ICS-SCADA reference model: The ICS reference model divides industrial systems into four levels, from Level 3 corporate networks to Level 0 process infrastructure.Level 2 supervises monitoring and control functions, while Level 3 contains conventional IT services and systems.
- SCADA components: SCADA systems use HMIs, engineering workstations, history logs, and PLCs to monitor, program, record, and control industrial processes.PLCs connect to sensors or actuators and function as slave stations in the SCADA architecture.
- Modbus protocol: Modbus is an application-layer Client/Server messaging protocol that uses function codes to provide services between devices on Ethernet networks.The protocol defines standard message formats for inter-device communications in SCADA environments.
- Modbus protocol: Modbus data references use four types identified by leading address digits and four-digit memory locations.The listed ranges are 0xxxx, 1xxxx, 3xxxx, and 4xxxx, with corresponding address intervals.
3. The SCADA System Testbed
The testbed emulates a real SCADA water-storage control system while remaining smaller than a complete industrial plant. Its sensors, PLC, pumps, and valve implement automated tank-level control and support realistic cyber-attack experiments.
- Testbed purpose: The testbed emulates real-world industrial systems without replicating an entire plant and enables cyber-attacks against a water-storage control process.The tank represents part of water treatment and distribution, and its components are commonly used in real SCADA environments.
- Tank-level control: Two level sensors monitor the tank and signal the PLC when maximum or minimum water levels are reached.LS1 detects the maximum level, while LS2 detects the minimum level.
- Tank-level control: At the maximum level, the PLC turns off Water Pump 1, opens the valve, and turns on Water Pump 2 to draw water from the tank.This control sequence responds to the LS1 signal.
- Protocol choice: Modbus was selected despite alternatives such as DNP3 because many industries still use SCADA systems based on Modbus.A cited study identified 17,546 Internet-connected Modbus devices worldwide.
- Implementation: The testbed uses a Schneider M241CE40 PLC programmed in LADDER, with sensors connected to digital inputs and pumps and valves connected to PLC outputs.The PLC controls the water-storage process through these input and output connections.
4. Machine Learning Algorithms and Performance Measurements
The paper uses traditional supervised machine-learning algorithms for attack detection and evaluates them with confusion-matrix-derived metrics suited to imbalanced intrusion-detection data.
- Five supervised algorithms are selected for attack detection: Logistic Regression, Random Forest, Naïve Bayes, SVM, and KNN.
- Confusion-matrix evaluation distinguishes correctly and incorrectly classified normal and abnormal flows through TN, TP, FP, and FN.
- Accuracy measures the percentage of correctly predicted flows among all predictions.
- False Alarm Rate measures normal flows misclassified as attacks, while UN-Detection Rate measures attacks misclassified as normal traffic.
- Because normal traffic dominates realistic IDS datasets, the study reports FAR and UND alongside accuracy to reduce biased evaluation.
5. Attack Scenarios, Features Selection, and Evaluation Scenarios
The study conducts reconnaissance and exploit attacks against a SCADA testbed, captures and analyzes traffic features, and evaluates models through offline training/testing and online deployment.
- 5.1. Attack Scenarios: The attack study focuses on reconnaissance, including network scanning and exploit-based mapping of SCADA vulnerabilities.Reconnaissance attacks inspect network topology, devices, and vulnerabilities before later attacks.
- 5.1. Attack Scenarios: Easy-to-detect scans generate large Modbus/TCP packet-rate variations, whereas sophisticated exploits can produce traffic nearly indistinguishable from normal behavior.
- 5.1. Attack Scenarios: Captured traffic includes control information from devices such as valves, pumps, and sensors and is analyzed with Wireshark and Argus.
- 5.2. Features Selection: Features are selected by analyzing their behavior during normal and attack traffic to distinguish anomalous from normal flows.
- 5.3. Evaluation Scenario: Offline evaluation labels traffic, uses 80% for training and 20% for testing, then online evaluation deploys trained models on real network traffic for comparison.
6. Numerical Results
Offline evaluations show strong performance across the selected models, but online results reveal important differences, especially for KNN. Because the dataset is unbalanced, FAR and UND provide critical complements to accuracy.
- Decision Tree and KNN achieved 100% offline accuracy, while all trained models performed well during offline evaluation.
- Online accuracy remained close to offline accuracy for Decision Tree, Random Forest, Naïve Bayes, and Logistic Regression, unlike KNN.The authors report that KNN provides poor practical accuracy because its online and offline results differ significantly.
- Accuracy is not ideal for this evaluation because the dataset is unbalanced, with normal traffic dominant.The paper therefore evaluates false alarm and un-detection rates in addition to accuracy.
- Random Forest and Decision Tree had the lowest false alarm rates, followed by KNN, indicating stronger detection of normal traffic.The authors note that low FAR may also reflect model bias toward the dominant normal class.
- Un-detection rate is more critical than false alarm rate because it represents attacks misclassified as normal and therefore missed.UND also exposes model bias toward normal traffic in the unbalanced dataset.
- Offline UND was zero for Decision Tree and Random Forest and small for Naïve Bayes, Logistic Regression, and KNN, while online KNN performed worst.The other models had online performance close to their offline performance.
7. Conclusion
The paper develops a water-storage-tank SCADA testbed and dataset for studying attacks and evaluating machine-learning detectors. Across offline and online evaluation, Random Forest and Decision Tree performed best on imbalance-sensitive metrics, supporting the feasibility of detecting reconnaissance attacks in ICS environments.
- The testbed controls a water storage tank within water treatment and distribution and supports cybersecurity research on SCADA attack effects.
- Five ICS reconnaissance attacks were conducted, and traffic involving valves, pumps, and sensors was captured for feature extraction.Argus and Wireshark were used to extract features for the machine-learning dataset.
- Random Forest, Decision Tree, Logistic Regression, Naïve Bayes, and KNN were evaluated both offline and after deployment using online network traffic.The online and offline performances were compared directly.
- Using accuracy, FAR, and UND on an unbalanced dataset, Random Forest and Decision Tree outperformed the other models in both phases.
- The results demonstrate the feasibility of detecting reconnaissance attacks in ICS environments.Future work includes generating more attacks, testing different environments, and evaluating unsupervised algorithms.