PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Industrial Control System Testbeds and Datasets for Security Research

Mauro Conti, Denis Donadel, Federico Turrin

arXiv:2102.05631v3cs.CR

TL;DR

ICS security research needs safe, data-rich platforms because attacks on critical physical processes can cause severe damage, while real-system data are difficult to collect and share. This survey synthesizes ICS architectures, protocols, testbeds, datasets, and IDS results, then distills design and usage guidance for future research.

  • Problem

    Attacks on interconnected ICSs can damage critical physical processes, but real-system data are difficult to collect because attacks risk damage and companies may withhold private information.

  • Method

    The paper surveys ICS architectures, devices, protocols, testbeds, datasets, and IDS performance, while organizing testbeds and describing dataset-generation requirements.

  • Results

    The survey categorizes testbeds as physical, virtual, or hybrid, compares datasets, and reports the best-performing IDS algorithms identified for each dataset.

  • Takeaways & Limitations

    Researchers can use the survey’s comparisons, documentation requirements, attack-design guidance, and reported IDS results when developing or selecting ICS security research resources.

  • Takeaways & Limitations

    Surveyed datasets can have limited realism or documentation, including attacks injected with off-the-shelf tools rather than designed around protocol vulnerabilities.

Abstract

from arXiv · show

The increasing digitization and interconnection of legacy Industrial Control Systems (ICSs) open new vulnerability surfaces, exposing such systems to malicious attackers. Furthermore, since ICSs are often employed in critical infrastructures (e.g., nuclear plants) and manufacturing companies (e.g., chemical industries), attacks can lead to devastating physical damages. In dealing with this security requirement, the research community focuses on developing new security mechanisms such as Intrusion Detection Systems (IDSs), facilitated by leveraging modern machine learning techniques. However, these algorithms require a testing platform and a considerable amount of data to be trained and tested accurately. To satisfy this prerequisite, Academia, Industry, and Government are increasingly proposing testbed (i.e., scaled-down versions of ICSs or simulations) to test the performances of the IDSs. Furthermore, to enable researchers to cross-validate security systems (e.g., security-by-design concepts or anomaly detectors), several datasets have been collected from testbeds and shared with the community. In this paper, we provide a deep and comprehensive overview of ICSs, presenting the architecture design, the employed devices, and the security protocols implemented. We then collect, compare, and describe testbeds and datasets in the literature, highlighting key challenges and design guidelines to keep in mind in the design phases. Furthermore, we enrich our work by reporting the best performing IDS algorithms tested on every dataset to create a baseline in state of the art for this field. Finally, driven by knowledge accumulated during this survey's development, we report advice and good practices on the development, the choice, and the utilization of testbeds, datasets, and IDSs.

I. INTRODUCTION

ICSs integrate information technology with operational technology to monitor and control physical processes, but IT/OT convergence has expanded their vulnerability surfaces. Because these systems govern potentially dangerous processes, security research depends on controlled testbeds and shareable datasets for developing and evaluating detection techniques.

  • ICS Context: ICSs combine OT components such as PLCs, sensors, and actuators with IT workstations, databases, and other information-processing machines.The two networks were originally disconnected.
  • ICS Context: IT/OT convergence interconnected previously separate networks to support process digitization while opening new vulnerability surfaces.
  • Security Motivation: ICS attacks can produce severe physical, operational, economic, and safety consequences, including equipment damage and infrastructure shutdowns.Reported consequences include operational shutdowns, business waste, intellectual property fraud, and health and safety risks.
  • Research Infrastructure: Testbeds reproduce real-world ICSs in controlled, scaled-down environments, while physical implementations trade higher reliability for greater cost and virtual implementations provide an alternative.Developing testbeds remains challenging because of implementation costs, sharing capability, and fidelity.
  • Research Infrastructure: Datasets generated from testbeds can contain sensor and actuator measurements, network traffic, or both, enabling researchers to compare detection algorithms without attacking critical real systems.Real-ICS data collection is constrained by risks to physical processes and devices, privacy concerns, and possible exposure of proprietary vulnerabilities.

A. Contribution

The paper surveys ICS architecture, protocols, testbeds, datasets, and IDSs to consolidate the field’s security-research resources. It also provides performance baselines, design guidance, and a resource-sharing website for researchers and institutions.

  • Survey Scope: The survey provides background on ICS architecture and the main components characterizing these systems.
  • Survey Scope: It reviews industrial communication protocols, emphasizing their intrinsic security features and proposed security expansions.
  • Survey Scope: The paper analyzes testbeds, datasets, and IDSs to provide design concepts and an overview of the ICS security state of the art.
  • Baselines: For each presented dataset, it describes the best-performing IDS and identifies a baseline for comparing detection results.The authors note that the field lacks a defined methodology and baseline for testing detection frameworks.
  • Guidance: The survey reviews challenges and good practices for designing ICS testbeds, datasets, and IDSs, and introduces a website to support resource sharing.

B. Survey Organization

The paper proceeds from prior-work review and ICS background to protocols, IDSs, testbeds, datasets, guidance, and conclusions. Its later sections address practical advice and good practices for researchers and organizations creating or using these resources.

  • Paper Structure: Section II reviews previous surveys, while Section III introduces ICS architecture and common devices.
  • Paper Structure: Section IV covers typical ICS communication protocols, including their characteristics, security features, and real-world diffusion.
  • Paper Structure: Section V recalls IDS concepts and conventional approaches before later sections address testbeds and datasets.
  • Paper Structure: Section VIII presents advice and good practices for researchers and institutions developing new datasets or testbeds, followed by the conclusion in Section IX.

II. RELATED ICS SURVEYS

Earlier surveys separately examined ICS testbeds, security, smart-grid platforms, datasets, or design guidance. This survey aims to combine detailed coverage of ICS testbeds and datasets with IDS analyses and dataset-level performance baselines.

  • Research Gap: Existing literature did not provide a detailed analysis combining ICS datasets, testbeds, and the principal IDSs implemented on them.
  • Prior Surveys: Prior surveys examined ICS testbeds, security assessment, smart-grid testbeds, datasets, or testbed-building guidelines from different perspectives.
  • Positioning: Smart-grid survey classification emphasized research areas, whereas this work classifies testbeds mainly by platform type.
  • Positioning: The survey collects platforms useful for ICS security research and reports the best IDS performance on every dataset as a potential research baseline.

III. INDUSTRIAL CONTROL SYSTEMS

This section introduces ICS background by outlining the architecture and components discussed in the paper.

  • The section provides background on Industrial Control Systems.
  • It compares ICS architecture with classical systems architecture.
  • It summarizes widely used ICS components.

A. ICS Architecture

ICSs interconnect heterogeneous computing, electrical, and mechanical devices to manage physical processes, but IT/OT convergence and weak separation expose them to security risks.

  • ICS composition: ICSs combine computers, electrical and mechanical devices with sensors, actuators, controllers, protocols, SCADA systems, and human interfaces.
  • ICS composition: Unlike classical IT systems, ICSs carry both TCP/IP traffic and physical-process data, creating vulnerability surfaces for new attacks.
  • Purdue Model: The Purdue Model divides ICS networks into logical segments including Enterprise, DMZ, Control, and Safety zones.
  • Purdue Model: Purdue levels range from sensors and actuators at Level 0 to PLCs, HMIs, and manufacturing operations systems at Levels 1–3.
  • Security considerations: Although the DMZ should carry all traffic between IT and OT networks, this separation is rarely respected in real-world systems.
  • Security considerations: ICS security prioritizes reliability and availability because failures or data compromises can affect critical physical processes.

B. ICS components

ICS testbeds replicate industrial environments with physical or simulated components, while controllers, interfaces, and data systems support process monitoring and control.

  • Components: ICS devices may be installed or simulated in testbeds to replicate industrial environments.
  • Controllers: A PLC reads sensor inputs, executes programmed instructions, and produces outputs that control actuators and physical processes.PLCs are designed for continuous operation and millisecond-scale input, execution, and output cycles.
  • Operator and data systems: HMIs let operators monitor process values, alarms, and trends while sending manual commands to controllers.
  • CPS closed loop: Field devices form a CPS feedback loop in which sensors monitor, PLCs decide, and actuators apply actions.
  • Component roles: PLCs, RTUs, and IEDs interact with field devices, while HMIs, historians, engineering workstations, and SCADA manage system data and connections.

IV. INDUSTRIAL PROTOCOLS SECURITY

Industrial protocols were developed for OT requirements such as fault tolerance, reliability, and real-time operation, but some originally lacked basic security features.

  • Protocol security: OT protocols support requirements including fault tolerance and reliability.
  • Protocol security: Many protocols were originally designed for air-gapped environments, so security received less attention than real-time constraints.
  • Protocol security: Some original industrial protocols provide no authentication or encryption features.
  • Protocol comparison: The survey table records manufacturers, standard ports, original protocol release information, and security properties.
  • Protocol comparison: Enhanced protocol versions are documented with their names, release years, and encryption, integrity, or authentication measures.

A. Industrial Protocols

The survey describes Modbus as a widely used ICS protocol and catalogs protocol support across analyzed testbeds and datasets. It also compares protocol characteristics and security measures, including manufacturer-proposed enhancements.

  • Modbus: Modbus supports request execution and response exchange between clients and servers, with Modbus/TCP adding TCP/IP encapsulation to the serial protocol.
  • Protocol security: Table I compares protocol characteristics and security measures, distinguishing encryption, integrity, and authentication in enhancement extensions.
  • Testbeds and datasets: Table II records which protocols each testbed or dataset supports, while noting when the specific Modbus variant is unspecified.

TETRIS

This section presents security mechanisms and vulnerabilities across industrial protocols, including Modbus Security, S7Comm, PROFINET, CIP, OPC-UA, and IEC 61850-related communications.

  • Modbus Security: Modbus Security enhances Modbus/TCP with TLS, certificates, confidentiality, integrity, authentication, and certificate-based authorization.
  • S7Comm: S7Comm lacks security features and is vulnerable to replay attacks, while S7CommPlus adds replay protection but remains vulnerable to reverse debugging attacks.
  • OPC-UA: OPC-UA provides configurable security classes, secure channels, encryption, signatures, certificates, authentication, and authorization, but with slightly higher complexity.
  • IEC 61850: IEC 61850 communications can use digital signatures and TLS, but heterogeneous networks remain exposed to protocol-mapping vulnerabilities.

B. Industrial Protocols Employment

The survey compares protocol usage in testbeds, datasets, Internet measurements, and industrial markets. Modbus/TCP dominates research platforms, but market trends reveal important representation gaps.

  • Temporal trends: During two years of recording, Modbus and S7Comm devices increased while DNP3 devices decreased.
  • Internet exposure: Internet measurements found many Modbus, MQTT, and Niagara Fox devices, while EtherNet/IP, S7Comm, and BACNet appeared in significantly lower samples.
  • Market analysis: In 2020, EtherNet/IP and Profinet each held 17% market share, while EtherCat held 7%; Profibus and DeviceNet declined by 5%.
  • Surveyed platforms: Modbus/TCP is the most employed protocol across the surveyed testbeds, datasets, and Internet measurements.
  • Design implications: Profinet, BACnet, TridiumFox, NiagaraFox, and EtherCAT are identified as candidates for future testing platforms because their market presence is underrepresented or absent.

V. ICS ATTACK AND DEFENCE

The paper surveys typical ICS attacks and the techniques used to detect and mitigate cyberattacks. It organizes this discussion around attacks implemented in testbeds and datasets and available defense approaches.

  • Section scope: Section V reviews attacks targeting ICSs that are implemented in testbeds and datasets, then surveys techniques for detecting and mitigating cyberattacks.

A. Typical Attacks

ICS attacks target both network infrastructure and physical processes, exploiting interconnected devices and potentially causing unsafe or damaging outcomes. Effective defenses therefore require testbeds and datasets that represent realistic abnormal operating conditions.

  • Testbeds should simulate verisimilar attacks and datasets should include normal and attack data, but dangerous attacks cannot all be replicated safely.
  • ICS attack surfaces span network components and physical processes, and the two categories can combine toward a specific target.
  • Network Attacks: Reconnaissance, MitM, injection, replay, and DoS attacks respectively identify victims, manipulate communications, supply malicious inputs, retransmit valid messages, or exhaust resources.
  • Physical Process Attacks: Physical-process attacks alter system relationships through field devices such as sensors and actuators, sometimes after network access is obtained.
  • Physical Process Attacks: Stealth attacks use static or dynamic small perturbations to create long-term production losses or device degradation and are generally difficult to detect.
  • Physical Process Attacks: Device tampering corrupts recorded measurements, whereas direct-damage attacks introduce large process variations that can make equipment or environments unsafe.

B. ICS Defence Techniques

ICS defenses include network protections and process-aware detection techniques. IDS approaches differ in whether they match known misbehavior or learn deviations from normal operation, with machine-learning methods requiring substantial data.

  • DMZs, network separation, boundary protections, firewalls, and ICS-specific rules support security-by-design network architectures.
  • Process-aware techniques use sensor data to predict ICS operations and can be installed without changing the system topology or replacing every network device.
  • Knowledge-based IDS match runtime sensor, actuator, or network features to specific misbehavior patterns but react only to known dangerous behavior.
  • Anomaly-based IDS detect runtime features that differ from normal behavior, learning regular traffic patterns through unsupervised or semi-supervised approaches.
  • Modern machine-learning and deep-learning anomaly detectors generally become more precise when provided with more training data.
  • Network-based IDS analyze packets across the network, whereas host-based IDS monitor information from individual devices and can detect threats from non-network sources.

VI. ICS TESTBEDS

The survey analyzes ICS testbeds through a classification, requirements, and challenges framework, then describes selected testbeds across three categories.

  • The survey classifies ICS testbeds, introduces requirements for effective designs, and discusses challenges in developing them.
  • It then provides detailed descriptions of selected testbeds divided into three categories.

A. Testbeds Classification

The survey classifies testbeds as physical, virtual, or hybrid and compares them using functional, access, scope, cost, and resource criteria. These categories represent trade-offs among realism, safety, cost, and development effort.

  • Testbeds are classified by functional elements as Physical, Virtual, or Hybrid, although boundaries between categories can be minimal.
  • Physical Testbeds: Physical testbeds use real hardware and software, providing realistic measurement variation, latencies, and device vulnerabilities at higher cost and longer construction time.
  • Virtual Testbeds: Virtual testbeds reproduce networks and components through software simulations or emulations, reducing cost while limiting physical-process fidelity.
  • Hybrid Testbeds: Hybrid testbeds combine physical devices with software simulations, reducing cost and development time while potentially reducing fidelity.
  • The survey’s geographic maps show testbed locations, with marker size representing estimated cost and color representing associated-reference citations.
  • The classification records sector, category, physical-process implementation, license, scope, cost, reference, and download resource availability.
  • License categories distinguish open-source, open-description, education, collaboration, and unavailable testbeds, while scope includes security, forensic, pedagogy, and general applications.
  • Estimated implementation cost is low below 500 $, medium from 500 $ to 10k $, and high above 10k $, while resource detail varies by dataset.

B. Testbeds Requirements

The survey compares ICS testbeds and datasets by their physical, virtual, and hybrid characteristics, while emphasizing fidelity, documentation, labeling, and reproducibility requirements. It also reviews IDS approaches and reported baselines across datasets.

  • Testbed designs: Commercial hardware and hardware-in-the-loop integration can support realistic multi-system scenarios but may require extensive and expensive infrastructure.HAI Testbed interconnects three independent real ICSs with a real-time HIL, while NIST emulates three industrial systems without replicating a complete plant.
  • Security evaluation: Testbed security studies include denial-of-service, spoofing, man-in-the-middle, data-injection, and vulnerability-discovery scenarios.The reviewed platforms also evaluate protections such as attack detection, whitelist-based host protection, and combined IT/ICS IDS designs.
  • Testbed designs: Physical, virtual, and hybrid testbeds differ in how closely they reproduce industrial processes and control components.Hybrid systems combine simulated physical processes with commercial control hardware and software, while some virtual designs remain entirely simulated.
  • Dataset requirements: Dataset construction requires planned attack selection, representative system relationships, privacy consideration, and labels identifying normal or abnormal behavior.Breaking physical or process relationships during traffic injection can create detection bias and reduce generalization to physical environments.
  • Dataset requirements: Realistic ICS datasets remain difficult to share because company data can expose proprietary configurations, legacy vulnerabilities, or industrial-site weaknesses.Consequently, many available datasets are generated from scaled-down testbeds rather than real industrial environments.
  • IDS evaluation: Reported IDS performance varies by dataset, with representative results including 0.804 F1-score and 0.919 precision for DAICS on WADI.Other reviewed results include 0.970 F1-score for a model-based BATADAL approach, 0.955 for JRipper with AdaBoost, and 0.78 F1-score for CNN-based HAI detection.

E. Network Level

The surveyed network-level datasets span diverse ICS testbeds, protocols, attack types, formats, and detection methods. Reported IDS performance varies by dataset, attack, data balance, and modeling approach, underscoring the need for careful dataset design and evaluation.

  • CyberCity Dataset: CyberCity provides over 170k unlabeled packets from a simulated city ICS, covering scanning, information disclosure, command injection, MitM, and DoS attacks.About 16% of the data is estimated to be under attack; Modbus/TCP, EtherNet/IP, and NetBIOS are used.
  • Gas Pipeline and RTU Datasets: 0.9991 detection rate and 0.001 false positive rate were achieved on a gas-pipeline dataset after algorithm fine-tuning, while attack difficulty varied by attack type.Burst-value and setpoint-value injection attacks were especially problematic in the cited evaluation.
  • Gas Pipeline and Water Storage Tank: 0.981 Precision, Accuracy, and F1-Score were reported by SOCCADF on the Water Storage Tank dataset, while the New Gas Pipeline result was 0.988 for all three metrics.The framework uses an evolving Spiking Neural Network and was also adapted to semi-supervised validation on GRYPHON.
  • Lemay SCADA: Lemay SCADA results vary by attack and method, with Random Forest and SVM exceeding 0.999 for both F1-Score and Accuracy across packet batches.The deep-autoencoder framework requires attacks to last beyond a minimum duration; Precision reaches 100% for longer attacks.
  • Lemay SCADA: 0.96 Precision, 0.81 Accuracy, and 0.77 F1-Score were achieved by AdaBoost and Random Forest for DoS detection on Lemay SCADA.The evaluation used an expanded Smod tool to generate DoS, MitM, and replay attacks, while the IDS targeted DoS attacks.
  • Good Practices: Dataset design and IDS evaluation require explicit documentation, realistic attacker models, accessible attack deployment, and attention to class imbalance.The survey highlights control logic, attack descriptions, configuration settings, and dataset characteristics as documentation requirements.
Loading 2102.05631v3…