Source-linked AI summary
A Roadmap to Available ICS Datasets and Testbeds for Cybersecurity Research
Ebtesam J. Alqahtani, Mohammad Hammoudeh
TL;DR
ICS cybersecurity research lacks suitable datasets and experimental environments amid increasingly connected OT and IT systems. This paper reviews and classifies datasets, testbeds, and Digital Twins in a unified roadmap, finding persistent gaps in realism, standardisation, diversity, and OT-specific AI validation. It identifies research gaps and proposes recommendations for developing realistic, scalable, and interoperable resources.
Problem
ICS cybersecurity solutions are difficult to develop and test because suitable datasets and experimental environments remain limited.
Method
The paper systematically reviews, analyses, and classifies ICS cybersecurity datasets, testbeds, and Digital Twins using taxonomies and selection criteria.
Results
The analysis identifies persistent gaps including limited realistic and diverse datasets, nonstandard benchmarking, insufficient modern attack scenarios, and difficulty validating AI-driven cybersecurity solutions.
Takeaways & Limitations
The roadmap provides research-gap identification and recommendations for creating realistic, scalable, and interoperable ICS cybersecurity resources.
Takeaways & Limitations
Testbed design must balance fidelity and scalability because physical environments are costly and virtualised environments can poorly reproduce hardware and process behaviour.
Abstract
from arXiv · showhide
Industrial Control Systems (ICS) are the backbone of many critical infrastructure sectors; however, their growing level of connectivity, long lifespan and integration with the Information Technology (IT) environment introduces numerous cybersecurity challenges. The merging of Operational Technology (OT) and IT along with the deployment of Industry 4.0 technologies increases the attack surface of ICS environments, which in turn makes them more vulnerable to advanced cyber threats. Therefore, many researchers have shown interest in the field of cybersecurity of ICS. The topics of intrusion detection, anomaly detection, threat intelligence, attack simulation and resilience assessment of ICS have received much attention. Nevertheless, the development and testing of cybersecurity solutions for ICS remains to be challenging due to the lack of appropriate datasets and experimental environment. The main objective of this paper is to provide the roadmap of existing ICS cybersecurity datasets, testbeds and digital twins. This paper presents various taxonomies along with systematic analysis of architecture, characteristics, capabilities, pros and cons of these tools. The results of the analysis demonstrate the presence of persistent problems such as lack of standardized benchmarking datasets, lack of modern attack scenarios, insufficient number of datasets based on real operational traffic and difficulty in validating artificial intelligence-driven cybersecurity solutions. In addition to summarizing current research on ICS cybersecurity datasets and testbeds, this roadmap provides the identification of research gaps and recommendations on creation of new tools.
1 Introduction
Connected, data-driven OT environments have expanded AI applications while creating dataset-related deployment difficulties. The paper responds with a unified roadmap that reviews and analyses ICS cybersecurity datasets, testbeds, and Digital Twins.
- AI now supports predictive maintenance, process optimisation, quality inspection, energy management, and cybersecurity across the industrial lifecycle.
- OT deployment of AI faces difficulties including a lack of high-quality datasets.
- The paper reviews and analyses ICS cybersecurity datasets, testbeds, and Digital Twins within a unified roadmap framework.
- The study addresses challenges in selecting, evaluating, and developing suitable resources for ICS cybersecurity research.
- It analyses resource characteristics and limitations, identifies research gaps, and proposes a roadmap for realistic, scalable, and interoperable resources.
2 Related Works
Prior work separately reviews ICS testbeds and Digital Twins, while this review integrates datasets, testbeds, and DTs with taxonomies, comparisons, and future directions. The comparison frames the proposed review as broader in scope than representative surveys.
- Existing literature broadly comprises reviews of physical, virtual, and hybrid ICS testbeds and surveys of Digital Twin architectures, applications, and adoption challenges.
- Testbed reviews examine design, architecture, implementation, and applications including vulnerability assessment, attack experimentation, IDS, and cybersecurity education.
- Digital Twin surveys address IDS, vulnerability assessment, cyber ranges, secure cyber-physical systems, anomaly detection, threat analysis, and cyber defence.
- The comparison evaluates review scope, coverage of datasets, testbeds, and DTs, taxonomies, comparative studies, research gaps, and future directions.
- The proposed review integrates datasets, testbeds, DTs, taxonomies, selection criteria, and future research directions within a wider scope.
3 ICS Cybersecurity Datasets
ICS cybersecurity datasets span vulnerability analysis, intrusion and anomaly detection, attack classification, and other AI-driven tasks across multiple industrial sectors. The review compares them by data, source, attack labels, and primary ML application.
- Public ICS cybersecurity datasets support intrusion detection, anomaly detection, attack classification, and other AI-driven cybersecurity techniques across multiple industrial sectors.
- Water-sector datasets such as WUSTL-IIoT-2018 capture network traffic and sensor data from realistic testbeds during normal operation and cyberattacks.
- Power-system datasets combine simulated cyberattacks with network traffic, system logs, and operational measurements for smart-grid and cyber-physical security research.
- Railway, manufacturing, building-automation, oil-and-gas, and IIoT datasets represent sector-specific processes, protocols, telemetry, and attack scenarios.
- ICS-LTU2022 compiles structured vulnerability information for vulnerability analysis, risk assessment, and vulnerability prediction rather than intrusion or anomaly detection.
- Cross-industry datasets include ICS-ADD and Cyber4OT, with security events or realistic network traffic supporting intrusion detection, anomaly detection, and security monitoring.
- The review categorises datasets by application domain and compares collected data, source, labelled attack availability, and primary ML application.
4 ICS Cybersecurity Testbeds
ICS testbeds address the experimental limits of static public benchmarks by enabling controlled cybersecurity testing and, in many cases, dataset generation. They range from physical and virtual systems to HIL, cloud/container, and domain-specific architectures.
- Static public benchmarks cannot support controlled experiments or data generation, motivating the adoption of ICS testbeds for cybersecurity testing.
- Physical testbeds use real industrial hardware and communication networks for high-fidelity attacks, defence validation, and dataset generation.
- Virtual testbeds simulate ICS in software, reducing hardware requirements while supporting education, attack simulation, defence validation, security evaluation, and dataset generation.
- HIL testbeds combine real industrial hardware with real-time simulation to provide high-fidelity cyber-physical experimentation with flexibility and safety.
- Cloud- and container-based testbeds improve scalability, portability, and reproducibility through virtualised industrial environments.
- Domain-specific testbeds emulate particular industrial sectors or communication technologies for sector-specific threat and defence evaluation.
- The review classifies testbeds by architecture and evaluates implementation method, protocol support, cybersecurity applications, and data-generation features.
5 Digital Twins for ICS Cybersecurity
Digital Twins (DTs) are reviewed as virtual representations that support monitoring, simulation, cybersecurity testing, AI-based detection, and incident response across ICS research.
- Operational Monitoring and Simulation: DTs provide dynamic virtual representations of physical industrial systems for real-time monitoring, simulation, and cybersecurity evaluation.The review categorizes DT platforms according to their primary cybersecurity purpose.
- Security Testing and Validation: Security-oriented DTs enable cyber ranges, attack experimentation, synthetic dataset generation, and validation of security countermeasures without affecting physical systems.Applications include process-aware attacks such as command injection, denial of service, and measurement manipulation.
- AI-Enabled Cybersecurity: AI-enabled DT frameworks generate simulated operational data for offline ML training and deploy models at the edge for real-time fault and cyberattack detection.Other frameworks combine sensor monitoring with knowledge graphs and ontology-based data modelling to identify abnormal behaviour.
- Incident Response and Mitigation: Process-based DT security frameworks integrate security simulations into SOC workflows to support cybersecurity monitoring, incident analysis, and incident response.Studies also examine how different DT modalities enhance incident response activities in cyber-physical systems.
- Comparison: The reviewed DT platforms are compared by implementation approach, AI-enabled capabilities, overall functionality, and primary cybersecurity purpose.Table 4 summarizes representative ICS cybersecurity DTs using these comparison dimensions.
6 Datasets, Testbeds, and DTs: When and How to Use Them
Datasets, testbeds, and DTs support different stages of AI-based ICS cybersecurity research, so resource selection should follow the research objective and lifecycle stage.
- Roles: Datasets provide benchmark data for model training and evaluation, testbeds enable controlled experimentation, and DTs support continuous virtual analysis and validation.The three resource types differ in realism, data generation, deployment cost, and intended applications.
- Selection by Objective: Datasets are most suitable for model development and benchmarking, while testbeds support attack execution, data creation, and experiment validation before implementation.DTs extend these capabilities through real-time monitoring, predictive analysis, and virtual representations of physical systems.
- Comparison Dimensions: DT implementation is classified as physical, virtual, or hybrid, while AI indicates whether AI or ML is a core component of the platform.Physical DTs connect to real industrial assets; virtual DTs are software-based, and hybrid DTs combine virtual models with physical hardware or HIL components.
- Integrated Lifecycle: Datasets, testbeds, and DTs should be treated as complementary technologies forming an integrated ecosystem for the complete AI-based ICS cybersecurity lifecycle.This complementary relationship forms the foundation of the roadmap.
7 Research Gaps and Open Challenges
ICS cybersecurity research faces persistent challenges in balancing testbed fidelity, scalability, diversity, operational realism, dataset quality, and standardized evaluation.
- Testbed Fidelity and Scalability: Physical testbeds offer high fidelity but are costly and difficult to maintain, whereas simulation and virtualization scale better but poorly represent hardware and process behaviour.Accurate modelling of processes and network environments is especially critical for cyber-physical systems.
- Testbed Diversity: ICS testbeds often cover only a subset of heterogeneous vendor devices, while legacy and modern technologies complicate testing and expose gaps in traditional IT security solutions.The diversity of industrial systems makes representative testbed coverage difficult.
- Operational Constraints: Latency, sampling rate, reliability, and availability requirements mean communication delays, packet loss, and network failures can affect system stability.Cybersecurity solutions should therefore be evaluated under operational conditions that include both cyber and physical impacts.
- Scalable Design: Physical infrastructure is costly to scale, while virtualization, HIL, and simulation cannot replace physical components in CPS analysis.Future testbeds should use modular and extensible designs that connect new devices, networks, and simulated processes.
- Dataset Availability: Realistic and diverse ICS cybersecurity datasets remain limited because real-environment collection is constrained by safety, confidentiality, and operational requirements.AI anomaly detection needs normal behaviour, process dynamics, and diverse attack scenarios, while logging choices trade computational overhead against detection effectiveness.
- Evaluation Standards: No standard evaluation approach exists because studies use different datasets, attack scenarios, and metrics, preventing direct comparison of security solutions.The paper identifies benchmarking frameworks as a needed response.
- AI in OT: AI deployment in OT remains problematic when approaches rely on IT assumptions rather than deterministic communication, process dependencies, safety constraints, and computational limitations.The paper highlights lightweight and explainable AI as a direction for further research.
8 Roadmap for Next-Generation ICS Dataset, Testbed and Digital Twins Development
The roadmap synthesizes prior frameworks into a systematic workflow for developing realistic, scalable, reproducible, and interoperable ICS cybersecurity resources. It guides researchers from objective setting through environment design, attack scenarios, dataset validation, and AI-security assessment.
- Roadmap foundation: Prior frameworks for datasets, cyber ranges, digital twins, and AI benchmarking are synthesized into a roadmap for future ICS resource development.These frameworks provide practical advice for creating more realistic datasets and testbeds, improving experiment replicability, and establishing AI-solution benchmarks.
- Systematic workflow: The roadmap guides researchers from defining objectives and selecting development environments to designing processes, creating attacks, validating datasets, and assessing AI techniques.
- Design requirements: Realism, scalability, reproducibility, protocol variety, and validation are emphasized as core requirements for datasets, testbeds, and digital twins.
9 Recommendations for Researchers and Practitioners
The recommendations align resource selection and evaluation with research objectives while promoting diverse attacks, protocol coverage, documentation, standardization, and multi-environment validation. A systematic procedure should precede benchmarking and deployment.
- Resource selection: Select datasets according to research aims, application domain, attack diversity, communication protocols, and data characteristics rather than popularity.
- Evaluation: Evaluate proposed solutions across public datasets, testbeds, and digital twins to combine reproducible benchmarking with realistic-environment validation.
- Resource development: Future resources should include diverse cyber-physical attacks, multiple industrial protocols, high-quality labels and ground truth, documentation, and standardized public benchmarks.
- Workflow: The recommended procedure defines objectives, selects relevant resources, creates attack scenarios, validates AI models in different environments, and then benchmarks and deploys them.
10 Conclusion
The paper reviews ICS cybersecurity datasets, testbeds, and digital twins through a unified taxonomy and identifies persistent deficiencies in benchmarking, process representation, coverage, and integration. It concludes that future resources require greater standardization, diversity, scalability, and realism.
- Review scope: The review categorizes ICS cybersecurity datasets, testbeds, and digital twins by use cases, data attributes, protocols, attack scenarios, and research purposes.
- Review scope: The taxonomy supports resource selection by documenting benefits, drawbacks, and applicability to different cybersecurity tasks.
- Research gaps: Persistent challenges include inconsistent benchmarking, limited industrial-process representation, coverage problems, and weak physical-virtual-data integration.
- Future direction: The analysis supports developing standardized and diverse datasets, scalable hybrid testbeds, AI-enabled experimental environments, and evaluation approaches.