Source-linked AI summary
A Taxonomy of Network Threats and the Effect of Current Datasets on Intrusion Detection Systems
Hanan Hindy, David Brosset, Ethan Bayne, Amar Seeam, Christos Tachtatzis, Robert Atkinson, Xavier Bellekens
TL;DR
Current IDS research relies on datasets that often lack recent real-world network characteristics and do not adequately represent changing threats. The paper surveys prominent datasets and past-decade IDS research while proposing a taxonomy of threats and associated tools. It finds that current IDS research covers only 33.3% of the taxonomy and that just 11% of examined IDSs use recent or real-life generated or simulated datasets.
Problem
Available datasets do not reflect recent network attacks or changing architectures, limiting their suitability for developing IDSs against new and zero-day threats.
Method
The paper evaluates network-based datasets and past-decade IDS research, then develops a threat taxonomy organized by source, OSI layer, and active or passive mode, mapping threats to associated tools.
Results
Current IDS research covers only 33.3% of the threats in the proposed taxonomy, while only 11% of examined IDSs use recent or real-life generated or simulated datasets.
Takeaways & Limitations
The taxonomy and dataset analysis identify underrepresented threats and tools to guide creation of more current, publicly available datasets.
Takeaways & Limitations
Only 33.3% of known attacks are covered in the reviewed publicly available datasets, and only 11% of examined IDSs use recent or real-life generated or simulated datasets.
Abstract
from arXiv · showhide
As the world moves towards being increasingly dependent on computers and automation, building secure applications, systems and networks are some of the main challenges faced in the current decade. The number of threats that individuals and businesses face is rising exponentially due to the increasing complexity of networks and services of modern networks. To alleviate the impact of these threats, researchers have proposed numerous solutions for anomaly detection; however, current tools often fail to adapt to ever-changing architectures, associated threats and zero-day attacks. This manuscript aims to pinpoint research gaps and shortcomings of current datasets, their impact on building Network Intrusion Detection Systems (NIDS) and the growing number of sophisticated threats. To this end, this manuscript provides researchers with two key pieces of information; a survey of prominent datasets, analyzing their use and impact on the development of the past decade's Intrusion Detection Systems (IDS) and a taxonomy of network threats and associated tools to carry out these attacks. The manuscript highlights that current IDS research covers only 33.3% of our threat taxonomy. Current datasets demonstrate a clear lack of real-network threats, attack representation and include a large number of deprecated threats, which together limit the detection accuracy of current machine learning IDS approaches. The unique combination of the taxonomy and the analysis of the datasets provided in this manuscript aims to improve the creation of datasets and the collection of real-world data. As a result, this will improve the efficiency of the next generation IDS and reflect network threats more accurately within new datasets.
4, (SENIOR MEMBER, IEEE), ROBERT ATKINSON
The paper reviews IDS research and introduces a threat taxonomy organized by threat source, OSI layer, and active or passive mode. It evaluates datasets and prior NIDS studies to identify shortcomings affecting real-world threat detection.
- Scope and contributions: The study evaluates network-based datasets and reviews NIDS research from the past decade.Its contributions include assessing dataset limitations, reviewing NIDS research, and presenting a threat taxonomy.
- Threat taxonomy: The threat taxonomy categorizes attacks by source, OSI layer, and active or passive mode.It also maps threats to associated tools that can support dataset creation.
- IDS foundations: IDSs monitor network traffic or systems to detect anomalies, intrusions, or privacy violations and may log, alert, and mitigate detected intrusions.NIDS specifically analyzes communication between network nodes.
- Detection methods: Signature-based IDSs detect known attacks accurately but cannot detect zero-day, metamorphic, or polymorphic threats.Anomaly-based IDSs compare traffic patterns with normal traffic and generally offer better zero-day detection, but often have higher false-positive rates.
- Evaluation approaches: IDS evaluation approaches include mathematical models, simulation, emulation, and real experiments, each with distinct advantages and disadvantages.Detection may also be stateless per packet or stateful per flow, with stateful analysis providing contextual information.
1) Metrics for IDS Evaluation
The paper emphasizes that IDS evaluation requires multiple metrics rather than detection rate alone. It discusses classification, discrimination, imbalance-aware, resource, and operational measures.
- Core evaluation requirements: High detection rates and low false-positive rates are key requirements, but detection rate alone does not reflect IDS performance.False alarms can substantially limit practical IDS performance.
- Classification metrics: Precision measures the proportion of positively classified incidents that are truly positive.It is defined using true positives and false positives.
- Classification metrics: F1 is the harmonic mean of precision and recall and is better suited to imbalanced IDS classes.AUROC visualizes the trade-off between sensitivity and fallout.
- Imbalance-aware metrics: Matthews correlation coefficient applies to binary IDS classification, where incidents are labeled attack or normal.The geometric mean of accuracy is intended for imbalanced datasets with relatively few attack samples.
- Operational metrics: CPU consumption, throughput, and power consumption matter when IDSs run on high-speed networks or resource-limited hardware.Memory, transparency, safety, interoperability, and ease of use are additional evaluation factors.
2) Feature Selection and IDS
The paper treats feature learning and engineering as central to IDS design because feature choices affect traffic representation and performance. It also positions the work as a complement to surveys that emphasize algorithms, features, or specific architectures.
- Feature processing: Feature learning and engineering are essential because the selected features strongly affect IDS performance.Feature processing includes construction, extraction, and selection, with filter, wrapper, and embedded selection techniques.
- Feature representations: Network features may be naïve or rich abstractions and are grouped into time-series, header, payload, and statistical categories.Time-series and statistical features remain available when header and payload information is unavailable through encryption.
- Research gap: Existing IDS studies use published datasets, but prior reviews generally do not discuss the challenges posed by those datasets.Related surveys often focus on algorithms, feature selection, IDS characteristics, or machine-learning challenges.
- Research gap: Other studies focus on individual architectures such as wireless sensor networks, industrial process automation, or software vulnerability detection.These perspectives do not provide a detailed overview of dataset shortcomings or attack tools.
- Contribution: The paper complements prior surveys by evaluating dataset limitations and providing information about tools used to conduct attacks.Its analysis is intended to support datasets that better reflect current network threats.
III. IDS AND DATASETS SURVEY
This section surveys prominent datasets and recent IDS research while identifying limitations that reduce dataset usefulness for realistic network-threat evaluation. It emphasizes missing real-network characteristics, difficult sharing and simulation, and insufficient information for fair dataset comparison.
- Survey scope: The survey summarizes prominent datasets, their limitations, and recent IDS algorithms and evaluation datasets.It also discusses algorithmic trends observed over the past decade.
- Dataset shortcomings: Available datasets often lack real-life characteristics of recent network traffic, limiting the applicability of anomaly-based IDSs to production environments.Networks also change through new nodes, traffic loads, and topology changes, which old datasets do not capture well.
- Dataset types: Datasets may be real or synthetic, with synthetic traffic used to introduce attacks or balance attack classes.The section identifies real production-like traffic, complete scenarios, and labeling as desirable dataset properties.
- Research constraints: Dataset research is constrained because sharing may be prohibited and realistic scenario simulation requires many model parameters.These problems affect research using both synthetic and non-synthetic datasets.
- Comparability limits: Sparse dataset documentation can make evaluation and ranking unfair because realism lacks a standard metric and is often not reported.A realistically represented background and attack traffic may therefore be difficult to compare systematically with other datasets.
B. IDS AND ASSOCIATED DATASETS ANALYSIS
This analysis reviews 85 IDS manuscripts, their datasets, detected attacks, and algorithmic trends. It finds heavy reliance on older datasets and a narrow attack coverage, motivating extendable datasets and a threat taxonomy organized by OSI layer and threat characteristics.
- Review scope: 85 IDS manuscripts published from 2008–2020 were analyzed for algorithms, evaluation datasets, and detected attacks.The review used IEEE Xplore and Google Scholar queries to seek broad coverage of datasets, machine-learning techniques, and attacks.
- Dataset usage: 11% of the discussed IDSs used generated or simulated datasets, while many datasets lacked real-life properties.KDD-99 was the most frequently used dataset, followed by DARPA, whose age limits representation of current attacks.
- Attack coverage: 12 attacks were listed, with KDD-99’s DoS/DDoS, Probing, R2L, and U2R categories receiving the greatest coverage.The limited attack range highlights potential difficulty addressing broader and zero-day attacks.
- Dataset development: Extendable datasets and standardized generation methods are proposed as a basis for training anomaly-detection models against zero-day attacks.The paper connects this recommendation with machine-learning advances and its later threat-coverage taxonomy.
- Algorithmic trends: Machine-learning algorithms dominate the reviewed IDSs, while statistical and knowledge-based methods are less represented.ANN, SVM, and k-means are identified as the most used algorithms.
- Threat taxonomy: The threat taxonomy classifies attacks by source, affected OSI layer, and active or passive mode, while associating threats with attack tools.It is intended to support IDS construction, dataset development, accuracy, and false-positive reduction.
1) Network Threats
The taxonomy organizes network threats by attack type and highlights how attacks can target services, packets, hosts, code, hardware, and users. It also identifies substantial gaps in IDS coverage and proposes the taxonomy as a basis for improving datasets and threat representation.
- Network threats: DoS and DDoS flood networks with requests, rendering services unresponsive and preventing legitimate users from accessing them.The taxonomy divides them into flood, amplification, protocol-exploit, and malformed-packet attacks.
- Network threats: Packet forging generates traffic that resembles normal network packets and can enable unauthorized actions or theft of sensitive data.The passage lists credentials, personal data, payment-card details, and Social Security numbers as potential targets.
- Network threats: Scanning searches networks for active nodes, operating systems, and software versions, while MAC flooding and VLAN hopping represent additional network attacks.The taxonomy includes TCP- and UDP-based scanning.
- Host threats: Host threats include malware such as worms, viruses, spyware, Trojans, ransomware, and camouflage malware, which can compromise systems or maintain attacker control.Camouflage malware may use polymorphic or metamorphic techniques, while ransomware encrypts files until a ransom is paid.
- Code attacks: Code injection includes SQL injection and cross-site scripting, which can expose or delete database data and steal cookies or credentials.The passage distinguishes persistent and reflected XSS categories.
- Physical and human threats: Physical and human attacks include configuration tampering, backdoors, masquerade, phishing, privilege escalation, repudiation, session hijacking, and sniffing.Phishing uses electronic messages to obtain credentials or confidential data.
- Coverage and contribution: 72% of the attacks in the taxonomy are not covered by the discussed IDSs, motivating datasets that address missing attacks and represent threats structurally.The authors state that the taxonomy is intended to help generate datasets covering nonaddressed attacks and measure IDS coverage.
B. ATTACKING TOOLS
The paper links attack-specific tools to IDS development and summarizes major shortcomings in current datasets. These shortcomings include limited coverage of real-life, emerging, specialized, and evolving threats.
- Attacking tools: Attack-associated tools can guide researchers when building IDSs for specific threats and selecting tools for evaluation.Examples include Caffe-Latte, Hirte, EvilTwin, and Cain and Abel for impersonation attacks.
- Dataset limitations: 33.3% of known attacks are covered in the reviewed publicly available datasets, limiting IDS use in real-life environments.The paper identifies attack coverage as one of the biggest dataset challenges.
- Dataset limitations: Only 11% of past-decade IDSs use recent or real-life generated or simulated datasets.The paper connects this finding with limited ability to cope with emerging needs.
- Dataset limitations: Attacks evolve faster than current datasets can accommodate, creating a need for more efficient dataset generation and public release.The paper states that faster dataset updates could support quicker IDS retraining.
- Dataset limitations: Public datasets for special-purpose IDSs, including IoT, SCADA, and Tor networks, remain insufficient.
- Dataset outlook: Datasets do not currently cover emerging technologies such as Blockchain, SDN, NFV, and Big-Data or their associated threats.The paper calls for dataset generation to follow networking and technology trends.
B. RECOMMENDATIONS
The recommendations prioritize representative, current, and extensible data alongside complex IDS models. They call for realistic traffic, validated datasets, broader threat coverage, precise evaluation, and adaptable attack injection.
- ML-First Vs Data-First: IDS research should prioritize valid, representative, and accurate data before machine-learning model construction.The paper warns that skewed and biased data produce models unfit for exploitation.
- Evaluation: Evaluation should report metrics beyond accuracy, including false positives, recall, geometric mean for imbalanced data, and throughput.The recommendation responds to limitations of relying only on conventional machine-learning loss and accuracy.
- Generation and Resilience: Attack-generation methods need standardized documentation, raw packet data, realistic attack injection, and dataset variations to extend dataset lifetime.The paper identifies inadequate documentation and limited traffic-injection proposals as barriers to combining datasets and adding attacks.
- Realistic Data: New datasets should use monitored real traffic or real traffic as ground truth for realistic background generation.The recommendation addresses benchmark datasets that lack real-life properties.
- Dataset Validation: Dataset validation should compare network-traffic properties and similarities between real and synthetic traffic.The paper recommends network traffic validation techniques and explicit real-versus-synthetic comparisons.
- Coverage and Extensibility: Datasets should cover wider and updated attack ranges, including bespoke networks and zero-day threats, while remaining modular and extendable.The taxonomy is intended to support broader attack coverage, and modular datasets can integrate new data as networks change.
- Evidence for Recommendations: The survey finds that current IDS research covers around 33.3% of the threats in the proposed taxonomy.The paper also reports that 97.25% of examined IDSs use machine learning, with ANN, k-means, and SVM dominant despite outdated datasets.
- Open Taxonomy: The threat taxonomy is open-sourced through GitHub so researchers can contribute to its future revision and relevance.The taxonomy is presented as a resource for extending threat coverage in future IDS research.
APPENDIX A
Appendix A compiles prominent intrusion-detection studies from 2008–2020, recording datasets, algorithms, and detected attacks. The entries show repeated use of established datasets and a broad range of algorithmic approaches and attack categories.
- Scope: The appendix table covers prominent intrusion-detection research from 2008–2020.Table A.1 is used in the paper’s analysis of IDS research over the past decade.
- Table Structure: Each research entry records the publication year, dataset, algorithms, detected attacks, and reference.The appendix describes rows as manuscripts linked to datasets, algorithms, and attack capabilities.
- Supporting Table: Table A.2 separately summarizes dataset publication years and attack remarks for datasets discussed in Section III-A.This table complements the manuscript-level inventory in Table A.1.
- Datasets: KDD-99 and NSL-KDD recur across entries, alongside datasets such as DARPA, Kyoto2006+, UNB-CIC, UNSW-NB15, and CICIDS2017.The listed studies use both legacy benchmark datasets and newer dataset families.
- Algorithms: The recorded methods include SVM, k-means, neural networks, decision trees, Bayesian methods, clustering, deep learning, and ensemble approaches.The appendix lists these algorithms across studies from multiple publication years.
- Detected Attacks: Many entries detect the recurring categories Probing, DoS, R2L, and U2R, while later entries also include web, brute-force, bot, scan, and Heartbleed attacks.The appendix includes both repeated legacy attack categories and broader later attack groupings.