Network Intrusion Datasets: A Survey, Limitations, and Recommendations

Motivation

• Public datasets support evaluation and comparison of intrusion detection systems, which must balance attack detection against false alarms.The introduction notes that data-driven approaches, especially machine learning, have increased reliance on data in intrusion detection.Source: §1 · p. 1
• The introduction describes a longstanding concern about dataset quality and says many newer datasets remain undiscovered and unanalyzed.It notes that dozens of datasets appeared after the 2019 survey, with no comprehensive study since then.Source: §1 · pp. 1–2

Contributions

• The survey compares 89 public NIDS datasets published through 2023, addressing the lack of a comprehensive, up-to-date resource.The authors describe it as the largest survey of publicly available NID datasets to their knowledge.Source: §1 · pp. 1–2
• The authors manually analyze 13 properties per dataset and examine dataset popularity in contemporary state-of-the-art NIDS research.They share analysis notebooks so readers can inspect dataset structures without downloading the full data.Source: §1 · p. 2
• The paper synthesizes dataset flaws in relation to NID-specific properties and discusses recommendations for data selection, creation, and use.Its stated aim is to support more effective data use and correct data creation and handling.Source: §1 · p. 2

2. Survey Methodology

• The survey follows a systematic literature review methodology to support comprehensive, unbiased, and reproducible results.The section introduces the survey scope and research questions, with further methodological details in Appendix A.Source: §2 · p. 2

2.1. Scope of the Survey

• The scope spans datasets from environments including IoT, CPS, SDNs, and SCADA, but includes only datasets with packet or flow traces.Datasets containing only environment-specific features, such as CPS or SCADA sensor values, are excluded.Source: §2.1 · p. 2
• The authors caution that the survey is not exhaustive and should not be the sole data reference for specific use cases.They recommend consulting surveys focused on relevant cybersecurity subdomains when needed.Source: §2.1 · p. 2

2.2. Research Questions

• The research questions are framed in relation to the paper’s goal and scope.The supplied passage does not enumerate the questions.Source: §2.2 · p. 2

• The section asks what datasets are available for network intrusion detection research and development.Source: RQ1: What datasets are available for… · p. 2

• The section frames RQ2 around the qualitative and quantitative properties of identified datasets.Source: RQ2: What are the key qualitative and… · p. 2

• The review covers scientific literature published through 2023 and synthesizes its findings.Source: RQ4: What were the most popular… · p. 2
• After filtering, the authors downloaded, manually analyzed, and compared 89 relevant NIDS datasets.Source: RQ4: What were the most popular… · p. 2
• The paper directs readers to Appendix A for the search and study-selection process.Source: RQ4: What were the most popular… · p. 2

3. Domain-Specific Properties and Their Impact on Data

• NID’s domain properties affect data collection, handling, and interpretation, and can violate the i.i.d. assumption underlying machine learning.Source: §3.1 · p. 3, Figure 1 · p. 3

3.1. When Hell Comes to Earth: Unpleasant Properties of the Network Intrusion Detection Domain

• Network diversity, changing traffic patterns, adversaries, and differences between networks make observed traffic variable across time and settings.Source: §3.1 · p. 3
• Errors carry high costs, while uncertain ground truth and difficult labeling make reliable intrusion labels hard to establish.A low false-positive rate can still yield many false alarms under severe traffic imbalance; experts may disagree, and infected-host traffic can include legitimate activity.Source: §3.1 · p. 3
• Data and concept drift can cause model performance to decline over time, constraining real-world NIDS use despite proposed updating approaches.Source: §3.1 · p. 4

3.2. Limitations of Existing Datasets

• Static datasets become less representative as networks and threats evolve, while privacy anonymization and traffic sampling can reduce realism or completeness.Source: §3.2 · p. 4
• Lab testbeds may not capture realistic behavior, and most datasets leave users to address class imbalance in training data.Source: §3.2 · p. 4
• Incomplete documentation, absent standard splits, and inconsistent feature sets hinder sound conclusions and comparisons across datasets.The authors identify documentation, explicit splits, feature-set discrepancies, reproducibility, and data-handling errors as limitations within dataset authors’ control.Source: §3.2 · pp. 4–5
• Limited attack coverage, mislabeling, data-handling errors, and scarce reproduction materials constrain dataset validity and reuse.Public datasets cover only 33% of known attacks; missing raw packets or extraction code can prevent validation and extensions.Source: §3.2 · p. 5

4. Network Intrusion Data and Its Properties

• The survey organizes its 13 properties into five categories to provide a comparative dataset view and a protocol for evaluating datasets beyond those reviewed.The categories are General information, Nature of data, Data volume, Network properties, and Evaluation.Source: §4 · p. 5, §4.5 · p. 9

4.1. General Information

• General information captures dataset scenario, benign-traffic and attack origins, distinguishable attack counts, and attack types; generated traffic offers flexibility but remains limited by realism.The survey distinguishes real, emulated, and synthetic traffic and uses attack categories as a concise reference rather than a comprehensive taxonomy.Source: §4.1 · pp. 5–7

4.2. Nature of Data

• Data formats include packets, flows, logs, and combined network-host features; packets offer detail, flows support throughput despite encryption, and combined features speed experimentation but reduce flexibility.Packet payloads enable deep packet inspection when retained; enriched network-host feature sets avoid custom multi-source extraction.Source: §4.2 · pp. 7–8

4.3. Data Volume

• Dataset volume is measured from the data itself, and reported capture duration excludes temporal gaps, with continuity assessed using a threshold set to 1% of the dataset span.Discontinuous data may prevent time-dependent detection systems from using sample dependencies and can bias evaluation.Source: §4.3 · pp. 8–9

4.4. Network Properties

• Network properties record network size and type and whether capture covers the whole network; incomplete captures can skew traffic patterns and call for specialized evaluation or NIDS types.For example, honeypot-only captures can contain disproportionately more malicious traffic than regular hosts.Source: §4.4 · p. 9

4.5. Evaluation

• Evaluation properties record train-test splits and labeling; splits aid transparent comparisons, while indirect or absent labels make evaluation error-prone and warrant caution.Without predefined splits, users must split data themselves, which can introduce evaluation bias and selective reporting.Source: §4.5 · p. 9
• Figure 3 classifies network intrusion datasets by focus and notes that special-purpose categories are nonexclusive, allowing one dataset to fit multiple categories.A dataset may combine a particular intrusion type with a specific environment.Source: Figure 3 · p. 10

5. Network Intrusion Datasets

• The survey selected datasets using its systematic literature review and analyzed 13 properties through documentation review and exploratory data analysis.The resulting notebooks and scripts are available through the paper’s GitHub repository.Source: §5 · p. 10

5.1. General-Purpose Datasets

• General-purpose datasets benchmark traditional LAN or enterprise environments with diverse intrusions, while the survey distinguishes them from special-purpose datasets.The section also previews reviews of DARPA, CIC, and other general-purpose datasets, plus sources, popularity, and research trends.Source: §5.1 · p. 10, §5 · p. 10

5.1.1. DARPA & KDD

• DARPA 1998 established the first public IDS benchmark generation with 32 attacks, and KDD Cup 1999 converted its raw data into 41 labeled features and became a de facto standard.NSL-KDD removed duplicate records and reduced dataset size, while GureKDDcup enriched records with packet payload data.Source: §5.1.1 · p. 10
• DARPA 1999 and LLDOS extended the DARPA family, but their raw-only format kept them from gaining significant community attention.LLDOS focused on infiltration and DoS attacks in the DARPA 1999 environment.Source: §5.1.1 · p. 10
• DARPA 1998/1999 and KDD'99/NSL-KDD faced criticism over validity, artifacts, attack stealthiness, and inconsistencies suggesting labeling issues or duplication, and are over 25 years old.The passage begins to state a consequence of these issues and the datasets’ age, but its final sentence is truncated, so no further conclusion is included.Source: §5.1.1 · p. 10
• Legacy datasets are no longer recommended for IDS benchmarking, although many recent studies still rely on them.Some datasets have feature files in which most merged features are NULL, captures combining whole-network traffic with isolated-device captures, or attack labels with fewer than 20 samples.Source: §5.1.1 · p. 16

5.1.2. Canadian Institute for Cybersecurity

• CIC-IDS2017 and CSE-CIC-IDS2018 introduced profile-based traffic generation and a popular CICFlowMeter feature set, but reported errors led to repaired or reimplemented alternatives.Reported problems include duplicated features or packets, incorrect flow construction, incoherent timestamps, labeling issues, and deprecated attack tools; resulting alternatives include Improved CIC-IDS2017, Improved CSE-CIC-IDS2018, and LYCOS-IDS2017.Source: §5.1.2 · p. 18

5.1.3. NetFlow Datasets

• NetFlow datasets use standardized features already deployed for monitoring, making them more practical through lower implementation costs and more realistic benchmark results.Their scope includes honeypot traces, emulated and real-world traffic, ISP captures, and datasets regenerated from popular NIDS datasets using nProbe.Source: §5.1.3 · p. 18
• Twente lacks benign traffic, while Appraise H2020 and Simargl2022 use minimalist or underdocumented scenarios and features.Twente’s honeypot capture limits its use to signature-based systems; Appraise H2020 has 19 features and lacks network-environment and attack-generation details, and Simargl2022 simulates reconnaissance, DoS, and botnet intrusions.Source: §5.1.3 · p. 18

5.1.4. Established Datasets

• UNSW-NB15 has documentation gaps and reported mislabeling, simulation artifacts, and limited diversity, while NGIDS-DS shares documentation issues and relies on indirect labels.UNSW-NB15 includes nine intrusion classes and 49 flow-based features; NGIDS-DS provides raw data labeled via host logs.Source: §5.1.4 · pp. 18–19
• GTCS provides four attack scenarios in a simple virtualized topology, while LITNET-2020 was captured over 10 months on a large academic network.CDX, Kyoto 2006+, and NetML have received limited attention because their labels or feature sets are constrained; LITNET-2020 also omits emulation-tool details.Source: §5.1.4 · p. 19

5.1.5. New Datasets

• Recent general-purpose datasets may better reflect current benign traffic and intrusions, but have not yet been thoroughly analyzed or commonly cited.The paper therefore introduces these newer datasets individually.Source: §5.1.5 · p. 19
• CUPID enables comparison of professional testers’ scripted and human-generated traffic but provides only binary labels; UKM-IDS20 contains 12.9 thousand flows.UKM-IDS20 emulates two small office networks over two weeks, with attacks introduced in the second week; its 48 features were extracted by a custom algorithm.Source: §5.1.5 · p. 19
• I-Sec’s simplistic topology and absent benign-traffic strategy make reliable benchmarks questionable, while OD-IDS2022 lacks environmental and normal-traffic details.I-Sec includes DoS and scanning traffic; OD-IDS2022 emulates 28 modern attacks in a virtualized network.Source: §5.1.5 · p. 19
• AIT LDS2.0 supports multi-step cyber-kill-chain research, whereas FLNET2023 was designed to collect traffic across locations for federated-learning benchmarks.AIT LDS2.0 captures logs in eight configurable testbed scenarios and has a related NetFlow dataset; FLNET2023 emulates a ten-router network and adds a location feature.Source: §5.1.5 · p. 19
• 99.9999% of UWF-ZeekData22’s malicious traffic is reconnaissance and discovery, making it unsuitable for systems focused on other intrusion types.The dataset labels traffic using MITRE ATT&CK phases and participants’ logs, but contains too few samples for other phases.Source: §5.1.5 · p. 19
• TII-SSRC-23’s benign traffic is only 0.015% of its 8.7 million flows, while VHS-22 combines five datasets into 115 classes.TII-SSRC-23 offers six benign traffic types and 26 malicious subtypes; its severe imbalance complicates use. VHS-22’s diverse replayed traffic may challenge anomaly detectors’ ability to learn normality.Source: §5.1.5 · pp. 19–20

5.2.1. Intrusion-Specific

• DoS/DDoS datasets span real-world attack samples and emulated flooding, while other examples target slow DoS, buffer overflow, SSH brute force, and HTTPS attacks.CAIDA DDoS Attack 2007 and Booters provide real-world samples; CIC-DDoS2019 emulates flooding, while USB-IDS-1 evaluates slow DoS against webserver defenses.Source: §5.2.1 · p. 20
• Botnet datasets range from 13 real-world CTU-13 scenarios to DNS-only detection in ISOT HTTP Botnet and merged traffic in other collections.CTU-13 runs unrestricted malware in a sandbox within an academic network; HogZilla and Botnet combine data from earlier datasets.Source: §5.2.1 · p. 20
• Anomaly datasets include enterprise flow and host logs, automatically labeled backbone traces, and traffic labeled with external threat intelligence.Kent and Unified Host and Network have heavy anonymization and no direct labels; MAWILab's inactive labeling project may have outdated detectors, and it has limited recent data. TrabID separates known, similar, and unknown data, but its environment details were not released.Source: §5.2.1 · p. 20
• APT datasets model sequenced incidents; DAPT 2020 includes stealthy, heavily imbalanced attacks, while later datasets vary in duration, complexity, labeling, and documentation.The Unraveled expands DAPT 2020 with a more complex network, longer duration, defender responses, attack tracks, and different attacker types; PWNJUTSU is unlabeled and CyberFORCE is indirectly labeled with sparse documentation.Source: §5.2.1 · p. 20
• Several intrusion-specific datasets offer tailored data, including custom flow features for buffer overflow detection, honeypot SSH brute-force traffic, and HTTPS-based attacks.NDSec-1 is designed for salting by injecting its traffic into other datasets, while HIKARI-2021 uses CICFlowMeter and Zeek features.Source: §5.2.1 · pp. 20–21

5.2.2. Environment-Specific

• Environment-specific datasets cover SDN, wireless, cloud, and VoIP networks, as well as IoT botnets and broader IoT attacks.IoT datasets include controlled botnet testbeds and scenarios such as scanning, denial of service, information theft, ransomware, and man-in-the-middle attacks.Source: §5.2.2 · p. 21
• Industrial datasets span small SCADA testbeds, device-agnostic IIoT emulation, and ICS scenarios that include replay, DDoS, and man-in-the-middle attacks.Westermo uses three capture points, making it suitable for federated learning; Edge-IIoTset offers both raw and pre-extracted data, and ICS-Flow provides custom flow features.Source: §5.2.2 · p. 21

5.3. Datasets Beyond This Survey

• The survey excludes unavailable datasets and points readers to repositories and lists as complementary sources, including Kaggle, Zenodo, CAIDA Data, and NETRESEC.Specialized repositories may restrict uploads, and some sources restrict access by location.Source: §5.3 · p. 21, §5.3.1 · p. 22, §5.3.2 · p. 22

5.4. Dataset Popularity

• Most surveyed NIDS studies use custom, often non-shared data; CIC-IDS2017 is the most-used public dataset, appearing in only 29% of papers.Most users rely on its original version despite known errors, while public dataset use is relatively evenly distributed.Source: §5.4 · pp. 22–23

5.5.1. Increase in Number While Narrowing the Scope

• 51 of 89 surveyed datasets were collected since 2020, while 83% of datasets published since 2016 are special-purpose.IoT is the most common environment-specific focus, present in 15% of datasets, or 20% when IIoT and industrial networks are included; botnet and DoS/DDoS are also common targets.Source: §5.5.1 · pp. 23–24, Figure 5 · p. 23

5.5.2. Traffic Generation Strategies

• Only 17% of datasets provide purely real normal traffic, while 71% rely solely on emulation and another 8% combine it with other traffic types.Real attacks appear in 19% of datasets overall and 13.7% of datasets since 2020; emulation is used for malicious traffic in 85% overall and 90% since 2020.Source: §5.5.2 · pp. 24–25
• Traffic generators can produce overly simplistic background activity that makes detection easier and may overestimate system capabilities; human-driven emulation is more realistic but more expensive.Other normal-traffic strategies include user profiles and excluding normal traffic, which can limit datasets to specific uses or require mixing in traffic from elsewhere.Source: §5.5.2 · p. 24
• Synthetic attacks can either oversample underrepresented classes or inject new attack classes, but realistic injection must account for interactions with background traffic.ID2T supports injection into raw PCAP files while accounting for potential packet drops or delays.Source: §5.5.2 · p. 25

5.5.3. Popular Formats and Feature Sets

• Flows appear in 60% of surveyed datasets, 87% of them bidirectional; CICFlowMeter is used by 31% of flow-based datasets despite feature incompatibilities and real-time concerns.No standard flow-feature set exists; although CICFlowMeter is widely adopted, one study found NetFlow v9 features outperform CIC-based features in attack detection.Source: §5.5.3 · p. 25

5.5.4. Other Observations

• 60% of datasets model networks under 100 hosts, 55% of recent datasets have discontinuous captures, and only 23% provide predefined train-test splits.The survey links small testbeds with cost efficiency and discontinuous captures with easier, more flexible collection; the limited availability of predefined splits leaves evaluation vulnerable to bias and selective reporting.Source: §5.5.4 · p. 25

6.1. Choosing a Dataset

• Define selection goals first, favor recent traffic, and assess quality through documentation and direct checks for errors such as incorrect timestamps and labels.The paper does not assess dataset quality or provide a standardized formal quality metric.Source: §6.1 · p. 26

6.2. Creating a Dataset

• The ten-step dataset-creation process spans preparation, collection, post-processing, and publication, and may need iteration for continual datasets.The process is ideally linear, but redesigning scenarios or repeating collection and processing can make it iterative.Source: §6.2 · p. 26, Figure 6 · p. 27

6.2.1. Preparation

• Set objectives, design realistic and varied scenarios, and configure the capture environment accordingly; insufficient traffic variability can make detection unrealistically easy.The objective should specify malicious traffic classes, environment type and size, and time span.Source: §6.2.1 · pp. 26–28

6.2.2. Data Collection

• Capture formats and sources should fit the dataset goals; multiple formats can support varied NIDS benchmarks, while feature choices should be practical and reproducible.Combining sources can require clock synchronization and event correlation, and the optimal feature set remains an open issue.Source: §6.2.2 · p. 28

6.2.3. Post-Processing

• Post-processing should provide direct labels, validate correctness and realism, and minimize cleaning changes because excessive anonymization can reduce realism.Validation can check capture errors, attack execution, feature extraction, processing artifacts, and labels.Source: §6.2.3 · p. 28

6.2.4. Publication

• Publish documentation and data alongside creation materials, use a static train-test split, and update datasets as traffic patterns change.Sharing raw packets as well as extracted features is desirable when possible.Source: §6.2.4 · pp. 28–29

6.3. Using a Dataset

• Avoid mixing datasets unless necessary, respect temporal order in evaluation, and check for artifacts that can inflate performance or make accuracy misleading.When merging is necessary, similar collection periods and networks and replay in a local environment can reduce potential bias; multiple metrics and repeated evaluations are also recommended.Source: §6.3 · pp. 29–30
• Start with simple models and evaluate across multiple datasets and runs, using several metrics suited to NID and class imbalance.Repeated runs should respect temporal dependencies.Source: §6.3 · pp. 29–30

7.1. Data Generation

• Only one in five surveyed datasets contains real traffic, and static real-world datasets can become outdated as network patterns evolve.The section calls for continual real-world data capture and annotation that preserves privacy and label accuracy; automated labeling in drifting environments remains an open problem.Source: §7.1 · p. 30
• Profile-driven benign traffic can lack real-world complexity, so research should compare realistic synthesis methods and combine emulated, synthetic, and real traffic.The section also calls for an extensible profile-based generation framework and domain-specific validity conditions for synthetic traffic.Source: §7.1 · pp. 30–31
• Virtual testbeds can support timely, reproducible data, but most datasets lack setup files or customization details, motivating standardized replicable and extensible frameworks.AIT-LDSv2.0 emphasizes reproducibility, but its environment customizability is limited.Source: §7.1 · p. 31
• Many attack and IoT scenarios remain uncovered, while public concept-drift data and long-term datasets with cyclostationary patterns are lacking or limited.The section recommends scenario-specific coverage and continual, extensible datasets to keep pace with emerging threats.Source: §7.1 · p. 31

7.2. Data Processing

• Because most datasets are flow-based and nearly half lack packet data, research should explore packet-level, window-statistic, and cross-packet features and their fusion.These alternatives could help detect attacks missed by flow-based methods.Source: §7.2 · p. 31

7.3. Data Validation

• Existing quality measures and frameworks do not yet provide a practical, clear assessment of intrinsic dataset quality, while manual validation is time-consuming and expert-dependent.The section recommends formalized validity and quality frameworks that integrate expert knowledge into (semi-)automated checks.Source: §7.3 · p. 32

7.4. Data Publishing

• Publication delays can make datasets outdated within months, and the section proposes centralized indexing to improve discovery, availability, and delivery speed.The authors report that no maintained repository or indexing system for network intrusion datasets is known to them.Source: §7.4 · p. 32
• Dataset documentation varies substantially, so guidelines, checklists, and templates could standardize metadata and improve credibility and usability.Some datasets provide only brief descriptions or lack documentation entirely.Source: §7.4 · p. 32

Comparison with related work

• Compared with Ring et al. (2019b), this study adds newer datasets and sections on data limitations and trends while refining some dataset properties.It removes data balance as a property because NID data are naturally imbalanced.Source: §8 · p. 32
• This study is the only SLR-based survey focused specifically on NID data and lists the highest number of public datasets, with detailed properties and transparent popularity analysis.Its popularity analysis is based exclusively on Tier 1 security conferences.Source: §8 · p. 32, Table 6 · p. 33
• Compared with Yang et al. (2022), Kenyon et al. (2020), and Hindy et al. (2020), this study covers more datasets through an SLR and emphasizes data properties and dataset popularity.The earlier studies respectively listed 52, compared 34, and examined 21 datasets; Hindy et al. focused more on threats and attack comparisons.Source: §8 · pp. 32–33

Scope

• The review includes datasets captured from networks but excludes datasets whose data primarily come from end-host devices or sensors, such as many ICS and IoT datasets.The paper points readers to specialized ICS and IoT dataset surveys for those areas.Source: §8 · p. 33

Conclusions

• The survey analyzes 89 NID datasets across 13 properties and identifies trends including more published datasets, greater use of emulation, and rising popularity of CICFlowMeter features.The comparative table is intended to aid selection for benchmarking new NIDS proposals.Source: §9 · p. 33
• DARPA-based datasets are no longer the most prevalent in contemporary state-of-the-art NIDS research; CIC-IDS2017 is frequently used, while researchers still often collect data that is not publicly shared.The authors state that this trend hinders reproducibility and validation.Source: §9 · p. 33
• Creating a perfect dataset is infeasible, and reliance on static datasets will lag behind evolving attacks and network trends.Continuously updated datasets are proposed as a step toward mitigating this timeliness issue.Source: §9 · p. 33

Appendix A. Survey Methodology in Detail

• Two researchers performed the SLR process, with the first author conducting it and the second author refining the protocol and validating a random 10% sample.The process covered the review protocol, resource search, filtering, and synthesis.Source: Appendix A · p. 41

Appendix A.1. Data Search and Filtering Process

• The search combined known datasets from existing surveys, an own literature search, and backward and forward reference searches.The survey sources contained 302 datasets with duplicates; the own search produced 1,477 papers with duplicates.Source: Appendix A.1 · p. 41
• The own search used title and keyword matches across SCOPUS, ACM Digital Library, IEEE Xplore, and Google Scholar, with publication years capped at 2023.The Google Scholar review was limited and not entirely replicable; omitting it would leave over a dozen datasets undiscovered.Source: Appendix A.1 · p. 41
• Relevance filtering retained novel network traces for cyberthreat intrusion or anomaly detection, published through 2023, while excluding unrelated host-based or non-network data.The criteria also excluded work presenting an IDS method using existing data or focusing on network traffic classification or noncyberthreat anomaly detection.Source: Appendix A.1 · p. 41
• Using Google Scholar, backward and forward citation searches covered papers from 2018–2023; applying the same filtering criteria as the previous step added 25 papers, bringing the total to 138.Source: Appendix A.1 · pp. 41–42
• Quality review used whether a dataset was useful to potential users as its premise, excluding datasets that were not public, heavily pre-processed, or auxiliary to other research.The authors used this approach because many resources lacked the format or experimental results needed for common research-quality assessments.Source: Appendix A.1 · p. 42
• 89 datasets remained after relevance filtering and quality checks.A dataset was excluded if it failed any quality criterion.Source: Appendix A.1 · p. 42

Appendix A.2. Data Popularity Determination

• Popularity analysis focused on network intrusion or anomaly detection research from Tier 1 and Tier 2 cybersecurity conferences.The venues were selected according to the MLSec group list and included 12 conferences.Source: Appendix A.2 · p. 42
• The analysis surveyed 12 conferences, including S&P, CCS, Security, NDSS, ACSAC, AsiaCCS, CSF, ESORICS, EuroS&P, PETS, RAID, and DIMVA.Source: Appendix A.2 · p. 42
• The review covered papers from 2020–2023, identifying datasets through manual proceedings review and full-text analysis.Source: Appendix A.2 · p. 42