PaperlayerView exact revision on arXiv

Source-linked AI summary

Survey of Network Intrusion Detection Methods from the Perspective of the Knowledge Discovery in Databases Process

Borja Molina-Coronado, Usue Mori, Alexander Mendiburu, José Miguel-Alonso

arXiv:2001.09697v1cs.CRcs.LGcs.NI

TL;DR

Evolving attacks, diverse traffic, growing data volumes, and encryption make network intrusion detection difficult, while data-mining algorithms alone do not address network-data particularities. This survey reviews NIDS methods across the KDD process and finds that evaluations often rely on accuracy over old-fashioned datasets, limiting their usefulness for current attacks.

  • Problem

    Evolving attacks and traffic, growing data volumes, encryption, and near-real-time demands complicate NIDS, while mining algorithms cannot be applied directly to network data.

    Source: §1 · p. 2

  • Method

    The survey reviews NIDS proposals through the KDD process, covering data techniques, mining approaches, model updates, and evaluation procedures.

    Source: §1 · p. 3

  • Results

    Many NIDS proposals are evaluated only for detection accuracy on old-fashioned databases, an approach the survey describes as of limited usefulness for current attacks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    NIDS assessment should consider properties beyond accuracy, including complexity, adaptation ability, and resistance to attacks against the detector itself.

    Source: §9.5 · p. 28

    Public datasets often have short, intermittent captures and may omit current traffic patterns, including encrypted traffic, constraining evaluation of model adaptation and relevance to current networks.

    Source: §4.3 · p. 11

Abstract

from arXiv · show

The identification of cyberattacks which target information and communication systems has been a focus of the research community for years. Network intrusion detection is a complex problem which presents a diverse number of challenges. Many attacks currently remain undetected, while newer ones emerge due to the proliferation of connected devices and the evolution of communication technology. In this survey, we review the methods that have been applied to network data with the purpose of developing an intrusion detector, but contrary to previous reviews in the area, we analyze them from the perspective of the Knowledge Discovery in Databases (KDD) process. As such, we discuss the techniques used for the capture, preparation and transformation of the data, as well as, the data mining and evaluation methods. In addition, we also present the characteristics and motivations behind the use of each of these techniques and propose more adequate and up-to-date taxonomies and definitions for intrusion detectors based on the terminology used in the area of data mining and KDD. Special importance is given to the evaluation procedures followed to assess the different detectors, discussing their applicability in current real networks. Finally, as a result of this literature review, we investigate some open issues which will need to be considered for further research in the area of network security.

1 Introduction

Network intrusion detection is difficult because attacks and traffic evolve, data volumes grow, encryption limits visibility, and detection must often be near real time. This survey uses the KDD process to review NIDS methods across data capture, preparation, mining, and evaluation, addressing aspects often neglected by prior surveys.

1 Introduction

  • NIDS faces evolving attacks and traffic, growing data volumes, encrypted traffic inaccessible to detectors, and near-real-time response demands.These combined challenges make intrusion detection a very hard problem.Source: §1 · p. 2
  • Off-the-shelf data-mining algorithms cannot be applied directly to network data; capture, preparation, and preprocessing are needed before detectors can deliver meaningful results.The survey also examines evaluation practices, noting that accuracy on old-fashioned databases has limited usefulness for current attacks.Source: §1 · pp. 2–3
  • Earlier surveys largely focus on detection algorithms, often omit data acquisition and curation or model updates, and may be restricted to particular detection methods or input types.The authors argue a broader view is needed to understand the building blocks of operational NIDSs.Source: §1 · pp. 2–3
  • The survey reviews NIDS proposals through the KDD process, covering data acquisition, preprocessing, transformation, mining, evaluation, and research challenges.It also discusses motivations, assumptions, and limitations of techniques, and mechanisms for updating models as traffic and attacks evolve.Source: §1 · p. 3

2 Basic Concepts

The section establishes how attacks manifest in network traffic and introduces KDD as an iterative process for deriving and evaluating useful patterns. It emphasizes that detection depends on both the traffic information captured and the methods applied to it.

2.1 Networks Under Attack

  • A network attack negatively affects confidentiality, integrity, or availability, and its network attack vector makes traffic analysis relevant to detection.The section classifies attack manifestations as point, contextual, or collective, depending on whether maliciousness appears in one sample, its context, or a set of events.Source: §2.1 · pp. 3–4

2 Basic Concepts

  • Different attack manifestations require different traffic views; detection depends on the captured and processed information as well as the data-mining algorithm.The section introduces cyberattacks and KDD concepts as foundations for understanding NIDS operation.Source: §2 · p. 3, §2.1 · p. 4

2.2 The Knowledge Discovery in Databases Process

  • KDD iteratively collects data, preprocesses features, reduces data, mines for patterns, and interprets and evaluates results.Steps may be revisited; for example, a previously excluded feature can be added during transformation to improve mining performance.Source: §2.2 · pp. 4–5

3 Data Collection Stage

NIDS data collection choices determine which traffic is visible, how much data must be processed, and which attacks can be detected. The section contrasts raw packets with more compact flow records and describes their respective capture characteristics.

3 Data Collection Stage

  • Capture location trades network coverage against traffic volume, throughput demands, and sensitivity to perturbations.Root-level capture covers many devices and Internet-related traffic but may miss horizontal attacks; lower-level capture is more segment-limited and requires more modest throughput.Source: §3 · p. 5
  • Distributed partial captures at several network points can be gathered centrally to address drawbacks of using root- or lower-level capture alone.Probes may supply full packet copies or traffic summaries such as per-flow samples, the latter providing a more limited view.Source: §3 · pp. 5–6

3.1 Capturing Raw Packet Data

  • Raw packets contain headers from multiple protocol layers and payloads, with fields and positions varying across protocols and messages sometimes split among packets.Sniffers can capture packets locally or through mirrored network-device ports, but monitoring performance is challenged by growing device counts and network speeds.Source: §3.1 · p. 6

3.2 Capturing Flow-level Data

  • A flow groups related packets by shared attributes and may be unidirectional or bidirectional, with the latter also called a session.Flow records can be derived from packets or computed and exported by network devices, often through IPFIX-related mechanisms and sampling.Source: §3.2 · pp. 6–7
  • Flow records include timing, byte and packet counts, and TCP flags; payload is optional and usually limited, so flow captures can be more compact than raw packet captures.The packet-encapsulation figure notes that higher-level protocol information is carried in lower-level payloads, while application and transport headers vary by protocol.Source: §3.2 · p. 7, Figure 2 · p. 7

4 Preprocessing Stage: Derivation of Features

Feature derivation turns captured traffic into structured records whose contents shape what a detector can find. The section surveys packet-, payload-, flow-, and dataset-level choices, including their operational trade-offs.

4.1 Derivation of Explanatory Variables

  • Detection capability depends heavily on the explanatory variables used to form structured traffic records.The section covers feature types, their derivation from captured data, and their usefulness for detecting attacks.Source: §4.1 · p. 7, §4 · p. 7

4.1.1 Packet Header Features

  • Packet headers expose protocol behavior and low-level attacks, but high throughput and encryption constrain real-time extraction and can make higher-level data more practical.A common feature vector may require missing values or dropping non-common fields; separate protocol-specific detectors are one proposed alternative.Source: §4.1.1 · p. 8

4.1.2 Payload Features

  • Payload features can reveal application-level attacks without distinctive header footprints, but require packet reassembly and substantial processing, storage, and access to plaintext.Character- and bigram-frequency vectors are common, and their length may require later dimensionality reduction; accessing encrypted contents is described as unrealistic in practice.Source: §4.1.2 · pp. 8–9

4.1.3 Single Flow Derived Features (SFD)

  • SFD records summarize flows without payload, enabling near-real-time operation on high-speed networks but limiting detection to attacks reflected in flow statistics or context.Some papers report many false alarms when using SFD features to detect attacks visible in payload.Source: §4.1.3 · p. 9

4.1.4 Multiple Flow Derived Features (MFD)

  • MFD features aggregate flow records into higher-level statistics that can reveal activity spread across connections, but add processing and memory costs and can hinder source tracing.Examples include byte, packet, and flow counts, entropy, covariance matrices, and wavelet-derived features.Source: §4.1.4 · pp. 9–10

4.2 Class Labeling: Derivation of Response Variables

  • Labels identify records as harmless or malicious, and may specify attack type; they support evaluation and are required to train some supervised methods.Manual labeling is expert-intensive but reliable; external tools can err, while scheduled attacks control labeling cost but produce less realistic traffic.Source: §4.2 · p. 10

4.3 Publicly Available Datasets for NIDS Research

  • Public datasets offer a common yardstick, but most provide preprocessed records rather than raw captures, limiting researchers’ ability to derive alternative features.Synthetic datasets may poorly reflect real network size and user variability; short captures, outdated traffic, and omitted encrypted traffic also constrain representativeness, while older DARPA derivatives remain in use despite criticism.Source: §4.3 · pp. 10–11

5 Preprocessing Stage: Data Cleaning and Feature Transformation

This section surveys cleaning and transformation methods used to make heterogeneous, noisy network records suitable for data mining. It also notes that sparse reporting of these procedures makes research harder to reproduce.

5 Preprocessing Stage: Data Cleaning and Feature Transformation

  • Network records mix categorical and numerical features and may contain noise, so preprocessing derives, modifies, or removes features and samples to build cleaner datasets.The authors note that NIDS papers often omit procedural details, hindering analysis and reproducibility.Source: §5 · p. 11

5.1 Noise Reduction

  • Noise reduction removes samples with extreme or missing values, but few NIDS papers explicitly describe their methods.Examples include Mahalanobis-distance removal of the 0.5% largest-ranked samples and feature-specific intervals based on 10% of the maximum-to-minimum range.Source: §5.1 · pp. 11–12

5.2 Transformation from Categorical to Numerical

  • Number encoding preserves dimensionality but can impose misleading numeric order on categories; one-hot and frequency encoding provide alternatives used in NIDS.One-hot encoding adds a binary feature per category, while frequency encoding weights values by their appearance frequency.Source: §5.2 · p. 12

5.3 Feature Scaling

  • Feature scaling can improve optimization and distance-based methods, but requires value ranges and may compress distinctions among narrow-range features when outliers occur.Reported approaches include mean normalization, logarithmic transformation, min-max normalization, and standardization.Source: §5.3 · p. 12

5.4 Feature Discretization

  • Discretization replaces continuous values with interval identifiers, potentially speeding learning and reducing noise at the cost of information loss.Equal-width, equal-frequency, and K-means methods require user choices of k; EMD and PKID avoid setting the interval count, and online methods approximate changing ranges.Source: §5.4 · p. 13

6 Data Reduction

Data reduction addresses large datasets and feature sets that can hinder learning through irrelevant or redundant variables. The section reviews feature and sample reduction methods, along with their trade-offs.

6 Data Reduction

  • The full traffic-derived feature set can be ineffective because irrelevant and redundant variables hinder data-mining algorithms; reduction aims to lower computation cost and improve detector accuracy.The survey covers reducing variables as well as reducing records.Source: §6 · pp. 13–14

6.1 Feature Dimensionality Reduction

  • Feature dimensionality reduction either removes explanatory variables or projects them into a lower-dimensional space.Both approaches seek to discard useless information or reduce redundancy.Source: §6.1 · p. 14

6.1.1 Manual Feature Removal

  • Manual feature removal relies on domain expertise or selected criteria, but dropping unsupported categorical variables can lose important information.Examples include removing constant features, retaining only MFD statistics for DoS detection, and excluding identifiers tied to a simulated environment.Source: §6.1.1 · p. 14

6.1.2 Feature Subset Selection

  • Feature subset selection chooses relevant variables to reduce model cost, help prevent overfitting, and remove redundancies; wrappers are costly, while filters are cheaper and model-independent.Embedded methods perform feature relevance ranking within the prediction algorithm’s training process.Source: §6.1.2 · pp. 14–15

6.1.3 Feature Projection

  • Feature projection compresses original variables into a lower-dimensional representation, but obscures original traffic values and therefore limits explainability.PCA captures linear components but not complex nonlinear relations; auto-encoders can model nonlinear relations, while ICA extracts statistically independent components.Source: §6.1.3 · pp. 15–16

6.2 Sample Dimensionality Reduction

  • Sample reduction can accelerate training by removing duplicate or similar records or replacing the dataset with representative samples such as cluster centroids.NIDS studies also use representative samples to reduce the cost of distance calculations.Source: §6.2 · p. 16

7 Data Mining Stage

This section surveys data-mining approaches for network intrusion detection, organizing detectors by goal and learning approach and covering misuse, anomaly, hybrid, batch, and incremental methods. It also addresses detector definitions and the challenges and operational constraints of adapting models to evolving network traffic.

7 Data Mining Stage

  • NIDS methods are classified by detection goal—misuse-based, anomaly-based, or hybrid—and by whether they are trained in batch or incrementally.This taxonomy differs from surveys that categorize proposals chiefly by algorithm type.Source: §7 · p. 16

7.1 Misuse-based, Anomaly-based and Hybrid Detectors

  • Inconsistent definitions have led surveys to classify some detectors differently and to mislabel classifiers as anomaly-based despite limits on detecting new traffic classes.The survey proposes updated definitions grounded in data-mining concepts; combining characteristics of misuse and anomaly detection yields a hybrid detector.Source: §7.1 · p. 17

7.1.1 Misuse-based NIDS

  • Misuse classifiers can model known attacks accurately with few false alarms, but struggle with zero-day or mutated attacks and require costly labeled training data.Because they recognize only what was learned, they need ongoing updates as new attack patterns appear.Source: §7.1.1 · p. 18

7.1.2 Anomaly-based NIDS

  • Anomaly detectors can identify unknown attacks without requiring attack labels, but often raise false alarms on benign changes and may need methods to explain or categorize alarms.Their normal-traffic profiles can be built from unlabeled or partially labeled data.Source: §7.1.2 · p. 19
  • Unsupervised anomaly methods flag infrequent patterns or transitions, correlation and PCA deviations, prediction errors, or observations distant from dense normal clusters.Frequency methods can model temporal relations but may perform poorly when normal traffic types are imbalanced; time-series methods require numerical data and can be difficult with non-stationary data.Source: §7.1.2 · pp. 19–20

7.1.3 Hybrid NIDS

  • Hybrid detectors combine misuse and anomaly methods to improve detection or control false alarms, at the cost of greater complexity.Parallel detectors can increase false positives; chained designs can prioritize either finding unusual patterns or reducing alarms, while one described two-stage scheme analyzes every record twice.Source: §7.1.3 · p. 21

7.2 Batch vs. Incremental Learning

  • Because network traffic evolves, the survey compares batch and incremental learning by their practical usability and ability to adapt to previously unseen patterns.Flexibility and adaptability matter, but the literature rarely discusses them.Source: §7.2 · pp. 21–22

7.2.1 Batch learning NIDS

  • Batch models are trained on static data and can be effective at deployment, but are costly in time and memory and do not evolve with changing traffic.They assume a stationary data distribution, limiting their ability to detect new or mutated attacks and new kinds of normal traffic; retraining requires data selection and storage, and supervised methods also require labeling.Source: §7.2.1 · p. 22

7.2.2 Incremental learning NIDS

  • Incremental NIDS update models as continuous data arrives, adapting gradually to changing networks at lower cost than retraining but with online resource and convergence constraints.Early accuracy may be poor, and continuous adaptation can increase sensitivity to noise or evasion perturbations; additional evaluation should monitor learning before deployment.Source: §7.2.2 · p. 23
  • The most common unsupervised stream methods flag incoming points that differ from prior observations, often using compact, incrementally updated summaries such as attribute-value frequencies.These methods typically process each point once and use small-memory representations that can be updated online.Source: §7.2.2 · p. 23
  • Incremental clustering NIDS detect observations far from or in sparse clusters, maintaining streaming summaries as windows, prototypes, microclusters, or grids.Some methods adapt cluster structures over time: uses bird-flocking optimization for merges and splits, while compares GMM microclusters with Kullback–Leibler divergence before merging.Source: §7.2.2 · p. 23
  • Less common unsupervised methods detect attacks as departures from learned stream evolution, using anomalous transitions between cluster assignments or LSTM-based forecasts.The transition method builds on incremental clustering [65]; the forecasting method flags observations far from predicted values.Source: §7.2.2 · p. 23
  • Rare supervised incremental NIDS require class labels during operation; examples update online AdaBoost weak classifiers or use incremental Very Fast Decision Trees.The tree method assumes classes do not evolve, and adds low-confidence records to capture new traffic classes.Source: §7.2.2 · pp. 23–24
  • To reduce early-model immaturity, some systems first train on offline data, then incrementally update the batch-trained model as new samples arrive.This provides a well-learned detector at deployment while avoiding the limitations of pure batch processing.Source: §7.2.2 · p. 24

8 Evaluation Stage

The survey focuses on predictive performance because it is readily measured and commonly used to compare NIDS proposals. It also notes that evaluation should cover deployment-relevant criteria, although comprehensive assessment of system complexity is beyond the survey’s scope.

8 Evaluation Stage

  • Evaluation ideally measures accuracy, complexity, adaptability, understandability, and security, but comprehensive complexity testing exceeds this survey’s scope.Algorithmic complexity captures only part of system cost; other KDD stages and implementation choices also affect processing ability.Source: §8 · p. 24
  • Predictive performance is readily quantified and used as a yardstick for comparing NIDS proposals, so the review focuses on that criterion.The survey notes that other evaluation characteristics are harder to assess with valid metrics and comparison criteria.Source: §8 · p. 24
  • The confusion matrix supports binary attack-versus-normal evaluation and one-vs-all evaluation for multiclass traffic; Table 6 lists metrics derived from it.Detection Rate, True Negative Rate, accuracy, and precision are interpreted as better when higher; False Positive Rate and False Negative Rate are better near zero.Source: §8 · p. 24, Table 6 · p. 25
  • ROC curves compare classifiers across decision thresholds by plotting the trade-off between True Positive Rate and False Positive Rate; F-measure is supplementary.ROC curves can help compare detectors, but similar curves may require additional metrics for clarification.Source: §8 · p. 25
  • Accuracy may misrepresent performance in imbalanced network traffic, where attacks are rare and missed attacks can severely affect victims.The survey therefore does not recommend accuracy for this setting, although several surveyed articles report it.Source: §8 · p. 25
  • Detection Rate (DR) and False Positive Rate (FPR) are the most common metrics; good NIDSs maximize DR and minimize FPR.DR measures detection of abnormal behavior as attacks, while FPR measures normal observations incorrectly flagged as attacks.Source: §8 · p. 25

9 Discussion, Conclusions and Open Issues

The review maps NIDS methods across KDD phases and finds recurring gaps in pipeline reporting, realistic data, adaptation, and evaluation. It identifies open issues for research across the KDD process.

9 Discussion, Conclusions and Open Issues

  • Main KDD paths often include “ND” because papers omit data-method details or apply algorithms directly to cooked public-dataset records.Figure 7 maps KDD phases and methods, with edge thickness showing the proportion of surveyed works using a method.Source: §9 · p. 25, Figure 7 · p. 26

9.1 Reproducible research

  • Most papers insufficiently report feature derivation, transformation, and validation, hindering analysis, evaluation, and comparison.The review recommends detailing the whole NIDS-building procedure to support reproducible research.Source: §9.1 · pp. 26–27

9.2 Data obtention, feature extraction and public datasets

  • Most NIDSs use public datasets for construction and evaluation, but success on one does not guarantee real-world performance; evaluation should use diverse datasets and network views.The review notes dataset limitations including short or discontinued captures, uncertain labeling, and limited realism.Source: §9.2 · p. 27
  • Public datasets’ fixed derived features may miss newer attacks or include features unavailable in real scenarios, while payload features are difficult to obtain in high-speed networks.Host probes may provide access to decrypted data, but increase network traffic and reduce some early-detection capabilities.Source: §9.2 · p. 27

9.3 Feature transformation and dimensionality reduction

  • Feature scaling and categorical-to-numerical transformations are more common in misuse systems; changing traffic can make static preprocessing choices obsolete.The review calls for further exploration of dynamic preprocessing and transformation techniques.Source: §9.3 · pp. 27–28

9.4 Data mining algorithms and learning schemes

  • The proposed taxonomy distinguishes detection approach from learning approach, addressing misuse of “anomaly detection” for supervised classifiers.Misuse systems mainly use supervised classifiers, whereas anomaly detection mainly uses unsupervised or weakly supervised methods.Source: §9.4 · p. 28
  • Most reviewed works use batch learning despite evolving traffic; incremental methods remain scarce and bring risks including labeling demands, false alarms, and evasion.The review also notes limited study of temporal dependencies, which could support earlier detection before an attack is underway.Source: §9.4 · p. 28

9.5 NIDS evaluation beyond accuracy

  • NIDS evaluation often omits suitable metrics and properties beyond accuracy, including complexity, adaptation, and resistance to attacks on the detector.Operational costs can arise in preprocessing and transformation as well as detection; security assessments are infrequent, and some reported protections are limited or unevaluated.Source: §9.5 · pp. 28–29
Loading 2001.09697v1…