Source-linked AI summary
Application of Data Mining to Network Intrusion Detection: Classifier Selection Model
Huy Nguyen, Deokjai Choi
TL;DR
The paper asks whether intrusion-detection classifiers should be selected by attack category rather than applied uniformly. It compares a broad classifier set on KDD99, selects category-specific algorithms, and reports improved detection performance from two selection models, while noting dataset and deployment limitations.
Problem
Prior intrusion-detection studies commonly applied one algorithm across attack categories, despite evidence that category-specific algorithms may perform differently.
Method
The authors compare a comprehensive classifier set on KDD99 and use the results to construct two classifier-selection models.
Results
The two proposed models produced minor TP improvements for DoS and Probe, significant TP improvements for U2R and R2L, and reasonably small FP across categories.
Takeaways & Limitations
Selecting classifiers by attack category offers a paper-supported approach for improving intrusion-detection performance and supporting real-time application.
Takeaways & Limitations
The authors discourage further KDD99 use for developing intrusion-detection algorithms and identify deployment flexibility and resource requirements as practical concerns.
Abstract
from arXiv · showhide
As network attacks have increased in number and severity over the past few years, intrusion detection system (IDS) is increasingly becoming a critical component to secure the network. Due to large volumes of security audit data as well as complex and dynamic properties of intrusion behaviors, optimizing performance of IDS becomes an important open problem that is receiving more and more attention from the research community. The uncertainty to explore if certain algorithms perform better for certain attack classes constitutes the motivation for the reported herein. In this paper, we evaluate performance of a comprehensive set of classifier algorithms using KDD99 dataset. Based on evaluation results, best algorithms for each attack category is chosen and two classifier algorithm selection models are proposed. The simulation result comparison indicates that noticeable performance improvement and real-time intrusion detection can be achieved as we apply the proposed models to detect different kinds of network attacks.
1 Introduction
The paper addresses intrusion detection under growing data volumes and varied attack behaviors. It evaluates classifiers by attack category and proposes selection models based on those comparisons.
- The study targets the burden of analyzing large audit datasets while improving intrusion-detection performance against complex, dynamic attacks.
- Prior work lacked broad classifier comparisons and often used one algorithm across attack categories, producing poor performance in some cases.
- The authors evaluate a comprehensive classifier set on KDD99 across Probe, DoS, U2R, and R2L attacks.
- The paper proposes two classifier-selection models after identifying algorithms that perform best for particular attack categories.
2 Related Works on KDD99 Dataset
The related-work survey reports varied results from intrusion-detection methods evaluated on KDD data. It motivates category-specific algorithm selection because prior studies commonly applied one algorithm to all major attack categories.
- Earlier methods reported varied detection rates across DoS, Probe, U2R, and R2L, with some approaches omitting false-positive results or category details.
- The survey finds that most prior studies used a single algorithm across all four major attack categories, often with poor performance.
- The authors report that different algorithms may perform differently across attack categories, motivating their study.
3 Empirical Study
The empirical study compares ten widely used classifier algorithms for intrusion detection on KDD99. The selected algorithms span Bayesian, decision-tree, rule-based, and function-based approaches.
- Ten distinct classifiers were selected to represent Bayesian, decision-tree, rule-based, and function-based approaches.
3.1 Evaluation Setup
The evaluation uses Weka and KDD99 to compare classifiers on shared, sampled intrusion-detection data. KDD99 provides labeled connections grouped into four attack categories, while the experiments use separate training and testing samples.
- The experiments use Weka so the selected machine-learning techniques can be compared within a common implementation environment.
- KDD99 was chosen as a shareable baseline because live network data are difficult to quantify and release for independent comparison.
- The dataset represents connections with 41 features and groups attacks into DoS, Probe, U2R, and R2L categories.
- The authors sampled 49,596 training instances and 15,437 independent testing instances from KDD99.
- The setup includes tables describing connection-type distributions in the standard 10% training data and the actual classifier-evaluation training data.
3.2 Classifier Algorithms
The study evaluates several classifier families, including Bayesian, tree-based, and hybrid methods, as candidates for intrusion detection on the KDD dataset.
- BayesNet learns Bayesian networks using nominal attributes, globally replaced missing values, SimpleEstimator, and K2 search without ADTree.
- NaïveBayes represents probabilistic knowledge by assuming predictive attributes are conditionally independent given the class.
- J48 is Weka’s optimized implementation of Quinlan’s C4.5 decision-tree algorithm, evaluated with specified pruning and fold parameters.
- NBTree combines decision trees with NaïveBayes classifiers at tree leaves, potentially trading increased performance for reduced speed.
3.2.5 Decision Table
The section describes rule-based classifiers used in the experiment, emphasizing feature-subset evaluation for Decision Table and reduced-error pruning for RIPPER.
- Decision Table builds a majority classifier and evaluates feature subsets with best-first search and cross-validation.The experiment uses crossVal = 1, BestFirst search, and disables IBk assistance.
- RIPPER generates class rules incrementally and prunes them using reduced-error pruning through Weka’s JRip implementation.JRip uses three folds, minimum number two, two optimizations, seed one, and pruning enabled.
- OneR creates a one-level decision tree represented as rules that test a single attribute.The method is described as simple and inexpensive while often producing useful data-characterizing rules.
3.2.8 Multilayer Perceptron (MLP)
The experiments include multilayer perceptron, support vector, and nearest-neighbor classifiers with explicitly specified architectures or parameter settings.
- MLP uses a three-layer feed-forward neural network with input, hidden, and output layers.The selected settings are learningRate = 0.3, momentum = 0.2, randomSeed = 0, and validationThreshold = 20.
- SMO trains a support vector classifier using polynomial or Gaussian kernels and a specified parameter configuration.The reported settings include c = 1.0, epsilon = 1.0E-12, PolyKernel, and randomSeed = 1.
- LBk is a lazy k-nearest-neighbor classifier evaluated with k = 1 and linear nearest-neighbor search.Cross-validation is disabled and windowSize = 0.
3.3 Performance Comparison
The comparison evaluates ten classifiers using detection, false-alarm, accuracy, and training-time measures to identify algorithms suited to different attack categories.
- No single algorithm detects every attack category with both high detection probability and low false-alarm rate.The results therefore support using different classifiers for different network attack types.
- For DoS attacks, most algorithms achieve approximately 95% TP, while NaïveBayes reaches 79.2% TP.
- For Probe attacks, NaïveBayes records 94.8% FP, while BayesNet and Decision Table each record 83.8%.
- For U2R attacks, BayesNet and Decision Table are the leading classifiers with FP values of 30.3% and 32.8%, respectively.
4 Classifier Selection Model
The paper generalizes classifier comparisons into attack-category-specific selection models, including a parallel model and a real-time model that accounts for training time.
- Table 4 observations identify JRip for DoS and Probe, Decision Table for U2R, and OneR for R2L as the best category-specific algorithms.
- The classifier-selection model uses parallel classification to choose the algorithm suited to the attack category at hand.
- The real-time model selects low-training-time algorithms: J48 for DoS, BayesNet for Probe and U2R, and OneR for R2L.
5 Model Evaluation and Discussion
The proposed models improve TP over the KDD Cup Winner, especially for U2R and R2L, while keeping FP reasonably small, but deployment raises flexibility, resource, and comparison concerns.
- TP improvement was minor for DoS and Probe but significant for U2R and R2L, while FP remained reasonably small across all attack categories.
- Practical deployment may require hardcoding multiple algorithms, which makes updating classifiers difficult when a more representative dataset changes the best choices.
- Running multiple intrusion-detection algorithms at multi-gigabit ingress points may impose resource demands that affect overall network performance.
- The authors identify comparison with a multiple classifiers selection system as an additional unresolved evaluation issue.
6 Conclusion
The paper surveys and compares classifiers on KDD99, proposes category-specific selection models, and reports promise for performance improvement and real-time applications while acknowledging KDD99’s problems.
- The study combines a KDD99 literature survey with an extensive Weka comparison of popular classifier algorithms.
- It proposes two algorithm-selection models intended to improve performance and support real-time intrusion-detection applications.
- The authors acknowledge problems with KDD99, discourage its further use for developing intrusion-detection data-mining algorithms, and plan evaluation on another dataset.