Source-linked AI summary
Evaluation of Machine Learning Algorithms for Intrusion Detection System
Mohammad Almseidin, Maen Alzubi, Szilveszter Kovacs, Mouhammd Alkasassbeh
TL;DR
Changing attack techniques make reliable intrusion detection difficult, motivating evidence on machine-learning classifiers for IDS. The paper evaluates seven classifiers on the KDD dataset using several metrics, emphasizing false-negative and false-positive rates. Decision Table achieves the lowest false-negative rate, while Random Forest achieves the highest average accuracy rate.
Problem
Changing attack tools and techniques make implementing an accepted intrusion detection system challenging.
Method
The paper evaluates seven machine-learning classifiers on the KDD intrusion dataset using performance metrics focused on false-negative and false-positive rates.
Results
Random Forest achieves the highest accuracy rate at 93.77, while Decision Table achieves the lowest false-negative rate at 0.002.
Takeaways & Limitations
Decision Table offers the lowest false-negative rate, whereas Random Forest provides the highest accuracy rate among the evaluated classifiers.
Abstract
from arXiv · showhide
Intrusion detection system (IDS) is one of the implemented solutions against harmful attacks. Furthermore, attackers always keep changing their tools and techniques. However, implementing an accepted IDS system is also a challenging task. In this paper, several experiments have been performed and evaluated to assess various machine learning classifiers based on KDD intrusion dataset. It succeeded to compute several performance metrics in order to evaluate the selected classifiers. The focus was on false negative and false positive performance metrics in order to enhance the detection rate of the intrusion detection system. The implemented experiments demonstrated that the decision table classifier achieved the lowest value of false negative while the random forest classifier has achieved the highest average accuracy rate.
I. INTRODUCTION
The paper frames IDS as essential but difficult because attacks continually evolve. It therefore evaluates machine-learning classifiers for intrusion detection, emphasizing false-negative and false-positive rates.
- Attackers continually change their tools and techniques, making it difficult to detect all vulnerabilities with one fixed solution.
- IDS monitors network traffic and generates alerts when attacks appear, either for a specific host or across network traffic.
- Anomaly-based IDS can detect new intrusions but produces many false-positive alarms from normal packets.
- Misuse-based IDS uses attack signatures and avoids false alarms, but new attack signatures can pass through undetected.
- The paper treats attack detection as classification and evaluates J48, Random Forest, Random Tree, Decision Table, MLP, Naive Bayes, and Bayes Network classifiers.
II. RELEVANT WORKS TO THE KDD DATASET
Prior work uses the KDD dataset and machine-learning methods to evaluate intrusion detection, while reporting varying performance across attack types and objectives. This paper compares seven classifiers with particular attention to false negatives and false positives.
- Previous studies used KDD to test classifiers including Naive Bayes, multilayer perceptron, neural networks, genetic algorithms, and extended classifier systems.
- Neural networks were reported as unsuitable enough for R2L and U2R attacks but acceptably accurate for DOS and PROBE attacks.
- A genetic-algorithm methodology recorded 97% accuracy while aiming for maximum intrusion detection and minimum false-positive rates.
- For a separate DDoS dataset, MLP recorded the highest accuracy rate at 98.36%.
- The paper evaluates J48, Random Forest, Random Tree, Decision Table, MLP, Naive Bayes, and Bayes Network on KDD, focusing on false-negative and false-positive rates.
III. KDD DATASET PREPROCESSING AND ANALYSIS
The study imports and analyzes the KDD intrusion dataset to characterize its attacks and attributes. The dataset is highly unbalanced but provides extensive connection features, including expert-suggested attributes for intrusion detection.
- The KDD dataset contains 4,898,431 instances and 41 attributes and was imported into SQL Server 2008 for statistical analysis.
- 21 attack types are categorized into four groups, with DOS attacks comprising 79% and normal packets 19% of the dataset.
- The dataset is unbalanced while containing 41 packet attributes.
- The attributes include basic TCP/IP connection information collected from network connections.
- Thirty-two expert-suggested attributes are included to help understand DOS, R2L, U2R, and PROBE attack behavior.
IV. BRIEF OVERVIEW OF MACHINE LEARNING CLASSIFIERS
The paper uses supervised machine learning because the imported KDD dataset has predefined classes. It reviews several classifier families, including neural, tree-based, probabilistic, and rule-based methods.
- The classifiers are supervised because the imported KDD dataset includes predefined classes.
- Multi-layer Perceptron (MLP) Classifier: MLP supports classification and regression, with short testing but typically long training, and can use sigmoid, linear, or hyperbolic transfer functions.
- Random Tree Classifier: Random Tree constructs a finite group of decision trees from randomly selected dataset attributes and chooses the class receiving the most tree votes.
- Random Forest Classifier: Random Forest uses a forest of classification trees and is described as able to handle noisy dataset values without re-modification during classification.
- J48 Classifier: J48 improves C4.5 decision-tree implementation while balancing computation time and accuracy.
- Naive Bayes Classifier: Naive Bayes classifies using conditional probabilities for classes and attributes and can process discrete and continuous attributes.
- Decision Table Classifier: Decision Table builds a lookup table of conditions and expected actions to identify the predicted output class.
V. TRAINING MODELS DATASET EXPERIMENTS
The study constructed and compared training models using a heavily imbalanced KDD dataset, then prepared the selected classifiers for comprehensive efficiency evaluation.
- Training dataset: 148753 extracted KDD records formed the training data.The dataset contains 21 attack types grouped into DOS, R2L, U2R, and PROBE categories.
- Training models: WEKA experiments used J48, Random Forest, Random Tree, Decision Table, MLP, Naive Bayes, and Bayes Network classifiers.The experiments were conducted on Ubuntu 13.10 with an Intel Core i5-4210U CPU and 6 GB RAM.
- Training models: All selected classifiers produced training models that were prepared and compared in a comprehensive efficiency study.The passage describes comparison across the studied machine learning models.
VI. MACHINE LEARNING CLASSIFIERS EXPERIMENTS, RESULTS AND DISCUSSION
Experiments on 60,000 randomized KDD records compared machine-learning classifiers using confusion-matrix metrics, with particular attention to false positives and false negatives. Random forest led overall accuracy, while decision table minimized false negatives but incurred the highest false-positive rate.
- Testing and metrics: 60,000 randomized KDD records were tested, and confusion matrices supported evaluation of accuracy, precision, false-negative, false-positive, true-negative, and true-positive rates.The testing data included all 21 KDD attack types.
- False-negative and false-positive results: 0.002 was the decision table classifier’s lowest false-negative rate, whereas random tree recorded the highest at 0.093.False negatives represent attacks classified as normal packets.
- False-negative and false-positive results: 0.073 was the decision table classifier’s highest false-positive rate, indicating normal packets classified as attacks.The decision table therefore traded fewer missed attacks for more false alarms.
- Training time: Random tree built its training model fastest, while MLP required 176 minutes, the longest reported training time.All selected classifiers except MLP built their models within accepted periods.
- Accuracy and error results: 90.73 was the random tree classifier’s lowest average accuracy rate, alongside the smallest ROC value.Bayes network recorded the highest ROC value of 0.999, while random tree recorded the lowest at 0.953.
VII. CONCLUSIONS
The experiments evaluated seven machine learning classifiers on the KDD intrusion detection dataset using multiple performance metrics. Results showed complementary strengths rather than one universally best classifier.
- Seven classifiers were evaluated on the KDD intrusion detection dataset through experiments.The classifiers were J48, Random Forest, Random Tree, Decision Table, MLP, Naive Bayes, and Bayes Network.
- 60000 randomly selected records were used in testing, with accuracy, precision, false negative, false positive, true negative, and true positive metrics computed.
- Random forest achieved the highest accuracy rate at 93.77%, while decision table achieved the lowest false negative value at 0.002.Random forest also had the smallest RMSE and false positive rate, but was not strongest on false negatives.
- No single machine learning algorithm efficiently handled all attack types.The conclusion emphasizes that true positive and average accuracy rates alone are insufficient; both false negative and false positive rates must also be considered.