PaperlayerView exact revision on arXiv

Source-linked AI summary

Performance Comparison of Intrusion Detection Systems and Application of Machine Learning to Snort System

Syed Ali Raza Shah, Biju Issac

arXiv:1710.04843v2cs.NIcs.CRcs.LG

TL;DR

The paper addresses the difficulty of detecting malicious traffic accurately while limiting false positives in open-source IDSs. It compares Snort and Suricata at 10 Gbps, then evaluates machine-learning enhancements for Snort. Suricata processes more traffic with fewer packet drops but uses more resources, while optimized SVM with a firefly algorithm achieves 8.6% FPR and 2.2% FNR.

  • Problem

    Snort detects known malicious traffic but triggers a high false positive rate, motivating an adaptive approach for improving detection accuracy.

  • Method

    The study compares Snort and Suricata at 10 Gbps and empirically evaluates machine-learning algorithms on three IDS datasets before integrating an adaptive plug-in with Snort.

  • Results

    Suricata processes more packets with a lower packet drop rate but uses more computational resources, while Snort shows higher detection accuracy and the optimized SVM achieves 8.6% FPR and 2.2% FNR.

  • Takeaways & Limitations

    The optimized SVM with firefly algorithm produced the best result among the evaluated approaches for reducing false positives and false negatives in the Snort plug-in.

  • Takeaways & Limitations

    The optimized SVM plug-in requires higher CPU and memory utilization, which the authors suggest addressing with better hardware.

Abstract

from arXiv · show

This study investigates the performance of two open source intrusion detection systems (IDSs) namely Snort and Suricata for accurately detecting the malicious traffic on computer networks. Snort and Suricata were installed on two different but identical computers and the performance was evaluated at 10 Gbps network speed. It was noted that Suricata could process a higher speed of network traffic than Snort with lower packet drop rate but it consumed higher computational resources. Snort had higher detection accuracy and was thus selected for further experiments. It was observed that the Snort triggered a high rate of false positive alarms. To solve this problem a Snort adaptive plug-in was developed. To select the best performing algorithm for Snort adaptive plug-in, an empirical study was carried out with different learning algorithms and Support Vector Machine (SVM) was selected. A hybrid version of SVM and Fuzzy logic produced a better detection accuracy. But the best result was achieved using an optimised SVM with firefly algorithm with FPR (false positive rate) as 8.6% and FNR (false negative rate) as 2.2%, which is a good result. The novelty of this work is the performance comparison of two IDSs at 10 Gbps and the application of hybrid and optimised machine learning algorithms to Snort.

1. INTRODUCTION

The introduction motivates accurate, scalable intrusion detection for high-speed networks and compares Snort and Suricata as open-source IDSs. It proposes machine-learning enhancements to reduce false positives and improve Snort’s detection accuracy.

  • Motivation: Accurate IDS detection is critical because attackers can hide malicious activity within large volumes of legitimate network traffic.The paper frames detection capability and false-alarm reduction as central requirements for network protection.
  • IDS Comparison: Snort and Suricata were selected because they offer comparable functions, rule sets, syntax, IPv6 support, and intrusion-prevention features.Suricata uses a multi-threaded architecture, whereas Snort uses a single-threaded architecture.
  • Performance Requirements: IDS performance depends on host CPU, memory utilization, and packet drop rate at the network interface.The paper links increasing network speed and malicious traffic with the need to inspect packets at high throughput.
  • Detection Limitations: Both IDSs primarily use predefined rules that trigger alarms when traffic matches known malicious patterns.The rule-based approach does not address unknown malicious traffic and can produce false positive alarms.
  • Study Aim: The study compares Snort and Suricata at 10 Gbps and applies collective and optimized machine-learning algorithms to Snort.The stated objectives include measuring performance and detection accuracy and reducing false positive alarms.

2. RELATED WORKS ON PERFORMANCE COMPARISON

Earlier studies compared Snort and Suricata across hardware platforms, traffic speeds, resource use, packet handling, and detection accuracy. Their findings generally associate Suricata with higher processing requirements and show that hardware can affect packet handling without necessarily improving detection capability.

  • Prior Comparisons: Prior work established that Snort–Suricata performance comparison is not new and commonly examines resource utilization and detection accuracy.The reviewed studies used different computer hosts, platforms, and experimental environments.
  • Resource Requirements: Suricata generally required more processing power than Snort, while higher processing power supported accurate malicious-traffic detection in one comparison.The reported findings were based on differing CPU, memory, and network-card specifications.
  • Snort Evaluations: A simulated three-IDS evaluation found that Snort used low computing resources but could negatively affect network traffic more than the other tested IDSs.The study used both physical and virtual computers.
  • High-Speed Processing: One real-network study found that Snort struggled with high-speed packet processing and dropped packets without accurately analysing them.The researchers introduced parallel IDS technology to reduce packet drops.
  • Hardware Effects: Better hardware improved Snort packet handling against DDoS but did not improve its detection capability.The evaluation compared three hardware configurations.

3. DESCRIPTION OF RESEARCH METHODOLOGY

The methodology uses repeatable quantitative experiments to compare Snort and Suricata at 10 Gbps across seven attack types. It measures resource use, packet handling, packet drops, and detection accuracy using controlled virtualized traffic generation.

  • Research Design: The study performs rigorous, repeatable, quantitative comparisons of both IDSs and evaluates machine-learning algorithms.The experiments were designed to make observations and take measurements.
  • Test Bed: The test bed compares Snort and Suricata at 10 Gbps using seven malicious-traffic types selected for consistent rule application.The attack types were also described as common and covering a good number of attacks.
  • Traffic and Rules: The experiments use default Snort and Suricata rule sets with combined legitimate and malicious traffic as input.The methodology asks how architecture, resource utilization, packet drops, and rule-set accuracy differ as traffic increases.
  • Experimental Infrastructure: Five virtual machines connected through a virtual switch and 10 Gbps Ethernet links generate and route traffic to high-performance Snort and Suricata VMs.The experiments used Snort v2.9.6.1 and Suricata v2.6.9.

4. EXPERIMENT SCENARIOS

The experiments compared Snort and Suricata under identical conditions, measuring resource use, packet processing, packet drops, and detection accuracy at high network speeds. Snort showed higher detection accuracy in the tested scenarios, while Suricata processed traffic faster with greater resource use.

  • Experiment setup: The test bed compared Snort and Suricata on identical virtual machines using identical rule sets and test conditions.The experiments used 10 Gbps Ethernet links and separate IDS installations.
  • Resource utilisation: 79.3% average CPU utilisation was recorded for Suricata, compared with 67.7% for Snort across the same traffic metrics.Suricata also used more memory, reaching 3.8 Gbytes at 10 Gbps versus Snort’s 3.2 Gbytes.
  • Packet processing: Suricata’s packet processing engine was faster than Snort’s when both processed 1,000,000 UDP, TCP, and ICMP packets over three four-hour periods.The comparison used variable network speeds and recorded CPU, memory, and processing behavior.
  • Packet drops: At 10 Gbps, Snort dropped more packets than Suricata, although CentOS 6.5 memory buffers contributed to packet loss for both systems.The buffer issue occurred at 10 Gbps but not at 100 Mbps or 1 Gbps.
  • Detection accuracy: Snort achieved higher detection accuracy than Suricata in legitimate-traffic testing, while Suricata produced a 39% FNR and 3% TPR.Snort produced 0% TPR and 0% FNR in that test and triggered fewer false positive alarms.
  • Detection accuracy: For mixed malicious traffic, Snort averaged 55.2% FPR and 6.7% FNR, while Suricata averaged 74.3% FPR and 16.7% FNR.Seven malicious traffic types were injected under default rule configurations.

5. SNORT IDS USING MACHINE LEARNING TECHNIQUES

The study selected machine-learning algorithms for use with Snort to improve classification of normal and malicious traffic and reduce false positive alarms.

  • Machine-learning integration: The study planned to combine Snort with machine-learning algorithms to reduce false positive alarms and improve performance.Candidate methods included SVM, Decision Trees, Fuzzy Logic, BayesNet, NaiveBayes, Genetic Algorithms, Neural Networks, and Sequential Minimal Optimization.
  • Machine-learning integration: Snort’s rule set was described as effective for known malicious traffic but limited when attack patterns were not represented in the rules.Machine learning was considered for classifying normal and malicious traffic beyond rule-based detection.
  • Machine-learning integration: The study sought the best-performing algorithms for conjunction with Snort, with emphasis on reducing false positive alarms.The selected algorithms were intended to improve Snort’s performance in traffic classification.

6. EVALUATION OF MACHINE LEARNING ALGORITHMS (MLAs)

The study empirically evaluated five machine-learning algorithms across three IDS datasets using Weka and cross-validation. SVM provided the strongest overall detection results, while Fuzzy Logic also performed well.

  • Evaluation design: Five high-performing algorithms were evaluated on three publicly available IDS datasets after data preprocessing and literature-based selection.Weka was used as the machine-learning and data-mining environment.
  • Evaluation design: The evaluation used 10-fold cross-validation, dividing each dataset into 90 labelled training data and 10 labelled testing data.The algorithms were assessed using three different datasets.
  • Algorithm comparison: SVM achieved superior Detection Rate and Detection Accuracy compared with Decision Trees, Fuzzy Logic, BayesNet, and NaiveBayes.Algorithms could be rejected when Detection Rate was poor despite favorable FPR or Detection Accuracy.
  • Algorithm comparison: 96% average DR, 3% average FPR, and 95% average DA were recorded for SVM across the three datasets.These averages were reported in the machine-learning output analysis.
  • Algorithm comparison: Fuzzy Logic achieved 93.8% average DR, 0.92% average FPR, and 93.4% average DA across the three datasets.Its results were better than those of Decision Tree, BayesNet, and NaiveBayes on the reported measures.

7. PROPOSED SNORT ADAPTIVE PLUG-IN

The proposed Snort adaptive plug-in was designed to address Snort’s high false positive rate by operating alongside its rule set and applying machine learning to traffic classification.

  • Motivation: Snort detected six of seven malicious traffic types but produced a 55.2% average false positive rate, motivating an adaptive plug-in.The plug-in architecture is shown in Figure 6.
  • Architecture: The adaptive plug-in operates in parallel with Snort’s rule set so it can address unknown or variant malicious traffic.The rule set handles known malicious traffic, while the plug-in receives preprocessed network traffic.
  • Architecture: The plug-in decodes network packets, classifies legitimate and malicious traffic, and applies a machine-learning algorithm.Decoded information includes addresses, ports, Ethernet-frame data, and packet size.
  • Implementation: The Snort v2.9 plug-in was implemented as a script that imports machine-learning algorithms and operates alongside Snort’s detection engine.A preprocessor sends decoded packets to the plug-in.
  • Algorithm selection: SVM, Fuzzy Logic, and Decision Tree required live background malicious-traffic testing to select the superior algorithm for the plug-in.The live experiment was intended to evaluate false positive and false negative alarm rates.

(a) Individual Algorithms

The study evaluated individual machine-learning algorithms as adaptive Snort plug-ins using multiple malicious-traffic experiments. SVM reduced false alarms and missed detections compared with Snort without the plug-in.

  • Individual algorithms: SVM, Fuzzy Logic, and Decision Tree plug-in performance was recorded in Table 12.The experiments injected seven types of malicious traffic into Snort for comparison.
  • Baseline: 55.2% FPR and 6.7% FNR were observed when Snort operated without the adaptive plug-in.The initial experiment used seven types of malicious traffic.
  • Individual algorithms: 16.9% FPR and 4.1% FNR were recorded for the SVM adaptive plug-in.The SVM plug-in was compared with Fuzzy Logic and Decision Tree plug-ins.

(b) Hybrid and Optimized Algorithms

The study compared hybrid SVM–Fuzzy Logic and firefly-optimized SVM plug-ins for Snort. Firefly optimization produced the best reported error rates, while requiring more CPU and memory.

  • Hybrid and optimized algorithms: The hybrid SVM–Fuzzy Logic and optimized firefly-SVM cases were evaluated as improved Snort plug-ins.Their FPR and FNR were recorded in Table 13.
  • Hybrid SVM and Fuzzy Logic: 13% FNR and 3.2% FNR were reported for the hybrid SVM with Fuzzy Logic approach.SVM first predicts two classes, after which fuzzy-logic decision rules generate the results.
  • Optimized SVM with firefly algorithm: 8.6% FPR and 2.2% FNR were achieved by the optimized SVM with firefly algorithm, the best reported result.The firefly algorithm optimized SVM parameters before classification.
  • Optimized SVM with firefly algorithm: The firefly algorithm selected SVM parameters by comparing fireflies and choosing the brightest location.The optimized parameters were C = 1.57 and γ = 0.58.
  • Resource utilization: Higher CPU and memory utilization made the optimized SVM plug-in more resource intensive.The study states that better memory and faster multi-core hardware could address this requirement.

8. CONCLUSION

At 10 Gbps, Suricata processed traffic faster with lower packet drops but used more computational resources, while Snort achieved higher detection accuracy. An adaptive Snort plug-in using machine learning reduced false positives, with SVM identified as the strongest evaluated algorithm.

  • IDS performance comparison: 82,223 packets/second was Suricata’s processing rate, compared with Snort’s 60,866 packets/second at 10 Gbps.Suricata’s higher throughput accompanied a minimum packet drop rate, whereas Snort used fewer computational resources.
  • IDS performance comparison: Suricata’s CPU utilisation was 10–15% higher than Snort’s, and its average memory use exceeded Snort’s by 600 Mbytes.Suricata used an average 3.8 Gbyte of memory because of its multi-threaded architecture and built-in detection stages.
  • Detection accuracy: Snort produced a 55.2% FPR and 6.7% FNR, compared with Suricata’s 74.3% FPR and 16.7% FNR.Both systems triggered high false-positive rates, but Snort had higher detection accuracy on the same malicious traffic.
  • Detection accuracy: Snort detected six of seven malicious-traffic types, while Suricata detected four, with each achieving a 99% true-positive rate for detected types.Suricata detected fewer types because it could not detect data-link-layer traffic.
  • Adaptive Snort plug-in: Support Vector Machine was the strongest of five evaluated learning algorithms for reducing Snort’s false-positive rate.The algorithms were compared across three datasets using Weka, and SVM achieved an average 96% detection rate with a low false-positive rate.

Author Biographies:

The authors are affiliated with Teesside University’s School of Computing and work in networking, network security, mobile communications, and AI-based applications.

  • Syed Ali Razah Shah completed an MSc at Teesside University and researches computer networks, network security, and AI-based applications.
  • Dr Biju Issac is a senior lecturer at Teesside University with research training in networking and mobile communications.He is also a Chartered Engineer and Senior Member of IEEE.
Loading 1710.04843v2…