Source-linked AI summary
Machine Learning Based Intrusion Detection Systems for IoT Applications
Abhishek Verma, Virender Ranga
TL;DR
IoT’s open, self-configuring nature creates exposure to cyber threats, while comparing classifiers across datasets remains difficult. The paper benchmarks supervised classifiers for anomaly-based DoS intrusion detection, evaluates validation and statistical methods, and finds performance varies by metric and validation scheme.
Problem
IoT’s open and self-configuring nature exposes applications to cyber threats, and classifier comparisons across datasets can be difficult to validate statistically.
Method
The study benchmarks ensemble and single machine-learning classifiers across CIDDS-001, UNSW-NB15, and NSL-KDD, using hold-out and repeated cross-fold validation with Friedman and Nemenyi tests.
Results
Classifier performance varies by metric and validation scheme: hold-out validation favors different classifiers across metrics, while 10f validation improves performance overall and changes the metric leaders.
Takeaways & Limitations
The results support testing multiple classifiers and using statistical assessment when developing anomaly-based IDSs for IoT applications.
Takeaways & Limitations
The study uses only supervised-learning classifiers, although unsupervised learning could also be used for intrusion detection.
Abstract
from arXiv · showhide
Internet of Things (IoT) and its applications are the most popular research areas at present. The characteristics of IoT on one side make it easily applicable to real-life applications, whereas on the other side expose it to cyber threats. Denial of Service (DoS) is one of the most catastrophic attacks against IoT. In this paper, we investigate the prospects of using machine learning classification algorithms for securing IoT against DoS attacks. A comprehensive study is carried on the classifiers which can advance the development of anomaly-based intrusion detection systems (IDSs). Performance assessment of classifiers is done in terms of prominent metrics and validation methods. Popular datasets CIDDS-001, UNSW-NB15, and NSL-KDD are used for benchmarking classifiers. Friedman and Nemenyi tests are employed to analyze the significant differences among classifiers statistically. In addition, Raspberry Pi is used to evaluate the response time of classifiers on IoT specific hardware. We also discuss a methodology for selecting the best classifier as per application requirements. The main goals of this study are to motivate IoT security researchers for developing IDSs using ensemble learning, and suggesting appropriate methods for statistical assessment of classifier's performance.
1 Introduction
IoT’s open and self-configuring nature creates security and privacy risks, with DoS attacks threatening service access and causing organizational losses. The paper evaluates machine-learning classifiers and ensembles for anomaly-based IoT intrusion detection using multiple datasets, validation methods, statistical tests, and Raspberry Pi response-time measurements.
- Motivation: IoT’s self-configuring and open nature makes it vulnerable to insider and outsider attackers that threaten security, privacy, and adoption.Attackers may seek personal information, monetary gains, or opportunities for eavesdropping.
- Motivation: DoS attacks can block legitimate service access, violate SLA terms, and cause substantial monetary losses for firms and organizations.
- Study objectives: The study assesses single classifiers and ensembles for IoT IDS using accuracy, specificity, sensitivity, false positive rate, and AUC.The evaluated methods include CART, MLP, RF, AB, XGB, GBM, and ETC.
- Study objectives: Classifiers are benchmarked on CIDDS-001, UNSW-NB15, and NSL-KDD using repeated hold-out and repeated cross-fold validation.
- Study objectives: Friedman and Nemenyi tests assess classifier significance and pairwise performance differences, while Raspberry Pi execution measures response time on IoT hardware.
2 Related Work
Prior IoT DoS defenses include specification-based, signature-based, and energy-consumption approaches. These works illustrate varied detection strategies and deployment targets for protecting IoT systems.
- Existing IoT DoS defenses: A specification-based IDS detects distributed DoS attacks against IoT middleware when incoming requests exceed a service threshold.The approach uses Learning Automata and targets the middleware layer rather than an individual device.
- Existing IoT DoS defenses: A signature-based IDS framework was evaluated with Scapy, but the reported work lacked simulation support and details on signature-database updates.
- Existing IoT DoS defenses: Another IDS tracks malicious nodes by analyzing energy consumption against models of normal energy use in mesh-routing networks.The passage indicates that the proposed system requires nodes to monitor energy-related behavior.
3 Classification Algorithms
The paper motivates testing multiple classifier families because no single learning algorithm performs best across all domains. It therefore compares ensemble methods with selected single classifiers for intrusion detection.
- Classifier selection: The no-free-lunch theorem motivates experimenting with different classifiers because no single learning algorithm universally performs best across domains.
- Classifier selection: The study compares ensemble methods including RF, AB, GBM, XGB, and ETC with the single classifiers CART and MLP.Single classifiers are included partly because ensembles can be prone to over-fitting when input features are numerous.
3.1 Classifier ensembles
The study uses diverse ensemble classifiers, including bagging, boosting, gradient boosting, regularized boosting, and randomized trees, with model-specific prediction and tuning strategies.
- Five ensemble families are examined: Random Forest, AdaBoost, Gradient Boosted Machine, Extreme Gradient Boosting, and Extremely Randomized Trees.
- Random Forest: Random Forest combines independently randomized decision trees and uses majority voting to improve accuracy while avoiding over-fitting.The assessment sets 500 estimators and maximum tree depth to 26.
- AdaBoost: AdaBoost sequentially reweights difficult or incorrectly classified instances so weak learners converge toward a strong learner.Each iteration adds a weighted weak learner while minimizing cumulative training error.
- Gradient Boosted Machine: GBM sequentially combines decision trees, using prior predictions to reduce errors while optimizing a differential loss function.Its tuned configuration uses 500 estimators, depth 3, minimum split size 100, and a 0.1 learning rate.
- Extreme Gradient Boosting: XGB extends GBM with regularization to control model complexity and improve generalization, while also supporting sparse, parallelized computation.The paper describes DMatrix-based sparsity-aware algorithms and low-memory, high-speed execution.
- Extremely Randomized Trees: ETC builds unpruned trees by randomly selecting features and cut-points, with K, nmin, and Tcount governing its construction.
3.2 Single classifiers
The study evaluates two single classifiers: CART, a recursively partitioning tree method, and MLP, a supervised feed-forward neural network that learns nonlinear mappings.
- The single-classifier comparison uses Classification and Regression Trees and Multi-Layer Perceptron.
- CART: CART recursively partitions training samples and fits a simple prediction model within each partition.It uses exhaustive search to identify splits that reduce child-node impurity and includes a mechanism for missing feature values.
- MLP: MLP is a supervised feed-forward neural network composed of input, middle, and output layers.It learns a nonlinear function approximator from input features to output classes.
- MLP: MLP transforms weighted sums through nonlinear activation functions before the output layer produces final predictions.
4 Experimental Design
The study benchmarks machine-learning classifiers for IoT intrusion detection across multiple datasets, validation methods, and performance metrics, while using statistical tests to compare classifiers. It combines dataset-based evaluation, hyperparameter tuning, and significance analysis to support robust classifier comparisons.
- Datasets: Three datasets—CIDDS-001, UNSW-NB15, and NSL-KDD—are used to validate classifiers for anomaly-based intrusion detection.CIDDS-001 and UNSW-NB15 are selected partly because they include recently generated or real traffic data.
- Hyperparameter tuning: Random search with cross-validated candidate settings is used to find optimal hyperparameters for the ensemble classifiers.RandomizedSearchCV from scikit-learn performs the hyperparameter search.
- Classifiers and metrics: Seven classifiers, including CART, MLP, Random Forest, AdaBoost, XGB, GBM, and extremely randomized trees, are assessed using prominent classification metrics.The metrics include accuracy, specificity, sensitivity, false positive rate, and AUC.
- Validation: Repeated hold-out and repeated 10-fold cross-validation are used to stabilize error estimation and reduce validation variance.Hold-out validation uses a 60:40 training-to-testing split.
- Statistical analysis: Friedman and Nemenyi tests assess whether classifier performances differ significantly and identify which classifier pairs account for differences.The study applies the statistical analysis to both hold-out and 10-fold validation results, using α values of 0.05 and 0.1.
5 Results and Analysis
Across four datasets, classifier performance varies by metric and validation method, with hold-out and 10f results showing different leaders and statistically significant differences. Response-time measurements on Raspberry Pi expose practical trade-offs for selecting an IoT IDS classifier.
- Hold-out validation: 94.94% accuracy and 91.6% specificity make RF the hold-out leader, while GBM leads sensitivity at 99.53% and XGB leads AUC at 98.76%.These are averages across CIDDS-001, UNSW-NB15, KDDTrain+, and KDDTest+ datasets.
- Hold-out validation: 8.89% FPR makes RF best under hold-out validation, whereas AB is worst at 13.26%.The comparison covers average FPR across four datasets.
- 10f validation: Under 10f validation, CART leads accuracy at 96.74%, AB specificity at 97.5%, RF and XGB sensitivity at 97.31%, and XGB AUC at 98.77%.The passage reports improved performance for all classifiers relative to hold-out validation.
- 10f validation: 3.78% FPR makes CART best under 10f validation, while RF is worst at 21.85%.These are average FPR values across the four evaluated datasets.
- Statistical assessment: Friedman tests find significant classifier differences across all considered metrics for hold-out validation, but only AUC for 10f validation.Both significance levels, α = 0.05 and α = 0.1, are reported for these assessments.
- Response time: On Raspberry Pi, CART has the shortest classification response time, while ETC is slowest for CIDDS-001 and KDDTest+; RF and XGB are nearly similar.GBM is the slowest on KDDTrain+ according to the reported experiment.
- Application-oriented selection: Classifier selection depends on application requirements: CART, RF, or XGB favor quick response, while accuracy and low FPR support several alternatives.The paper frames selection as a trade-off among accuracy, FPR, response time, and other performance measures.
- Scope boundary: The study evaluates only supervised-learning classifiers, which rely on known target values during training.The authors motivate this scope by using network traffic patterns for model training and prediction.
6 Conclusion
The study evaluates seven machine-learning classifiers for anomaly-based IoT intrusion detection against DoS attacks. It benchmarks their performance, statistical differences, and Raspberry Pi response times to identify suitable trade-offs.
- Seven classifiers are evaluated for anomaly-based intrusion detection against IoT DoS attacks.
- Performance is measured using accuracy, specificity, sensitivity, false positive rate, and AUC across CIDDS-001, UNSW-NB15, and NSL-KDD.
- Friedman and Nemenyi post-hoc tests assess significant differences among classifier performance results.
- Raspberry Pi experiments evaluate the classifiers' average response times on IoT-specific hardware.
- CART and XGB show the best trade-off between prominent metrics and response time.