Source-linked AI summary
An Experimental Analysis of Attack Classification Using Machine Learning in IoT Networks
Andrew Churcher, Rehmat Ullah, Jawad Ahmad, Sadaqat ur Rehman, Fawad Masood, Mandar Gogate, Fehaid Alqahtani, Boubakr Nour, William J. Buchanan
TL;DR
Resource-constrained IoT devices and limited security make intrusion detection difficult as attacks increase, while existing research lacks IoT and multi-class evidence. The paper compares machine-learning algorithms for binary and multi-class attack classification on Bot-IoT, finding that leading algorithms vary by classification setting and dataset weighting.
Problem
Resource-constrained IoT devices, overlooked security, and limited IoT and multi-class datasets make intrusion-detection research difficult and important.
Method
The paper surveys and compares state-of-the-art machine-learning algorithms using accuracy, precision, recall, F1 score, log loss, and other metrics for binary and multi-class Bot-IoT classification.
Results
Performance varied by setting: RF was best for accuracy and precision on the non-weighted dataset, ANN achieved higher binary-classification accuracy on weighted data, and KNN and ANN led specified multi-class settings.
Takeaways & Limitations
The results indicate that algorithm choice for IoT attack classification depends on the classification task and whether the dataset is weighted.
Takeaways & Limitations
Machine-learning intrusion detection can produce false positives that may render an IDS unusable by triggering alerts on normal data.
Abstract
from arXiv · showhide
In recent years, there has been a massive increase in the amount of Internet of Things (IoT) devices as well as the data generated by such devices. The participating devices in IoT networks can be problematic due to their resource-constrained nature, and integrating security on these devices is often overlooked. This has resulted in attackers having an increased incentive to target IoT devices. As the number of attacks possible on a network increases, it becomes more difficult for traditional intrusion detection systems (IDS) to cope with these attacks efficiently. In this paper, we highlight several machine learning (ML) methods such as k-nearest neighbour (KNN), support vector machine (SVM), decision tree (DT), naive Bayes (NB), random forest (RF), artificial neural network (ANN), and logistic regression (LR) that can be used in IDS. In this work, ML algorithms are compared for both binary and multi-class classification on Bot-IoT dataset. Based on several parameters such as accuracy, precision, recall, F1 score, and log loss, we experimentally compared the aforementioned ML algorithms. In the case of HTTP distributed denial-of-service (DDoS) attack, the accuracy of RF is 99%. Furthermore, other simulation results-based precision, recall, F1 score, and log loss metric reveal that RF outperforms on all types of attacks in binary classification. However, in multi-class classification, KNN outperforms other ML algorithms with an accuracy of 99%, which is 4% higher than RF.
1. Introduction
IoT networks face increasing automated attacks while resource-constrained devices make adequate security difficult. The paper therefore surveys and experimentally compares ML methods for binary and multi-class attack classification.
- Automated attack traffic increasingly targets IoT networks, whose limited processing power and bandwidth complicate adequate security.
- Machine learning can monitor network traffic for suspicious activity and support more autonomous, accurate intrusion detection systems.
- The paper identifies the best ML methods for IoT attack detection using both binary and multi-class classification.
- The study surveys ML-based IoT attack detection and compares algorithms using confusion matrices, accuracy, precision, recall, F1 score, log loss, ROC AUC, and CKC.
- The paper examines both binary-class and multi-class testing as part of its algorithm evaluation.
2. Background and Related Work
This section reviews IDS, ML-based attack detection, related work, candidate algorithms, and datasets to establish the design of the experiments.
- The section reviews intrusion detection systems and ML methods used for attack detection.
- Related work informs the selection of algorithms and datasets for testing the models.
- Each algorithm is examined for its suitability to the paper’s experimental design.
2.1. Intrusion Detection System
IDS can use signature-based or anomaly-based detection, with ML particularly suited to identifying deviations from normal traffic. However, ML-based IDS can generate false positives that undermine usability.
- An IDS monitors networks for potentially harmful traffic through signature-based or anomaly-based detection.
- Signature-based IDS compares incoming traffic with known attack signatures and cannot detect attacks without matching signatures.
- Anomaly-based IDS identifies traffic that differs from normal network behavior and is better suited to ML training.
- False positives from normal data can trigger alerts and render an ML-based IDS unusable.
2.2. IoT Intrusion Detection Using Machine Learning
This section reviews machine-learning approaches for IoT intrusion detection and identifies gaps in existing evaluation, especially for multi-class classification. It introduces several candidate algorithms and describes ANN processing components.
- Research gaps: Existing IoT intrusion-detection studies often evaluate limited algorithm sets, use outdated or unavailable datasets, or omit multi-class testing.DARPA-based research used datasets over 20 years old and did not perform multi-class testing; another Bot-IoT study also omitted multi-class evaluation.
- Machine-learning models: The reviewed algorithms include KNN, SVM, DT, NB, LR, RF, and ANN for machine-learning analysis of IoT attack data.The section presents these models as candidate methods for the analysis.
- Machine-learning models: KNN classifies data using nearby training points and assumes similar data points form groups.KNN is described as a supervised lazy learner because training data are used during prediction rather than through a separate training phase.
- Machine-learning models: SVM separates classes with a maximum-margin hyperplane and can transform inputs using linear, nonlinear, polynomial, Gaussian, RBF, or sigmoid kernels.The prediction uses the sign of the decision function to determine the class.
- Machine-learning models: Decision trees classify data through hierarchical attribute tests, while random forests aggregate predictions from trees trained on varied data and feature subsets.Randomized sampling and feature selection are described as mechanisms intended to lower variance and prevent overfitting.
- Artificial neural networks: ANNs use input, hidden, and output layers; training combines weighted inputs with biases, applies activations, and minimizes loss by changing weights and biases.The section identifies ReLU and Softmax as activation functions and describes gradient-based loss minimization.
2.3. Internet of Things Attacks
IoT attacks span data theft, denial of service, keylogging, and probing techniques that exploit device and network weaknesses. The section also motivates comprehensive ML evaluation for realistic IoT attack data.
- IoT attack context: IoT protocols target devices with limited computation, storage, communication, and battery resources, complicating security provision.Examples include ZigBee, RFID, and smart Bluetooth.
- Attack types: Data exfiltration attacks involve gaining private-network access and stealing stored information such as payment-card and personal data.Prior detection approaches include partially observable Markov decision processes and file-system metadata capture.
- Attack types: DoS uses one attacking system and connection, whereas DDoS uses multiple systems and connections, typically through botnets.HTTP floods, TCP SYN attacks, and UDP floods are identified as protocol-dependent variants.
- Attack types: The Mirai botnet reportedly used up to 400,000 devices to disrupt major websites, while prior DoS/DDoS detection algorithms lacked ML techniques.The section links this research motivation to weak IoT-device security.
- Probing attacks: Probing includes passive and active OS scans and service scans that identify operating systems, open ports, device types, and possible network entry points.Studies reported identifying IoT devices and vulnerable printer ports through scanning.
- Research motivation: Existing work often lacks detailed evaluation across algorithms or focuses on special environments, whereas this study evaluates real and simulated IoT attack data.The stated comparison includes multiple ML models and realistic network data recorded at the University of New South Wales.
3.1. Benchmark Data
The benchmark selection prioritizes attack-data variety and recency, leading the evaluation to use the Bot-IoT datasets.
- Dataset selection: The Bot-IoT datasets were selected because they provide attack-data variety and are up to date.These were the two stated factors considered when choosing datasets for evaluating several ML models.
3.2. Performance Evaluation Metrics
The evaluation uses confusion matrices and standard classification metrics to assess model predictions in binary and multi-class settings. Confusion matrices expose correct classifications and misclassification patterns across classes.
- Confusion matrix: A confusion matrix displays where model predictions are correct or incorrect.For binary classification, it includes TP, TN, FP, and FN.
- Confusion matrix: In confusion-matrix tables, columns represent correct classifications and rows represent available classifications.The binary example is identified as Table 2.
- Confusion matrix: Multi-class confusion matrices show all classes, allowing misclassification between classes to be observed.The multi-class example is identified as Table 3.
- Confusion matrix: Correct classifications form a diagonal path from the top-left to the bottom-right of the multi-class table, while incorrect classifications are represented by W.C denotes correct classifications and W denotes incorrect classifications.
- Performance metrics: Accuracy measures the percentage of predictions classified correctly.It can be expanded using TP, TN, FP, and FN.
- Performance metrics: Precision measures correctly predicted positive outcomes relative to all predicted positive outcomes, while recall measures them relative to all outcomes in the class.These metrics are introduced as evaluation measures for classification performance.
3.2.5. F1 Score
The paper describes evaluation metrics for comparing machine-learning models, emphasizing how metric choice depends on class distribution and how several metrics capture different aspects of performance.
- F1 Score: F1 score combines precision and recall into a value between 0 and 1.It is presented as a performance metric that can be preferable to accuracy.
- Metric selection: F1 score is preferable to accuracy when classes are highly unbalanced, whereas accuracy is suited to similarly distributed classes.Accuracy may lead to wrong conclusions when it does not account for class distribution.
- Log Loss: Log loss evaluates model performance using the probability assigned to the expected outcome, with lower scores indicating better performance.For multi-class classification, losses are calculated for each class label and summed.
- Log Loss: In binary log loss, M = 2; in multi-class log loss, M > 2 and separate class-label losses are summed.M denotes the number of possible classes, while y_i indicates the correct class and p_i is the model’s prediction probability.
- ROC: ROC plots true-positive and false-positive rates across prediction thresholds.These rates are used to examine model results at varying decision thresholds.
- Cohen’s Kappa Coefficient: Cohen’s kappa coefficient measures inter-rater reliability and compares model agreement with frequency-based guessing.This makes it useful for evaluating disparity in multi-class datasets with varying numbers of attack records.
3.3. Dataset Description
The study uses the Bot-IoT dataset, which contains records for multiple IoT attacks and no-attack cases. The experiments select relevant features, inspect class representation, and construct a randomized multi-class dataset from the binary datasets.
- Dataset contents: Bot-IoT contains ten CSV files covering data exfiltration, DoS, DDoS, keylogging, OS scan, and service scan attacks.The dataset was created by UNSW and includes both real and simulated attack data.
- Feature selection: The experiments use selected columns from the dataset after removing empty, textual, or irrelevant features.Table 4 identifies the features retained for the experiments.
- Class representation: Class representation is examined because over- or under-represented classes can adversely affect the experiments.Table 5 reports attack and no-attack data amounts for each dataset.
- Multi-class dataset: The multi-class dataset is created by collecting and randomizing rows from the binary-classification datasets.Table 6 shows the class representation of the resulting training and test data.
- Multi-class dataset: Binary and multi-class datasets do not give all classes equal representation.The study notes that weighted-class testing can examine the effects of equal representation.
3.4. Implementation
The implementation uses Python with scikit-learn and Keras, applies feature removal and normalization, and prepares randomized, size-limited data split into training and test sets.
- Tools Used: Python 3.7.4, scikit-learn, and Keras implement the machine-learning models.Keras is used for the ANN, while scikit-learn implements the other models; default classifier hyperparameters are used for comparison.
- Preprocessing: Preprocessing removes features considered uninformative or irrelevant to classification, including address, protocol, state, and metadata fields.The listed removals include flgs, proto, dir, state, saddr, daddr, and several identifier and category fields.
- Preprocessing: Feature values are normalized to a defined range such as −1 to 1 because their magnitudes vary substantially.The normalized value is produced from the original value and the minimum and maximum feature values.
- Data preparation: The multi-class data are formed by shuffling rows from all datasets and using roughly 25% of the full data, or 1,500,000 rows.The reduction is made because of the dataset’s large size and expected testing runtime.
- Training and testing: The data are split 80:20, with 80% used for training and 20% for testing.The split is implemented with train_test_split from the model_selection module.
3.5. Results and Discussion
The binary results show that the strongest model varies by attack dataset and evaluation setting, with class weighting and test-data size materially affecting performance. RF, DT, ANN, and KNN each lead under different metric priorities or classification conditions.
- Binary classification: RF is the best model without and with weighted classes for data exfiltration, while DT also has perfect scores but higher log loss.The higher log loss indicates lower confidence for DT than RF.
- Binary classification: 7.25 is DT’s log loss for DDoS HTTP despite perfect performance scores; RF has perfect recall, lowest log loss, and highest ROC AUC without weighting.With weighted classes, ANN becomes the best model because it retains perfect recall and has low log loss.
- Binary classification: RF is best for DDoS TCP with and without weighted classes, retaining perfect scores while slightly lowering log loss under weighting.LR has very low log loss with weighting but loses perfect recall.
- Binary classification: KNN is best for DDoS UDP without weighting because it has perfect scores and lower log loss than DT; RF is best with weighting.NB is second best without weighting because it combines perfect precision with low log loss.
- Binary classification: DT leads key logging without weighting, but increasing test data changes its results and weighting causes performance drops for several models.At 30% test data, recall becomes perfect instead of accuracy; at 50%, recall and precision are no longer perfect.
- Binary classification: RF leads OS and service scan without weighting, whereas ANN retains perfect recall under weighting but has the poorest ROC AUC for service scan.For OS scan with weighting, DT and RF outperform ANN on several metrics despite ANN retaining perfect recall.
4. Conclusions
The paper compares machine-learning algorithms on weighted and non-weighted Bot-IoT datasets across binary and multi-class attack classification. Results vary by task and dataset weighting, with ANN strongest for weighted binary classification and KNN and ANN highly accurate in multi-class settings.
- The study compares state-of-the-art ML algorithms using accuracy, precision, recall, F1 score, and log loss on weighted and non-weighted Bot-IoT datasets.
- RF performs best for accuracy and precision on the non-weighted dataset, while ANN achieves higher binary-classification accuracy on the weighted dataset.
- In multi-class classification, KNN is highly accurate on weighted datasets, whereas ANN is highly accurate on non-weighted datasets.
- For weighted datasets containing all attack types, ANN predicts the attack type with higher accuracy.
- The authors plan to test the models in an IDS prototype using diverse mixed-attack data to validate multi-class functionality.