PaperlayerView exact revision on arXiv

Source-linked AI summary

Multiclass Classification Procedure for Detecting Attacks on MQTT-IoT Protocol

Hector Alaiz-Moreton, Jose Aveleira-Mata, Jorge Ondicol-Garcia, Angel Luis Muñoz-Castañeda, Isaías García, Carmen Benavides

arXiv:2402.03270v1cs.LGcs.CR

TL;DR

Public datasets for evaluating network traffic using IoT protocols are lacking. The paper compares ensemble and recurrent models for classifying MQTT attack and normal frames, reporting that ensemble models maintained the highest results.

  • Problem

    Public datasets based on network traffic using IoT protocols are lacking, and research-ready traffic dissection for specialized IoT protocols is unavailable.

    Source: §1 · p. 2, §2 · p. 2

  • Method

    The paper evaluates ensemble and recurrent deep-learning models to classify three attacks and normal frames in an IoT environment dataset.

    Source: §1 · p. 2, §6 · p. 7

  • Results

    Ensemble models maintained the highest metrics, with XGBoost achieving the highest accuracy and less loss than SVM, GRU, and LSTM.

    Source: §5 · pp. 6–7

  • Takeaways & Limitations

    The authors propose LSTM, GRU, and XGBoost models to classify MQTT frames as attacks or normal for an IoT-oriented IDS.

    Source: §6 · p. 7

    Despite balancing, large class differences remained and may have negatively affected accuracy in some models, specifically GRU and LSTM.

    Source: §5 · p. 7

Abstract

from arXiv · show

The large number of sensors and actuators that make up the Internet of Things obliges these systems to use diverse technologies and protocols. This means that IoT networks are more heterogeneous than traditional networks. This gives rise to new challenges in cybersecurity to protect these systems and devices which are characterized by being connected continuously to the Internet. Intrusion detection systems (IDS) are used to protect IoT systems from the various anomalies and attacks at the network level. Intrusion Detection Systems (IDS) can be improved through machine learning techniques. Our work focuses on creating classification models that can feed an IDS using a dataset containing frames under attacks of an IoT system that uses the MQTT protocol. We have addressed two types of method for classifying the attacks, ensemble methods and deep learning models, more specifically recurrent networks with very satisfactory results.

1. Introduction

IoT devices’ limited computing and encryption capabilities, combined with heterogeneous networks, create cybersecurity challenges. This paper uses machine-learning models and an IoT dataset to classify three attacks and normal frames simultaneously.

1. Introduction

  • IoT devices’ low computing capacity and reduced encryption capability, alongside heterogeneous networks, create vulnerabilities and cybersecurity challenges.The introduction cites Mirai as an example of attacks exploiting these vulnerabilities, including distributed denial of service attacks involving up to 400,000 simultaneously connected devices.Source: §1 · p. 1
  • Machine learning can improve IDS detection, while anomaly-based systems can identify attacks including zero-day attacks.The paper distinguishes signature-based detection, which matches patterns of known attacks, from anomaly-based detection, which compares system or traffic behavior with a predefined normal status.Source: §1 · p. 1
  • The paper uses a dataset from the authors’ previous research to develop machine-learning models that classify three attacks and normal frames simultaneously in an IoT environment.The authors state that no public datasets based on network traffic using IoT protocols were available.Source: §1 · p. 2

2. Related Work

Prior work applies machine learning and deep learning to network intrusion detection, often using general-purpose datasets. The paper focuses on an IoT protocol dataset without traffic dissections ready for research use.

Related Work

  • Machine-learning studies on KDD99 and NSL-KDD use SVM, Random Forest, K-Centroid clustering, and class balancing for network anomaly detection.Reported techniques also include fuzziness-based semisupervised learning with accuracy of 84 and sequential extreme learning machine with accuracies around 95.Source: §2 · p. 2
  • Deep-learning approaches use DBNs or Fisher Score with autoencoders for feature selection, while LSTM recurrent networks classify attacks using temporal sequences [19–22].The cited work also frames machine learning as based on deep learning.Source: §2 · p. 2
  • IoT IDS studies combine fog and mobile edge computing or modify rule-based IDS with KNN and SVM, with simulations using NSL-KDD.The passage notes that existing IoT IDS solutions do not cover a wide range of attacks or IoT technologies.Source: §2 · p. 2
  • AWID supports Wi-Fi intrusion studies using a neural-network classifier, while CICIDS2017 is used to validate algorithms trained with recurrent neural networks.AWID contains TCP frames captured from a WLAN network with attacks on 802.11 security.Source: §2 · p. 2
  • The study uses a dataset specialized for an IoT protocol and specific vulnerabilities, addressing the lack of research-ready traffic dissection for that protocol.The authors describe IoT IDS research as incipient, with existing solutions not covering a wide range of attacks or IoT technologies.Source: §2 · p. 2

3. Methods and Materials · 3.1. MQTT Dataset.

The study develops methods and experimental materials for classifying MQTT anomalies using network traffic captured in a sensor-and-actuator test environment. Frames are labeled as attack or normal to support supervised learning.

3. Methods and Materials

  • The Methods and Materials section frames the work around experimental methods and materials, with the MQTT dataset obtained in a sensor-and-actuator test environment.A server hosts the management application and also acts as the MQTT broker.Source: §3 · p. 2, §3.1 · p. 2

3.1. MQTT Dataset.

  • MQTT uses a central broker in a star topology to manage topic-based publish-subscribe communication, which can be one-to-one or one-to-many.Source: §3.1 · p. 2
  • The researchers captured network-level attack traffic and all generated traffic in the test environment.Source: §3.1 · p. 2
  • DoS attacks saturated the broker with messages and new connections per second using MQTT-malaria.Source: §3.1 · p. 2
  • MitM attacks intercepted and modified sensor data between a sensor and broker using Kali Linux and Ettercap.Source: §3.1 · p. 2
  • Intrusion attempts used MQTT’s port 1883 and the special character “#” to discover active topics available for subscription.Source: §3.1 · p. 2
  • Selected fields and protocol data were labeled as attack or normal, generating three attack-specific CSV files for supervised learning.Source: §3.1 · p. 3
  • The CSVs contain 94.625 DoS frames (45.513 attack, 49.112 normal), 110668 MitM frames (3855 attack, 106.813 normal), and 80,893 intrusion frames (1898 attack, 78,995 normal).Source: §3.1 · p. 3

3.2. Classification Methods.

The study uses XGBoost and recurrent networks to classify MQTT-IoT attacks, motivated by prior XGBoost results and the temporal information in sequential network frames. It describes XGBoost, general RNNs, and LSTM and GRU variants.

3.2. Classification Methods.

  • XGBoost and recurrent networks were selected for attack classification, with prior XGBoost results and the importance of frame sequence and timing motivating the choices.Frames are produced sequentially, and their sequence and the time between them provide relevant information for detecting attacks.Source: §3.2 · p. 3

3.2.1. XGBoost (Gradient Boosting).

  • XGBoost builds additive models by fitting regression trees to negative gradients of binomial or multinomial loss, and the study implements it with the Python library [38].The described system supports sparse data, approximate tree learning with instance weights, and out-of-core computation.Source: §3.2.1 · p. 3

3.2.2. Recurrent Neural Networks.

  • RNNs use activations from prior time steps to learn mappings over time, but can suffer vanishing or exploding gradients [40].Variations such as GRU and LSTM cells can address these training problems.Source: §3.2.2 · pp. 3–4
  • LSTM memory blocks use input, output, and forget gates to regulate cell information flow and stabilize the cell state.The implementation uses Keras with TensorFlow, GPU processing, and CudnnLSTM.Source: §3.2.2 · p. 4
  • GRU units adaptively capture dependencies across time scales [44] and omit the LSTM’s data-flow exposure control, making them faster and more adaptable to changes in time flow [45] [46].The implementation uses Keras with TensorFlow, GPU processing, and CudnnGRU.Source: §3.2.2 · p. 4

3.3. Optimization Methods.

The authors tested RMSprop, Adam, and Nadam for loss optimization and obtained their best results with Nadam. They describe Nadam as an Adam variant with Nesterov momentum and cite its faster minibatch learning as beneficial for their complex network and limited epochs.

3.3. Optimization Methods.

  • Nadam is an Adam variant that applies Nesterov momentum [48].Source: §3.3 · p. 4
  • Adam uses adaptive estimates of lower-order moments and is described as efficient, low-memory, and suitable for large problems, changing objectives, and noisy or sparse gradients [47].Its hyperparameters are described as intuitive and usually requiring little tuning.Source: §3.3 · p. 4
  • Nadam gave the best results among tested RMSprop, Adam, and Nadam for loss optimization, with faster minibatch learning and better results for the complex network in fewer epochs.The authors sought a faster optimizer to learn more in fewer epochs without risking a rapid decay into overfitting.Source: §3.3 · p. 4

3.4. Batch Normalization. · 3.5. Evaluation Metrics.

The section describes batch normalization’s training benefits and defines metrics used to evaluate the classification models. These metrics cover probability-based loss, misclassification rate, the precision–recall balance, and categorical accuracy.

3.4. Batch Normalization.

  • Batch normalization normalizes each training minibatch to address internal covariate shift, enabling higher learning rates and less dependence on initialization; it can also regularize models [49].The passage says internal covariate shift arises as input-layer values change during training and makes models harder to train with saturating nonlinearities.Source: §3.4 · p. 4

3.5. Evaluation Metrics.

  • The evaluation uses several metrics because each measures model efficiency differently, providing a more accurate view of performance.Source: §3.5 · p. 4

3.5.1. Multiclass Logarithmic Loss and Categorical Cross Entropy.

  • Logarithmic loss evaluates classification predictions expressed as probabilities between 0 and 1, and equals cross-entropy when measuring error rates in that range.For multiclass problems, the passage extends the metric to one-of-K encoded true labels and a matrix of predicted probabilities.Source: §3.5 · p. 4

3.5.2. Multiclass Classification Error Rate.

  • The multiclass classification error rate is the percentage of a model’s predictions that are misclassifications.Source: §3.5.2 · p. 5

3.5.3. F-Beta Score.

  • The F-beta score is a weighted harmonic average of precision and recall, with beta below 1 favoring precision and beta above 1 favoring recall.Its best value is 1 and its worst value is 0.Source: §3.5.3 · p. 5

3.5.4. Categorical Accuracy.

  • Categorical accuracy calculates the average accuracy rate across all predictions in a multiclass problem.Source: §3.5.4 · p. 5

3.6. Dropout.

Dropout is used to prevent overfitting by randomly dropping neural-network units and their connections during training.

3.6. Dropout.

  • Dropout randomly drops neural-network units and their connections during training to limit excessive adaptation to the problem.The stated purpose is to prevent overfitting.Source: §3.6 · p. 5

4. Experiments

The experiments combine balanced datasets for three attacks into a multiclass dataset, select features with a hybrid importance method, and evaluate XGBoost, LSTM, and GRU classifiers. Recurrent networks use four timesteps, with model settings guided by training logs to address underfitting or overfitting.

4. Experiments

  • Three attack datasets were balanced with Scikit-learn resampling, combined, and feature-selected using a mutual-information filter-wrapper method that reduces redundancy.The experiments also set beta = 1 for their custom F-beta score to increase recall’s value.Source: §4 · p. 5

4.1. XGBoost.

  • XGBoost used the fast-histogram tree method and threefold cross-validation grid search to select parameters for four-class classification.The method is described as faster and more memory-efficient than other methods, but requires a specific CUDA version.Source: §4.1 · p. 5

4.2. Recurrent LSTM.

  • The LSTM used categorical cross entropy, Nadam, categorical accuracy and F-score, with a batch size of 128, 15 epochs, and 10% validation split.The encoder-decoder network used dropout of 0.3–0.4 and batch normalization; Nadam was found more efficient than Adam and RMSprop for this model.Source: §4.2 · p. 6

4.3. Recurrent GRU.

  • The GRU used categorical cross entropy, Nadam, categorical accuracy and F-score, with a batch size of 256, 17 epochs, and 10% validation split.Its linear structure used dropout of 0.2–0.3 and batch normalization; Nadam was more efficient than Adam, and the GRU cell was described as slower to learn but faster than LSTM.Source: §4.3 · p. 6

5. Results and Discussion

XGBoost achieved the highest accuracy and less loss than linear models and the tested neural networks. The LSTM and GRU results were stronger overall than linear models but remained below ensemble results, with class imbalance still affecting performance.

  • XGBoost achieved the highest accuracy and less loss than linear models and SVM, GRU, and LSTM in IoT attack classification.Source: §5 · pp. 6–7
  • Overall, LSTM and GRU outperformed linear models but trailed ensembles; against SVM, both were worse on DoS and better on intrusion and MitM.The paper notes that multiclass classification is harder than binary classification.Source: §5 · p. 7
  • Despite addressing class imbalance, large class differences may have negatively affected GRU and LSTM accuracy, while accounting for sequence helped maintain good results.Source: §5 · p. 7

6. Conclusion

The paper proposes MQTT-specific classification models to support IoT intrusion detection, using LSTM, GRU, and XGBoost to classify network frames as attack or normal. It concludes that ensembles achieved the highest results and proposes deploying a model in an IDS on a standard computer.

  • Ensemble methods achieved the highest results; deep learning models generally outperformed linear models but did not match ensembles.Source: §6 · p. 7
  • The authors chose LSTM and GRU for the importance of timing and sequence in network attacks, and XGBoost because the problem structure benefits hierarchical ensembles.Source: §6 · p. 7
  • The proposed models classify MQTT network frames as attack or normal and are intended to provide an IoT-oriented IDS with attack detection capability.Source: §6 · p. 7
  • The proposed IDS would use a model deployed on a standard computer, with router port mirroring.Source: §6 · pp. 7, 10

Hindawi www.hindawi.com Volume 2018 Operations Research Advances in

The supplied excerpt consists of Hindawi and Volume 2018 labels and journal names; it contains no substantive discussion of the MQTT-IoT classification study.

Loading 2402.03270v1…