PaperlayerView exact revision on arXiv

Source-linked AI summary

Revolutionizing Cyber Threat Detection with Large Language Models: A privacy-preserving BERT-based Lightweight Model for IoT/IIoT Devices

Mohamed Amine Ferrag, Mthandazo Ndhlovu, Norbert Tihanyi, Lucas C. Cordeiro, Merouane Debbah, Thierry Lestable, Narinderjit Singh Thandi

arXiv:2306.14263v2cs.CRcs.AI

TL;DR

IoT networks need accurate, computationally efficient cyber-threat detection as attacks and connected devices increase. The paper introduces SecurityBERT, a lightweight privacy-preserving BERT model using PPFLE with BBPE for structured network-traffic representation, and reports 98.2% accuracy across fourteen attack types with fast CPU inference and a compact model size.

  • Problem

    Growing IoT deployment and increasingly diverse cyber threats require accurate network-based attack detection with minimal computational requirements.

    Source: §I · p. 1

  • Method

    SecurityBERT combines a lightweight BERT architecture with privacy-preserving PPFLE encoding and BBPE tokenization for structured network-traffic representation.

    Source: §I · pp. 1–2

  • Results

    98.2% overall accuracy was achieved for fourteen attack types, with 0.15 sec CPU inference time and a 16.7 MB model size.

    Source: 1) Inference time · p. 14, §V · p. 15

  • Takeaways & Limitations

    The reported accuracy, inference speed, and compact size support SecurityBERT’s use for real-life traffic analysis and deployment on resource-constrained IoT devices.

    Source: 1) Inference time · p. 14, §V · p. 15

Abstract

from arXiv · show

The field of Natural Language Processing (NLP) is currently undergoing a revolutionary transformation driven by the power of pre-trained Large Language Models (LLMs) based on groundbreaking Transformer architectures. As the frequency and diversity of cybersecurity attacks continue to rise, the importance of incident detection has significantly increased. IoT devices are expanding rapidly, resulting in a growing need for efficient techniques to autonomously identify network-based attacks in IoT networks with both high precision and minimal computational requirements. This paper presents SecurityBERT, a novel architecture that leverages the Bidirectional Encoder Representations from Transformers (BERT) model for cyber threat detection in IoT networks. During the training of SecurityBERT, we incorporated a novel privacy-preserving encoding technique called Privacy-Preserving Fixed-Length Encoding (PPFLE). We effectively represented network traffic data in a structured format by combining PPFLE with the Byte-level Byte-Pair Encoder (BBPE) Tokenizer. Our research demonstrates that SecurityBERT outperforms traditional Machine Learning (ML) and Deep Learning (DL) methods, such as Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs), in cyber threat detection. Employing the Edge-IIoTset cybersecurity dataset, our experimental analysis shows that SecurityBERT achieved an impressive 98.2% overall accuracy in identifying fourteen distinct attack types, surpassing previous records set by hybrid solutions such as GAN-Transformer-based architectures and CNN-LSTM models. With an inference time of less than 0.15 seconds on an average CPU and a compact model size of just 16.7MB, SecurityBERT is ideally suited for real-life traffic analysis and a suitable choice for deployment on resource-constrained IoT devices.

I. INTRODUCTION

SecurityBERT addresses rising IoT cyber threats with a lightweight, privacy-preserving BERT architecture designed for accurate, practical network-based attack detection. It combines PPFLE and BBPE to structure and protect network traffic data while targeting efficient multi-category classification.

  • The architecture targets rapid inference, confidentiality of extracted network data, and higher accuracy than previous machine-learning models.Source: §I · p. 1
  • PPFLE combines with the BBPE tokenizer to represent network traffic in a structured format while preserving privacy.Source: §I · p. 2
  • SecurityBERT is a 15-layer BERT-based architecture with 11 million parameters for multi-category cyber threat classification.Source: §I · p. 2
  • 98.2% overall accuracy was reported for identifying fourteen attack types on the Edge-IIoTset cybersecurity dataset.Source: §I · p. 2
  • Prior research applied BERT to cybersecurity tasks including CAN anomaly detection, Android malware analysis, system-log diagnosis, and malicious HTTP-request detection.Source: §II · p. 2
  • Existing studies often combined packet-data detection with either LLMs or conventional methods, while SecurityBERT was designed to integrate lightweight detection, packet-data handling, and privacy considerations.Source: §II · p. 3

III. SECURITYBERT ARCHITECTURE DESIGN

SecurityBERT is built through a staged workflow that progresses from dataset use and feature extraction to privacy-preserving encoding, tokenization, representation learning, and fine-tuning. The final classifier produces fifteen outputs for fourteen attack categories plus normal traffic.

  • The architecture workflow includes dataset utilization, feature extraction, PPFLE, BBPE tokenization, embedding, contextual representation, and model training.Source: §III · p. 3
  • Fine-tuning uses a Softmax activation function after the training stages.Source: §III · p. 3

A. Dataset Utilization (Edge-IIoTset Dataset)

The Edge-IIoTset dataset provides realistic network-security data for evaluating attack detection, while SecurityBERT converts its numerical and categorical features into privacy-preserving fixed-length representations.

Dataset rationale

  • Realistic cybersecurity datasets are crucial because generating traffic is time-consuming and simulated attacks may be incomplete.Open-source sources often rely heavily on artificial scenarios, which can produce unrealistic outcomes when used exclusively for training.Source: A. Dataset Utilization (Edge-IIoTset… · p. 3

Dataset composition

  • The Edge-IIoTset dataset combines network traffic, logs, system resources, and alerts, with 61 distinct features retained after null-feature removal.The first 15 features are listed in Table II, while the complete feature set is referenced separately.Source: B. Features extraction · p. 4, Table II · p. 4

Feature extraction

  • SecurityBERT processes each PCAP time window by separating network flows, extracting predefined features, and organizing them into CSV format.The resulting features are intended to distinguish patterns associated with network-based cyberattacks.Source: B. Features extraction · p. 4

Privacy-Preserving Fixed-Length Encoding

  • PPFLE transforms numerical and categorical values by adding column names, hashing each feature representation, and concatenating hashed values within each instance.This creates a structured textual representation while encoding network data so sensitive information is hidden and classification features are preserved.Source: C. Privacy-Preserving Fixed-Length… · p. 5, PPFLE description · p. 5
  • PPFLE repeats the hashed feature construction across matrix rows to create DataList, whose inner lists contain fixed-length encoded values for each row.The paper presents the procedure as a row-and-column algorithm and notes that the representation mirrors natural-language characteristics for machine-learning use.Source: PPFLE description · p. 5

D. Byte-level BPE (BBPE) Tokenizer

The BBPE tokenizer converts PPFLE-encoded network data into byte-level subword sequences so SecurityBERT can process hashed traffic representations without receiving sensitive information.

Tokenization rationale

  • Tokenization is applied to PPFLE-encoded data, ensuring that sensitive network information is not directly fed to the model during training.Each feature is encoded independently so adjacent hashed values retain information about attack types, whereas hashing all features together would destroy useful structure.Source: D. Byte-level BPE (BBPE) Tokenizer · p. 6

Encoded attack patterns

  • PPFLE produces distinct hash sequences for differing feature patterns, including port-443 GET traffic and DNS poisoning representations.The paper reports that the 61 encoded features effectively represent attack types and that the model recognizes patterns even after hashing.Source: D. Byte-level BPE (BBPE) Tokenizer · p. 6

BBPE operation

  • Byte-level subword processing breaks the network-traffic representation into byte sequences likely to occur in the tokenizer’s human-language vocabulary.This allows traffic data to be processed using BERT’s language-modeling capabilities.Source: D. Byte-level BPE (BBPE) Tokenizer · p. 6

Tokenizer configuration

  • A 5000-token vocabulary and special tokens including <s>, <pad>, </s>, <unk>, and <mask> were used during tokenizer training.The tokenizer was trained with a minimum frequency of 2.Source: D. Byte-level BPE (BBPE) Tokenizer · p. 6

E. SecurityBERT embedding

SecurityBERT embedding prepares evaluation sequences for transformer processing by encoding them in chunks, collecting input IDs and attention masks, and concatenating the results.

Sequence encoding

  • The embedding procedure initializes a chunk_size of 5000 and computes the number of evaluation-data chunks by rounding up the length-to-chunk ratio.Each chunk is then encoded separately before the resulting components are combined.Source: E. SecurityBERT embedding · p. 7

Transformer inputs

  • Encoded evaluation chunks are unpacked into input IDs and attention masks, which are appended to separate lists and concatenated along dimension 0.The resulting arrays form a complete set of transformer inputs for the evaluation data.Source: E. SecurityBERT embedding · p. 7

Attention masking

  • Attention masks identify real tokens with 1s and padding tokens with 0s, preventing the transformer from attending to padding.Masks typically have the same length as the corresponding input-ID sequence.Source: E. SecurityBERT embedding · p. 7

F. Contextual representation

SecurityBERT pre-trains a transformer-based BERT architecture on tokenized network data to produce contextual representations for cyber threat classification.

F. Contextual representation

  • SecurityBERT represents each token as an embedding vector before processing the sequence through transformer encoder layers.Each encoder layer uses multi-head self-attention and positionwise feed-forward operations.Source: F. Contextual representation · p. 7
  • Self-attention captures the importance of different tokens according to their semantic and syntactic relationships.The resulting contextual embeddings undergo feed-forward operations and layer normalization.Source: F. Contextual representation · p. 7
  • PPFLE-encoded data makes text normalization unnecessary in SecurityBERT’s training process.The paper states that normalization from the original BERT architecture adds no extra value for PPFLE data.Source: 1) Text Normalization · p. 7, G. Training SecurityBERT · p. 7
  • Tokenization breaks each document into words or tokens that form the basic units processed by the machine-learning model.The tokenization function applies to documents in the dataset.Source: 2) Tokenization · pp. 7–8

3) Frequency Filtering:

The tokenizer pipeline filters infrequent tokens, builds a frequency-ranked vocabulary, adds special tokens, and maps future text into processable token sequences for SecurityBERT.

3) Frequency Filtering:

  • The high-pass filter removes tokens whose document frequency is below the minimum frequency F.This filtering excludes rare words considered less informative for subsequent processing or training.Source: 3) Frequency Filtering · p. 8
  • The vocabulary function selects the top V words from the documents according to their frequency rank.The selected words form the vocabulary recognized by the model.Source: 4) Vocabulary Creation · p. 8

5) Special Token Addition:

  • The expanded vocabulary v′ combines the original vocabulary v with special tokens S.These tokens support operations such as sentence marking, unknown-word handling, and padding.Source: 5) Special Token Addition · p. 8

3) Frequency Filtering:

  • The trained tokenizer maps future text segments into token sequences from the established vocabulary v′.This mapping converts unstructured text into a form SecurityBERT can process.Source: 6) Tokenizer Training · p. 8
  • A linear layer followed by Softmax is added after pre-training, and the full network is fine-tuned with labeled data for threat classification.Fine-tuning adapts the learned contextual representations to the specific detection task.Source: H. Fine tuning SecurityBERT · p. 8

1) Training setup:

SecurityBERT uses a compact 15-layer architecture built around four encoder layers and regularized BERT components to support efficient classification of PPFLE data.

1) Training setup:

  • The final SecurityBERT model comprises 15 layers and uses four encoder layers adapted to the target problem.Additional final-stage layers include dropout and a classifier layer, with dropout used to mitigate overfitting.Source: §I · p. 8

1) BERT Embeddings:

  • The embedding block contains word, position, and token type embeddings followed by layer normalization and dropout.The layer normalization size is 128 and the dropout rate is 0.1.Source: 1) BERT Embeddings · p. 8

2) BERT Self Attention:

  • BERT self-attention uses three linear transformations for keys, queries, and values, with input and output features sized at 128.A dropout layer with rate 0.1 is included in this component.Source: 2) BERT Self Attention · pp. 8, 10

1) Training setup:

  • The self-output block applies a 128-dimensional dense layer, layer normalization, and dropout for regularization.The dense layer preserves a feature size of 128 from input to output.Source: 3) BERT Self Output · p. 10
  • The intermediate block expands features from 128 to 512 using a dense layer and GELU activation, while the output block returns them to 128.These transformations are part of the BERT encoder structure.Source: 4) BERT Intermediate · p. 10, 5) BERT Output · p. 10

6) BERT Pooler+BERT Final:

The pooler and final classifier reduce contextual features to fifteen outputs for fourteen attack categories plus normal traffic, followed by evaluation on Edge-IIoTset.

6) BERT Pooler+BERT Final:

  • The final linear layer reduces the pooled representation to 15 outputs for 14 attack classes and normal traffic.A Tanh activation precedes the final dimensionality reduction.Source: 6) BERT Pooler+BERT Final · p. 10
  • 98.2% overall accuracy was reported for SecurityBERT in the performance evaluation.The paper describes this as the highest accuracy it knew of for an ML algorithm detecting IoT attacks on realistic real-world network traffic.Source: §IV · p. 10
  • The evaluation uses Precision, Recall, F1-Score, and Support as standard performance measurements.These metrics are used to assess classification performance comprehensively.Source: A. Experimental Results · p. 10
  • The Edge-IIoTset data was split into 80% training samples and 20% evaluation samples unseen during training.The distribution covers different cyberattack classes across the two partitions.Source: A. Experimental Results · p. 10
  • The reported evaluation materials include a classification report and an accuracy-and-loss history over four training epochs.These materials are presented in Table VI and Figure 6, respectively.Source: A. Experimental Results · p. 10, Figure 6 · p. 11

1) ROC AUC Scores for Cyber Threat Classification:

SecurityBERT achieved near-perfect or perfect class-level discrimination across the cyber-threat classification task. Performance was strong overall, though ransomware and fingerprinting remained notable sources of misclassification.

  • 1.0 AUC was achieved for the Normal, UDP, TCP, and MITM classes.Most remaining classes had AUC values from approximately 0.9976 to 0.999988.Source: 1) ROC AUC Scores for Cyber Threat… · p. 11
  • 0.991569 AUC for Fing was the lowest reported class-level ROC AUC, while still indicating strong classification.Source: 1) ROC AUC Scores for Cyber Threat… · p. 11
  • 0.40 recall and 0.57 F1-score for Ransomware indicate that the model missed a significant portion of actual ransomware instances.Ransomware precision was 1.00.Source: 2) Confusion Matrix · p. 12
  • 0 precision, recall, and F1-score for Fingerprinting indicate complete misclassification of those instances.A considerable proportion of Fingerprinting misclassifications pertained to ICMP.Source: 2) Confusion Matrix · p. 12
  • 0.982 overall accuracy was reported for identifying different types of network attacks.Source: 2) Confusion Matrix · p. 12
  • α values stabilized around 2 to 3 in deeper layers, with many layers near 2, a value associated with better generalization in the cited analysis.The first layer had an α value of around 10.43.Source: 3) WeightWatcher - Empirical Spectral… · p. 12

B. Performance Comparison

SecurityBERT was compared with traditional ML and DL models on the 14-attack Edge-IIoTset task. The reported 98.2% accuracy was presented as the highest achieved for multiclassifying these attack categories.

1) Inference time:

SecurityBERT was evaluated across GPUs, a TPU, and a general-purpose CPU to assess deployment feasibility on IoT devices. The reported CPU inference time supports its use for rapid traffic analysis and embedded deployment.

  • 0.15 seconds was the pivotal CPU inference time reported for SecurityBERT.Source: 1) Inference time · p. 14
  • A100, T4, and V100 GPUs, a TPU, and a general-purpose CPU were evaluated for SecurityBERT inference.Source: 1) Inference time · p. 14
  • Real-time identification of malicious activity within internal networks is described as possible when SecurityBERT is embedded in an IoT network.The paper links this deployment scenario to a potential reduction in Mean Time to Remediate.Source: 2) Reducing MTTR · p. 14
  • SecurityBERT can be integrated into existing real-world systems to augment their accuracy and detection rate.Source: 2) Reducing MTTR · p. 15
  • Future work includes continuous updating and training on the latest real-world datasets to maintain efficacy as cyber threats evolve.Source: §V · p. 15

Related papers

Loading 2306.14263v2…