PaperlayerView exact revision on arXiv

Source-linked AI summary

Parameters efficient language model with latent cross-attention for IoT intrusion detection

Frederic Adjewa, Moez Esseghir, Leila Merghem-Boulahia, Cheikh Kacfah

arXiv:2610.30222v1cs.CR

TL;DR

Large language models show promise for intrusion detection, but their computational and memory demands make IoT deployment difficult. The paper proposes a compact Transformer architecture that compresses inputs into a latent space; it reports competitive results against SecurityBERT with fewer parameters and faster simulated-edge inference.

  • Problem

    Large language models have strong potential for intrusion detection, but their memory and computational demands make deployment difficult in resource-constrained IoT environments.

    Source: §I · p. 1

  • Method

    The model compresses input sequences through cross-attention into a learnable latent space, then processes the representation with Transformer blocks using stacking or parameter sharing.

    Source: §I · p. 1

  • Results

    With the same PPFLE encoding, the model gains about 2 macro F1-score points on Edge-IIoTset and is within 0.7 points on CICIoT2023; it is about 89× faster than SecurityBERT.

    Source: §VI · p. 12

  • Takeaways & Limitations

    A single Transformer layer is sufficient in this setting, and parameter sharing matches stacking with up to 66% fewer parameters when more depth is needed.

    Source: §VI · p. 12

Abstract

from arXiv · show

Language models have shown strong potential for intrusion detection, but their high memory and computational demands make them difficult to deploy in resource-constrained IoT environments. In this work, we propose a compact Transformer-based architecture that compresses the input into a small latent space before processing it with a minimal Transformer core. We analyze nine compression configurations, compare parameter sharing with layer stacking, and evaluate our approach against SecurityBERT on two IoT intrusion detection datasets, Edge-IIoTset and CICIoT2023. With only 306k non-embedding parameters and 1.6M parameters in total (6MB), our model is about 7x smaller than SecurityBERT (11.2M parameters, 45MB). With the same privacy-preserving encoding, it outperforms SecurityBERT by about 2 points of macro F1-score on Edge-IIoTset and is within 0.7 points on CICIoT2023. It is also about 89x faster on a simulated edge device. We further show that a single Transformer layer is sufficient, and that parameter sharing matches layer stacking with up to 66% fewer parameters. These findings show that lightweight Transformer-based architectures can provide a practical and accurate solution for intrusion detection in resource-constrained IoT settings

I. Introduction

Language models offer a promising route to IoT intrusion detection, but their computational and memory demands complicate deployment. The paper proposes a compact Transformer-inspired model and situates it among prior approaches that often rely on large pretrained models or multi-component architectures.

I. Introduction

  • Large language models are difficult to deploy in IoT environments because of their substantial computational and memory requirements.Prior efforts reduce Transformer depth or offload processing to edge or cloud devices.Source: §I · p. 1
  • The paper asks whether a compact Transformer-inspired model can learn directly from network data without pretrained weights while retaining competitive detection performance.The work also frames questions about compression trade-offs, parameter sharing versus stacking, and comparison with a larger Transformer-based IDS.Source: §I · pp. 1–2

II. Related Work

  • Prior work spans Transformer-based IDSs, IoT-tailored and hybrid models, multi-model systems, and approaches that offload processing to edge or cloud devices.Examples include BERT-based systems, IoT-focused designs, multi-model frameworks, and cloud or edge offloading.Source: §II · p. 2
  • Many existing methods rely on large pretrained language models or multi-component architectures with substantial computational and memory requirements.This limitation motivates compact Transformer-inspired designs intended to retain detection capability for resource-constrained IoT deployments.Source: §II · p. 2

III. Methodology

The methodology motivates a compact attention design by identifying sequence-length-dependent computation as a Transformer bottleneck. It frames input compression into a smaller latent set as a way to reduce attention cost while retaining attention-based modeling.

III. Methodology

  • The methodology identifies notation for the model and provides a qualitative comparison framework for related work.The comparison caption defines IoT focus, pretrained-language-model use, homogeneity, and deployability as its criteria.Source: §III · p. 2, Table I · p. 3

A. The transformer models and the computation bottleneck

  • The paper notes that standard Transformer architectures are generally infeasible in IoT settings with limited computation, memory, and power budgets.It emphasizes the use of Transformer mechanisms rather than the necessity of deploying pretrained BERT in these environments.Source: A. The transformer models and the… · p. 3
  • For input length N, computing QK^T costs O(N^2d_k), and this quadratic scaling limits Transformers on long sequences and resource-constrained systems.The bottleneck comes primarily from self-attention's dependence on input sequence length.Source: A. The transformer models and the… · pp. 3–4
  • Compressing an input of length N into M learnable latent vectors, with M ≪ N, changes attention complexity to O(NM) in input length and O(M^2) in latent size.The paper presents this reformulation as enabling efficient processing while preserving attention-based expressive power.Source: A. The transformer models and the… · p. 4

B. Proposed architecture

The proposed architecture compresses inputs through cross-attention into a latent representation, then processes that representation with Transformer blocks. It considers both distinct stacked blocks and repeated use of a parameter-shared block.

  • Cross-attention projects input and latent representations into a shared feature space, and its learned projection matrices are independent of input length and latent-set size.The resulting computation scales linearly with input length N for fixed latent size M and latent dimension e.Source: B. Proposed architecture · p. 4
  • The cross-attention stage produces a compact representation whose dimensionality depends on latent size M, decoupling downstream processing from original input length N.The latent attention computation uses the projected input and latent representations.Source: B. Proposed architecture · p. 4
  • After cross-attention, Transformer blocks process the latent representation using self-attention with O(M^2) complexity, independent of original input length N.The model does not employ multi-head attention, although the authors encourage it for richer representations.Source: B. Proposed architecture · p. 4
  • Stacking uses distinct sequential Transformer blocks, so parameter count grows linearly with depth L; sharing reapplies one block and keeps parameter count constant as L increases.The shared-block formulation is described as iterative refinement of the latent representation and as suitable for resource-constrained deployment.Source: B. Proposed architecture · p. 4

C. Model paramters assessing

The parameter assessment identifies embeddings as a common component of sequence models and describes token and positional embedding choices. The proposed architecture figure also depicts stacking and parameter sharing as alternative downstream paths.

C. Model paramters assessing

  • The architecture figure contrasts downstream processing by sequential distinct Transformer blocks with recurrent reuse of a single parameter-shared block.Source: Figure 1 · p. 5

1) Embeddings:

  • Token embeddings map each discrete token identifier to an E-dimensional vector, with parameter count scaling linearly with vocabulary size and maximum sequence length.The embedding layer is common to sequence-based models.Source: C. Model paramters assessing · p. 4, 1) Embeddings · p. 5
  • Deterministic sinusoidal positional embeddings encode token order without adding trainable parameters.The paper adopts them following and states they achieve comparable empirical performance to learned positional encodings.Source: 1) Embeddings · p. 5

2) Cross-Attention module:

The cross-attention module compresses the input by attending from a small latent representation to the full sequence. Its parameterization combines query and key projections into one matrix to reduce attention-score parameters.

2) Cross-Attention module:

  • Cross-attention compresses the input by having a small set of latent vectors attend to the full input sequence.The passage says this alleviates standard self-attention’s quadratic scaling issue.Source: 2) Cross-Attention module · p. 6
  • The cross-attention module projects input and latent representations into a shared feature space of dimension e.Its parameter count is summarized as a total in the section.Source: 2) Cross-Attention module · p. 6
  • Naive separate query and key projections would require e^2 + Ee parameters.The section contrasts this parameterization with the single-matrix reformulation.Source: 2) Cross-Attention module · p. 6
  • Rewriting the query–key interaction lets one projection matrix WQK replace separate WQ and WK matrices.This reduces the learnable parameters associated with computing attention scores.Source: 2) Cross-Attention module · p. 6

3) Latent transformer module:

The latent Transformer processes the enriched latent representation with a single-head self-attention layer and an MLP. Reusing a block across iterations keeps its parameter count independent of depth, while weight sharing shifts the main parameter bottleneck to embeddings.

3) Latent transformer module:

  • Each latent Transformer block applies single-head self-attention followed by an MLP to the enriched latent representation.The block operates exclusively on the output produced by cross-attention.Source: 3) Latent transformer module · p. 6
  • A single latent self-attention layer has 3e^2 parameters, followed by an output block with e^2 parameters.The MLP uses hidden dimension h = 4e and has 8e^2 learnable parameters.Source: 3) Latent transformer module · p. 6
  • One latent Transformer block has 12e^2 parameters, while stacking L blocks uses distinct parameter sets.Bias terms, residual connections, and normalization layers are omitted as negligible or residual contributions in the count.Source: 3) Latent transformer module · p. 6
  • Reusing one latent Transformer block across L iterations makes the parameter count independent of L.The paper notes that increasing latent-processing depth under sharing can refine representations without increasing model size.Source: 3) Latent transformer module · p. 6
  • The model’s rough total parameter count includes embeddings, cross-attention terms, latent Transformer parameters, and residual parameters Δ(e).Δ(e) represents omitted bias, residual-connection, and normalization-layer parameters.Source: 3) Latent transformer module · p. 6
  • Under weight sharing, embedding layers become the main parameter bottleneck as attention and cross-attention parameter contributions decrease.This is the reported effect of adopting the weight-sharing strategy.Source: 3) Latent transformer module · p. 6

D. Other internal improvements

RMSNorm replaces layer normalization to reduce computational overhead in the model. The experiments use IoT intrusion-detection benchmarks, including a stratified subset of CICIoT2023.

D. Other internal improvements

  • RMSNorm replaces layer normalization and focuses only on rescaling, which the authors say improves running time.The passage attributes computational overhead to layer normalization’s rescaling and recentering.Source: D. Other internal improvements · p. 6

IV. Experimental design

  • CICIoT2023 contains about 33 attack types and more than 46 million records; the study samples a stratified subset preserving class distribution.The benchmark is described as collected from a realistic experimental testbed.Source: §IV · p. 7
  • EdgeIIoTset includes 14 attack types besides benign traffic, and the two datasets provide an IoT intrusion-detection evaluation setting.Source: §IV · p. 7

A. Data preparation

Data preparation retains sufficiently represented attack classes and limits the vocabulary to support compact inputs. Evaluation emphasizes macro-F1 on imbalanced data and measures computational complexity with FLOPs.

A. Data preparation

  • Attack classes with at least 1000 samples are retained, datasets are split 70:30, and only a fraction of benign traffic is kept.The stated purpose of retaining a fraction of benign traffic is to focus primarily on detecting malicious traffic.Source: A. Data preparation · p. 7
  • Each dataset’s vocabulary is limited to 5000 tokens, with a word-level tokenizer trained on 70% of encoded data.The passage says this vocabulary limit is used to maintain a compact, computationally efficient representation.Source: A. Data preparation · p. 7

B. Experimental configuration and evaluation metrics

V. Experimental Results

Across nine latent-space configurations on Edge-IIoTset, e128M128 offers the best reported balance of macro F1-score and computational cost. Smaller configurations save compute but can lose detection quality, while increasing latent size can raise compute without comparable storage growth.

A. Which compression configuration provides the best

  • 95.66 ± 2.78% macro F1-score for e64M64 is 2.4 percentage points below e128M128, with greater variation across seeds.The authors note that the compact configuration may need more than the 10-epoch training budget to converge.Source: A. Which compression configuration… · p. 8
  • At fixed e, doubling M raises GFLOPs by 1.7–1.9×; at fixed M, doubling e raises them by 1.9–2.5×.Model size is driven mainly by e: increasing M from 64 to 256 adds at most 0.19 MB at fixed e.Source: A. Which compression configuration… · p. 8
  • e64M256 and e128M128 use nearly the same compute, 0.145 vs. 0.147 GFLOPs per sample, but e128M128 is 0.76 MB larger.The configurations cluster by latent embedding size when macro F1-score is plotted against core parameters.Source: A. Which compression configuration… · p. 8, Figure 3 · p. 8
  • 98.06±0.08% macro F1-score makes e128M128 the best trade-off among the evaluated configurations.The highest-scoring e256M64 configuration offers only a negligible gain at about 3.4× more core parameters and 1.5× more compute.Source: A. Which compression configuration… · p. 8, Figure 2 · p. 8

B. What are the trade-offs between parameters sharing and Transformer stacking?

The experiments compare shared and stacked Transformer layers, then assess the compact model against SecurityBERT for IoT intrusion detection. The proposed model offers a favorable performance-efficiency trade-off, while parameter sharing reduces parameter count without increasing compute at a given depth.

B. What are the trade-offs between parameters sharing and Transformer stacking?

  • 39% fewer non-embedding parameters at L=2 and 66% fewer at L=4 are achieved by sharing, with the same compute cost as stacking at each depth.Sharing keeps 306k non-embedding parameters at every L, whereas stacking adds about 198k parameters per layer.Source: B. What are the trade-offs between… · p. 9, Table V · p. 9
  • 98.06% macro F1-score at L=1 on Edge-IIoTset changes by less than 0.05 points at L=2, and falls at L=4 under both strategies.The L=4 training curves were still improving when the 10-epoch budget ended, so the decrease may reflect insufficient training.Source: B. What are the trade-offs between… · p. 9
  • On CICIoT2023, macro F1-scores range from 93.20% to 93.59%; increasing L from 1 to 4 improves scores by 0.35 points with sharing and 0.39 with stacking.Recon-OSScan, Recon-PortScan, and DNS Spoofing remain difficult to detect at every L.Source: B. What are the trade-offs between… · p. 9

C. Can a compact model match the detection perfor-

  • The proposed architecture ranks favorably against Edge-IIoTset baselines: conventional machine learning and standard deep neural networks remain below 78% F1-score, while Transformer-based approaches perform better.SecurityBERT-based models require about 11.2 M parameters, compared with about 1.6 M for the proposed architecture.Source: C. Can a compact model match the… · pp. 9–10, Table VI · p. 10
  • Without PPFLE, the model reaches 99.9% accuracy and 98.06% macro F1-score; with PPFLE, it reaches 98.88% accuracy and 94.55% macro F1-score.PPFLE is also the input encoding used by SecurityBERT.Source: C. Can a compact model match the… · p. 10
  • On both datasets, the models perform similarly on most per-class results; both achieve the same binary intrusion-detection performance.Fingerprinting is difficult for both models on Edge-IIoTset, while SecurityBERT has an approximately 1.71% advantage on benign traffic in CICIoT2023.Source: C. Can a compact model match the… · p. 10
  • 98.88% accuracy and 94.55% macro F1-score on Edge-IIoTset with PPFLE outperform SecurityBERT, while CICIoT2023 is within about 0.7 percentage points of its macro F1-score.The proposed model uses 1.6 M parameters and 6 MB, versus SecurityBERT's 11.2 M parameters and 45 MB; simulated-edge batch latency is 9.07 ms versus 810.69 ms.Source: C. Can a compact model match the… · p. 10

D. Discussion

The proposed model remains competitive with SecurityBERT while using fewer parameters and less inference time. The discussion finds that moderate compression preserves detection performance, parameter sharing reduces model size when depth is needed, and evaluation remains limited to two IoT datasets.

D. Discussion

  • e128M128 reaches 98.06% macro F1-score, close to e256M64, which requires about 3.4× more core parameters and 1.5× more compute.The most compact e64M64 reduces compute by 3.2× but loses 2.4 macro F1 points and is less stable across seeds; all configurations exceed 98% accuracy, which does not reveal these differences.Source: D. Discussion · p. 12
  • Parameter sharing and stacking differ by at most 0.33 macro F1 points and have identical compute and peak training memory at a given depth, while sharing holds non-embedding parameters at 306k.Sharing reduces stored model size by 39% at L=2 and 66% at L=4; additional depth brought little benefit, and one layer is sufficient in this setting.Source: D. Discussion · p. 12
  • The evaluation covers two IoT datasets, and the authors caution that results may not generalize to broader network environments.Latency was measured on a simulated edge device; some classes remained difficult to detect, and deeper L=4 configurations had not converged after at most 10 training epochs.Source: D. Discussion · p. 12
  • On Edge-IIoTset, the model exceeds SecurityBERT by about 2 macro F1 points and is within 0.7 points on CICIoT2023, while using about 7× fewer parameters and running about 89× faster.The comparison uses the same privacy-preserving encoding; Table IX reports model size and inference efficiency for batch size 10.Source: Table IX · p. 11, §VI · p. 12, D. Discussion · p. 12

VI. Conclusion

  • The retained e128M128 configuration has 1.6M total parameters, including 306k non-embedding parameters, and a model size of about 6 MB.The authors conclude that this model is well suited to resource-constrained IoT environments and identify physical-hardware validation and additional datasets as future work.Source: §VI · p. 12

Related papers

Loading 2610.30222v1…