PaperlayerView exact revision on arXiv

Source-linked AI summary

ESL-former: Event Sparse Linear Transformer for FPGA-Accelerated Malware Detection

Jun Yang, Yumeng Cao, Yingxuan Xiao, Changhong Li, Shangru Zhou

arXiv:2610.19665v1cs.AR

TL;DR

High computational and memory costs make Transformer-based malware detection difficult to deploy on resource-constrained edge platforms. ESL-former scores tokens and channels from Linear Transformer’s accumulated key-value state and reorganizes selected data for hardware execution; it reports up to 41.56× end-to-end speedup over ViT4Mal while maintaining competitive detection performance.

  • Problem

    Transformer self-attention’s quadratic computation and memory costs complicate malware detection on resource-constrained edge devices.

    Source: §1 · pp. 1–2

  • Method

    ESL-former scores token and channel contributions from Linear Transformer’s accumulated key-value state and uses Top-K buffering and channel compaction for sparse execution.

    Source: §1 · p. 3

  • Results

    Up to 41.56× end-to-end speedup over ViT4Mal while maintaining competitive detection performance.

    Source: §1 · p. 3, §5 · p. 18

  • Takeaways & Limitations

    On PYNQ-Z2, event sparsity achieves up to 1.20× measured kernel speedup over mask-based sparsity under the same token selection results.

    Source: §5 · p. 18

Abstract

from arXiv · show

Transformer based intrusion detection has shown strong potential for identifying unknown malware variants, but its deployment on resource constrained edge devices remains difficult due to the high computational and memory cost of self attention. This paper presents an event sparse Linear Transformer framework for malware detection on FPGA edge platforms. Instead of relying on an explicit attention matrix, the proposed method derives token and channel importance directly from the accumulated key value state of Linear Transformer. A contribution aware scoring function is introduced to measure the perturbation of each token or kernel channel to the linear attention state, allowing redundant tokens and low contribution channels to be removed without constructing quadratic attention maps. To translate sparsity into practical hardware acceleration, the proposed design further replaces conventional binary mask based execution with a Top K active token buffer and a hardware compaction unit. The selected active tokens and channels are reorganized into dense fixed length data streams, enabling downstream processing elements to execute regular multiply accumulate operations with reduced redundant memory access and control overhead. This design is suitable for low power FPGA based malware detection, where both latency and on chip resource usage are critical. Experimental evaluation demonstrates that the proposed sparse Linear Transformer can reduce hardware cost while preserving malware classification accuracy.

1 Introduction

Transformer deployment for malware detection on edge platforms is constrained by attention’s computational and memory costs, while existing sparse execution can retain hardware overhead. ESL-former derives token and channel importance from Linear Transformer’s key-value state and uses hardware-oriented sparse execution.

1 Introduction

  • Standard self-attention’s computation and memory costs grow quadratically with sequence length, complicating Transformer deployment on resource-constrained edge platforms.Existing mask-based or irregular sparse execution can add control overhead, memory accesses, and index handling on small edge FPGAs.Source: §1 · p. 2
  • ESL-former scores token and channel contributions from the accumulated key-value state, removing low-contribution elements without constructing quadratic attention maps.The scoring is applied after kernel feature mapping and before the key-value state update.Source: §1 · p. 3
  • Top-K active-token buffering and channel compaction replace binary mask-based sparsity with hardware-oriented sparse execution.The paper’s contributions also include a contribution-aware token and channel scoring method derived from Linear Transformer state accumulation.Source: §1 · p. 3
  • Up to 41.56× end-to-end speedup over ViT4Mal while maintaining competitive detection performance.The evaluation implements ESL-former on the PYNQ-Z2 FPGA platform for IMG_DS malware image detection.Source: §1 · p. 3

2 Related Works

Prior work explores FPGA security acceleration and efficient or sparse Transformer execution, but small edge-FPGA malware detection still faces memory, control, and execution-efficiency constraints. ESL-former couples sparsity to Linear Transformer’s key-value state and reorganizes selected data for hardware execution.

2.1 FPGA Accelerated Malware Detection

  • FPGA security work spans rule-based packet inspection and machine-learning malware or intrusion detection, but the reviewed methods leave FPGA-specific resource and execution constraints insufficiently addressed.Examples include graph neural networks [10], LSTM, XGBoost ransomware detection, OmBNNet, and MalViT.Source: §2.1 · pp. 3–4

2.2 Efficient Transformer Models and FPGA Deployment

  • Transformer acceleration on FPGA uses attention approximation, dataflow optimization, quantisation, and specialized mapping, yet small edge security platforms retain memory and control bottlenecks.The related work includes SWAT, FlightLLM, SATNet, cosFormer [3], and FastViT.Source: §2.2 · pp. 4–5

2.3 Sparse Transformer Acceleration

  • Sparse Transformer methods use attention patterns, token redundancy, or weight sparsity, but masks and irregular execution can add control and memory overhead on FPGA.DynamicViT and EViT [20] rely on standard-attention-linked scores, which do not directly transfer to Linear Transformer’s accumulated key-value state.Source: §2.3 · pp. 5–6

2.4 Position of This Work

  • ESL-former derives token and channel importance from Linear Transformer key-value state updates and compacts selected activations into dense fixed-length streams for regular FPGA processing.Downstream processing elements can use fewer effective loop iterations and reduced memory traffic.Source: §2.4 · p. 6

3 Methodology

The methodology adapts Linear Transformer attention for resource-constrained malware detection and defines token and channel sparsity from contributions to its key-value state. It then compacts selected tokens into dense data for more regular FPGA execution.

3 Methodology

  • For IMG_DS, images are resized to 32 × 32, divided into 8 × 8 patches, and encoded as a 16-token sequence before binary malware classification.Each patch is a 64-dimensional token projected to hidden/kernel-feature width d = 16; the decoder uses an MLP and Softmax.Source: §3 · pp. 6–7

3.1 Linear Transformer

  • Linear Transformer replaces the explicit N × N attention matrix with kernel-based factorization and a compact key-value state, reducing dominant computation from O(N^2d) to O(Nd^2).The compact state is reusable across query tokens.Source: §3.1 · pp. 7–8

3.2 Channel Contribution Importance Evaluation

  • Channel contribution scores select high-contribution Q/K kernel channels, while each token’s score measures the Frobenius norm of its contribution to the accumulated key-value state.The token score is derived from the forward computation rather than an external predictor, and its vector-based decomposition avoids explicitly constructing the outer-product matrix.Source: §3.2 · pp. 8–9

3.3 Token Sparsification and Hardware Compression

  • Mask-based sparsification can retain full-sequence loop and control overhead; Top-K instead selects informative tokens and reorders them into a compact active-token buffer.Scoring and selection still consider the original N candidates, while subsequent state updates and backend computation use the K selected tokens.Source: §3.3 · pp. 9–11, Figure 2 · p. 10
  • The hardware compaction unit gathers selected tokens and channels into a dense fixed-length block, allowing downstream processing elements to execute continuous dense operations.The resulting loop bound is K rather than N, reducing redundant iterations and memory traffic.Source: §3.3 · p. 11

4 Results and Discussion

Across four Token-Channel configurations on IMG_DS, the PYNQ-Z2 experiments assess detection performance, FPGA resources, and latency. Event sparsity reduces kernel and end-to-end latency, with the strongest reported speedup over ViT4Mal at the most compressed configuration.

4.1 Experimental Setup

  • The IMG_DS evaluation uses 14,733 malware and 2,486 benign samples, with N = 16 tokens and d = 16 kernel-feature width; evaluated retention settings range from K16/D16 to K11/D8.K denotes retained active tokens and D retained Q/K kernel-feature channels; identities are dynamically selected for each input.Source: §4.1 · pp. 11–12

4.2 Case Study 1: Sparse Unit Ablation

  • Q16 is selected for subsequent token-channel sparse experiments: latency falls from 3166.3 µs at Q32 to 1146.1 µs, with less than 1 percentage point accuracy degradation.Q8 further reduces latency but has 93.00% accuracy, a larger degradation than Q16.Source: §4.2 · p. 12, Table 1 · p. 12
  • At K11/D8, Event Sparse reduces kernel latency by 16.4% versus mask-based sparsity, whereas masking from K16/D16 to K11/D8 reduces latency by only 36 µs.Event Sparse has a 6.6% latency overhead at full retention, but its advantage grows as effective tokens decrease.Source: §4.2 · p. 13, Figure 3 · p. 13

4.3 Case Study 2: On-Board Hardware Performance Evaluation

  • K16/D16, K14/D13, and K13/D10 retain stable detection performance; K11/D8 shows lower Accuracy and F1 at the higher compression level.Fig. 4 presents detection performance and FPGA resource usage under different event-sparse configurations.Source: §4.3 · p. 14, Figure 4 · p. 15
  • As compression increases, LUT utilization falls to 0.78 and FF utilization to 0.80 at K11/D8, while BRAM remains approximately 0.53.DSP and BRAM usage are reported as relatively stable across configurations.Source: §4.3 · p. 14, Figure 4 · p. 15
  • Across K16/D16 to K11/D8, total end-to-end latency falls from 26272.0 µs to 19486.9 µs, while PL Kernel latency falls from 1146.1 µs to 868.4 µs.Input Preparation also varies, whereas PS→PL and PL→PS transfer stages remain relatively stable.Source: Table 2 · p. 14, §4.3 · p. 15

4.4 Case Study 3: Malware Detection Performance Analysis

  • Across the reported Token-Channel sparse configurations, K16/D16 achieves 94.13% Accuracy and 96.53% F1; at more aggressive K11/D8 sparsity, these fall to 92.78% and 95.08%, respectively.Source: §4.4 · p. 16
  • Compared with ViT4Mal, the four proposed configurations achieve end-to-end speedups of 30.83×, 36.88×, 39.90×, and 41.56×, respectively.Source: §4.4 · p. 17

5 Conclusion

ESL-former targets Transformer malware detection on resource-constrained edge FPGAs. On IMG_DS and PYNQ-Z2, it reports lower latency and BRAM usage than the FPGA-based ViT4Mal baseline, with competitive accuracy.

5 Conclusion

  • ESL-former is presented as an event-sparse Linear Transformer framework for FPGA-accelerated malware image detection addressing deployment on resource-constrained edge FPGAs.Future work will focus on streaming inference and evaluation on larger FPGA platforms and additional malware datasets.Source: §5 · pp. 17–18
  • K13/D10 reduces end-to-end latency from 810 ms to 20.30 ms and BRAM usage from 92 to 5, while improving accuracy from 92.41% to 93.26% versus FPGA-based ViT4Mal.Evaluated on IMG_DS using the PYNQ-Z2 platform.Source: §5 · p. 18
  • Event sparsity achieves up to 1.20× measured kernel speedup over mask-based sparsity under the same token selection results.The paper attributes the latency gain to reduced redundant hardware execution overhead.Source: §5 · p. 18
  • K11/D8 achieves up to 41.56× end-to-end latency speedup while maintaining 92.78% accuracy.Reported as an edge deployment configuration.Source: §5 · p. 18
Loading 2610.19665v1…