Source-linked AI summary
Bytes to Threats: Byte-Sequence Transformer For Malware Detection
Abdelrahman Yassin Eljamal, Abdeltawab Hendawi
TL;DR
Handcrafted-feature malware detection requires domain expertise and specialized analysis, and capturing long-range dependencies in executable bytes remains challenging for CNN-based methods. MalwareFormer fine-tunes a Transformer on raw bytes, without handcrafted features. It outperforms two CNN baselines across threshold-dependent metrics, achieving 98.35% accuracy and a 1.94% false positive rate.
Problem
Existing malware-detection approaches rely heavily on handcrafted executable features that require domain expertise, while CNN filters have limitations capturing long-range dependencies.
Source: §I · p. 1, §VI · p. 6Method
MalwareFormer fine-tunes a Transformer-based ByteFormer model to detect malware directly from raw executable bytes without handcrafted features or domain expertise.
Source: §I · p. 1Results
At 50K-byte input, MalwareFormer achieves 98.35% accuracy, 98.32% balanced accuracy, 98.32% macro-F1, and a 1.94% false positive rate, outperforming both CNN baselines across threshold-dependent metrics.
Source: §I · p. 2Takeaways & Limitations
MalwareFormer demonstrates a transformer-based approach to byte-level malware detection that avoids handcrafted features and is reported to perform consistently better than CNN baselines on threshold-dependent metrics.
Source: §VII · p. 6
Abstract
from arXiv · showhide
Existing malware detection approaches heavily rely on handcrafted features derived from static or dynamic analysis of the executable files. These methods require specialized domain expertise, dedicated analysis tools, and time-consuming analysis, which limits their scalability and robustness in the face of the increasing complexity and diversity of modern malware. Consequently, there is a growing interest in approaches that operate directly on raw bytes, which contain rich semantic and structural information. However, capturing long-range dependencies within executable files remains challenging for conventional static and CNN-based methods. This paper presents MalwareFormer, a transformer-based, byte-level malware detection framework that operates directly on raw bytes, without requiring domain knowledge or feature extraction. MalwareFormer achieves an accuracy of 98.35%, a balanced accuracy of 98.32%, a macro-F1 of 98.32%, and a false positive rate of 1.94%, outperforming both baselines across all threshold-dependent classification metrics and demonstrating that transformers provide a powerful and scalable approach for byte-level malware detection.
I. INTRODUCTION
Handcrafted-feature malware detection depends on expertise and specialized tools, while CNNs can struggle to capture long-range byte dependencies. MalwareFormer applies a transformer directly to raw bytes and is reported to outperform two CNN baselines across threshold-dependent metrics.
I. INTRODUCTION
- Existing approaches rely on expert-derived executable features; dynamic analysis can be slow and resource-intensive, while static analysis still requires handcrafted features and specialized tools,.CNN-based models can learn from raw bytes, but local convolutional filters limit their ability to capture long-range dependencies.Source: §I · p. 1
- MalwareFormer operates directly on raw executable bytes without handcrafted features or domain expertise, fine-tuning ByteFormer to capture long-range dependencies.The paper compares fine-tuning with training from scratch and examines convergence, performance, and varying input sequence lengths.Source: §I · pp. 1–2
- At 50K-byte input, MalwareFormer achieves 98.35% accuracy, 98.32% balanced accuracy, 98.32% macro-F1, and a 1.94% false positive rate, outperforming both CNN baselines across threshold-dependent metrics.MalConv reports 95.71% accuracy, 95.64% macro-F1, and 3.56% false positive rate; AvastNet reports 97.98% accuracy, 97.94% macro-F1, and 2.46% false positive rate.Source: §I · p. 2
- The paper investigates whether pretrained ByteFormer representations transfer to malware detection and whether accurate classification is possible with truncated executable inputs.Its research questions also examine CNN-baseline effectiveness and entropy-related error patterns.Source: §I · p. 2
- In high-entropy regions (entropy 7–8), the reported false positive rate reaches 14.24% while the false negative rate falls to 0.08%.The authors describe this asymmetry as a limitation of byte-level detection in entropy-ambiguous regimes and motivate future entropy-aware approaches.Source: §I · p. 2
II. RELATED WORK
Related work reviews malware detection from raw bytes, beginning with CNN approaches and moving toward Transformer architectures.
II. RELATED WORK
- The review is organized around raw-byte malware detection, progressing from CNN-based approaches to more recent Transformer architectures.Source: §II · p. 2
A. CNN Based Models
Prior work established that CNNs can learn malware representations from raw executable bytes, while hybrid CNN–LSTM models combine local and sequential pattern modeling.
A. CNN Based Models
- MalConv processes Windows PE files as byte sequences using gated convolution, global max pooling, and a classifier; later CNN work studied deeper architectures and raw-byte generalization.Memory-efficient sequence-classification methods addressed inputs that can exceed several million bytes [8].Source: A. CNN Based Models · p. 2
- A hybrid approach combined CNN layers for local feature extraction with LSTM layers for long-range dependencies, modeling spatial and sequential byte-stream patterns.Source: A. CNN Based Models · p. 2
B. Transformer Based Models
Transformer malware-detection studies use assembly sequences, graphs, visual encodings, and higher-level features, while most rely on extracted or engineered inputs. This work instead applies a Transformer to raw executable bytes.
B. Transformer Based Models
- Prior Transformer malware methods model assembly instructions, function-call graphs, binary images, manifest information, or network behavior rather than a single common input representation –.A survey noted that most existing approaches rely on engineered or extracted features.Source: B. Transformer Based Models · p. 3
1) Byte Only Transformer Models:
- ByteEnc uses byte-level pretraining objectives and relative positional encodings for non-executable documents, while this work focuses on raw executable bytes and long-range dependencies.ByteEnc targets PDF, HWP, and MS Office formats; this work uses a transformer-based architecture within MalwareFormer.Source: 1) Byte Only Transformer Models · p. 3
III. METHODOLOGY
MalwareFormer’s methodology checks dataset integrity and label consistency, models raw byte sequences, and evaluates classification and entropy-stratified behavior. Its ByteFormer encoder combines token reduction with windowed transformer layers and pooled classification.
III. METHODOLOGY
- The framework verifies dataset integrity and label consistency, feeds raw byte sequences to the model, then evaluates classification metrics and entropy-stratified predictions.Label consistency is cross-checked using VirusTotal; the analysis uses six entropy bins.Source: §III · p. 3, Figure 1 · p. 3
A. Model Overview
- ByteFormer uses positional embeddings, an initial convolutional token-reduction module, and 12 windowed transformer layers with six distributed token-merging operations.After the final transformer block, global mean pooling feeds a linear classification head.Source: A. Model Overview · p. 3
B. Dataset
The dataset comprises executable files from multiple public repositories, with benign and malicious examples. Its file-size distributions substantially overlap, while entropy distributions differ in their shape and variability.
B. Dataset
- 201,549 executable files form the dataset, including 86,812 benign and 114,737 malicious files collected from multiple public repositories.Repositories include VirusShare, MalShare, and TheZoo.Source: B. Dataset · p. 3
- The benign and malicious file-size distributions overlap substantially, so file size alone is insufficient to reliably distinguish the classes.Malware includes smaller binaries, but the curves converge and remain close across most of the distribution.Source: B. Dataset · pp. 3–4, Figure 2 · p. 4
- Entropy is computed using Shannon entropy with byte-level probabilities estimated by maximum likelihood, and serves as an indicator of randomness, packing, compression, and encryption.The paper uses entropy to characterize statistical properties of the dataset.Source: B. Dataset · p. 4
- The benign entropy distribution peaks around 5.8 to 6.3, while malware is more dispersed and has localized spikes in the high-entropy tail.The spikes are suggested to reflect multiple malware sub-populations, potentially associated with different families or packing strategies.Source: B. Dataset · p. 4
IV. EXPERIMENTAL SETUP
Experiments compare fine-tuning pretrained ByteFormer with training from scratch and test several input lengths. Baselines are trained in a shared framework using their reported configurations where possible.
IV. EXPERIMENTAL SETUP
- The model is evaluated using the first 50K, 100K, 150K, and 200K bytes of each executable, reflecting dataset file sizes from about 500 bytes to over 2 million bytes.Training strategies include fine-tuning a model pretrained on image bytes and training from scratch on executable files.Source: §IV · p. 4
- MalConv and AvastNet are integrated into CoreNet and trained using their original published hyperparameters whenever possible; all models train for 30 epochs.The baseline configuration uses a 50,000-byte input, Adam, a 1 × 10^-3 learning rate, and batch size 128.Source: §IV · p. 4
- ByteFormer uses AdamW with weight decay 0.05 and a cosine learning-rate schedule, including 1,000 warmup iterations and a peak learning rate of 3 × 10^-5.Batch-normalization layers and bias parameters are excluded from weight decay.Source: §IV · p. 4
V. RESULTS
Fine-tuning converges faster than training from scratch: validation loss falls below 0.1 by about epoch 11, and training loss becomes lower by epoch two. The reported observations associate pretrained representations with faster convergence and improved stability.
V. RESULTS
- Validation loss for the fine-tuned model falls below 0.1 by approximately epoch 11 of the 30-epoch schedule, while the scratch-trained model maintains higher validation loss.The comparison is shown in Fig. 4.Source: §V · p. 4, Figure 4 · p. 5
- Despite starting with higher training loss in epoch one, the fine-tuned model reaches lower training loss than the scratch-trained model by epoch two.The authors describe pretrained representations as beneficial for accelerating convergence and improving model stability.Source: §V · pp. 4–5
A. Effect of Input Sequence Length
MalwareFormer remains stable on truncated byte sequences, with larger inputs offering modest gains. At 50K bytes, it outperforms both CNN baselines across threshold-dependent metrics.
A. Effect of Input Sequence Length
- Increasing fine-tuned input size from 50K to 100K bytes consistently improves accuracy and AUC, while 150K or 200K bytes yield only marginal gains.MalwareFormer also shows stable performance with truncated byte sequences.Source: A. Effect of Input Sequence Length · p. 5, Table I · p. 5
B. Comparison with CNN-Based Malware Detectors
- On the shared test set, MalwareFormer misses 46 fewer malware samples and produces 67 fewer benign-file false positives than AvastNet.MalwareFormer detects 16,966 malware samples with 245 misses and classifies 12,769 benign files with 253 false alarms.Source: B. Comparison with CNN-Based Malware… · p. 5
- At 50K bytes, MalwareFormer achieves 98.35% accuracy, 98.32% balanced accuracy, 98.32% macro-F1, and the lowest FPR at 1.94%, outperforming both CNN baselines on threshold-dependent metrics.The comparison is relevant to deployment, where false alarms and missed detections carry operational costs.Source: Table II · p. 5, B. Comparison with CNN-Based Malware… · p. 5
VI. DISCUSSION
Fine-tuning ByteFormer transfers to executable analysis and converges quickly despite an initial byte-domain mismatch. Entropy-stratified results reveal elevated false positives in the highest-entropy bin, while the architecture combines local and global byte structures.
VI. DISCUSSION
- ByteFormer pretraining provides a strong initialization for malware classification, and the fine-tuned model reaches lower training loss than training from scratch by epoch two.The first-epoch higher loss is consistent with semantic differences between image bytes and executable bytes; pretraining used TIFF byte sequences, which are structured binary containers.Source: §VI · p. 5
- In the 7–8 input-window entropy bin, FPR reaches 14.24% while FNR is 0.08%, and 5,011 samples are predominantly packed or encrypted content.The model over-flags high-entropy benign files such as compressed archives or resource-heavy executables; full-file entropy stratification shows the same asymmetric pattern.Source: §VI · pp. 5–6, Table III · p. 6
- MalwareFormer combines token reduction with windowed self-attention to aggregate local and global byte-level structures.The discussion contrasts this with CNNs' fixed receptive fields and local convolutions, which limit long-range dependency capture.Source: §VI · p. 6
VII. CONCLUSION
The conclusion presents MalwareFormer as a raw-byte transformer for malware detection that avoids handcrafted features and domain expertise. It reports strong performance on 50K-byte inputs, stable results with truncated sequences, and an entropy-related false-positive bias at the highest entropy.
VII. CONCLUSION
- On 50K-byte inputs, MalwareFormer achieves 98.35% accuracy, 98.32% macro-F1, and 1.94% FPR while outperforming CNN baselines across threshold-dependent metrics.Larger byte sequences provide modest gains, but performance remains stable with truncated inputs.Source: §VII · p. 6
- MalwareFormer operates directly on raw executable bytes without handcrafted features or domain expertise, and the paper characterizes transformers as a practical, scalable approach to byte-level malware analysis.The conclusion identifies an entropy-induced false-positive bias in the 7–8 bin, with 0.08% FNR and 14.24% FPR, and points to entropy-aware training and hybrid approaches as future work.Source: §VII · p. 6