Source-linked AI summary
Adapter-Based Few-Shot Continual Learning for Malicious Packet Recognition
Kyle Stein, Guillermo Francia, III Eman El-Sheikh, Andrew Arash Mahyari
TL;DR
Malware FSCIL must learn emerging classes from scarce labels while avoiding catastrophic forgetting and the security risks of retaining raw samples. The paper addresses this with domain-specific self-supervised pre-training, LoRA adaptation, and rehearsal-free prototype classification, and reports improved performance and stability over prior baselines. The main scope boundary is that frozen features may limit plasticity under substantial domain shift.
Problem
Malware FSCIL remains comparatively underexplored despite requiring adaptation to new malware classes from few labeled examples without catastrophic forgetting or retained samples.
Method
The framework pre-trains a malware-packet transformer with self-supervision, uses LoRA while freezing the backbone, and applies prototype-based classification during incremental sessions.
Results
The method outperforms existing baselines on CIC-IDS2017 and UNSW-NB15, improving final-session accuracy by +8.75% and +8.15% against SOA rehearsal-free baselines.
Takeaways & Limitations
A robust frozen feature space and targeted parameter tuning support rehearsal-free adaptive malware detection when data retention is prohibited.
Takeaways & Limitations
Frozen features may limit plasticity when novel attacks undergo significant domain shift, and the approach depends on diverse unlabeled malware data and costly initial pre-training.
Abstract
from arXiv · showhide
The continual evolution of malware variants necessitates detection systems that can adapt to new threats without retraining from scratch. However, continually updating models on new data often leads to catastrophic forgetting, where previously learned knowledge is overwritten. While continual learning has been increasingly explored for malware detection, the specific setting of Few-Shot Class-Incremental Learning (FSCIL), where new malware classes must be learned from only a small number of labeled examples, remains comparatively underexplored. Therefore, this work investigates the FSCIL setting for malware classification. To address the stability-plasticity dilemma, we propose a hybrid framework that leverages a Self-Supervised Learning (SSL) backbone initialized through domain-specific pre-training on malware packets. Our method incorporates Low-Rank Adaptation (LoRA) to efficiently adapt the model during the base session while freezing the core backbone to preserve previously learned representations, alongside a prototype-based classification head for incremental sessions to establish robust decision boundaries from limited samples. Extensive experiments across several datasets demonstrate that our approach consistently outperforms prior malware FSCIL baselines and achieves state-of-the-art performance.
I. INTRODUCTION
The paper frames rehearsal-free FSCIL for malware detection as a response to evolving threats, scarce labels, catastrophic forgetting, and security risks from retaining malware samples. It proposes domain-specific self-supervision, LoRA adaptation, and prototype classification, with empirical evaluation across benchmark datasets.
- Evolving malware families require classifiers that adapt to novel attack patterns without full retraining or large labeled datasets.
- FSCIL extends continual learning by requiring adaptation to new classes from only a few labeled samples while preserving previously learned knowledge.
- Rehearsal-free learning supports immediate disposal of malware samples, avoiding privacy, security, accidental-execution, and leakage risks from persistent buffers.
- The framework pre-trains a transformer on malware packets, adapts it with LoRA, and classifies incremental classes using prototypes.
- The study evaluates fine-grained attack subtypes as independent classes and reports state-of-the-art performance across multiple benchmark datasets.
II. RELATED WORK
Related work describes a shift from rehearsal and full fine-tuning toward frozen pre-trained models with parameter-efficient adaptation. The paper positions adapter-based methods as a way to reduce forgetting while avoiding prompt-optimization difficulties and malware-sample retention.
- FSCIL research has moved from metric learning and full fine-tuning toward parameter-efficient adaptation of frozen pre-trained transformers.
- Rehearsal preserves decision boundaries but storing raw malware creates security and privacy risks, motivating rehearsal-free approaches based on pre-trained models.
B. Continual Learning for Malware Classification
Malware continual-learning methods do not always transfer cleanly from computer vision, motivating malware-specific FSCIL research. FSCIL evaluates cumulative classes over sequential sessions while later classes provide only scarce examples.
- Many continual-learning techniques developed for computer vision fail to prevent catastrophic forgetting on malware datasets.
- The paper combines domain-specific self-supervised pre-training with adapter-based FSCIL for rehearsal-free adaptation under scarce incremental data.
- FSCIL begins with a data-rich base session and then introduces new classes sequentially with scarce examples.
- After each session, evaluation uses the cumulative label set, requiring new-class learning without catastrophic forgetting of earlier classes.
IV. PROPOSED METHOD
The proposed framework has three stages: self-supervised transformer pre-training, LoRA-based base-session adaptation, and rehearsal-free prototype classification for novel malware families. Its encoder represents byte sequences using embeddings, positional information, self-attention, and mean pooling.
- The framework performs self-supervised pre-training, LoRA-based parameter-efficient adaptation, and rehearsal-free prototype classification in three stages.
- Self-supervised transformer pre-training learns protocol semantics from raw malware packets before class labels are introduced.
- A. Transformer Encoder: The encoder maps packet bytes into embeddings, adds positional encodings, and uses multi-head self-attention to model contextual dependencies.
- A. Transformer Encoder: Mean pooling over final hidden states produces a fixed-size global representation vector for each packet.
B. Self-Supervised Pre-training
The backbone learns general malware-packet representations from unlabeled bytes through masked language modeling, producing a feature extractor for later family-specific adaptation.
- Masked Language Modeling masks 15% of packet positions and requires reconstructing masked byte values from surrounding context.The objective uses contextual information from unmasked bytes to learn packet representations without explicit class labels.
- The model projects masked-position hidden states into the byte-token vocabulary and minimizes cross-entropy against the original byte indices.
- The resulting feature extractor is trained on malicious packet bytes and then adapted to malware families with LoRA during the base session.AdamW, a 1 × 10−4 learning rate, and weight decay are used during pre-training.
C. Base Session Adaptation with LoRA
Base-session adaptation uses LoRA to specialize the pre-trained packet encoder while freezing its original weights, reducing the risk of forgetting under few-shot updates.
- LoRA freezes pre-trained weights and injects trainable low-rank matrices into transformer attention layers for efficient adaptation.
- For a pre-trained weight matrix W0, LoRA decomposes the update ΔW into low-rank factors rather than replacing the full matrix.
- The adaptation rank is set to 8, constraining task-specific updates while requiring fewer trainable parameters than full weight-matrix updates.LoRA is applied to the query, key, and value projections of the transformer packet encoder.
D. Rehearsal-Free Incremental Learning
Incremental sessions retain a frozen feature space and classify novel malware families with normalized prototypes and scaled cosine similarities, avoiding replay and further gradient updates.
- Continuing model updates during few-shot incremental sessions can cause feature drift and degrade previously learned decision boundaries.
- The method averages newly introduced-class features to construct lightweight prototypes without storing privacy-sensitive historical samples.The frozen feature extractor is established through SSL pre-training and base-session LoRA adaptation.
- Prototype inference uses normalized embeddings and a scaled cosine-similarity classifier rather than Euclidean distance.The query embedding is obtained by normalizing the frozen extractor output.
- Logits are converted to class probabilities with softmax, and the predicted label is the class with maximum probability.
E. Rehearsal-Based Incremental Learning
The rehearsal-based variant stores exemplars and continues updating parameters with current and replayed samples, providing a comparison against the proposed static-feature-space approach.
- The rehearsal-based variant stores a small exemplar buffer and mixes previously seen samples with new few-shot data during incremental training.Unlike the proposed method, it continues parameter updates in incremental sessions.
V. EXPERIMENTAL RESULTS
The evaluation follows standard 5-shot FSCIL protocols on CIC-IDS2017 and UNSW-NB15, using base and sequential incremental sessions. Performance is measured cumulatively across observed classes with accuracy and average forgetting, averaged over three seeds.
- Datasets and Metrics: Experiments use CIC-IDS2017 and UNSW-NB15 with a robust base session followed by sequential incremental sessions.SSL pre-training uses the Session 0 split to initialize the encoder.
- Datasets and Metrics: Evaluation uses a cumulative test set containing query samples from all classes observed through the current session.
- Datasets and Metrics: Results are averaged over three random seeds using cumulative accuracy and Average Forgetting.Average Forgetting measures the decline in knowledge retention for previously learned classes.
A. Main Experimental Results
The proposed rehearsal-free method outperforms rehearsal-free baselines on both datasets and substantially reduces forgetting. Rehearsal-based and rehearsal-free variants achieve comparable final accuracy, but the rehearsal-free design provides greater stability without retaining samples.
- Main Results: +8.75% final-session accuracy on CIC-IDS2017 and +8.15% on UNSW-NB15 versus SOA rehearsal-free baselines.The corresponding accuracies are 61.05% versus 52.30% and 58.59% versus 50.44%, respectively.
- Main Results: The proposed method reduces average forgetting by roughly half compared with MalFSCIL.The authors attribute the stability to SSL pre-training combined with LoRA adapters.
- Stability–Plasticity Trade-off: Comparable final CIC-IDS2017 accuracy is achieved by Rehearsal-Based Buffer=5 and Rehearsal-Free Buffer=0 variants: 61.05% versus 59.54%.
- Stability–Plasticity Trade-off: Average forgetting is 19.95% with rehearsal and 8.59% without rehearsal on CIC-IDS2017.The rehearsal-based variant updates LoRA weights, whereas the rehearsal-free variant freezes the weights during incremental sessions.
- Stability–Plasticity Trade-off: The rehearsal-free approach avoids the security risks associated with storing a replay buffer while achieving superior or equivalent overall performance.
VI. LIMITATIONS
The approach has computational, data-dependence, and scope limitations. Its frozen feature space may restrict adaptation under substantial domain shift, and continual learning is less suitable when malware families change infrequently.
- Limitations: Self-supervised pre-training from scratch incurs significant initial computational costs and depends on the diversity of the unlabeled malware corpus.SSL removes the need for labeled data during pre-training, but corpus quality remains important.
- Limitations: Freezing the feature space assumes that the pre-trained backbone and base-session adapters remain discriminative for future malware variants.
- Limitations: Under significant domain shift, the frozen encoder may limit plasticity compared with methods that continuously fine-tune the backbone.
- Limitations: In static environments where malware families change infrequently, a continual-learning system may be more complex than a simpler static classifier.
- Limitations: Experiments inject diverse incremental classes to assess robustness to significant semantic shifts.