PaperlayerView exact revision on arXiv

Source-linked AI summary

Adapter-Based Few-Shot Continual Learning for Malicious Packet Recognition

Kyle Stein, Guillermo Francia, III Eman El-Sheikh, Andrew Arash Mahyari

arXiv:2608.23536v1cs.CRcs.AI

TL;DR

Malware FSCIL must learn emerging classes from scarce labels while avoiding catastrophic forgetting and the security risks of retaining raw samples. The paper addresses this with domain-specific self-supervised pre-training, LoRA adaptation, and rehearsal-free prototype classification, and reports improved performance and stability over prior baselines. The main scope boundary is that frozen features may limit plasticity under substantial domain shift.

  • Problem

    Malware FSCIL remains comparatively underexplored despite requiring adaptation to new malware classes from few labeled examples without catastrophic forgetting or retained samples.

    Source: §I · p. 1

  • Method

    The framework pre-trains a malware-packet transformer with self-supervision, uses LoRA while freezing the backbone, and applies prototype-based classification during incremental sessions.

    Source: §I · pp. 1–2, C. Base Session Adaptation with LoRA · p. 3, B. Self-Supervised Pre-training · p. 3

  • Results

    The method outperforms existing baselines on CIC-IDS2017 and UNSW-NB15, improving final-session accuracy by +8.75% and +8.15% against SOA rehearsal-free baselines.

    Source: A. Main Experimental Results · p. 4

  • Takeaways & Limitations

    A robust frozen feature space and targeted parameter tuning support rehearsal-free adaptive malware detection when data retention is prohibited.

    Source: §VII · p. 5

    Frozen features may limit plasticity when novel attacks undergo significant domain shift, and the approach depends on diverse unlabeled malware data and costly initial pre-training.

    Source: §VI · p. 5

Abstract

from arXiv · show

The continual evolution of malware variants necessitates detection systems that can adapt to new threats without retraining from scratch. However, continually updating models on new data often leads to catastrophic forgetting, where previously learned knowledge is overwritten. While continual learning has been increasingly explored for malware detection, the specific setting of Few-Shot Class-Incremental Learning (FSCIL), where new malware classes must be learned from only a small number of labeled examples, remains comparatively underexplored. Therefore, this work investigates the FSCIL setting for malware classification. To address the stability-plasticity dilemma, we propose a hybrid framework that leverages a Self-Supervised Learning (SSL) backbone initialized through domain-specific pre-training on malware packets. Our method incorporates Low-Rank Adaptation (LoRA) to efficiently adapt the model during the base session while freezing the core backbone to preserve previously learned representations, alongside a prototype-based classification head for incremental sessions to establish robust decision boundaries from limited samples. Extensive experiments across several datasets demonstrate that our approach consistently outperforms prior malware FSCIL baselines and achieves state-of-the-art performance.

I. INTRODUCTION

The paper frames rehearsal-free FSCIL for malware detection as a response to evolving threats, scarce labels, catastrophic forgetting, and security risks from retaining malware samples. It proposes domain-specific self-supervision, LoRA adaptation, and prototype classification, with empirical evaluation across benchmark datasets.

  • Evolving malware families require classifiers that adapt to novel attack patterns without full retraining or large labeled datasets.Source: §I · p. 1
  • FSCIL extends continual learning by requiring adaptation to new classes from only a few labeled samples while preserving previously learned knowledge.Source: §I · p. 1
  • Rehearsal-free learning supports immediate disposal of malware samples, avoiding privacy, security, accidental-execution, and leakage risks from persistent buffers.Source: §I · p. 1
  • The framework pre-trains a transformer on malware packets, adapts it with LoRA, and classifies incremental classes using prototypes.Source: §I · pp. 1–2
  • The study evaluates fine-grained attack subtypes as independent classes and reports state-of-the-art performance across multiple benchmark datasets.Source: §I · pp. 1–2

II. RELATED WORK

Related work describes a shift from rehearsal and full fine-tuning toward frozen pre-trained models with parameter-efficient adaptation. The paper positions adapter-based methods as a way to reduce forgetting while avoiding prompt-optimization difficulties and malware-sample retention.

B. Continual Learning for Malware Classification

Malware continual-learning methods do not always transfer cleanly from computer vision, motivating malware-specific FSCIL research. FSCIL evaluates cumulative classes over sequential sessions while later classes provide only scarce examples.

IV. PROPOSED METHOD

The proposed framework has three stages: self-supervised transformer pre-training, LoRA-based base-session adaptation, and rehearsal-free prototype classification for novel malware families. Its encoder represents byte sequences using embeddings, positional information, self-attention, and mean pooling.

  • The framework performs self-supervised pre-training, LoRA-based parameter-efficient adaptation, and rehearsal-free prototype classification in three stages.Source: §IV · p. 2
  • Self-supervised transformer pre-training learns protocol semantics from raw malware packets before class labels are introduced.Source: A. Transformer Encoder · p. 2

A. Transformer Encoder

  • The encoder maps packet bytes into embeddings, adds positional encodings, and uses multi-head self-attention to model contextual dependencies.Source: A. Transformer Encoder · pp. 2–3
  • Mean pooling over final hidden states produces a fixed-size global representation vector for each packet.Source: A. Transformer Encoder · p. 3

B. Self-Supervised Pre-training

The backbone learns general malware-packet representations from unlabeled bytes through masked language modeling, producing a feature extractor for later family-specific adaptation.

  • Masked Language Modeling masks 15% of packet positions and requires reconstructing masked byte values from surrounding context.The objective uses contextual information from unmasked bytes to learn packet representations without explicit class labels.Source: B. Self-Supervised Pre-training · p. 3
  • The model projects masked-position hidden states into the byte-token vocabulary and minimizes cross-entropy against the original byte indices.Source: B. Self-Supervised Pre-training · p. 3
  • The resulting feature extractor is trained on malicious packet bytes and then adapted to malware families with LoRA during the base session.AdamW, a 1 × 10−4 learning rate, and weight decay are used during pre-training.Source: B. Self-Supervised Pre-training · p. 3

C. Base Session Adaptation with LoRA

Base-session adaptation uses LoRA to specialize the pre-trained packet encoder while freezing its original weights, reducing the risk of forgetting under few-shot updates.

D. Rehearsal-Free Incremental Learning

Incremental sessions retain a frozen feature space and classify novel malware families with normalized prototypes and scaled cosine similarities, avoiding replay and further gradient updates.

E. Rehearsal-Based Incremental Learning

The rehearsal-based variant stores exemplars and continues updating parameters with current and replayed samples, providing a comparison against the proposed static-feature-space approach.

  • The rehearsal-based variant stores a small exemplar buffer and mixes previously seen samples with new few-shot data during incremental training.Unlike the proposed method, it continues parameter updates in incremental sessions.Source: E. Rehearsal-Based Incremental Learning · p. 4

V. EXPERIMENTAL RESULTS

The evaluation follows standard 5-shot FSCIL protocols on CIC-IDS2017 and UNSW-NB15, using base and sequential incremental sessions. Performance is measured cumulatively across observed classes with accuracy and average forgetting, averaged over three seeds.

Datasets and Metrics

  • Experiments use CIC-IDS2017 and UNSW-NB15 with a robust base session followed by sequential incremental sessions.SSL pre-training uses the Session 0 split to initialize the encoder.Source: §V · p. 4
  • Evaluation uses a cumulative test set containing query samples from all classes observed through the current session.Source: §V · p. 4
  • Results are averaged over three random seeds using cumulative accuracy and Average Forgetting.Average Forgetting measures the decline in knowledge retention for previously learned classes.Source: §V · p. 4

A. Main Experimental Results

The proposed rehearsal-free method outperforms rehearsal-free baselines on both datasets and substantially reduces forgetting. Rehearsal-based and rehearsal-free variants achieve comparable final accuracy, but the rehearsal-free design provides greater stability without retaining samples.

Main Results

  • +8.75% final-session accuracy on CIC-IDS2017 and +8.15% on UNSW-NB15 versus SOA rehearsal-free baselines.The corresponding accuracies are 61.05% versus 52.30% and 58.59% versus 50.44%, respectively.Source: A. Main Experimental Results · p. 4
  • The proposed method reduces average forgetting by roughly half compared with MalFSCIL.The authors attribute the stability to SSL pre-training combined with LoRA adapters.Source: A. Main Experimental Results · p. 4

Stability–Plasticity Trade-off

  • Comparable final CIC-IDS2017 accuracy is achieved by Rehearsal-Based Buffer=5 and Rehearsal-Free Buffer=0 variants: 61.05% versus 59.54%.Source: UNSW-NB15 · p. 5
  • Average forgetting is 19.95% with rehearsal and 8.59% without rehearsal on CIC-IDS2017.The rehearsal-based variant updates LoRA weights, whereas the rehearsal-free variant freezes the weights during incremental sessions.Source: UNSW-NB15 · p. 5
  • The rehearsal-free approach avoids the security risks associated with storing a replay buffer while achieving superior or equivalent overall performance.Source: UNSW-NB15 · p. 5

VI. LIMITATIONS

The approach has computational, data-dependence, and scope limitations. Its frozen feature space may restrict adaptation under substantial domain shift, and continual learning is less suitable when malware families change infrequently.

Limitations

  • Self-supervised pre-training from scratch incurs significant initial computational costs and depends on the diversity of the unlabeled malware corpus.SSL removes the need for labeled data during pre-training, but corpus quality remains important.Source: §VI · p. 5
  • Freezing the feature space assumes that the pre-trained backbone and base-session adapters remain discriminative for future malware variants.Source: §VI · p. 5
  • Under significant domain shift, the frozen encoder may limit plasticity compared with methods that continuously fine-tune the backbone.Source: §VI · p. 5
  • In static environments where malware families change infrequently, a continual-learning system may be more complex than a simpler static classifier.Source: §VI · p. 5
  • Experiments inject diverse incremental classes to assess robustness to significant semantic shifts.Source: §VI · p. 5

Related papers

Loading 2608.23536v1…