PaperlayerView exact revision on arXiv

Source-linked AI summary

Parameter Efficient Continual Learning for Sparse Event-Based Transformers

Vaishnavi Nagabhushana, Kartikay Agrawal, Ayon Borthakur

arXiv:2608.26720v1cs.CV

TL;DR

Continual learning for pretrained sparse event-based transformers is underexplored despite the need for adaptation under strict memory and energy constraints. The paper introduces sLoTh, which combines shared threshold modulation with seLoRA while freezing the backbone and avoiding replay buffers. Across continual-learning benchmarks, sLoTh delivers competitive rehearsal-free performance, updates less than 1% of parameters, and uses approximately 6.5× less energy than dense vision transformers.

  • Problem

    Pretrained sparse event-based transformers have remained largely unexplored for rehearsal-free continual learning, especially in online settings requiring adaptation under memory and energy constraints.

  • Method

    sLoTh freezes the pretrained backbone and combines shared neuronal threshold modulation with scalable-efficient low-rank attention updates, distillation, and regularisation without replay buffers.

  • Results

    sLoTh achieves competitive rehearsal-free performance across CIL and OCL benchmarks, updates less than 1% of parameters, and uses approximately 6.5× lower energy than dense vision transformers.

  • Takeaways & Limitations

    Parameter-efficient plasticity tailored to sparse event-driven transformers can support continual adaptation under strict memory and computational constraints without storing past samples.

Abstract

from arXiv · show

Robotic and edge intelligence systems operate in dynamic environments where data arrives continuously, requiring models to adapt while preserving previously learned knowledge under strict memory and energy constraints. While parameter-efficient fine-tuning has shown promise for continual learning with vision transformers, conventional architectures rely on dense computation and remain costly for real-world deployment. Sparse event-based vision transformers provide energy-efficient event-driven computation, yet their continual learning capabilities remain largely unexplored. We here introduce sLoTh, a parameter-efficient continual learning framework for pretrained sparse event-based (spiking) vision transformers. sLoTh freezes the backbone and restricts plasticity to scalable-efficient low-rank attention updates (seLoRA) and shared neuronal threshold modulation, enabling adaptation without replay buffers by updating less than 1% of model parameters. Experiments across CIFAR-100, Tiny-ImageNet, ImageNet-100, and ImageNet-R with up to 100 tasks demonstrate competitive rehearsal-free performance in class-incremental learning and online continual learning, while enabling approximately 6.5x lower energy consumption than conventional dense vision transformers.

1 Introduction

Continual learning must preserve prior knowledge as data arrives incrementally, but dense transformers are energy-intensive and PEFT for pretrained sparse event-based transformers remains underexplored. sLoTh addresses this gap with rehearsal-free adaptation and reports strong continual-learning performance with low parameter and energy costs.

  • Continual learning addresses catastrophic forgetting when sequential updates overwrite previously acquired knowledge.
  • Dense attention makes standard vision transformers energy-intensive, while PEFT continual learning has focused primarily on artificial neural networks.
  • sLoTh combines shared threshold modulation, seLoRA, and multi-objective distillation and regularisation for rehearsal-free adaptation without network expansion or exemplar buffers.
  • sLoTh provides a systematic study of PEFT for pretrained sparse event-based vision transformers and uses shared adaptation rather than task-specific modules or parameter growth.
  • Less than 1% of model parameters are updated, while sparse event-driven transformers achieve 6.5× lower energy consumption than conventional dense vision transformers.

2 Related Work

Continual learning methods constrain updates, replay prior data, or modify architectures, while PEFT adapts pretrained transformers by updating few parameters. Spiking transformers offer event-driven efficiency but have received limited attention for continual learning.

  • Continual learning methods are commonly grouped into regularisation-based, rehearsal-based, and architecture-based approaches.
  • PEFT keeps a pretrained transformer backbone frozen while adapting only a small subset of parameters through prompts, adapters, or low-rank methods.
  • Spiking neural networks use membrane accumulation and threshold-based firing for event-driven computation, offering improved energy efficiency over conventional neural networks.

3 Methodology

The methodology freezes a pretrained sparse event-based transformer and restricts plasticity to threshold modulation and low-rank attention updates. Distillation, parameter anchors, cosine classification, prototype inference, and loss-based novelty detection support continual adaptation across task settings.

  • Problem Formulation: The framework evaluates Full sLoTh, combining seLoRA and threshold modulation, across CIL, TIL, and OCL protocols.
  • Model Plasticity: Plasticity is restricted to lightweight parameters while the pretrained backbone remains fixed to reduce representation drift and catastrophic forgetting.
  • Channel-wise Excitability Modulation: Channel-wise excitability modulation adjusts neuronal firing thresholds instead of synaptic weights, preserving pretrained feature projections while adapting feature responses.
  • Low-Rank Attention Modulation: seLoRA uses one shared low-rank attention module to provide parameter-efficient plasticity without task-specific adapters or task-dependent parameter growth.
  • Stabilised Training Objective: The training objective combines current-task cross-entropy with teacher-student distillation and local and global parameter anchors.
  • Classification: Cosine classification trains new class weights while freezing prior classifier columns, and nearest-class-mean inference reduces recency bias.
  • Online Novelty Detection: OCL novelty detection uses an exponential moving average of loss and triggers adaptation when the current-to-average loss ratio exceeds a threshold, but is presented as an initial rather than fully robust strategy.

4 Experiments

Experiments evaluate sLoTh across class-incremental and online continual learning, task granularities, datasets, backbones, and component configurations. Results show stronger scalability and competitive rehearsal-free performance alongside energy and parameter efficiency.

  • Evaluation setup: sLoTh generalizes across continual-learning settings, datasets, and transformer backbones with limited architectural modifications and hyperparameter tuning.Experiments cover CIL and OCL, including QKFormer and SpikingFormer backbones.
  • Class Incremental Learning: 84.65% accuracy for 20 tasks and 87.22% for 50 tasks on CIFAR-100, outperforming compared methods as task granularity increases.Full sLoTh reaches 81.23% for 10 tasks, then improves under finer splits.
  • Class Incremental Learning: 77.10% accuracy for T = 20 on ImageNet-R, outperforming compared methods under finer task granularity despite 71.61% for T = 5.The comparison evaluates CIL under domain shift across task splits.
  • Online Continual Learning: 55.12% accuracy with 12.79 forgetting on CIFAR-100 OCL, closely matching MOE-MOSE’s 55.62% despite using no replay buffer.sLoTh retains only lightweight per-class prototype statistics, whereas replay baselines store up to 5k samples.
  • Online Continual Learning: 42.90% accuracy on Tiny-ImageNet with 100 tasks, exceeding OCM, OnPro, and MOE-MOSE replay baselines by approximately 4.5% over the strongest replay baseline.The framework also narrows the boundary-aware versus boundary-free OCL performance gap without replay.
  • Online Continual Learning: Full sLoTh outperforms prior ViT-backbone methods by at least 1.76% on ImageNet-100 OCL with 50 tasks.Transferred Online-LoRA reaches 61.87, approximately equivalent to Full sLoTh.
  • Component Analysis: Combining seLoRA and threshold modulation improves longer-sequence CIL, while threshold-only or seLoRA-only variants favor immediate streaming adaptation.Full sLoTh achieves 81.23% in the reported CIL component analysis; KL, local-consistency, and global-consistency terms also support stable updates.

5 Discussion

sLoTh is presented as a rehearsal-free framework for continual learning with sparse event-based transformers, combining structured attention modulation with excitability control. The discussion highlights its potential for resource-constrained continual learning while identifying shift detection as an area for further improvement.

  • sLoTh combines structured attention modulation with excitability control to maintain stability without memory buffers.
  • sLoTh is designed for sparse event-based transformers, where threshold dynamics support learning with smaller parameter updates without changing underlying features.
  • The framework is reported as effective for both online and fine-grained class-incremental learning under strict memory and computational constraints.
  • More advanced shift-detection strategies may improve robustness and scalability in online continual learning.

A Extended Method Details

The method formulates continual learning over sequential tasks and uses a frozen pretrained sparse event-based transformer with restricted plasticity. Threshold modulation controls neuronal excitability, while shared low-rank attention updates provide scalable adaptation without task-specific parameter growth.

  • Problem formulation: Continual learning is modeled as sequential tasks with disjoint label sets and cumulative prediction over previously introduced classes.
  • Continual learning protocols: Online continual learning lacks explicit task identities, whereas class- and task-incremental settings use different task-boundary and prediction assumptions.
  • Plasticity modules: The pretrained sparse event-based transformer remains frozen, with plasticity restricted to a small set of adaptation parameters.
  • Threshold modulation: Channel-wise threshold offsets modulate neuronal excitability while keeping synaptic weights frozen and largely preserving pretrained feature geometry.
  • Low-rank attention adaptation: seLoRA adds low-rank residuals to attention projections using one shared module across tasks, preventing parameter growth as the task sequence expands.

A.3 Stabilized Training Objective

The stabilized training objective combines current-task classification with distillation and parameter anchoring. These components preserve earlier knowledge and limit both abrupt and long-term adaptation drift, followed by prototype-based inference.

  • Training objective: The objective includes cross-entropy on current-task classes, logit distillation from a frozen teacher, and local and global parameter anchors.
  • Stabilization: Distillation preserves knowledge of previous classes, while local and global anchors constrain short-term and long-term parameter drift.
  • Training objective: The distillation temperature is represented by τ in the objective.
  • Classification: A cosine classifier is used during training, and inference uses nearest class mean classification with cosine similarity.
  • Online shift detection: Online shift detection monitors training loss and activates a novelty trigger when the specified loss condition is met.
  • Online shift detection: When triggered, the framework snapshots adaptation parameters, registers newly discovered classes, and updates the teacher model.

A.6 Full Training Procedure

The full online training procedure initializes a pretrained backbone, adaptation parameters, and an exponential moving average of loss. Each batch updates the loss estimate, conditionally snapshots teacher parameters, and then updates the adaptation parameters.

  • Protocol difference: Unlike online continual learning triggers, class- and task-incremental learning capture an adaptation snapshot after every task.
  • Initialization: Training initializes the pretrained backbone, adaptation parameters, and exponential moving-average loss before processing batches.
  • Batch processing: For each batch, the model computes features and loss, updates the exponential moving average, and checks the loss ratio against γ.
  • Trigger and update: When the trigger condition is met, teacher parameters are snapshotted before adaptation parameters are updated.

B Datasets

The paper evaluates continual learning across four image datasets and multiple settings, using class-disjoint task partitions. Online continual learning additionally requires tuning a novelty-detection threshold for teacher snapshots.

  • Datasets: Evaluations cover CIFAR-100, Tiny-ImageNet, ImageNet-100, and ImageNet-R.ImageNet-100 provides a reduced-scale benchmark, while ImageNet-R provides a domain-shift benchmark.
  • Datasets: Each dataset is partitioned into disjoint class subsets to simulate continual learning scenarios.
  • Continual learning settings: In online continual learning, a novelty-detection threshold triggers teacher model snapshots.Very small thresholds cause frequent false triggers and added computational overhead, whereas overly large thresholds delay distribution-shift detection.
  • Experimental configuration: The experiments use hyperparameters specified across OCL, CIL, and TIL settings.
  • Experimental configuration: Each experiment is conducted three times with seeds 0, 1, and 2.

D.1 PEFT for Sparse Event-Based Vision Transformers

The PEFT comparison shows a trade-off between adaptation accuracy and efficiency: adapters perform best but add architectural overhead, while threshold-only adaptation is most parameter-efficient. This motivates combining low-rank attention modulation with threshold-based plasticity.

  • PEFT comparison: 87.21% accuracy: adapters achieve the highest CIFAR-100 accuracy on a pretrained QKFormer backbone.They tune small feed-forward bottleneck networks, but introduce additional learnable modules.
  • PEFT comparison: 80.89% accuracy: LoRA provides lightweight updates to attention projections as a more efficient alternative.
  • PEFT comparison: 70.56% accuracy: threshold-only adaptation modifies spiking-neuron firing thresholds without adding weights.It is the most parameter-efficient strategy but offers lower accuracy than adapters and LoRA.
  • sLoTh motivation: The observed trade-off motivates sLoTh, which combines low-rank attention modulation with threshold-based intrinsic plasticity.The stated goal is efficient adaptation without affecting inference efficiency.

D.2 Task-Incremental Learning and Class Incremental Learning

The paper evaluates sLoTh under task-incremental, class-incremental, and non-spiking settings. sLoTh maintains strong task-incremental performance as task counts increase, while seLoRA-only adaptation also generalizes to a conventional ViT backbone.

  • Task-Incremental Learning: Task-incremental learning provides task boundaries during training and evaluation, making it comparatively easier than CIL and OCL.
  • Task-Incremental Learning: sLoTh maintains strong performance across Tiny-ImageNet, CIFAR-100, ImageNet-100, and ImageNet-R as the number of tasks increases.Results are reported across different task granularities.
  • Class-Incremental Learning: CIFAR-100 CIL comparisons include pretrained ViT PEFT methods and pretrained QKFormer and SpikingFormer models with full sLoTh.The table varies task granularity, with T denoting the number of tasks.
  • Non-spiking generalization: 55.68 ± 0.97 accuracy: seLoRA-only adaptation on an 86M-parameter ViT-B/16 reaches this result for 100 Tiny-ImageNet tasks.This evaluates whether the adaptation mechanism transfers to a non-spiking architecture.
Loading 2608.26720v1…