Source-linked AI summary
Spikformer: When Spiking Neural Network Meets Transformer
Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, Li Yuan
TL;DR
Vanilla self-attention does not fit SNN computation because it uses floating-point multiplication, softmax, and quadratic sequence complexity. The paper introduces SSA and Spikformer, which use spike-form Query, Key, and Value without softmax; Spikformer outperforms state-of-the-art SNNs on static and neuromorphic datasets, including 74.81% top1 accuracy on ImageNet with 4 time steps.
Problem
Vanilla self-attention uses floating-point matrix multiplication, softmax, and quadratic sequence complexity that do not meet SNN calculation characteristics and efficiency requirements.
Method
The paper proposes Spiking Self Attention and builds Spikformer using sparse spike-form Query, Key, and Value without softmax or multiplication.
Results
Spikformer outperforms state-of-the-art SNNs on static and neuromorphic datasets, achieving 74.81% top1 accuracy on ImageNet with 4 time steps.
Takeaways & Limitations
SSA provides an efficient spike-friendly self-attention mechanism, while Spikformer extends self-attention and Transformer models to directly trained SNNs.
Takeaways & Limitations
For static datasets, a 2D image must be repeated T times to form a sequence of images.
Abstract
from arXiv · showhide
We consider two biologically plausible structures, the Spiking Neural Network (SNN) and the self-attention mechanism. The former offers an energy-efficient and event-driven paradigm for deep learning, while the latter has the ability to capture feature dependencies, enabling Transformer to achieve good performance. It is intuitively promising to explore the marriage between them. In this paper, we consider leveraging both self-attention capability and biological properties of SNNs, and propose a novel Spiking Self Attention (SSA) as well as a powerful framework, named Spiking Transformer (Spikformer). The SSA mechanism in Spikformer models the sparse visual feature by using spike-form Query, Key, and Value without softmax. Since its computation is sparse and avoids multiplication, SSA is efficient and has low computational energy consumption. It is shown that Spikformer with SSA can outperform the state-of-the-art SNNs-like frameworks in image classification on both neuromorphic and static datasets. Spikformer (66.3M parameters) with comparable size to SEW-ResNet-152 (60.2M,69.26%) can achieve 74.81% top1 accuracy on ImageNet using 4 time steps, which is the state-of-the-art in directly trained SNNs models.
1 INTRODUCTION
The paper introduces SSA and Spikformer to bring self-attention into SNNs while respecting spike-based, multiplication-avoiding computation. Experiments report stronger performance than state-of-the-art SNNs on static and neuromorphic datasets.
- Vanilla self-attention conflicts with SNN computation because it uses floating-point products, softmax exponentiation and division, and substantial overhead.These operations do not match SNN calculation characteristics and make direct application difficult.
- SSA requires 5.5 × 10^6 versus 77 × 10^6 operations for VSA and consumes less energy than VSA, whose energy is reported as 354.2µJ.The comparison uses theoretical synaptic operations and energy for one Query-Key-Value calculation in one time step.
- SSA uses sparse spike-form Query, Key, and Value without softmax, making attention non-negative while avoiding multiplications.Its computation can use logical AND operations and addition.
- Spikformer combines SSA with a Transformer architecture for SNNs and is presented as the first implementation of self-attention and Transformer in SNNs.The framework is based on the proposed spike-compatible attention mechanism.
- Spikformer outperforms state-of-the-art SNNs on both static and neuromorphic datasets, exceeding 74% ImageNet accuracy with four time steps.The result is reported for a directly trained SNN model.
2 RELATED WORK
The related work connects Vision Transformers' global feature modeling with SNNs' discrete spike-based computation. The paper focuses on testing self-attention and developing a spiking Transformer architecture for image classification.
- Vision Transformers: Vision Transformers use patch splitting, Transformer encoders, and linear classification heads, with self-attention capturing global dependence through query-key weighting and softmax.Prior work also studies convolutional patch splitting and reduced-complexity attention.
- Spiking Neural Networks: SNNs transmit information as discrete spike sequences rather than continuous decimal values and can be developed through conversion or direct training.The paper uses this spike-based computation as the architectural setting for its Transformer exploration.
- Spiking Neural Networks: Spikformer uses LIF spike neurons whose membrane dynamics accumulate input current and trigger spikes when membrane potential exceeds a threshold.The post-trigger membrane potential is either retained or reset according to spike generation.
- Spiking Transformer: The proposed overview contains a spiking patch splitting module, Spikformer encoder, and linear classification head, using batch normalization instead of layer normalization.The paper reports empirically that layer normalization does not apply to SNNs.
3 METHOD
Spikformer combines spike-based processing with Transformer self-attention through Spiking Self Attention (SSA), designed to match SNN computation constraints. Its architecture converts image sequences into spike-form patches, applies spike-based attention and encoder blocks, and evaluates efficiency and accuracy against alternative attention variants.
- Spikformer overview: Spikformer incorporates self-attention and Transformer components into spiking neural networks through an encoder containing SSA and an MLP block.The framework is designed to enhance learning capability while retaining SNN-compatible processing.
- Spiking Patch Splitting: The SPS module linearly projects image sequences into D-dimensional spike-form features and splits them into N flattened patches.Static images are repeated across T time steps, while neuromorphic inputs already have temporal shape T × C × H × W.
- Motivation for SSA: VSA is unsuitable for SNNs because float-point matrix multiplication, softmax exponentiation and division, and quadratic sequence complexity violate efficient SNN computation requirements.These operations do not conform to SNN calculation rules and impose substantial computational overhead.
- Spiking Self Attention: SSA converts learnable Query, Key, and Value projections into spike sequences and computes attention directly without softmax.Spike-form representations are non-negative, so the attention map does not require softmax normalization.
- Analysis of SSA: Replacing SSA with alternative attention variants while preserving the remaining Spikformer structure evaluates accuracy, operation count, and theoretical energy consumption.The comparison covers CIFAR10-DVS and CIFAR10/100, with ASSA measured in SOPs and other variants in FLOPs.
- Spiking Self Attention: Sparse spike-form Query, Key, and Value reduce attention computation to logical AND and addition, lowering operations and theoretical energy consumption.When sequence length N exceeds head dimension d, the alternative K^T V-first order has complexity O(Nd^2) instead of O(N^2d).
4 EXPERIMENTS
Experiments evaluate Spikformer from scratch on static and neuromorphic datasets, including accuracy, computational cost, SSA, and time-step robustness. Spikformer achieves strong results across these settings, with low reported operations and energy in an ImageNet comparison.
- Experimental setup: Spikformer is evaluated from scratch on CIFAR, ImageNet, CIFAR10-DVS, and DVS128 Gesture, with ablations on SSA and simulation time steps.The experiments compare Spikformer with current SNN models on static and neuromorphic datasets.
- Static datasets classification: 70.24% top-1 accuracy with 16.81M parameters lets Spikformer-8-384 outperform SEW-ResNet-152 at 69.26% with 60.19M parameters on ImageNet.Spikformer-8-384 also reports 6.82G SOPs and 7.734mJ theoretical energy, versus 13.72G and 12.891mJ for SEW-ResNet-152.
- Static datasets classification: 95.19% accuracy on CIFAR10 makes Spikformer-4-384 better than TET at 94.44% and ResNet-19 ANN at 94.97%.Performance improves as embedding dimensions or transformer blocks increase.
- Neuromorphic datasets classification: 98.2% accuracy on DVS128 Gesture with 16 time steps exceeds SEW-ResNet at 97.9%, while CIFAR10-DVS accuracy exceeds DSR by 1.6% with 10 steps and 3.6% with 16 steps.These results use a 2.59M-parameter model.
- Ablation study: 70.14% accuracy with one time step shows that Spikformer remains robust under fewer simulation steps.On CIFAR10, the one-step result is 1.87% lower than the network using four time steps.
5 CONCLUSION
The paper explores self-attention and Transformer implementation in spiking neural networks through Spikformer and its Spiking Self-Attention mechanism. SSA removes softmax and directly operates on spike-form representations, supporting efficient computation and strong performance on static and neuromorphic datasets.
- Conclusion: The work proposes Spikformer based on a new Spiking Self-Attention mechanism for implementing self-attention and Transformer in spiking neural networks.SSA is designed specifically for SNNs and spike data.
- Conclusion: SSA drops softmax and performs matrix dot products directly on spike-form Query, Key, and Value, avoiding multiplications.The paper characterizes this self-attention mechanism as efficient and reports strong performance across static and neuromorphic datasets.
- Conclusion: Directly trained Spiking Transformer models outperform state-of-the-art SNN models on the reported static and neuromorphic datasets.The authors present this work as a step toward transformer-based SNN research.
REPRODUCIBILITY STATEMENT
The paper describes its implementation and reproducibility materials, including the software foundations, planned code release, and multi-head SSA organization. Time is independent in spiking-neuron layers but merged with batch size elsewhere.
- Reproducibility statement: The implementation uses SpikingJelly and the PyTorch image models library, with training and augmentation details provided in the paper and appendix.The Spikformer model code is supplied as supplementary material and planned for GitHub release after review.
- Multi-head SSA: Multi-head SSA reshapes Q, K, and V into T × H × N × d, splits them into H parts, and runs H SSA operations in parallel.The embedding dimension satisfies D = H × d.
- Time-step handling: T is an independent dimension for spiking-neuron layers and is merged with batch size in other layers.This describes how the time-step dimension is handled across the implementation.
C.1 TRAINING
Spikformer training uses architectural and surrogate-gradient choices tailored to spiking computation, while its energy analysis counts spike-based operations and firing activity.
- Spikformer replaces dropout, droppath, and pre-block layer normalization with batch normalization after each linear layer.
- MLP blocks use a hidden dimension of 4 × D, and the Sigmoid surrogate function uses α = 4.
- Theoretical synaptic operations equal firing rate × simulation time steps × layer FLOPs, with FLOPs defined as multiply-and-accumulate operations.
- Spikformer energy estimates use spike-based accumulate operations for SNN components and compare them with multiply-and-accumulate operations for ANNs.
D ADDITIONAL RESULTS
Additional analyses show that SSA produces sparse Query, Key, and Value representations, while larger Spikformer configurations reduce both training and testing losses.
- Query, Key, and Value are very sparse in SSA, resulting in sparse SSA computation.
- Training and testing losses decrease as either the number of Spikformer blocks or the embedding dimension increases.
- CIFAR10/100 results include Spikformer-4-384w IF, which uses the Integrate-and-Fire neuron.
D.3 ADDITIONAL ACCURACY RESULTS ON CIFAR
The paper reports additional experiments on CIFAR, presented in Table 6.
- Additional CIFAR10/100 experiments are reported in Table 6.
D.4 ANALYSIS OF SELF-ATTENTION VARIANTS NOT CONVERGING ON IMAGENET
The analysis attributes non-convergence in several self-attention variants to vanishing surrogate gradients, while SSA keeps QKTV outputs in a suitable range for stable training.
- Vanishing sigmoid-surrogate gradients occur when the difference between average input Vi and firing threshold Vth is too large or too small.
- SSA keeps QKTV outputs in a suitable range, providing stable surrogate gradients during training and easier convergence than the other variants.
- Figure 6 shows the sigmoid surrogate function and gradient curve alongside the QKTV output values for the compared variants.
D.5 TRANSFER LEARNING
The paper evaluates transferring ImageNet-pretrained Spikformer models to CIFAR10/100 through 60-epoch fine-tuning. Spikformer shows high transfer ability in this setting.
- D.5 TRANSFER LEARNING: ImageNet-pretrained Spikformer-4-384 and Spikformer-8-384/512 models are fine-tuned on CIFAR for 60 epochs.The CIFAR input size is 224 × 224, with remaining hyperparameters matching direct CIFAR training.
- D.5 TRANSFER LEARNING: Spikformer shows high transfer ability on the downstream CIFAR dataset.
- D.5 TRANSFER LEARNING: The transfer-learning evaluation covers CIFAR10 and CIFAR100.