PaperlayerView exact revision on arXiv

Source-linked AI summary

SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks

Rui-Jie Zhu, Qihang Zhao, Guoqi Li, Jason K. Eshraghian

arXiv:2302.13939v5cs.CLcs.LGcs.NE

TL;DR

Large language models impose substantial computational and energy costs, while SNNs had not yet demonstrated effective language generation despite their efficiency advantages. SpikeGPT adapts recurrent linear-attention ideas from RWKV into a binary, event-driven spiking language model, achieving competitive tested performance and substantially lower reported computational or energy costs. The paper presents this as evidence that large SNNs can support natural-language generation while retaining efficient spike-based computation.

  • Problem

    SNNs offer energy-efficient computation but had not achieved comparable success in generative language modeling, where training large spiking models is difficult.

  • Method

    SpikeGPT combines binary spiking activations with recurrent linear attention, sequential token streaming, autoregressive training, and binary embeddings for direct-SNN language modeling.

  • Results

    SpikeGPT achieves competitive performance on tested datasets; its 46-million-parameter variant is estimated to use 33.2× less energy than comparable transformer models on asynchronous hardware.

  • Takeaways & Limitations

    Event-driven spiking activations can support language generation with fewer high-cost operations while enabling larger SNN models for neuromorphic computing.

  • Takeaways & Limitations

    The paper identifies a need for more sophisticated training methods, such as knowledge distillation, for larger-scale corpora.

Abstract

from arXiv · show

As the size of large language models continue to scale, so does the computational resources required to run it. Spiking Neural Networks (SNNs) have emerged as an energy-efficient approach to deep learning that leverage sparse and event-driven activations to reduce the computational overhead associated with model inference. While they have become competitive with non-spiking models on many computer vision tasks, SNNs have also proven to be more challenging to train. As a result, their performance lags behind modern deep learning, and we are yet to see the effectiveness of SNNs in language generation. In this paper, inspired by the Receptance Weighted Key Value (RWKV) language model, we successfully implement `SpikeGPT', a generative language model with binary, event-driven spiking activation units. We train the proposed model on two model variants: 45M and 216M parameters. To the best of our knowledge, SpikeGPT is the largest backpropagation-trained SNN model to date, rendering it suitable for both the generation and comprehension of natural language. We achieve this by modifying the transformer block to replace multi-head self attention to reduce quadratic computational complexity O(N^2) to linear complexity O(N) with increasing sequence length. Input tokens are instead streamed in sequentially to our attention mechanism (as with typical SNNs). Our preliminary experiments show that SpikeGPT remains competitive with non-spiking models on tested benchmarks, while maintaining 20x fewer operations when processed on neuromorphic hardware that can leverage sparse, event-driven activations. Our code implementation is available at https://github.com/ridgerchu/SpikeGPT.

1 Introduction

SpikeGPT addresses the energy burden and training challenges of applying SNNs to language generation by combining sequential processing with recurrent, linear-attention design. It demonstrates competitive language-model performance while reducing computational complexity and energy use.

  • Motivation: SNNs offer energy-efficient spike-based computation, but their prior success was concentrated in computer vision rather than generative language modeling.The paper identifies sparse gradients, binarized activations, and language-specific encoding as barriers to language generation.
  • Method: SpikeGPT aligns language sequence positions with the SNN temporal dimension, uses autoregressive training, and applies stateful neurons to mitigate binary-activation bandwidth constraints.These techniques avoid an additional encoder and support direct spiking language processing.
  • Contribution: SpikeGPT provides a first demonstration of language generation using direct-SNN training, addressing a domain where SNNs had not achieved comparable generative success.The approach targets challenges including language encoding, large-scale training, sparse gradients, and recurrent structure.
  • Architecture: Replacing multi-head self-attention with recurrent linear attention reduces sequence complexity from O(T 2) to O(T) while enabling event-driven spike representations.Tokens are processed sequentially rather than through quadratic attention over the full sequence.
  • Results: 33.2× less energy consumption is estimated for the 46-million-parameter SpikeGPT variant on asynchronous hardware while performing competitively against similar transformer models.The result is reported for the small-scale model variant.

2 Related Works

Prior SNN research addressed basic NLP and vision-oriented Transformer applications, but language-generation-oriented SNN Transformers remained underexplored. Added temporal processing and SNN-specific training constraints made these models more computationally expensive and harder to scale.

  • SNNs for NLP: Earlier NLP-oriented SNNs used spiking encoders or converted word embeddings for sentiment classification, machine translation, and text classification.These approaches reported energy savings or comparable performance to ANN counterparts in their respective tasks.
  • Transformer-based SNNs: Prior spiking Transformer variants introduced spiking self-attention or spatial-temporal attention primarily to model visual features.These methods illustrate Transformer adaptation to SNNs without establishing language-generation capability.
  • Research gap: SNN-based Transformers had succeeded mainly on vision tasks, while their potential for language generation remained underexplored.The related work contrasts vision demonstrations with the absence of prior SNN language generation.
  • Transformer-based SNNs: Adding a temporal dimension can increase Transformer complexity from quadratic O(T 2) to cubic O(T 3), making training more expensive.The added cost compounds challenges from sparse activations, non-differentiable operators, approximate gradients, and single-bit activations.

3 Methods

SpikeGPT combines binary spiking neurons with recurrent RWKV-based token mixing and gated feed-forward channel mixing, replacing self-attention with a sequentially compatible design for language modeling. Its architecture supports binary embeddings, residual spiking blocks, and separate generation or classification heads.

  • Leaky Integrate-and-Fire Neuron: The model uses Leaky Integrate-and-Fire neurons with binary spikes, membrane-state decay, thresholding, reset dynamics, and an arctangent surrogate gradient for backpropagation.The default settings are U_threshold = 1, U_reset = 0, and β = 0.5.
  • Model Architecture: Each SpikeGPT block combines Spiking RWKV token mixing with a Spiking Receptance Feed-Forward Network for channel mixing, using residual connections throughout the architecture.The SRFFN uses gated linear transformations and feeds its output into a LIF neuron to preserve block-level sparsity and binary characteristics.
  • Binary Embedding: A Binary Embedding layer converts continuous token embeddings into binary spikes, using a Heaviside function during forward propagation and a surrogate function during backpropagation.This preserves compatibility between the embedding layer and the model’s binary spiking activations.
  • Spiking RWKV: SpikeGPT replaces self-attention with Spiking RWKV, which recurrently processes token inputs without requiring access to the full sequence at once.Self-attention is incompatible with recurrent SNN processing because it requires full-sequence access and dynamic matrix operations.
  • Spiking RWKV: Spiking RWKV uses recurrent decay vectors to blend token dimensions over time, allowing long-range dependencies while avoiding dynamic attention-map matrix multiplication.The decay vectors separately weight current information and decay the influence of previous information.
  • Training and Inference: SpikeGPT is pretrained autoregressively and then fine-tuned for downstream natural language generation or understanding using task-specific heads.Generation uses a language-modeling output head, while understanding tasks use a classification head.

4 Experiments

SpikeGPT is evaluated on language generation, understanding, architectural variants, and scaling. It is competitive with several baselines, though performance varies by dataset and recurrent dynamics are important for binarized activations.

  • 4.2 Results on Natural Language Generating Tasks: SpikeGPT surpasses LSTM-based models and approaches or exceeds simplified Transformers, while remaining behind the vanilla Transformer on generative performance.The comparison uses BPC on Enwik8 and includes Reformer, Synthesizer, Linear Transformer, Performer, Stacked LSTM, and SHA-LSTM baselines.
  • 4.2 Results on Natural Language Generating Tasks: SpikeGPT surpasses GPT-2 on WikiText-2 after fine-tuning but falls behind GPT-2 on the larger WikiText-103 dataset.The authors suggest more refined training methods may be needed for larger corpora.
  • 4.3 Results on Natural Language Understanding Tasks: The fine-tuned 216M SpikeGPT achieves the second-highest NLU accuracy, behind BERT, while the untuned 46M model remains competitive with baseline classifiers.SpikeGPT also uses recurrent and spiking features without an additional temporal dimension for feed-forward processing.
  • 4.4 A Study of SpikeGPT and RWKV Variants: SpikeGPT-B has marginally better test BPC than SpikeGPT, whereas Heaviside RWKV performs worst without recurrent dynamics.The ablation indicates that recurrence helps compensate for performance degradation from binarized activations.

5 Conclusion

The paper concludes that event-driven spiking activations can support language generation with fewer high-cost operations. Its recurrent, serialized attention design is presented as a direction for more effective large-scale SNNs.

  • 5 Conclusion: Event-driven spiking activations support language generation while using fewer high-cost operations.The conclusion frames this as evidence that large SNNs can benefit from transformer advances and serialized attention mechanisms.
  • 5 Conclusion: The authors expect SpikeGPT’s approach to open new directions for large-scale spiking neural networks.The stated direction combines recurrent dynamics with a serialized version of attention mechanisms.

A.1 Derivation and Analysis of the LIF Neuron

The LIF neuron models membrane-potential dynamics with leak, weighted input, threshold-based spiking, and reset, while surrogate gradients enable backpropagation through its binary outputs.

  • A.1 Derivation and Analysis of the LIF Neuron: LIF neurons accumulate weighted inputs in a membrane potential, emit a spike when it exceeds threshold θ, and reset after firing.The spike output is binary, with reset subtracting θ only when a spike occurs.
  • A.1 Derivation and Analysis of the LIF Neuron: The membrane potential follows a leaky dynamical system whose discrete-time approximation uses a decay factor β and learnable input weight W.The decay rate is defined as β = e−1/τ, and the input current is simplified to Iin[t] = WX[t].
  • A.1 Derivation and Analysis of the LIF Neuron: Backpropagation through LIF neurons recursively propagates gradients across time while using a surrogate derivative for the non-differentiable Heaviside spike function.The surrogate function σ(x) supplies the derivative used for the Heaviside function Θ(x).

B.1 Datasets

The experiments evaluate SpikeGPT on language generation and understanding using established Wikipedia, sentiment, subjectivity, and review-classification datasets with stated dataset-specific splits and properties.

  • B.1 Datasets: SpikeGPT is evaluated on three NLG datasets—Enwik8, WikiText-2, and WikiText-103—and four NLU classification datasets: MR, SST-5, SST-2, and Subj.The NLG datasets assess text generation, while the NLU datasets assess text classification performance.
  • B.1 Datasets: Enwik8 contains the first 100 million bytes of a 2006 English Wikipedia dump and uses a 90%/5%/5% train-validation-test split.It is commonly used to measure data-compression ability.
  • B.1 Datasets: WikiText-2 contains 2 million Wikipedia-derived tokens, whereas WikiText-103 contains over 100 million tokens and targets long-term dependencies and richer vocabulary.WikiText-103 is described as a larger and more diverse version of WikiText-2.
  • B.1 Datasets: The NLU datasets cover binary and five-way sentiment, movie-review polarity, and subjective-versus-objective sentence classification.SST-2 is the binary version of SST-5, while Subj distinguishes subjective from objective sentences.
  • B.1 Datasets: When standard train-test segmentation is unavailable, 10% of the full dataset is randomly selected as the test set.

B.2 Baselines

The study compares SpikeGPT with recurrent, convolutional, transformer, and efficient-attention baselines while describing the RWKV-based mechanism used to process sequences recurrently.

  • B.2 Baselines: NLG comparisons include Stacked LSTM, SHA-LSTM, Transformer, Reformer, Synthesizer, Linear Transformer, and Performer.These baselines span recurrent, self-attention, and efficient-transformer variants.
  • B.2 Baselines: NLU comparisons include LSTM, TextCNN, TextSCNN, and BERT, covering recurrent, convolutional, spiking-convolutional, and encoder-only transformer architectures.
  • B.2 Baselines: RWKV replaces self-attention with a recurrent convolution-like operation that sweeps over time, reducing reliance on matrix-matrix multiplication and supporting limited-resource execution.SpikeGPT modifies this operation to process input data recurrently for compatibility with recurrent SNNs.
  • B.2 Baselines: RWKV forms time-varying receptance, key, and value representations from token-shifted embeddings, with receptance controlling acceptance of past information.The matrices MR, MK, and MV are learnable parameters.
  • B.2 Baselines: Its positional weight decay encodes sequential importance, while slow learned decay can preserve long-range dependencies and the receptance acts as a forget gate.The weighted history is combined with the current receptance through an element-wise product.

C.3 Self-Attention and RWKV

The paper distinguishes RWKV’s decomposed matching mechanism from self-attention and examines spiking-layer activity, including firing-rate differences and membrane-potential outliers.

  • C.3 Self-Attention and RWKV: RWKV decomposes token matching into receptance, positional decay, and key terms, producing an embedding-wise matching vector αij.The formulation can be interpreted as an E-headed mechanism with one-dimensional hidden size per head.
  • C.3 Self-Attention and RWKV: Spike and membrane-potential visualizations show distinct information-representation patterns between Spiking RWKV and SRFFN layers.
  • C.3 Self-Attention and RWKV: The SRFFN layer has a higher firing rate, suggesting it may retain more information similarly to Transformer feed-forward layers.
  • C.3 Self-Attention and RWKV: SpikeGPT cannot express activation outliers as ANN activation values, but prominent often-negative outliers appear in individual neurons’ membrane potentials.The observation suggests a different mechanism for accommodating and preserving outliers.
  • C.3 Self-Attention and RWKV: The positional weight bias matrix combines decay, time-step, and initial-decay components, with Wd and Wf learnable and Wc pre-calculated.The hyperparameter pk is set to 0.3 in this paper.

D Visualization of Spike and Membrane Potential

SpikeGPT visualizations compare spike and membrane-potential patterns in its Spiking RWKV and SRFFN layers, revealing distinct information representations and membrane-potential outliers.

  • D Visualization of Spike and Membrane Potential: SRFFN layers exhibit higher firing rates than Spiking RWKV layers, suggesting greater information retention similar to Transformer FFN layers.The comparison is based on visualized spike and membrane-potential patterns in the two layer types.
  • D Visualization of Spike and Membrane Potential: Figure 4 visualizes membrane potentials in Spiking RWKV layers and spike patterns in SRFFN layers, with each dot representing a spike event.The figure distinguishes membrane-potential plots from spike-pattern plots across the two layer families.
  • D Visualization of Spike and Membrane Potential: The Parallelized RWKV demonstration uses large-kernel convolution with decay during sliding-window computation to manage temporal dependencies.The illustrated example uses sequence length N = 4 and embedding size E = 3.
  • D Visualization of Spike and Membrane Potential: Membrane-potential outliers, many negative, provide a way for SpikeGPT to accommodate outlier behavior that binary activations cannot express.The paper contrasts these membrane-potential outliers with activation-value outliers in conventional artificial neural networks.

E Generation Examples

The paper presents generated-text examples from SpikeGPT across material trained on OpenWebText2 and BookCorpus contexts.

  • E Generation Examples: Generated-text examples are shown for models trained on OpenWebText2 and BookCorpus.The captions identify OpenWebText2 for most examples and BookCorpus for one example.
  • E Generation Examples: The generation figures document sample outputs rather than reporting quantitative evaluation results.Their captions identify training corpora and generated-text examples without supplying metrics or comparisons.
  • E Generation Examples: The examples condition generation on diverse contexts, including fictional reports, game descriptions, and historical-art descriptions.The supplied contexts include dragons in Tibet, Pac-Man behavior, and prehistoric cave art in France.
Loading 2302.13939v5…