PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models with At Most One Spike per Neuron

Zhuoya Zhao, Parsa Omidi, Aref Jafari, Richard Naud

arXiv:2609.05151v1cs.NEcs.CL

TL;DR

Conventional TTFS coding cannot directly represent several signed or complex LLM operations, limiting fully spiking language models. The paper introduces reference-based TTFS encoding for core LLM modules and trains a fully TTFS architecture, achieving competitive understanding and reasoning performance while retaining language-modeling and hardware-validation gaps.

  • Problem

    Conventional TTFS SNNs are restricted to specific structures, making signed LLM components such as LayerNorm and matrix multiplication difficult to encode.

  • Method

    The paper introduces reference-based TTFS mechanisms for embedding, LayerNorm, matrix multiplication, attention-related operations, and dropout, then trains the resulting SNN end-to-end.

  • Results

    The method reaches ANN-level performance on natural-language-understanding and commonsense-reasoning benchmarks while producing lower spike-related cost than rate-coded spiking baselines under a spike-count proxy.

  • Takeaways & Limitations

    Reference-based TTFS provides a route to fully TTFS-based LLM computation with high sparsity and competitive performance on understanding and commonsense tasks.

  • Takeaways & Limitations

    Language-modeling performance remains clearly worse, and the energy analysis is a spike-count proxy rather than a physical neuromorphic-hardware measurement.

Abstract

from arXiv · show

Leveraging their inherent sparse event-driven computation, spiking neural networks (SNNs) offer a promising path toward energy-efficient large language models (LLMs). Time-to-first-spike (TTFS) coding generates at most one spike per neuron within a time window, yielding extremely low firing rates. However, conventional TTFS SNNs are restricted to specific structures, making it challenging to encode certain blocks in LLM -- such as layer normalization and matrix multiplication --using TTFS. To overcome this limitation, we introduce a reference-based strategy specifically to encode the four core LLM components: embedding layers, layer normalization, attention-related operations and dropout. We construct a fully TTFS-based SNN architecture and train it end-to-end. Experiments on modern LLMs like BERT and GPT-2 demonstrate that our approach achieves performance comparable to ANN counterparts on natural language understanding and common-sense reasoning, while a clear gap remains on language modeling perplexity. To the best of our knowledge, this is the first work to scale a spiking LLM to 1.5 billion parameters using TTFS coding. We also report an estimate of spike-related energy; this is a spike-count proxy under an established cost model rather than a measurement on neuromorphic hardware.

1 Introduction

SNNs use sparse, event-driven spikes to target energy-efficient LLMs, but conventional TTFS coding cannot directly represent several LLM components. The paper introduces reference-based TTFS mechanisms for full-pipeline computation and reports ANN-level performance on language-understanding and commonsense benchmarks.

  • SNNs exploit sparse binary spikes and in-memory neuromorphic computing, motivating their use for energy-efficient LLMs.
  • TTFS allows at most one spike per neuron, but conventional mappings support restricted structures and output timings, complicating embedding and normalization.
  • A reference-time mechanism increases TTFS expressiveness by enabling signed activations.
  • Reference-based TTFS is extended to embedding, LayerNorm, matrix multiplication, and dropout, enabling computation throughout the LLM pipeline.
  • BERT and GPT-2 experiments reach ANN-level performance on natural-language-understanding and commonsense-reasoning benchmarks.
  • The approach yields substantially lower spike counts and estimated spike-related energy than rate-coded spiking baselines.

2 Related works

TTFS is a sparse latency code that represents information by first-spike timing, with relative timing offering greater stability for scaling. Conventional TTFS nevertheless struggles with signed LLM operations and accurate, energy-efficient matrix multiplication.

  • TTFS encodes information using the timing of at most one spike, making it sparser than rate and multi-spike temporal codes.
  • Relative TTFS measures output timing against a reference, avoiding the across-layer accumulation of absolute spike times.
  • Conventional TTFS supports non-negative outputs, preventing direct representation of the signed activations required by LayerNorm and matrix multiplication.
  • Existing TTFS matrix-multiplication strategies either add logarithmic hardware overhead or rely on temporal accumulation prone to estimation errors.

3 Preliminary

The TTFS neuron integrates incoming spikes until threshold crossing and emits at most one spike within a time window. Its formulation provides the basis for exact ReLU-to-TTFS mapping and the paper’s reference-based extension.

  • TTFS uses first-spike latency as its coding variable and models membrane-potential accumulation followed by threshold crossing.
  • The neuron integrates presynaptic spikes through linear synaptic dynamics to generate membrane potential over time.
  • At most one spike occurs within the time window, after which the membrane potential is reset or the process ends at t_max.
  • The B1-model fixes the synaptic parameter A_i^(n) to 1 as a special TTFS case.
  • The linear-layer formulation derives output first-spike timing over the maximum time interval and supports exact mapping from ReLU-based ANNs.

4 Methods

The method extends TTFS SNNs with reference-based timing so they can represent signed values and encode LLM components beyond conventional ReLU-compatible structures. It maps embedding, normalization, attention, and dropout operations into TTFS layers, using fixed or approximate submodules where exact dynamics are insufficient.

  • Conventional TTFS output timing is bounded and its linear ReLU dynamics cannot directly represent LayerNorm, matrix multiplication, or attention components.
  • Reference-based TTFS: Reference-based TTFS uses a reference time to encode negative values through relative temporal shifts.
  • Reference-based TTFS: Shift-ReLU establishes an exact mapping between reference-based TTFS layers and ReLU ANNs in the shift-ReLU linear regime.
  • Embedding layers: Embedding layers one-hot encode token IDs or positions as spikes, with active entries emitting at 1ms and inactive entries remaining silent.
  • Layer normalization: LayerNorm is unrolled into TTFS steps for mean, variance, subtraction, squaring, inverse-square-root approximation, and affine transformation.
  • Attention and nonlinearities: The inverse-square-root nonlinearity is approximated by two fully connected TTFS layers trained with MSE, while attention uses weighted spike accumulation for matrix products and direct softmax processing.
  • Dropout: TTFS dropout follows the same timing principle as Equation 3 but randomly drops neurons according to a specified dropout rate.

5 Experiments

Experiments show that TTFS models can approach ANN performance on natural language understanding and commonsense reasoning, while language-modeling performance remains substantially weaker.

  • BERT: TTFS-BERT achieves an average GLUE score of 80.5, compared with 65.1 for SpikeBERT and 81.3 for SpikeLM.On QQP, TTFS-BERT scores 87.0 versus 83.9 for SpikeLM.
  • BERT: TTFS-BERT Large reaches an average GLUE score of 82.3, 3.0 points below full-precision BERT-Large at 85.3.
  • Evaluation setup: Table 2 compares conventional and TTFS-based GPT-2 models on language-modeling and commonsense-reasoning benchmarks using lmevaluation-harness.
  • GPT-2: TTFS-GPT-2 Small outperforms the ANN baseline on commonsense reasoning, with average accuracy of 42.5 versus 40.7.The XL model remains comparable at 49.5 versus 51.6.
  • GPT-2: On language modeling, TTFS-GPT-2 shows a substantial gap: WikiText perplexity rises from 20.4 to 26.6 for XL, while LAMBADA perplexity rises from 10.6 to 23.8.LAMBADA accuracy also declines from 51.2 to 39.4.

6 Discussion

The energy analysis estimates spike-related cost rather than total system energy, because hardware, neuron dynamics, communication, storage, and timing circuitry can materially affect consumption.

  • Scope of energy claims: Energy conclusions for SNNs depend strongly on the underlying hardware, and GPU execution may incur higher latency and energy costs.
  • Scope of energy claims: The reported analysis excludes physical neuromorphic measurements and quantifies only spike-related energy under an established cost model.
  • Spike-count comparison: TTFS with at most one spike has lower spike-related cost than rate-coded methods under the stated cost model.
  • Spike-count comparison: The proxy uses Ns · Tr and excludes weight storage, off-core memory traffic, inter-core communication, and timing-circuitry costs.These omitted terms can dominate on digital asynchronous hardware and do not necessarily scale with spike count.
  • Spike-count comparison: Table 3 reports estimated average spike-related energy cost per neuron per inference, with baseline entries derived from firing rate multiplied by time steps.

7 Conclusion

The paper proposes reference-based TTFS coding for LLM components and reports competitive ANN-level performance on language understanding and commonsense reasoning, alongside unresolved language-modeling and hardware-validation gaps.

  • R-TTFS represents or approximates embedding, LayerNorm, attention, and dropout operations while constraining each neuron to emit at most one spike.
  • Under a spike-count proxy, the approach has lower spike-related cost than rate-coded spiking baselines.
  • The models remain competitive with ANN counterparts on natural language understanding and commonsense reasoning.
  • The paper identifies a clear language-modeling gap and the absence of physical-neuromorphic-hardware measurements as remaining limitations.

A.0.1 Theoretical implementation on neuromorphic chips

The reference-based neuron uses a gating signal to control the sign of integration, allowing negative inputs to be represented correctly as processing proceeds chronologically.

  • Reference-based TTFS neurons add a gating signal that modulates the sign of integration.
  • This mechanism correctly represents negative inputs when tref − tj ≤ 0 during chronological integration.

B Implementation details

Pre-training experiments were conducted on a cluster of eight NVIDIA H100 GPUs, each with 80GB of memory.

  • Eight NVIDIA H100 GPUs with 80GB memory each were used for all pre-training experiments.

B.1 TTFS-BERT

The TTFS-BERT experiments use large-scale pre-training configurations and quantify spike activity per transformer block. The reported construction yields approximately 0.80 spikes per neuron per inference for TTFS-BERT-Base.

  • Training configuration: 1,000,000 update steps, a 1 × 10−4 peak learning rate, 2% warm-up, and batch size 512 define BERT pre-training.BERT uses WordPiece tokenization with vocabulary size 30,522 and maximum sequence length 512.
  • Spike-count analysis: 18.28d + N + 260 spikes per transformer block results from weighting each block component by its average spike count.The non-integer terms arise from the 1/√x approximator and TTFS_ReLU, while remaining sub-blocks emit one spike per neuron by construction.
  • Spike-count analysis: 14,811 spikes over 18,436 neurons yield ≈0.80 spikes per neuron per inference for TTFS-BERT-Base.This is the reported 0.80 Tr entry in Table 3.
  • Spike-count analysis: The expanded TTFS LayerNorm has lower estimated spike activity than a rate-coded block for the hidden sizes considered.The comparison is based on 18.28d + N + 260 versus approximately 19.2d + 1.2N spikes.
  • Spike-count analysis: The 1/√x approximator contributes 512 neurons per LayerNorm independently of hidden size.Its relative cost therefore decreases as models scale up.
  • Evaluation: Evaluation reports accuracy, F1, and Spearman correlation for task-specific BERT metrics, alongside perplexity and accuracy for GPT-2.Accuracy measures correct predictions, F1 addresses class imbalance, Spearman correlation evaluates semantic similarity, and PPL evaluates language modeling.

C Detailed Results

TTFS-BERT shows almost no performance degradation relative to BERT on reported BERT training and task accuracies, while using highly sparse firing. The analysis also highlights a latency trade-off from sequential layer processing.

  • Sparsity and task performance: At most one spike per unit lets TTFS-LLMs remain competitive on natural language understanding and commonsense reasoning.This sparsity distinguishes them from SNN language models using multiple spikes per unit.
  • Temporal configuration: TTFS does not bring longer latency per unit because its distinguishable states are T/δt, matching the rate-coding expression.Across layers, however, TTFS processes layers sequentially, which can increase total latency relative to rate coding.
  • Temporal configuration: Sequential layer processing can produce longer total latency for TTFS than for rate coding.The comparison concerns network-level latency rather than the latency of an individual unit.
  • Firing rate: 8.92 × 10−4kHz is the reported approximate firing rate across embedding layers, LayerNorm, dropout, and transformer blocks.The time window is set to 1000 ms; cited rate-based SNN-LLMs report values between 0.1 and 0.6, with unspecified units in the original papers.
Loading 2609.05151v1…