PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models with At Most One Spike per Neuron

Zhuoya Zhao, Parsa Omidi, Aref Jafari, Richard Naud

arXiv:2609.05151v1cs.NEcs.CL

TL;DR

Conventional TTFS coding cannot directly represent several signed or complex LLM operations, limiting fully spiking language models. The paper introduces reference-based TTFS encoding for core LLM modules and trains a fully TTFS architecture, achieving competitive understanding and reasoning performance while retaining language-modeling and hardware-validation gaps.

  • Problem

    Conventional TTFS SNNs are restricted to specific structures, making signed LLM components such as LayerNorm and matrix multiplication difficult to encode.

    Source: §1 · p. 2, §2.3 · p. 3

  • Method

    The paper introduces reference-based TTFS mechanisms for embedding, LayerNorm, matrix multiplication, attention-related operations, and dropout, then trains the resulting SNN end-to-end.

    Source: §1 · p. 2, §7 · p. 10

  • Results

    The method reaches ANN-level performance on natural-language-understanding and commonsense-reasoning benchmarks while producing lower spike-related cost than rate-coded spiking baselines under a spike-count proxy.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Reference-based TTFS provides a route to fully TTFS-based LLM computation with high sparsity and competitive performance on understanding and commonsense tasks.

    Source: §1 · p. 2, §7 · p. 10

    Language-modeling performance remains clearly worse, and the energy analysis is a spike-count proxy rather than a physical neuromorphic-hardware measurement.

    Source: §6.1 · pp. 9–10, §7 · p. 10

Abstract

from arXiv · show

Leveraging their inherent sparse event-driven computation, spiking neural networks (SNNs) offer a promising path toward energy-efficient large language models (LLMs). Time-to-first-spike (TTFS) coding generates at most one spike per neuron within a time window, yielding extremely low firing rates. However, conventional TTFS SNNs are restricted to specific structures, making it challenging to encode certain blocks in LLM -- such as layer normalization and matrix multiplication --using TTFS. To overcome this limitation, we introduce a reference-based strategy specifically to encode the four core LLM components: embedding layers, layer normalization, attention-related operations and dropout. We construct a fully TTFS-based SNN architecture and train it end-to-end. Experiments on modern LLMs like BERT and GPT-2 demonstrate that our approach achieves performance comparable to ANN counterparts on natural language understanding and common-sense reasoning, while a clear gap remains on language modeling perplexity. To the best of our knowledge, this is the first work to scale a spiking LLM to 1.5 billion parameters using TTFS coding. We also report an estimate of spike-related energy; this is a spike-count proxy under an established cost model rather than a measurement on neuromorphic hardware.

1 Introduction

SNNs use sparse, event-driven spikes to target energy-efficient LLMs, but conventional TTFS coding cannot directly represent several LLM components. The paper introduces reference-based TTFS mechanisms for full-pipeline computation and reports ANN-level performance on language-understanding and commonsense benchmarks.

  • SNNs exploit sparse binary spikes and in-memory neuromorphic computing, motivating their use for energy-efficient LLMs.Source: §1 · p. 1
  • TTFS allows at most one spike per neuron, but conventional mappings support restricted structures and output timings, complicating embedding and normalization.Source: §1 · pp. 1–2
  • A reference-time mechanism increases TTFS expressiveness by enabling signed activations.Source: §1 · p. 2
  • Reference-based TTFS is extended to embedding, LayerNorm, matrix multiplication, and dropout, enabling computation throughout the LLM pipeline.Source: §1 · p. 2
  • BERT and GPT-2 experiments reach ANN-level performance on natural-language-understanding and commonsense-reasoning benchmarks.Source: §1 · p. 2
  • The approach yields substantially lower spike counts and estimated spike-related energy than rate-coded spiking baselines.Source: §1 · p. 2

2 Related works

TTFS is a sparse latency code that represents information by first-spike timing, with relative timing offering greater stability for scaling. Conventional TTFS nevertheless struggles with signed LLM operations and accurate, energy-efficient matrix multiplication.

  • TTFS encodes information using the timing of at most one spike, making it sparser than rate and multi-spike temporal codes.Source: §2.1 · pp. 2–3
  • Relative TTFS measures output timing against a reference, avoiding the across-layer accumulation of absolute spike times.Source: §2.2 · p. 3
  • Conventional TTFS supports non-negative outputs, preventing direct representation of the signed activations required by LayerNorm and matrix multiplication.Source: §2.3 · p. 3
  • Existing TTFS matrix-multiplication strategies either add logarithmic hardware overhead or rely on temporal accumulation prone to estimation errors.Source: §2.3 · p. 3

3 Preliminary

The TTFS neuron integrates incoming spikes until threshold crossing and emits at most one spike within a time window. Its formulation provides the basis for exact ReLU-to-TTFS mapping and the paper’s reference-based extension.

  • TTFS uses first-spike latency as its coding variable and models membrane-potential accumulation followed by threshold crossing.Source: §3.1 · p. 3
  • The neuron integrates presynaptic spikes through linear synaptic dynamics to generate membrane potential over time.Source: §3.1 · pp. 3–4
  • At most one spike occurs within the time window, after which the membrane potential is reset or the process ends at t_max.Source: §3.1 · p. 4
  • The B1-model fixes the synaptic parameter A_i^(n) to 1 as a special TTFS case.Source: §3.1 · p. 4
  • The linear-layer formulation derives output first-spike timing over the maximum time interval and supports exact mapping from ReLU-based ANNs.Source: §3.1 · p. 4

4 Methods

The method extends TTFS SNNs with reference-based timing so they can represent signed values and encode LLM components beyond conventional ReLU-compatible structures. It maps embedding, normalization, attention, and dropout operations into TTFS layers, using fixed or approximate submodules where exact dynamics are insufficient.

  • Conventional TTFS output timing is bounded and its linear ReLU dynamics cannot directly represent LayerNorm, matrix multiplication, or attention components.Source: §4 · p. 4

Reference-based TTFS

  • Reference-based TTFS uses a reference time to encode negative values through relative temporal shifts.Source: §4.1.1 · pp. 4–5
  • Shift-ReLU establishes an exact mapping between reference-based TTFS layers and ReLU ANNs in the shift-ReLU linear regime.Source: §4.1.2 · p. 5

Embedding layers

  • Embedding layers one-hot encode token IDs or positions as spikes, with active entries emitting at 1ms and inactive entries remaining silent.Source: §4.2.1 · p. 5, Figure 3 · p. 5

Layer normalization

  • LayerNorm is unrolled into TTFS steps for mean, variance, subtraction, squaring, inverse-square-root approximation, and affine transformation.Source: §4.2.2 · pp. 5–6

Attention and nonlinearities

  • The inverse-square-root nonlinearity is approximated by two fully connected TTFS layers trained with MSE, while attention uses weighted spike accumulation for matrix products and direct softmax processing.Source: §4.2.2 · p. 7, §4.2.3 · pp. 7–8

Dropout

  • TTFS dropout follows the same timing principle as Equation 3 but randomly drops neurons according to a specified dropout rate.Source: §4.2.4 · p. 8

5 Experiments

Experiments show that TTFS models can approach ANN performance on natural language understanding and commonsense reasoning, while language-modeling performance remains substantially weaker.

BERT

  • TTFS-BERT achieves an average GLUE score of 80.5, compared with 65.1 for SpikeBERT and 81.3 for SpikeLM.On QQP, TTFS-BERT scores 87.0 versus 83.9 for SpikeLM.Source: §5.2 · p. 8
  • TTFS-BERT Large reaches an average GLUE score of 82.3, 3.0 points below full-precision BERT-Large at 85.3.Source: §5.2 · pp. 8–9

Evaluation setup

  • Table 2 compares conventional and TTFS-based GPT-2 models on language-modeling and commonsense-reasoning benchmarks using lmevaluation-harness.Source: Table 2 · p. 9

GPT-2

  • TTFS-GPT-2 Small outperforms the ANN baseline on commonsense reasoning, with average accuracy of 42.5 versus 40.7.The XL model remains comparable at 49.5 versus 51.6.Source: §5.2 · p. 9
  • On language modeling, TTFS-GPT-2 shows a substantial gap: WikiText perplexity rises from 20.4 to 26.6 for XL, while LAMBADA perplexity rises from 10.6 to 23.8.LAMBADA accuracy also declines from 51.2 to 39.4.Source: §5.2 · p. 9

6 Discussion

The energy analysis estimates spike-related cost rather than total system energy, because hardware, neuron dynamics, communication, storage, and timing circuitry can materially affect consumption.

Scope of energy claims

  • Energy conclusions for SNNs depend strongly on the underlying hardware, and GPU execution may incur higher latency and energy costs.Source: §6.1 · p. 9
  • The reported analysis excludes physical neuromorphic measurements and quantifies only spike-related energy under an established cost model.Source: §6.1 · p. 9

Spike-count comparison

  • TTFS with at most one spike has lower spike-related cost than rate-coded methods under the stated cost model.Source: §6.1 · p. 10
  • The proxy uses Ns · Tr and excludes weight storage, off-core memory traffic, inter-core communication, and timing-circuitry costs.These omitted terms can dominate on digital asynchronous hardware and do not necessarily scale with spike count.Source: §6.1 · p. 10
  • Table 3 reports estimated average spike-related energy cost per neuron per inference, with baseline entries derived from firing rate multiplied by time steps.Source: Table 3 · p. 10

7 Conclusion

The paper proposes reference-based TTFS coding for LLM components and reports competitive ANN-level performance on language understanding and commonsense reasoning, alongside unresolved language-modeling and hardware-validation gaps.

  • R-TTFS represents or approximates embedding, LayerNorm, attention, and dropout operations while constraining each neuron to emit at most one spike.Source: §7 · p. 10
  • Under a spike-count proxy, the approach has lower spike-related cost than rate-coded spiking baselines.Source: §7 · p. 10
  • The models remain competitive with ANN counterparts on natural language understanding and commonsense reasoning.Source: §7 · p. 10
  • The paper identifies a clear language-modeling gap and the absence of physical-neuromorphic-hardware measurements as remaining limitations.Source: §7 · p. 10

A.0.1 Theoretical implementation on neuromorphic chips

The reference-based neuron uses a gating signal to control the sign of integration, allowing negative inputs to be represented correctly as processing proceeds chronologically.

  • Reference-based TTFS neurons add a gating signal that modulates the sign of integration.Source: §A.0.1 · p. 13
  • This mechanism correctly represents negative inputs when tref − tj ≤ 0 during chronological integration.Source: §A.0.1 · p. 13

B Implementation details

Pre-training experiments were conducted on a cluster of eight NVIDIA H100 GPUs, each with 80GB of memory.

B.1 TTFS-BERT

The TTFS-BERT experiments use large-scale pre-training configurations and quantify spike activity per transformer block. The reported construction yields approximately 0.80 spikes per neuron per inference for TTFS-BERT-Base.

Training configuration

  • 1,000,000 update steps, a 1 × 10−4 peak learning rate, 2% warm-up, and batch size 512 define BERT pre-training.BERT uses WordPiece tokenization with vocabulary size 30,522 and maximum sequence length 512.Source: §B.1 · p. 13

Spike-count analysis

  • 18.28d + N + 260 spikes per transformer block results from weighting each block component by its average spike count.The non-integer terms arise from the 1/√x approximator and TTFS_ReLU, while remaining sub-blocks emit one spike per neuron by construction.Source: §B.1 · p. 13
  • 14,811 spikes over 18,436 neurons yield ≈0.80 spikes per neuron per inference for TTFS-BERT-Base.This is the reported 0.80 Tr entry in Table 3.Source: §B.1 · p. 13
  • The expanded TTFS LayerNorm has lower estimated spike activity than a rate-coded block for the hidden sizes considered.The comparison is based on 18.28d + N + 260 versus approximately 19.2d + 1.2N spikes.Source: §B.1 · p. 13
  • The 1/√x approximator contributes 512 neurons per LayerNorm independently of hidden size.Its relative cost therefore decreases as models scale up.Source: §B.1 · p. 13

Evaluation

  • Evaluation reports accuracy, F1, and Spearman correlation for task-specific BERT metrics, alongside perplexity and accuracy for GPT-2.Accuracy measures correct predictions, F1 addresses class imbalance, Spearman correlation evaluates semantic similarity, and PPL evaluates language modeling.Source: §B.1 · p. 14, §B.2 · p. 14

C Detailed Results

TTFS-BERT shows almost no performance degradation relative to BERT on reported BERT training and task accuracies, while using highly sparse firing. The analysis also highlights a latency trade-off from sequential layer processing.

Sparsity and task performance

  • At most one spike per unit lets TTFS-LLMs remain competitive on natural language understanding and commonsense reasoning.This sparsity distinguishes them from SNN language models using multiple spikes per unit.Source: §C.1 · pp. 14–15

Temporal configuration

  • TTFS does not bring longer latency per unit because its distinguishable states are T/δt, matching the rate-coding expression.Across layers, however, TTFS processes layers sequentially, which can increase total latency relative to rate coding.Source: §C.1.2 · p. 15
  • Sequential layer processing can produce longer total latency for TTFS than for rate coding.The comparison concerns network-level latency rather than the latency of an individual unit.Source: §C.1.2 · p. 15

Firing rate

  • 8.92 × 10−4kHz is the reported approximate firing rate across embedding layers, LayerNorm, dropout, and transformer blocks.The time window is set to 1000 ms; cited rate-based SNN-LLMs report values between 0.1 and 0.6, with unspecified units in the original papers.Source: §C.1.3 · p. 16

Related papers

Loading 2609.05151v1…