PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Batch Normalization for Training Low-latency Deep Spiking Neural Networks from Scratch

Youngeun Kim, Priyadarshini Panda

arXiv:2010.01729v5cs.CVcs.AIcs.NE

TL;DR

SNN training is hindered by non-differentiable spiking neurons and batch normalization that does not capture temporal spike statistics. The paper introduces BNTT with time-specific parameters, enabling direct training from scratch at low latency and an early-exit inference procedure. BNTT is reported to support deep SNNs on large datasets, including Tiny-ImageNet, while reducing inference latency through parameter-based exit decisions.

  • Problem

    Non-differentiable spiking neurons hinder gradient-descent training, while standard BN uses shared statistics that fail to capture temporal spike dynamics.

  • Method

    BNTT expands BN across time with local learnable parameters and temporal statistics, training the resulting SNN with surrogate-gradient backpropagation.

  • Results

    BNTT enables direct training of SNNs from scratch on large datasets, including Tiny-ImageNet, while reducing latency through temporal parameterized spike control.

  • Takeaways & Limitations

    BNTT supports low-latency, low-energy SNN operation and enables temporal early exit by monitoring its learnable parameters during inference.

  • Takeaways & Limitations

    The mathematical analysis assumes γ values have similar behavior across adjacent time intervals, allowing the analysis to focus on γ_t.

Abstract

from arXiv · show

Spiking Neural Networks (SNNs) have recently emerged as an alternative to deep learning owing to sparse, asynchronous and binary event (or spike) driven processing, that can yield huge energy efficiency benefits on neuromorphic hardware. However, training high-accuracy and low-latency SNNs from scratch suffers from non-differentiable nature of a spiking neuron. To address this training issue in SNNs, we revisit batch normalization and propose a temporal Batch Normalization Through Time (BNTT) technique. Most prior SNN works till now have disregarded batch normalization deeming it ineffective for training temporal SNNs. Different from previous works, our proposed BNTT decouples the parameters in a BNTT layer along the time axis to capture the temporal dynamics of spikes. The temporally evolving learnable parameters in BNTT allow a neuron to control its spike rate through different time-steps, enabling low-latency and low-energy training from scratch. We conduct experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet and event-driven DVS-CIFAR10 datasets. BNTT allows us to train deep SNN architectures from scratch, for the first time, on complex datasets with just few 25-30 time-steps. We also propose an early exit algorithm using the distribution of parameters in BNTT to reduce the latency at inference, that further improves the energy-efficiency.

1. Introduction

SNNs offer energy-efficiency benefits but remain difficult to train because spiking neurons are non-differentiable. The paper proposes BNTT, whose time-specific parameters capture spike dynamics and support accurate, low-latency training and early inference exit.

  • SNNs process visual information as discrete spikes over multiple time-steps and can achieve 1–2 orders of magnitude greater energy efficiency than ANNs on neuromorphic hardware.
  • Non-differentiable spiking neurons make SNNs incompatible with ordinary gradient-descent training.
  • ANN-SNN conversion and hybrid approaches require hundreds of time-steps, while surrogate-gradient training has poor optimization on larger datasets.
  • BNTT decouples batch-normalization parameters across time-steps to capture temporal spike dynamics during surrogate-gradient training.
  • BNTT supports stable training from scratch on large-scale datasets with approximately 25–50 time-steps and controls spike activity for low-energy inference.
  • The proposed temporal early-exit algorithm monitors BNTT parameters so an SNN can stop inference before the full time period.

2. Batch Normalization

Batch normalization standardizes mini-batch representations using batch statistics and learnable affine parameters. It improves optimization in ANNs, but its effectiveness in temporal SNNs had not been observed.

  • BN computes mini-batch mean and variance to standardize intermediate representations instead of using statistics from the entire dataset.
  • The normalized features are transformed with learnable parameters γ and β, expressed as BN(x_i) = γ_b x_i + β.
  • At inference, BN uses running averages of the mean and variance collected during training.
  • BN generally improves ANN performance and reduces the iterations needed for convergence, but its effectiveness in bio-plausible SNNs had not been observed.

3. Methodology

The methodology models temporal spike dynamics with LIF neurons and trains SNNs using BNTT, whose time-specific parameters support temporal normalization and early inference exit.

  • 3.1. Spiking Neural Networks: LIF neurons integrate weighted incoming spikes, leak membrane potential over time, and emit binary spikes when the membrane potential exceeds threshold θ.After spiking, the soft reset subtracts θ and preserves residual voltage for the next time-step.
  • 3.1. Spiking Neural Networks: The output layer accumulates membrane potentials across all time-steps without thresholding, allowing softmax classification without discarding temporal information.Output neurons use a fixed leak parameter λ = 1 and match the dataset’s number of classes.
  • 3.1. Spiking Neural Networks: Because hidden-layer spikes are non-differentiable, training uses an approximated gradient and backpropagation through time to update network weights.The damping factor α is set to 0.3 because large values can make gradients unstable when summed across time-steps.
  • 3.2. Batch Normalization Through Time (BNTT): Standard BN fails to capture time-specific spike statistics because it uses common parameters across time-steps, motivating temporally adaptive normalization.Early layers can produce zero-valued signals until spikes propagate from preceding layers.
  • 3.2. Batch Normalization Through Time (BNTT): BNTT expands batch normalization across time by assigning local learning parameters to each time-step, enabling gradients and normalization to reflect temporal input statistics.It is inserted after convolutional or linear operations, while β is omitted because it would add redundant voltage to the membrane potential.
  • 3.4. Early Exit Algorithm: The temporal distribution of γt supports early exit when averaged γt values across every layer fall below a predetermined threshold.For VGG9 on CIFAR-10, all averaged γt values fall below 0.1 after t > 20, yielding an early exit at t = 20 without additional computation.

4. Experiments

Experiments across static and event-driven datasets show that BNTT enables accurate, low-latency SNN training while improving spike sparsity, energy efficiency, early exiting, and robustness.

  • Experimental setup: BNTT is evaluated on CIFAR-10, CIFAR-100, Tiny-ImageNet, and event-driven DVS-CIFAR10 datasets.The study uses three static classification datasets and one neuromorphic dataset.
  • Comparison with previous methods: BNTT trains SNNs from scratch on larger datasets while reducing latency compared with conversion and hybrid methods.It trains with 25 time-steps on CIFAR-10, achieves about 40× and 2× faster inference on CIFAR-100 than conversion and hybrid methods, and outperforms conversion on Tiny-ImageNet with shorter latency.
  • Comparison with previous methods: BNTT improves training stability and achieves state-of-the-art performance on event-driven DVS-CIFAR10 data.ANN-SNN conversion is not applicable to this dataset, while BNTT improves stability over the surrogate-gradient baseline.
  • Temporal parameter analysis: BNTT parameters exhibit layer-specific temporal dynamics, with gamma distributions peaking at different times before shrinking at later time-steps.In VGG9, conv1 gamma values decrease from high initial values, while conv4 and conv7 peak at t = 9 and t = 13, respectively.
  • Early exit: Higher gamma thresholds enable earlier inference exits with only marginal accuracy drops across CIFAR-10, CIFAR-100, and Tiny-ImageNet.The early-exit rule stops inference when average gamma values in every layer fall below a predetermined threshold.
  • Robustness analysis: BNTT achieves higher performance than the compared methods across all tested Gaussian noise intensities.The authors associate this robustness with reducing the overall number of time-steps.

5. Conclusion

The paper proposes extending batch normalization across time with time-specific learnable parameters to train SNNs from scratch. It reports that optimizing γ enables sequential information flow and direct training on large datasets such as Tiny-ImageNet.

  • BNTT extends batch normalization to the temporal dimension using time-specific learnable parameters and statistics.
  • Optimizing γ during training enables visual information to pass sequentially through network layers.
  • The approach directly trains SNNs on large datasets such as Tiny-ImageNet.

A. Appendix: Backward Gradient of BNTT

The appendix derives the backward gradient for a BNTT layer and explains that gradients use time-specific input statistics. This lets training account for temporal dynamics when updating weight connections.

  • The appendix computes the backward gradient of a BNTT layer at each time-step for a mini-batch sample.
  • The derivation separates the right-hand-side terms of the gradient into successive components before reformulating the result.
  • For every time-step, gradients use time-specific input statistics to incorporate temporal dynamics into weight-connection training.

B. Appendix: Rate Coding

The appendix uses rate coding to convert static images into spike sequences. Accumulated spikes increasingly represent the original image over time.

  • As time progresses, accumulated spikes represent an image similar to the original CIFAR-10 image.
  • Rate coding generates spikes proportional to the pixel intensity of the input image.The method compares each pixel value with a random number to implement the encoding.

C. Appendix: DVS-CIFAR10 dataset

For DVS-CIFAR10, the data are downsampled, divided into 20 temporal intervals, and accumulated within intervals. The appendix uses a five-layer feature extractor and classifier architecture.

  • The 128 ×128 DVS-CIFAR10 images are downsampled to 42×42 and divided into 20 time intervals.
  • Spikes are accumulated within each temporal interval from the original event timeframe data.
  • The DVS-CIFAR10 architecture consists of a 5-layered feature extractor and a classifier.
  • The network structure illustration identifies AP as average pooling and FC as fully connected configuration.

D. Appendix: Energy Calculation

The appendix estimates ANN and SNN computational cost from layer operations and spiking activity, then maps those costs to energy using a 45 nm CMOS model.

  • ANN FLOPS are computed from kernel size, output feature-map size, and input/output channels for convolutional and linear layers.The appendix treats FLOPS as approximately equivalent to Matrix-Vector Multiplication operations.
  • SNN FLOP count incorporates each layer’s average firing rate because neurons consume energy when they spike.The spiking rate is defined as the average firing rate per neuron at layer l.
  • Total ANN and SNN inference energy is obtained across layers using energy costs from a standard 45 nm CMOS process.Table 4 provides the energy table for the 45 nm CMOS process.

E. Appendix: Energy Comparison in Neuromorphic Architecture

The appendix evaluates BNTT energy efficiency on the TrueNorth neuromorphic architecture using dynamic and static energy components, with conversion as the comparison reference.

  • TrueNorth energy is divided into dynamic energy for computing cores and routers and static energy for maintaining CMOS circuit state.The normalized-energy calculation follows prior work and distinguishes Edyn from Esta.
  • Table 5 compares normalized energy on CIFAR-10 with a VGG9 architecture, using conversion as the reference.The appendix states that BNTT has a substantial energy-efficiency advantage on neuromorphic hardware.
  • Total energy is calculated as #Spikes × Edyn + #Time-step × Esta, using Edyn = 0.4 and Esta = 0.6.The formula accounts for both spike-driven computation and time-dependent state maintenance.

F. Appendix: Adversarial Robustness

The appendix tests robustness to adversarial inputs by varying FGSM attack intensity and comparing classification performance across ANN and SNN approaches.

  • FGSM adversarial samples are generated by perturbing input images according to the loss gradient and attack strength ϵ.For SNNs, the experiments use an SNN-crafted FGSM method proposed in prior work.
  • Across varying FGSM intensities, BNTT and Surrogate BP are reported as more robust than ANN, with BNTT showing the highest robustness.Figure 9 plots classification accuracy against attack intensity.
  • The appendix also identifies Layer Normalization as a comparison method for BNTT on CIFAR-10 using a VGG9 architecture.Table 6 is labeled as a comparison with Layer Normalization on CIFAR-10.
Loading 2010.01729v5…