PaperlayerView exact revision on arXiv

Source-linked AI summary

A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibration

Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, Shi Gu

arXiv:2106.06984v1cs.LG

TL;DR

ANN-to-SNN conversion is computationally attractive but can suffer activation mismatch, conversion error, and high latency. The paper introduces layer-wise SNN calibration to adapt converted parameters, reports state-of-the-art low-latency results on challenging ImageNet architectures, and identifies threshold outliers and an initialization approximation as scope constraints.

  • Problem

    ANN-to-SNN conversion is cheaper than training SNNs from scratch, but existing methods can suffer activation mismatch and require high latency for accurate results.

  • Method

    SNN Calibration analyzes conversion-error propagation and adjusts weights, bias, and initial membrane potential layer by layer.

  • Results

    The method establishes state-of-the-art SNN conversion performance and converts MobileNet and RegNetX-4GF at less than 256 time steps.

  • Takeaways & Limitations

    Calibration provides a low-cost way to improve converted SNNs using only a few training images.

  • Takeaways & Limitations

    Maximum pre-activation thresholds can be outliers, increasing flooring error and forcing prior methods to use very large T.

Abstract

from arXiv · show

Spiking Neural Network (SNN) has been recognized as one of the next generation of neural networks. Conventionally, SNN can be converted from a pre-trained ANN by only replacing the ReLU activation to spike activation while keeping the parameters intact. Perhaps surprisingly, in this work we show that a proper way to calibrate the parameters during the conversion of ANN to SNN can bring significant improvements. We introduce SNN Calibration, a cheap but extraordinarily effective method by leveraging the knowledge within a pre-trained Artificial Neural Network (ANN). Starting by analyzing the conversion error and its propagation through layers theoretically, we propose the calibration algorithm that can correct the error layer-by-layer. The calibration only takes a handful number of training data and several minutes to finish. Moreover, our calibration algorithm can produce SNN with state-of-the-art architecture on the large-scale ImageNet dataset, including MobileNet and RegNet. Extensive experiments demonstrate the effectiveness and efficiency of our algorithm. For example, our advanced pipeline can increase up to 69% top-1 accuracy when converting MobileNet on ImageNet compared to baselines. Codes are released at https://github.com/yhhhli/SNN_Calibration.

1. Introduction

SNNs offer energy-efficient, spatial-temporal inference, but training from scratch is difficult and conventional ANN-to-SNN conversion can suffer activation mismatch and high latency. The paper proposes layer-wise SNN calibration to improve low-latency conversion, including for challenging ImageNet architectures.

  • Motivation: SNNs binarize activations and repeat forwarding across T time steps, enabling energy efficiency and a latency–accuracy trade-off.Specialized hardware can provide orders-of-magnitude energy savings compared with ANNs.
  • Motivation: Training SNNs from scratch remains difficult to scale to ImageNet and may require T× more time than ANN training.Binary activations are nondifferentiable, while GPU training cannot exploit binary acceleration.
  • Motivation: Conventional ANN-to-SNN conversion uses copied parameters, but BN-containing ANNs can fail at low latency and activation mismatch remains a concern.Prior methods are reported to fail below or at 256 time steps for ANNs with BN layers.
  • Proposed approach: SNN calibration adjusts converted parameters to match ANN activations and narrows the activation-distribution gap.The method targets extremely low latency, defined here as less than 256 time steps, and low cost.
  • Proposed approach: The paper decomposes conversion error into flooring and clipping components, analyzes layer-wise propagation, and calibrates weights, bias, and initial potential.Light and Advanced Pipelines balance accuracy and practical utility.
  • Results: Up to 69% accuracy is increased in Spiking MobileNet conversion with 256 time steps on ImageNet.The experiments also include MobileNet and RegNetX-4GF architectures.

2. Related Work

Related work follows two main directions: training SNNs directly or converting pretrained ANNs. Conversion is cheaper and more scalable, but existing methods face accuracy–latency trade-offs and limitations on complex architectures.

  • Training-based methods: Training-based SNN methods include synaptic plasticity and surrogate-gradient approaches.Synaptic plasticity updates weights using firing-time intervals, while surrogate gradients replace the hard step function with a relaxed function.
  • Training-based methods: Surrogate-gradient training can be computationally expensive and slow on complex network architectures.This limitation concerns training SNNs directly rather than ANN-to-SNN conversion.
  • Conversion methods: ANN-to-SNN conversion requires less computation and memory than training from scratch but traditionally needs more than 2k time steps for accurate results.Recent methods reduce conversion loss or simulation length through normalization, threshold selection, reset mechanisms, and frequency-based adjustment.
  • Conversion methods: This work extends conversion experiments beyond VGG and ResNet to MobileNet and RegNet architectures.The supplied related-work passage identifies this as an extension of prior tested architectures.

3. Preliminaries

The preliminaries describe ANN and IF-neuron computation, temporal averaging in SNNs, and the conversion errors caused by binary spikes. Accurate ANN–SNN equivalence generally requires long simulations, which increases latency.

  • ANN model: An ANN layer applies a ReLU activation to a weighted input, with optional bias parameters.The layer may be fully connected or convolutional.
  • SNN model: An IF neuron accumulates binary input, emits a threshold-valued spike when its temporary potential exceeds V_th, and otherwise emits no spike.Its membrane potential is updated using the soft-reset, or reset-by-subtraction, mechanism.
  • SNN model: Weight normalization represents intermediate spikes with binary values across layers.The paper recursively applies equalization so that 0,1 spikes can represent intermediate activations.
  • SNN model: The first layer directly encodes static images into temporal spikes, while the last layer integrates inputs without firing spikes.The last-layer choice accommodates outputs that may be positive or negative.
  • Conversion: SNN conversion repeats forwarding for T steps so temporal spike averages approximate the ANN input-output mapping.The final result is based on the expected output across time steps.
  • Conversion: The approximation may require T to reach 1k or higher, while high T increases inference latency and harms practical utility.Figure 1 attributes conversion error to flooring and clipping error.

4. Methodology

The method models ANN-to-SNN conversion loss, then uses adaptive thresholds and greedy layer-wise calibration to reduce errors progressively. Light and Advanced Pipelines trade calibration cost against accuracy by adjusting bias alone or also weights and initial membrane potentials.

  • Conversion-loss analysis: Conversion loss is decomposed into flooring and clipping errors, whose propagation through layers motivates progressive layer-wise correction.The final-layer error is bounded by a weighted linear combination of layer-wise errors.
  • Adaptive threshold: MMSE selects thresholds that balance flooring and clipping errors under the simulation length T using layer-wise grid search.The threshold is selected from 100 linearly sampled grids over [0, max(x^(ℓ+1))].
  • Conversion pipeline: The converted SNN accumulates presynaptic inputs over time, while the final-layer objective minimizes the difference between expected SNN input and ANN input.Average-pooling layers are replaced with depthwise convolutions to avoid non-binary information from spike inputs.
  • Light Pipeline: Bias Calibration adds expected conversion error to the bias term and requires only one batch of training images to compute the reduced mean.This forms the Light Pipeline and uses little memory because it stores bias terms for different T.
  • Advanced Pipeline: Potential Calibration sets the initial membrane potential from the expected error, using v^(ℓ)(0) = T × e^(ℓ+1) without computing a spatial mean.This calibration corrects output-distribution mismatch caused by conversion.
  • Advanced Pipeline: Weights Calibration optimizes each SNN layer’s weight tensor to reduce conversion error, using stored ANN inputs and expected SNN inputs rather than repeated T-step convolutions.The optimization uses stochastic gradient descent with a straight-through estimator for the floor operation.

5. Experiments

Experiments evaluate threshold selection, calibration components, conversion accuracy, computational and memory cost, data requirements, and firing sparsity across CIFAR and ImageNet models. The results show that calibration improves accuracy substantially while retaining low conversion cost and latency.

  • Threshold selection: MMSE thresholding outperformed maximum-activation and percentile thresholds at T = 16, while channel-wise MMSE further improved VGG-16 from 43.5% to 47.4% and ResNet-20 from 72.1% to 73.5%.For VGG-16, MMSE was 15.7% higher in accuracy than percentile thresholding.
  • Light Pipeline: Bias Calibration consistently improved converted-SNN accuracy across thresholds, boosting VGG-16 by 22% with percentile thresholding and up to 35% with MMSE at T = 16.The light pipeline requires little memory and supports accuracy–latency trade-offs.
  • Advanced Pipeline: Potential Calibration raised MMSE-plus-BC VGG-16 accuracy from 44.95% to 59.52%, while Weight Calibration produced lower result variance.The advanced pipeline changes potential and weight parameters to adapt the ANN to spiking operation.
  • Comparison to Previous Work: On ImageNet, the advanced pipeline improved BN ResNet-34 conversion by 64.4% at T = 32 and achieved 69.02 top-1 accuracy for MobileNet at T = 256.The MobileNet baseline crashed at T ≤ 256 and required T = 2048 for acceptable accuracy; RegNetX-4GF reached 75.8% at T = 128.
  • Complexity Study: Bias and Potential Calibration require limited time, whereas MobileNet ImageNet Weight Calibration takes 30 minutes; the advanced pipeline remains cheaper than 20-epoch Hybrid Train.Runtime inference adds no additional inference time, but calibration itself requires computation.
  • Complexity Study: Calibrating ResNet-34 bias on ImageNet requires 0.3653MB, compared with 83.25MB for weights and 18.76MB for potential.The light pipeline therefore has substantially lower memory requirements than the advanced calibration components.

Conclusion

The paper analyzes conversion error and introduces adaptive thresholding plus layer-wise calibration to reduce the activation gap between ANN and SNN. The method achieves state-of-the-art SNN conversion, including low-latency conversion of challenging architectures and high accuracy with Batch Normalization.

  • Conclusion: Layer-wise calibration reduces the gap between ANN and SNN activation and significantly improves performance over simple-copy conversion methods.The method requires only a few training images.
  • Conclusion: The method establishes state-of-the-art performance for SNN conversion.
  • Conclusion: The method converts MobileNet and RegNetX-4GF at less than 256 time steps.These architectures are described as challenging conversion targets.
  • Conclusion: The method preserves high classification accuracy when converting ANNs with Batch Normalization layers.

A. Conversion Error Analysis

The analysis represents conversion error as the difference between ANN ReLU activation and an SNN clipfloor function, then studies how this error propagates through layers. With identical ANN and SNN image inputs, the first term in the recursive error expression is zero.

  • A. Conversion Error Analysis: The paper uses || · || to denote the Frobenius norm in its conversion-error analysis.
  • A. Conversion Error Analysis: The conversion error is defined as the difference between h(W(n−1)¯s(n−1)) and g(W(n−1)¯s(n−1)).Here h is the ANN ReLU function and g is the clipfloor function.
  • A. Conversion Error Analysis: The recursive analysis relies on h(·) being piecewise linear with gradient less than or equal to 1.
  • A. Conversion Error Analysis: The first term in Eq. (23) is zero when ANN and SNN use the same image input.

B. Converting Average Pooling Layers

The paper expresses AvgPool2d forwarding as an average over kernel positions and implements it through an equivalent depthwise Conv2d representation. An ablation shows that converting AvgPool2d can affect architectures differently.

  • B. Converting Average Pooling Layers: AvgPool2d receives inputs shaped [N, C, H, W] and uses a kernel specified by [kW, kH].N, C, H, and W denote batch size, channel number, input width, and input height, respectively.
  • B. Converting Average Pooling Layers: The AvgPool2d output averages input values sampled across kernel positions using the specified strides.
  • B. Converting Average Pooling Layers: AvgPool2d is represented as a depthwise Conv2d with kernel size [kW, kH], constant kernel weights 1/(kW × kH), and zero bias.
  • B. Converting Average Pooling Layers: On ResNet-34, converting AvgPool2d causes only a 2% accuracy drop, whereas its impact is much larger on VGG-16.The ablation uses ImageNet with T = 32 and the Light Pipeline.

C.1. ImageNet

The ImageNet experiments use standard image preprocessing and augmentation, while replacing Max Pooling with Average Pooling across tested architectures. The MobileNet pretrained model is downloaded from pytorchcv2.

  • C.1. ImageNet: ImageNet contains 120M training images and 50k validation images.
  • C.1. ImageNet: Training images are randomly cropped and resized to 224×224, with ColorJitter augmentation using specified brightness, contrast, saturation, and hue settings.
  • C.1. ImageNet: Test images are center-cropped to 224×224, and Max Pooling layers are replaced with Average Pooling layers for all tested architectures.
  • C.1. ImageNet: The MobileNet pretrained model is downloaded from pytorchcv2.

D. Results on CIFAR

Table 9 compares the proposed algorithm with existing SNN conversion methods on CIFAR10 and CIFAR100, while distinguishing architectural and implementation settings.

  • Table 9 compares the proposed algorithm with other existing SNN conversion works on CIFAR10 and CIFAR100.
  • “Use BN” indicates that BN layers are used to optimize the ANN.
  • “Convert AP” indicates that convolutional layers replace average pooling layers.
  • An asterisk marks results obtained through self-implementation.
Loading 2106.06984v1…