PaperlayerView exact revision on arXiv

Source-linked AI summary

Comprehensive SNN Compression Using ADMM Optimization and Activity Regularization

Lei Deng, Yujie Wu, Yifan Hu, Ling Liang, Guoqi Li, Xing Hu, Yufei Ding, Peng Li, Yuan Xie

arXiv:1911.00822v3cs.NEcs.LGeess.SP

TL;DR

SNN compression is less studied than ANN compression, despite the memory and compute costs that hinder deployment and the accuracy sensitivity and dynamic spikes that complicate compression. The paper combines STBP and ADMM for pruning and quantization, and adds activity regularization to reduce spike events. It reports better compression performance than prior work and identifies trade-offs among memory, spike, and overall compression.

  • Problem

    SNN compression is less studied than ANN compression, while accuracy sensitivity and event-driven dynamic spikes complicate reducing memory and compute costs.

    Source: §I · p. 1

  • Method

    The paper combines STBP and ADMM for connection pruning and weight quantization, and adds activity regularization to reduce dynamic spikes.

    Source: §I · p. 2, D. Activity Regularization · p. 6

  • Results

    The authors report much better compression performance than prior work and find trade-offs between memory compression and spike reduction, with joint-way compression recommended for better overall performance.

    Source: §V · p. 13

  • Takeaways & Limitations

    Gentle compression of multiple information types usually incurs less accuracy loss than aggressive compression of only one type, supporting joint-way compression for better overall performance.

    Source: §V · p. 13

    The authors report limited testing because of page and time budgets and call for a wider range of experiments on joint-way compression.

    Source: §V · p. 13

Abstract

from arXiv · show

As well known, the huge memory and compute costs of both artificial neural networks (ANNs) and spiking neural networks (SNNs) greatly hinder their deployment on edge devices with high efficiency. Model compression has been proposed as a promising technique to improve the running efficiency via parameter and operation reduction. Whereas, this technique is mainly practiced in ANNs rather than SNNs. It is interesting to answer how much an SNN model can be compressed without compromising its functionality, where two challenges should be addressed: i) the accuracy of SNNs is usually sensitive to model compression, which requires an accurate compression methodology; ii) the computation of SNNs is event-driven rather than static, which produces an extra compression dimension on dynamic spikes. To this end, we realize a comprehensive SNN compression through three steps. First, we formulate the connection pruning and weight quantization as a constrained optimization problem. Second, we combine spatio-temporal backpropagation (STBP) and alternating direction method of multipliers (ADMM) to solve the problem with minimum accuracy loss. Third, we further propose activity regularization to reduce the spike events for fewer active operations. These methods can be applied in either a single way for moderate compression or a joint way for aggressive compression. We define several quantitative metrics to evaluation the compression performance for SNNs. Our methodology is validated in pattern recognition tasks over MNIST, N-MNIST, CIFAR10, and CIFAR100 datasets, where extensive comparisons, analyses, and insights are provided. To our best knowledge, this is the first work that studies SNN compression in a comprehensive manner by exploiting all compressible components and achieves better results.

I. INTRODUCTION

The introduction asks how to compress SNNs while preserving functionality, given their accuracy sensitivity and event-driven computation. It proposes combining supervised learning, ADMM-based pruning and quantization, and activity regularization, against a background of SNN-specific temporal and spike-based processing.

A. Spiking Neural Networks

  • SNNs communicate between neurons with binary spike events and integrate temporal information through neuronal dynamics, unlike ANNs’ layerwise continuous signals.The paper also describes a spiking neuron as synapses and dendrites feeding the soma, which produces spikes transmitted by the axon.Source: §I · p. 1, A. Spiking Neural Networks · p. 2

I. INTRODUCTION

  • SNN compression is less studied than ANN compression because accuracy is sensitive to compression and event-driven execution adds dynamic spikes as a compression dimension.The paper frames its question around preserving SNN functionality while compressing the model.Source: §I · pp. 1–2
  • Earlier SNN-compression work either tested small-scale or fully connected models, adapted ANNs, or reported limited performance on CIFAR10.The authors identify a lack of work on comprehensive compression of standard SNNs.Source: §I · pp. 1–2
  • The paper combines STBP and ADMM for connection pruning and weight quantization, then adds activity regularization to reduce spike events and active operations.The approaches can be used separately for moderate compression or jointly for aggressive compression.Source: §I · p. 2
  • The authors present their work as the first comprehensive SNN-compression study to exploit all compressible components and define quantitative evaluation metrics.They validate the methods on pattern-recognition tasks across MNIST, N-MNIST, CIFAR10, and CIFAR100.Source: §I · p. 2

B. STBP Supervised Learning

This section introduces direct supervised STBP for learning in SNNs and describes ADMM as a way to divide constrained optimization into tractable subproblems. The resulting STBP and ADMM framework supports gradient-based updates alongside an analytical or heuristic constraint step.

B. STBP Supervised Learning

  • STBP is selected as a directly supervised SNN training method because compression needs accurate learning and ADMM relies on supervised learning.The authors avoid indirect supervision because it requires transforming between ANN and SNN models.Source: B. STBP Supervised Learning · p. 3
  • STBP uses rate coding and measures loss between ground truth and output-layer firing rates over simulation window T.Recognition uses the output neuron with the largest average firing rate.Source: B. STBP Supervised Learning · p. 3
  • Because spike outputs are thresholded, STBP uses a boxcar approximation to the derivative of the spike output with respect to membrane potential.The boxcar width is controlled by hyper-parameter a.Source: B. STBP Supervised Learning · p. 3

C. ADMM Optimization Tool

  • ADMM decomposes constrained optimization into a differentiable subproblem solved by gradient descent and a nondifferentiable subproblem handled analytically or heuristically.Its basic formulation minimizes f(X) + g(Z) subject to AX + BZ = C.Source: C. ADMM Optimization Tool · p. 3

III. SPIKING NEURAL NETWORK COMPRESSION

SNN inference compression targets memory and computation through synapse compression and spike-activity reduction. The section combines STBP supervised learning with ADMM for pruning and quantization, and describes an ADMM procedure for connection pruning.

A. Possible Compression Ways

B. ADMM-based Connection Pruning

C. ADMM-based Weight Quantization

ADMM-based weight quantization replaces the sparse-weight constraint with a quantized one. Its retraining keeps Z in a discrete weight space while W approaches Z, and the reported weight-space evolution illustrates this process.

C. ADMM-based Weight Quantization

D. Activity Regularization

Activity regularization adds a penalty on average spike rate to the loss, targeting fewer active operations. Unlike the cited ANN-to-SNN approach, it regularizes spikes directly in the SNN.

D. Activity Regularization

  • The regularizer adds the average spike rate per neuron per timestep to the vanilla loss, weighted by penalty coefficient λ.Source: D. Activity Regularization · p. 6
  • Using average spike rate rather than total spike count is intended to unify the exploration of λ settings across different networks.Source: D. Activity Regularization · p. 6
  • Directly regularizing SNN spikes does not rely on the assumption that ANN activation is proportional to the spike rate of a converted SNN neuron.Source: D. Activity Regularization · p. 6

E. Compression Strategy: Single-way or Joint-way

The paper distinguishes single-way compression, which applies one method, from joint-way compression, which combines methods. It describes four joint combinations and says they can usually achieve a more aggressive overall compression ratio.

E. Compression Strategy: Single-way or Joint-way

F. Quantitative Evaluation Metrics for SNN Compression

The evaluation metrics track residual memory, spike, and operation costs to quantify compression. They distinguish the effects of pruning, quantization, and activity regularization, while operation-cost estimates are coarse.

F. Quantitative Evaluation Metrics for SNN Compression

  • Pruning lowers memory and baseline operations, quantization lowers memory and the cost of each addition, and activity regularization mainly lowers compute cost by reducing dynamic spikes.The evaluation counts weight storage for memory and addition operations for compute, since multiplications can be removed with binary spike representation.Source: F. Quantitative Evaluation Metrics for… · p. 6
  • Activity regularization is evaluated for compute reduction; any memory-traffic benefit from fewer spikes is left out because it depends on hardware architecture.The paper limits this analysis to algorithm-level compute cost.Source: F. Quantitative Evaluation Metrics for… · p. 7
  • Residual memory cost is measured as the remaining fraction of memory after connection sparsity and weight bitwidth reduction.The supplied passages identify the metric but do not include its displayed formula.Source: F. Quantitative Evaluation Metrics for… · p. 7
  • Residual operation cost accounts for reduced baseline operations and the average spike rate, linking dynamic spike reduction to compute compression.The operation-cost estimate is described as coarse because bitwidth does not affect operation cost linearly.Source: F. Quantitative Evaluation Metrics for… · p. 7

A. Experimental Setup

Experiments evaluate compression on MNIST, N-MNIST, CIFAR10, and CIFAR100 using dataset-appropriate SNN architectures and input encoding. The study tests individual and joint compression methods while leaving the first and last layers uncompressed.

A. Experimental Setup

  • Experiments use LeNet-5 for MNIST and N-MNIST and a 10-layer spiking CNN for CIFAR10 and CIFAR100.For the CIFAR datasets, stride-2 convolutions replace pooling operations.Source: A. Experimental Setup · p. 7
  • MNIST inputs use Bernoulli sampling, while CIFAR10 and CIFAR100 use an encoding layer to convert normalized images into spike trains.The stated motivation for the CIFAR encoding layer is to improve baseline accuracy.Source: A. Experimental Setup · p. 7
  • The experiments first assess pruning, quantization, and activity regularization individually, then test two- or three-method combinations; the first and last layers remain uncompressed.Those layers are excluded because of accuracy sensitivity and their insignificant share of parameters and operations.Source: A. Experimental Setup · pp. 7–8

B. Single-way Compression

Single-way pruning, quantization, and activity regularization compress different parts of an SNN, while joint compression combines them for more aggressive reductions. Across these approaches, accuracy tolerance depends on the compression level, dataset, and mix of methods.

B. Single-way Compression

  • Pruning below roughly 40–50% causes negligible accuracy loss or can improve accuracy, while loss exceeds 2% at 75% sparsity.The reported pruning sensitivity is greater on N-MNIST than MNIST, and CIFAR10 accuracy drops faster as task difficulty and model size increase.Source: B. Single-way Compression · p. 8, Table II · p. 8
  • Weight quantization causes negligible accuracy loss at b ≥ 4 across datasets, and loss remains ≤0.52% at b = 1.At b = 2, the number of discrete levels exceeds five because scaling factors differ across layers.Source: B. Single-way Compression · p. 8, Table III · p. 8, Figure 6 · p. 8
  • Activity regularization reduces spike events and active operations, with stronger regularization producing a lower spike rate.Robustness weakens from MNIST to N-MNIST and CIFAR10; a higher baseline spike rate is associated with more room to reduce activity without much accuracy loss.Source: B. Single-way Compression · p. 9, Table IV · p. 9, Figure 7 · p. 9

C. Joint-way Compression

  • Weight quantization contributes more to memory reduction than pruning, while activity regularization contributes most to spike reduction; together Rmem and Rs determine Rops.The paper notes that its b-bit weights have 2b + 1 discrete levels.Source: C. Joint-way Compression · p. 9
  • Aggressive compression of both synapse memory and spikes is difficult without accuracy loss, and accuracy is only globally—not locally—monotonic with Rops.A too-aggressive spike-compression baseline or memory-compression baseline can cause substantial loss when the other axis changes slightly.Source: C. Joint-way Compression · p. 9
  • A MNIST combination of 25% pruning, 1-bit weights, and λ = 0.01 activity regularization yields 0.26% accuracy loss with Rmem = 2.34% and Rops = 0.91%.These ratios correspond to 42.74× memory compression and 109.89× operation compression; the authors report joint compression can reduce more operations than single-way compression at the same accuracy loss.Source: C. Joint-way Compression · pp. 9–10, Table VI · p. 11
  • Under three-way compression, N-MNIST accuracy loss is about 2% at Rmem = 0.78% and Rops = 0.04%, whereas the same compression on MNIST causes over 20% loss.The authors attribute this contrast to the temporal scattering of sparse event-driven features; they report that it can heighten sensitivity to slight compression while reducing loss under aggressive compression.Source: C. Joint-way Compression · pp. 10–11, Table VI · p. 11

D. Effectiveness of ADMM optimization

The section compares compressed SNNs trained with and without ADMM optimization. Accuracy scores are consistently better with ADMM across the reported single-way and joint-way cases.

D. Effectiveness of ADMM optimization

  • ADMM optimization produces consistently better accuracy than hard compression without ADMM across the reported single-way and joint-way cases.The authors present this comparison as evidence of ADMM's effectiveness for SNN compression.Source: D. Effectiveness of ADMM optimization · p. 11

E. Comparison with Existing SNN Compression

The paper compares its SNN compression with prior work and ANN-to-SNN conversion, then summarizes compression-strategy insights and remaining experimental and hardware constraints.

E. Comparison with Existing SNN Compression

  • Considering compression ratio and recognition accuracy, the approach achieves much higher overall compression performance than prior SNN compression works.Table VIII presents the comparison; ReStoCNet is not a pure SNN, and the models in use ANN-to-SNN conversion with usually much larger T.Source: E. Comparison with Existing SNN… · pp. 11–12, Table VIII · p. 12

F. Comparison with ANN-to-SNN-Conversion Methodology

  • With 25% connection sparsity and 3-bit weights, direct SNN compression achieves 98.59% accuracy on MNIST; converted SNN accuracy can fall when T is insufficient.The authors report that conversion methodology usually needs T tens of times larger than their T = 10 to maintain accuracy.Source: F. Comparison with… · pp. 12–13, Table IX · p. 13

V. CONCLUSION AND DISCUSSION

  • STBP and ADMM compress connections and weights, while activity regularization reduces dynamic spikes; the three methods can be used singly or jointly.The paper reports that these approaches shrink memory space, baseline operations, and active operations, respectively.Source: §V · p. 13
  • Weight quantization contributes most to memory reduction, activity regularization to spike reduction, and joint-way compression usually loses less accuracy than aggressive single-way compression.The authors also report a trade-off between Rmem and Rs, say Rops approximately reflects post-compression accuracy, and observe tolerance to weight quantization.Source: §V · p. 13
  • On event-driven datasets such as N-MNIST, accuracy drops more than on static-image datasets at low compression ratios, but the drop is described as graceful under aggressive compression.Source: §V · p. 13
  • Testing was limited by page and time budgets; the authors call for broader experiments to understand joint-way compression and uncover further insights.They specifically mention varying compression hyperparameters and benchmarking datasets.Source: §V · p. 13
  • The study uses irregular element-wise sparsity, which impedes efficient execution through indexing overhead; hardware-aware design is proposed for practical latency and energy savings.The authors suggest structured sparsity may help optimize execution performance.Source: §V · p. 13

Related papers

Loading 1911.00822v3…