PaperlayerView exact revision on arXiv

Source-linked AI summary

STDP Based Pruning of Connections and Weight Quantization in Spiking Neural Networks for Energy Efficient Recognition

Nitin Rathi, Priyadarshini Panda, Kaushik Roy

arXiv:1710.04734v1cs.NE

TL;DR

Large SNNs challenge crossbar hardware and energy efficiency as synapse counts grow. The paper prunes and quantizes connections using unsupervised STDP, reporting improvements in energy, area, and testing speed.

  • Problem

    Large SNNs strain memristive crossbar size limits and increase computation, making networks slower and less energy-efficient.

    Source: §1 · p. 1

  • Method

    The approach uses unsupervised STDP weights to identify critical connections, prune others, and quantize and share the retained weights during training.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · pp. 4–5

  • Results

    The sparse SNN reports 2–3x improvement in energy, 2–4x in area, and 2–3x in testing speed.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The authors position the compressed topology as a way to improve crossbar scalability while maintaining classification accuracy.

    Source: §1 · pp. 1–2

Abstract

from arXiv · show

Spiking Neural Networks (SNNs) with a large number of weights and varied weight distribution can be difficult to implement in emerging in-memory computing hardware due to the limitations on crossbar size (implementing dot product), the constrained number of conductance levels in non-CMOS devices and the power budget. We present a sparse SNN topology where non-critical connections are pruned to reduce the network size and the remaining critical synapses are weight quantized to accommodate for limited conductance levels. Pruning is based on the power law weight-dependent Spike Timing Dependent Plasticity (STDP) model; synapses between pre- and post-neuron with high spike correlation are retained, whereas synapses with low correlation or uncorrelated spiking activity are pruned. The weights of the retained connections are quantized to the available number of conductance levels. The process of pruning non-critical connections and quantizing the weights of critical synapses is performed at regular intervals during training. We evaluated our sparse and quantized network on MNIST dataset and on a subset of images from Caltech-101 dataset. The compressed topology achieved a classification accuracy of 90.1% (91.6%) on the MNIST (Caltech-101) dataset with 3.1x (2.2x) and 4x (2.6x) improvement in energy and area, respectively. The compressed topology is energy and area efficient while maintaining the same classification accuracy of a 2-layer fully connected SNN topology.

1. INTRODUCTION

Large SNNs challenge energy efficiency and emerging crossbar hardware as their size and synapse counts grow. The paper proposes pruning and quantizing synapses using unsupervised STDP.

1. INTRODUCTION

  • Larger SNNs can compromise energy efficiency, while numerous synapses increase computation and make networks slower and less energy-efficient.The introduction also notes that improving MNIST accuracy by 12% in one cited study required 64x more neurons in a two-layer SNN.Source: §1 · p. 1
  • Large SNNs on memristive crossbars face crossbar-size limits, voltage degradation, process-variation noise, and sneak paths.Numerous synapses also increase computation, making networks slower and less energy-efficient.Source: §1 · p. 1
  • The proposed sparse, quantized SNN is reported to improve energy by 2–3x, area by 2–4x, and testing speed by 2–3x.The introduction presents these gains as relevant to efficient implementation in crossbar arrays, which face size and device-nonideality constraints.Source: §1 · p. 1
  • The approach uses unsupervised STDP weights to identify critical connections, temporarily prunes connections below a preset threshold, and quantizes critical weights.The introduction motivates this approach by noting that unsupervised SNN learning lacks the defined error function used to identify low-saliency parameters in supervised networks.Source: §1 · p. 2

2. BACKGROUND

The background describes the LIF neuron and STDP learning rule, then outlines the network topology and its unsupervised training and testing procedure.

2.1. Neuron & Synapse Model and STDP Learning

  • LIF neurons integrate input spikes into membrane potential, which decays toward rest; crossing threshold triggers a spike, reset, and refractory inactivity.The paper adopts a power-law weight-dependent STDP model in which synaptic weight changes depend on pre- and post-neuron spike timing.Source: §2.1 · p. 2

2.2. Network Topology

  • The input layer connects fully to excitatory neurons, which connect one-to-one to inhibitory neurons; inhibitory feedback supports lateral inhibition.The excitatory layer uses an adaptive threshold that rises when a neuron fires and decays over time, promoting distinct learned features.Source: §2.2 · pp. 2–3, Figure 2 · p. 2
  • Pruning and weight quantization are applied to excitatory synapses, while the fully connected topology serves as the baseline.Figure 2 specifies that pruning applies only to input-to-excitatory connections.Source: Figure 2 · p. 2, §2.2 · p. 3

2.3. Training & Testing

  • Training is unsupervised and converts image pixels into Poisson spike trains; testing predicts the label associated with the strongest average excitatory response.Neuron labels are assigned from their average spiking activity across images.Source: §2.3 · p. 3

3. COMPRESSION TECHNIQUES

The paper compresses a fully connected SNN by using STDP to identify critical synapses, pruning other connections, and quantizing retained weights. Pruning and quantization recur during batch training, with quantization levels representing available conductances.

3.1. STDP Based Pruning

  • STDP changes synaptic weight according to pre- and post-neuron spike timing, with weight change also dependent on the previous weight.The model’s update uses spike timing, a learning rate, a time constant, a depression offset, a maximum weight, and a weight-dependence constant.Source: §3.1 · p. 3, Figure 3 · p. 3

3. COMPRESSION TECHNIQUES

  • The figures depict STDP weight change over temporal spike correlation, the compression algorithm, rearranged input-to-excitatory weights, and connectivity and accuracy across pruning settings.Fig. 6 also compares accuracy across different numbers of excitatory neurons.Source: Figure 3 · p. 3, Figure 4 · p. 4, Figure 5 · p. 5

3.1. STDP Based Pruning

  • STDP-trained weights above a threshold mark critical synapses; other weights are zeroed during training and remaining non-critical connections are permanently removed at the final step.Zeroed connections continue participating in later training batches and may become critical for new inputs.Source: §3.1 · p. 3

3.2. Weight Sharing and Quantization

  • In 2-level quantization, critical synapses share their average weight, while 3-level quantization assigns two conductance values using the median of critical weights.The 2-level network uses 0 and one shared value; the 3-level network uses 0, a lower value, and a higher value.Source: §3.2 · pp. 4–5
  • The algorithm starts with a fully connected, randomly weighted network, delays its first pruning step for M batches, then prunes and quantizes after each remaining batch.The delay allows randomly initialized synapses time to capture input features before critical connections are identified.Source: §3.2 · p. 5

4. EXPERIMENTAL METHODOLOGY

The proposed topology was simulated in BRIAN, using the same neuron and synapse model parameters as a cited prior work. Evaluation covered MNIST digit recognition and image recognition on a subset of Caltech 101.

4. EXPERIMENTAL METHODOLOGY

  • The network was simulated in the open-source BRIAN spiking-neuron simulator implemented in Python.BRIAN supports biologically plausible neurons and synapses defined by differential equations.Source: §4 · p. 6
  • The neuron and synapse model parameters matched those used in.Source: §4 · p. 6
  • Evaluation tested digit recognition on MNIST and image recognition on a subset of Caltech 101 images.Source: §4 · p. 6

A. MNIST Dataset

The MNIST setup uses 28×28 grayscale digit images and a 784-input, 100-excitatory-neuron network. The compressed network begins pruning and quantization after three batches, then repeats both after each subsequent batch.

A. MNIST Dataset

  • MNIST contains 28×28 grayscale digits, giving the input layer 784 neurons.Source: A. MNIST Dataset · p. 6
  • The compressed MNIST network begins pruning and quantization after three batches totaling 15,000 training images, then repeats both after each later batch.The 60,000 training images are divided into 12 batches of 5,000; the baseline is trained on all 60,000 images in sequence.Source: A. MNIST Dataset · p. 6
  • Figure 5(c) shows rearranged synaptic weights of a network pruned at threshold 0.3 and quantized to 2 levels; the same network is also described with continuous weight values.Source: A. MNIST Dataset · p. 7

B. Caltech 101 Dataset

The Caltech 101 evaluation uses a balanced 10-category subset, resized to 28×28 grayscale images, and applies pruning and 2-level quantization after an initial training period.

B. Caltech 101 Dataset

  • The selected Caltech 101 subset contains 10 categories, with 1,000 training images per category and 80%/20% training/testing splits.The images are copied to balance category counts; preprocessing converts to grayscale, smooths with a 3×3 Gaussian kernel, and resizes to 28×28 pixels.Source: B. Caltech 101 Dataset · p. 7
  • The compressed Caltech network starts pruning and quantization after ten batches totaling 5,000 images, then repeats them after each remaining batch.The training set has 10,000 images divided into 20 batches of 500; the pruning threshold is 0.2 and weights are quantized to 2 levels.Source: B. Caltech 101 Dataset · p. 7

5. RESULTS & ANALYSES

The results compare pruning thresholds, excitatory-neuron counts, and pruning during training, then assess energy savings against the baseline. Compressed networks reduce connectivity substantially while retaining much of baseline accuracy.

5.1. Comparison with varying pruning threshold

  • Compressed accuracy reached 79.5% (MNIST) and 82.8% (Caltech 101), with 75% and 36% greater sparsity than baseline, respectively.Higher pruning thresholds produce sparser networks; early weight sharing can reduce the effect of STDP depression and lower the probability of connection removal.Source: §5.1 · pp. 7–8

5.2. Comparison with varying number of neurons

  • With 6,400 baseline neurons, accuracy was 93.2% (MNIST) and 94.2% (Caltech 101); pruned networks reached 91.5% at 8% and 92.8% at 12% connectivity, respectively.The best-performing compressed settings used 27% connectivity for MNIST and 42% for Caltech 101, with weights quantized to 3 levels.Source: §5.2 · p. 8

5.3. Pruning while training

  • Pruning connections during training performs better than training an initially sparse network with the same connectivity, because it removes non-critical connections rather than random ones.The authors interpret this comparison as evidence that STDP identifies non-critical connections.Source: §5.3 · p. 8

5.4. Reduction in spike count or energy

  • 3.1x and 2.2x improvement in energy is reported for the pruned topology; the 2-level quantized network achieves 2.4x and 1.92x for MNIST and Caltech101, respectively.Fig. 8 presents normalized energy improvement relative to the baseline topology.Source: §5.4 · p. 8, Figure 8 · p. 8

6. CONCLUSIONS

The conclusion presents STDP-guided pruning and weight quantization as SNN compression techniques. The compressed topology reports energy gains and faster training convergence on MNIST and Caltech 101.

6. CONCLUSIONS

  • STDP determines which connections to prune and quantizes the weights of retained critical connections to levels suited to device and technology requirements.The paper compares the compressed topology with a 2-layer fully connected SNN for MNIST and Caltech 101 recognition.Source: §6 · p. 8
  • 3.1x and 2.2x improvement in energy is reported on MNIST and Caltech 101, respectively, compared with the baseline fully connected SNN.Source: §6 · p. 8
  • Training time was reduced by 3x on MNIST and 2x on Caltech 101 through faster training convergence.Source: §6 · p. 8

Related papers

Loading 1710.04734v1…