PaperlayerView exact revision on arXiv

Source-linked AI summary

Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks

Jibin Wu, Chenglin Xu, Daquan Zhou, Haizhou Li, Kay Chen Tan

arXiv:2007.01204v1cs.NE

TL;DR

Deep SNNs are efficient but difficult to train and convert for rapid inference. The paper proposes progressive tandem learning, which combines spike-count ANN-to-SNN conversion, layer-wise adaptive fine-tuning, and progressive hardware constraints. It demonstrates competitive performance across object recognition, image reconstruction, and speech separation while targeting rapid, efficient neuromorphic inference.

  • Problem

    Deep SNN training is difficult because BPTT over sparse spike trains is computation- and memory-inefficient, limiting scalability.

  • Method

    Progressive tandem learning uses spike-count ANN-to-SNN conversion followed by layer-wise adaptive fine-tuning and progressive hardware-aware training.

  • Results

    The trained SNNs show competitive classification and regression across ImageNet-12 object recognition, image reconstruction, and speech separation tasks.

  • Takeaways & Limitations

    The framework supports rapid and efficient inference on low-power neuromorphic devices by using the available encoding time window.

  • Takeaways & Limitations

    Conversion can incur quantization error from the encoding window and spike-count approximation error caused by temporally structured input spike trains.

Abstract

from arXiv · show

Spiking neural networks (SNNs) have shown clear advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency, due to their event-driven nature and sparse communication. However, the training of deep SNNs is not straightforward. In this paper, we propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition, which is referred to as progressive tandem learning of deep SNNs. By studying the equivalence between ANNs and SNNs in the discrete representation space, a primitive network conversion method is introduced that takes full advantage of spike count to approximate the activation value of analog neurons. To compensate for the approximation errors arising from the primitive network conversion, we further introduce a layer-wise learning method with an adaptive training scheduler to fine-tune the network weights. The progressive tandem learning framework also allows hardware constraints, such as limited weight precision and fan-in connections, to be progressively imposed during training. The SNNs thus trained have demonstrated remarkable classification and regression capabilities on large-scale object recognition, image reconstruction, and speech separation tasks, while requiring at least an order of magnitude reduced inference time and synaptic operations than other state-of-the-art SNN implementations. It, therefore, opens up a myriad of opportunities for pervasive mobile and embedded devices with a limited power budget.

1 INTRODUCTION

Deep SNNs promise efficient event-driven pattern recognition but remain difficult to train at scale because spike generation is discrete and standard backpropagation is not directly applicable. The paper introduces progressive tandem learning to combine spike-count conversion, layer-wise correction, adaptive scheduling, and hardware-aware training.

  • 1 INTRODUCTION: Deep SNN training remains challenging because discrete, non-differentiable spike generation prevents direct application of standard backpropagation.
  • 1 INTRODUCTION: ANN-to-SNN conversion methods approximate analog activations with firing rates, but long time windows create an accuracy–latency trade-off.
  • 1 INTRODUCTION: Progressive tandem learning combines layer-wise ANN-to-SNN conversion with adaptive weight fine-tuning to compensate conversion errors.
  • 1 INTRODUCTION: Spike-count representation reframes conversion as discrete activation matching, reducing inference speed and computational costs relative to firing-rate approximation.
  • 1 INTRODUCTION: The framework progressively incorporates constraints such as limited weight precision and fan-in connections during training for neuromorphic deployment.
  • 1 INTRODUCTION: The study evaluates deep SNNs on speech separation, presenting the cocktail-party problem as an application of the proposed learning framework.

2 RELATED WORK

Earlier ANN-to-SNN conversion methods primarily use firing-rate approximations, whose accumulated errors force long encoding windows. The paper instead uses bounded spike counts and layer-wise learning to target rapid, accurate deep-SNN inference.

  • 2 RELATED WORK: Prior methods address conversion errors through threshold selection, weight normalization, reset-by-subtraction, and residual-network error compensation.
  • 2 RELATED WORK: Firing-rate conversion methods approximate continuous ANN activations but typically require hundreds to thousands of time steps for competitive deep-SNN accuracy.
  • 2 RELATED WORK: Surplus membrane potential and approximation errors accumulate across layers, motivating larger thresholds or longer encoding windows.
  • 2 RELATED WORK: The proposed method uses spike count to approximate discrete ANN representations, exploiting the finite encoding window instead of relying on stable firing rates.
  • 2 RELATED WORK: Layer-wise learning fine-tunes weights after primitive conversion to counteract conversion errors and support rapid, efficient recognition.

3 RETHINKING ANN-TO-SNN CONVERSION

The paper reframes ANN-to-SNN conversion as mapping between discrete activation and spike-count representations, enabling shorter encoding windows while addressing conversion errors through threshold selection and layer-wise compensation.

  • Neural Discretization vs Activation Quantization: The encoding time window defines the SNN’s discrete representation space, so mapping quantized ANN representations can reduce inference time and improve efficiency.The paper connects reduced spatial representation precision in ANNs with shorter temporal encoding windows in SNNs.
  • Neural Discretization vs Activation Quantization: Activation quantization and IF-neuron discretization provide a correspondence between quantized ReLU outputs and spike counts, forming the basis for ANN-to-SNN conversion.The mapping uses a layer-specific quantization scale for analog activations and firing threshold for spiking neurons.
  • Neural Discretization vs Activation Quantization: The activation distribution is heavily concentrated at low values, while neuron counts decrease rapidly as activation values increase.The figure marks the 99th percentile for each layer’s activation distribution.
  • Threshold LayerNorm: Primitive conversion introduces quantization and spike-count approximation errors, which are mitigated by threshold normalization and layer-wise training.The approximation error can depend on encoding-window size and the temporal structure of input spike trains.
  • Threshold LayerNorm: A layer’s firing threshold is determined from its activation upper bound and encoding window, with weights copied from the pretrained ANN and biases converted into constant injecting currents.The activation upper bound can be obtained from the maximum activation value or a random training batch.

4 PROGRESSIVE TANDEM LEARNING

Progressive tandem learning converts ANN layers into SNN layers sequentially and fine-tunes each stage to reduce conversion errors. The framework also adapts training schedules and progressively incorporates hardware constraints.

  • 4 PROGRESSIVE TANDEM LEARNING: Layer-wise conversion and fine-tuning completes an L-layer ANN-to-SNN conversion in L stages, addressing errors that accumulate across layers.Each stage converts one ANN layer into an SNN layer and fine-tunes its weights.
  • 4.1 Tandem Learning: A layer’s firing threshold is determined with Threshold LayerNorm, and its bias-driven injecting current is scaled by the encoding window size.All spiking neurons in the same layer share the threshold, while the bias term determines the constant input current.
  • 4.1 Tandem Learning: Spike counts bridge the coupled SNN and ANN layers: spiking neurons produce counts, while analog neurons approximate those counts for weight training.This tandem configuration supports error-gradient approximation from the ANN layer.
  • 4.2 Adaptive Training Scheduler: The adaptive scheduler updates validation-loss patience after each epoch and automates when each progressive training stage proceeds.The patience counter resets after validation improvement and otherwise increases.
  • 4.3 Optimizing for Other Hardware Constraints: PTL progressively imposes hardware constraints, including low-precision weights and limited fan-in, during training for hardware-algorithm co-design.Quantization-aware training shares quantized weights and biases with the SNN while retaining full-precision copies in the ANN.

5 EXPERIMENTS ON PATTERN CLASSIFICATION

The experiments evaluate progressive tandem learning for scalable, accurate, and efficient deep SNN classification under reduced time, hardware precision, and synaptic-operation budgets.

  • 5.2 Accumulated Errors with Spike-based Learning Methods: Deep spike-based learning becomes difficult beyond 10 layers, whereas progressive layer-wise fine-tuning scales to SNNs with 16 layers.The experiments attribute the degradation to accumulated gradient-approximation errors across layers.
  • 5.3 Object Recognition on Cifar-10 and ImageNet-12: After each ANN-to-SNN layer conversion, validation and test accuracy recover within a few epochs and remain stable, eventually surpassing the pretrained ANN.The layer-wise learning method compensates for conversion errors during progressive training.
  • 5.3 Object Recognition on Cifar-10 and ImageNet-12: 90.86% and 91.24% test accuracy were achieved by spiking AlexNet and VGG-11 on Cifar-10, outperforming their pretrained ANN baselines by 1.27% and 0.65%.These models achieved state-of-the-art accuracy among comparable SNN implementations.
  • 5.3 Object Recognition on Cifar-10 and ImageNet-12: ImageNet-12 spiking AlexNet improved top-1 and top-5 accuracy by 3.39% and 2.21% over an earlier constrain-then-train method while reducing time steps from 200 to 16.Spiking VGG-16 remained competitive while using at least 25 times fewer time steps.
  • 5.5 Rapid and Efficient Classification with SNN: On Cifar-10, a single encoding time step can produce reliable predictions, while larger time windows improve performance.The reported accuracy increases positively with the encoding time window.
  • 5.5 Rapid and Efficient Classification with SNN: A patience period of 1 yields accuracy above the pretrained ANN with an average of only 18 training epochs, while longer patience can further improve accuracy.The adaptive scheduler therefore trades training duration against classification accuracy.

6 EXPERIMENTS ON SIGNAL RECONSTRUCTION

The paper extends progressive tandem learning beyond classification to signal reconstruction, targeting high-fidelity image and speech outputs that are challenging for SNNs.

  • 6 EXPERIMENTS ON SIGNAL RECONSTRUCTION: Signal reconstruction is presented as a challenging SNN application because regression requires high-precision outputs from spikes.The experiments therefore cover image reconstruction and speech separation, both requiring high-fidelity signal reconstruction.
  • 6.2 Time-domain Speech Separation: The speech-separation system takes a two-speaker mixture as input and produces two independent output streams, one for each speaker.Figure 7 depicts the approach and its network architecture.

6.1 Image Reconstruction with Autoencoder

The paper frames image reconstruction through an autoencoder that compresses signals into a compact latent representation before reconstructing the original input.

  • 6.1 Image Reconstruction with Autoencoder: An autoencoder learns a compact latent representation through a lower-dimensional bottleneck and uses it to reconstruct the original signal.This representation can ignore variation, remove noise, and disentangle mixed information.

6.2 Time-domain Speech Separation

The proposed time-domain SNN separates mixed speech by estimating speaker-specific masks over an encoded representation and reconstructing each speaker’s waveform.

  • 6.2 Time-domain Speech Separation: Speech separation addresses the cocktail-party problem of selectively recovering an individual speaker from a multi-talker mixture.The paper motivates this task as a challenging auditory scene-analysis problem for machines.
  • 6.2 Time-domain Speech Separation: Dilated convolutional layers capture long-range speech dependencies with a manageable number of parameters.The model is optimized with SI-SDR loss for high-fidelity speech reconstruction.
  • 6.2 Time-domain Speech Separation: The network uses an encoder, separator, and decoder to transform a mixture waveform into separate speaker signals.The encoder forms a high-dimensional representation, the separator estimates masks, and the decoder reconstructs time-domain outputs.

6.3 Experimental Setup

The experiments evaluate PTL-converted SNNs for MNIST image reconstruction and two-talker speech separation, using task-specific architectures, training procedures, and metrics.

  • Image Reconstruction: MNIST reconstruction uses a 784-128-64-32-64-128-784 autoencoder with sigmoid output and free aggregate membrane potential for high-resolution decoding.The final SNN-layer membrane potential is used as the sigmoid pre-activation rather than spike count.
  • Image Reconstruction: The image-reconstruction networks train for 100 epochs with MSE loss, scheduler patience T_p=6, and varied encoding windows for test-set evaluation.
  • Speech Separation: The WSJ0-2mix evaluation contains 20,000 training, 5,000 development, and 3,000 test utterances sampled at 8 kHz.The corpus comprises approximately 30, 8, and 5 hours for training, development, and test sets, respectively.
  • Speech Separation: Speech separation uses a Conv-TasNet-inspired SNN with 512 filters, 20-sample windows, 10-sample stride, and normalized encoded representations.The system first applies a one-dimensional convolution and ReLU before separator processing.
  • Speech Separation: The converted speech-separation SNN uses aggregate membrane potential at the final 1×1 convolution, while SI-SDR with permutation-invariant training evaluates separation quality.The ANN is trained first, then its separator is converted into an SNN; SI-SDR compares original and converted systems.

6.4 Experimental Results

PTL-converted SNNs achieve high-quality MNIST reconstruction and speech separation close to ANN performance, with longer encoding windows benefiting regression reconstruction.

  • Image Reconstruction: 0.00662 MSE at N_s=32 slightly improves on the pre-trained ANN’s 0.00667, while reconstructed MNIST images remain high quality.The results indicate that regression reconstruction may require a larger discrete representation space or encoding window than classification.
  • Comparative Evaluation: Table 3 compares image-reconstruction MSE across encoding-window sizes, while Table 4 compares ANN and SNN separation under closed and open conditions.Table 4 distinguishes different-gender, same-gender, and overall mixtures.
  • Speech Separation: 12.2 dB SI-SDR and 2.85 PESQ for the SNN remain close to the ANN’s 12.8 dB and 2.94 under open-condition speech separation.The open condition evaluates unseen speakers, and the comparison covers both SI-SDR and perceptual quality.
  • Speech Separation: SNN-separated speech examples show high-fidelity similarity to ANN outputs, with a male-male mixture spectrum similar to the clean ground truth.

7 CONCLUSION

The paper presents PTL as an ANN-to-SNN conversion and layer-wise learning framework that uses spike counts, adaptive scheduling, and progressive hardware constraints for efficient deep-SNN inference.

  • Contributions: PTL uses spike count to represent analog activation space, improving use of limited representation space and inference speed.
  • Contributions: Layer-wise learning compensates for primitive conversion errors, while an adaptive scheduler automates flexible and efficient training.
  • Hardware Co-design: PTL supports progressive algorithm-hardware co-design by imposing constraints such as limited weight precision and fan-in during training.Quantization-aware training illustrates how hardware constraints can be introduced before deployment on neuromorphic hardware.
  • Results: The trained SNNs show competitive classification and regression across ImageNet-12 recognition, image reconstruction, and speech separation tasks.
  • Implications: The framework targets rapid, efficient inference on low-power pervasive devices by combining deep-SNN computation with neuromorphic hardware.
Loading 2007.01204v1…