PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting Neuron and Synapse Filter Dynamics in Spatial Temporal Learning of Deep Spiking Neural Network

Haowen Fang, Amar Shrestha, Ziyi Zhao, Qinru Qiu

arXiv:2003.02944v2cs.NEcs.LGstat.ML

TL;DR

Rate-based SNN training overlooks temporal spike structure, while robust training of high-performance models remains challenging. The paper models SNNs as IIR-filter networks and trains synaptic weights and filter kernels; it reports state-of-the-art accuracy across varied tasks and datasets.

  • Problem

    Rate-based SNN methods focus on spike-count statistics despite information in spike timing, and robust training algorithms for high-performance models remain a challenge.

    Source: §1 · p. 1

  • Method

    The paper represents SNNs as IIR-filter networks with neuron nonlinearities and trains synaptic weights and synapse-filter kernels.

    Source: §1 · p. 2

  • Results

    The model and method achieve state-of-the-art accuracy across evaluated tasks including associative memory, MNIST, NMNIST, DVS 128 gesture, and TIDIGITS.

    Source: §6 · p. 7

  • Takeaways & Limitations

    The model is described as independent of neuron type and capable of representing complex neuron and synapse dynamics.

    Source: §6 · p. 7

    The paper assumes homogeneous synapse types, although neurons can have heterogeneous synapses with varying feedforward and feedback orders.

    Source: §3 · p. 3

Abstract

from arXiv · show

The recent discovered spatial-temporal information processing capability of bio-inspired Spiking neural networks (SNN) has enabled some interesting models and applications. However designing large-scale and high-performance model is yet a challenge due to the lack of robust training algorithms. A bio-plausible SNN model with spatial-temporal property is a complex dynamic system. Each synapse and neuron behave as filters capable of preserving temporal information. As such neuron dynamics and filter effects are ignored in existing training algorithms, the SNN downgrades into a memoryless system and loses the ability of temporal signal processing. Furthermore, spike timing plays an important role in information representation, but conventional rate-based spike coding models only consider spike trains statistically, and discard information carried by its temporal structures. To address the above issues, and exploit the temporal dynamics of SNNs, we formulate SNN as a network of infinite impulse response (IIR) filters with neuron nonlinearity. We proposed a training algorithm that is capable to learn spatial-temporal patterns by searching for the optimal synapse filter kernels and weights. The proposed model and training algorithm are applied to construct associative memories and classifiers for synthetic and public datasets including MNIST, NMNIST, DVS 128 etc.; and their accuracy outperforms state-of-art approaches.

1 Introduction

SNN training faces a gap between rate-based methods and the temporal information carried by spike timing, alongside difficulties training deep networks. The paper models SNNs as IIR filters with neuron nonlinearities and proposes learning synaptic weights and filter kernels; it reports state-of-the-art performance across several datasets.

1 Introduction

  • Rate-based SNN methods focus on spike-count statistics, although distinct temporal spike patterns can carry different information.The passage also identifies non-differentiable spike outputs and a lack of unified, robust training algorithms as challenges.Source: §1 · p. 1
  • Earlier approaches trained individual neurons on temporal patterns or used multilayer BPTT, but the latter approaches ignored synapse dynamics and filter effects.The cited neuron-level approaches were described as not scalable to multiple layers.Source: §1 · pp. 1–2
  • The paper formulates LIF neuron dynamics as IIR filters and represents SNNs as IIR-filter networks with neuron nonlinearities.The formulation exploits both synapse and neuron filter effects.Source: §1 · p. 2
  • The proposed algorithm learns synaptic weights and optimizes synapse-filter impulse-response kernels for rate-based and spatial-temporal patterns.It is described as applicable to simple LIF neurons and neurons with alpha or dual-exponential synapses.Source: §1 · p. 2
  • The algorithm outperforms state-of-the-art approaches on MNIST, neuromorphic MNIST, DVS128 gesture, TIDIGITS, and Australian Sign Language.The listed evaluations span conventional and neuromorphic datasets.Source: §1 · p. 2

2 Neuron Model

The neuron model treats membrane and synapse states as a hybrid system: they evolve continuously, while spikes trigger state updates. It interprets synaptic responses as filtered input spikes and accounts for membrane reset and its adaptive effect.

2 Neuron Model

  • The LIF neuron with dual-exponential synapses is modeled as a hybrid system whose membrane potential and synapse status evolve continuously, with spike-triggered state updates.The passage gives the model equations with reference to Brette et al. (2007).Source: §2 · p. 2
  • The linear time-invariant ODE system can be interpreted as convolving an impulse response with the input spike train to produce the spike-response model.The model relates membrane potential and output spikes to historical input spikes.Source: §2 · p. 2
  • Each synapse's postsynaptic potential is the convolution of its input spikes with a kernel, and the sub-threshold membrane potential sums the synaptic responses.This summation is stated for the case without reset.Source: §2 · p. 2
  • The biological reset can be modeled as a negative current impulse, adding a decaying reset-voltage response to the summed postsynaptic potentials.The cited reset impulse has a decay speed controlled by τr.Source: §2 · p. 2
  • Treating reset as a negative impulse enables adaptive thresholds; frequent spikes increase reset voltage, inhibiting activity and preventing over-activation.The passage says this makes weight-threshold balancing unnecessary.Source: §2 · p. 2
  • Causal synapse filters decay over time, so the current postsynaptic potential depends on all earlier input spikes and calls for temporal error propagation.Source: §2 · p. 2

3 Neuron and Synapse as IIR Filters

The section represents neuron and synapse dynamics as a network of IIR filters with neuronal nonlinearity. This formulation captures temporal behavior, can be unfolded for backpropagation through time, and accommodates different synapse models.

3 Neuron and Synapse as IIR Filters

  • For computational efficiency, the SNN is simulated in discrete time, with network states evaluated at each unit time.Source: §3 · p. 2
  • The infinite-convolution form is impractical to implement, so the system is expressed using linear constant-coefficient difference equations.The synapse filter is described as a second-order IIR filter.Source: §3 · pp. 2–3
  • Synapses sharing a fan-in neuron can share state and computation because they have identical spike histories.This avoids separately tracking the states of those synapses.Source: §3 · p. 3
  • The general formulation interprets the SNN as a network of IIR filters with nonlinear neurons, incorporating neuron and synapse dynamics.The neuron may depend on its previous states, while synapses are represented by filters.Source: §3 · p. 3
  • The general IIR form allows arbitrary postsynaptic-potential shapes and can represent more complex neuron models than LIF.The stated parameter settings model an alpha synapse or remove the synapse filter to obtain a simple LIF neuron.Source: §3 · p. 3
  • Axonal delay is modeled through delayed input spikes, enabling more complex temporal behavior; heterogeneous synapse orders are possible, though this paper assumes homogeneous types.Source: §3 · p. 3
  • The iterative equations allow the network to be unfolded over time so that backpropagation through time can be applied.The section describes a spatial and temporal data flow with a second-order synapse filter.Source: §3 · p. 3

4 Spatial Temporal Error Propagation

The section describes learning both output classes and target spike patterns by propagating errors through the spatial and temporal dynamics. It also introduces a smooth gradient surrogate for the non-differentiable spike activation.

4 Spatial Temporal Error Propagation

  • For classification, the neuron that fires most represents the predicted result, and the task uses cross-entropy loss with output spike counts.Source: §4 · p. 3
  • For temporal learning, the loss compares actual and target spike trains after filtering them into continuous traces.The target is a specified output spike pattern with spatial and temporal structure.Source: §4 · p. 4
  • Axonal delays can create dependencies from a later layer to an earlier layer at multiple time steps, and errors propagate recursively.The learning rule also computes gradients for synaptic weights and can optimize synapse-filter coefficients.Source: §4 · p. 4
  • Within each training iteration, loss calculation, error propagation, and parameter updates assume the SNN is a linear time-invariant system.Source: §4 · p. 4

4.1 Spike Function Gradient Approximation

  • The forward path retains hard-threshold spikes, while the backward path replaces their non-differentiable gradient with a smooth surrogate.The section describes spike probability as one possible surrogate, with stochasticity obtained from noise.Source: §4.1 · p. 4

5 Experiments

The experiments test associative memory, vision classification, and time-series classification. They report spatiotemporal pattern reconstruction and classification results across vision and temporal datasets.

5.1 Associative Memory

  • After 50 training epochs, the associative-memory network reconstructed noisy spatiotemporal patterns, with a 200-neuron intermediate layer learning their representation.The noisy inputs included pattern obfuscation and deletion, timing jitter, and random background spikes.Source: §5.1 · p. 5

5.2 Vision Tasks

  • MNIST results are reported as state of the art in the SNN domain, while N-MNIST matches 99.35% accuracy using a significantly smaller network without voting or WTA circuits.The MNIST comparison notes that the next-highest accuracy, 99.42%, uses an ensemble of 64 spiking CNNs.Source: §5.2 · p. 5
  • 96.09% accuracy on DVS128 Gesture is reported as state of the art in the spiking domain, with spike streams fed directly to the network without preprocessing.The model also requires no specialized neuron circuits.Source: §5.2 · p. 5, Table 1 · p. 7
  • 96.09% versus 94.14% accuracy on DVS128 followed synapse-kernel training, which also sped convergence; MNIST changed from 99.43% to 99.46%, and N-MNIST from 99.24% to 99.39%.The authors attribute the lack of meaningful MNIST improvement to its static images and absence of temporal information.Source: §5.2 · p. 6, Figure 6 · p. 6

5.3 Time Series Classification

  • On Australian Sign Language, the method outperformed vanilla LSTM and DNN approaches while classifying all 95 classes; TIDIGITS' best reported accuracy belongs to a non-spiking approach.For sign language, the input is converted to spikes on the fly rather than flattened into a vector.Source: §5.3 · p. 7, Table 2 · p. 7

6 Conclusion

The conclusion presents a general SNN model based on IIR filters with neuron nonlinearity and a learning rule for synapse weights and filter kernels. It reports evaluation on associative memory and several classification tasks, achieving state-of-the-art accuracy.

  • The proposed model formulates SNNs as networks of IIR filters with neuron nonlinearity.Source: §6 · p. 7
  • The learning rule trains synapse weights and synapse-filter impulse-response kernels.Source: §6 · p. 7
  • Evaluation spans associative memory and MNIST, NMNIST, DVS 128, and TIDIGITS tasks, with state-of-the-art accuracy reported.Source: §6 · p. 7
Loading 2003.02944v2…