Source-linked AI summary
Supervised learning based on temporal coding in spiking neural networks
Hesham Mostafa
TL;DR
The paper tackles the difficulty of applying gradient descent to feedforward spiking networks with discrete spike generation. It trains temporal-code networks directly through differentiable spike-time relationships, obtaining sparse activity and rapid classification while retaining a limitation in MNIST accuracy and generalization.
Problem
Feedforward spiking neural networks lack a general gradient-based training technique because discrete spike generation and spike communication hinder direct application of ANN backpropagation.
Method
The paper directly relates output spike times to causally influential input spike times and minimizes differentiable spike-time costs using gradient descent.
Results
The temporal-code networks spike sparsely and classify rapidly, selecting an MNIST class after an average of 25 hidden-layer spikes in the 784-800-10 network.
Takeaways & Limitations
Temporal coding provides a way to train multilayer spiking networks directly while enabling detailed spike-time control and efficient neuromorphic processing.
Takeaways & Limitations
MNIST performance falls short of state-of-the-art fully connected ANNs, while overfitting indicates that further regularization is needed for better generalization.
Abstract
from arXiv · showhide
Gradient descent training techniques are remarkably successful in training analog-valued artificial neural networks (ANNs). Such training techniques, however, do not transfer easily to spiking networks due to the spike generation hard non-linearity and the discrete nature of spike communication. We show that in a feedforward spiking network that uses a temporal coding scheme where information is encoded in spike times instead of spike rates, the network input-output relation is differentiable almost everywhere. Moreover, this relation is piece-wise linear after a transformation of variables. Methods for training ANNs thus carry directly to the training of such spiking networks as we show when training on the permutation invariant MNIST task. In contrast to rate-based spiking networks that are often used to approximate the behavior of ANNs, the networks we present spike much more sparsely and their behavior can not be directly approximated by conventional ANNs. Our results highlight a new approach for controlling the behavior of spiking networks with realistic temporal dynamics, opening up the potential for using these networks to process spike patterns with complex temporal information.
1 Introduction
The paper addresses the lack of a general training method for feedforward spiking networks by using temporal coding and directly optimizing differentiable spike-time objectives. This approach supports multilayer networks with sparse, temporally controlled activity rather than rate-coded ANN emulation.
- Research gap: Feedforward spiking networks lack a general training technique comparable to ANN backpropagation, especially for multilayer architectures.Prior approaches often require direct input-output connections or face scalability concerns in stochastic formulations.
- Approach: The proposed method directly relates each output spike time to causally influential input spike times and minimizes a differentiable cost with gradient descent.It uses deterministic neural and synaptic dynamics without explicit delay elements.
- Approach: Temporal coding replaces discrete spike counts or rates with continuous spike times, making gradient-based optimization and detailed single-spike control possible.The method therefore targets spike timing directly instead of first training an ANN and translating its weights.
- Temporal coding: The networks can spike sparsely because spike timing carries substantial information, unlike rate-based systems that use spike rates to encode analog quantities.Their behavior also differs substantially from conventional ANNs because each neuron depends on a dynamically changing subset of preceding neurons.
- Related approaches: The resulting framework extends temporal learning to multilayer networks using realistic deterministic dynamics rather than pre-specified delay combinations.This distinguishes it from SpikeProp-style models based on multiple delayed sub-connections.
2 Network model
The model uses non-leaky integrate-and-fire neurons with exponentially decaying synaptic currents, representing each neuron’s output by its first spike time. After transforming spike times into the z-domain, the input-output relation is locally linear and changes when the dynamically determined causal input set changes.
- Neuron and synapse dynamics: Non-leaky integrate-and-fire neurons integrate exponentially decaying synaptic currents, spike at threshold 1, and reset their membrane potential to 0.Synaptic current decays with time constant τsyn, normalized to 1; neurons may have positive or negative weights and may briefly have negative membrane potential.
- Causal input set: Only input spikes arriving before the output spike belong to the causal set and influence the neuron’s first spike time.The causal input weights must sum to more than 1 for the neuron to cross threshold.
- z-domain transformation: Exponentiating spike times, exp(tx) → zx, converts the neuron’s input-output relation into a linear relation in the z-domain.This transformation is used because spike times appear exponentiated in the integrated membrane-potential expression.
- Spiking assumption: Each neuron emits at most one spike per input presentation and then remains refractory until the network is reset.This assumption keeps activity sparse and makes the first spike time the neuron’s output value.
- Piece-wise linearity: The neuron’s local effective weights depend on the causal set, so changing spike times can remove inputs or give previously irrelevant inputs non-zero influence.Early strong positive spikes can make later spikes irrelevant, while late source spikes may leave the causal set entirely.
- Continuity and scope: The input-to-output spike-time transformation is generally continuous and piece-wise linear in the z-domain, but it can fail to remain continuous when perturbations change whether the neuron spikes.Scaling all transformed input times shifts the output spike time correspondingly because the network has no internal time reference.
3 Training
Training uses a layer-wise feedforward implementation that computes causal sets and first-spike outputs in the z-domain, then applies backpropagation through the resulting derivatives. Additional cost and gradient controls keep neurons spiking and prevent unstable updates.
- Forward pass: The forward pass processes layer-wise feedforward networks by sorting input spike times, finding each neuron’s causal set, and evaluating its first-spike output.If no valid causal set exists, the neuron’s output is set to infinity.
- Gradient computation: Backpropagation computes gradients through the piece-wise differentiable spike-time relation, despite derivative discontinuities when causal sets change.The paper treats these discontinuities as compatible with gradient-based training, analogous to derivative discontinuities in ReLU networks.
- Weight constraints: A weight-cost term penalizes incoming weight sums below 1 so each neuron spikes when all of its input neurons spike.Without this constraint, the network can become quiescent and stop spiking.
- Weight constraints: Large positive weights can let one source neuron nearly control a target neuron’s spike time, reducing integration of information from the other inputs.The training formulation therefore identifies excessive positive weights as problematic for multi-input integration.
- Gradient stabilization: Gradient normalization clips oversized matrix gradients before each descent step to limit large parameter jumps.Gradients become large when the causal-set weight sum approaches 1 because the output time and derivatives can diverge as the denominator shrinks.
4 Results
The experiments train feedforward spiking networks on XOR and permutation-invariant MNIST using gradient descent, with MNIST results showing rapid, sparse decisions and overfitting despite strong training-set performance.
- Tasks and training: The experiments evaluate fully connected feedforward networks on XOR and permutation-invariant MNIST, using standard gradient descent to minimize the loss.XOR tests nonlinear transformation, while MNIST tests generalization to unseen input patterns.
- XOR task: In the XOR task, two input spikes arrive early or late, and the first output neuron must spike before the second when exactly one input is early.The network uses two spike sources, a four-neuron hidden layer, and two output neurons.
- MNIST classification task: MNIST inputs are binarized into spikes at times 0 or ln(6) = 1.79, and the tested topologies use either one 800-neuron hidden layer or two 400-neuron hidden layers.The dataset contains 60,000 training and 10,000 test images, with no validation set.
- MNIST classification task: Small training errors show sufficient representational power and trainability, but substantially higher test errors reveal overfitting; noisy inputs reduce test error, while dropout often prevents neurons from spiking.The authors conclude that further regularization is needed for better generalization.
- MNIST classification task: Hidden neurons are broadly tuned across classes, and no sampled hidden neuron is redundant because removing any one affects output spike times across the MNIST test set.The tuning analysis examines 30 randomly selected hidden-layer neurons in the 784-800-10 network.
5 Conclusion
The presented spiking networks can be trained with gradient descent using temporal coding, yielding piece-wise linear input-output transformations and rapid, sparse classification. Their MNIST accuracy remains below state-of-the-art ANN performance, while the framework extends toward multiple spikes and recurrent processing.
- Conclusion: Temporal spike coding enables effective gradient-descent training of feedforward spiking networks despite discontinuous spike generation and discrete spike counts.The network input-output relation becomes differentiable almost everywhere and piece-wise linear after transforming the time variable.
- Conclusion: The transformed input-output relation changes when neurons’ causal input sets change, analogous to the piece-wise linear behavior of ReLU networks.The authors describe this as the first demonstration that spiking networks effectively implement a piece-wise linear transformation from input to output spike times.
- Conclusion: Standard stochastic gradient descent was used because second-order methods are more computationally demanding than first-order methods.The paper notes that gradient-history information can often eliminate the performance gap between first- and second-order methods.
- Conclusion: The network’s MNIST performance falls short of state-of-the-art fully connected ANNs, which achieve error rates between 0.9% and 2%.The paper suggests that producing an early classification decision causes the network to ignore much hidden-neuron activity, potentially reducing accuracy.
- Conclusion: Each neuron was allowed to spike once, although the training scheme can be extended to multiple spikes and potentially recurrent configurations.Later spike times can be differentiably related to all causal input spikes, supporting possible processing of continuous input event streams.
- Conclusion: 25 hidden-layer spikes per input, on average, are sufficient for the 784-800-10 MNIST network to classify an input.Processing can stop as soon as an output spike is generated, so only a small fraction of hidden-layer spikes need to be dispatched and processed.