Temporal Coding in Spiking Neural Networks with Alpha Synaptic Function: Learning with Backpropagation

Iulia M. Comsa; Krzysztof Potempa; Luca Versari; Thomas Fischbacher; Andrea Gesmundo; Jyrki Alakuijala

Source-linked AI summary

Temporal Coding in Spiking Neural Networks with Alpha Synaptic Function: Learning with Backpropagation

Iulia M. Comsa, Krzysztof Potempa, Luca Versari, Thomas Fischbacher, Andrea Gesmundo, Jyrki Alakuijala

arXiv:1907.13223v3Jul 30, 2019cs.NEcs.LGq-bio.NC

TL;DR

Conventional artificial networks lack intrinsic temporal coding, motivating a spiking model that learns from relative spike timing. The paper combines alpha synaptic transfer, temporal backpropagation, and synchronisation pulses, and reports successful Boolean and MNIST learning with strong benchmark performance. The authors also identify optimization discontinuities as a central challenge that training can overcome.

Problem
Conventional artificial networks lack intrinsic temporal coding, while temporal spiking networks remain difficult to train with standard supervised-learning methods.
Method
The paper trains an alpha-function spiking network by backpropagating exact temporal derivatives, encoding classes by the first output neuron to spike and using learned synchronisation pulses.
Results
The network successfully solves noisy Boolean logic and MNIST, improving non-convolutional spiking-network accuracy and achieving comparable quality to fully connected atemporal networks.
Takeaways & Limitations
Temporal coding with biologically plausible alpha synapses can support complex nonlinear computation and serve as an early building block for richer biologically inspired architectures.
Takeaways & Limitations
Optimization faces discontinuities when output neurons stop spiking or presynaptic and postsynaptic spike order changes, although training overcame these challenges.

Abstract

from arXiv · show

The timing of individual neuronal spikes is essential for biological brains to make fast responses to sensory stimuli. However, conventional artificial neural networks lack the intrinsic temporal coding ability present in biological networks. We propose a spiking neural network model that encodes information in the relative timing of individual neuron spikes. In classification tasks, the output of the network is indicated by the first neuron to spike in the output layer. This temporal coding scheme allows the supervised training of the network with backpropagation, using locally exact derivatives of the postsynaptic spike times with respect to presynaptic spike times. The network operates using a biologically-plausible alpha synaptic transfer function. Additionally, we use trainable synchronisation pulses that provide bias, add flexibility during training and exploit the decay part of the alpha function. We show that such networks can be trained successfully on noisy Boolean logic tasks and on the MNIST dataset encoded in time. The results show that the spiking neural network outperforms comparable spiking models on MNIST and achieves similar quality to fully connected conventional networks with the same architecture. We also find that the spiking network spontaneously discovers two operating regimes, mirroring the accuracy-speed trade-off observed in human decision-making: a slow regime, where a decision is taken after all hidden neurons have spiked and the accuracy is very high, and a fast regime, where a decision is taken very fast but the accuracy is lower. These results demonstrate the computational power of spiking networks with biological characteristics that encode information in the timing of individual neurons. By studying temporal coding in spiking networks, we aim to create building blocks towards energy-efficient and more complex biologically-inspired neural architectures.

1 Introduction

The paper addresses the limited temporal coding and training capabilities of conventional and spiking neural networks by proposing backpropagation for an alpha-function network encoding information in spike timing.

Motivation: Conventional artificial networks lack biological networks’ intrinsic ability to encode information through temporal spike timing.Biological spike timing supports rapid perceptual processing, whereas atemporal networks require additional processing for temporal information.
Motivation: Training temporal spiking networks is difficult because asynchronous spike sequences prevent direct application of standard supervised-learning techniques.Existing approaches include rate coding, approximate gradients, custom learning rules, and conversion from atemporal networks.
Approach: The proposed model uses alpha synaptic transfer, relative spike timing, exact temporal gradients, and learned synchronisation pulses for supervised learning.Synchronisation pulses provide input-independent learned timing signals that act as bias and increase training flexibility.
Results: The model learns temporally encoded Boolean logic and MNIST, improving state-of-the-art accuracy among non-convolutional spiking networks while matching comparable atemporal networks.The MNIST experiments use evolutionary-neural hybrid agents to search for effective hyperparameters.
Significance: The work presents temporal coding as an early building block for more complex recurrent, spike-based, and artificial-biological neural architectures.The stated aim is to increase familiarity with temporal coding in spiking neural networks and invite further exploration.

2.1 Temporal coding

The network represents features and classifications through the relative timing of single spikes, with the earliest output spike determining the predicted class.

Input encoding: Each input neuron represents one feature, with earlier spikes encoding more salient feature values.Information propagates through hidden and output layers as temporally ordered spikes.
Classification: For n possible classes, the first-spiking output neuron identifies the predicted class.An example from class k is correct exactly when output neuron k spikes first.
Learning objective: Learning changes synaptic weights so the output neurons produce the target spike order.Earlier winning spikes can indicate greater classification confidence, while non-winning spike times can represent alternative predictions.

2.2 Alpha activation function

The neuron model uses an alpha synaptic response with gradual rise and slow decay, computes threshold-crossing spike times from presynaptic inputs, and is sufficiently expressive for well-behaved functions.

Alpha response: The alpha synaptic response has the form te^-τt, rising gradually before decaying slowly and peaking at tmax = τ^-1.It is obtained by integrating exponentially decaying synaptic current kernels and provides a biologically plausible temporal response.
Membrane dynamics: The membrane potential sums weighted alpha responses from presynaptic spikes arriving no later than the evaluation time.The decay rate scales induced potential in amplitude and time, whereas synaptic weights scale amplitude only.
Spike generation: The neuron spikes when the membrane potential crosses threshold, and its spike time is computed from the minimal sufficient presynaptic subset.Inputs are sorted and added iteratively, recomputing the predicted spike when later relevant inputs must be included.
Spike-time solution: The threshold equation can have rising- and decaying-branch solutions, with the earlier valid solution selected as the neuron’s spike time.The spike time is obtained using the main branch of the Lambert W function when its argument is valid.
Expressivity: The model is powerful enough to represent any sufficiently well-behaved function.

2.3 Error backpropagation

The model trains temporal classifications by making the target output neuron spike earliest, using a loss over softmax-transformed negative spike times and exact temporal derivatives. Optimization remains possible despite changing presynaptic contributions and discontinuities when neurons stop spiking.

The correct-label neuron is trained to spike earliest, while non-target neurons are delayed through a loss based on output spike-time ordering.For n classes, the target spike time is minimized and non-target spike times are maximized.
Softmax over negative output spike times converts earlier spikes into larger class probabilities for cross-entropy training.Negating positive spike times ensures that minimizing cross-entropy favors the correct label's earliest spike.
Changing network weights advances or delays spike times throughout the network during cross-entropy minimization.
Exact derivatives relate each postsynaptic spike time to presynaptic spike times and weights, enabling temporal backpropagation.
Optimization faces changing loss landscapes when presynaptic spikes enter or leave the contributing set and discontinuities when output neurons stop spiking.The authors report that optimization remains possible despite these challenges, using a penalty described later.

2.4 Synchronisation pulses

Trainable synchronisation pulses provide temporal bias by adding adjustable spike inputs across the network or individual layers. Their timing and connections are learned alongside ordinary network weights.

Synchronisation pulses act as additional inputs that delay or advance spikes, providing temporal bias for adjusting class boundaries.The pulses are motivated partly by internally generated rhythmic activity in biological networks.
The model uses one pulse set per network for easier problems or one set per layer for more difficult problems.Per-neuron bias is avoided because it is described as biologically implausible and more computationally demanding.
Pulse connections are fully connected to all non-input neurons or to every neuron in their assigned non-input layer.
Each pulse has a predefined, trainable spike time and trainable pulse-to-neuron weights, initialized with pulse times evenly distributed between 0 and 1.Pulse times are updated with Eq. 4, while pulse weights are trained with Eq. 5.

2.5 Hyperparameters

Training uses fully connected feedforward networks optimized with Adam, specialized initialization, and mechanisms to prevent quiescent neurons and unstable gradients. Hyperparameter search selects MNIST settings, while additional tested heuristics did not improve results.

Adam minimizes mini-batch cross-entropy with separate learning rates for pulse spike times and network weights.The networks use a fixed firing threshold and decay constant, with topology represented by hidden-layer sizes.
Modified Glorot initialization uses a custom mean multiplier so initial weights can make neurons spike and pulse and non-pulse weights can specialize differently.
A fixed penalty on presynaptic weights discourages quiescence in neurons that have not fired during training.Some neurons may remain unnecessary because they spike too late to affect classification.
Table 1 organizes default Boolean-task parameters, search ranges, and the values selected for MNIST, with asterisks marking logarithmic searches.
Derivative clipping addresses gradients that approach infinity when a spike nearly reaches the postsynaptic firing threshold.This singular behavior arises as the Lambert W function approaches its minimum.
Additional heuristics, including noise, weight decay, alternative inputs, and presynaptic-neuron removal, did not improve results and were excluded.

2.6 Experiments

The experiments test temporal spike-time coding on noisy Boolean logic, concentric circles, and MNIST. MNIST brightness is encoded in input spike delays, and classification is determined by the earliest output spike after hyperparameter search and final training.

Boolean logic: Noisy AND, OR, and XOR inputs are encoded as two spike times, with the result represented by the first output neuron to spike.True and False values are sampled from separated time intervals between 0 and 1, using one synchronisation pulse.
Concentric circles: The concentric-circles task represents 2D coordinates sampled from either an inner circle or an outer annulus.
Non-convolutional MNIST: MNIST pixel brightness is linearly encoded into spike times for 784 input neurons, with darker pixels producing earlier spikes and white pixels producing none.Input spikes occur between 0 and 1, and pixels are flattened in row-major order.
Non-convolutional MNIST: The output class is the first neuron to spike among the 10 output neurons, with layer-specific pulse sets supporting deeper architectures.
Non-convolutional MNIST: An evolutionary-neural hybrid agent searches MNIST architecture and hyperparameters using reinforcement-learning-guided mutations.
Non-convolutional MNIST: The study makes no claim that the encoded temporal scales have biological meaning relative to the model's decay constant.Temporal encoding is used to test whether spiking models can solve nonlinear, complex problems.
Non-convolutional MNIST: The MNIST search uses 90% of 60000 training examples for training, 10% for validation, and selects parameters after 3394 trials based on validation accuracy after 100 epochs.
Non-convolutional MNIST: Three selected networks are then trained for 1000 epochs on all MNIST training data and evaluated on the test set.

2.7 Feature visualisation

The network’s learned representations were examined by optimizing blank or random input images toward selected output spike-time targets. This procedure produced recognizable target digits under non-negative spike-time constraints.

Feature visualisation: Feature visualisation adjusts a blank input image to minimise a selected neuron’s spike time, optionally maximising other neurons’ spike times.A one-hot target is assigned to a non-input layer, and derivatives with respect to spike time are backpropagated to the input.
Feature visualisation: The visualisation procedure starts from pixels spiking at t = 0.0 and updates the image once per epoch with learning rate 0.1.Optimization continues until the target class is produced for 10 consecutive epochs, while spike times remain non-negative.

3 Results

The temporal spiking networks solved Boolean logic tasks and achieved high MNIST accuracy, while training revealed distinct slow and fast classification regimes. Synchronisation pulses supported temporal flexibility, including exploitation of the alpha function’s decay phase.

3.1 Boolean logic problems: Adam updates applied only to misclassified examples trained the Boolean networks faster while discouraging overfitting and reducing training operations.The networks were trained using the default parameters from Table 1.
3.1 Boolean logic problems: 100% accuracy was achieved on all four Boolean logic problems by small networks with 2 hidden neurons and 1 synchronisation pulse.Training used up to 100 epochs on 1000 examples, with 150 randomly generated test examples; Figure 3 shows example temporal class boundaries.
3.2 Non-convolutional MNIST: 99.96% train accuracy and 97.96% test accuracy were obtained on MNIST with the best hyperparameters.The best candidate settings generally used batch sizes up to 5, decay rates between 0.1 and 0.3, and non-pulse learning rates usually below 10^-3.
3.2 Non-convolutional MNIST: The network spontaneously developed slow and fast operating regimes during training, with the slow regime reporting the best accuracy and the fast regime making earlier decisions.In the slow regime, output neurons usually wait for all hidden neurons and pulses; in the fast regime, the first output spike precedes the mean hidden-layer spike.
3.2 Non-convolutional MNIST: Epoch 384 marked a sudden transition from slow to fast classification in the learning dynamics shown for the test set.The transition was associated with synchronised input-layer pulses reaching spike time 0.0, after which hidden-layer pulses spiked considerably earlier.
3.2 Non-convolutional MNIST: The winning output neuron often spiked before most or all hidden-layer pulses, while all output neurons were initially inhibited.The authors suggest pulses produced late spikes from otherwise non-spiking neurons, allowing gradients from non-target neurons during training.
3.2 Non-convolutional MNIST: The learned decay constant 0.181769 placed the alpha-function maximum at t = 5.56, and later synchronisation pulses caused output neurons to spike after only some pulses.This indicates that the network exploited the decaying part of the alpha function.

4 Discussion

The discussion presents temporal coding with biologically plausible spiking dynamics as a trainable and computationally capable alternative for benchmark tasks. It also identifies biological learning, computational cost, and the complementary relationship between temporal and rate coding as important boundaries and directions.

Model and contribution: The model combines relative single-spike timing, an alpha synaptic function, differentiable temporal dynamics, and trainable synchronisation pulses.The alpha function provides a biologically plausible decay component, while synchronisation pulses provide temporal bias and training flexibility.
Empirical results: The network successfully solves noisy Boolean logic problems and MNIST, with MNIST results comparing favourably against recent spiking models, including larger or deeper architectures.The comparison concerns non-convolutional spiking networks reported in the literature.
Limitations: The model faces discontinuous loss landscapes and computationally expensive activation and training procedures, with training times ranging from a few seconds to a few hours.The discontinuities arise when neurons stop spiking or when pre- and postsynaptic spike order changes; threshold computation involves exponentials and the Lambert W function.
Biological relevance: Temporal coding enables complex, nonlinear computation with biologically plausible activation functions that encode information in individual spike timing.The discussion connects this capacity to precise spike timing observed in sensory systems such as the retina and fingertips.
Future directions: The authors frame temporal coding as a computationally powerful alternative that may complement rate coding rather than replace it in biological brains.They also propose further study of recurrent architectures, spike-based state machines, and interfaces between artificial and biological networks.
Learning mechanism: The study investigates representational capacity rather than whether backpropagation itself is a biologically plausible learning mechanism.The authors note that biological backpropagation remains debated, including requirements such as symmetrical interlayer connections.

A Appendix

The appendix establishes that alpha-function spiking networks can conditionally detect temporal input intervals and approximate sufficiently well-behaved functions. It also states a causality-based boundary on the functions temporal coding can represent.

Interval detection: A one-output spiking network with n inputs can emit a spike in (t, t + ϵ) exactly when every input lies in specified intervals.The inputs are constrained to [0, 1], t ≥ 2 + 2/τ, and the construction uses 2n + 4 neurons.
Temporal comparison: One neuron with three auxiliary pulses can conditionally produce a spike according to whether an input spike occurs before or after a fixed time.The construction uses the alpha-function potential and can implement either temporal inequality direction.
Network construction: Combining interval detectors yields an output spike if and only if the input belongs to a specified product of intervals.The shared-pulse construction requires at most 2n + 3 + 1 neurons.
Approximation theorem: For every ϵ > 0, a two-hidden-layer network can approximate any continuous function on the stated domain within ϵ.The theorem further gives a neuron-count bound when the function is Lipschitz.
Proof strategy: The proof partitions the compact input domain into small boxes, assigns each box a temporally localized detector, and combines their outputs to approximate the target function.The resulting output spike time is at most ϵ away from f(x).
Scope boundary: Temporal coding cannot represent arbitrary input-output timing relations when they violate causality, such as producing x + y/2 from inputs x and y.The appendix states that the function-domain restriction is necessary for this reason.