Source-linked AI summary
Learning Delays in Spiking Neural Networks using Dilated Convolutions with Learnable Spacings
Ilyass Hammouamri, Ismail Khalfaoui-Hassani, Timothée Masquelier
TL;DR
Spike arrival-time coincidence helps SNNs detect temporal patterns, while connection delays also matter to their expressive capacity. This paper learns delays alongside weights in deep feedforward SNNs using DCLS-based temporal convolutions. The method outperforms prior state of the art on three benchmarks with fewer parameters.
Problem
Connection delays affect spike arrival-time coincidence, which supports detection of complex spatiotemporal patterns.
Source: §1 · p. 1Method
Results
The method outperforms prior state of the art on SHD, SSC, and GSC-35 while using fewer parameters.
Source: §5 · p. 9Takeaways & Limitations
The results demonstrate the potential of delay learning for accurate and precise temporal-data models.
Source: §5 · p. 9For clarity, the delay formulation assumes one synapse between each pair of neurons; multiple synapses are handled in experiments.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
Spiking Neural Networks (SNNs) are a promising research direction for building power-efficient information processing systems, especially for temporal tasks such as speech recognition. In SNNs, delays refer to the time needed for one spike to travel from one neuron to another. These delays matter because they influence the spike arrival times, and it is well-known that spiking neurons respond more strongly to coincident input spikes. More formally, it has been shown theoretically that plastic delays greatly increase the expressivity in SNNs. Yet, efficient algorithms to learn these delays have been lacking. Here, we propose a new discrete-time algorithm that addresses this issue in deep feedforward SNNs using backpropagation, in an offline manner. To simulate delays between consecutive layers, we use 1D convolutions across time. The kernels contain only a few non-zero weights - one per synapse - whose positions correspond to the delays. These positions are learned together with the weights using the recently proposed Dilated Convolution with Learnable Spacings (DCLS). We evaluated our method on three datasets: the Spiking Heidelberg Dataset (SHD), the Spiking Speech Commands (SSC) and its non-spiking version Google Speech Commands v0.02 (GSC) benchmarks, which require detecting temporal patterns. We used feedforward SNNs with two or three hidden fully connected layers, and vanilla leaky integrate-and-fire neurons. We showed that fixed random delays help and that learning them helps even more. Furthermore, our method outperformed the state-of-the-art in the three datasets without using recurrent connections and with substantially fewer parameters. Our work demonstrates the potential of delay learning in developing accurate and precise models for temporal data processing. Our code is based on PyTorch / SpikingJelly and available at: https://github.com/Thvnvtos/SNN-delays
1 INTRODUCTION
Connection delays shape spike arrival times, enabling neurons to detect complex spatiotemporal patterns. This work uses temporal convolutions and DCLS to learn delays alongside weights in deep SNNs.
1 INTRODUCTION
- Heterogeneous connection delays align spike arrival times, letting neurons detect complex spatiotemporal patterns rather than only synchrony patterns.The Figure 1 example contrasts a delayed connection, which brings inputs to the threshold together, with an undelayed one.Source: §1 · p. 1, Figure 1 · p. 2
- Biological connection delays sum axonal, synaptic, and dendritic delays, vary in duration, and make learning in the brain more than synaptic plasticity.The passage notes axonal delay can be reduced through myelination, an adaptive process required for some tasks.Source: §1 · p. 2
- A theoretical result shows that an SNN with k adjustable delays can compute a richer function class than a threshold circuit with k adjustable weights.Source: §1 · p. 2
- Most neuromorphic chips have programmable synaptic delays.Source: §1 · p. 2
- The paper relates connection delays to 1D temporal convolutions and learns them with DCLS alongside weights using backpropagation in arbitrarily deep SNNs.The method is integrated with PyTorch automatic differentiation; DCLS was previously proposed for receptive-field sizing in non-spiking 2D CNNs.Source: §1 · p. 2
2 RELATED WORK
Prior deep-SNN work improved training through surrogate gradients, ANN conversion, and neuron or architecture changes. Delay-learning methods were fewer, often shallow, and some deep joint-learning approaches used finite-difference delay gradients.
2.1 DEEP LEARNING FOR SPIKING NEURAL NETWORKS
- Surrogate gradients and ANN-to-SNN conversion have enabled training increasingly deep SNNs, while other work adapts neuron dynamics, thresholds, residual networks, or attention.The cited approaches include learnable membrane time constants, dynamic thresholds, deep Spiking ResNets, and spike-based Transformers.Source: §2.1 · pp. 2–3
2.2 DELAYS IN SNNS
- Several delay-learning rules targeted shallow SNNs; other proposals used weightless STDP, optical SNNs, or multiple fixed delays followed by pruning.Source: §2.2 · p. 3
- The authors identify SLAYER and its derivatives as the only prior methods jointly learning delays and weights in deep SNNs, with finite-difference delay gradients unless using an SRM.They question gradient precision and compare this with similar performance from fixed random delays in their experiments.Source: §2.2 · p. 3
- The paper proposes a control test requiring learned delays to outperform equivalent uniformly distributed fixed random delays, especially with sparse connectivity.Source: §2.2 · p. 3
3 METHODS
The method models spiking neurons with discrete-time leaky integrate-and-fire dynamics and represents synaptic delays as temporal convolutions. It learns delay positions with smoothed kernels during training, then uses discrete sparse kernels for evaluation.
3.1 SPIKING NEURON MODEL
- The neuron model uses leaky integrate-and-fire dynamics, emitting a spike when membrane potential exceeds threshold and then resetting to the resting potential.The continuous-time dynamics are discretized with Euler’s method, using ureset = 0 and R = τ.Source: §3.1 · pp. 3–4
- Backpropagation uses a surrogate gradient, defining the threshold derivative as the derivative of a surrogate arctangent function during the backward step.The discrete-time formulation uses Euler’s method approximation.Source: §3.1 · p. 4
3.2 SYNAPTIC DELAYS AS A TEMPORAL CONVOLUTION
- Synaptic connections are modeled as 1D temporal convolutions, with kernel positions encoding delays and the resulting currents summed at each postsynaptic neuron.The kernel size is the maximum delay plus one; input spike trains use zero left-padding of size Td −1.Source: §3.2 · p. 4
- DCLS with Gaussian interpolation learns delay positions by smoothing the kernel around each delay, enabling gradients with respect to those positions.Weights and delays are learned, while all kernels share a standard deviation σ that is not learned.Source: §3.2 · p. 5
- The shared Gaussian width σ is exponentially reduced during training from a high value to 0.5, then discrete kernels are used for validation and test.The schedule starts with broader temporal dependencies and later refines weights and delays toward the discrete inference kernel; the authors report it outperforms constant σ.Source: §3.2 · p. 5
4 EXPERIMENTS
The experiments evaluate feedforward SNNs on temporal speech benchmarks and compare learned-delay models with prior methods and controlled variants. The method outperforms prior state of the art across three benchmarks, while ablations examine how delay learning and connectivity affect performance.
4.1 EXPERIMENTAL SETUP
- The evaluation uses SHD and SSC/GSC speech datasets, which require temporal pattern recognition, with feedforward SNNs containing two or three hidden fully connected layers.Spiking inputs use 140 neurons and 10 ms time steps; GSC uses 140-bin Mel spectrograms and approximately 100 time steps.Source: §4.1 · pp. 5–6
4.2 RESULTS
- The method outperforms prior state-of-the-art accuracy on SHD, SSC, and GSC-35 with substantially fewer parameters, no recurrent connections beyond LIF self-recurrence, and vanilla LIF neurons.Accuracy plateaued after two hidden layers for SHD and three for SSC/GSC; dense convolutions performed worse than DCLS.Source: §4.2 · p. 7
4.3 ABLATION STUDY
- The ablation compares the full Decreasing σ model with Constant σ, fixed-random-delay, delay-only, and no-delay models, controlling parameter counts and initializations.The no-delay controls use wider or deeper networks to match parameter counts; models share weight and, when applicable, delay initializations.Source: §4.3 · pp. 7–8
- Models with delays significantly outperform no-delay models on SHD and SSC, while fixed random delays perform nearly on par with trained delays in the fully connected setting.The Decreasing σ model only slightly outperforms Fixed random delays.Source: §4.3 · p. 8, Figure 4 · p. 9
- With only ten synaptic connections per neuron, corresponding to 96% sparsity on SHD and 98% on SSC, random delay initialization is less likely to cover the temporal range.The sparse control compares fixed random delays with Decreasing/Constant σ models and is reported to support the hypothesis and the method's effectiveness.Source: §4.3 · p. 9, Figure 4 · p. 9
- Learning both weights and delays is preferable overall, while learning weights is preferable when sparse connectivity requires choosing between weight and delay learning.Learning only delays gives acceptable results in the fully connected case but not in the sparse case.Source: §4.3 · p. 9
5 CONCLUSION
The paper proposes learning delays in feedforward SNNs with DCLS, representing each synapse as a Gaussian kernel whose position is learned. The approach outperforms prior state of the art on three benchmarks with fewer parameters, and its learned discrete delays could potentially suit neuromorphic implementations.
5 CONCLUSION
- DCLS learns Gaussian-kernel positions as synaptic delays in feedforward SNNs, with the Gaussian width reduced during training to yield one discrete delay per synapse.The method outperforms prior proposals on SHD, SSC, and GSC-35 while using fewer parameters; future work may test other kernels and architectures.Source: §5 · p. 9
A.1 SUPPLEMENTARY FIGURE
Figure 5 depicts Gaussian convolution kernels for synaptic connections and the delayed spike trains produced by convolving with them.
A.1 SUPPLEMENTARY FIGURE
- Each Gaussian is centered at its connection's delay position.Source: Figure 5 · p. 14
- The area under each Gaussian corresponds to its synaptic weight.Source: Figure 5 · p. 14
- The right side shows delayed spike trains after convolution with the kernels.The figure omits the −1 for clarity.Source: Figure 5 · p. 14