Source-linked AI summary
The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks
Benjamin Cramer, Yannik Stradmann, Johannes Schemmel, Friedemann Zenke
TL;DR
The paper addresses the lack of shared, standardized benchmarks for comparing spiking neural networks. It introduces two public spike-based datasets and a standardized audio-to-spike conversion, finding that spike timing and recurrent architectures support classification performance.
Problem
Shared benchmarks for spiking neural networks are needed to enable quantitative, unbiased comparisons and standardized evaluation across platforms.
Method
The authors create two public spike-based speech datasets, standardize raw-audio-to-spike conversion, and train spiking and non-spiking classifiers as baseline systems.
Results
Suitable classifiers leveraged spike timing for better classification, while explicitly recurrent architectures such as LSTMs and RSNNs performed best among tested architectures.
Takeaways & Limitations
The datasets, conversion software, and initial baselines provide a reference for more quantitative comparison of functional spiking neural networks on conventional computers and neuromorphic hardware.
Takeaways & Limitations
More challenging speech tasks with sentence-level alignment, and larger or imbalanced word-level corpora requiring additional preprocessing, were left for future research.
Abstract
from arXiv · showhide
Spiking neural networks are the basis of versatile and power-efficient information processing in the brain. Although we currently lack a detailed understanding of how these networks compute, recently developed optimization techniques allow us to instantiate increasingly complex functional spiking neural networks in-silico. These methods hold the promise to build more efficient non-von-Neumann computing hardware and will offer new vistas in the quest of unraveling brain circuit function. To accelerate the development of such methods, objective ways to compare their performance are indispensable. Presently, however, there are no widely accepted means for comparing the computational performance of spiking neural networks. To address this issue, we introduce two spike-based classification datasets, broadly applicable to benchmark both software and neuromorphic hardware implementations of spiking neural networks. To accomplish this, we developed a general audio-to-spiking conversion procedure inspired by neurophysiology. Further, we applied this conversion to an existing and a novel speech dataset. The latter is the free, high-fidelity, and word-level aligned Heidelberg digit dataset that we created specifically for this study. By training a range of conventional and spiking classifiers, we show that leveraging spike timing information within these datasets is essential for good classification accuracy. These results serve as the first reference for future performance comparisons of spiking neural networks.
1 Introduction
The paper addresses the lack of shared, unbiased benchmarks for comparing spiking neural networks. It proposes audio-based spike classification tasks designed to remain accessible, general, and sensitive to temporal coding.
- Why benchmarks?: Shared benchmarks enable quantitative, unbiased comparisons of SNN approaches and foster constructive competition across implementations.The authors contrast community benchmarks with private, problem-specific evaluations.
- Why benchmarks?: MNIST-to-spike conversion leaves fundamental preprocessing decisions to modelers, putting comparability at risk.
- Why benchmarks?: Existing benchmarks may be inaccessible, too specialized, saturated, or burdensome to preprocess.
- Why benchmarks?: Ideal SNN benchmarks should be unsaturated, minimally preprocessed, general, easy to obtain, and free to use.
- Previous work: Temporal XOR is limited for quantitative comparison because its low dimensionality and few input patterns make it saturated.
- Benchmark design: The proposed audio benchmarks exploit speech’s natural temporal structure and lower bandwidth, while requiring spike timing for high-accuracy classification.
2 Methods
The study constructs two speech-based spiking datasets and standardizes their audio-to-spike conversion. It also establishes baseline classifiers using recurrent and feed-forward LIF networks.
- Datasets: The study creates the Heidelberg Digits dataset and converts it alongside Speech Commands into spike-based classification data.The methods cover dataset descriptions, audio-to-spike conversion, publication format, and classifier training.
- Heidelberg Digits: HD contains 10 420 high-quality spoken-digit recordings in English and German from 12 speakers.The recordings include six female and six male speakers and approximately 40 digit sequences per language.
- Heidelberg Digits: HD recordings were made in a sound-shielded room with three microphones at 48 kHz and 24-bit precision.
- Heidelberg Digits: The HD dataset has balanced per-class counts and variable recording durations, with recordings cut to minimal duration.
- Audio-to-spike conversion: The conversion pipeline models auditory processing through a hydrodynamic basilar membrane, hair cells, and bushy cells that produce phase-coded spikes.The basilar membrane separates frequencies, while bushy-cell integration increases phase locking.
- Audio-to-spike conversion: The biologically inspired conversion reduces user-specific preprocessing variability while retaining low computational cost.
- Classifier training: LIF networks were trained with surrogate gradients and BPTT, using feed-forward or recurrent hidden layers and non-spiking linear readouts.
3 Results
Across SHD and SSC, high classification accuracy required access to spike timing, while SNN performance depended on recurrent connectivity, loss design, and dataset. SNNs established useful baselines but generally remained below LSTM accuracy.
- Spike-count classifiers reached at most 60% on SHD and 30% on SSC, showing that spike counts alone were insufficient for high accuracy.
- LSTMs using temporal histograms achieved (85.7 ± 1.4)% on SHD and (75.0 ± 0.2)% on SSC, exceeding spike-count classifiers.
- CNNs had the least overfitting, with test accuracy drops of 1.4% on SHD and 1.5% on SSC.
- Max-over-time loss generally improved SNN and LSTM performance, whereas only LSTMs generalized well with last-time-step loss.
- 3.1 Training spiking neural networks: Accuracy was highest across a broad range of surrogate steepness values β ≥40, while convergence speed depended strongly on β and learning rate η.
- 3.1 Training spiking neural networks: On SHD, increasing feed-forward depth barely improved performance, whereas SSC accuracy increased monotonically with the number of layers.
- 3.1 Training spiking neural networks: RSNNs outperformed feed-forward SNNs, reaching (71.4 ± 1.9)% on SHD and (50.9 ± 1.1)% on SSC, but remained below LSTMs.
- 3.2 Generalization across speakers and datasets: CNNs generalized best across held-out SHD speakers, with an accuracy drop of 8%, followed by LSTMs at 10% and RSNNs at 21%.
3.3 Improving generalization performance through data augmentation and larger networks
The study improved SHD generalization by enlarging recurrent networks and applying input, temporal, and spatial perturbations. Combining these strategies produced the strongest reported test accuracy.
- 76.5 ± 1.0 % test accuracy was reached with 1024 neurons, as increasing network size improved SHD validation and test performance.
- 72.2 ± 1.6 % test accuracy followed compression of 700 input channels to 70, while reducing the network’s computational footprint.
- 79.9 ± 2.8 % test accuracy was achieved after scaling neuronal time constants by 4, which also reduced overfitting.
- 78.7 ± 2.2 % test accuracy was obtained with channel-jitter noise of σu = 20, which decreased overfitting.
- 83.2 ± 1.3 % test accuracy resulted when the strategies were combined, exceeding the previous RSNN result of 71.4 ± 1.9 %.
4 Discussion
The discussion presents the datasets and baselines as tools for more standardized SNN evaluation, while identifying preprocessing choices, training methods, and future comparisons as important boundaries. Results indicate that temporal information and loss-function selection materially affect classification performance.
- Two public-domain spike-based datasets and baseline spiking and non-spiking classifiers were introduced to facilitate quantitative SNN comparisons.
- The benchmarks use auditory tasks because audio supplies a temporal dimension and fewer input channels than video, making the resulting datasets more tractable.
- The authors selected SC and created HD because available public alternatives differed in licensing, recording quality, alignment precision, size, or preprocessing requirements.
- A neurophysiology-inspired audio-to-spike pipeline standardizes preprocessing and reduces performance variability caused by end-user conversion differences.
- Across SHD and SSC, temporal spike information improved classification with suitable architectures, and recurrent LSTMs and RSNNs performed best among tested models.
- SNN training used surrogate gradients with BPTT, while comparisons with alternative learning approaches and STDP were left for future work.
- Loss-function choice markedly affected performance: LSTMs performed best with last-time-step loss, whereas SNNs performed best with max-over-time loss.
A Inner ear model
The audio-to-spike conversion uses a staged model of the inner ear and ascending auditory pathway. Basilar-membrane, hair-cell, and bushy-cell components prepare auditory signals for spike generation.
- Audio is converted into spikes through a basilar membrane model, a hair-cell population, and a bushy-cell population.
A.1 Basilar membrane model
The basilar membrane model represents cochlear fluid–membrane interaction with spatially varying mechanical impedance, under simplified linear and long-wave assumptions. The resulting equations are used to model the stimulus-driven membrane response across frequency channels.
- A.1 Basilar membrane model: The model simplifies cochlear geometry while retaining fluid–membrane interaction as the source of spatial frequency dispersion.The fluid is assumed inviscid and incompressible, with small oscillations permitting a linear description.
- A.1 Basilar membrane model: The basilar membrane impedance depends on position and angular frequency through stiffness, resistance, and effective mass.Stiffness and resistance vary exponentially along the membrane, while damping is determined from these parameters and the effective mass.
- A.1 Basilar membrane model: Stiffness variation over several orders of magnitude allows the model to cover the full audible-frequency range.
- A.1 Basilar membrane model: The pressure field is derived subject to chamber boundary conditions while retaining only the principal excitation mode in the z-direction.The model sets n = 1 for the principal mode.
- A.1 Basilar membrane model: For long waves with kh ≪1, the governing equations are combined and solved approximately using Fourier-transformed pressure and analytical response functions.The formulation uses the Fourier transform of p(x, 0, 0), with functions g(x, ω) and G(x, ω) specified in the derivation.
- A.1 Basilar membrane model: The stimulus is transformed into the frequency domain and used to evaluate the membrane velocity over 700 equally spaced channels spanning (0, 3.5 cm].Input recordings were normalized to an RMS value of 0.3 cm s−1 before applying the model.
A.2 Hair cell model
The hair-cell model converts basilar-membrane motion into spikes through transmitter release, synaptic-cleft dynamics, and stochastic spike generation. Forty independent hair cells are simulated at each basilar-membrane position.
- A.2 Hair cell model: Basilar-membrane movement is transformed into spikes by a hair-cell model based on the key steps of Meddis [1986].
- A.2 Hair cell model: Hair-cell permeability depends on basilar-membrane velocity and controls release from a free transmitter pool into the synaptic cleft.
- A.2 Hair cell model: Transmitter in the cleft decreases through chemical destruction, diffusion, and re-uptake into the cell.
- A.2 Hair cell model: Reuptaken transmitter is partly returned to the free transmitter pool, while a manufacturing base replenishes that pool.
- A.2 Hair cell model: A cleft transmitter quantum generates a spike with probability P_spike = h·c(x, t) dt, and events within 1 ms of a previous event are rejected.At each basilar-membrane position, the model simulates N_HC = 40 independent hair cells.
A.3 Bushy cell model
The bushy-cell stage increases phase locking by integrating hair-cell spike trains with standard leaky integrate-and-fire neurons. One bushy cell processes the outputs of 40 hair cells for each basilar-membrane channel.
- A.3 Bushy cell model: Bushy cells increase phase locking by receiving hair-cell spike outputs.
- A.3 Bushy cell model: The bushy-cell layer contains standard LIF neurons without recurrent connections.The implementation uses a single layer with l = 1.
- A.3 Bushy cell model: One bushy cell integrates the spike trains of N_HC = 40 hair cells for each basilar-membrane channel.The connection weight is V_ij^(l) = 0.54/N_HC for all i and j.
B Non-spiking classifiers
The study validates its spiking datasets with conventional time-series classifiers, including support vector machines, LSTMs, and CNNs. For SVMs, inputs are channel-wise spike counts standardized before classification.
- B Non-spiking classifiers: Three conventional time-series classifiers—SVMs, LSTMs, and CNNs—were applied for validation.
- B Non-spiking classifiers: Both linear and nonlinear SVMs were trained using polynomial kernels up to third degree and RBF kernels.
- B Non-spiking classifiers: Each sample was represented by an N_ch-dimensional vector of spike counts from the bushy cells.
- B Non-spiking classifiers: SVM features were standardized by removing the mean and scaling to unit variance.
B.2 Long short term memories
LSTMs were used to validate performance on temporal spike-train data, with inputs binned into 10 ms intervals.
- LSTMs were used for validation on temporal spike-train data.
- The LSTM inputs were Nch spike trains emitted by the BCs and binned into 10 ms time bins.
- The model used a single LSTM layer with 128 cells.
B.3 Convolutional neural networks
CNNs were used to test dataset separability by binning spike trains in both time and space, then processing them with a convolutional architecture.
- CNNs were applied to further test the separability of the datasets.
- Spike trains were binned temporally at 10 ms and spatially into 64 distinct input units.
- The network used three convolutional blocks with 32 filters of size 11 × 11 and ReLU activation in the first block.
- Each block was finalized with 2 × 2 max pooling and dropout at rate 0.2.
- The model ended with a 128-node dense ReLU layer and softmax readout, trained using Adamax and categorical cross entropy.