Conjoint Audio-to-Spikes Encoding and Processing for Efficient Neuromorphic Speech Recognition

Valentin M. Meunier; Amélie Gruel; Pierre Lewden; Adrien F. Vincent; Sylvain Saïghi

Source-linked AI summary

Conjoint Audio-to-Spikes Encoding and Processing for Efficient Neuromorphic Speech Recognition

Valentin M. Meunier, Amélie Gruel, Pierre Lewden, Adrien F. Vincent, Sylvain Saïghi

arXiv:2608.30792v1Aug 31, 2026cs.NEcs.AIcs.LGeess.AS

TL;DR

Neuromorphic speech pipelines need efficient audio-to-spikes encoding that remains informative for downstream SNN classification and practical for digital hardware. This paper evaluates a programmable encoder with hardware-neutral efficiency metrics, jointly optimizes encoder and classifier, and reports 99.77% accuracy on spike-encoded Heidelberg Digits.

Problem
Highly bio-mimetic audio encoders can be computationally demanding, energy-intensive, difficult to tune, and challenging to implement on digital hardware.
Method
The paper combines a non-learnable, high-level programmable audio-to-spikes encoder with a feedforward SNN classifier and hardware-agnostic metrics based on spike activity.
Results
99.77% accuracy was achieved on the spike-encoded Heidelberg Digits dataset, outperforming the neuromorphic state of the art on this benchmark.
Takeaways & Limitations
Jointly optimizing encoder parametrization and classifier architecture can combine high accuracy with low spiking activity in the evaluated neuromorphic pipeline.

Abstract

from arXiv · show

Obtaining data from neuromorphic sensors and processing it with Spiking Neural Networks is a promising solution to lower the energy cost of artificial intelligence. The current rarity of natively neuromorphic datasets promotes the development of software tools to translate input sensory data into spikes. However, highly bio-mimetic simulators can be challenging to implement on digital hardware. In this work, we evaluate the neuromorphic encoding and subsequent classification of audio into spikes using a non-learnable, high-level, programmable encoder targeting hardware implementation on FPGA. We quantify the pipeline's efficiency with hardware-agnostic metrics based on the quantitative spiking activity. Our study focuses on the simultaneous optimisation of encoder and classifier: the first provides efficient and informative data so that the latter achieves a better performance with an overall lower energy cost at learning and inference. This work introduces the first end-to-end neuromorphic spike-encoding and evaluation of the TIMIT dataset. Our simple feedforward network reaches a classification accuracy of 99.77% on a spike-encoded Heidelberg Digits, overcoming the neuromorphic state of the art on this benchmark dataset.

I. INTRODUCTION

The paper investigates end-to-end neuromorphic audio classification with a programmable encoder that balances bio-inspiration, efficiency, and FPGA feasibility. It jointly optimizes encoding and SNN processing across benchmark datasets, including the first such evaluation of TIMIT.

Lauscher provides bio-mimetic audio encoding but is computationally demanding, energy-intensive, difficult to tune, and challenging to implement efficiently on embedded digital hardware.
The HLP strategy uses programmable filterbank encoding to balance bio-inspiration with practical implementation trade-offs in FPGA area, speed, and power.
The study evaluates how audio-to-spikes encoding affects downstream SNN accuracy and energy efficiency using hardware-agnostic metrics based on synaptic activity.
The work introduces the first end-to-end neuromorphic encoding and SNN processing of TIMIT, while the best simulated HD result reaches 99.77% accuracy.
The pipeline excludes learnable front-ends and non-spiking inputs, retaining a reusable encoder and benchmarking end-to-end processing of fixed spike trains.

B. HLP cochlear encoder

The HLP cochlear encoder converts audio into spikes through a simple, programmable filter-and-neuron structure designed for efficient hardware realization while retaining bio-inspiration.

Overall mechanism: Each HLP channel applies a band-pass filter, rectifies its output, and feeds the absolute value into a Leaky Integrate-and-Fire neuron.
The HLP design uses few parameters and avoids the higher-order, memory-intensive computations of more bio-mimetic cochlear models, improving embedded FPGA feasibility.
Parametrisation: The encoder uses logarithmically spaced Butterworth filters spanning the human hearing range, with dataset-dependent upper frequencies near the Nyquist limit.
Parametrisation: A filter’s quality factor controls bandwidth, while filter order affects computation time and memory footprint.
Parametrisation: Firing threshold and membrane decay rate are key parameters because they shape output spike density and define neuromorphic variants of each dataset.

C. SNN classifier

The classifier is a feedforward fully connected SNN trained on spikes produced by the encoder and evaluated across HD, GSC, and TIMIT benchmark datasets. TIMIT provides the paper’s first end-to-end neuromorphic treatment of that dataset.

SNN classifier: The SNN uses affine layers followed by LIF neurons, and predicts the class whose output unit emits the highest spike count.
SNN classifier: Training uses surrogate-gradient backpropagation through time with Adam and a mean-square-error spike-count loss, with repeated random-seed runs for reliability.
SNN classifier: The architecture varies hidden-layer depth and width while matching the classifier input size to the encoder’s number of filters.
Datasets: The experiments sweep HLP encoder parameters across HD, GSC, and TIMIT, using standardized dataset-specific audio preparation and spike variants.
Datasets: TIMIT is segmented into 0.25-second single-phoneme samples and reduced from 61 to 39 phonemes; no published baseline exists for the resulting end-to-end neuromorphic evaluation.

E. Evaluation metrics

The paper evaluates accuracy together with energy-related spike activity using hardware-neutral efficiency metrics. These metrics support comparison across datasets and devices while exposing the effects of encoder and classifier optimization.

The evaluation jointly measures classification accuracy and energy cost, which depends strongly on data transfer and is approximated through spike activity.
Training efficiency divides test accuracy at epoch e by cumulative synaptic operations performed by the classifier through that epoch.
Inference efficiency divides test accuracy at epoch e by classifier spikes emitted during inference at that epoch.
The study extends prior encoder-only tuning toward conjoint optimization of encoder parameters and downstream classifier design.

1) Channel parametrisation:

The study simplifies HLP encoder parameters by removing membrane decay, selecting fourth-order filters, and limiting the filter count to 13 or fewer without sacrificing supported accuracy.

Channel parametrisation: Removing encoder membrane decay yielded the best downstream accuracy and permits simpler IF-neuron hardware implementation.Threshold tuning controls spike density when decay is absent.
Channel parametrisation: Fourth-order filters are adopted because orders four and above showed no significant downstream-accuracy difference, whereas order two reduced accuracy.Filter implementation cost is proportional to filter complexity.
Optimisation of filter distribution: Nf = 700 provided no benefit in the studied setting, increasing energy and resource usage while producing lower accuracy than leaner configurations.The analysis therefore focuses on Nf ≤13.
Optimisation of filter distribution: For NSC, variants with Nf ≥3 surpassed SSC in accuracy and training efficiency, while Nf = 7 slightly outperformed Nf = 8 and Nf = 9.The passage notes that filter placement can matter as much as filter count.
Optimisation of filter distribution: Accuracy saturated at Nf ≈13 across tested Fmax settings and datasets, so Nf = 13 became the robust default.Smaller values are reserved for explicit accuracy-efficiency trade-offs.

B. Selection of the SNN architecture

The authors select a compact SNN architecture by comparing depth and width across NHD, NSC, and NTIMIT, finding a strong accuracy-efficiency trade-off in Nf-300-300-No.

Selection of the SNN architecture: The architecture search varies one to three hidden layers and three widths while keeping total hidden units fixed to compare compute-budget allocation.This isolates the effects of network depth and capacity.
Selection of the SNN architecture: The highest accuracies used two hidden layers for NHD and NSC, and three hidden layers for TIMIT.The corresponding hidden-layer widths were 300, 600, and 400 neurons, respectively.
Selection of the SNN architecture: 99.77% accuracy was reached on the clean NHD variant, compared with 79.96% for NSC and 63.98% for NTIMIT.The passage attributes the lower latter accuracies to task complexity, label count, and inter-sample variation.
Selection of the SNN architecture: Nf-300-300-No stays within 2% of the best configuration while matching or exceeding its efficiency across the studied datasets.It also showed stable learning without abrupt test-accuracy drops or overfitting signs.
Comparison with the state-of-the-art: Parameter count is used as a proxy for network complexity and theoretical energy use when comparing end-to-end neuromorphic pipelines.The comparison covers spiked-encoded HD and GSC.

D. Optimising encoding for efficiency

The efficiency study combines audio down-sampling with filter clipping, showing dataset-dependent gains that preserve accuracy on easier tasks but can remove critical information from harder ones.

Optimising encoding for efficiency: Down-sampling lowers the Nyquist frequency, requiring the encoder’s highest band Fmax to be lowered or clipped accordingly.The study evaluates both changes before spike encoding.
Optimising encoding for efficiency: The experiments use Nf-100-100-No classifiers to magnify the effects of down-sampling and lowering the filter-bank ceiling on NHD-English.NHD-English contains only English-spoken digits and has ten labels.
Optimising encoding for efficiency: Fmax is set to 16.667 kHz at 48 kHz sampling, 6.65 kHz at 16 kHz, and 3.333 kHz at 8 kHz.These settings keep the highest filter below the relevant Nyquist frequency.
Optimising encoding for efficiency: Logarithmic Nf = 13 placement produces markedly fewer spikes in the three lowest and two highest filters at the highest Fmax.This motivates investigating removal of low-activity filters.
Optimising encoding for efficiency: Efficiency improved by up to a factor 2 while accuracy stayed within one thousandth of baseline, but gains were inconsistent across datasets.Easier NHD tasks tolerated clipping and 8 kHz down-sampling; NSC variants lost critical learning data.

IV. DISCUSSIONS

The discussion explains dataset and metric differences, prioritizes inference efficiency, and links larger filter banks to slower learning while retaining the study’s benchmark scope.

Dataset comparisons: NHD-English scored 12.7 points higher than NSC-digit despite matched pipelines and similar content, implicating dataset-specific factors including speaker composition.The compared subsets both contain digit samples, but GSC-digit recordings involve 2519 speakers with unequal representation.
Efficiency metrics: The study defines training and inference efficiency as accuracy-to-spike-count trade-offs corresponding to cumulative training spikes and per-epoch inference spikes.These metrics are intended to assess accuracy alongside energy-related activity.
Efficiency metrics: Training efficiency can favor datasets or configurations that reach a low-accuracy peak early, motivating exclusive reporting of inference efficiency after Fig. 3.NSC with Nf = 2 reached 10.61% accuracy at epoch 8 of 200.
Learning dynamics: NHD with Nf = 700 reached its best epoch at 197 of 200 and showed no clear accuracy plateau, indicating a slower learning phase with more encoding channels.This observation reinforces the choice of a low filter count.

C. Comparison with state-of-the-art models

The HLP encoder produces dense, informative spike activity that supports a simple linear SNN, outperforming more complex models on homogeneous HD but losing robustness on heterogeneous datasets. Encoder choice therefore determines which classifier structures are effective.

The HLP encoder yields denser, more contiguous activity with fewer isolated spikes than Lauscher, improving efficiency despite lower bio-mimicry.Lauscher produces broader, sparser activity with many singleton events.
99.77% accuracy is reached on NHD, while the same classifier reaches 79.96% on NSC and less than 72% on SHD.The reported accuracies vary substantially across encoded datasets and task conditions.
A simple linear SNN can outperform more complex spiking models on the easier, well-controlled HD dataset.The comparison includes recurrent and other state-of-the-art architectures.
The pipeline underperforms more bio-mimetic or alternative front-end pipelines on heterogeneous GSC, where data variability challenges the specialised HLP encoder.Restricting GSC to speakers with stronger representation substantially closes the performance gap.
Encoder choice affects classifier effectiveness: complex SNNs can exploit informative patterns that simpler classifiers do not capture.The paper relates this contrast to the differing spike statistics produced by HLP and Lauscher.

V. PROOF OF CONCEPT IN HARDWARE

The complete audio-to-spikes and SNN pipeline was implemented on a Zynq-7000 FPGA for online-compatible processing of the English HD subset. Hardware constraints preserved high accuracy while limiting spontaneous activity and implementation footprint.

Implementation: The FPGA implementation processes the English HD subset with eight HLP-derived filters and ten output classes.The implemented architecture uses the Nf-100-100-No design with Nf = 8 and No = 10.
Implementation: Fixed-point SOS filters and low thresholds were used to prevent spontaneous encoder activity caused by recursive residual values.The filter coefficients use signed Q9.22 format.
Implementation: Zero biases prevent spontaneous SNN spikes during continuous operation and reduce the effective parameter count by 210.The choice targets online low-power operation without extra computation from unsolicited spikes.
Results: 152 ms average runtime supports processing a 48 kHz microphone stream within the sensor sampling interval.The measured runtime includes file loading, data transfer, and result retrieval.
Results: 99.63% accuracy was obtained on the complete English HD dataset using hardware-derived processing and final weights from the reference simulation.The reference simulation achieved 99.42% train accuracy and 99.21% test accuracy over 100 epochs.
Results: The constrained hardware architecture remains within the state of the art and is presented as a candidate for online smart-sensor deployment.Its constraints prevent strong intrinsic spontaneous activity.
Limitations: The implementation is generic rather than application-dedicated, and future work targets a smaller footprint and higher simulation accuracies.Planned changes include removing bias computation states and memories and evaluating quantisation and hardware parameter adjustments.

VI. CONCLUSION

The paper presents a simple, hardware-friendly neuromorphic pipeline that jointly optimises spike encoding and SNN processing for accuracy and efficiency. It performs best on controlled Heidelberg Digits data but is less tolerant of heterogeneous datasets, motivating further resilience and hardware evaluation.

The pipeline combines a non-learnable HLP cochlear encoder with a linear SNN classifier for FPGA-oriented audio classification.The encoder uses IF neurons, sparse events, and a small filter bank to reduce implementation complexity and memory traffic.
Down-sampling or filter clipping can improve efficiency by up to a factor 2 while keeping accuracy within one thousandth on less complex datasets.The study uses hardware-neutral metrics based on spike activity to assess this trade-off.
The pipeline outperforms state-of-the-art strategies on Heidelberg Digits despite limited performance on heterogeneous datasets.Its encoder trades robustness for specialisation toward well-controlled data.
Joint exploration of encoder parametrisation and classifier architecture is required to achieve high accuracy with low spike activity.The optimisation may use global or channel-wise encoder settings.
Future work will improve resilience while preserving simplicity and hardware implementability on high-variability datasets.The authors also plan to translate spike-count savings into real latency, energy, and memory measurements through FPGA evaluation.