PaperlayerView exact revision on arXiv

Source-linked AI summary

Bio-Inspired Auditory Processing: Coupling Parallel Resonate-and-Fire Dynamics with Ternary Recurrent Networks for Speech Recognition

Pavlos Poulos, Katelyn Sadorf, Themos Stafylakis, Costas A. Anastassiou

arXiv:2610.29720v1cs.SDcs.AIcs.NEeess.ASq-bio.NC

TL;DR

Spiking networks are promising for energy-efficient inference but remain constrained on continuous, context-dependent speech recognition, while ternary recurrent models may still require separate activation quantization. stGRU combines ternary gated filtering with RF-generated binary event communication trained end-to-end; it reports competitive outcomes on LibriSpeech and Aishell-1 alongside a favorable modeled efficiency trade-off.

  • Problem

    SNNs remain constrained on continuous, context-dependent ASR, and ternary recurrent models with continuous activations require separate activation quantization for low-precision neuromorphic deployment.

    Source: §1 · pp. 1, 3

  • Method

    stGRU pairs ternary gated recurrent filtering with downstream RF neurons that generate a binary event stream during training, avoiding a separate deployment-time activation-quantization step.

    Source: §1 · p. 3

  • Results

    6.44 ± 0.17% LibriSpeech test-clean WER without an external language model and 10.88% Aishell-1 test CER were reported for the Large model and separately trained Aishell-1 model, respectively.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The findings suggest efficient event-driven communication can coexist with rich internal organization, including deep-layer error-to-confidence selectivity and structured sparsity.

    Source: §6 · pp. 9–10

    Energy estimates use analytical operation counts and 45 nm CMOS constants, and element-wise multiplications remain, requiring specialized arithmetic units or approximations on addition-only or integer-only edge silicon.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

Spiking Neural Networks (SNNs) promise energy-efficient inference through sparse, event-driven computation, yet remain constrained on continuous, context-dependent tasks such as automatic speech recognition (ASR). A key bottleneck is the Leaky Integrate-and-Fire (LIF) neuron, whose membrane decay erases information across pauses and phonetic boundaries while it must both extract features and retain long-range context. Inspired by cortical separation of nonlinear filtering and somatic integration, we introduce the Spiking Ternary GRU (stGRU): a ternary-weight gated recurrent unit performs temporal filtering, while downstream Resonate-and-Fire (RF) neurons generate the binary spike stream driving the next block's event pathway. This makes binary event transmission a native training constraint rather than a separate deployment-time activation quantization step; together with ternary weights, learned projections avoid dense floating-point matrix MACs. Trained end-to-end on LibriSpeech-960, Small, Medium, and Large stGRUs reach 8.09%, 7.04%, and $6.44 \pm 0.17\%$ WER on test-clean without an external language model. To our knowledge, these are the first matrix-MAC-free spiking networks without a learned ANN sequence decoder to achieve single-digit WER on LibriSpeech. The architecture also achieves 10.88% test CER on Aishell-1 (9.28% on dev) when trained from scratch. Internally, trained networks develop a depth-graded hierarchy: shallow layers act as fast acoustic filters, while deep layers form slower semantic integrators whose latent representations segregate into part-of-speech manifolds and word-level spike events. Deep-layer firing rates follow lognormal distributions with a sparsity gradient, mirroring hierarchical processing observed in human cortex in vivo. Our biologically grounded architecture combines competitive ASR with favorable modeled compute and memory cost.

1 INTRODUCTION

stGRU addresses spiking ASR’s context and compute constraints by separating ternary temporal filtering from a train-time binary event pathway. It reaches single-digit LibriSpeech test-clean WER without an external language model while exhibiting depth-dependent neural representations.

1 INTRODUCTION

  • SNNs face context loss across pauses and phonetic boundaries, while prior spiking ASR often restores dense ANN decoders and ternary recurrent models retain continuous activations.These limitations motivate learning discrete event communication while retaining gated temporal filtering.Source: §1 · pp. 1–3
  • stGRU uses ternary MLGRU filtering followed by RF neurons that emit binary spikes to drive the next block’s event pathway, with the spike constraint present during optimization.This separates temporal filtering from discrete event communication and mirrors cortical separation of gated processing and somatic integration.Source: §1 · p. 3
  • 8.09%, 7.04%, and 6.44 ± 0.17% test-clean WER were achieved by Small, Medium, and Large stGRUs, respectively, without an external LM; Aishell-1 test CER was 10.88%.The paper identifies these as the first matrix-MAC-free spiking networks without a learned ANN sequence decoder to reach single-digit LibriSpeech WER.Source: §1 · p. 3
  • 1.95 mJ/s for Small and 3.42 mJ/s for Large are below the reported Transformer (8.80 mJ/s) and Conformer (11.28 mJ/s) rates; Large ternary INT2 models use 122.22 MB on LibriSpeech.For the Large Conv1D–MLGRU–ReLU baseline, ternarization changes test-clean WER from 5.83% to 6.49% while modeled energy changes from 33.48 to 4.10 mJ/s.Source: §1 · p. 4
  • Layer L11 shows an error-monitoring signature with peak sparsity of 0.83, while L14 shows peak firing for high-confidence correct words and a shift to Gaussian rate distributions.Figure 1 depicts internal spiking activity and progressively separated part-of-speech clusters across layers.Source: Figure 1 · p. 2, §1 · p. 4

2 RELATED WORK

Prior ASR systems often combine spiking encoders with dense ANN decoders, while ternary recurrent models can retain continuous activations. stGRU combines ternary filtering with a train-time binary event interface and direct CTC decoding.

2 RELATED WORK

  • Dense continuous ASR models incur compute and memory costs from dense MACs, while MatMul-free MLGRU retains continuous activations and requires separate activation quantization for Loihi 2.The cited Loihi 2 mapping retains 16-bit precision after 8-bit quantization substantially degrades accuracy.Source: §2 · p. 4
  • Prior spiking speech systems include hybrid decoders that reintroduce floating-point computation; QT-SNN targets keyword spotting, and DyTr-SNN lacks reported WER and uses a dense Transformer decoder.Leading SNN ASR baselines also rely on continuous ANN decoders.Source: §2 · p. 4
  • stGRU combines ternary MLGRU filtering with an end-to-end trained RF-generated binary event interface, deferred scaling, and direct CTC decoding without a learned ANN sequence decoder.This makes event discretization part of the model rather than a posttraining hardware conversion.Source: §2 · p. 4

3 METHODOLOGY

The method pairs ternary-weight temporal filtering with downstream spiking to make event transmission part of training. Deferred scaling and parallel recurrent and membrane computations support matrix-MAC-free projections and faster training.

3.1 TERNARY PROJECTIONS WITH DEFERRED WEIGHT SCALING

  • Deferred scaling applies the weight scale after ternary accumulation, avoiding floating-point learned weights while retaining an output scaling operation.In the temporal depthwise projection, additions and subtractions form the sum before γdw is applied.Source: §3.1 · p. 5

3.2 PARALLEL SPIKING-TERNARY BLOCK ARCHITECTURE

  • The stGRU block uses an associative MLGRU scan and FFT-based PRF integration to enable global parallel unrolling in O(L log L) time.The continuous residual stream supplies input current to the PRF neuron, whose membrane integration is independent of output spikes.Source: §3.2 · pp. 5–6
  • Each block extracts temporal features from incoming discrete spikes with a ternary depthwise temporal convolution followed by a pointwise channel projection.The MLGRU uses ternary gate-specific projections and a ternary output projection; its element-wise recurrence supports an associative parallel scan.Source: §3.2 · p. 5
  • The architecture also uses a continuous residual stream to stabilize variance across depth and provide input current to the PRF neuron.The residual update is introduced to prevent vanishing gradients and stabilize variance across depth.Source: §3.2 · p. 6
  • Training is 3.1×–3.4× faster than parameter-matched LIF variants, with surrogate gradients training the binary event interface end-to-end.The next block is optimized on the same discrete spike stream used at inference.Source: §3.2 · p. 6

4 EXPERIMENTS

Experiments evaluate standalone stGRU acoustic models on LibriSpeech-960 and Aishell-1, alongside efficiency benchmarks and parameter-matched ablations. The models reach single-digit LibriSpeech test-clean WER without an external language model, while the reported arithmetic-energy figures are modeled estimates.

4 EXPERIMENTS

  • The evaluation uses 80-dimensional log-Mel inputs at 50 Hz, 4× temporal subsampling, and end-to-end CTC training; an optional 4-gram LM is excluded from neural energy profiling.The acoustic model uses a ternary CNN frontend and an MLGRU layer coupled to a PRF pre-encoder.Source: §4 · p. 6
  • 10.88% test CER on Aishell-1 and 9.28% on dev are reported for a separately trained stGRU.The Aishell-1 model is trained from scratch.Source: §4 · p. 6
  • At Large scale, discrete PRF spikes achieve 6.14% WER at 3.42 mJ/s versus 6.49% at 4.10 mJ/s for continuous ternary ReLUs.The parameter-matched ablations also report 7.4×–10.8× lower memory from FP32-to-ternary weight quantization, with WER changing from 5.83% to 6.49%.Source: §4 · p. 6
  • 6.44±0.17% WER on LibriSpeech test-clean is achieved by the Large stGRU without an external LM, among Small (8.09%) and Medium (7.04%) models.The authors report these as the first single-digit WERs for matrix-MAC-free SNNs without a learned ANN sequence decoder; Table 1 compares architectures on LibriSpeech-960.Source: §4 · p. 6, Table 1 · p. 7
  • The reported energy figures use a 45 nm arithmetic-cost model, not measured hardware energy.The estimate allocates 2.73 mJ/s to IML-Spikeformer’s FP32 frontend and 1.68 mJ/s to its ANN decoder, while the stGRU estimate includes 0.05 mJ/s for active-weight accumulation in its FP32 CTC readout.Source: §4 · p. 7

5 NEUROMORPHIC INTERPRETATION

Deep layers develop distinct outcome-related codes: L11 has an ACC-like error-monitoring signature, while L14 has a PFC-like confidence and commitment signature. Their transition is gradual and is associated with MLGRU gating.

5.1 ERROR-MONITORING SIGNATURE IN L11 MIMICS THE HUMAN MEDIAL FRONTAL CORTEX

  • At L11, incorrect words produce significantly higher population firing than either high- or low-confidence correct words; incorrect-preferring neurons number 194, versus 64 high-confidence-correct neurons.Incorrect-preferring neurons exhibit a sharp post-onset transient that decays before the CTC emission window. Per-neuron firing rates at L11 follow a lognormal distribution.Source: §5.1 · pp. 7–8

5.2 DECISION-CONFIDENCE SIGNATURE IN L14 MIMICS THE PREFRONTAL CORTEX

  • At L14, high-confidence-correct words elicit the highest firing, and high-confidence-correct neurons dominate incorrect-preferring neurons (254 vs. 58).Sparsity decreases from L11, and firing-rate distributions shift from lognormal to near-Gaussian (∆R2 = −0.011), consistent with broad recruitment.Source: §5.2 · pp. 8–9, Figure 3 · p. 8

5.3 AUTONOMOUS ACC-TO-PFC HANDOFF WITHIN A SINGLE NETWORK

  • The L11-to-L14 handoff proceeds through L12–L13, where selectivity, sparsity, firing directionality, and firing-rate statistics change gradually.L12 responses become similar across outcome categories, while lognormality declines from L11 (+0.091) through L12 (+0.064) and L13 (+0.041) before reaching −0.011 at L14.Source: Figure 3 · p. 8, §5.3 · p. 9
  • Without MLGRU gating, deep-layer outcome selectivity collapses by 95.5% after controlling for empty-emission trials, versus 8.1% in the full model.This ablation supports a role for gating in the observed outcome-selective activity.Source: §5.3 · p. 9

6 CONCLUSION

The conclusion presents stGRU as a matrix-MAC-avoiding ASR architecture with competitive reported results and favorable modeled efficiency. Its variants scale capacity through deeper stages and progressively larger temporal receptive fields.

6 CONCLUSION

  • The Large stGRU reaches 6.44 ± 0.17% LibriSpeech test-clean WER without an external language model, while a separately trained Aishell-1 model reaches 10.88% test CER.The efficiency trade-off is based on modeled arithmetic-energy and parameter-storage results, subject to the paper’s accounting limits.Source: §6 · p. 9
  • At inference, learned projections avoid dense matrix MACs, using deferred-scaling ternary projections, gated recurrent filtering, and resonant spiking integration.Ternary projections constrain weights while RF neurons make the inter-block event stream binary during training rather than relying on separate deployment-time activation quantization.Source: §6 · p. 9

A ARCHITECTURAL CONFIGURATIONS AND STREAMING SETUP

  • The Small, Medium, and Large variants scale capacity by increasing depth within linguistic stages, with the 15-layer Large model adding a fifth stage with receptive field K = 63.Across variants, the receptive field expands progressively across stages.Source: A ARCHITECTURAL CONFIGURATIONS AND… · p. 13, Table S1 · p. 13

A.1 STREAMING IMPLEMENTATION VIA ASYMMETRIC PADDING

The streaming implementation bounds future acoustic context to 320 ms by using asymmetric padding. The bound follows from the 12.5 Hz internal frame rate and restricting right-sided padding to selected layers.

A.1 STREAMING IMPLEMENTATION VIA ASYMMETRIC PADDING

  • 320 ms of future context is the maximum look-ahead, implemented as four future frames at 80 ms each.The 80 ms step follows 4× temporal subsampling of 50 Hz acoustic features to 12.5 Hz.Source: §A.1 · p. 14
  • Asymmetric padding eliminates right-sided padding across most layers and restricts it to selected intermediate layers.Standard symmetric padding would continually draw from future acoustic frames and violate streaming constraints.Source: §A.1 · p. 14
  • The padding strategy bounds look-ahead without altering the core matrix-MAC-free MLGRU dynamics.Source: §A.1 · p. 14

B REPRODUCIBILITY, TRAINING INFRASTRUCTURE, AND HYPERPARAMETERS

Training and evaluations ran on a single LUMI-G node using eight AMD Instinct MI250X graphics compute dies. Table S2 covers hyperparameters for PRF dynamics, optimization, tokenization, and CTC decoding.

B REPRODUCIBILITY, TRAINING INFRASTRUCTURE, AND HYPERPARAMETERS

  • Training and empirical evaluations used one LUMI-G node equipped with four AMD Instinct MI250X modules, exposing eight GCDs as independent runtime devices.Source: B REPRODUCIBILITY, TRAINING… · p. 14
  • Distributed Data Parallel training ran across the eight GCDs using torchrun with the c10d backend.Source: B REPRODUCIBILITY, TRAINING… · p. 14
  • Table S2 lists key hyperparameters for PRF neuronal dynamics, optimization, tokenization, and CTC decoding.Source: Table S2 · p. 14

C ALGORITHMIC DERIVATIONS AND PSEUDOCODE

The appendix specifies how stGRU trains and stabilizes ternary weights, schedules learning rates, and estimates modeled arithmetic energy. It also details how ternary projections defer scaling and how the energy estimate treats the spiking readout.

C.1 QUANTIZATION-AWARE TRAINING (QAT) WARMUP STRATEGY

  • QAT warmup gradually shifts effective weights from continuous values toward ternary representations, with gradients updating latent weights through the Straight-Through Estimator.The schedule parameter αqat progresses from 0 to 1.Source: §C.1 · pp. 14–15

C.2 DEEP RESIDUAL STABILIZATION VIA CHANNEL-WISE SCALING

  • Channel-wise residual scaling modulates each block’s transformation, with α initialized to zero or 10^-4 to preserve identity gradient flow early in training.The 1D vector requires C sparse floating-point multiplications per timestep and bypasses dense matrix operations.Source: §C.2 · p. 15

C.3 SEMI-RESONANT COSINE ANNEALING (SRCA)

  • SRCA holds the learning rate initially, then runs 2.75 cosine cycles with exponentially decaying peaks and a valley-phase starting offset.The holding period is 150; the peak decay factor and initial phase offset are both 0.75.Source: §C.3 · p. 15, Figure S1 · p. 15

C.4 DEFERRED SCALING IN TEMPORAL DEPTHWISE PROJECTIONS

  • Deferred scaling accumulates temporal depthwise projections through ternary additions and subtractions, then applies the scale after the sum.This avoids floating-point multiplication by learned kernel weights inside the temporal sum, but retains post-accumulation scaling.Source: §C.4 · p. 16

D.1 DYNAMIC ENERGY FORMULATION AND READOUT DERIVATION

  • The derivation confirms the reported 0.96 J figure in Tables S3 and S4 (S8); the baseline calculation, 2.488 × 10^12 at 4.6 pJ each, is approximately 11.45 J (0.59 mJ/s) (S7).Source: §D.1 · p. 17

D.2 THEORETICAL ENERGY ESTIMATION METHODOLOGY FOR BASELINES

The appendix estimates dynamic arithmetic energy by profiling active operations and applying stated operation costs. It describes how continuous, hybrid-spiking, and stGRU baselines are decomposed for comparison.

D.2 THEORETICAL ENERGY ESTIMATION METHODOLOGY FOR BASELINES

  • The baseline decomposition accounts for Transformer attention and feed-forward operations, Conformer convolution and dual feed-forward modules, Transformer++ frame stacking, and IML-Spikeformer’s spiking encoder and dense decoder.The IML-Spikeformer description notes six simulation timesteps per frame, whereas stGRU operates at T = 1.Source: §D.2 · p. 18
  • Energy profiling scales theoretical synaptic operations by input firing rate and nonzero-weight density, then applies accumulation costs; PRF neuron updates receive a separate per-output cost.The procedure records forward-pass activations and sums layer costs across the model and dataset.Source: §D.2 · p. 18
  • Continuous baselines are modeled with an FP32 CTC readout, while IML-Spikeformer uses a six-layer ANN decoder and stGRU uses active-weight accumulation in an FP32 CTC readout driven by binary spikes.The estimates omit normalization, activation functions, residual scaling, memory traffic, and decoding/search.Source: §D.2 · p. 17, Table S3 · p. 18

E NEUROMORPHIC INTERPRETATION AND ANALYSIS

The appendix analyzes firing-rate distributions and outcome-selective neurons across network depth and model variants. It also identifies the scope of architectural, seed, energy, and throughput comparisons presented in supplementary tables.

E NEUROMORPHIC INTERPRETATION AND ANALYSIS

  • For firing-rate fits, neurons below 0.1 Hz were excluded as recording-duration artifacts; 1,022–1,024 neurons per layer were analyzed using squared Pearson correlation between empirical and fitted CDFs.The analysis covered the LibriSpeech evaluation set.Source: E NEUROMORPHIC INTERPRETATION AND… · p. 19
  • Lognormal firing-rate fits peak in superiority at L11 (+0.091), persist at L12–L13 (+0.064, +0.041), and slightly lose to Gaussian at L14 (−0.011).Figure S2 compares lognormal and Gaussian fits to per-layer firing-rate histograms.Source: E NEUROMORPHIC INTERPRETATION AND… · pp. 19–20, Figure S2 · p. 20
  • Supplementary comparisons cover Aishell-1 CER and theoretical memory, six-seed Large-model LibriSpeech WER, component ablations, scale placement, and PRF-versus-LIF training throughput.Tables S5–S9 identify these comparison scopes; Table S8 describes energy rates as modeled and excluding an external language model.Source: Table S5 · p. 20, Table S6 · p. 20, Table S7 · p. 21
  • In Large variants, restricting selective-neuron counts to attempted emissions reduces the aggregate by 8.1% with MLGRU, but by 95.5% without it.Empty-emission rates on incorrect words are 13.82% with MLGRU and 57.76% without it; the passage interprets this as supporting a role for MLGRU gating in the emergent error-monitoring property.Source: Figure S3 · p. 22

Related papers

Loading 2610.29720v1…