PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Spiking Neural Networks for Large Vocabulary Automatic Speech Recognition

Jibin Wu, Emre Yilmaz, Malu Zhang, Haizhou Li, Kay Chen Tan

arXiv:1911.08373v1cs.NEcs.CL

TL;DR

ANN-based ASR delivers strong recognition performance but carries substantial computational demands, and cloud-based processing raises concerns about security and speed. This paper uses deep SNNs for acoustic modeling across large-vocabulary recognition tasks. The results report competitive accuracy to ANN systems, with potential for efficient local inference.

  • Problem

    ANN acoustic models require substantial computation, while cloud ASR raises data-security and processing-speed concerns; SNN use for large-vocabulary continuous ASR remained unexplored.

    Source: §1 · pp. 1–2

  • Method

    The paper trains deep SNN acoustic models to map frame-level speech features to senone posteriors and combines acoustic scores with language-model and pronunciation information during decoding.

    Source: §3.4 · p. 8, §5 · p. 12

  • Results

    SNN acoustic models achieved competitive accuracy to ANN counterparts across ASR scenarios, with lower WERs on FAME and comparable results on LibriSpeech.

    Source: §4.1 · p. 10, §4.2 · p. 11, §4.3 · p. 11

  • Takeaways & Limitations

    The initial results are a first step toward integrating ASR engines into power-restricted mobile and embedded devices.

    Source: §5.3 · p. 14

    TIMIT results are not directly comparable with the cited system because this system uses both acoustic and language models during decoding.

    Source: §4.1 · p. 11

Abstract

from arXiv · show

Artificial neural networks (ANN) have become the mainstream acoustic modeling technique for large vocabulary automatic speech recognition (ASR). A conventional ANN features a multi-layer architecture that requires massive amounts of computation. The brain-inspired spiking neural networks (SNN) closely mimic the biological neural networks and can operate on low-power neuromorphic hardware with spike-based computation. Motivated by their unprecedented energyefficiency and rapid information processing capability, we explore the use of SNNs for speech recognition. In this work, we use SNNs for acoustic modeling and evaluate their performance on several large vocabulary recognition scenarios. The experimental results demonstrate competitive ASR accuracies to their ANN counterparts, while require significantly reduced computational cost and inference time. Integrating the algorithmic power of deep SNNs with energy-efficient neuromorphic hardware, therefore, offer an attractive solution for ASR applications running locally on mobile and embedded devices.

1 INTRODUCTION

ANN-based ASR performs well but requires substantial computation, while cloud processing raises security and speed concerns for local use. This work explores deep SNN acoustic models for large-vocabulary ASR and reports competitive accuracy alongside prospects for rapid, energy-efficient inference.

1 INTRODUCTION

  • ANN acoustic modeling has advanced ASR, but its performance gains come with substantial computation, while cloud processing raises data-security and speed concerns.On-device ASR processes speech locally using mobile-device resources.Source: §1 · pp. 1–2
  • SNNs use asynchronous, event-driven processing whose energy consumption matches sensory activity, offering a computationally efficient alternative to synchronous ANN processing.Neuromorphic hardware supports spike-based processing with low-power, massively parallel computing units.Source: §1 · p. 2
  • Deep SNN acoustic models are applied to large-vocabulary ASR, a setting earlier SNN speech systems had not reached.The work uses tandem learning and reports competitive accuracy against baseline ANN models.Source: §1 · p. 2
  • The paper reports prospects for rapid inference and energy efficiency from its preliminary SNN-based acoustic model study.The paper positions deep SNNs with neuromorphic hardware as a potential approach for local ASR applications.Source: §1 · pp. 2–3

2 FUNDAMENTALS AND RELATED WORK

The section introduces event-driven SNN processing, conventional large-vocabulary ASR components, and prior SNN speech systems. It motivates deep SNN acoustic modeling for LVCSR because earlier SNN speech work was limited to phone or small-vocabulary tasks.

2.1 Spiking Neural Networks

  • SNNs process information asynchronously, and classification decisions can begin with the first output spike but typically improve as evidence accumulates.This differs from conventional synchronous ANNs, whose output waits for preceding layers to update.Source: §2.1 · p. 3

2.2 Large Vocabulary Automatic Speech Recognition

  • Conventional ASR combines acoustic, language, and pronunciation models to choose a word sequence from speech features.The acoustic model estimates speech-unit likelihoods, while the language model supplies word-sequence priors.Source: §2.2 · pp. 3–4

2.3 Speech Recognition with Spiking Neural Network

  • Prior SNN speech systems addressed phone recognition or small-vocabulary tasks, whereas this work applies deep SNNs to LVCSR and reports competitive accuracy against comparable ANN systems.Earlier approaches include spike-based feature extractors, SOM-SNNs, and fully SNN-based recognition frameworks.Source: §2.3 · p. 5

3 METHODS

The acoustic model processes frame-based speech features with integrate-and-fire neurons and a reset-by-subtraction scheme. Neurons accumulate synaptic input and emit spikes when their membrane potential crosses a threshold.

3 METHODS

  • The model uses integrate-and-fire neurons with reset by subtraction to process short-time-stationary frame-based speech features at minimal computational cost.The neuron’s membrane potential starts at zero for each new frame-based input.Source: §3.1 · pp. 5–6
  • Incoming spikes are converted to synaptic current using connection weights and a bias-derived constant current, which contributes to each neuron’s membrane potential.The aggregate membrane potential links SNN layers to coupled ANN layers for parameter optimization.Source: §3.1 · p. 6
  • A neuron emits an output spike when its membrane potential crosses the firing threshold, set to 1 in the experiments.The threshold setting assumes synaptic weights are normalized with respect to that threshold.Source: §3.1 · p. 6

3.2 Neural Coding Scheme

The coding scheme transforms continuous speech features into spike trains for tandem learning. It uses a ReLU encoding layer and decodes output activity from free aggregate membrane potentials.

3.2 Neural Coding Scheme

  • A weighted ReLU layer transforms continuous input features such as MFCC or FBANK into activations for spike encoding.The input features are represented as a vector before this transformation.Source: §3.2 · p. 6
  • The encoding layer distributes ReLU activation across the encoding window and represents the transformed input as spike trains and spike counts.This encoding supports rapid inference by encoding input within a short time window.Source: §3.2 · pp. 6–7
  • Free aggregate membrane potential at the output neurons is used for decoding to support smooth learning with high-precision output error gradients.The input and output layers have much smaller dimensionality than hidden layers, limiting their deployment computation on edge devices.Source: §3.2 · p. 7

3.3 Tandem Learning for Training Deep SNNs

The paper trains deep SNN acoustic models with tandem learning, using a coupled ANN to approximate SNN gradients; only the SNN is used at inference. The acoustic model maps contextualized speech features to senone posteriors, which are decoded with language-model and pronunciation-lexicon information.

3.3 Tandem Learning for Training Deep SNNs

  • The tandem approach treats IF-neuron membrane potential as the main information carrier and relates it to an artificial neuron's pseudo spike count, while neglecting temporal dynamics.The paper notes that spike timings play an insignificant role in its neural representation.Source: §3.3 · p. 7
  • The coupled ANN and SNN layers synchronize forward-propagated information, allowing ANN-derived gradients to approximate those of the SNN; only the SNN is used at inference.SNN layers determine exact spikes and pass spike counts to ANN layers and spike trains to subsequent SNN layers.Source: §3.3 · p. 7
  • Tandem learning had previously demonstrated competitive ImageNet classification accuracies, as reported by Wu et al. (2019).Source: §3.3 · p. 8

3.4 SNN-based Acoustic Modeling

  • Tandem learning trains the SNN to map contextualized speech features to senone posteriors using frame-level alignments from a conventional GMM-HMM system.The input frames are spliced to provide more temporal context before passing through multiple spiking-neuron layers.Source: §3.4 · p. 8
  • At inference, SNN acoustic scores are combined with language-model and pronunciation-lexicon information in a WFST search graph, and the output is selected from a hypothesis lattice.The search uses a weighted sum of acoustic and language-model scores.Source: §3.4 · p. 8

3.5 Training and Evaluation

The evaluation spans phone recognition, low-resource bilingual speech recognition, and large-vocabulary speech recognition. It describes the model setup, error metrics, and synaptic-operation measures used to compare ANN and SNN computation.

3.5.1 Datasets

  • The evaluation covers TIMIT phone recognition, FAME Frisian-Dutch low-resource ASR, and Librispeech large-vocabulary continuous ASR.Librispeech models use 100- or 360-hour training subsets; FAME includes Frisian, Dutch, and code-switching speech.Source: §3.5.1 · pp. 8–9

3.5.2 Implementation Details

  • Experiments use PyTorch-Kaldi, with tandem learning implemented in PyTorch and Kaldi handling alignments, feature extraction, graph creation, and decoding.The ANN and SNN models use five hidden layers of 2,048 units, and the evaluated features include MFCC, FBANK, and FMLLR variants.Source: §3.5.1 · p. 8, §3.5.2 · p. 9

3.5.3.1 ASR Performance

  • TIMIT performance is measured by phone error rate, while FAME and Librispeech performance use word error rate; both count insertion, deletion, and substitution errors against reference transcriptions.Source: §3.5.3.1 · pp. 9–10

3.5.3.2 Energy Efficiency: Counting Synaptic Operations

  • ANN SynOps count multiply-and-accumulate operations per classification, whereas SNN SynOps count accumulate operations linked to firing rate, fan-out, and simulation time.For ANN, the operation count is constant for a given network configuration; the SNN count also depends on generated spikes.Source: §3.5.3.2 · p. 10

4 RESULTS

Across TIMIT, FAME, and Librispeech, SNN acoustic models approach ANN recognition performance, outperforming them on FAME test WERs but trailing slightly in 100-hour Librispeech experiments. Synaptic-operation counts vary by input feature, and projected hardware savings depend on the chip architecture and materials.

4.1 Phone Recognition on TIMIT Corpus

  • On TIMIT, the 13-dimensional FBANK ANN reached 16.9% (18.5%) PER and its SNN counterpart 17.3% (18.7%) on development (test).State-of-the-art systems had approximately 1% lower PER, largely attributed to the Li-GRU's longer time context; the 26.4% Bellec comparison is not direct because decoding setups differ.Source: §4.1 · pp. 10–11

4.2 Low-resourced ASR on FAME Corpus

  • 36.9% vs 39.0% overall WER: SNNs beat ANNs on FAME test data using 40-dimensional FBANK, with relative gains of 4.7%, 5.2%, and 8.2% on fy, nl, and cs.The authors suggest noisy weight updates from tandem learning may explain the gains, but leave this mechanism for future investigation.Source: §4.2 · p. 11

4.3 LVCSR experiments on Librispeech Corpus

  • At 360 hours, best SNN WERs fell from 10.0% (10.3%) to 9.2% (9.4%) for development (test), with comparable ANN LVCSR results.At 100 hours, ANN systems had absolute WER advantages of 0.1% to 0.6% across features.Source: §4.3 · pp. 11–12

4.4 Energy Efficiency of SNN-based ASR Systems

  • For 40-dimensional MFCC, FBANK, and FMLLR features, SNNs required 1.72, 1.10, and 0.68 times ANN synaptic operations, respectively, measured per feature classification across five randomly chosen TIMIT utterances.The authors expect at least an order of magnitude in energy and chip-area savings on emerging neuromorphic chips; actual energy savings depend on chip architectures and materials.Source: §4.4 · p. 12

5 DISCUSSION

The paper applies SNNs to acoustic modeling for large-vocabulary ASR and reports recognition results near ANN performance on some benchmarks, with low-resource results favoring SNNs. It also discusses neuromorphic efficiency and future development directions.

5 DISCUSSION

  • The proposed framework uses SNNs to map frame-level features to acoustic units, which are combined with language-model information to identify likely word sequences.Source: §5 · p. 12

5.1 Superior Speech Recognition Performance with SNNs

  • SNN recognition is comparable to or slightly worse than matched ANNs on TIMIT and Librispeech, while SNNs outperform ANNs in the low-resource scenario.The authors suggest reduced spike-count representation power as a possible reason for the benchmark gap, and noisy tandem-learning training as a possible reason for the low-resource result.Source: §5.1 · pp. 12–13
  • Offline-trained SNNs on neuromorphic chips show at least an order of magnitude in energy and chip-area savings over equivalent ANNs on TIMIT.The neural encoding scheme places input features within a short encoding window for rapid processing.Source: §5.1 · p. 13

5.2 Development of SNN-based ASR Systems

  • SNN acoustic models can be developed in PyTorch and integrated into the PyTorch-Kaldi speech-recognition toolkit.Source: §5.2 · p. 13

5.3 Future Directions

  • The authors plan to explore recurrent spiking networks for large-vocabulary ASR to improve recognition performance.Source: §5.3 · p. 13
  • The authors propose applying ANN compression and quantization techniques to SNNs to further reduce memory footprint and computation cost.Source: §5.3 · p. 13
  • Prior silicon-cochlea ASR studies were limited in task scale, while a cited auditory encoding scheme reduced spiking events by about 50% with negligible perceptual interference.Source: §5.3 · pp. 13–14
  • The authors frame these initial results as a step toward ASR on power-restricted mobile and embedded devices, with further gains anticipated from neuromorphic front ends, SNN architectures, hardware, and memory.Source: §5.3 · p. 14

FUNDING

The work acknowledges support from a Singapore Government research and enterprise grant and partial support for JBW from Zhejiang Lab.

FUNDING

  • The research was supported by Programmatic Grant No. A1687b0033.Source: FUNDING · p. 14
  • The grant was from Singapore’s Research, Innovation and Enterprise 2020 plan in the Advanced Manufacturing and Engineering domain.Source: FUNDING · p. 14
  • JBW was also partially supported by Zhejiang Lab’s International Talent Fund for Young Professionals.Source: FUNDING · p. 14

ALGORITHM

The algorithm trains SNN acoustic models through tandem ANN and SNN processing: it encodes features, computes outputs and loss, then updates ANN parameters and copies them to the SNN.

ALGORITHM

  • The algorithm takes frame-based feature vectors, target labels, network parameters, and an encoding-window size as inputs, and returns updated parameters.Source: ALGORITHM · p. 19
  • Neural encoding converts input features to an initial representation, after which ANN and SNN layers process them and the SNN spike counts are aggregated.Source: ALGORITHM · p. 19
  • The final ANN layer decodes the aggregate representation into an output used to calculate the loss against the target label.Source: ALGORITHM · p. 19
  • Gradients update the ANN parameters, which are copied to corresponding SNN layers; inference performs state updates entirely on SNN layers.Source: ALGORITHM · p. 19

TABLES

The tables cover recognition performance on TIMIT, FAME, and Librispeech, alongside a comparison of ANN and SNN computational costs.

TABLES

  • Table 1 reports PER on the TIMIT development and test sets, with literature results above and this work’s ANN and SNN results below.Source: Table 1 · p. 20
  • Table 2 reports WER on FAME monolingual and mixed test segments, with word counts, reference ANN systems, and this work’s models in separate panels.Source: Table 2 · p. 20
  • Table 3 reports Librispeech development and test WER, separating repository results from this work’s ANN and SNN results at 100 and 360 training hours.Source: Table 3 · p. 21
  • Table 4 compares ANN and SNN computational costs using the ratio of their required total synaptic operations.Source: Table 4 · p. 21

FIGURE

The figures compare ANN and SNN computation, diagram a conventional ASR system and an integrate-and-fire neuron, and show the SNN tandem-training flow and layer spike counts.

FIGURE

  • Figure 1 compares the synchronous computational paradigm of ANNs with the asynchronous paradigm of SNNs.Source: Figure 1 · p. 22
  • Figure 2 diagrams a conventional ASR system in which acoustic and linguistic components jointly determine the most likely hypothesis.Source: Figure 2 · p. 22
  • Figure 3 depicts an integrate-and-fire neuron receiving asynchronous spike trains and generating output spikes when membrane potential crosses the firing threshold.Source: Figure 3 · p. 23
  • Figure 4 shows tandem SNN training, with SNN layers determining spike counts and trains in the forward pass and ANN layers approximating gradients through error back-propagation.Source: Figure 4 · p. 23
  • Figure 5 plots average spike counts per neuron across TIMIT SNN layers, color-codes input features, and shows sparse neuronal activity.Source: Figure 5 · p. 24
Loading 1911.08373v1…