PaperlayerView exact revision on arXiv

Source-linked AI summary

FPGA-Based Low-Power Speech Recognition with Recurrent Neural Networks

Minjae Lee, Kyuyeon Hwang, Jinhwan Park, Sungwook Choi, Sungho Shin, Wonyong Sung

arXiv:1610.00552v1cs.CLcs.LGcs.SD

TL;DR

Large HMM-based speech recognition can require large networks and substantial DRAM access, while neural approaches also demand extensive computation. This paper combines acoustic and language RNNs with statistical word-level rescoring on an FPGA; the small model achieves 9.24 W at ×4.12 real-time speed, while its best floating-point WER is 8.79% versus 7.34% for the cited state-of-the-art result.

  • Problem

    HMM-based recognition can require networks of hundreds of megabytes and power-consuming off-chip DRAM access, complicating small-footprint, low-power implementation.

    Source: §I · p. 1, A. Large Vocabulary Continuous Speech… · p. 2

  • Method

    The FPGA system combines acoustic and character-level RNNs with statistical word-level language modeling in N-best search, using six-bit weights stored on-chip.

    Source: §I · p. 1

  • Results

    9.24 W and ×4.12 real-time speed were measured for the small model; its best floating-point WER was 8.79%, versus 7.34% for the cited state-of-the-art result.

    Source: B. FPGA Implementation Performance · p. 5, A. Recognition Performance · p. 5

  • Takeaways & Limitations

    The character-level RNN can dictate OOV words, and the unidirectional design supports low-latency decoding, with slight weakness in recognition accuracy.

    Source: A. Algorithm Overview · p. 2

    The unidirectional acoustic model has a slight recognition-performance disadvantage compared with a bidirectional model.

    Source: §I · p. 1

Abstract

from arXiv · show

In this paper, a neural network based real-time speech recognition (SR) system is developed using an FPGA for very low-power operation. The implemented system employs two recurrent neural networks (RNNs); one is a speech-to-character RNN for acoustic modeling (AM) and the other is for character-level language modeling (LM). The system also employs a statistical word-level LM to improve the recognition accuracy. The results of the AM, the character-level LM, and the word-level LM are combined using a fairly simple N-best search algorithm instead of the hidden Markov model (HMM) based network. The RNNs are implemented using massively parallel processing elements (PEs) for low latency and high throughput. The weights are quantized to 6 bits to store all of them in the on-chip memory of an FPGA. The proposed algorithm is implemented on a Xilinx XC7Z045, and the system can operate much faster than real-time.

I. INTRODUCTION

Conventional speech recognition often relies on large HMM-based networks and substantial memory access, while RNN-based systems still demand many arithmetic operations. This work develops an FPGA-based real-time system combining acoustic and language RNNs with a statistical word-level model.

I. INTRODUCTION

  • The work addresses the memory burden of HMM-based recognition, whose large networks typically require GPU or multicore systems with DRAM and are described as poorly power efficient.Recent fully neural RNN systems avoid conventional HMMs but still demand many arithmetic operations and are mostly implemented on GPUs.Source: §I · p. 1
  • The system combines acoustic and character-level language RNNs with a statistical word-level LM using tree-structured N-best beam search, rather than an HMM network.At beam width 128, the search data structure requires about 197 KB, compared with a few hundred megabytes for a conventional HMM-based network.Source: §I · p. 1
  • The unidirectional acoustic RNN trades a slight recognition-performance disadvantage against bidirectional models for online real-time operation.It avoids accessing future context when immediate reaction to an utterance is desired.Source: §I · p. 1
  • Six-bit weight quantization stores both RNNs’ weights and character-LM contexts in FPGA on-chip memory, avoiding DRAM accesses for the RNNs.The system uses DRAM only for tri-gram language modeling and employs highly parallel arithmetic arrays.Source: §I · p. 1

II. RELATED WORKS

Large vocabulary recognition commonly uses large WFST networks, making small-footprint, low-power hardware difficult. Prior work also explores end-to-end RNN recognition and quantized neural-network implementations to address decoding and memory demands.

A. Large Vocabulary Continuous Speech Recognition

  • WFST decoding networks commonly exceed a few hundred megabytes, while Viterbi search and off-chip DRAM access make low-power hardware implementation difficult.The network combines HMM acoustic, pronunciation lexicon, and word-level language models; DRAM access consumes substantial power.Source: A. Large Vocabulary Continuous Speech… · pp. 1–2
  • End-to-end RNN acoustic models, character-level language models, and prefix or tree-based decoding have been explored to reduce WFST size and decoding complexity.A phoneme-level CTC RNN was reported to reduce WFST size to about half that of DNN-HMM hybrid models.Source: A. Large Vocabulary Continuous Speech… · p. 2

B. FPGA-Based Neural Network Implementation

  • Neural networks offer hardware-friendly parallelism but often require external DRAM because of their parameter counts and low temporal locality of weight accesses.Prior work explored retraining-based quantization, including two-bit DNN weights [12] and on-chip SRAM storage.Source: B. FPGA-Based Neural Network… · p. 2

III. SPEECH RECOGNITION WITHOUT HMM

The recognition algorithm combines an online CTC acoustic RNN, a character-level language RNN, and a statistical word-level language model in N-best search. Its unidirectional character model supports low-latency decoding and OOV dictation, with a slight recognition-accuracy disadvantage.

A. Algorithm Overview

  • The algorithm combines acoustic RNN, character-level RNN language-model, and statistical word-level tri-gram outputs to select a hypothesis through N-best search.The acoustic model generates character probabilities from input frames, while the two language models predict following characters and words, respectively.Source: A. Algorithm Overview · p. 2
  • The deep LSTM acoustic model is end-to-end trained with online CTC and uses a unidirectional structure that cannot access future context for real-time operation.Recent RNN-based systems use bidirectional structures to improve recognition performance [17],,.Source: A. Algorithm Overview · p. 2
  • The deep unidirectional character-level LM enables low-latency decoding and OOV dictation, but is slightly weaker at recognizing words in its dictionary.The character-level LM does not use lexicon information.Source: A. Algorithm Overview · p. 2
  • The statistical word-level tri-gram LM uses improved Kneser-Ney smoothing and is rescored on the fly at blank or end-of-sentence nodes during N-best search.A word insertion bonus is included when applying the word-level LM.Source: A. Algorithm Overview · p. 2

B. Beam Search Algorithm

The decoder represents N-best hypotheses in a prefix tree and seeks the label sequence with maximum acoustic-model posterior probability. CTC transitions are represented with label and blank states, and the growing tree is pruned in depth and width.

B. Beam Search Algorithm

  • N-best hypotheses are generated with the acoustic and character-level language RNNs, then rescored on the fly by the statistical word-level LM.The beam search uses a simple prefix-tree structure.Source: B. Beam Search Algorithm · p. 2
  • Given input features x1:t, beam search seeks the label sequence with maximum posterior probability from the acoustic RNN, excluding the CTC blank from output labels.L denotes the output labels other than the CTC blank.Source: B. Beam Search Algorithm · p. 2
  • Each tree node represents an output label, while low-level CTC state transitions decompose it into a label state followed by a blank state.The state set is L′ = L ∪ {CTC blank}.Source: B. Beam Search Algorithm · pp. 2–3
  • Because the search tree grows indefinitely, the decoder periodically prunes it in both depth and width.The pruning procedure is described in.Source: B. Beam Search Algorithm · p. 3

C. Retraining Based Fixed-Point Optimization

The FPGA system uses one LSTM tile and one output tile, operated intermittently, while quantization fits the RNN weights and signals into on-chip memory. Weights use 6 bits, signals 8 bits, and internal LSTM cells 16 bits.

C. Retraining Based Fixed-Point Optimization

  • Weights are grouped and directly quantized by layer, then the whole network is retrained in the quantized domain.The method reduces word length for storing LSTM parameters on the FPGA.Source: C. Retraining Based Fixed-Point… · p. 3
  • Weights and internal signals use 6 and 8 bits, respectively, while internal LSTM cells use 16 bits.The paper reports that internal LSTM cells demand higher precision.Source: C. Retraining Based Fixed-Point… · p. 3

A. Overview of the FPGA System

  • The XC7Z045 design implements one LSTM tile and one output tile that operate intermittently under control signals.The acoustic model runs once per input speech frame, normally 10 ms, while the character-level LM runs more frequently to generate N-best hypotheses.Source: A. Overview of the FPGA System · p. 3

B. Architecture and Algorithm

The architecture separates LSTM matrix-vector multiplication from the remaining gate and activation calculations, then stores layer outputs as contexts for decoding. The PE arrays and output tile support this processing flow.

B. Architecture and Algorithm

  • The peephole LSTM equations define input, forget, and output gates, cell and candidate-cell activations, bias, and model-parameter matrices.The equations also use prior-time-step data, element-wise multiplication, a logistic sigmoid, and tanh.Source: B. Architecture and Algorithm · p. 3
  • The PE array consists of 512 PEs and calculates matrix-vector multiplications; each PE multiplies input Din by weight W, adds the result to an accumulator-stored partial sum, with bias preloaded.The LSTM tile consists of two main processing modules: the PE array and the LSTM extra processing unit.Source: B. Architecture and Algorithm · p. 3
  • The PE multiplies input Din by weight W and adds the result to an accumulator with bias values preloaded.The eight matrix-vector multiplication results are stacked in PE output buffers PEi, PEf, PEo, and PEc.Source: B. Architecture and Algorithm · p. 3, Figure 5 · p. 4
  • The PE array stacks results from eight matrix-vector multiplications in four buffers, while the LSTM EPU handles remaining element-wise and activation operations.The EPU applies peephole connections; lookup tables implement its activation functions, and the EPU and PE array can operate independently.Source: B. Architecture and Algorithm · p. 3
  • The LSTM EPU output is stored in context memory, with one stored context per beam-search hypothesis; the output tile is a fully connected layer fed by that memory.The context vectors are used in subsequent operations and beam-search decoding.Source: B. Architecture and Algorithm · pp. 3–4

C. Throughput of the LSTM tile

The LSTM tile’s processing time depends on input-vector dimension because the outer product method supplies one input element per clock. The reported three-layer acoustic model requires 2,806 cycles, and the two-layer language model requires 1,596.

C. Throughput of the LSTM tile

  • 2,806 cycles are required for the three-layer RNN AM, compared with 1,596 cycles for the two-layer RNN LM.The totals include a small synchronization overhead.Source: C. Throughput of the LSTM tile · p. 4
  • Each RNN layer needs four matrix-vector multiplication cycles because eight matrix-vector multiplications are handled by two PE arrays.Each array has 256 PEs and uses the outer product method.Source: C. Throughput of the LSTM tile · p. 4
  • Processing time depends on input-vector dimension, with 123 inputs in the first AM layer and 256 inputs in subsequent layers.The first AM layer requires 246 cycles for xt and 512 cycles for ht−1 matrix-vector multiplications; subsequent layers require 1,024 cycles.Source: C. Throughput of the LSTM tile · p. 4

A. Recognition Performance

The section evaluates recognition performance across model size, beam width, language-model rescoring, and weight precision. The best floating-point system reports 8.79% WER, compared with 7.34% for the state-of-the-art result, while supporting delay-free real-time recognition.

A. Recognition Performance

  • The evaluation concatenates WSJ eval92 utterances into a single 42-minute speech stream.The RNN AM training set is roughly 71 hours of WSJ SI-284 data.Source: A. Recognition Performance · p. 4
  • WER and CER improve as beam width or network size increases, and word-level LM combination helps especially with small networks.The reported evaluation uses WSJ eval92; the small-model uses 3×256 AM and 2×256 LM RNNs, while the large-model uses 4×512 AM and 2×512 LM RNNs.Source: A. Recognition Performance · pp. 4–5
  • 8.79% WER for the best floating-point system, versus 7.34% for the state-of-the-art result, with the proposed system supporting delay-free real-time recognition.The comparison is reported against a system that is not for real-time recognition and uses a bidirectional model and WFST decoding.Source: Table I · p. 4, A. Recognition Performance · p. 5
  • 6-bit weights use about 1/5 the memory of floating-point implementations, with about a 1.5% WER increase; the parameters occupy about 1.1 MB.The reported parameter size fits in the XC7Z045 on-chip memory.Source: A. Recognition Performance · p. 5

B. FPGA Implementation Performance

The FPGA implementation runs faster than real time and is compared with a high-end GPU on the 42-minute WSJ eval92 stream. The FPGA’s measured power includes DRAM and peripherals.

B. FPGA Implementation Performance

  • For real-time operation, the estimated requirement is about 6.4 million clock cycles per second, including 100 AM operations and about 3,840 character-LM operations per second.The character-level LM runs on character transitions, usually no more than 30 per second, across 128 beams.Source: B. FPGA Implementation Performance · p. 5
  • 9.24 W measured power accompanies ×4.12 real-time speed for the FPGA system, including DRAM and peripherals.The implementation consumes some extra cycles for communication.Source: B. FPGA Implementation Performance · p. 5
  • The GPU takes 12.5 minutes to evaluate the 42-minute WSJ eval92 set, achieving ×3.36 real-time speed while using about 30% of GPU resource.The FPGA system reports better recognition speed on a single speech stream; GPU throughput can increase by processing multiple utterances.Source: B. FPGA Implementation Performance · pp. 5–6
  • The GPU-based system consumes about 80 W, much higher than the FPGA system.The comparison is against an NVIDIA GeForce Titan X implementation.Source: B. FPGA Implementation Performance · pp. 5–6

VI. CONCLUDING REMARKS

The implemented FPGA speech recognizer combines acoustic and character-level recurrent models with a statistical word-level language model, using a simple tree-based search without HMMs or WFSTs. With 6-bit weights and on-chip RNN memory, it reaches approximately 4.12× real-time speed at 9.24 W on the XC7Z045.

VI. CONCLUDING REMARKS

  • The system combines acoustic and character-level RNNs with a statistical word-level LM through simple tree-based search, without HMMs or WFSTs.The character-level RNN can dictate out-of-vocabulary words; the word-level LM is used to improve recognition performance.Source: §VI · p. 6
  • At 100 MHz, the XC7Z045 implementation achieves approximately 4.12× real-time speed while consuming 9.24 W.The RNNs use 6-bit weights and only on-chip memory.Source: §VI · p. 6
  • The paper reports about 10 times higher power efficiency than a high-end GPU-based system.Source: §VI · p. 6
Loading 1610.00552v1…