PaperlayerView exact revision on arXiv

Source-linked AI summary

ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGA

Song Han, Junlong Kang, Huizi Mao, Yiming Hu, Xin Li, Yubin Li, Dongliang Xie, Hong Luo, Song Yao, Yu Wang, Huazhong Yang, William J. Dally

arXiv:1612.00694v2cs.CL

TL;DR

Large LSTM speech-recognition models are computation- and memory-intensive, making deployment power-hungry. The paper combines load-balance-aware compression, scheduling, and sparse-model FPGA hardware; ESE reaches 282 GOPS and substantially outperforms CPU and GPU implementations in speed and energy efficiency.

  • Problem

    LSTM’s high computation complexity and memory footprint make speech-recognition deployment power-intensive, with memory references consuming more than two orders of magnitude more energy than ALU operations.

  • Method

    The paper combines load-balance-aware pruning and quantization with hardware-friendly mapping, dependency-aware scheduling, and FPGA hardware operating directly on compressed sparse LSTM models.

  • Results

    ESE achieves 282 GOPS on a Xilinx XCKU060 FPGA, and is 43× faster than the CPU and 3× faster than the GPU implementation.

  • Takeaways & Limitations

    ESE demonstrates that cross-layer compression and acceleration can improve LSTM speech-recognition inference speed and energy efficiency.

Abstract

from arXiv · show

Long Short-Term Memory (LSTM) is widely used in speech recognition. In order to achieve higher prediction accuracy, machine learning scientists have built larger and larger models. Such large model is both computation intensive and memory intensive. Deploying such bulky model results in high power consumption and leads to high total cost of ownership (TCO) of a data center. In order to speedup the prediction and make it energy efficient, we first propose a load-balance-aware pruning method that can compress the LSTM model size by 20x (10x from pruning and 2x from quantization) with negligible loss of the prediction accuracy. The pruned model is friendly for parallel processing. Next, we propose scheduler that encodes and partitions the compressed model to each PE for parallelism, and schedule the complicated LSTM data flow. Finally, we design the hardware architecture, named Efficient Speech Recognition Engine (ESE) that works directly on the compressed model. Implemented on Xilinx XCKU060 FPGA running at 200MHz, ESE has a performance of 282 GOPS working directly on the compressed LSTM network, corresponding to 2.52 TOPS on the uncompressed one, and processes a full LSTM for speech recognition with a power dissipation of 41 Watts. Evaluated on the LSTM for speech recognition benchmark, ESE is 43x and 3x faster than Core i7 5930k CPU and Pascal Titan X GPU implementations. It achieves 40x and 11.5x higher energy efficiency compared with the CPU and GPU respectively.

Keywords

The paper concerns deep learning, speech recognition, model compression, hardware acceleration, software-hardware co-design, and FPGA implementation.

  • The paper addresses deep learning and speech recognition.
  • Its approach combines model compression with hardware acceleration.
  • The design emphasizes software-hardware co-design on FPGA platforms.

1. INTRODUCTION

The paper targets the computation- and memory-intensive LSTM component of speech recognition, proposing compression and cross-stack acceleration to improve efficiency. Its contributions span load-balanced pruning, scheduling, sparse-model hardware, and joint accuracy-latency optimization.

  • Motivation: LSTM is central to large-scale speech recognition but is difficult to deploy because of high computation and memory demands.Memory references consume more than two orders of magnitude more energy than ALU operations.
  • Model Compression: The proposed flow compresses LSTM models to 5% of their original size while retaining similar accuracy, then maps and accelerates them in hardware.The compressed representation uses 10% density and 2× narrower weights.
  • Deployment Flow: The deployment paradigm changes from training followed by inference to training, compression, and accelerated inference.The paper presents LSTM as a case study for this flow.
  • Contributions: Load-balance-aware pruning and dynamic-precision quantization form the paper’s LSTM compression algorithm.The pruning method is highlighted as hardware-aware, and the quantization flow is automatic.
  • Contributions: A scheduler overlaps memory references with computation to manage recurrent LSTM dependencies and expose parallelism.The scheduler is designed for the complex operations produced by recurrent dataflow.
  • Contributions: ESE operates directly on the sparse model through hardware that balances and partitions computation and storage.The architecture also supports concurrent processing of multiple users’ speech data.
  • Contributions: The work jointly analyzes prediction accuracy and latency across algorithm, software, and hardware boundaries.

2. BACKGROUND

Speech recognition converts speech signals into word sequences through front-end feature extraction and back-end acoustic, language, and decoding components. The LSTM acoustic model is the pipeline’s computational and memory bottleneck, with recurrent dependencies and gated state updates defining its data flow.

  • Speech Recognition Pipeline: Speech recognition combines front-end feature extraction with back-end acoustic modeling, language modeling, and decoding to produce word sequences.The decoder uses acoustic and language models with maximum a posteriori estimation.
  • Speech Recognition Pipeline: LSTM is used in the acoustic model to compute acoustic output probabilities for large-scale speech recognition.
  • LSTM Bottleneck: LSTM acceleration must address complicated data dependencies created by recurrence and memory references intertwined with computation.
  • LSTM Bottleneck: More than 90% of the whole speech recognition pipeline’s execution time is consumed by LSTM.
  • LSTM Architecture: LSTM’s recurrent hidden layer uses memory cells and multiplicative input, output, and forget gates to store and control temporal state.At time T, the input depends on the output at T −1.
  • LSTM Architecture: An LSTM maps an input sequence x = (x1; . . . ; xT ) to an output sequence y = (y1; . . . ; yT ) through iterative updates from t = 1 to T.The update equations include gate and cell computations involving current inputs and recurrent state.

3. MODEL COMPRESSION

The paper compresses LSTM models through pruning and quantization while preserving accuracy and improving hardware parallelism. Load-balance-aware pruning distributes nonzero weights evenly across processing elements, while quantization reduces numerical representation size.

  • 3.1 Pruning: Pruning trains, removes low-saliency weights, and retrains the model under a sparsity constraint.Weights are pruned when their absolute values fall below an empirical threshold.
  • 3.1 Pruning: 90% parameter pruning avoids hurting word error rate on a larger proprietary speech dataset.On TIMIT, PER increased dramatically only after more than 93% of parameters were pruned.
  • 3.2 Load Balance-Aware Pruning: Load-balance-aware pruning assigns equal sparsity quotas to submatrices, evenly distributing nonzero weights across processing elements.This prevents one submatrix from requiring substantially more computation than another.
  • 3.2 Load Balance-Aware Pruning: 3 versus 5 cycles are required when four processing elements receive evenly balanced rather than unevenly distributed nonzero weights.The two pruning methods retain nearly identical prediction accuracy, with differences within the pruning-process variance on TIMIT.
  • 3.3 Weight and Activation Quantization: 12-bit weight and activation quantization achieves no accuracy loss on TIMIT.The method uses linear quantization for weights and activations, with lookup tables and interpolation for sigmoid and tanh.
  • 3.3 Weight and Activation Quantization: 20.4% PER for the original network becomes 20.7% after pruning and fine-tuning, remaining 20.7% under 16/12-bit quantization.PER deteriorates to 84.5% with 8-bit quantization.

4. ENCODING AND COMPILING

The encoding and compiling flow maps compressed LSTM data onto processing elements for parallel execution. It uses row-wise partitioning, sparse storage, and aligned representations to support the accelerator’s data flow.

  • 4. ENCODING AND COMPILING: The scheduler organizes sparse matrix multiplication, element-wise multiplication, and memory references for accelerator execution.It is designed to make full use of the hardware accelerator.
  • 4. ENCODING AND COMPILING: Rows are assigned cyclically across n processing elements so the first matrix portion is available immediately for the next computation.The first n rows go to distinct processing elements, and row n + 1 returns to the first.
  • 4. ENCODING AND COMPILING: Sparse matrices store only nonzero weights using relative row indices and column pointers.Column pointers identify where new columns begin, while relative indices encode each nonzero weight’s position.
  • 4. ENCODING AND COMPILING: 16-bit storage combines each 12-bit quantized weight with a 4-bit relative row index to satisfy DDR byte alignment.This representation packages weight values and sparse-position information together.
  • 4. ENCODING AND COMPILING: CSC encoding locates weight columns through pointers and derives absolute addresses by accumulating relative indices.Zero-padding supports aligned storage and processing in the compressed representation.

5. HARDWARE IMPLEMENTATION

ESE combines sparse-LSTM-aware hardware components with a scheduler that overlaps memory access and computation while managing irregular workloads and dependencies. Its architecture partitions computation across processing elements and uses buffering, sparse-matrix readers, arithmetic units, and nonlinear-function units.

  • Hardware challenges: Load imbalance from sparsity forces processing elements to synchronize, causing waiting when their non-zero-weight workloads differ.
  • Hardware challenges: ESE addresses compressed-model irregularity with customized decoding, relative-index accumulation, padding zeros for large offsets, and dynamic-precision data handling.
  • System overview: The CPU+FPGA system separates the FPGA accelerator, CPU software, and external memory, which stores parameters and voice vectors beyond on-chip BRAM capacity.
  • ESE channel architecture: ESE channels implement independent voice-vector sequences using processing elements and units for SpMV, element-wise multiplication, accumulation, addition, and sigmoid/tanh activation.
  • ESE controller: The scheduler overlaps memory references with computation while executing horizontally dependent operations sequentially and vertically independent operations concurrently.
  • ESE channel architecture: ActQueue FIFOs decouple processing elements with unequal workloads so faster elements can fetch new input-vector elements instead of waiting.

6. EXPERIMENTAL RESULTS

Experiments evaluate ESE on FPGA hardware and speech datasets against CPU and GPU baselines. Results show high resource utilization, improved utilization with FIFO buffering, faster sparse execution, lower power, and substantial speed and energy-efficiency gains.

  • Experimental setup: ESE runs on an XCKU060 FPGA at 200 MHz with two external 4GB DDR3 DRAMs and host-controlled parameter and vector transfers.
  • Experimental setup: The evaluation uses TIMIT and a proprietary speech dataset containing 1000 hours of training, 100 hours of validation, and 10 hours of test data.
  • Resource utilization: 32 channels with 32 processing elements each almost fully utilize the FPGA hardware resources.
  • Resource utilization: 80% utilization without FIFO rises above 90% at depth 4, while depths 8 and 16 provide marginal additional gain; ESE therefore selects depth 8.
  • Accuracy, speed and energy efficiency: 6.2× speedup over the dense baseline is achieved at 10% density, compared with 5.5× without load-balance-aware pruning.
  • Accuracy, speed and energy efficiency: 41 watts for ESE compares with 111 watts for CPU and 202 watts for GPU; ESE is 43× faster than CPU, 3× faster than GPU, and 40×/11.5× more energy efficient.

7. RELATED WORK

Related work includes compression methods and accelerators for CNNs, sparse matrix-vector multiplication, GRUs, and LSTMs. ESE differs by targeting compressed sparse LSTM speech recognition with load balancing and FPGA hardware designed for the resulting irregular computation.

  • Deep compression: Deep Compression targets CNN and image-recognition models, whereas ESE targets LSTM speech recognition with load-balance-aware pruning, linear quantization, and no Huffman coding.
  • CNN accelerators: DianNao-series accelerators target CNNs with uncompressed dense weights, while ESE is designed for compressed sparse LSTM execution.
  • EIE accelerator: EIE processes compressed networks with sparse matrix-vector multiplication, but ESE applies this direction specifically to sparse LSTM speech-recognition workloads.
  • Recurrent-network accelerators: Prior FPGA GRU and LSTM implementations demonstrate FPGA advantages but did not support sparse LSTM networks; ESE is specifically designed for sparse LSTM.

8. CONCLUSION

ESE jointly optimizes LSTM compression, software mapping, scheduling, and FPGA architecture to execute compressed sparse models directly. On XCKU060, it achieves high throughput and substantially higher speed and energy efficiency than CPU and GPU implementations.

  • 20× compression combines pruning and quantization while preserving prediction accuracy, reducing memory bandwidth requirements for FPGA inference.
  • ESE schedules complex LSTM operations, handles compression-induced irregularity, balances computation and storage, and supports concurrent processing of multiple speech inputs.
  • 282 GOPS on compressed LSTM corresponds to 2.52 TOPS on dense LSTM, with 43× and 3× speedups over CPU and GPU and 40× and 11.5× energy-efficiency gains.
Loading 1612.00694v2…