Source-linked AI summary
ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGA
Song Han, Junlong Kang, Huizi Mao, Yiming Hu, Xin Li, Yubin Li, Dongliang Xie, Hong Luo, Song Yao, Yu Wang, Huazhong Yang, William J. Dally
TL;DR
LSTM’s high computation complexity and memory footprint make it difficult to deploy and lead to high power consumption. ESE compresses LSTM models to 5% of their original size while retaining similar accuracy, then uses hardware-friendly mapping and specialized hardware to process the compressed models. ESE is 43× faster than CPU and 3× faster than GPU; for sparse LSTM, it is 40.0× and 11.5× more energy efficient than CPU and GPU, respectively.
Problem
LSTM’s high computation complexity and memory footprint make it difficult to deploy and lead to high power consumption.
Source: §1 · p. 1Method
ESE compresses LSTM models to 5% of their original size while retaining similar accuracy, then uses hardware-friendly mapping and specialized hardware to process the compressed models.
Source: §1 · p. 1Results
ESE is 43× faster than CPU and 3× faster than GPU; for sparse LSTM, it is 40.0× and 11.5× more energy efficient than CPU and GPU, respectively.
Source: §6.3 · p. 8Takeaways & Limitations
For the evaluated speech-recognition LSTM, ESE demonstrates FPGA inference on compressed sparse models with speed and energy-efficiency advantages over CPU and GPU implementations.
Source: §6.3 · p. 8, §8 · p. 10
Abstract
from arXiv · showhide
Long Short-Term Memory (LSTM) is widely used in speech recognition. In order to achieve higher prediction accuracy, machine learning scientists have built larger and larger models. Such large model is both computation intensive and memory intensive. Deploying such bulky model results in high power consumption and leads to high total cost of ownership (TCO) of a data center. In order to speedup the prediction and make it energy efficient, we first propose a load-balance-aware pruning method that can compress the LSTM model size by 20x (10x from pruning and 2x from quantization) with negligible loss of the prediction accuracy. The pruned model is friendly for parallel processing. Next, we propose scheduler that encodes and partitions the compressed model to each PE for parallelism, and schedule the complicated LSTM data flow. Finally, we design the hardware architecture, named Efficient Speech Recognition Engine (ESE) that works directly on the compressed model. Implemented on Xilinx XCKU060 FPGA running at 200MHz, ESE has a performance of 282 GOPS working directly on the compressed LSTM network, corresponding to 2.52 TOPS on the uncompressed one, and processes a full LSTM for speech recognition with a power dissipation of 41 Watts. Evaluated on the LSTM for speech recognition benchmark, ESE is 43x and 3x faster than Core i7 5930k CPU and Pascal Titan X GPU implementations. It achieves 40x and 11.5x higher energy efficiency compared with the CPU and GPU respectively.
1 Stanford University, 2 DeePhi Tech, 3 Tsinghua University, 4 NVIDIA · 1. INTRODUCTION
The introduction presents ESE as a cross-stack approach to reduce LSTM memory demands through compression, hardware-friendly mapping, and specialized hardware. It identifies compression, scheduling, and sparse-model hardware as its main contributions.
1. INTRODUCTION
- Memory references consume more than two orders of magnitude more energy than ALU operations, while LSTM’s computation complexity and memory footprint make deployment power-intensive.The introduction therefore focuses on optimizing memory footprint.Source: §1 · p. 1
- The proposed flow compresses LSTM models to 5% of their original size using 10% density and 2× narrower weights while retaining similar accuracy, then maps and runs them directly on specialized hardware.This spans algorithm, software, and hardware optimization.Source: §1 · p. 1
- The paper proposes a shift from Training=>Inference to Training=>Compression=>Accelerated Inference for faster, more energy-efficient deep-learning inference.LSTM serves as the case study for this flow.Source: §1 · pp. 1–2
- The LSTM compression algorithm combines load-balance-aware pruning with an automatic flow for dynamic-precision data quantization.Source: §1 · p. 2
- The scheduler addresses recurrent LSTM data dependencies by overlapping memory references with computation when scheduling operations.Source: §1 · p. 2
- ESE runs directly on the irregular sparse model, balancing and partitioning computation and storage, and supports concurrent processing of multiple users’ speech data.Source: §1 · p. 2
- The work studies the LSTM speech-recognition system across algorithm, software, and hardware, jointly analyzing the trade-off between prediction accuracy and latency.Source: §1 · p. 2
2. BACKGROUND
Speech recognition uses a front end to extract features and a back end with acoustic and language models plus a decoder to produce words. Because LSTM acoustic modeling is computation- and memory-intensive and accounts for over 90% of pipeline execution time, the paper focuses on accelerating it.
2. BACKGROUND
- The back end processes front-end features with acoustic and language models, then uses MAP estimation to generate a word sequence; the LSTM serves as the acoustic model.The acoustic and language models provide P(X|W) and P(W), respectively.Source: §2 · p. 2
- Over 90% of the speech-recognition pipeline’s execution time is spent in LSTM, the pipeline’s most computation- and memory-intensive part, motivating the paper’s focus on accelerating it.Source: Figure 3 · p. 2, §2 · p. 2
- The standard LSTM uses recurrent memory cells to retain temporal state, with input, output, and forget gates controlling input flow, output flow, and adaptive forgetting.Its input at time T depends on the output at T −1.Source: §2 · p. 2, Figure 4 · p. 2
3. MODEL COMPRESSION
Model compression reduces computation and memory by removing redundant weights and quantizing parameters. The section presents pruning, load-balance-aware pruning, and weight and activation quantization.
3.1 Pruning
- On TIMIT, PER increased dramatically only after more than 93% of parameters were pruned; a larger 1000-hour training corpus allowed 90% pruning without hurting WER.The pruning process trains to identify necessary weights, removes weights below an empirical absolute-value threshold, then retrains under the sparsity constraint.Source: §3.1 · p. 3
3.2 Load Balance-Aware Pruning
- With the same total nonzero weights, load-balance-aware pruning reduced processing from five cycles to three in the illustrated four-PE example, with very similar performance and accuracy.It assigns the same sparsity quota to each submatrix to distribute nonzero weights evenly; the accuracy margin versus original pruning was within the pruning process's variance.Source: §3.2 · p. 3, Figure 5 · p. 3, Figure 6 · p. 3
3.3 Weight and Activation Quantization
- PER was 20.4% for the original TIMIT network, 20.7% after pruning and fine-tuning with 32-bit floats, and remained 20.7% under 16/12-bit quantization, but rose to 84.5% at 8 bits.Weights and activations use linear quantization; sigmoid and tanh lookup tables use 2048 samples over [-64, 64] and [-128, 128], respectively, with 16-bit outputs and 15-bit decimals.Source: §3.3 · p. 4
4. ENCODING AND COMPILING
This section explains how sparse LSTM weights are encoded and partitioned, and how a data-flow scheduler coordinates computation across processing elements (PEs).
4. ENCODING AND COMPILING
- A data-flow scheduler coordinates sparse matrix multiplication, element-wise multiplication, and memory references to use the accelerator fully.Source: §4 · p. 4
- Partitioning successive matrix rows across n PEs lets the first n rows be read in one cycle and used immediately in the next computation step.The n + 1 row is assigned to the first PE again.Source: §4 · p. 4
- CSC encoding stores only nonzero weights, using relative row indexes and column pointers to locate columns; Figure 7 illustrates zero-padding for data alignment.Absolute weight addresses are calculated by accumulating relative indexes from the column pointer.Source: §4 · p. 4, Figure 7 · p. 4
- Weights use 16-bit storage to satisfy DDR byte alignment, packing each 12-bit quantized weight with a 4-bit relative row index.Source: §4 · p. 4
5. HARDWARE IMPLEMENTATION
ESE implements sparse LSTM acceleration on a CPU–FPGA system, using scheduled processing elements, on-chip buffers, and external memory to manage computation and data movement.
5.1 Motivation
- Sparse LSTM hardware must balance synchronized PE workloads and handle relative-index decoding, mismatched data widths, dynamic-precision shifts, and dependency-aware scheduling.When some PEs have fewer non-zero weights, they wait for the busiest PE; 4-bit relative offsets above 16 require padding zeros.Source: §5.1 · p. 5, Figure 9 · p. 5
5.2 System Overview
- The CPU–FPGA system places PEs, controllers, and input/output buffers on the FPGA, while external DRAM stores model parameters and voice vectors beyond BRAM capacity.The ESE Controller schedules data fetches and the accelerator pipeline; PEs compute LSTM tasks for voice-vector slices.Source: §5.2 · p. 5
5.3 ESE Controller (Scheduler)
- The ESE scheduler overlaps memory fetches and computation, executing dependent operations sequentially and independent operations concurrently, subject to DDR resource conflicts.Element-wise operations and nonlinear functions run in parallel with matrix-vector multiplication and weight fetching.Source: §5.3 · pp. 5–6, Figure 11 · p. 6
5.4 ESE Channel Architecture
- Within each channel, ActQueue FIFOs decouple imbalanced PE workloads, while encoded-weight buffers feed SpMV and accumulation alongside element-wise multiplication, summation, and activation units.The ActQueue FIFO depth ranges from 1 to 16; encoded SpmatRead entries contain a 4-bit index and a 12-bit weight.Source: §5.4 · pp. 6–7
5.5 Memory System
- The memory system uses ping-pong double buffers to overlap transfer and computation, with DDR3 and interface-width conversions handling data movement between the accelerator and software.Eight 16-bit results are assembled into a 128-bit value for the PCIe interface, while a 512-bit input vector is split into thirty-two 16-bit values.Source: §5.5 · p. 7
6. EXPERIMENTAL RESULTS
ESE’s speech-recognition LSTM results show higher speed and energy efficiency than CPU and GPU baselines, while its hardware design addresses load imbalance in sparse execution.
6.1 Experimental Setup
- Experiments used an XCKU060 FPGA at 200 MHz with two 4GB DDR3 DRAMs, plus TIMIT and proprietary 1000/100/10-hour train/validation/test data.Baselines were an i7-5930k CPU and Pascal Titan X GPU; dense and sparse matrix operations used MKL BLAS/cuBLAS and MKL SPARSE/cuSPARSE, respectively.Source: §6.1 · p. 7
6.2 Resource Utilization
- With 32 channels of 32 PEs, ESE nearly fully utilizes FPGA resources; 32 PEs per channel balances computation with DRAM transfer at 200 MHz.The design requires data-transfer speed at least equal to computation speed to avoid starving the DSPs.Source: §6.2 · pp. 7–8, Table 6 · p. 8
- FIFO depth 1 yields 80% utilization, while depth 4 exceeds 90%; depths 8 and 16 add only marginal gain, so ESE uses depth 8.The last matrix, Wym, remains poorly utilized because it has few rows and few elements per PE, which the FIFO cannot fully address.Source: Figure 13 · p. 8, §6.2 · p. 8
6.3 Accuracy, Speed and Energy Efficiency
- 41 watts for ESE compared with CPU power of 111 watts (38 watts with MKLSparse) and GPU power of 202 watts (136 watts with cuSparse).ESE power was measured as the difference between socket power with and without the FPGA board.Source: Figure 14 · p. 8, §6.3 · p. 8
- 6.2× speedup over the dense baseline resulted from pruning to 10% of parameters; load-balance-aware pruning raised speedup from 5.5× to 6.2×.Source: §6.3 · p. 8
- ESE is 43× faster than CPU and 3× faster than GPU, with 197.0×/40.0× (dense/sparse) CPU and 14.3×/11.5× (dense/sparse) GPU energy efficiency.Source: §6.3 · p. 8, Table 8 · p. 9
- 282 GOPS sparse-LSTM throughput on ESE corresponds to 2.52 TOPS dense-equivalent; for a 1024-hidden LSTM, ESE takes 82.7 us versus CPU 6017.3/3569.9 us and GPU 240.2/287.4 us (dense/sparse).The model was pruned to 10% non-zeros, or 11.2% including padding zeros. At batch=32, CPU sparse is faster than dense while GPU sparse is slower; without batching, sparse is faster on both.Source: §6.3 · p. 8, Table 8 · p. 9
7. RELATED WORK
The section positions ESE as an FPGA accelerator for sparse LSTM speech recognition, distinguishing its load-balanced compression and scheduling from prior CNN, SpMV, and recurrent-network accelerators.
Deep Compression
CNN Accelerators
- DianNao, DaDianNao, and ShiDianNao accelerate CNNs with dense uncompressed weights, while Angel-Eye also targets CNNs despite supporting compression.ESE instead targets LSTM speech recognition and supports compressed data.Source: §7 · p. 9
EIE Accelerator
- EIE [10] accelerates compressed-network sparse matrix-vector multiplication with weight sharing; unlike its ASIC and codebook-quantization design, ESE targets FPGA LSTM with linear quantization.EIE is described as a general deep-learning building block rather than being designed specifically for LSTM speech recognition.Source: §7 · p. 9
Sparse Matrix-Vector Multiplication Accelerators
- Prior FPGA SpMV accelerators exploit static weight sparsity, whereas ESE uses relative-indexed CSC storage and a scheduler to map complex LSTM networks.The cited work includes FPGA-based SpMV designs by Zhuo et al., Fowers et al., and Dorrance et al..Source: §7 · p. 9
GRU on FPGA
- Nurvitadhi et al. demonstrate FPGA performance-per-watt advantages for GRU over CPU and GPU, while ESE accelerates sparse LSTM.Their accelerator runs on Stratix V and Arria 10 FPGAs.Source: §7 · pp. 9–10
LSTM on FPGA
- Chang [1] and Lee [15] accelerate LSTM or RNNs on FPGA, but their implementations do not support sparse LSTM, which ESE supports for additional speedup.Chang's Zynq 7020 implementation uses two layers and 128 hidden units and is 21 times faster than its embedded ARM Cortex-A9 CPU.Source: §7 · p. 10
8. CONCLUSION
ESE combines a 20× LSTM compression method, an FPGA scheduler, and hardware designed for compressed sparse models. On Xilinx XCKU060, it achieves 282 GOPS and outperforms CPU and GPU implementations in speed and energy efficiency.
8. CONCLUSION
- ESE is 43× faster than the Core i7 CPU and 3× faster than the Pascal Titan X GPU, with 40× and 11.5× higher energy efficiency, respectively.The speed and energy-efficiency comparisons are for the CPU and GPU, respectively.Source: §8 · p. 10
- On Xilinx XCKU060, ESE achieves 282 GOPS on the compressed model, equivalent to 2.52 TOPS for dense LSTM.Source: §8 · p. 10
- ESE compresses LSTM models by 20× without sacrificing prediction accuracy, then uses a scheduler for FPGA parallelism and hardware that handles compression-induced irregularity.The compression reduces FPGA memory-bandwidth requirements.Source: §8 · p. 10