C-LSTM: Enabling Efficient LSTM using Structured Compression Techniques on FPGAs

Shuo Wang; Zhe Li; Caiwen Ding; Bo Yuan; Yanzhi Wang; Qinru Qiu; Yun Liang

Source-linked AI summary

C-LSTM: Enabling Efficient LSTM using Structured Compression Techniques on FPGAs

Shuo Wang, Zhe Li, Caiwen Ding, Bo Yuan, Yanzhi Wang, Qinru Qiu, Yun Liang

arXiv:1803.06305v1Mar 14, 2018cs.LGcs.AR

TL;DR

Large LSTM models improve acoustic recognition accuracy but exceed FPGA on-chip resource constraints, while pruning introduces irregular computation and memory access. C-LSTM uses block-circulant compression, FFT acceleration, and automated synthesis to create efficient FPGA implementations. It reports up to 18.8X performance and 33.5X energy-efficiency gains with very small accuracy degradation.

Problem
Increasing LSTM model size improves acoustic recognition accuracy but creates inefficient FPGA designs because on-chip resources are limited and pruning produces irregular sparse computation and memory accesses.
Method
C-LSTM combines block-circulant structured compression, FFT-based circulant convolution, and automatic optimization and synthesis for LSTM variants on FPGAs.
Results
Up to 18.8X performance and 33.5X energy-efficiency gains are achieved compared with the state-of-the-art LSTM implementation, with very small accuracy degradation.
Takeaways & Limitations
Structured compression reduces model storage and eliminates irregular computation and memory accesses, enabling efficient FPGA acceleration of LSTM models.
Takeaways & Limitations
The reported ESE storage comparison is pessimistic because compressed ESE indices may use fewer bits than weights.

Abstract

from arXiv · show

Recently, significant accuracy improvement has been achieved for acoustic recognition systems by increasing the model size of Long Short-Term Memory (LSTM) networks. Unfortunately, the ever-increasing size of LSTM model leads to inefficient designs on FPGAs due to the limited on-chip resources. The previous work proposes to use a pruning based compression technique to reduce the model size and thus speedups the inference on FPGAs. However, the random nature of the pruning technique transforms the dense matrices of the model to highly unstructured sparse ones, which leads to unbalanced computation and irregular memory accesses and thus hurts the overall performance and energy efficiency. In contrast, we propose to use a structured compression technique which could not only reduce the LSTM model size but also eliminate the irregularities of computation and memory accesses. This approach employs block-circulant instead of sparse matrices to compress weight matrices and reduces the storage requirement from $\mathcal{O}(k^2)$ to $\mathcal{O}(k)$. Fast Fourier Transform algorithm is utilized to further accelerate the inference by reducing the computational complexity from $\mathcal{O}(k^2)$ to $\mathcal{O}(k\text{log}k)$. The datapath and activation functions are quantized as 16-bit to improve the resource utilization. More importantly, we propose a comprehensive framework called C-LSTM to automatically optimize and implement a wide range of LSTM variants on FPGAs. According to the experimental results, C-LSTM achieves up to 18.8X and 33.5X gains for performance and energy efficiency compared with the state-of-the-art LSTM implementation under the same experimental setup, and the accuracy degradation is very small.

1 INTRODUCTION

LSTM models improve recognition accuracy but challenge FPGA implementation because their size, irregular sparsity, skewed computation, and complex dependencies strain limited resources. C-LSTM addresses these constraints with structured compression and automated synthesis while preserving regular hardware execution.

Challenges: FPGA LSTM designs are constrained by insufficient on-chip memory, sparse-index overhead, skewed operator complexity, and complicated data dependencies.These constraints make balanced resource allocation and regular memory access difficult.
Structured compression: Block-circulant compression reduces storage from O(k^2) to O(k) while retaining dense structure for FPGA acceleration.Each block-circulant matrix can be represented by a vector, avoiding sparse-matrix irregularities.
Structured compression: FFT acceleration reduces circulant-convolution complexity from O(k^2) to O(klogk).The FFT is applied to the most computation-intensive circulant-convolution operator.
Framework: C-LSTM automatically optimizes and synthesizes a wide range of LSTM variants for FPGA implementations.Its framework combines model training with template generation and automatic LSTM synthesis.
Results: 18.8X performance and 33.5X energy-efficiency gains are reported over the state-of-the-art LSTM implementation, with very small accuracy degradation.These are the paper’s reported peak gains under the same experimental setup.

2 LSTM BACKGROUND

LSTM is a recurrent architecture used in speech recognition and other sequence tasks, with gates, cell states, and projected outputs connected through recurrent computations. The Google LSTM variant illustrates these operations through matrix-vector products, element-wise operations, and nonlinear activations.

LSTM background: LSTM is a widely deployed RNN variant used in large-scale automatic speech recognition and other sequence-processing applications.The paper uses Google LSTM as its representative variant.
LSTM background: The LSTM equations define input, forget, and output gates together with cell state, cell output, and projected output variables.The symbols i, f, o, c, m, and y denote these components, respectively.
LSTM background: The model uses logistic activation σ and hyperbolic tangent h, with ⊙ denoting element-wise multiplication and + denoting element-wise addition.Peephole connections use diagonal weight matrices and can be computed through element-wise multiplication.
LSTM background: Nine matrix-vector multiplications remain after excluding peephole connections, and input/recurrent terms can be fused by concatenating matrices and vectors.The fusion combines W_*x x_t and W_*r y_{t−1} into one matrix-vector multiplication.

3 STRUCTURED COMPRESSION

The paper compresses LSTM weight matrices with block-circulant structure, trading block size against model size, computational complexity, and prediction accuracy. FFT-based inference and training preserve regular computation while reducing the cost of circulant convolution.

3.1 Block-Circulant Matrix: Block-circulant representation reduces each weight matrix’s parameter count by a factor determined by block size k.A k × k circulant block is represented by one row vector rather than all k^2 entries.
3.1 Block-Circulant Matrix: Larger block sizes increase compression ratio but may degrade prediction accuracy, whereas smaller blocks provide higher accuracy with less compression.Block size 1 corresponds to no compression.
3.2 Inference and Training Algorithms: FFT accelerates circulant convolution, reducing LSTM inference complexity from O(pqk^2) to O(pqk log k).The circulant blocks are represented by vectors and multiplied in the frequency domain using DFT, element-wise multiplication, and inverse DFT.
3.2 Inference and Training Algorithms: The block-circulant formulation also supports backpropagation through derivatives that retain block-circulant structure.The training procedure applies the same structured representation to backward propagation.
3.3 Compression and Accuracy Trade-offs: For Google LSTM, block sizes 8 and 16 reduce model size by 7.6X and 14.6X, while computational complexity falls by 2.6X and 3.7X and PER increases by only 0.32% and 1.23%.These compressed models are selected for further study.

4 FPGA ACCELERATION

The FPGA acceleration techniques optimize circulant convolution, quantize datapaths and activations, and schedule dependent LSTM operators across balanced coarse-grained pipelines. These choices reduce computation and memory irregularities while fitting the design within FPGA resource constraints.

Circulant Convolution Optimization: FFT-based circulant convolution reduces LSTM inference complexity from O(pqk^2) to O(pqk logk).The method targets the computation-intensive circulant convolution operator.
Circulant Convolution Optimization: DFT-IDFT decoupling reduces IDFT calls for each circulant convolution from q to 1.DFT and IDFT linearity allows the inverse transform to move outside the accumulation.
Circulant Convolution Optimization: Precomputed weight DFT values stored in FPGA BRAMs reduce runtime DFT calls from 2qk to qk per circulant convolution.The weights are fixed after training, so their transforms can be computed before inference.
Datapath and Activation Quantization: 16-bit datapath quantization and piece-wise linear sigmoid and tanh approximations improve resource utilization while preserving accuracy closely.The activation approximations use 22 segments and have less than 1% error relative to the original functions.
Operator Scheduling: The scheduler converts LSTM equations into a directed acyclic dependency graph and allocates operators to stages under FPGA resource constraints.It computes operator priorities, determines parallelism, and creates current or new pipeline stages.
Performance and Resource Models: Coarse-grained pipeline stages overlap execution using double buffers, while throughput is constrained by the slowest stage.The performance model uses stage execution cycles, operator workloads, parallelism, input size, and pipeline depth.

5 C-LSTM FRAMEWORK

C-LSTM combines compressed-model training with automatic FPGA implementation for a broad range of LSTM variants. Its implementation flow generates optimized operators and synthesizable hardware from the trained model.

Model Training: C-LSTM combines block-circulant compression during LSTM training with an automatic flow for generating FPGA inference designs.The framework separates model training from implementation on FPGAs.
C-LSTM Framework: C-LSTM is designed to automatically optimize and implement a wide range of LSTM variants on FPGAs.Its framework overview includes both training and implementation flows.
Model Training: The training flow iteratively explores the trade-off between compression ratio and prediction accuracy before exporting the trained inference model.The compressed model is configured with trained weights and passed to implementation.
Model Implementation: C-LSTM provides templates for tanh, sigmoid, element-wise addition, element-wise multiplication, and circulant convolution operators.These templates are intended to support almost any LSTM variant studied in the work.
Model Implementation: The implementation flow generates a dependency graph, schedules a multi-stage pipeline under resource constraints, and automatically emits C/C++ for HLS synthesis.The code generator integrates the selected primitive-operator templates before commercial HLS produces FPGA hardware.

6 EXPERIMENT EVALUATION

C-LSTM is evaluated against ESE on two FPGA platforms and across Google and Small LSTM configurations. The structured designs deliver substantial performance and energy gains with very small PER degradation.

6.1 Experiment Setup: C-LSTM is evaluated on Xilinx KU060 and ADM-7V3 FPGA platforms, using comparisons with ESE under the same dataset, LSTM algorithm, and platforms.The evaluation also includes a smaller bidirectional LSTM model with 39-dimensional inputs and 512-dimensional gate/cell layers.
6.2 Experimental Results of Google LSTM: 3.6X and 4.3X latency reductions and 11X and 13X performance speedups are achieved with FFT8 and FFT16 on KU060, respectively.The latency and throughput differ because the coarse-grained accelerator pipelines frames in stages.
6.2 Experimental Results of Google LSTM: The comparison’s ESE storage estimate includes weights and indices and is explicitly described as pessimistic because indices may use fewer bits than weights.C-LSTM stores the compressed weights and projection matrix in BRAM.
6.2 Experimental Results of Google LSTM: 10.2X and 18.8X performance speedups and 19.1X and 33.5X energy-efficiency gains are achieved with FFT8 and FFT16 on ADM-7V3, respectively.C-LSTM power consumption is reported as half that of ESE; the energy-efficiency comparison is described as pessimistic because the platforms use different manufacturing processes.
6.2 Experimental Results of Google LSTM: 0.32% and 1.23% relative PER degradation is reported for FFT8 and FFT16 Google LSTM designs, respectively.The proposed designs use fewer LUT, FF, and BRAM resources than ESE, while structured compression exposes more parallelism and keeps the model on-chip.
6.3 Experimental Results of Small LSTM: 19.3X and 35.9X KU060 performance speedups are achieved for Small LSTM with FFT8 and FFT16, while ADM-7V3 reaches 17.5X and 31.9X speedups.ADM-7V3 energy-efficiency gains are 34.2X and 59.4X, with PER degradation of 0.29% and 1.16% for FFT8 and FFT16.

7 RELATED WORK

Prior FPGA research spans CNN and RNN acceleration, but the paper distinguishes C-LSTM from existing approaches by targeting compressed LSTM models with structured block-circulant computation.

CNN Accelerators: Prior FPGA CNN accelerators use techniques including systolic arrays, Winograd convolution, and frequency-domain FFT processing.The cited CNN FFT approach differs from C-LSTM because it accelerates CNN convolution layers rather than LSTM circulant convolution operators.
RNN Accelerators: Existing FPGA RNN work includes GRU and LSTM accelerators, but the cited LSTM studies do not use compression techniques to reduce model size.The GRU accelerator targets an architecture different from LSTM-based RNNs.
Compressed LSTM Accelerators: ESE is identified as the most relevant prior study because it combines software and hardware co-design for compressed sparse LSTM models.C-LSTM instead addresses structured compression and regular computation and memory access patterns.

8 CONCLUSION

The paper concludes that block-circulant compression, FFT-based computation, and automatic synthesis enable efficient FPGA implementations across LSTM variants. C-LSTM achieves large performance and energy-efficiency gains with small accuracy degradation.

8 CONCLUSION: Block-circulant matrices compress LSTM models enough to fit FPGA BRAM while eliminating irregular computation and memory accesses.The regular structure supports efficient hardware organization without the irregularities of sparse matrices.
8 CONCLUSION: FFT-based fast circulant convolution reduces computational and storage complexity for LSTM inference.The FFT acceleration is integrated with the structured block-circulant representation.
8 CONCLUSION: C-LSTM provides an automatic optimization and synthesis framework for a wide range of LSTM variants on FPGAs.The framework generates implementations that combine the compression and acceleration techniques.
8 CONCLUSION: 18.8X performance and 33.5X energy-efficiency gains are reported against the state-of-the-art LSTM implementation, with small accuracy degradation.These are the maximum gains stated in the conclusion.