Source-linked AI summary
FTRANS: Energy-Efficient Acceleration of Transformers using FPGA
Bingbing Li, Santosh Pandey, Haowen Fang, Yanjun Lyv, Ji Li, Jieyang Chen, Mimi Xie, Lipeng Wan, Hang Liu, Caiwen Ding
TL;DR
Large pre-trained transformer language representations are difficult to deploy on computation-, memory-, and energy-constrained hardware. Ftrans combines enhanced BCM-based compression with FPGA acceleration, reducing model size while improving performance and energy efficiency against CPU and GPU references.
Problem
Large pre-trained language representations create computation, weight-storage, and FPGA-resource challenges for deployment on constrained devices.
Method
Ftrans uses enhanced BCM-based model compression together with an FPGA architecture and design-automation approach to explore parallelism and achieve high throughput.
Results
Ftrans reduces NLP model size by up to 16 times and achieves 27.07× throughput, 81× energy efficiency versus CPU, and up to 8.80× energy-efficiency improvement versus GPU.
Takeaways & Limitations
The FPGA-based implementation significantly outperforms CPU and GPU in energy efficiency while maintaining small accuracy loss on Transformer.
Takeaways & Limitations
RoBERTa experiences a slightly higher accuracy drop because its parameters are more sensitive to compression after pretraining on 160GB of text.
Abstract
from arXiv · showhide
In natural language processing (NLP), the "Transformer" architecture was proposed as the first transduction model replying entirely on self-attention mechanisms without using sequence-aligned recurrent neural networks (RNNs) or convolution, and it achieved significant improvements for sequence to sequence tasks. The introduced intensive computation and storage of these pre-trained language representations has impeded their popularity into computation and memory-constrained devices. The field-programmable gate array (FPGA) is widely used to accelerate deep learning algorithms for its high parallelism and low latency. However, the trained models are still too large to accommodate to an FPGA fabric. In this paper, we propose an efficient acceleration framework, Ftrans, for transformer-based large scale language representations. Our framework includes enhanced block-circulant matrix (BCM)-based weight representation to enable model compression on large-scale language representations at the algorithm level with few accuracy degradation, and an acceleration design at the architecture level. Experimental results show that our proposed framework significantly reduces the model size of NLP models by up to 16 times. Our FPGA design achieves 27.07x and 81x improvement in performance and energy efficiency compared to CPU, and up to 8.80x improvement in energy efficiency compared to GPU.
1 INTRODUCTION
Transformers address limitations of recurrent sequence models through self-attention, but their computational and storage demands hinder deployment on constrained devices. Ftrans combines enhanced BCM-based compression with FPGA acceleration, reducing model size and improving performance and energy efficiency.
- Transformers replace sequence-aligned RNNs and convolution with self-attention, achieving notable improvements for sequence-to-sequence tasks.
- Pre-trained transformer language representations impose intensive computation and power demands, limiting their use on computation- and energy-constrained edge devices.
- Ftrans combines enhanced BCM-based weight compression with an FPGA architecture-level acceleration design for large-scale transformer language representations.
- Up to 16 times compression reduces NLP model size with small accuracy loss, while Ftrans targets lower hardware cost and energy consumption than CPU and GPU references.
- 27.07× throughput and 81× energy-efficiency improvements over CPU accompany up to 8.80× higher energy efficiency than GPU.
2 RELATED WORK
Attention mechanisms have become central to sequence modeling and transduction, motivating the development of transformer-based models across NLP applications.
- Attention mechanisms have become an integral part of sequence modeling and transduction models across various tasks.
3 TRANSFORMER WORKLOAD ANALYSIS
The Transformer uses stacked self-attention and fully connected layers in encoder and decoder stacks. Its workload includes single-head scaled dot-product attention, multi-head attention, and parallel attention configurations.
- Transformer architecture: The encoder maps input symbols to continuous representations, while the decoder generates output symbols autoregressively from previously generated symbols.
- Encoder: Each encoder layer combines multi-head self-attention with a fully connected feed-forward network, residual connections, and layer normalization.
- Decoder: Each decoder layer adds encoder-output attention and masked self-attention so position i depends only on previous positions.
- Attention: Scaled dot-product attention maps queries, keys, and values to outputs by computing query-key dot products, scaling them, and applying softmax weights.
- Attention: Multi-head attention concatenates multiple single-head outputs and projects them, enabling attention across different representation subspaces and positions.
- Configurations: The implementation uses 2 parallel attention layers with 4 heads for the shallow Transformer and 12 layers with 12 heads for RoBERTa.
4 TRANSFORMER COMPRESSION USING ENHANCED BLOCK-CIRCULANT MATRIX
Transformer models’ intensive computation and weight storage motivate compression. Ftrans enhances block-circulant matrix representations to reduce storage while preserving prediction accuracy through better parameter retention and FFT-based multiplication.
- Large pre-trained language representations create substantial computation and weight-storage challenges for hardware implementation.
- Block-circulant matrices replace weight matrices with circulant blocks, storing each block through an index vector instead of all rows or columns.
- The enhanced BCM method modifies the index-vector formulation because prior approaches had not addressed large-scale language representations.
- The modified representation better preserves parameter information by avoiding the loss of effective representations from other rows or columns.
- FFT-based multiplication computes BCM matrix-vector products equivalently through circular convolution, reducing the multiplication complexity relative to direct matrix-vector multiplication.
5 ARCHITECTURE
Ftrans partitions Transformer execution across FPGA resources, placing computational encoder/decoder stacks on chip while off-loading embeddings. Its architecture combines pipelined computation, specialized processing elements, buffering, and controller-managed data flow.
- Because Transformer parameters can exceed FPGA on-chip memory, Ftrans off-loads the embedding layer and keeps encoder/decoder stacks on chip.The embedding layer contributes 30.89% of parameters and is treated as a lookup table with less computation than the encoder and decoder.
- The overall FPGA architecture includes encoder/decoder computation units, on-chip memory banks, a transformer controller, and off-chip DDR memory.The host transfers tokenized sentences through PCIe, while the controller coordinates FPGA modules.
- The computation units decompose Transformer execution into primitives such as matrix multiplications, vectorized exponentials, multi-head attention, feed-forward layers, linear operations, and add/norm.
- Multi-Head Attention Design: Multi-head attention uses processing-element banks, K/Q/V buffers, normalization, masking, and softmax modules to coordinate attention computation.Local controllers manage data flow and mask future tokens in masked multi-head attention.
- Processing Elements: Ftrans uses PE-A and PE-B for matrix-vector multiplication and FFT/IFFT-based PEs for BCM-compressed fully connected layers.
- Processing Elements: PE-A and PE-B stream multiplier outputs into accumulators or dividers, allowing scaling and softmax to overlap with matrix multiplication.
- FFT/IFFT-Based PE and Softmax: The FFT/IFFT-based PE combines an FFT/IFFT kernel, an accumulator, and an adder, with the accumulator implemented as an N-input adder tree.
- FFT/IFFT-Based PE and Softmax: The softmax module approximates exponential functions with piece-wise linear functions, then buffers exponentials, accumulates their sum, and performs division.
6 DESIGN AUTOMATION & OPTIMIZATION
Ftrans automates FPGA design exploration by modeling dependencies, scheduling operations, and balancing resources across Transformer layers to improve pipeline throughput.
- Workflow: A data dependency graph captures computation flow, after which operators are scheduled into a pipeline under design constraints.A code generator converts the schedule into C/C++ for HLS synthesis.
- Operation decomposition: Major operations are decomposed into processing elements, while lighter operations such as division and softmax are merged or overlapped.Different layers can share processing elements, but unequal operation times create utilization imbalance.
- Resource optimization: The first optimization stage minimizes the maximum layer execution time by allocating resources to balance the pipeline.The formulation estimates each layer’s resource use and execution time under FPGA resource constraints.
- Resource optimization: The scheduling procedure identifies the slowest layer, reallocates resources toward it, and repeats until no further speedup is available.It can reclaim resources from substantially faster layers for the bottleneck layer.
7 EVALUATION
Ftrans is evaluated on shallow Transformer and RoBERTa models using compressed, fixed-point weights, then implemented on FPGA and compared with CPU and GPU baselines.
- Experimental setup: Enhanced BCM compression and 16-bit fixed-point weights are evaluated on shallow Transformer and RoBERTa language representations.Experiments cover WikiText-2 language modeling and IMDB sentiment classification.
- Compression accuracy: The shallow Transformer loses 0% accuracy with block size 4 and 0.6% with block size 8, while RoBERTa loses 4.2% and 4.3%, respectively.Changing 32-bit floating point to 16-bit fixed point causes no accuracy loss.
- FPGA evaluation: FPGA evaluation reports resource utilization, latency, throughput, and power across seven encoder/decoder stages, revealing a latency–power trade-off.The best trade-off occurs at batch size 8; larger batches significantly increase latency without increasing throughput.
- Compression accuracy: RoBERTa’s compression accuracy drop is slightly higher because its parameters were pretrained on 160GB of text and are more sensitive to compression.This caveat explains the difference relative to the shallow Transformer results.
8 CONCLUSION
The paper presents Ftrans as an energy-efficient FPGA framework combining enhanced BCM compression, architecture support, and automated parallelism optimization for large Transformer representations.
- Conclusion: Ftrans combines enhanced BCM-based model compression with FPGA architecture design and design automation to explore parallelism and achieve high throughput.The framework reduces NLP model size with small accuracy loss on Transformer and outperforms CPU and GPU in energy efficiency.
- Conclusion: Ftrans significantly reduces NLP model size while its FPGA implementation significantly outperforms CPU and GPU in energy efficiency.These conclusions are reported as experimental results for the proposed framework.