PaperlayerView exact revision on arXiv

Source-linked AI summary

FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAs

Shulin Zeng, Jun Liu, Guohao Dai, Xinhao Yang, Tianyu Fu, Hongyi Wang, Wenheng Ma, Hanbo Sun, Shiyao Li, Zixiao Huang, Yadong Dai, Jintao Li, Zehao Wang, Ruoyu Zhang, Kairui Wen, Xuefei Ning, Yu Wang

arXiv:2401.03868v2cs.ARcs.AI

TL;DR

LLM inference is constrained by heavy computation, memory demands, and hardware inefficiencies in processing compressed models. FlightLLM provides a complete FPGA mapping flow using sparse DSP computation, on-chip mixed-precision decoding, and adaptive compilation. It reports higher energy and cost efficiency than commercial GPUs and higher throughput than NVIDIA A100 in the evaluated settings.

  • Problem

    Compressed LLMs remain difficult for existing hardware because inference has low computational efficiency, underutilized memory bandwidth, and large compilation overheads.

  • Method

    FlightLLM combines a configurable sparse DSP chain, an always-on-chip mixed-precision decode scheme, and length-adaptive compilation for FPGA inference.

  • Results

    6.0× higher energy efficiency and 1.8× better cost efficiency than NVIDIA V100S are reported, while Versal VHK158 achieves 1.2× higher throughput than NVIDIA A100.

  • Takeaways & Limitations

    FlightLLM demonstrates that FPGAs are promising candidates for efficient LLM inference using FPGA-specific resources and a complete mapping flow.

Abstract

from arXiv · show

Transformer-based Large Language Models (LLMs) have made a significant impact on various domains. However, LLMs' efficiency suffers from both heavy computation and memory overheads. Compression techniques like sparsification and quantization are commonly used to mitigate the gap between LLM's computation/memory overheads and hardware capacity. However, existing GPU and transformer-based accelerators cannot efficiently process compressed LLMs, due to the following unresolved challenges: low computational efficiency, underutilized memory bandwidth, and large compilation overheads. This paper proposes FlightLLM, enabling efficient LLMs inference with a complete mapping flow on FPGAs. In FlightLLM, we highlight an innovative solution that the computation and memory overhead of LLMs can be solved by utilizing FPGA-specific resources (e.g., DSP48 and heterogeneous memory hierarchy). We propose a configurable sparse DSP chain to support different sparsity patterns with high computation efficiency. Second, we propose an always-on-chip decode scheme to boost memory bandwidth with mixed-precision support. Finally, to make FlightLLM available for real-world LLMs, we propose a length adaptive compilation method to reduce the compilation overhead. Implemented on the Xilinx Alveo U280 FPGA, FlightLLM achieves 6.0$\times$ higher energy efficiency and 1.8$\times$ better cost efficiency against commercial GPUs (e.g., NVIDIA V100S) on modern LLMs (e.g., LLaMA2-7B) using vLLM and SmoothQuant under the batch size of one. FlightLLM beats NVIDIA A100 GPU with 1.2$\times$ higher throughput using the latest Versal VHK158 FPGA.

1 INTRODUCTION

LLMs face substantial computation and memory overheads, while compressed models remain difficult for current hardware to process efficiently. FlightLLM addresses FPGA inference challenges with sparse computation, on-chip decoding, and adaptive compilation, achieving strong efficiency and throughput results.

  • GPT-3 has 175 billion parameters, requiring about 660TOPS for a single inference and approximately 350GB in FP16.
  • Current hardware struggles to efficiently support compressed LLMs because GPUs favor structured sparsity while unstructured sparsity does not provide end-to-end acceleration.
  • LLM inference on FPGAs faces low computation efficiency, underutilized memory bandwidth of 29-43%, and compilation overhead reaching approximately TB for 2048 input tokens.
  • FlightLLM uses a configurable sparse DSP chain, always-on-chip mixed-precision decoding, and length-adaptive compilation to address these three challenges.
  • 6.0× higher energy efficiency and 1.8× better cost efficiency than NVIDIA V100S are reported on modern LLMs, with 1.2× higher throughput than NVIDIA A100 on Versal VHK158.

2 BACKGROUND AND RELATED WORK

Transformer LLMs combine multi-head attention and feed-forward networks, operating through prefill and iterative decode stages. Prior compression and accelerator approaches use sparsity or quantization, but existing methods have limitations for LLM decode acceleration.

  • Transformer blocks generally contain multi-head attention and feed-forward networks.
  • Multi-head attention projects token embeddings into query, key, and value matrices before computing attention for each head.
  • The feed-forward network applies two fully connected layers and a nonlinear activation to generate the next block's token embedding.
  • Prefill processes all prompt tokens simultaneously, whereas decode processes one newly generated token iteratively through matrix-vector multiplications.
  • Compression methods include low-bit quantization, sparse attention, and weight pruning across patterns such as block, N:M, and unstructured sparsity.
  • Existing transformer accelerators emphasize specialized sparse-attention or mixed-precision designs, but cannot accelerate the LLM decode stage effectively.

3 COMPUTING ARCHITECTURE

FlightLLM’s computing architecture combines configurable FPGA DSP resources with unified matrix engines to support dense, sparse, matrix-matrix, and matrix-vector operations. Its hardware/software co-design targets efficient sparse computation and FPGA memory utilization.

  • Overall Architecture: FlightLLM organizes each core around an MPE, MMU, SFU, and instruction scheduler for matrix, memory, miscellaneous, and control operations.The MPE handles matrix operations, while the MMU and SFU address memory access and non-matrix functions.
  • Unified Matrix Processing Engine: The unified MPE supports GEMM, SpMM, GEMV, SpMV, and SDDMM through configurable MPU and VPU structures.MPUs can operate in matrix-matrix or matrix-vector mode, while VPUs perform vector dot products.
  • MPU Design: The configurable sparse DSP chain divides DSP48 resources into groups with configurable interconnections, supporting sparse reduction while reducing hardware overhead.Sparse multiplexers select nonzero inputs, reduction nodes separate outputs, and overflow adjustment manages long accumulations.
  • MPU Design: The CSD-Chain executes dense and sparse multiplications while fully using DSP48 cores, with reduction nodes splitting sparse execution into separate MAC chains.The sparse case produces two MAC outputs by breaking the cascade path into two chains.
  • MPU Design: N:M sparsity maps to CSD-Chain groups so one cycle produces one dense MAC output or N sparse MAC outputs.The mapping constrains nonzero positions while retaining flexible sparsity; the cited example uses a 2:4 pattern.
  • Matrix-Vector Multiplication Analysis: Redesigned computational parallelism allows MPUs to maximize GEMV and SpMV execution while maintaining off-chip memory-bandwidth utilization in MV mode.For double buffering, the design targets memory time below computation time.

4 ALWAYS-ON-CHIP DECODE

FlightLLM improves decode efficiency by keeping activation vectors on chip and fusing operations across layers, while using FPGA memory hierarchy and mixed-precision support to reduce memory traffic. The same fusion principles extend to prefill attention and miscellaneous operations.

  • On-chip Decode Dataflow: 65.9% HBM bandwidth utilization is achieved through decode-stage operator fusion, up from about 35.6%.The fusion targets frequent off-chip accesses caused by small activation vectors.
  • On-chip Decode Dataflow: Fusing all decode-stage layer computations keeps activation vectors on chip and writes results off chip only after each inference.Because decode activations are small vectors, they fit in the FPGA’s on-chip buffer.
  • On-chip Decode Dataflow: The design retains intermediate activations in on-chip buffers so successive layers can pass data directly to the MPE or SFU.This avoids writing each layer’s activation to off-chip memory.
  • Fusion: MPE and SFU computations are fused because their hardware resources do not conflict, reducing off-chip traffic for intermediate results.Softmax and LayerNorm follow MV computation, while element-wise operations can execute in the SFU as soon as their inputs are available.
  • Fusion in the Prefill: Prefill fusion uses MM rather than MV, allowing Softmax and LayerNorm to start after an activation vector is produced and element-wise operations after each MM.The prefill schedule adapts the fusion pattern to matrix-matrix computation.
  • Fusion in the Prefill: Sparse prefill attention skips loads and matrix multiplications when the zero attention mask covers all corresponding results.Partial mask coverage also limits the computed regions.

5 SOFTWARE DESIGN

FlightLLM’s software design maps compressed LLMs to FPGA instructions through adaptive compilation, platform-aware hardware generation, and a complete deployment flow. Length sharing and instruction compression reduce storage requirements for variable-length inference.

  • Instruction Set Architecture: FlightLLM uses an ISA with LD, ST, MM, MV, MISC, and SYS instructions to connect the LLM workload with accelerator hardware.These instructions cover data movement, matrix operations, miscellaneous functions, and synchronization.
  • Length Adaptive Compilation: Static compilation would require different instructions for each growing token length, creating large instruction volumes for generative LLM inference.Each decode inference increases token length by one, expanding the set of required input shapes.
  • Length Adaptive Compilation: Length-adaptive compilation reuses instructions across token-length ranges, such as using the 16-token instructions for lengths from 1 through 16.Threshold ranges reduce the need to store instructions for every possible prefill or decode length.
  • Length Adaptive Compilation: Refined decode thresholds and configurable SLR base addresses reduce the instruction size to 4.77 GB, which fits in U280 DDR.Decode receives finer thresholds because its memory-access bottleneck scales with token length.
  • Instruction Compression: Combining similar transfers across eight HBM channels into one instruction lets the hardware decoder launch eight channel operations concurrently.This optimization further reduces instruction size while preserving parallel memory access.
  • Platform-Aware Generation: The RTL generator selects computation parallelism and buffer sizes from FPGA DSP, HBM/DDR, and on-chip RAM resources.It generates platform-specific RTL and compilation configurations to use available resources.
  • Overall Mapping Flow: The complete mapping flow converts a PyTorch LLM into ISA through sparsification, quantization, automated intermediate-representation export, and subsequent optimization.The exported representation includes model structure, weights, sparse indexes, and attention masks.

6.1 Evaluation Setup

FlightLLM is evaluated on compressed OPT-6.7B and LLaMA2-7B models using FPGA platforms, GPU baselines, and latency, throughput, and accuracy metrics.

  • Models and Datasets: OPT-6.7B and LLaMA2-7B are evaluated after fine-tuning compressed models on 8192 RedPajama rows containing 56M tokens.Accuracy is measured on WikiText-103 and WikiText-2.
  • Metrics: Latency measures end-to-end inference time, while throughput measures decode-stage output tokens per unit time.Unless otherwise noted, experiments use batch size 1 for latency-sensitive scenarios.
  • FPGA Platforms: Experiments use Xilinx Alveo U280 hardware and Versal VHK158 evaluation, with U280 implemented on a real system and VHK158 evaluated using a cycle-accurate simulator.The U280 provides 8GB HBM at 460GB/s and 32GB DDR at 38GB/s; VHK158 provides 32GB HBM at 819GB/s.
  • Baselines: DFX, FACT, and CTA serve as accelerator baselines whose hardware parameters are aligned for fair comparison.Because open-source implementations and recent-LLM support are unavailable, C++ simulators reproduce the corresponding hardware designs with less than 5% deviation from original data.

6.2 Evaluation Results

FlightLLM combines compression-aware FPGA mechanisms with optimized memory access and achieves strong latency, throughput, energy-efficiency, and cost-efficiency results across models and baselines.

  • Comparison with Accelerators: 2.7× and 4.6× geomean latency speedups over DFX are achieved on U280 and VHK158, respectively, for OPT-6.7B.Corresponding geomean throughput speedups are 2.6× and 4.6×; FlightLLM has the highest throughput among the compared architectures.
  • Energy and Cost Efficiency: 6.0× and 4.2× higher energy efficiency than V100S-opt and A100-opt, respectively, are achieved on U280 for OPT-6.7B.For LLaMA2-7B, the corresponding gains are 5.5× and 3.8×.
  • Energy and Cost Efficiency: 1.9× and 1.5× higher geomean cost efficiency than V100S-opt and A100-opt are achieved for OPT-6.7B.For LLaMA2-7B, the corresponding gains are 2.3× and 1.4×.
  • Latency Breakdown: 1.6–1.7× performance improvement follows use of the always-on-chip decoder after flexible sparsity and the sparse DSP chain.The reported improvement is attributed to reducing off-chip memory-access overhead.
  • Comparison with gpt-fast: 92.5 tokens/s on VHK158 is reported for LLaMA2-7B, with 2.9× better energy efficiency than gpt-fast on A100.FlightLLM also reports higher bandwidth utilization: 64.8% versus 44.6%.
  • Multi-batch Performance: FlightLLM’s performance advantage over GPU decreases as batch size increases because GPUs provide more memory bandwidth and computing resources.The multi-batch comparison reports GPU-opt results because gpt-fast does not support multi-batch processing.

7 CONCLUSION

FlightLLM uses FPGA-specific resources and compression-aware mapping to make FPGA-based LLM inference efficient, demonstrating strong energy and cost efficiency against commercial GPUs.

  • Conclusion: FlightLLM demonstrates that FPGA-specific resources can support efficient LLM inference with a complete mapping flow.The conclusion highlights 6.0× higher energy efficiency and 1.8× better cost efficiency than commercial GPUs on modern LLMs such as LLaMA2.
Loading 2401.03868v2…