PaperlayerView exact revision on arXiv

Source-linked AI summary

EdgeLLM: A Highly Efficient CPU-FPGA Heterogeneous Edge Accelerator for Large Language Models

Mingqiang Huang, Ao Shen, Kai Li, Haoxiang Peng, Boyu Li, Yupeng Su, Hao Yu

arXiv:2407.21325v2cs.AR

TL;DR

Deploying LLMs on edge devices is difficult because of heavy computation, memory access, complex operator graphs, and compilation requirements. EdgeLLM addresses these constraints with CPU-FPGA heterogeneous execution, mix-precision and sparse hardware, unified data handling, and end-to-end compilation. The deployed accelerator reports higher throughput and energy efficiency than GPU and FlightLLM baselines.

  • Problem

    LLM edge deployment is constrained by heavy computation and memory access, complex operator graphs, and compilation challenges.

    Source: §I · pp. 1, 3

  • Method

    EdgeLLM combines mix-precision group-systolic hardware, structured sparsity, unified data formats, and dynamic end-to-end compilation on a CPU-FPGA system.

    Source: §I · p. 3, B. Design of Mix-Precision Computing PEs · p. 5

  • Results

    1.91× higher throughput and 7.55× higher energy efficiency are reported versus the NVIDIA A100-SXM4-80G GPU.

    Source: §I · p. 3

  • Takeaways & Limitations

    EdgeLLM supports heterogeneous edge deployment across common MHA+FFN LLM structures while improving reported throughput and energy efficiency.

    Source: §I · p. 3, B. Analysis on the Hardware Efficiency · p. 12

Abstract

from arXiv · show

The rapid advancements in artificial intelligence (AI), particularly the Large Language Models (LLMs), have profoundly affected our daily work and communication forms. However, it is still a challenge to deploy LLMs on resource-constrained edge devices (such as robots), due to the intensive computation requirements, heavy memory access, diverse operator types and difficulties in compilation. In this work, we proposed EdgeLLM to address the above issues. Firstly, focusing on the computation, we designed mix-precision processing element array together with group systolic architecture, that can efficiently support both FP16*FP16 for the MHA block (Multi-Head Attention) and FP16*INT4 for the FFN layer (Feed-Forward Network). Meanwhile specific optimization on log-scale structured weight sparsity, has been used to further increase the efficiency. Secondly, to address the compilation and deployment issue, we analyzed the whole operators within LLM models and developed a universal data parallelism scheme, by which all of the input and output features maintain the same data shape, enabling to process different operators without any data rearrangement. Then we proposed an end-to-end compiler to map the whole LLM model on CPU-FPGA heterogeneous system (AMD Xilinx VCU128 FPGA). The accelerator achieves 1.91x higher throughput and 7.55x higher energy efficiency than the commercial GPU (NVIDIA A100-SXM4-80G). When compared with state-of-the-art FPGA accelerator of FlightLLM, it shows 10-24% better performance in terms of HBM bandwidth utilization, energy efficiency and LLM throughput.

I. INTRODUCTION

EdgeLLM targets the computation, memory-access, operator-format, and compilation challenges that hinder LLM deployment on edge devices. It combines heterogeneous CPU-FPGA execution with specialized hardware, unified data formats, and end-to-end compilation.

  • LLMs impose heavy computation and memory-access demands because billions of parameters must be processed in matrix operations.Source: §I · p. 1
  • Diverse Transformer operators require different data formats, causing reshaping and transposition overhead that reduces system efficiency and throughput.Source: §I · p. 3
  • Complex operator graphs require compilers that exploit parallelism, manage memory movement, and optimize transitions between hundreds or thousands of operators.Source: §I · p. 3
  • EdgeLLM assigns core computational operators to the FPGA and dynamic compilation to the CPU in a heterogeneous acceleration system.Source: §I · p. 3
  • The design combines mix-precision processing, group systolic architecture, unified operator data formats, and end-to-end compilation for CPU-FPGA deployment.Source: §I · p. 3

A. Transformers and LLMs

Transformers replaced recurrent and convolutional sequence processing with self-attention, enabling greater parallelization and long-range dependency modeling. Their success led to billion-parameter LLMs and broader multimodal applications.

  • Self-attention enabled Transformers to process sequence information with improved parallelization and long-range dependency capture.Source: A. Transformers and LLMs · p. 3
  • Transformer-based models expanded from language generation and understanding to multimodal tasks including image captioning and video understanding.Source: A. Transformers and LLMs · pp. 3–4

B. AI Chips for Transformers and LLMs

Existing AI chips provide high compute and memory bandwidth but often rely on power-intensive GPU-style systems, motivating FPGA-based edge acceleration. Prior FPGA work demonstrates promise, while FlightLLM leaves bandwidth-utilization headroom.

III. HARDWARE DESIGN

The hardware design combines CPU-FPGA control, HBM-backed high-bandwidth operators, DDR-backed nonlinear operators, and roofline-guided parallelism. Its architecture matches memory traffic and computation for FP16*INT4 FFN and FP16*FP16 MHA workloads.

A. Overall Architecture

  • The CPU controls the accelerator through PCIe and AXI interfaces, while HBM, DDR, and the LLM accelerator provide the system’s main architectural components.Source: A. Overall Architecture · p. 4
  • MatMUL and MHA use HBM for weight and KV-cache traffic, whereas normalization, rotary embedding, and nonlinear operators use DDR with customized DMA strategies.Source: A. Overall Architecture · p. 4
  • Both FFN and MHA are designed to operate at the roofline model’s most effective point.Source: Figure 3 · p. 5
  • The roofline model compares memory-bound and compute-bound regions using operational intensity on the horizontal axis and achievable performance on the vertical axis.Source: A. Overall Architecture · p. 5, Figure 3 · p. 5
  • The design sets 4096-way FP16*INT4 parallelism for FFN and 1024-way FP16*FP16 parallelism for MHA, matching 16384 bits per cycle of HBM bandwidth.Source: A. Overall Architecture · p. 5

B. Design of Mix-Precision Computing PEs

The design combines a grouped vector systolic array with a mix-precision processing element that supports the distinct multiplication modes required by FFN and MHA layers. The resulting unit improves error, area, frequency, and power characteristics over the two baseline adder-tree designs.

  • The grouped vector systolic array transfers inputs and weights row by row, reducing power consumption and circuit area relative to TPU-style or broadcast-style arrays.Source: B. Design of Mix-Precision Computing PEs · p. 5
  • The mix-precision unit supports FP16*INT4 for FFN layers and FP16*FP16 for MHA blocks, using shared bandwidth and time-division-multiplexed DSPs.It also multiplies the result by an FP16 scale value for block-level INT4 quantization.Source: B. Design of Mix-Precision Computing PEs · p. 5, Figure 4 · p. 6
  • The four-stage dataflow splits operands, multiplies mantissas, aligns and accumulates results, then converts and scales the output to FP16.The design preserves mantissa components and uses a 19-bit adder tree to balance resource use and computational accuracy.Source: B. Design of Mix-Precision Computing PEs · pp. 5–6
  • The proposed module records 0.047% error in FP16*INT4 mode and 0.0044% in FP16*FP16 mode across 100,000 random tests, outperforming both baselines in error.The comparison uses FP16 and customized FP20 intermediate formats as the two baseline schemes.Source: B. Design of Mix-Precision Computing PEs · p. 6, Table I · p. 7
  • The design reaches 1.11GHz at 28nm, uses 71,664 μm2 area, and reduces area by 33.2% versus baseline 1 and 49.1% versus baseline 2.Its power is only 2% above baseline 1 while remaining lower than baseline 2.Source: B. Design of Mix-Precision Computing PEs · p. 7

C. Further Optimization with Log-scale Sparsity

The accelerator combines log-scale structured sparsity, block quantization, and hybrid mask encoding to reduce weight storage and memory-access overhead while preserving hardware utilization. Its sparse design supports layer-specific sparsity and reports substantial effective-bit-width and performance improvements.

  • Log-scale structured sparsity uses 1/2, 1/4, and 1/8 patterns to reduce memory-access overhead while enabling 100% hardware utilization across a broad sparsity range.The approach uses a time-unrolled micro-architecture.Source: C. Further Optimization with Log-scale… · p. 7, Figure 5 · p. 7
  • The scheme symmetrically quantizes 128 adjacent parameters per scale and defines each eight-block group to contain at least N zero blocks.At 75% sparsity, each group contains at least six zero blocks and at most two non-zero blocks; scales, masks, and weights are stored in HBM.Source: C. Further Optimization with Log-scale… · pp. 7–8
  • The packaged sparse representation reaches 3.66× performance enhancement at 87.5% sparsity, where the total storage cost is 2304 bits per 2048-channel group.Address-in-block encoding becomes more efficient than one-hot encoding at this sparsity.Source: Figure 5 · p. 7, C. Further Optimization with Log-scale… · p. 8
  • Effective weight bit-widths are 4.125, 3.125, 1.875, and 1.125 bits, with corresponding performance enhancements of 1.32×, 2.2×, and 3.67× at different sparsity levels.The reported benefits are higher memory efficiency, higher computation speed, and better bandwidth optimization.Source: C. Further Optimization with Log-scale… · p. 8, Table II · p. 8
  • Mixed sparsity assigns different sparsity levels to operator layers while power-of-two block dimensions preserve 100% computational efficiency across those levels.Hybrid encoding uses one-hot masks at lower sparsity and address-in-block encoding at higher sparsity.Source: C. Further Optimization with Log-scale… · p. 8

IV. SOFTWARE DESIGN

This section introduces compilation strategies intended to execute the accelerator's hardware operators optimally.

  • The software design focuses on compilation strategies for optimally executing the hardware operators.Source: §IV · p. 8
  • The section frames operator execution as a compilation problem rather than only a hardware-design problem.Source: §IV · p. 8
  • The stated software objective is optimal execution of the accelerator's hardware operators.Source: §IV · p. 8

A. Unified Data Format based LLM Operator Graph

The unified data format keeps LLM operator inputs and outputs in compatible tensor shapes, enabling seamless operator transitions without data rearrangement. It also supports attention-specific dimensions, transpose execution, and efficient AXI burst transfers.

  • The optimized operator graph fuses one ChatGLM-6B block into 17 hardware steps while maintaining consistent input and output data structures.This consistency supports smooth transitions between operators.Source: A. Unified Data Format based LLM… · p. 8, Figure 6 · p. 9, Figure 7 · p. 9
  • A universal data-parallel scheme represents text and image activations with compatible tensorization, using [CH/Tout, token, Tout] and [CH/Tout, H, W, Tout].Head or batch dimensions can be added analogously when required.Source: A. Unified Data Format based LLM… · p. 9
  • The unified structure supports QKV formation, attention reshaping, and segmented continuous transpose execution without changing the data format.Contiguous [token, Tout] addresses help preserve access continuity for the transposed data.Source: A. Unified Data Format based LLM… · p. 9
  • Fixed AXI read and write widths of Tout*16 align with the smallest tensor package, enabling sequential burst transfers along width or token dimensions.Source: A. Unified Data Format based LLM… · p. 9

B. Software Architecture and LLM Inference Systems

EdgeLLM combines compiler-based deployment with runtime optimizations for CPU-FPGA LLM inference. Its software supports dynamic tokens, KVcache management, serialized instruction execution, and latency hiding.

B. Software Architecture and LLM Inference Systems

  • The compiler imports sparse, quantized LLMs and generates instructions, preprocessed weights, and runtime control code for deployment on the VCU128 platform.The surrounding inference framework uses the FPGA as the server and Python as the client for token interaction.Source: B. Software Architecture and LLM… · p. 9, Figure 8 · p. 10
  • The deployment flow is illustrated by FPGA compilation and inference-latency-hiding views, while the experimental setup also reports decode speed and zero-shot accuracy across four benchmarks.Source: Figure 8 · p. 10, Figure 9 · p. 10, Figure 10 · p. 10
  • MAX Token static-address allocation and DAG-based instruction expressions adapt hardware execution to dynamic token lengths while conserving KVcache space.Directly evaluable parameters are resolved at compile time; others become simplified runtime expressions.Source: B. Software Architecture and LLM… · p. 10
  • The compiler removes inactive data from earlier tokens after attention, reducing subsequent computation by retaining the actual offset of the last token.Source: B. Software Architecture and LLM… · p. 10
  • Pre-configured register mode streams serialized operator instructions from DDR to buffers, allowing host control updates to overlap accelerator execution.After the initial complete instruction update, later dynamic-control updates are hidden during the preceding model inference.Source: B. Software Architecture and LLM… · p. 10, Figure 9 · p. 10

V. EXPERIMENTAL RESULTS

Experiments deploy EdgeLLM on an AMD Xilinx VCU128 FPGA and evaluate its hardware configuration, operating frequencies, sparse-model accuracy, and decoding speed. The setup compares GLM-6B variants and also measures Qwen-7B.

A. Experiment Setup

  • The VCU128 implementation provides 1303K LUTs, 2607K FFs, 9024 DSPs, 2016 BRAMs, and 8GB HBM at 460GB/s, with CPU control over memory and accelerator registers.Source: A. Experiment Setup · p. 10
  • The matrix-multiplication operator runs at 280MHz through HBM, while other operators run at 140MHz; timing optimization could increase these frequencies.Source: A. Experiment Setup · p. 10
  • Sparse optimization increases GLM-6B speed by approximately 63% with a slight accuracy decrease, while the original model incurs almost no accuracy loss on the reported benchmarks.Source: A. Experiment Setup · p. 11
  • Qwen-7B decoding reaches 42.5 token/s to 69.4 token/s and is slower than GLM because it has fewer FFN parameters but highly shared MHA weight heads.Source: A. Experiment Setup · p. 11

B. Analysis on the Hardware Efficiency

EdgeLLM sustains high HBM utilization and strong LLM throughput and energy efficiency, while performance depends on memory type, sequence length, and operator characteristics. Its comparison with FlightLLM is favorable, but model-specific operators and temporal execution remain limitations.

B. Analysis on the Hardware Efficiency

VI. CONCLUSION

EdgeLLM combines mix-precision sparse hardware with universal data handling and end-to-end compilation for CPU-FPGA LLM deployment. On VCU128, it outperforms GPU and FlightLLM comparisons while targeting edge applications.

VI. CONCLUSION

  • EdgeLLM combines FP16/FP16 and FP16/INT4 mix-precision processing, group systolic architecture, and log-scale structured sparsity to improve LLM hardware efficiency.Source: §VI · p. 13
  • Its universal data format and end-to-end compiler map diverse LLM operators without data rearrangement onto a CPU-FPGA heterogeneous system.Source: §VI · p. 13
  • 1.91× higher throughput and 7.55× higher energy efficiency than an NVIDIA A100-SXM4-80G are reported, with about 10–24% higher performance than FlightLLM.Source: §VI · p. 13
Loading 2407.21325v2…