PaperlayerView exact revision on arXiv

Source-linked AI summary

A Tensor-Train Decomposition based Compression of LLMs on Group Vector Systolic Accelerator

Sixiao Huang, Tintin Wang, Ang Li, Ao Shen, Kai Li, Keyao Jiang, Mingqiang Huang, Hao Yu

arXiv:2501.19135v1cs.AR

TL;DR

LLM deployment on resource-constrained hardware is challenged by storage and computation demands, especially the memory access demands of linear layers. The paper applies TTD compression to LLM linear layers and implements inference on an FPGA accelerator. The accelerator reduces first token delay for both evaluated models compared with a baseline without TTD inference.

  • Problem

    LLM deployment on edge devices is constrained by limited storage and computing resources, while linear-layer memory access limits accelerator performance.

    Source: §I · p. 1

  • Method

    The paper applies TTD compression to LLM linear layers and maps TTD inference to an FPGA group vector systolic array with DSP-shared parallel vector processing elements.

    Source: §I · p. 1, §IV · p. 4

  • Results

    1.45× and 1.57× reduction in first token delay for ChatGLM3-6B and LLaMA2-7B, respectively, versus an accelerator baseline without TTD inference.

    Source: C. Hardware Performance Evaluation · p. 6

  • Takeaways & Limitations

    The implementation demonstrates FPGA deployment of TTD-compressed ChatGLM3-6B and LLaMA2-7B models.

    Source: §I · p. 1

Abstract

from arXiv · show

Large language models (LLMs) are both storage-intensive and computation-intensive, posing significant challenges when deployed on resource-constrained hardware. As linear layers in LLMs are mainly resource consuming parts, this paper develops a tensor-train decomposition (TTD) for LLMs with a further hardware implementation on FPGA. TTD compression is applied to the linear layers in ChatGLM3-6B and LLaMA2-7B models with compression ratios (CRs) for the whole network 1.94$\times$ and 1.60$\times$, respectively. The compressed LLMs are further implemented on FPGA hardware within a highly efficient group vector systolic array (GVSA) architecture, which has DSP-shared parallel vector PEs for TTD inference, as well as optimized data communication in the accelerator. Experimental results show that the corresponding TTD based LLM accelerator implemented on FPGA achieves 1.45$\times$ and 1.57$\times$ reduction in first token delay for ChatGLM3-6B and LLaMA2-7B models, respectively.

I. INTRODUCTION

LLM edge deployment is challenged by storage limits and linear-layer memory access; this work applies TTD compression to LLM linear layers and implements their inference on FPGA.

A. Linear Layers in LLMs

  • Linear layers dominate LLM computation, while their low ratio of operations to weights makes memory access a performance limit for accelerators.The paper identifies compression of these layers as important for efficient edge deployment.Source: §I · p. 1, A. Linear Layers in LLMs · p. 2

I. INTRODUCTION

  • Because TTD changes linear-layer computational structure, efficient acceleration requires hardware redesign.Source: §I · p. 1
  • TTD compression of linear layers gives whole-network compression ratios of 1.94× for ChatGLM3-6B and 1.60× for LLaMA2-7B.Source: §I · p. 1
  • 1.45× and 1.57× reduction in first token delay for ChatGLM3-6B and LLaMA2-7B, respectively, versus the baseline without TTD inference.Source: §I · p. 2

B. Tensorization

Linear-layer weight matrices are tensorized without changing their element count so TTD can be applied; TTD then represents the higher-dimensional tensor as ordered products of lower-dimensional cores.

B. Tensorization

  • Tensorization reshapes a linear-layer weight matrix into a higher-dimensional tensor while preserving the number of elements.The tensor dimensions satisfy M1M2 = product of the v_i values.Source: B. Tensorization · p. 2

C. Tensor-Train Decomposition

D. TTD Compression and Inference

The method tensorizes a linear layer’s input and weight matrix, then decomposes the weight tensor into cores for compressed inference. Inference applies those cores to the tensorized input and sums over input dimensions to produce the output.

D. TTD Compression and Inference

  • Rank affects both the compression ratio and the similarity between the matrix reconstructed from the cores and the original weight matrix.The passage states that a higher rank gives a lower compression ratio; its description of the corresponding similarity change is incomplete.Source: D. TTD Compression and Inference · p. 2
  • Algorithm 1 factorizes the input and output dimensions, tensorizes x and W, and applies TTD to produce the input tensor X and weight cores G1…Gd.Its inputs include x, W, the TTD dimension d, and rank r.Source: D. TTD Compression and Inference · p. 3
  • TTD inference multiplies the tensorized input by the weight cores and sums across input dimensions to produce the tensorized output Y.Each Gk is a core of the weight tensor, and Y is the high-dimensional output format.Source: D. TTD Compression and Inference · p. 3

A. Operator Fusion of LLMs Inference

Operator fusion groups adjacent inference operations to reduce redundant transfers between off-chip and on-chip memory. The accelerator configures TTD linear, regular linear, and nonlinear operations, with serialized instructions chaining them in the execution order shown for the two models.

A. Operator Fusion of LLMs Inference

  • Operator fusion merges adjacent operations to reduce redundant off-chip/on-chip data transfers and improve throughput.The accelerator configures TTD linear, regular linear, and nonlinear operations.Source: A. Operator Fusion of LLMs Inference · p. 3
  • The TTD linear operation integrates batch normalization and residual operations, while the nonlinear operation handles layer normalization, embedding, activation, and softmax.The regular linear operation handles ordinary linear layers and related operations.Source: A. Operator Fusion of LLMs Inference · p. 3
  • Serialized compiler instructions configure operations and data addresses in HBM/DDR, chaining the network’s operators in the execution order shown for ChatGLM3-6B and LLaMA2-7B.Fig. 4 depicts the operator execution order for inference of TTD-compressed LLMs.Source: A. Operator Fusion of LLMs Inference · p. 3, Figure 4 · p. 3

B. GVSA Architecture

GVSA uses parallel vector processing elements for matrix computation. For TTD inference, its staged data mapping reuses partial computations and hides intermediate reordering in ping-pong buffer transfers.

B. GVSA Architecture

  • GVSA uses a row-by-row systolic style with Tout PEs, each performing Tin MAC operations, and loads one weight vector per cycle.The loaded weight vector remains stationary in the PEs for Tout cycles while feature vectors pass through groups of PEs.Source: B. GVSA Architecture · p. 3, Figure 5 · p. 3

C. TTD Inference Data Mapping on GVSA

  • Reusing cores across outputs with shared leading output dimensions avoids redundant partial products and memory access in TTD inference.The passage’s example shares G1, G2, and G3 when j1, j2, and j3 match, while G4 differs.Source: C. TTD Inference Data Mapping on GVSA · p. 4
  • The computation proceeds from input dimension i1 through the final input dimension, multiplying each core by the preceding partial result and summing over ik.P0 is X and Pd is Y; intermediate Pk is reordered for the next stage.Source: C. TTD Inference Data Mapping on GVSA · p. 4
  • Stage 1 reads X and G1 from the input buffer and writes P1 to ping-pong buffers; Stage 2 reorders P1 to match GVSA’s Tin-parallel input format.The staged core and feature mappings and operation stages are depicted in Fig. 6.Source: C. TTD Inference Data Mapping on GVSA · p. 4, Figure 6 · p. 4
  • Ping-pong buffer writes and reads hide inter-stage reordering, avoiding both waiting time and a separate reordering unit.After the remaining stages, results pass through BN and Res modules and are written to external DDR by DMA.Source: C. TTD Inference Data Mapping on GVSA · p. 4

IV. FPGA IMPLEMENTATION

The FPGA implementation maps TTD linear operations onto a group vector systolic array with DSP-shared parallel vector processing elements. Its data arrangement supports two FP16×INT4 products per DSP, while a pipeline handles exponent alignment and accumulation.

IV. FPGA IMPLEMENTATION

  • Fig. 7 depicts the DSP-shared GVSA vector PEs, the DSP design on Xilinx, and the DSP input arrangement.The implementation architecture also includes DMA, input and ping-pong buffers, a GVSA, and a control unit.Source: §IV · p. 4, Figure 7 · p. 5
  • DSP-shared vector PEs arrange inputs so each DSP produces two 16-bit multiplication results from FP16 and INT4 operands.The approach uses the DSP input width to multiply a two’s-complement mantissa with two INT4 weights.Source: §IV · pp. 4–5
  • One feature vector and two weight vectors enter a pipeline that separates FP16 components, aligns exponents, accumulates shifted products, and applies the input scale.The exponent comparison determines the shift amount for the adder trees.Source: §IV · p. 5

A. Experiment Setup

The accelerator was synthesized for an AMD Alveo V80 and evaluated on TTD-compressed ChatGLM3-6B and LLaMA2-7B. The deployment combines selected TTD-compressed, INT4-quantized linear layers with other quantized network components and emulates operator execution in sequence.

A. Experiment Setup

  • The accelerator was implemented with Vivado 2024.1 on an AMD Alveo V80 FPGA, which integrates 32GB of HBM.The platform has 12574K LUTs, 5148K FFs, 3741 BRAMs, and 10848 DSPs.Source: A. Experiment Setup · p. 5
  • The benchmarks were TTD-compressed ChatGLM3-6B and LLaMA2-7B, with selected linear layers excluding Attention(Q, K, V) receiving TTD compression and INT4 quantization.The remaining parts use INT4 quantization only, and the blocks are concatenated with quantized output layers to form the deployed networks.Source: A. Experiment Setup · p. 5
  • Inference operators were deployed and executed sequentially in the Fig. 4 operator-chaining order to emulate the accelerator system on V80.The setup is described as similar to the FlightLLM experiment setup.Source: A. Experiment Setup · p. 5

B. Network Performance Evaluation

TTD compression reduces whole-network size for both evaluated models, with reported benchmark quality changes. Against an accelerator without TTD inference, the FPGA system reduces first-token delay; the evaluation also reports throughput comparisons and decoding-stage trends.

B. Network Performance Evaluation

C. Hardware Performance Evaluation

  • Fig. 9(a) compares decoding speed across token counts, and the TTD accelerator performs better than the baseline while KV caching mitigates the speed decline for longer outputs.The figure also includes normalized throughput comparisons across hardware platforms.Source: C. Hardware Performance Evaluation · p. 6, Figure 9 · p. 6
  • 1.45× and 1.57× reduction in first token delay was achieved for ChatGLM3-6B and LLaMA2-7B, respectively, versus the accelerator without TTD inference.MLP inference speedups were 3.22× and 3.88×, and single-block speedups were 2.19× and 1.78×, respectively.Source: Figure 8 · p. 6, C. Hardware Performance Evaluation · p. 6
  • 27% and 19% higher throughput than FlightLLM on U280 was achieved by the TTD accelerator; throughput was also 59% and 49% higher than the A100 GPU.The passage attributes the FlightLLM comparison to TTD compression, optimized TTD data flow, and GVSA.Source: C. Hardware Performance Evaluation · p. 6, Figure 9 · p. 6

VI. CONCLUSION

The paper implements TTD-compressed LLMs on FPGA, mapping TTD inference to a GVSA with DSP-shared parallel vector PEs. The accelerator reports speedups in MLP and single-block inference and lower first-token delay for both evaluated models.

VI. CONCLUSION

  • The implementation deploys TTD-compressed LLM inference on FPGA using a highly efficient GVSA with DSP-shared parallel vector PEs.Source: §VI · p. 6
  • 3.22× and 3.88× MLP inference speedups were reported for ChatGLM3-6B and LLaMA2-7B, respectively.Source: §VI · p. 6
  • 1.45× and 1.57× reduction in first token delay was reported for ChatGLM3-6B and LLaMA2-7B, respectively.Source: §VI · p. 6

Related papers

Loading 2501.19135v1…