PaperlayerView exact revision on arXiv

Source-linked AI summary

DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation

Seongmin Hong, Seungjae Moon, Junsoo Kim, Sungjae Lee, Minsub Kim, Dongsoo Lee, Joo-Young Kim

arXiv:2209.10797v1eess.SYcs.ARcs.LGcs.SY

TL;DR

GPT text generation combines parallel context processing with sequential token generation, where GPUs degrade, while partial accelerators can leave GPT-2 operations to host systems. DFX combines multi-FPGA model parallelism with GPT-aware compute and dataflow; on four U280s, it reports performance, energy-efficiency, and cost-effectiveness gains over conventional multi-GPU appliances.

  • Problem

    GPT text generation combines parallel context processing with sequential single-token generation, where GPUs degrade, while partial accelerators leave GPT-2 operations to host systems.

    Source: A. Sequential Characteristic · p. 4, B. End-to-End Acceleration · p. 4

  • Method

    DFX is a multi-FPGA GPT appliance combining intra-layer model parallelism with token-oriented compute and HBM-aware tiling and dataflow for end-to-end inference.

    Source: §I · p. 2, B. Homogeneous Multi-FPGA Cluster · p. 5, B. Direct Memory Access · p. 7

  • Results

    5.58× performance, 3.99× energy-efficiency, and 8.21× cost-effectiveness improvements versus conventional multi-GPU appliances.

    Source: §IX · p. 13

  • Takeaways & Limitations

    DFX offers a reprogrammable FPGA path for evolving transformer operations and dimensions, with minimum redesign cost relative to an ASIC-based accelerator.

    Source: §I · p. 2

Abstract

from arXiv · show

Transformer is a deep learning language model widely used for natural language processing (NLP) services in datacenters. Among transformer models, Generative Pre-trained Transformer (GPT) has achieved remarkable performance in text generation, or natural language generation (NLG), which needs the processing of a large input context in the summarization stage, followed by the generation stage that produces a single word at a time. The conventional platforms such as GPU are specialized for the parallel processing of large inputs in the summarization stage, but their performance significantly degrades in the generation stage due to its sequential characteristic. Therefore, an efficient hardware platform is required to address the high latency caused by the sequential characteristic of text generation. In this paper, we present DFX, a multi-FPGA acceleration appliance that executes GPT-2 model inference end-to-end with low latency and high throughput in both summarization and generation stages. DFX uses model parallelism and optimized dataflow that is model-and-hardware-aware for fast simultaneous workload execution among devices. Its compute cores operate on custom instructions and provide GPT-2 operations end-to-end. We implement the proposed hardware architecture on four Xilinx Alveo U280 FPGAs and utilize all of the channels of the high bandwidth memory (HBM) and the maximum number of compute resources for high hardware efficiency. DFX achieves 5.58x speedup and 3.99x energy efficiency over four NVIDIA V100 GPUs on the modern GPT-2 model. DFX is also 8.21x more cost-effective than the GPU appliance, suggesting that it is a promising solution for text generation workloads in cloud datacenters.

I. INTRODUCTION

DFX addresses the generation-stage bottleneck on GPUs with a multi-FPGA appliance designed for end-to-end GPT inference. Its design combines programmable compute, GPT-aware memory dataflow, and model parallelism to target low latency and high throughput.

I. INTRODUCTION

  • GPU performance degrades in generation because its sequential token processing underutilizes massively parallel compute units.GPUs can process input tokens simultaneously during summarization, but generation feeds each output token into the next iteration.Source: §I · pp. 1–2
  • Prior transformer accelerators often focus on attention, leaving other GPT operations to host or extra compute modules and adding processing overhead.The paper argues that practical language services need a unified programmable architecture supporting the whole GPT operation sequence.Source: §I · p. 2
  • DFX is a multi-FPGA appliance for end-to-end inference across differently sized GPT models, with compute optimized for single-token processing.The design also uses GPT-aware tiling and dataflow to use HBM bandwidth, and model parallelism to distribute workloads across devices with minimal synchronization.Source: §I · p. 2
  • DFX applies model parallelism across FPGAs, evenly distributing model parameters to reduce synchronization and enable parallel computation.Its compute core uses custom instructions, while GPT-aware tiling and dataflow target high HBM bandwidth use, low latency, and high throughput.Source: §I · p. 2
  • The appliance is presented as having lower upfront and operating costs while delivering multiple-times better performance and efficiency than a conventional GPU platform.The authors describe it as promising for text-generation workloads in datacenters.Source: §I · p. 2

II. BACKGROUND

GPT-2 uses a decoder-only transformer to generate tokens from context. Its summarization stage processes the full context, while generation feeds each prior output back as the next input.

A. GPT Language Model

  • GPT uses a decoder-only transformer, with token embedding converting input words to vectors and the LM head selecting the highest-probability next token.The embedding combines word-token and word-position vectors; the LM head applies a matrix multiplication with the transpose of WTE and selects a token using softmax.Source: A. GPT Language Model · pp. 2–3
  • Each of GPT-2’s N decoder layers combines self-attention, a two-layer GELU feed-forward network, layer normalization, and residual operations.The number of decoder layers N depends on model size; multi-head attention executes independent matrix operations in parallel.Source: A. GPT Language Model · p. 3
  • Summarization processes n×emb context and creates the first token plus Key and Value context; generation feeds 1×emb tokens and appends Key and Value rows iteratively.The generated tokens are detokenized together to form the output text.Source: A. GPT Language Model · p. 3
  • Workload token ratios vary, and datacenter language services tend to generate more output tokens than input context, changing computation across the two stages.The amount of computation in each stage depends on the length of the input context and output sequence.Source: A. GPT Language Model · p. 3

B. Parallelism in Deep Learning

Data and model parallelism distribute work differently: model parallelism can reduce operation time but may require synchronization. GPT-2 generation is sequential, making GPU latency especially sensitive to output length.

B. Parallelism in Deep Learning

  • Data parallelism splits input batches across workers, making it unsuitable for inference settings that commonly use a single or non-batched input.Source: B. Parallelism in Deep Learning · p. 3
  • Model parallelism partitions parameters across workers; intra-layer parallelism divides operations such as matrix multiplication to reduce execution time, while pipelining targets throughput rather than latency.Intra-layer parallelism may require synchronization, so performance depends on synchronization and device counts.Source: B. Parallelism in Deep Learning · pp. 3–4

A. Sequential Characteristic

  • 75.45ms per additional output token versus 0.02ms per additional input token on GPU, on average, for GPT-2 1.5B.The figure varies input tokens leftward and output tokens rightward.Source: A. Sequential Characteristic · p. 4, Figure 3 · p. 4
  • Datacenters may avoid fully batching different users because gathering their inputs increases latency; unfilled batch slots also reduce utilization linearly.This motivates low-latency processing of a single input token.Source: A. Sequential Characteristic · p. 4

B. End-to-End Acceleration

The section argues for accelerating all GPT-2 operations, distributing large models across devices, and using a multi-FPGA appliance architecture. GPU execution spends a disproportionate share of time on layer normalization and residual operations.

B. End-to-End Acceleration

  • Prior accelerators often target attention, leaving other GPT-2 operations to the host, where repeated PCIe transfers can become a performance bottleneck.GPT-2 also includes token embedding, layer normalization, residual, and LM head operations.Source: B. End-to-End Acceleration · p. 4
  • Layer normalization and residual account for 22.8% of GPU latency despite only 0.11% of raw operations, motivating acceleration of all GPT-2 operations.Figure 4 presents the latency and operation breakdown.Source: B. End-to-End Acceleration · p. 4, Figure 4 · p. 4

C. Parallel Computing

  • Large GPT-2 models require multiple devices for model parallelism and an efficient network to share computation while limiting added latency.Source: C. Parallel Computing · p. 4

A. Architecture Overview

  • A DFX workload uses one CPU and a homogeneous cluster of four FPGAs, with one compute core on each FPGA; the cluster connects to the host via PCIe Gen 3 and uses a ring network between FPGAs.The host link transfers at 16 GB/s and FPGA links at 100 Gb/s; four-FPGA clusters can scale within an appliance subject to cost.Source: A. Architecture Overview · pp. 4–5

B. Homogeneous Multi-FPGA Cluster

DFX partitions GPT-2 across a homogeneous FPGA cluster and synchronizes partial results between devices. Its custom instruction set supports end-to-end inference operations, while memory placement and instruction scheduling target efficient execution.

B. Homogeneous Multi-FPGA Cluster

  • Intra-layer parallelism partitions attention weights by head and fully connected weights by columns, letting FPGA cores process assigned parameters in parallel.Cores exchange output subvectors over a ring; synchronization occurs four times per decoder layer.Source: B. Homogeneous Multi-FPGA Cluster · p. 5, Figure 6 · p. 5
  • DFX stores frequently accessed partitioned weight matrices in HBM and places tokens, bias vectors, and other less frequently accessed parameters in DDR.The FPGA has 8 GB HBM at 460 GB/s theoretical peak bandwidth and 32 GB DDR at 38 GB/s; model parameters use FP16.Source: B. Homogeneous Multi-FPGA Cluster · p. 5

C. Instruction Set Architecture

  • The custom DFX ISA supports end-to-end GPT-2 inference with compute, DMA, and router instruction types for processing and data movement.Instructions can chain dependencies while independent compute, DMA, and router work proceeds in parallel, helping keep memory and communication bandwidth in use.Source: C. Instruction Set Architecture · pp. 5–6
  • The decoder-layer algorithm sequences normalization, self-attention, feed-forward operations, and residual additions, with synchronization between distributed computations.Source: C. Instruction Set Architecture · p. 6
  • Matrix instructions implement operations including Conv1D, masked matrix multiplication, and matrix multiplication, while vector instructions compose LayerNorm and Softmax from basic operations.MaskedMM applies a future-context mask; LayerNorm and Softmax use combinations of vector instructions.Source: C. Instruction Set Architecture · p. 6

V. MICROARCHITECTURE

The compute core is designed for sequential, non-batched text generation and combines matrix and vector processing units. Its control unit coordinates dataflow and instruction execution across the core.

V. MICROARCHITECTURE

  • The compute core centers on matrix and vector processing units designed for sequential workloads with non-batched input.Figure 7 depicts the compute-core microarchitecture.Source: §V · p. 6, Figure 7 · p. 7

A. Control Unit

  • The control unit tracks unit states and arbitrates which modules run, using a controller, scheduler, and scoreboard.Source: A. Control Unit · pp. 6–7
  • The controller configures core identity, model partition, decoder layers, and token counts; the scheduler checks resource status, while the scoreboard stalls instructions with overlapping source and destination addresses.Source: A. Control Unit · p. 7

B. Direct Memory Access

DFX configures DMA to use HBM bandwidth and combines model-aware tiling, zigzag traversal, and write-time transposition to balance throughput with hardware constraints.

B. Direct Memory Access

  • The DMA connects to all 32 HBM channels, handling 32×512 bits per cycle and streaming tiled weights, keys, and values to the matrix-processing units.It also reads input tokens, biases, and embeddings from DDR and writes the final output token there.Source: B. Direct Memory Access · p. 7
  • The model-and-hardware-aware tiling scheme selects tile dimensions, lane count, and weight-loading direction for large matrices and matrix-vector multiplication.For FP16 data, the design rearranges the HBM read dimension into d × l × BWdata weight bits.Source: B. Direct Memory Access · p. 7
  • d=64 and l=16 provide near-best multi-head-attention performance with negligible difference from d=16 or d=32, while requiring the least MPU resources.The selected configuration avoids underutilization for attention heads around 64 dimensions and limits lane-dependent resource growth.Source: B. Direct Memory Access · pp. 7–8, Figure 8 · p. 8
  • A zigzag d×d tile traversal balances input reuse against the on-chip buffers required for partial sums.Horizontal traversal maximizes reuse but needs many buffers; vertical traversal uses one buffer but increases register-file access and reduces throughput.Source: B. Direct Memory Access · p. 8
  • DFX transposes Value tiles as they are written to off-chip memory and reorders computation so the transpose latency is hidden while Query and Key are generated.This avoids transposing the full, high-capacity Value matrix in on-chip memory.Source: B. Direct Memory Access · p. 8

C. Processing Units

DFX’s matrix and vector units execute GPT-2’s core operations through pipelined FP16 functional units, while special-function units handle nonlinear operations and bypasses reduce unnecessary cycles.

C. Processing Units

  • The matrix function unit performs matrix-vector multiplication with d-dimensional tree MACs across l parallel lanes, processing d×l multiplications in parallel.For d=64 and l=16, the MFU uses 3,072 DSPs.Source: C. Processing Units · pp. 8–9
  • The vector function unit supports element-wise vector addition, subtraction, and multiplication, plus exponential operations, with a bypass path that lets load and store instructions take one cycle.A scoreboard handles data hazards in the asynchronous dataflow.Source: C. Processing Units · p. 9
  • Special-function units process nonlinear operations after the matrix and vector units, including masking, GELU, vectorization, and reduce-max operations.The matrix-side GELU uses a lookup table with linear approximation over [−8,8], sampled at 2,048 inputs with reported zero mean squared error in half precision.Source: C. Processing Units · p. 9
  • Bypasses let operations skip unneeded hardware without cycle penalties, and alternating matrix and vector instructions supports sequential GPT-2 generation.The special-function units also use bypasses for operations that do not require their hardware.Source: C. Processing Units · p. 9

D. Register File Manager

The register-file manager and lightweight router coordinate operand delivery and inter-device synchronization, while the appliance maps DFX cores across four U280 FPGAs to meet routing constraints and use HBM bandwidth.

D. Register File Manager

  • The matrix operand collector generates microcodes at runtime, distributes inputs, weights, and biases to the MPU, and uses double buffering to reduce latency and sustain throughput.The vector operand collector generates VPU microcodes and can access vector and scalar register files, DMA buffers, and router buffers.Source: D. Register File Manager · pp. 9–10

E. Router

  • The lightweight router synchronizes register-file data across the ring because model-parallel cores compute only portions of output rows needed by later operations.It transfers 64 × 16-bit output vectors peer-to-peer and reorders them identically at each core.Source: E. Router · p. 10

VI. APPLIANCE IMPLEMENTATION

  • The prototype uses four U280 cards, one DFX core per device; splitting kernels across SLRs and placing DMA and MPU near HBM mitigates routing congestion and achieves maximum bandwidth utilization.The design runs at 200 MHz kernel and 410 MHz memory-interface frequencies, using 39.15% of DSP resources.Source: §VI · p. 10, Figure 13 · p. 10

VII. EVALUATION

The evaluation compares DFX with a four-V100 GPU appliance across GPT-2 models and reports closely matched accuracy on three word-prediction datasets. DFX’s accuracy differences are attributed to the GELU approximation difference between the systems.

VII. EVALUATION

  • The evaluation compares DFX with a four-NVIDIA-V100 GPU appliance and also runs cloud TPU; it tests 345M, 774M, and 1.5B GPT-2 models.The 1.5B model’s attention-head count is adjusted from 25 to 24 to make parallelization feasible on both hardware platforms.Source: §VII · p. 11

A. Inference Accuracy

  • Accuracy is assessed on WSC, CBT-CN, and CBT-NE, datasets that predict a word from the given context.The comparison uses the 345M model and the baseline V100 GPU.Source: A. Inference Accuracy · p. 11
  • DFX has no loss, 0.3% loss, and 0.15% gain in accuracy for WSC, CBT-CN, and CBT-NE, respectively, compared with the baseline GPU.Both systems use FP16; the passage attributes the differences to the subtle approximation difference in GELU.Source: A. Inference Accuracy · p. 11

B. Performance Analysis

Across the evaluated GPT-2 sizes, DFX reports lower latency than an equivalent-GPU appliance, alongside higher throughput and energy efficiency on the 1.5B model. Its performance analysis also compares accelerator behavior across generation stages and FPGA counts.

B. Performance Analysis

  • Evaluation covers 345M, 774M, and 1.5B GPT-2 models with 32–128 input tokens and 1–256 output tokens; layer normalization and residual are not parallelized.The paper also reports a marginal throughput drop with each additional FPGA due to synchronization.Source: B. Performance Analysis · pp. 11, 13
  • 5.58× average speedup on the 1.5B model, 4.46× on 774M, and 3.20× on 345M versus the equivalent-GPU appliance; the 1.5B 32:256 workload reaches 10.03×.The comparison uses equivalent accelerator counts for each model size.Source: B. Performance Analysis · p. 11
  • Self-attention and feed-forward consume 72.6% of four-FPGA 1.5B latency, while synchronization accounts for 17.3%.The paper attributes the synchronization share to faster execution of other operations, alongside 5.58× lower overall latency than the GPU appliance.Source: B. Performance Analysis · p. 12, Figure 15 · p. 12
  • 3.78× throughput and 3.99× energy efficiency versus the GPU appliance on the 1.5B model.Throughput is output tokens divided by text-generation latency.Source: B. Performance Analysis · p. 12, Figure 16 · p. 12
  • 184.1 GFLOPS average across both stages for one DFX FPGA, while GPU and TPU throughput falls from 1632.1 to 40.6 and 674.5 to 8.2 GFLOPS, respectively.The comparison uses the 345M model with 64:64 tokens; Figure 17 compares GPU, TPU, and one DFX FPGA.Source: B. Performance Analysis · p. 12, Figure 17 · p. 12
  • 93.10, 146.25, and 207.56 tokens/sec for 1, 2, and 4 FPGAs, respectively, with performance increasing at a reported rate of 1.5.Figure 18 reports scalability on the 345M model with 64:64 tokens.Source: B. Performance Analysis · p. 12, Figure 18 · p. 12

C. Cost Analysis

DFX is reported as more cost-effective than the GPU appliance under the paper’s stated four-device comparison. The paper situates this result among datacenter accelerators and concludes with prototype-level performance, energy-efficiency, and cost-effectiveness improvements.

C. Cost Analysis

  • DFX is 8.21× more cost-effective than the GPU appliance, with four-device upfront cost $14,652 lower; the comparison excludes non-accelerator components and uses the 1.5B model at 64:64 tokens.The selected token ratio is described as representative of chatbot service.Source: C. Cost Analysis · p. 13

VIII. RELATED WORK

  • Several prior transformer accelerators target attention rather than end-to-end inference; DFX instead emphasizes model-parallel dataflow for end-to-end text-generation workloads.The cited examples include A3, GOBO, SpAtten, EdgeBERT, and ELSA.Source: §VIII · p. 13
  • Few domain-specific accelerators are designed at datacenter scale; the paper notes that Brainwave lacks high-bandwidth off-chip memory for memory-intensive GPT inference.It also states that prior datacenter architectures supporting transformers lack text-generation-specific optimizations.Source: §VIII · p. 13

IX. CONCLUSION

  • The four-U280 prototype reports 5.58× performance, 3.99× energy-efficiency, and 8.21× cost-effectiveness improvements over conventional multi-GPU appliances.Source: §IX · p. 13
Loading 2209.10797v1…