PaperlayerView exact revision on arXiv

Source-linked AI summary

Recent Developments in Transformer Inference Deployment on FPGA Platforms: A Survey

Arjan Blankestijn, Uraz Odyurt, Amirreza Yousefzadeh

arXiv:2609.01212v1cs.LGcs.AR

TL;DR

Transformer inference on FPGAs faces substantial computational and memory demands, while deployment methodology and tooling remain incomplete. The paper systematically reviews recent 2024–May 2025 research, extracts a taxonomy of implementation and optimisation choices, and finds no absolute preference across examined design dimensions. Its conclusions also highlight comparison difficulties caused by non-standardised benchmarking and inconsistent reporting.

  • Problem

    Transformer inference has substantial computational and memory demands, while FPGA deployment methodology and tooling remain incomplete for real-time and resource-constrained use.

  • Method

    The paper conducts a systematic literature review of recent Transformer inference deployment on FPGAs and extracts a taxonomy of recurring implementation and optimisation choices.

  • Results

    The review finds no absolute preference across abstraction level, memory organisation, pipelining, or quantisation, with choices remaining implementation-specific and application-driven.

  • Takeaways & Limitations

    FPGA flexibility supports diverse Transformer inference designs, but standardised benchmarking and complete architectural reporting are needed for direct comparisons.

  • Takeaways & Limitations

    Inconsistent accuracy baselines make robust comparative analysis of innovation impact impractical.

Abstract

from arXiv · show

With the rapid and continuous growth in the incorporation of machine learning models based on the Transformer architecture, capable deployment is in high demand. In this context, capable deployment refers to operational performance aspects, e.g., throughput and latency, as well as efficiency aspects, e.g., energy consumption. When it comes to the task of inference using such models, purpose-built hardware accelerators provide a lucrative alternative to common deployment choices, such as Central Processing Units (CPUs) and Graphics Processing Units (GPUs). The Field Programmable Gate Array (FPGA) platforms category is an example of such alternative accelerators, promising implementation flexibility, energy efficiency, improved latency and suitability for on-site deployment. We investigate the most recent advances, trends, and design choices for Transformer inference on FPGA platforms. We perform a systematic literature review, extracting and delving into preferred techniques for implementation and optimisation. This study and the provided taxonomy of topics could act as a guide for researchers from the academia and industry alike.

1 Introduction

Transformers offer powerful dependency modeling and parallel computation, but their inference workloads impose substantial computational and memory demands. This survey addresses the incomplete methodology and tooling for deploying Transformer inference on FPGAs through a recent systematic review.

  • Motivation: FPGAs provide reconfigurability and custom hardware-level optimisation for latency, throughput, power, size, and on-site deployment requirements.These properties make FPGA platforms an alternative to CPU and GPU accelerators for machine-learning inference.
  • Motivation: Transformer models capture distant dependencies and parallelise computation, but inference has substantially higher computational and memory demands than established architectures.The challenge is especially pronounced for real-time applications and resource-constrained environments.
  • Research focus: Deployment methodology and tooling for Transformer inference on FPGAs remain incomplete relative to those for older, established machine-learning architectures.The survey therefore examines performance indicators, improvement strategies, implementation and resource-management choices, optimisation techniques, and tooling limitations.
  • Scope: The review focuses on inference research published from 2024 through May 2025 rather than providing a broad historical survey.Its purpose is to capture rapidly evolving implementation practices, optimisation strategies, and target workloads while complementing earlier reviews.
  • Structure: The paper proceeds from background and methodology to taxonomy, storage, tooling, compression, IP design, performance analysis, and conclusions.These topics are organised across Sections 2–10.

2 Background and motivation

Transformer inference on FPGAs is motivated by rising research activity and by distinct architectural, workload, and hardware constraints. The survey frames recent work as a focused update and organises the background around Transformer variants and FPGA capabilities.

  • Survey motivation: Research on Transformer inference on FPGAs has increased sharply, motivating a focused examination of emerging trends and recent conclusions.The survey targets the 2024–May 2025 period, which earlier broad surveys did not cover.
  • Survey positioning: The survey uses a systematic, reproducible selection methodology and is intended as a recent update rather than a replacement for broader prior surveys.It covers FPGA-based Transformer inference articles without restricting analysis to specific use-case categories.
  • Transformer architecture: Transformers use encoder–decoder designs built from attention mechanisms, while many modern applications use encoder-only or decoder-only configurations.Encoder layers combine multi-head self-attention with feed-forward networks; decoder layers add masked self-attention and cross-attention.
  • Common Transformer model types: Vision Transformers process image patches and are constrained mainly by quadratic self-attention complexity and activation memory.High-resolution inputs increase the cost associated with attention maps and intermediate activations.
  • Common Transformer model types: Large Language Model inference is primarily memory-bound because weights stream from off-chip memory, while autoregressive decoding limits parallelism and KV caches add memory demands.Thus, ViTs and LLMs require different FPGA-oriented optimisation strategies.
  • FPGA technology: FPGAs combine programmable logic and hardened DSP, BRAM, URAM, and memory-interface resources for workload-specific parallel computation and storage.Their bespoke dataflows and memory hierarchies support streaming, systolic, pipelined, tiling, caching, and weight-buffering strategies.

3 Survey method and pruning protocols

The survey collects literature from three repositories using a standard query, then applies reproducible filtering and relevance ranking to focus on recent Transformer inference research. The staged process reduces the initial collection to the most relevant articles for review.

  • Collection: IEEE Xplore, Scopus, and arXiv were selected to balance coverage, reproducibility, and accessibility across FPGA research, broad indexing, and recent preprints.ArXiv collection uses its API dynamically.
  • Collection: A standard query combining transformer, fpga, and inference-related terms returned 411 articles from the three repositories.The query was applied consistently across the sources.
  • Pruning: Abstract-focus filtering retained 113 articles, survey removal retained 108, and training-focused article removal retained 96.These filters enforce Transformer relevance, exclude surveys, and preserve the inference focus.
  • Ranking: Relevance ranking assigns keyword-frequency points by subtopic and uses 25 points as the threshold, producing a top set of 57 articles.The most relevant articles from this set are processed by the authors.

4 Taxonomy of Transformer inference on FPGAs

The survey extracts a taxonomy of Transformer inference approaches on FPGAs from recurring implementation and optimisation themes. The taxonomy provides a conceptual hierarchy for organising the reviewed literature, while remaining non-exhaustive.

  • Taxonomy scope: The taxonomy is derived from the reviewed literature and summarises approaches and techniques for deploying Transformer models on FPGAs for inference.It is presented as Figure 4.
  • Taxonomy structure: Implementation and Optimisation form the taxonomy’s two main aspects and organise the remainder of the survey.The hierarchy reflects recurring themes and design choices in the collected articles.
  • Taxonomy scope: The taxonomy is high-level and not exhaustive, but it represents dominant implementation and optimisation techniques reported in the surveyed literature.The authors also use knowledge of similar implementations to make the structure more meaningful.
  • Review foundation: The article-collection and pruning workflow provides the selection context underlying the extracted taxonomy.Figure 3 records the protocols and resulting article counts at each stage.

5 Model weights storage

Transformer FPGA accelerators trade inference performance against model flexibility through their choice of on-chip or off-chip weight storage. Persistent on-chip storage generally improves throughput and latency, whereas off-chip storage supports broader model coverage but introduces memory-transfer costs.

  • Storage approaches: Off-chip storage supports more models and model sizes because capacity is not limited by available on-chip memory.Runtime-adaptive architectures can load inputs, weights, and biases through separate units.
  • Storage approaches: On-chip storage minimises memory transfers and can enable extensive pipelining, but limited FPGA memory prevents storing all weights and activations for some models.HG-PIPE exemplifies pipelining enabled by keeping weights and activations on-chip.
  • Storage approaches: On-chip buffers can load different model parameters before inference, providing runtime configurability without requiring persistent on-chip storage for one fixed model.ME-ViT loads parameters from off-chip memory into on-chip buffers before inference.
  • Performance comparison: Off-chip memory access generally consumes more FF, LUTs, DSPs, and BRAMs, while on-chip weights achieve higher throughput and lower latency.The survey attributes the performance difference to off-chip memory-access time.
  • Performance comparison: Persistent on-chip storage generally yields lower inference latency and higher throughput, but off-chip weights may be necessary for accelerators supporting a wider variety of models.The survey notes that the storage comparison is difficult because many articles do not clearly distinguish preloading from access during inference.
  • Performance comparison: Quantisation, sparsity, and architectural restructuring are analysed separately from the storage-based comparison.The storage analysis isolates the relationship between storage approach, resource usage, and performance.

6 Incorporated tooling

Transformer FPGA implementations use either Register-Transfer Level or High-Level Synthesis tooling, with surveyed usage nearly evenly divided between the two. Aggregate trends suggest trade-offs between HLS throughput and RTL power efficiency, although accelerator architecture strongly affects the figures.

  • Tooling categories: FPGA development tooling is organised into Register-Transfer Level and High-Level Synthesis design levels.HLS simplifies development by reducing the need for low-level RTL and hardware-description-language expertise.
  • Tooling usage: 12 surveyed articles use RTL tooling and 13 use HLS tooling, indicating a nearly even distribution among articles specifying their tooling.Verilog is the dominant RTL choice; SystemVerilog and VHDL appear in two other articles.
  • Frameworks: hls4ml was extended to support TensorFlow-based Transformer models and translate them into HLS code for FPGA deployment.The implementation used Vivado for the Transformer supported by the extended framework.
  • Comparison basis: The survey compares RTL and HLS resource usage, throughput in GOPS, and power in Watts, while noting that many articles do not specify tooling.Table 4 explicitly includes an Unknown tooling category for unspecified implementations.
  • HLS versus RTL: HLS designs use more FFs and LUTs on average and consume roughly twice the average power of RTL designs.These are overall trends rather than architecture-independent effects.
  • HLS versus RTL: RTL designs tend toward lower power, while HLS designs show higher average throughput in the surveyed comparisons.The survey says the underlying cause of the power pattern is unclear and may reflect design control or use-case selection.

7 Model compression

The survey examines quantisation and pruning as model-compression strategies for reducing Transformer size, resource use, and inference cost on FPGAs. Reported approaches trade precision or sparsity against accuracy while targeting memory, throughput, latency, and energy efficiency.

  • Quantisation: 8-bit quantisation is the most common reported precision, appearing in 13 of 30 articles, while other works use higher, lower, or mixed bit-widths.Higher precisions aim to retain accuracy, whereas lower precisions target lower computational latency.
  • Quantisation: 3-bit LUT-based multiplication reduces required DSPs from more than 10 000 to 744, while reducing accuracy from 73.30% to 69.60% compared with 4-bit operands.The multiplication is decomposed into six Boolean functions implemented with six LUT-6s.
  • Quantisation: 5.71-bit average weight precision with 0.526% accuracy loss is reported for HEAT’s hybrid scheme, which assigns 5 bits to normal values and 8 bits to outliers.The threshold is selected by line search using a constant outlier ratio of 3.5%.
  • Quantisation: 14.57 times higher energy efficiency is reported for a row-wise Hessian-driven accelerator that assigns higher precision only to the top 1% of weight rows.The architecture supports both low- and high-precision calculations.
  • Quantisation: 2.44 times weight compression, versus 1.95 times with 8-bit fixed-point quantisation, is achieved by dynamic mixed-precision allocation while maintaining identical model accuracy.The method allocates layer bit-widths according to input magnitude to avoid overflow.
  • Pruning: FNM-Trans reaches up to 93.75% attention sparsity and 75% weight sparsity with a 3.43% reduction in model accuracy.Its staged pruning algorithm applies structured N:M sparsity to attention and weights, retraining as sparsity increases.

8 IP design optimisations

The survey covers FPGA computation and architecture optimisations for Transformer inference, including SoftMax restructuring, processing-element organisation, dataflow, and pipelining. These designs seek better throughput and resource utilisation while managing memory traffic, dependencies, precision, and control complexity.

  • Computation: SoftMax and non-linear functions: SoftMax is a major FPGA inference bottleneck because exponentiation and division are expensive, while its sequential nature limits parallelism and optimisation.Its repeated use in attention layers increases computational and memory requirements.
  • Computation: SoftMax and non-linear functions: Threshold-based SoftMax approximation discards scores sufficiently below the maximum and stores valid scores in a smaller table.This approach addresses the excessive memory required by a full 16-bit SoftMax lookup table.
  • Computation: SoftMax and non-linear functions: Kernel fusion preserves SoftMax’s mathematical equivalence while moving normalisation outside matrix multiplication, reducing intermediate storage and off-chip memory traffic.The approach enables streaming execution and better hardware utilisation but requires additional accumulators, wider data paths, and synchronisation logic.
  • Architecture designs: PE design: At least 25 surveyed designs use systolic-array processing elements, while at least 14 use architectures composed of several distinct processing elements.These approaches may be combined to balance parallelism, resource utilisation, and design complexity.
  • Architecture designs: Pipelining: Pipelining divides computation into sequential stages that process different data concurrently, allowing new data to enter before preceding data finishes.Transformer implementations may pipeline individual computation steps or larger architectural blocks.
  • Architecture designs: Pipelining: Effective Transformer pipelining requires balancing dataflow, buffering, and memory access because dependencies, bandwidth, data movement, and workload characteristics can cause stalls.Sequence length, token granularity, and encoder-style versus autoregressive workloads affect computation overlap.
  • Architecture designs: Pipelining: 68% of collected articles explicitly mention pipelining, while 32% make no comment about it.The survey includes both intra-step and larger-scale pipelining approaches.

9 Insights and discussion

The survey compares FPGA Transformer inference primarily through throughput and energy efficiency, while emphasizing that model, hardware, precision, and implementation differences limit definitive cross-paper rankings.

  • Performance metrics: GOPS measures raw compute throughput, while GOPS/W provides an additional energy-efficiency metric for comparing reported FPGA inference results.The survey uses top-10 listings for throughput and energy efficiency, with available reporting determining inclusion in the latter.
  • Throughput: HG-PIPE achieves the highest reported throughput, but its LUT-based, on-chip implementation also consumes the highest number of LUTs.Among the top ten throughput implementations, only four use on-chip storage, although HG-PIPE substantially outperforms the second-best implementation.
  • Energy efficiency: The two most energy-efficient implementations use binarised quantisation, while five of the top ten use RTL-level design rather than HLS.Binarised weights are not feasible for every model, so this pattern does not imply universal applicability.
  • Model performance: Quantisation can improve performance and efficiency but may degrade accuracy, and different quantisation methods impose different access and computational requirements.The survey notes that accuracy sensitivity is model- and data-dependent and that some researchers use already quantised models as baselines.
  • Multi-objective analysis: Pareto analysis identifies a small set of designs balancing throughput and energy efficiency, while energy efficiency varies non-monotonically across throughput-ranked designs.High throughput therefore does not necessarily imply poor energy efficiency, motivating joint consideration of both metrics.
  • Comparison limitations: Meaningful comparisons remain difficult because implementations evaluate different Transformer models and configurations, so reported results reflect model complexity as well as accelerator innovation.The survey therefore presents broad tendencies rather than definitive rankings across fundamentally different implementations.

10 Conclusion

The survey systematically reviews recent FPGA-based Transformer inference research and finds implementation choices remain application-driven rather than governed by an absolute preference. It highlights quantisation and flexibility–performance trade-offs as central considerations for future designs.

  • Review contribution: The review applies a repeatable systematic protocol to recent Transformer inference deployment studies and organizes the field through a comprehensive taxonomy.It also derives top-10 listings from commonly reported throughput and power metrics, including GOPS/W as a power-efficiency measure.
  • Design diversity: No absolute preference emerges for HLS versus RTL, memory organization, pipelining, or quantisation; choices remain implementation-specific and application-driven.This diversity reflects FPGA flexibility while making direct comparisons across published works difficult.
  • Quantisation trade-off: Reduced precision can improve performance and energy efficiency but may cause non-negligible accuracy degradation, making some models non-viable after quantisation.The survey identifies this trade-off, improved reporting, and benchmark standardisation as future-work opportunities.
  • Flexibility and performance: Highly specialised accelerators can maximize throughput and energy efficiency but support fewer models, whereas flexible designs incur overheads and lower peak performance.The appropriate flexibility–performance balance depends on the application.
Loading 2609.01212v1…