PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of FPGA-Based Neural Network Accelerator

Kaiyuan Guo, Shulin Zeng, Jincheng Yu, Yu Wang, Huazhong Yang

arXiv:1712.08934v3cs.AR

TL;DR

Neural-network inference is computation- and storage-intensive, motivating alternatives to CPUs and GPUs. This survey synthesizes FPGA accelerator techniques across software and hardware and evaluates their achievable performance. It reports that software-hardware co-design can provide more than 10× better speed and energy efficiency than state-of-the-art GPUs, while noting remaining gaps in narrow-bit quantization, technique integration, tools, and scaling.

  • Problem

    Neural-network models require substantial computation and storage, while FPGA accelerators face performance and flexibility challenges that motivate systematic design analysis.

  • Method

    The paper reviews FPGA accelerator techniques from software and hardware perspectives, develops a performance model, compares state-of-the-art designs, and surveys automated design methods.

  • Results

    Software-hardware co-design enables FPGA accelerators to achieve more than 10× better speed and energy efficiency than state-of-the-art GPUs.

  • Takeaways & Limitations

    FPGA is a promising candidate for neural-network acceleration, with current development flows supporting high performance and run-time network switching.

  • Takeaways & Limitations

    A gap remains between current designs and estimates because extremely narrow quantization, combined techniques, commercial tools, and scaling still require further research.

Abstract

from arXiv · show

Recent researches on neural network have shown significant advantage in machine learning over traditional algorithms based on handcrafted features and models. Neural network is now widely adopted in regions like image, speech and video recognition. But the high computation and storage complexity of neural network inference poses great difficulty on its application. CPU platforms are hard to offer enough computation capacity. GPU platforms are the first choice for neural network process because of its high computation capacity and easy to use development frameworks. On the other hand, FPGA-based neural network inference accelerator is becoming a research topic. With specifically designed hardware, FPGA is the next possible solution to surpass GPU in speed and energy efficiency. Various FPGA-based accelerator designs have been proposed with software and hardware optimization techniques to achieve high speed and energy efficiency. In this paper, we give an overview of previous work on neural network inference accelerators based on FPGA and summarize the main techniques used. An investigation from software to hardware, from circuit level to system level is carried out to complete analysis of FPGA-based neural network inference accelerator design and serves as a guide to future work.

1 INTRODUCTION

Neural-network inference offers strong machine-learning performance but demands substantial computation, storage, and platform support. This survey reviews FPGA-based accelerator techniques and evaluates their potential for high-performance, energy-efficient inference.

  • CNNs and RNNs improve image, video, and speech processing, with CNN ImageNet top-5 accuracy rising from 73.8% to 84.7% in 2012.
  • The largest CNN models require up to 39 billion FLOP and more than 500MB of parameters for 224 × 224 image classification.Higher-resolution images may require more than 100 billion operations, while smaller models can incur accuracy loss.
  • CPU platforms struggle with neural-network performance and power requirements, whereas GPUs provide up to 10TOP/s and accessible development frameworks.
  • FPGAs support neural-network-oriented parallelism and hardware specialization, offering a potential energy-efficiency advantage over CPUs and GPUs.However, lower operating frequencies, reconfigurability overhead, and limited development frameworks create performance and flexibility challenges.
  • The survey develops a performance model, reviews software- and hardware-level optimization techniques, compares accelerator designs, and examines automated design methods.
  • FPGA accelerators use optimization methods spanning algorithms, models, architectures, and system design to address computation and storage constraints.
  • A typical FPGA accelerator combines a CPU host with an FPGA, while external memory is needed because on-chip storage is smaller than neural-network models.Common models use 100–1000MB of parameters, whereas the largest available FPGA chip has under 50MB of on-chip SRAM; external-memory bandwidth and power constrain performance.

3 DESIGN METHODOLOGY AND CRITERIA

The survey frames accelerator design around throughput, latency, and energy efficiency. Its criteria connect hardware resources, model properties, parallel execution, memory traffic, and energy costs.

  • Accelerator design targets high speed, including high throughput and low latency, together with high energy efficiency.
  • Speed: Peak throughput can increase by using more computation units or raising frequency, while utilization depends on parallelism, memory systems, and model access patterns.Reducing computation-unit size through lower precision may hurt model accuracy and requires hardware-software co-design.
  • Speed: Throughput comparisons use OPSact to reduce differences caused by evaluating accelerators on different neural-network models.Layer pipelining and batch processing are common concurrent designs considered with loop unrolling.
  • Energy Efficiency: Energy efficiency is defined as operations per total energy, so for a fixed workload, improving it means reducing the energy required per input.
  • Energy Efficiency: Total energy includes computation, SRAM access, DRAM access, and static energy; quantization, sparsification, on-chip memory, and scheduling can reduce dynamic costs.Static energy is difficult to improve for a fixed FPGA chip and design scale.
  • The methodology treats neural-network acceleration as requiring coordinated optimization of both models and hardware.

4 HARDWARE ORIENTED MODEL COMPRESSION

Hardware-oriented model compression reduces neural-network computation and storage by narrowing numerical representations or reducing weights. The survey reviews quantization, low-rank and pruning methods, emphasizing their accuracy trade-offs and hardware benefits.

  • Quantization: Quantization reduces computation and storage by representing activations and weights with fewer bits or a small set of trained values.The simplified representation also reduces hardware cost for each operation.
  • Quantization: Linear quantization assigns fixed-point representations, with layer-specific fractional bit-widths addressing the differing dynamic ranges of weights and activations.Layer-dependent scaling helps limit overflow and underflow from floating-point to fixed-point conversion.
  • Quantization: Non-linear quantization maps binary codes to independently assigned values through a lookup table, enabling further bit-width reduction.The surveyed methods obtain lookup-table values using hashing or clustering.
  • Quantization: 8-bit linear quantization is a clear bound for negligible accuracy loss, while at least 8 × 8 or listed non-linear methods stay within 1% loss.With 6 or fewer bits, fine-tuning or training weights from the beginning causes noticeable accuracy degradation.
  • Weight reduction: Low-rank decomposition replaces an m × n weight matrix with Am×pBp×n, reducing weights when p is sufficiently small.The resulting compression targets fully connected layers.
  • Weight reduction: Pruning removes zero or small-magnitude weights, but increasing sparsity trades against model accuracy; sparse group-lasso removed 90% of AlexNet weights after training.The hardware speed improvement from weight reduction can reach 10×, equal to the reciprocal of the compression ratio.

5 HARDWARE DESIGN: EFFICIENT ARCHITECTURE

The section surveys hardware techniques that improve FPGA neural-network accelerator performance and energy efficiency, spanning computation units, fast convolution, frequency, and memory-system design.

  • Architecture levels: Hardware techniques are organized into computation-unit, loop-unrolling, and system levels.This classification structures the discussion of performance and energy-efficiency techniques.
  • Computation unit designs: Reducing bit-width shrinks computation units, with 8-bit logic-only operators scaling to about 1/10 multiplier and 1/50 adder size versus 32-bit floating point.Using 4-bit or smaller operators can provide further benefits but may cause significant accuracy loss.
  • Computation unit designs: Narrow fixed-point operators remain resource-efficient, but DSP-based designs may underutilize wide hardware when using 8- or 4-bit data.Fixed-point functions with 16-or-less-bit data fit into one DSP on either Xilinx or Altera FPGA.
  • Fast convolution: Fast convolution transforms convolution into frequency- or Winograd-domain operations, reducing multiplication requirements for suitable layer sizes.Winograd reduces multiplications from 36 to 16 for the cited 4 × 4 feature-map and 3 × 3-kernel case.
  • Fast convolution: FFT- and Winograd-based convolution can provide up to 4× theoretical performance gain, although resource and kernel-size constraints limit applicable convolution sizes.The paper also reports that not all layers can use the most optimized fast-convolution method.
  • System-level design: Increasing working frequency and reducing off-chip transfers complement computation-unit optimization under FPGA resource and bandwidth constraints.Layer fusion avoids intermediate-result transfers and reduces off-chip data transfer by 95% with 20% extra on-chip memory cost.

6 EVALUATION

The evaluation compares state-of-the-art FPGA accelerator designs using performance, energy efficiency, and resource utilization, highlighting strong results for extremely low-bit-width designs.

  • Evaluation methodology: The evaluation reviews FPGA accelerator designs published across major FPGA, EDA, architecture, and related conferences since 2015.Because techniques, FPGA chips, and experiments vary, the comparison requires a trade-off.
  • Evaluation methodology: Designs are compared using data format, performance, resource utilization, and a speed-versus-power plot whose coordinate difference represents energy efficiency.GPU experimental results are included as standards for measuring FPGA designs.
  • Evaluation findings: 1-2-bit designs show outstanding speed and energy efficiency, but their linear-quantized network models suffer great accuracy loss.The paper therefore identifies model development as more important than further accelerator development for these designs.

Bit-width Reduction.

The evaluation finds that bit-width reduction improves accelerator efficiency, but its hardware advantage depends on DSP utilization and accuracy constraints.

  • Bit-width Reduction: INT16 generally has an obvious advantage over FP32, except when hard-core floating-point DSPs are utilized.The comparison suggests that fully utilizing on-chip DSPs is important.
  • Bit-width Reduction: INT16/8 and INT16 are commonly adopted, but their performance difference is not obvious because DSPs can be underutilized.For designs using 8 or more bits, within 1% accuracy loss can be achieved according to the cited results.
  • Bit-width Reduction: The 6 × 6 Winograd design is 1.7× faster and 2.6× more energy efficient than the compared 16-bit design.The cited comparison concerns designs and.

Fast Convolution Algorithm.

The evaluation shows that fast convolution can improve FPGA accelerator speed and energy efficiency, while practical gains remain below theoretical limits and system design quality also matters.

  • System-level comparison: On the same XC7VX690T platform, one 16-bit design achieves 2.5× the energy efficiency of another without fast convolution or sparsity.The comparison attributes differences partly to overall system optimization and HDL design quality.

System Level Optimization.

Resource-utilization analysis shows that current FPGA accelerator designs favor DSP resources over BRAM and logic, indicating computation-bounded designs. FPGA designs achieve GPU-comparable energy efficiency, but GPU speed remains higher.

  • Resource utilization: DSP utilization exceeds BRAM and logic utilization across current FPGA accelerator designs, indicating a preference for computation resources.The comparison uses resource-utilization ratios plotted for DSP, BRAM, and logic.
  • Resource utilization: The random preference between logic and BRAM reflects differing choices between using logic with DSPs for parallelism and using DSPs alone for higher frequency.These alternatives explain why logic–BRAM placement lacks the consistent pattern seen in DSP comparisons.
  • Power and performance: FPGA-based designs achieve 10-100GOP/J, comparable energy efficiency to GPUs, while GPU speed still surpasses FPGA speed.The comparison is stated at the general design level rather than for one specific accelerator.

Comparision with GPU.

The survey estimates that combining quantization, frequency scaling, sparsity, and related optimizations could substantially improve FPGA accelerator speed and energy efficiency. However, jointly applying these techniques remains unresolved, especially because sparsity complicates convolution processing, memory access, and scheduling.

  • Achievable design: 24× speedup and 12× better energy efficiency are estimated for an ideal FPGA design, reaching 72TOP/s at about 50W.The estimate combines 8-bit quantization, double frequency, a sparse model with 10% non-zero values, and a 16-bit fixed-point baseline.
  • Achievable design: The estimate suggests over 10× higher energy efficiency on FPGA than 32-bit floating-point processing on GPU.This is an ideal-design estimate, not a reported result from a completed accelerator.
  • Open issues: Applying double MAC, sparsification, quantization, fast convolution, and double frequency together remains an open design question.The survey identifies interactions among these techniques as unresolved rather than assuming their gains combine directly.
  • Open issues: Sparse processing introduces irregular data access, while increased parallelism challenges memory-system design and scheduling.Pruning individual elements may not benefit fast convolution, whereas pruning whole 2D kernels can reduce accuracy relative to fine-grained pruning.

7 TECHNIQUE DISCUSSION

The survey organizes FPGA neural-network accelerator techniques by their effects on hardware and their relationship to neural-network model levels. Future opportunities center on newer network structures and scaling FPGA systems beyond individual chips.

  • Technique discussion: The survey evaluates each technique by its effect on hardware design and the neural-network level it exploits.Figure 8 summarizes software and hardware techniques using these two dimensions.
  • Technique discussion: Datapath choices determine peak operations and operation energy, while memory and scheduling affect utilization and memory-access costs.The hardware is described as a datapath, memory, and scheduling system.
  • Technique discussion: Existing research exploits neural-network features from the single-neuron level through the whole-network level.The survey frames these levels as the range of current design approaches.
  • Future opportunities: Most techniques currently operate at neuron and convolution levels because few usable features exist at layer and network levels, although depth-wise convolutions and complex branches may offer new opportunities.The survey notes that relatively little work has focused on these newer model structures.
  • Future opportunities: Further opportunities arise from evolving network structures and scaling FPGA systems through larger or multiple chips.Scaling introduces challenges involving loop unrolling and bandwidth.

8 DESIGN AUTOMATION AND FLEXIBILITY

Design automation addresses the difficulty of mapping networks onto FPGA accelerators and the need to support different models. Approaches range from directly generating hardware to running multiple networks through instruction sequences, with hybrid methods combining hardware templates and compiled software.

  • Motivation and hardware automation: FPGA accelerator design automation is important because mapping CNNs onto hardware requires more work than using established deep-learning frameworks and applications may need multiple models.The survey discusses toolflows that vary in supported models, interfaces, and hardware architecture.
  • Motivation and hardware automation: Hardware automation generates HDL from network parameters, using intermediate descriptions to bridge high-level network specifications and low-level hardware.The intermediate representation is the main distinction among several approaches.
  • Hardware automation: Template-based hardware flows optimize parameters for handcrafted Verilog or OpenCL designs, and OpenCL-based development can integrate with Caffe and target different platforms.These approaches modify or specialize hardware for the supplied network and platform constraints.
  • Hardware automation: Network descriptions can use data-flow graphs or virtual instruction sets, which are translated into hardware mappings or finite-state-machine states.Venieris et al. use a DFG, while DnnWeaver translates a network into an instruction sequence mapped to FSM states.
  • Hardware automation: Hardware design automation can achieve target-platform performance but suits settings where network switching is infrequent and reconfiguration overhead is acceptable.The survey gives large-scale cloud services as an example scenario.
  • Software automation: Software automation runs different networks on the same accelerator by changing inputs, commonly instruction sequences, while instruction granularity determines scheduling flexibility and hardware cost.Examples range from three instruction types to layer-level configurable finite-state machines and flexible platform mappings.
  • Software automation: Instruction-based methods support runtime network switching without hardware modification, avoiding reconfiguration overhead in applications such as real-time video processing.This requires the instructions for the relevant networks to be prepared in memory.
  • Hybrid automation: Hybrid automation optimizes hardware templates and compiles software binaries, allowing a new network to use existing hardware by changing the binaries.The hardware is assembled from predefined HDL templates using optimized parameters.

9 CONCLUSION

The survey finds that software–hardware co-design can make FPGA neural-network accelerators substantially faster and more energy-efficient than state-of-the-art GPUs, while current designs remain short of estimated potential. It identifies narrow-bit-width quantization, technique integration, commercial tooling, and scaling as continuing challenges.

  • More than 10× better speed and energy efficiency than state-of-the-art GPU are achievable with software–hardware co-design.This evaluation result supports FPGA-based acceleration as a promising candidate for neural-network processing.
  • Current designs still lag behind estimated potential because extremely narrow quantization is constrained by model accuracy.Further algorithm research is needed to address this limitation.
  • Combining optimization techniques, improving commercial tools, and scaling designs remain open research challenges.The paper specifically notes that these challenges require additional software and hardware research.
Loading 1712.08934v3…