PaperlayerView exact revision on arXiv

Source-linked AI summary

Toolflows for Mapping Convolutional Neural Networks on FPGAs: A Survey and Future Directions

Stylianos I. Venieris, Alexandros Kouris, Christos-Savvas Bouganis

arXiv:1803.05900v1cs.CVcs.ARcs.LG

TL;DR

CNNs deliver strong AI performance but increasingly complex models demand more computation and memory, motivating alternatives to conventional execution platforms. This paper surveys CNN-to-FPGA toolflows, compares their architectures and design methods, and proposes a uniform evaluation methodology. The survey identifies performance patterns linked to RTL optimisation, design-space customisation, and network structure, while noting that current benchmarking prevents conclusive QoR comparisons.

  • Problem

    Increasing CNN complexity raises computational and memory requirements, while existing evaluation procedures do not support conclusive comparisons of CNN-to-FPGA toolflows.

  • Method

    The paper surveys CNN-to-FPGA toolflows, compares their characteristics and mapping techniques, identifies emerging challenges, and proposes a benchmark suite with uniform evaluation methodology.

  • Results

    Highly optimised RTL designs, finer-grained design-space exploration, and architectures matched to uniform CNN structure are associated with stronger reported performance.

  • Takeaways & Limitations

    CNN-to-FPGA research needs comprehensive benchmarks and evaluation metrics to support further development of automated toolflows.

  • Takeaways & Limitations

    Ad-hoc and non-uniform benchmarking prevents conclusive, meaningful comparison of generated accelerator Quality of Results.

Abstract

from arXiv · show

In the past decade, Convolutional Neural Networks (CNNs) have demonstrated state-of-the-art performance in various Artificial Intelligence tasks. To accelerate the experimentation and development of CNNs, several software frameworks have been released, primarily targeting power-hungry CPUs and GPUs. In this context, reconfigurable hardware in the form of FPGAs constitutes a potential alternative platform that can be integrated in the existing deep learning ecosystem to provide a tunable balance between performance, power consumption and programmability. In this paper, a survey of the existing CNN-to-FPGA toolflows is presented, comprising a comparative study of their key characteristics which include the supported applications, architectural choices, design space exploration methods and achieved performance. Moreover, major challenges and objectives introduced by the latest trends in CNN algorithmic research are identified and presented. Finally, a uniform evaluation methodology is proposed, aiming at the comprehensive, complete and in-depth evaluation of CNN-to-FPGA toolflows.

1 INTRODUCTION

CNNs achieve strong AI-task performance but increasingly complex networks raise computational and memory demands. The paper motivates FPGAs as an alternative platform and surveys toolflows that automate CNN-to-hardware mapping.

  • CNNs have achieved high accuracy across applications including object detection, classification, drone navigation, and autonomous driving.
  • Deeper and more expressive CNNs increase computational and memory requirements as task complexity grows.
  • Existing deep-learning frameworks primarily target power-costly CPUs, GPUs, DSPs, or specialised ASICs.
  • FPGAs are presented as a promising alternative target platform for integrating CNN acceleration into the deep-learning ecosystem.
  • Automated CNN-to-FPGA toolflows can generate customised hardware implementations without requiring users to have hardware-design expertise.
  • The paper surveys toolflows by supported models, interfaces, architectures, design-space exploration, arithmetic precision, and performance.

2 CNN-TO-FPGA TOOLFLOW CHARACTERISTICS

The survey evaluates CNN-to-FPGA toolflows from three complementary perspectives: applicability, design methodology, and reported performance.

  • Applicability: Applicability is assessed through supported neural-network models, input interfaces, and portability.
  • Design methodology: Design methodology is examined through hardware architecture, design-space exploration, and arithmetic-precision choices.
  • Performance: Performance is analysed using the results reported for each toolflow.

2.1 Supported Neural Network Models

Existing CNN-to-FPGA toolflows mainly target CNN inference, while Finn focuses specifically on binarised neural networks. The surveyed frameworks support the common CNN layer types identified in the passage.

  • Most existing toolflows focus on automatically mapping CNN inference.
  • Finn targets the more specific application area of Binarised Neural Networks.
  • The common CNN layer types are convolutional, nonlinear, pooling, and fully-connected layers.

2.2 Interface

Toolflow interfaces affect accessibility and portability affects the range of FPGA targets. Caffe is widely supported, whereas custom front ends complicate integration with existing deep-learning toolchains.

  • Input interface: Caffe is the most widely supported front end, used by seven FPGA frameworks because of its structured syntax, pretrained models, and user community.
  • Input interface: fpgaConvNet and Snowflake also provide Torch back ends, while FP-DNN provides an additional interface.
  • Input interface: SysArrayAccel, AutoCodeGen, and FFTCodeGen use custom front ends based on C, proprietary network descriptors, or YAML with a Python wrapper.
  • Input interface: Custom front ends make integration with existing deep-learning toolchains more difficult and require additional infrastructure for practitioners.
  • Portability: Portability measures whether a toolflow can target different vendors, families, deployment setups, and FPGA sizes.
  • Portability: DnnWeaver has demonstrated the highest portability by targeting SoCs and server-grade FPGAs from both Xilinx and Intel.

2.3 Hardware Architecture

CNN-to-FPGA toolflows generate either streaming architectures with dedicated pipelined layer stages or single computation engines that execute layers sequentially. These choices trade customisation and concurrent execution against flexibility and hardware reuse.

  • Streaming architectures: Streaming architectures assign distinct hardware blocks to CNN layers and chain them into pipelines for concurrent layer execution.Each block is separately optimised for its layer’s parallelism, while new CNNs require new bitstreams.
  • Streaming architectures: fpgaConvNet builds coarse pipelines from reusable convolution, pooling, sliding-window and irregular-dataflow blocks tailored to the input CNN.For latency-sensitive applications, it can time-share datapath stages through soft run-time reconfiguration.
  • Streaming architectures: DeepBurning combines library components into a streaming architecture and uses run-time data-driven control with time-sharing of blocks.A central controller manages data movement between off-chip and on-chip memory.
  • Streaming architectures: Finn maps each BNN layer to a dedicated pipelined computation engine whose processing rate is matched to neighbouring engines.Its MVTU performs matrix-vector-threshold operations, while on-chip weight storage limits the addressable network size.
  • Single computation engines: Single computation engines use a scalable systolic-array or matrix-multiplication template, with software controlling sequential layer execution and scheduling.Angel-Eye reuses its hardware across CNNs by compiling models into instruction sequences, whereas ALAMO customises the engine to the input CNN.

2.4 Design Space Exploration

Toolflows explore parameterised hardware designs under FPGA resource constraints using methods ranging from heuristic constrained optimisation to exhaustive enumeration and deterministic mapping. Their design spaces expose different trade-offs among throughput, latency, resource use and customisation.

  • Design Space Formulation and Search: Design Space Exploration selects an architecture point by trading performance, resource consumption and power efficiency under target-FPGA constraints.Performance is commonly characterised by latency and throughput, while tunable parameters define the explored design space.
  • Parameter Space: Toolflows expose different customisation parameters, including layer parallelism, block organisation, tiling, batch size, data reuse, array shape and frequency-domain transform size.Some systems tailor hardware to the CNN, while Haddoc2 deterministically maps network actors without configurable parameters and succeeds only when the result fits device resources.
  • Parameter Space: Haddoc2’s direct hardware mapping follows the exact CNN topology independently of target-device specifications, so feasibility depends on fitting within the resource budget.It also operates with batch size 1.
  • Design Space Formulation and Search: fpgaConvNet and DnnWeaver formulate constrained optimisation problems and use heuristic searches because their parameter spaces are too large for brute-force enumeration.fpgaConvNet supports throughput, latency and multiobjective objectives; DnnWeaver focuses on throughput with batch processing.
  • Design Space Formulation and Search: Caffeine uses an enhanced roofline model with burst-length-dependent bandwidth estimates, producing a small design space suitable for exhaustive enumeration.The model guides selection of the design point.
  • Design Space Formulation and Search: SysArrayAccel models performance and resources analytically for data-reuse patterns and systolic-array shapes, then maximises throughput in a high-dimensional design space.Its tunable parameters explore different parallelism and resource configurations.

2.5 Arithmetic Precision

Low-precision arithmetic reduces FPGA CNN implementation costs while preserving comparable accuracy, and toolflows differ in whether precision is uniform or layer-adaptive. Several frameworks expose bitwidth and scaling as compile-time design choices.

  • Arithmetic Precision: Few-bit fixed-point representations can achieve comparable accuracy to floating point while increasing performance and reducing external memory bandwidth requirements.Lower bandwidth can also decrease power consumption.
  • Arithmetic Precision: Angel-Eye applies automated dynamic quantisation by selecting a different scaling factor for each layer under a predefined network wordlength.The scaling-selection problem is solved using a greedy method.
  • Arithmetic Precision: ALAMO and AutoCodeGen adjust wordlength and scaling per unit at compile time to allocate fractional precision according to each layer’s dynamic range.DnnWeaver encodes floating- or fixed-point format, fractional bits and total bitwidth in generated instructions.
  • Arithmetic Precision: Caffeine, fpgaConvNet, FP-DNN, SysArrayAccel and FFTCodeGen support floating- and fixed-point data but apply uniform wordlength and scaling across layers.Haddoc2 supports only fixed point, while Snowflake uses uniform 16-bit quantisation.

2.6 Performance

Performance comparisons show that CNN-to-FPGA toolflows trade throughput, latency, resource use, precision, and automation differently across CNN–FPGA pairs. Fair evaluation therefore requires matching networks and devices, while comparisons with hand-tuned designs expose automation trade-offs.

  • Evaluation methodology: Fair comparisons require toolflows to map the same CNN on the same FPGA, using throughput and latency as primary performance metrics.Quality of results is also assessed against hand-tuned accelerators for identical CNN–FPGA pairs.
  • Zynq comparisons: On Zynq platforms, fpgaConvNet generally delivers higher throughput and lower latency than DeepBurning, while Angel-Eye leads fpgaConvNet on VGG16.For VGG16 on Zynq XC7Z045, Angel-Eye achieves 1.20× higher feature-extractor throughput, 136.97 GOp/s for the whole network, and 1.52× lower latency.
  • Zynq comparisons: Haddoc2 achieves 1.71× higher LeNet-5 throughput and 2.63× higher CIFAR-10 throughput than fpgaConvNet on Zynq XC7Z045.Haddoc2 uses 3-bit and 6-bit representations, whereas fpgaConvNet uses 16-bit representations.
  • Stratix comparisons: FP-DNN reaches 364.36 GOp/s on VGG19 and outperforms DnnWeaver by 2.31×, although the compared networks have different workloads.VGG19 contains three additional convolutional layers compared with VGG16, so its performance is used as an indicator for VGG16 throughput.
  • Stratix comparisons: ALAMO and FFTCodeGen often achieve higher DSP-normalised throughput, but this metric omits device bandwidth and ALAMO also uses ALMs alongside DSPs.FFTCodeGen performs convolutions in the frequency domain, while ALAMO uses DSPs and logic for compute units.
  • Hand-tuned comparisons: Against hand-tuned designs, automated flows can approach or exceed selected baselines, but highly specialised RTL and on-chip-memory assumptions limit direct generalisation.FFTCodeGen exceeds Suda et al. by 38.07× on AlexNet and 21.37× on VGG16, while DLA’s caching assumption does not hold for larger models such as VGG16 and ResNet-152.

2.7 Discussion: Quality of Results

Comparative QoR conclusions are limited by inconsistent benchmarking, but available same-pair comparisons reveal trade-offs among RTL optimisation, design-space flexibility, and architecture regularity.

  • Ad-hoc benchmarking prevents conclusive comparisons of generated accelerators’ comparative Quality of Results.The survey therefore proposes a uniform evaluation methodology to address current evaluation shortcomings.
  • Highly optimised RTL-based designs tend to outperform HLS counterparts, although manual RTL optimisation reduces automation and productivity.ALAMO outperformed a more extensively explored HLS design by 3.6×, illustrating the performance-productivity trade-off.
  • Finer-grained design-space exploration generally improves Quality of Results by enabling more customised accelerator configurations.Analytical exploration outperformed a more restricted space, which outperformed heuristic mapping; FFTCodeGen’s CaP sustained utilisation across differently sized convolution layers.
  • Single-computation-engine architectures perform well on structurally uniform CNNs but can underutilise hardware on networks with varied kernel sizes.VGG16’s uniform layers benefit DnnWeaver, SysArrayAccel, and Angel-Eye, whereas AlexNet’s varied kernels reduce utilisation; FFTCodeGen avoids this penalty with tiled FFT-based convolution.

2.8 Discussion: Suitability for Deep Learning Application Challenges

CNN-to-FPGA toolflows make different trade-offs among throughput, latency, portability, precision flexibility, and hardware specialisation. Their suitability depends strongly on application requirements, target devices, and network scale.

  • Application and performance trade-offs: Finn targets binarised neural networks, trading broad applicability for very high throughput and low latency on Xilinx devices.Its lightweight designs support embedded SoCs and standalone devices, but binarisation can affect accuracy.
  • Application and performance trade-offs: DnnWeaver prioritises portability across FPGA vendors and device classes, but is restricted to high-throughput applications with large batch sizes.Its parameterised RTL templates support variable precision across CNN layers.
  • Architectural choices: Caffeine and SysArrayAccel optimise systolic arrays for high-throughput CNNs, while their different target devices prevent meaningful performance comparison.Caffeine targets Xilinx FPGAs and SysArrayAccel targets Intel FPGAs.
  • Architectural choices: ALAMO combines RTL-like throughput and low latency with layer-wise precision flexibility, but manual quantisation and Intel-specific IPs limit automation and portability.Its generated designs are tailored to Intel FPGAs.
  • Design-space exploration: Angel-Eye co-optimises latency and throughput through automatic dynamic quantisation and unrolling of input and output feature maps.The selected parameter space determines both feature-map unrolling directions.
  • Application and performance trade-offs: fpgaConvNet supports high-throughput, low-latency, and multiobjective objectives, but Vivado HLS currently restricts it to Xilinx devices.Its optimisation methodology is selected according to application-level performance needs.
  • Architectural choices: DeepBurning supports multiple neural-network types and layer-wise precision flexibility, optimising batch-size-1 accelerators for both throughput and latency.Its RTL blocks include support for RNNs and LSTMs.
  • Architectural choices: Haddoc2’s deterministic direct mapping delivers high throughput and low latency when weights fit on-chip and sufficient logic exists, but cannot handle large-scale models without time-sharing.The approach suits small networks and aggressively quantised models.

2.9 Other Related Work

Related FPGA CNN work includes hand-tuned accelerators and memory-bandwidth-focused designs, but some lack the components required for complete automated CNN-to-FPGA toolflows.

  • Other FPGA-based designs: FPGA research includes hand-tuned RTL, HLS, and mixed RTL-HLS accelerators for particular CNN-FPGA pairs.Other designs focus specifically on optimising external memory-bandwidth utilisation.
  • Benchmark context: The benchmark suite is presented as a collection of CNN models with distinct computational and memory-mapping challenges.Its title identifies the models and their computational challenges, while the accompanying discussion motivates varied stress cases.

3 THE FUTURE OF CNN-TO-FPGA TOOLFLOWS

Future CNN-to-FPGA toolflows should be evaluated uniformly and extended to newer, irregular, sparse, recurrent, and hardware-aware learning workloads. The proposed direction couples representative benchmarks and metrics with broader framework integration and co-design.

  • Uniform evaluation methodology: A uniform evaluation methodology combines a benchmark suite with guidelines for evaluation metrics to address non-uniform prior assessments.The goal is thorough and comparative evaluation of CNN-to-FPGA toolflows.
  • Uniform evaluation methodology: The benchmark suite should include CNNs that stress computation, off-chip bandwidth, on-chip capacity, layer dependency, and irregular or sparse computation.Each model should be widely used and extensively studied for accuracy.
  • Uniform evaluation methodology: The proposed suite contains AlexNet, ZFNet, VGG16, GoogLeNet, ResNet-152, Inception-v4, and DenseNet-161.These models represent different hardware-mapping challenges.
  • Uniform evaluation methodology: VGG16 represents a deep CNN with high computation and memory requirements and is widely used as a pretrained model.Its large computational load, weight count, and depth create several mapping challenges.
  • Uniform evaluation methodology: Evaluation should report throughput, latency, resource consumption, power efficiency, and application-level accuracy.Together, these metrics expose strategic trade-offs made by each toolflow.
  • Uniform evaluation methodology: Resource-normalised performance metrics may omit memory bandwidth and capacity, while FPGA-family-specific normalisation limits cross-device comparison.A single DSP- or logic-normalised metric cannot capture hardware quality across differently optimised devices.
  • Emerging model support: Future tools should support compound CNN structures including residual, Inception, and dense blocks.ALAMO, Snowflake, FP-DNN, and fpgaConvNet support some of these structures, with fpgaConvNet providing dense-block hardware.
  • Emerging model support: RNNs and LSTMs require distinct mapping approaches because their matrix-vector multiplications create inherently memory-bounded workloads.DeepBurning and FP-DNN provide some automated support, but end-to-end recurrent-network frameworks remain an objective.

4 CONCLUSION

The paper surveys CNN-to-FPGA toolflows, compares their characteristics and mapping techniques, and proposes a comprehensive benchmark to address limitations in prior comparisons.

  • Conclusion: The survey identifies strengths and weaknesses across CNN-to-FPGA toolflows and their mapping techniques.Its comparison covers the main characteristics and features of the surveyed frameworks.
  • Conclusion: A comprehensive benchmark is proposed because prior evaluation methods were non-uniform and omitted both computational and memory resources.The stated aim is to improve comparison between toolflows.
Loading 1803.05900v1…