Accelerating CNN inference on FPGAs: A Survey

Kamel Abdelouahab; Maxime Pelcat; Jocelyn Serot; François Berry

Source-linked AI summary

Accelerating CNN inference on FPGAs: A Survey

Kamel Abdelouahab, Maxime Pelcat, Jocelyn Serot, François Berry

arXiv:1806.01683v1May 26, 2018cs.DCcs.ARcs.CVcs.NE

TL;DR

CNN inference requires substantial computation, motivating dedicated FPGA accelerators as CNN workloads increasingly exploit sparsity and compact data types. This survey synthesizes recent FPGA methods across computational transforms, datapaths, approximate computing, and layer-level optimization, reporting diverse throughput and efficiency results. It concludes that combining algorithmic and datapath optimizations can provide additive hardware performance gains, while numerical-range variation and memory bandwidth constrain some designs.

Problem
CNN inference can require up to 38 GOP/s per frame, creating a need for efficient dedicated hardware as CNN development increasingly uses sparsity and compact data types.
Method
The paper surveys FPGA CNN inference accelerators, analyzing workloads, parallelism, memory access, layer optimizations, datapaths, approximate computing, and recent implementation techniques.
Results
The surveyed approaches include 2.94 TOPs on VGG convolutional layers, 207.8 GOP/s at 4.7 W for CIFAR10 binary inference, and projected 12 TOP/s for pruned CNNs.
Takeaways & Limitations
FPGA CNN acceleration can exploit custom data types, local streams, dedicated processors, approximate computing, and jointly implemented algorithmic and datapath optimizations.
Takeaways & Limitations
Some FPGA designs remain constrained by off-chip memory bandwidth, while shared fixed-point exponents are ill-suited to deep networks with differing numerical ranges.

Abstract

from arXiv · show

Convolutional Neural Networks (CNNs) are currently adopted to solve an ever greater number of problems, ranging from speech recognition to image classification and segmentation. The large amount of processing required by CNNs calls for dedicated and tailored hardware support methods. Moreover, CNN workloads have a streaming nature, well suited to reconfigurable hardware architectures such as FPGAs. The amount and diversity of research on the subject of CNN FPGA acceleration within the last 3 years demonstrates the tremendous industrial and academic interest. This paper presents a state-of-the-art of CNN inference accelerators over FPGAs. The computational workloads, their parallelism and the involved memory accesses are analyzed. At the level of neurons, optimizations of the convolutional and fully connected layers are explained and the performances of the different methods compared. At the network level, approximate computing and datapath optimization methods are covered and state-of-the-art approaches compared. The methods and tools investigated in this survey represent the recent trends in FPGA CNN inference accelerators and will fuel the future advances on efficient hardware deep learning.

1 Introduction

CNNs deliver near-human accuracy but require substantial computation, motivating dedicated FPGA acceleration. This survey focuses on recent FPGA techniques as CNN sparsity and compact data types increasingly favor reconfigurable hardware.

CNNs achieve near-human accuracy across vision and speech applications, including classification, detection, segmentation, and recognition.
38 GOP/s may be required to classify a single frame, creating demand for dedicated CNN hardware.
GPUs reach up to 11 TFLOP/s, whereas FPGA solutions are more energy efficient and target both data centers and embedded applications.
FPGA performance has reached 9.2 TFLOP/s, while CNN sparsity and compact data types favor FPGA irregular parallelism and custom arithmetic.
The survey reviews recent FPGA-based CNN accelerator techniques not covered by earlier surveys, distinguishing them from ASIC-focused reviews.
The paper covers CNN computations, FPGA computational transforms, datapath optimization, and approximate computing across Sections 2–5.

2 Background on CNNs

This section introduces CNN inference by emphasizing the computations and parallelism patterns involved in processing these networks.

The section overviews CNN features while focusing on inference computations and their parallelism patterns.

2.1 General Overview:

CNNs are feed-forward, deep, sparsely connected networks with shared weights, organized as pipelines that transform feature maps into higher-level representations.

CNNs are feed-forward, deep, sparsely connected neural networks that implement weight sharing.
A typical CNN is a layer pipeline in which each layer transforms input feature maps into new maps with higher-level semantics.

2.2 Inference vs Training:

CNNs are trained on annotated data and then use the learned model for repeated inference on unseen samples. Because inference runs for every classified sample, the survey focuses on accelerating it, especially for image classification.

Training uses annotated samples and back-propagation to create a model whose learned parameters can be fine-tuned for new constraints.
Inference applies the learned model to classify previously unseen data samples.
Because inference executes whenever a new sample is classified, the literature primarily focuses on accelerating inference rather than training.
The survey focuses on image-classification accelerators while noting that its methods can also support detection, segmentation, and speech recognition.

2.3 Inference of CNNs

CNN inference is feed-forward propagation of input images through layers that transform feature maps using convolution, activation, pooling, and optionally fully connected operations. The section describes these layer computations and batch normalization as an inference-relevant operation.

CNN inference: CNN inference propagates B input images across L layers, with layer parameters and feature maps represented as tensors.The tensor dimensions are organized for each layer in Table 1.
Convolution layers: Convolution applies 3D filters to input volumes, producing two-dimensional feature maps and N output maps.A bias may be added to convolution results, although some models discard it.
Activation layers: Activation layers apply nonlinear functions to feature-map values, with recent CNNs commonly using ReLU for faster training and lower computational complexity.Earlier CNNs used TanH or Sigmoid functions.
Pooling layers: Pooling subsamples each feature-map channel by selecting an average or, more commonly, a maximum over a neighborhood, reducing feature-map dimensionality.Pooling layers are inserted periodically between successive convolutional layers.
Fully connected layers: Fully connected layers operate like convolution layers without weight sharing and apply a nonlinear function to their outputs.They are commonly used near the end of classification pipelines.
Batch normalization: Batch normalization linearly shifts and scales batch inputs toward zero mean and unit variance, with training statistics and hyperparameters used during processing.In binary neural networks, it can reduce quantization error relative to arbitrary input distributions.

2.4 Workload of a CNNs inference

CNN inference combines heavy computation, extensive parallelism, and substantial memory traffic. FPGA accelerators therefore exploit concurrency, caching, specialized hardware, and mappings that balance available resources against CNN workloads.

Computational workload: Convolutional layers account for more than 90% of typical CNN inference time, while fully connected layers contain most weights and create a different memory burden.This imbalance motivates distinct implementation strategies for convolutional and fully connected layers.
Parallelism in CNNs: Real-time CNN inference is challenging on low-energy devices because of its high computational demand, motivating exploitation of CNN concurrency.The section formalizes several parallelism sources.
Parallelism in CNNs: Batch and inter-layer parallelism reuse filters across grouped frames and pipeline successive data-dependent layers.Batch processing can reduce external memory accesses, while inter-layer execution overlaps adjacent layers.
Parallelism in CNNs: Inter-feature-map, intra-feature-map, inter-convolution, and intra-convolution parallelism expose progressively finer-grained opportunities for concurrent computation.These forms parallelize output maps, pixels, summed 2D convolutions, and multiplication pipelines, respectively.
Memory accesses: CNN vectorization can be inefficient without caching, while memory bandwidth often becomes the bottleneck for both weight-heavy fully connected layers and MAC-intensive convolutional layers.Each MAC requires at least two memory reads and one write in the stated best-case pipeline scenario.
Memory accesses: External DRAM accesses increase latency and energy consumption, so accelerators use on-chip buffers and nearby register files as a two-level caching hierarchy.The hierarchy reduces the number of external memory accesses.
Hardware, libraries and frameworks: FPGA CNN acceleration maps CNN computation onto DSP blocks and in-situ memories that support parallel execution and reduce external memory traffic.The mapping problem reflects the need to match the CNN computational model with the FPGA execution model.
Hardware, libraries and frameworks: Because FPGA resources cannot fully unroll modern CNN workloads, state-of-the-art designs map a limited number of reusable processing elements.The survey presents this resource-aware mapping alongside other accelerator strategies.

3 Algorithmic Optimizations for FPGA-Based CNN Acceleration

Algorithmic transforms accelerate FPGA CNN inference by converting convolutional and fully connected workloads into more parallel forms, while trading computation against memory overhead. GEMM, Winograd, and FFT methods offer different benefits depending on layer structure, kernel size, and implementation strategy.

Transform-based acceleration: Computational transforms vectorize convolutional and fully connected layers to reduce inference arithmetic and map CNN workloads onto FPGA architectures.These transforms operate on feature maps and kernels and are also implemented through CPU, GPU, and FPGA software or hardware libraries.
GEMM Transformation: GEMM maps fully connected layers to matrix-vector or batched matrix multiplication, allowing weights to be loaded once per batch and improving throughput at constant memory bandwidth.Batching concatenates feature maps into a CHW × B matrix; the benefit increases with FC-weight sparsity.
GEMM Transformation: GEMM maps 3D convolutions by flattening filters and rearranging input feature maps into matrices whose multiplication produces the output feature maps.The transformation can use an N × CKJ filter matrix and a CKJ × UV input matrix, but redundant input data can increase storage or complicate memory access.
Winograd Transform: Winograd filtering reduces multiplications from u^2 × k^2 to (u+k−1)^2; for u = 2 and k = 3, this gives a 2.25× arithmetic-complexity reduction while increasing additions.The method is especially efficient for stride-1 convolutions with small kernels, particularly K ≤ 3.
Winograd Transform: FPGA Winograd accelerators combine loop unrolling, tiling, and on-chip intermediate buffers to raise throughput and improve memory-bandwidth utilization.Reported results include 46 GOPs for AlexNet convolution layers and 2.94 TOPs for VGG convolutional layers, with the latter using 5.7× less power than TitanX at half its performance.
Fast Fourier Transform: FFT transforms 2D convolutions into frequency-domain element-wise matrix multiplications, reducing arithmetic complexity to O(W^2 log_2(W)) and favoring large kernels such as K > 5.Overlap-and-Add can further reduce complexity to O(W log_2(K)) when the signal is much larger than the filter; an FPGA implementation reports 83 GOPs for AlexNet.

4 Data-path Optimizations for FPGA-Based CNN Accelerators

FPGA CNN data-path optimization balances parallel execution against limited computational, on-chip memory, and external-memory resources. The surveyed approaches range from configurable PE architectures and loop transformations to dataflow mappings, with reported gains depending on workload and hardware capacity.

4 Data-path Optimizations for FPGA-Based CNN Accelerators: FPGA resource limits prevent fully unrolling many CNN execution patterns, so accelerators reuse processing elements temporally.This motivates configurable data paths and loop-based mappings rather than complete spatial expansion.
4.1 Systolic Arrays: Static systolic arrays support only bounded kernel sizes, waste DSP capacity on smaller kernels, and remain limited by off-chip memory bandwidth.For one reported design, 3 × 3 convolutions used only 9% of DSP blocks.
4.2 SIMD Accelerators: Configurable SIMD accelerators stream feature maps and weights through programmable PEs, making PE count, DSP allocation, cache size, and scheduling design variables.The mapping objective is to maximize computational throughput while fitting device resources and data movement constraints.
4.2 Loop Optimization: Loop unrolling increases parallel execution but consumes resources, while tiling partitions feature maps and weights so blocks fit on-chip buffers and can be reused.Unrolling factors determine PE and arithmetic resources; tiling factors determine buffer sizes.
4.2 Loop Optimization: Design-space models predict resource allocation, throughput, and bandwidth for loop factors, which are explored using brute-force or heuristic searches under FPGA constraints.Roofline analysis is used to select feasible designs matching available computational throughput and memory bandwidth.
4.2.4 FPGA Implementations: 61.62 GOPs were reported for AlexNet convolutional layers, while later implementations achieved ×2.2 and ×13.4 improvements through fixed-point arithmetic, RTL design, pipelining, and larger FPGA resources.The ×13.4 improvement used a deeply pipelined cluster of four Virtex7-XV960t devices with 16-bit fixed-point arithmetic.
4.2.4 FPGA Implementations: Unrolling large-kernel loops improved AlexNet throughput by ×1.36, but becomes ineffective for recent small-kernel CNNs and can increase control complexity.The reported ×1.36 comparison used the same VX485T device and 32-bit floating-point arithmetic.
4.2.4 FPGA Implementations: Expanding design-space exploration to additional loops produced a ×1.22 improvement, while cross-layer tiling reduced DRAM accesses by 95% at the cost of 362KB extra on-chip memory.These results illustrate the trade-off between computational parallelism, memory traffic, and FPGA storage.

5 Approximate Computing of CNN Models

Approximate computing accelerates CNN inference on FPGAs by trading a minimal amount of accuracy for higher computational throughput and energy efficiency. The surveyed strategies reduce arithmetic precision or the number of model operations, and can be integrated with training.

5 Approximate Computing of CNN Models: Approximate computing trades a minimal amount of CNN accuracy for improved computational throughput and energy efficiency on FPGAs.The paper identifies approximate computing as complementary to computational transforms and data-path optimization.
5 Approximate Computing of CNN Models: Approximate arithmetic reduces precision, while model-level methods reduce operations without critically affecting modeling performance.Both strategies can be integrated into the learning phase to jointly optimize the resulting model and execution.

5.1 Approximate Arithmetic for CNNs

Approximate arithmetic reduces CNN representation and computation costs through fixed-point, dynamic fixed-point, binary, ternary, and stochastic methods. FPGA implementations exploit compact representations and logic-based operations, trading resource efficiency and throughput against accuracy or runtime constraints.

Fixed point arithmetic: Fixed-point quantization reduces CNN operand precision using compact bit-widths selected for numerical range and desired precision.Compared with floating point, compact fixed-point computing improves FPGA hardware utilization and power consumption.
Dynamic Fixed Point: Dynamic fixed point assigns distinct scale factors and bit-widths to network parts to balance accuracy loss and computational load.Reported AlexNet accuracy drops were 1%, 0.4%, and 2% for different FPGA accelerators using layer-specific quantization.
Binary and pseudo-Binary Nets: Binary neural networks replace multiplications with XNOR and pop-count operations, while batch normalization reduces information loss before binarization.FPGA implementations reported 207.8 GOP/s at 4.7 W on CIFAR10, 2.4 TOP/s at 11 W on CIFAR10, and 1.9 TOP/s on ImageNet.
Binary and pseudo-Binary Nets: Ternary neural networks extend extreme quantization beyond binary representations, reaching 8.36 TMAC/s at 13 W for CIFAR10 classification.The reported result concerns peak performance for an FPGA ternary accelerator.
Stochastic Computing: Stochastic computing represents numbers as random bit sequences and maps multiplication to an AND gate, but long bit-streams increase runtime.An MNIST FPGA accelerator achieved 15.44 TOP/s with a 2.40% misclassification rate, while representing an n-bit number requires a 2^n-length stream.

5.2 Reduce Computations in CNNs

CNN computation can be reduced through pruning and low-rank approximation, then exploited in FPGA data paths by skipping zero operations or using compressed representations. These methods can substantially reduce parameters and cycles, but sparse formats are effective only within particular sparsity regimes.

Weight Pruning: Weight pruning removes low-value or energy-inefficient weights and fine-tunes the remaining model to preserve classification accuracy.One AlexNet implementation removed 53% of convolutional weights and 85% of fully connected weights with less than 0.5% accuracy loss.
Low-Rank Approximation: Low-rank approximation replaces filters or fully connected weight matrices with fewer components, reducing multiplication or parameter counts.A VGG16-SVD model achieved 87.96% ImageNet accuracy with 63% fewer parameters.
FPGA Implementations: FPGA sparse processing can skip multiplications involving zero weights, using fully unrolled designs or sparse matrix formats such as CSC and CSR.The suitable representation depends on device resources, matrix dimensions, and the likelihood of detecting zero computations.
FPGA Implementations: A zero-skip scheduler reduced cycle count 4x for 85%-sparse AlexNet layers and projected 12 TOP/s on Intel Stratix10 FPGAs.The projected throughput was reported as 10% above state-of-the-art GPU computational throughput.

6 Conclusion

The survey compares neuron-level and network-level methods for mapping CNNs onto FPGAs. It concludes that FPGA-specific algorithmic and datapath optimizations can be jointly applied, with approximate computing remaining central to acceleration.

Conclusion: The survey compares convolutional and fully connected layer optimizations with approximate computing and datapath optimization methods for FPGA CNN inference.The methods exploit custom data types, local data streams, and dedicated processors.
Conclusion: Algorithmic and datapath optimizations can be jointly implemented to obtain additive hardware performance gains.This conclusion covers the combined use of the FPGA degrees of freedom reviewed throughout the paper.
Conclusion: Weight pruning and fixed-point computation are identified as approximate computing techniques particularly suited to overparameterized CNNs.The survey characterizes approximate computing as a key contributor to CNN acceleration on hardware.