PaperlayerView exact revision on arXiv

Source-linked AI summary

FPGA-based Accelerators of Deep Learning Networks for Learning and Classification: A Review

Ahmad Shawahna, Sadiq M. Sait, Aiman El-Maleh

arXiv:1901.00121v1cs.NEcs.ARcs.CVcs.LG

TL;DR

CNNs deliver effective image-recognition performance but demand intensive computation and memory bandwidth as models grow. This paper reviews FPGA-based acceleration techniques, design challenges, and recommendations, reporting substantial performance and power-efficiency gains across representative CNN implementations. Its scope centers on recent FPGA accelerator techniques and their design-space and resource-utilization strategies.

  • Problem

    Growing CNNs require intensive computation, memory bandwidth, and resources, while ASICs lack flexibility and large on-chip-weight designs cannot support realistic large-scale models.

  • Method

    The paper reviews FPGA techniques including parallelism, loop tiling and unrolling, data reuse, pipelining, reduced data sizes, RTL-generation tools, and design-space analytics.

  • Results

    Reported FPGA implementations achieved strong CNN acceleration, including 5.5× and 9.5× speedups for VGG-16 and AlexNet on P395-D8, and up to 473.4 GOPS for VGG-S.

  • Takeaways & Limitations

    The review identifies FPGA parallelism, memory reuse, pipelining, data-size optimization, and automated RTL generation as practical directions for efficient CNN acceleration.

Abstract

from arXiv · show

Due to recent advances in digital technologies, and availability of credible data, an area of artificial intelligence, deep learning, has emerged, and has demonstrated its ability and effectiveness in solving complex learning problems not possible before. In particular, convolution neural networks (CNNs) have demonstrated their effectiveness in image detection and recognition applications. However, they require intensive CPU operations and memory bandwidth that make general CPUs fail to achieve desired performance levels. Consequently, hardware accelerators that use application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and graphic processing units (GPUs) have been employed to improve the throughput of CNNs. More precisely, FPGAs have been recently adopted for accelerating the implementation of deep learning networks due to their ability to maximize parallelism as well as due to their energy efficiency. In this paper, we review recent existing techniques for accelerating deep learning networks on FPGAs. We highlight the key features employed by the various techniques for improving the acceleration performance. In addition, we provide recommendations for enhancing the utilization of FPGAs for CNNs acceleration. The techniques investigated in this paper represent the recent trends in FPGA-based accelerators of deep learning networks. Thus, this review is expected to direct the future advances on efficient hardware accelerators and to be useful for deep learning researchers.

I. INTRODUCTION

Deep learning and CNNs have advanced image and other data-driven applications, but increasingly complex networks create substantial computational and memory demands. The paper reviews FPGA acceleration techniques as a flexible, energy-efficient response to these demands.

  • Deep Learning: Deep learning learns network structures and weights from data, enabling systems to address complex problems using layered representations.Layers transform inputs into progressively more abstract features.
  • CNN Applications: CNNs achieved strong computer-vision results, with ILSVRC classification accuracy reaching 88.8%, 93.3%, and 96.4% in 2013, 2014, and 2015.AlexNet’s 2012 success reduced ImageNet classification error from 26% to 15%.
  • Computational Challenges: Larger CNNs require billions of operations, millions of parameters, and substantial resources for training and evaluation, challenging general-purpose processors.The paper notes that increasing layers improves accuracy and real-time recognition but also increases structural complexity.
  • Hardware Acceleration: GPUs provide high bandwidth and throughput but consume substantial power and benefit from processing large image batches in parallel.This creates challenges for cloud services and battery-operated devices.
  • Hardware Acceleration: FPGAs and ASICs have fewer resources than GPUs but can provide moderate performance with lower power; FPGAs also offer parallel, adaptable acceleration.The review focuses on FPGA techniques and recommendations for improving CNN acceleration and simplifying FPGA use.
  • Memory Considerations: CNN accelerator designs must account for data movement because off-chip DRAM access can cost more energy than computation.Efficient architectures therefore need to consider both time and power.

II. BACKGROUND AND TERMINOLOGY

CNNs combine convolution, activation, normalization, and other operations in layered networks. Convolution dominates computation, motivating loop unrolling and related FPGA optimizations.

  • CNN Structure: A CNN commonly stacks convolutional and ReLU layers with pooling, followed by fully connected classification layers.The paper introduces these operations as core CNN terminology.
  • 1) Convolution (CONV): Convolution slides a k × k kernel over an input feature map, multiplying and summing values to produce output feature neurons.The operation scans the feature map using a stride S.
  • 1) Convolution (CONV): CONV computation has four loop levels: kernel MACs, input-feature-map scanning, input-feature-map accumulation, and output-feature-map processing.These are Loop-1 through Loop-4, respectively.
  • 1) Convolution (CONV): CONV layers often constitute more than 90% of CNN operations, making loop unrolling a central acceleration technique.Unrolling can target kernel dimensions, feature-map dimensions, and input or output feature-map counts.
  • 2) Activation Functions (AFs): ReLU is defined as f(x) = max(x, 0) and is described as faster to train and less computationally complex than sigmoid and tanh.It also does not require input normalization to prevent saturation.
  • 3) Normalization: Normalization adjusts neuron responses within or across neighboring features, particularly for unbounded ReLU activations.The adjustment depends on neighboring neurons.

4) Pooling

Pooling reduces CNN representation size and computation by spatially subsampling feature maps. The section also situates pooling within common CNN architectures such as AlexNet and VGG.

  • 4) Pooling: Pooling progressively reduces spatial representation size, thereby reducing network parameters and computation.Pooling layers are periodically inserted between successive convolutional layers.
  • 4) Pooling: Common max pooling uses a 2 × 2 filter and retains the maximum of four samples, discarding 75 percent of activations.Pooling operates independently on each depth slice.
  • 4) Pooling: Some CNNs use average or minimum pooling in addition to the commonly used maximum operation.These alternatives are described as functions performed by pooling units.
  • CNN Architecture: A common CNN architecture repeats convolutional, ReLU, and pooling stages before fully connected layers produce class scores.The final fully connected layer serves as the classification layer.
  • Known Networks: AlexNet contains five convolutional layers, three fully connected layers, ReLU activations, and three pooling layers, and won the 2012 ImageNet challenge.It classifies 224 × 224 color images into 1,000 classes.
  • Known Networks: VGG resembles AlexNet in fully connected-layer count but organizes convolutional layers into five groups whose sizes depend on the model version.The paper identifies multiple commonly used VGG variants.

2 normalization

ResNets use deep, irregular directed-acyclic-graph structures with shortcut connections, unlike the sequential structures of AlexNet and VGG.

  • ResNets: ResNets contain shortcut connections that compute residual functions through non-adjacent layers.These connections contribute to their irregular and complex structures.
  • ResNets: ResNet depths range from 50 to 1000 convolutional layers, substantially exceeding the shallower sequential architectures described for AlexNet and VGG.ResNet-50 and ResNet-152 are widely used for image classification.
  • ResNets: Unlike AlexNet and VGG, whose layers are connected sequentially, ResNet interconnections form a directed acyclic graph.ResNet-50/152 contain 53/155 convolutional layers, respectively.

C. FIELD PROGRAMMABLE GATE ARRAYS (FPGAS)

FPGAs provide programmable hardware with fine-grained parallelism, specialized embedded resources, and energy-efficient execution for deep-learning acceleration.

  • FPGAs combine configurable logic blocks, programmable interconnections, programmable I/O cells, and embedded DSP, BRAM, and LUT resources.DSP blocks support arithmetic-intensive operations such as multiply-and-accumulate.
  • FPGAs support flexible fine-grained parallelism for computationally intensive applications.
  • FPGA acceleration models include data streaming, associative computing, highly parallel memory access, standard hardware structures, and functional parallelism.Examples of standard structures include FIFO buffers, stacks, and priority queues.
  • FPGAs offer high performance per Watt and are used for deep-learning acceleration because their flexible architectures support substantial parallelism and high execution speeds.These properties make them suitable for battery-operated devices and cloud servers.
  • OpenCL programming models increase FPGA accessibility, while feed-forward algorithms support deeply pipelined, inherently multithreaded circuits and partial dynamic configuration.

D. CHALLENGES OF FPGA-BASED IMPLEMENTATION OF DEEP LEARNING NETWORKS

FPGA implementations of deep-learning networks must manage large storage, bandwidth, computation, and resource-allocation demands that vary across CNN layers.

  • CNN implementations require substantial storage, external memory bandwidth, and computational resources reaching billions of operations per second.AlexNet has over 60 million parameters and requires 250 MB for 32-bit floating-point weights.
  • Because commercial FPGAs cannot store all network weights, implementations must use external memory and transfer weights during computation.Insufficient resource sharing can prevent a network from fitting on the FPGA.
  • VGG-16 has 138 million weights and requires over 30 GOPS, illustrating how complexity increases for deeper CNN models.
  • Different CNN layers require different degrees of intra-output and inter-output parallelism and different memory-access patterns.Intra-output parallelism computes one output image from multiple input-kernel convolutions, whereas inter-output parallelism computes multiple output feature maps concurrently.
  • Efficient FPGA implementations require computational-resource reuse, internal storage of partial results, and careful ordering of operations and parallelism selection.

III. ACCELERATION OF DEEP LEARNING NETWORKS: CURRENT STATUS

The review covers CNN compression and hardware acceleration approaches for deep-learning algorithms, including ASIC- and FPGA-based techniques.

  • The section discusses CNN compression techniques and hardware acceleration techniques for deep-learning algorithms and CNNs using ASICs and FPGAs.Compression is introduced because it affects CNN implementation complexity.

A. CNNS COMPRESSION

CNN compression methods reduce operations, memory footprint, or storage requirements while aiming to preserve accuracy and implementation practicality.

  • CNN compression techniques target lower operation counts and memory footprints with minimal impact on accuracy.
  • SVD and filter clustering accelerated convolutional layers by a factor of 2 and reduced network-weight memory requirements.The reported speedup was relative to a CPU Eigen3-based library implementation.
  • Pruning redundant and less influential connections achieved 9× compression for AlexNet and 13× for VGG-16 with zero accuracy loss for both.
  • Deep compression combined pruning, trained-weight quantization, and Huffman coding to reduce AlexNet and VGG-16 storage requirements by 35× and 49×, respectively, without affecting accuracy.

B. ASIC-BASED ACCELERATORS

ASIC accelerators exploit specialized hardware and locality to improve neural-network performance, but their fixed designs limit adaptation to changing applications and network sizes. FlexFlow addresses CNN workload diversity by supporting multiple parallelism styles and reports higher utilization and performance than representative architectures.

  • ASIC accelerator families: DianNao minimizes memory transfers by exploiting neural-network locality, while fixed-point arithmetic reduces computation resources and memory footprint.DianNao uses 16-bit fixed-point arithmetic units; 16-bit multipliers reduce multiplier area and power relative to 32-bit multipliers.
  • ASIC accelerator families: DaDianNao extends the DianNao family with multi-chip capacity for on-chip storage of all CNN weights and uses 16-bit fixed-point inference.The design targets large-scale deployments requiring greater memory capacity.
  • ASIC accelerator families: PuDianNao accelerates multiple machine-learning techniques by extracting locality and computational primitives with on-chip storage and seven novel functional units.The reported comparison states that PuDianNao is 1.20ˆ and 128.41ˆ faster and energy-efficient, respectively, than an NVIDIA K20M GPU accelerator.
  • ASIC limitations: ASIC accelerators cannot efficiently adapt to different application demands, and designs storing all weights on chip cannot support realistic large-scale CNN models.The review also identifies long ASIC development cycles and limited flexibility for varying deep-learning network designs.
  • FlexFlow: CNN workloads expose feature-map, neuron, and synapse parallelism, but systolic, 2D-mapping, and tiling architectures generally exploit only one type.The dominant parallel type varies with input and output feature-map counts, output feature-map size, and kernel size.
  • FlexFlow: FlexFlow supports all processing styles through flexible data paths, modified processing elements, and hierarchical dataflow across CNN layers.Its processing elements are arranged in rows, with each row completing one convolution and serving one output neuron; the architecture determines layer-specific unrolling parameters.
  • Evaluation: Over 80% computing-resource utilization and over 420 GOPS at 1 GHz were reported for FlexFlow, which also outperformed other architectures in data reusability and power efficiency.The comparison used six practical workloads, including AlexNet and VGG, against systolic, 2D-mapping, and tiling architectures.

C. FPGA-BASED ACCELERATORS

FPGA-based CNN accelerators combine parallel processing, optimized dataflows, buffering, compression, and memory-centric techniques to improve throughput and energy efficiency. Reported results span substantial gains over CPU and GPU baselines, while some designs remain limited by network coverage or resource constraints.

  • Parallel convolution engines: CNP’s 3 × 3 convolution module performs K^2 MAC operations simultaneously per clock cycle through parallel vector arithmetic units.The module supports convolution, pooling, and nonlinear activation operations.
  • Parallel convolution engines: MAPLE uses a 2D grid of vector processing elements and smart memory blocks to support parallel workloads and reduce CNN off-chip memory traffic by 76×.Its smart memory processes intermediate data on-the-fly using in-memory processing.
  • Performance and energy efficiency: Specialized architectures achieve high application-level throughput, with DC-CNN reaching up to 30 frames per second while consuming 14 Watts versus more than 150 Watts for the GPU.DC-CNN reported speedups from 4.0× to 6.5× across five vision workloads.
  • Dataflow and system optimization: Reported limitations include architectures evaluated without fully connected layers and trade-offs such as DeepBurning being 1.13× slower while 1.45× more energy-efficient.The DC-CNN and coprocessor evaluations omitted FC layers, limiting applicability to full CNN models.
  • Performance and energy efficiency: FPGA accelerator designs report large gains across workloads, including 11× faster memory-centric processing, 17.42× CPU speedup, and 24.6× higher energy efficiency.The 17.42× speedup and 24.6× energy-efficiency result were reported for a Vivado-based implementation on a VC707 FPGA.
  • Dataflow and system optimization: Design-space and dataflow optimization improve resource use: Caffeine selects IMM for CONV and WMM for FC layers, while multi-CLP designs raise throughput over single-CLP baselines.Caffeine’s roofline-based selection especially benefits data reuse and bandwidth utilization at small batch sizes.

IV. METAHEURISTICS IN THE DESIGN OF CONVOLUTIONAL NEURAL NETWORKS

CNN designs are increasingly treated as optimization problems because architecture choices affect complexity and may be improved with limited accuracy impact.

  • CNN structures are designed by selecting layer counts, fully connected layers, feature-map sizes, and other operators.
  • Eliminating many fully connected-layer weights can have minimal impact on accuracy, motivating optimization of expert-designed CNN structures.
  • CNN design includes NP-hard combinatorial problems such as structure design, weight and bias selection, and runtime-variable optimization.

A. CNN STRUCTURE OPTIMIZATION

Genetic algorithms can search CNN structure spaces by encoding candidate networks and evaluating their recognition accuracy after training.

  • Genetic algorithms encode CNN structures as fixed-length binary chromosomes to search for networks minimizing error rate.
  • Candidate structures are assessed by recognition accuracy obtained through training and validation, with demonstrations on MNIST and CIFAR-10.

B. CNN WEIGHTS AND BIAS VALUES OPTIMIZATION

Metaheuristic optimization can tune last-layer weights and biases to improve accuracy while minimizing estimated error.

  • Simulated annealing, differential evolution, and harmony search optimize weights and biases in the CNN’s last layer.
  • The optimization stops when the iteration limit is reached or the cost function falls below a prespecified value.

C. CNN DESIGN VARIABLES OPTIMIZATION

CNN accelerator design can be optimized by exploring parallelism and resource-allocation variables under FPGA constraints. A genetic-algorithm methodology achieved peak convolution and whole-network performance for AlexNet and VGG implementations.

  • Suda et al. formulate FPGA accelerator exploration around convolution, normalization, pooling, and fully connected design variables under hardware constraints.
  • The variables include filter size, convolution vectorization, normalization operations per cycle, and parallel pooling outputs.
  • The objective is to minimize runtime using analytical models for convolution, normalization, pooling, and fully connected layers, subject to DSP, logic, and memory constraints.
  • The analytical models were validated at selected design points through full synthesis and execution on the FPGA accelerator.
  • Exhaustive search is infeasible for large variable or resource spaces, so iterative nondeterministic heuristics can efficiently traverse the design space.
  • Peak performance was achieved for convolution operations and the entire CNN when optimizing AlexNet and VGG on two resource-different Altera Stratix-V platforms.
  • Large memory requirements and costly objective evaluation make heuristic CNN design challenging, motivating estimation techniques and analytical formulations.

V. SUMMARY AND RECOMMENDATIONS

The reviewed FPGA accelerators primarily improve CNN performance by exploiting parallel computation, data reuse, memory optimization, and automated hardware design. The paper recommends scalable, user-oriented tools that optimize architectures for available resources and desired performance.

  • Acceleration focus: CONV operations consume around 90% of computational time, so reviewed techniques prioritize parallel multiply-accumulate operations and efficient data access.Internal memory structures and data reuse reduce memory-bandwidth requirements for feature maps and weights.
  • Parallelization: Workload analysis, polyhedral dependence analysis, loop unrolling, and loop pipelining are used to organize convolution computations into parallel streams.Roofline-model accelerators determine layer-specific unrolling factors to utilize FPGA computational resources.
  • Computational reduction: Winograd transformation reduces convolutional computational complexity by around 50%, improving resource efficiency.
  • Parallelization: Multiple convolution-layer processors pipeline different layers, achieving layer-level parallelism and improving resource utilization over a single processor.
  • Memory optimization: Memory footprint and bandwidth are reduced through fractional-length selection, SVD for fully connected weights, tiling, data reuse, and efficient buffering.Tiling partitions large inputs into subsets that trade hardware cost against speedup, while buffering overlaps weight transfer with computation.
  • Automation: Automation tools use RTL libraries, configuration scripts, HLS, and loop unrolling to generate parameterized hardware for ASIC or FPGA platforms.Tables 4 and 5 list optimization mechanisms used by the reviewed techniques to maximize performance and throughput.
  • Recommendations: The paper recommends a user-friendly framework that reads CNN specifications and weights, performs resource-aware parallelism and data-reuse optimization, and generates hardware for a given FPGA.The framework should report resource utilization, memory size, bandwidth, and power, while analytical models recommend platforms for specified performance targets.

VI. CONCLUSION

The paper reviews FPGA acceleration techniques for deep learning, emphasizing CNNs because their applications require CPU- and memory-intensive processing. It synthesizes optimization methods, automation tools, analytical techniques, and recommendations for improving FPGA utilization.

  • Scope: The review focuses on recent FPGA techniques for accelerating deep learning networks, particularly CNNs used in image detection and recognition.
  • Acceleration techniques: The reviewed methods exploit loop tiling, loop unrolling, internal memory, data reuse, operation pipelining, and data-size optimization.These techniques target parallelism, lower memory footprint, and improved FPGA resource utilization.
  • Automation: RTL-generation tools automate hardware design, support design-space exploration, and suggest efficient hardware configurations.
  • Analysis and recommendations: The paper also reviews workload analysis, loop-unrolling selection, data-locality analysis, and non-deterministic heuristics for CNN hardware optimization.It concludes by summarizing the key features of FPGA-based techniques and recommending ways to improve FPGA use for CNN acceleration.
Loading 1901.00121v1…