Source-linked AI summary
Deep Learning on FPGAs: Past, Present, and Future
Griffin Lacey, Graham W. Taylor, Shawki Areibi
TL;DR
Deep learning achieves strong performance by learning from massive datasets, but its growing data and model sizes require better hardware acceleration than current GPU-centered solutions provide. This review examines FPGA-based acceleration, emphasizing flexible architectures, evolving high-level tools, and deployment considerations. It concludes that FPGAs can accommodate deep-learning trends while offering architectural freedom and efficient parallel computation, although design size and hardware-programming expertise remain constraints.
Problem
Deep-learning models require substantial data and compute power, creating a need for hardware acceleration that scales beyond current data and algorithm sizes.
Method
The review surveys deep learning on FPGAs from a hardware-acceleration perspective, covering FPGA architectures, design tools, current support, limitations, and future directions.
Results
FPGAs can accommodate deep-learning trends by exploiting pipeline parallelism, efficient power consumption, mature design tools, and architectural freedom for exploration.
Takeaways & Limitations
FPGAs are a promising alternative to GPUs and GPPs for deep-learning acceleration, particularly where customizable architectures and performance per watt matter.
Takeaways & Limitations
FPGA deployment remains constrained by design size and the hardware-specific knowledge required for low-level programming.
Abstract
from arXiv · showhide
The rapid growth of data size and accessibility in recent years has instigated a shift of philosophy in algorithm design for artificial intelligence. Instead of engineering algorithms by hand, the ability to learn composable systems automatically from massive amounts of data has led to ground-breaking performance in important domains such as computer vision, speech recognition, and natural language processing. The most popular class of techniques used in these domains is called deep learning, and is seeing significant attention from industry. However, these models require incredible amounts of data and compute power to train, and are limited by the need for better hardware acceleration to accommodate scaling beyond current data and model sizes. While the current solution has been to use clusters of graphics processing units (GPU) as general purpose processors (GPGPU), the use of field programmable gate arrays (FPGA) provide an interesting alternative. Current trends in design tools for FPGAs have made them more compatible with the high-level software practices typically practiced in the deep learning community, making FPGAs more accessible to those who build and deploy models. Since FPGA architectures are flexible, this could also allow researchers the ability to explore model-level optimizations beyond what is possible on fixed architectures such as GPUs. As well, FPGAs tend to provide high performance per watt of power consumption, which is of particular importance for application scientists interested in large scale server-based deployment or resource-limited embedded applications. This review takes a look at deep learning and FPGAs from a hardware acceleration perspective, identifying trends and innovations that make these technologies a natural fit, and motivates a discussion on how FPGAs may best serve the needs of the deep learning community moving forward.
1. INTRODUCTION
Deep learning has shifted AI toward learning composable representations from massive data, but scaling these models requires better hardware acceleration. The review presents FPGAs as a flexible, increasingly accessible alternative whose parallelism and performance per watt address researchers’ and application scientists’ needs.
- Deep learning automatically learns composable feature extraction systems from massive example datasets, producing strong results in vision, speech, and language tasks.
- Researchers and application scientists are constrained by the need for hardware acceleration that can scale beyond current data and algorithm sizes.
- GPUs currently dominate deep-learning acceleration, while OpenCL extends parallel programming across GPUs, GPPs, FPGAs, and DSPs.
- FPGAs offer flexible hardware configuration and often better performance per watt for deep-learning subroutines, including sliding-windows computation.
- Higher-abstraction tools, reconfigurability, and parallelism make FPGAs more accessible for model design and potentially faster execution.
- The review surveys FPGA support, identifies limitations, and recommends future hardware-acceleration directions for deep learning.
2. DEEP LEARNING
Deep learning includes feed-forward MLPs and spatially structured CNNs, with both architectures composed of layers that transform inputs into predictions. CNNs use local, shared filters and organized feature maps to process data with spatial or temporal continuity.
- 2.1 Multi-layer Perceptrons: MLPs are simple feed-forward deep networks whose layers transform inputs into predictions through weighted connections and nonlinear units.
- 2.1 Multi-layer Perceptrons: In MLPs, weights are adjusted during training, while ReLU nonlinearities prevent the network from collapsing into a linear transformation.
- 2.2 Convolutional Neural Networks: CNNs are especially common for pixel-based visual recognition and are designed for data with spatial or temporal continuity.
- 2.2 Convolutional Neural Networks: CNN feature maps are two-dimensional arrangements of units, and convolution layers apply learnable filters through local receptive fields shared across input locations.
- 2.2 Convolutional Neural Networks: CNNs typically use fully connected output layers to reduce organized feature-map information to decisions such as classification or regression.
3. FPGAs
FPGAs occupy a middle ground between general-purpose processors and fixed-function hardware, combining reconfigurability with hardware-level acceleration. The section describes FPGA architectures, high-level programming tools, OpenCL support, and deployment challenges for deep learning.
- FPGA Architecture: FPGAs combine integrated-circuit performance with the reconfigurable flexibility of general-purpose processors.They implement logic with flip-flops and lookup tables and may include hardened components for common functions.
- FPGA Architecture: FPGAs can reduce deep-learning bottlenecks through distributed on-chip memory, pipeline parallelism, and data paths that avoid traditional Von Neumann memory traffic.Partial dynamic reconfiguration can also reprogram part of the device while another part remains in use.
- Trade-offs: Unlike GPUs, FPGAs permit customized hardware architectures, but this flexibility incurs long place-and-route compile times that hinder rapid design iteration.The review identifies compile time as problematic for researchers who repeatedly modify designs.
- Programming Tools: High-level synthesis tools translate high-level designs into RTL or HDL code, reducing the need for low-level hardware programming.The review focuses on parallel-computing frameworks such as CUDA and OpenCL as a path connecting deep learning with FPGAs.
- Programming Tools: OpenCL provides standardized parallel programming across GPPs, GPUs, DSPs, and FPGAs, but FPGA implementations currently support only a subset of OpenCL functions.OpenCL is open source and royalty-free, whereas CUDA is proprietary to NVIDIA.
- Deployment Flow: FPGA OpenCL kernels typically compile offline in tens of minutes to hours, although reusable precompiled deep-learning kernels make this compromise more practical for deployment.Profiling, debugging, optimization, and software simulation can help researchers develop kernels despite long hardware compilation times.
4. A REVIEW OF CNNS ON FPGAS
FPGA CNN research has progressed from early size- and arithmetic-constrained implementations to substantially higher-throughput systems, while architecture-specific differences still complicate identifying optimal designs.
- Historical development: Modern FPGAs have enabled deep networks to be implemented on single-FPGA systems through higher density and hardened computational units.
- Historical development: Early FPGA CNN implementations were constrained by limited device size and used low-precision arithmetic because dense hardened multiply-accumulate units were unavailable.These constraints also made arithmetic slow.
- Performance: 134 images/second was reported for FPGA CNN forward propagation on ImageNet 1K at 25 W, roughly 3x the next closest competitor.An Arria 10 projection estimated roughly 233 images/second at roughly the same power.
- Performance: GPU implementations achieved 500-824 images/second at 235 W, compared with the reported FPGA result on Stratix V D5.
- Architecture: FPGA CNN designs commonly use off-chip memory, configurable software layers, buffered input and output, and parallel processing elements for convolution.
- Architecture: Differences in memory systems, data transfer, soft-cores, LUT types, frequencies, and FPGA devices make specific optimal architecture decisions difficult to determine.The passage states that more research is needed.
5. LOOKING FORWARD
Future FPGA deep learning research centers on scalability, broader software accessibility, flexible model exploration, and efficient heterogeneous computing for growing workloads.
- Scalability: FPGA scalability depends on larger memory, smaller feature sizes, improved interconnects, and support for multi-FPGA configurations.The FPGA landscape may also see integration into consumer and data-center applications.
- Scalability: Design tools are expected to move toward higher abstraction and software-like experiences to attract a wider range of users.
- Software tools: OpenCL efforts are extending deep learning software beyond CUDA, although no deep learning tools explicitly supporting FPGAs were identified in the review.Examples include unofficial OpenCL support for Caffe, Torch, and Theano, plus the DeepCL library.
- Software tools: Caffe was recommended for newcomers because of its popularity, support, easy-to-use interface, and model zoo of pre-trained models.
- Model exploration: Fixed architectures limit exploration of possible model configurations, whereas FPGAs can be programmed with a different hardware structure at runtime.
- Low-power compute clusters: FPGAs may suit heterogeneous large-scale workloads through application-dependent reconfigurability and attractive performance per watt.The passage frames this as a possible alternative to GPU clusters with Infiniband and MPI.
6. CONCLUSION
FPGAs are presented as an attractive alternative for deep learning hardware because they combine pipeline parallelism, efficient power consumption, mature design tools, and architectural freedom.
- Conclusion: FPGAs exploit pipeline parallelism and efficient power consumption, offering a distinct advantage over GPUs and GPPs for common deep learning practices.
- Conclusion: Mature design tools make integrating FPGAs into popular deep learning frameworks possible.
- Conclusion: FPGA architectural freedom can accommodate deep learning trends while supporting hardware exploration and research.