PaperlayerView exact revision on arXiv

Source-linked AI summary

Maximizing CNN Accelerator Efficiency Through Resource Partitioning

Yongming Shen, Michael Ferdman, Peter Milder

arXiv:1607.00064v2cs.AR

TL;DR

CNN accelerators using one CLP struggle with layers of different dimensions, wasting FPGA arithmetic resources. The paper partitions the same resource budget among specialized concurrent CLPs and develops an automated optimization method. This achieves higher throughput across AlexNet, SqueezeNet, and GoogLeNet than the state-of-the-art Single-CLP approach.

  • Problem

    A single CLP processes differently dimensioned CNN layers sequentially, limiting arithmetic-unit utilization and accelerator performance.

  • Method

    The method optimizes a partition of FPGA resources into multiple specialized CLPs, then uses their computed dimensions to construct a complete CNN implementation.

  • Results

    3.8x higher throughput is achieved for AlexNet, with speedups of 2.2x for SqueezeNet and 2.0x for GoogLeNet.

  • Takeaways & Limitations

    Partitioning FPGA resources among specialized CLPs improves dynamic utilization and throughput compared with a Single-CLP design using the same resource budget.

  • Takeaways & Limitations

    The Multi-CLP approach can require one CLP per convolutional layer, reducing each CLP’s BRAM and increasing memory-bandwidth, coordination, and overhead costs.

Abstract

from arXiv · show

Convolutional neural networks (CNNs) are revolutionizing machine learning, but they present significant computational challenges. Recently, many FPGA-based accelerators have been proposed to improve the performance and efficiency of CNNs. Current approaches construct a single processor that computes the CNN layers one at a time; the processor is optimized to maximize the throughput at which the collection of layers is computed. However, this approach leads to inefficient designs because the same processor structure is used to compute CNN layers of radically varying dimensions. We present a new CNN accelerator paradigm and an accompanying automated design methodology that partitions the available FPGA resources into multiple processors, each of which is tailored for a different subset of the CNN convolutional layers. Using the same FPGA resources as a single large processor, multiple smaller specialized processors increase computational efficiency and lead to a higher overall throughput. Our design methodology achieves 3.8x higher throughput than the state-of-the-art approach on evaluating the popular AlexNet CNN on a Xilinx Virtex-7 FPGA. For the more recent SqueezeNet and GoogLeNet, the speedups are 2.2x and 2.0x.

1 INTRODUCTION

CNN accelerators face rising computational demands, while a single CLP is inefficient across layers with different dimensions. The proposed Multi-CLP approach partitions FPGA resources among specialized processors and improves throughput.

  • FPGAs are attractive for CNN acceleration because they combine programmable parallelism with power-efficient computation.
  • A conventional accelerator uses one parameterized CLP to process all CNN layers sequentially, optimizing aggregate throughput across the layer ensemble.
  • Different layer dimensions make the jointly optimized CLP sub-optimal for individual layers, causing dynamic FPGA resource underutilization.
  • Multi-CLP partitions FPGA resources among specialized CLPs that process different images concurrently and match distinct layer dimensions.The specialized CLPs can process layers with little idle hardware while operating in parallel.
  • 3.8x higher throughput is achieved for AlexNet on a Xilinx Virtex-7, with speedups of 2.2x for SqueezeNet and 2.0x for GoogLeNet.The results also report over 90% arithmetic unit utilization, sometimes close to 100%.

2 CNN BACKGROUND

CNNs compute convolutional layers by combining input feature maps with learned filters to produce output feature maps. Non-linear and fully connected layers complete the typical recognition pipeline.

  • Typical CNNs interleave convolutional layers with non-linear layers such as sub-sampling or activation functions.
  • CNNs commonly end with one or more fully connected layers that perform multiple dot-products.
  • A convolutional layer takes N input feature maps and convolves them with filters to produce output feature maps.Each of M filter sets contains N × K × K weights and produces one output feature map.
  • Each output point is computed as a dot-product between filter weights and a corresponding region of the input feature maps.The region slides across the inputs to compute all points in an output feature map.

3 RESOURCE UTILIZATION PROBLEM

Single-CLP accelerators reuse one computational grid across layers with different dimensions, creating arithmetic-unit underutilization. Loop transformations and tiling define the CLP, but mismatched tile and layer sizes limit performance.

  • 3 RESOURCE UTILIZATION PROBLEM: A conventional CLP computes CNN convolutional layers sequentially using the same hardware despite differences in layer dimensions.The dimensions include M, N, R, C, K, and S.
  • 3.1 State of the Art Design: Loop reordering, tiling, and unrolling transform the convolution loops into a datapath intended to increase throughput and reduce data transfer.On-chip input, output, and weight buffers support tiled computation, while double-buffering overlaps transfers with computation.
  • 3.1 State of the Art Design: Tm vector dot-product units of width Tn produce TmTn multipliers and adders through unrolling the inner loops.The tiling factors Tr, Tc, Tm, and Tn control data transfers and compute-module construction.
  • 3.2 Arithmetic Unit Utilization Problem: A single CLP optimized for collective layer performance can remain poorly matched to individual layers, leaving arithmetic units idle.The utilization penalty is quantified by the fraction of time the arithmetic units perform work rather than remain idle.
  • 3.2 Arithmetic Unit Utilization Problem: 22.2% arithmetic unit utilization occurs for SqueezeNet layer two when tile parameters Tn,Tm mismatch loop sizes N,M; overall SqueezeNet utilization is 76.4%.For layer one, utilization is 33.3% because N = 3 is less than Tn = 9.
  • 3.2 Arithmetic Unit Utilization Problem: Less than 24% overall arithmetic unit utilization is observed for AlexNet on a Virtex-7 690T using 16-bit fixed-point arithmetic.Fixed-point arithmetic allows more arithmetic units within the same DSP budget, exacerbating mismatch effects.

4 MULTI-CLP DESIGN

Multi-CLP accelerators partition FPGA resources among specialized processors, improving layer compatibility and utilization while balancing throughput, memory, and resource costs.

  • 4.1 Multi-CLP Accelerators for CNNs: Multi-CLP partitions resources among smaller CLPs with different sizes, allowing processors to match distinct CNN layer dimensions more closely.Layers assigned to the same CLP can be processed sequentially, while different CLPs operate concurrently on independent inputs.
  • 4.1 Multi-CLP Accelerators for CNNs: A static layer-to-CLP assignment minimizes the number of processors while permitting each CLP to compute multiple CNN layers.Layers assigned to one CLP need not be adjacent in the CNN structure.
  • 4.1 Multi-CLP Accelerators for CNNs: Latency can be reduced by constraining each CLP to adjacent layers, but this lowers throughput.Under this constraint, a CLP can process multiple layers for one image within a single epoch.
  • 4.1 Multi-CLP Accelerators for CNNs: System throughput is limited by the CLP with the longest assigned workload, so optimization targets balanced schedules and compatible layer dimensions.The design must also account for on-chip memory allocation, bandwidth, and arithmetic-unit utilization.
  • 4.2 Modeling CLP Cost and Performance: The design model estimates CLP resource use and speed from processor dimensions, layer tiling parameters, and CNN layer dimensions.It models DSP slices, BRAMs, memory bandwidth, and computation cycles, including bandwidth-bound execution.
  • 4.2 Modeling CLP Cost and Performance: DSP and BRAM models account for arithmetic-unit structure, memory banking, buffering, ports, capacity, word size, and layer-specific buffer requirements.Input and weight buffers are banked and double-buffered; weight-bank capacity is determined by the largest filter size among assigned layers.

5 DESIGN AND IMPLEMENTATION

The implementation uses an HLS-generated accelerator template whose parameters define CLP computation, buffering, and streaming interfaces, with pipelined data movement and computation.

  • 5 DESIGN AND IMPLEMENTATION: The optimizer produces parameters for a C++ HLS template that Vivado HLS compiles into synthesizable Verilog.The template supports arbitrary CNN layer sets, resource budgets, and data types.
  • 5 DESIGN AND IMPLEMENTATION: Nine parameters configure CLP compute dimensions, on-chip buffers, and AXI stream port counts for input, weight, and output transfers.Each CLP also receives an AXI4-Lite start interface and AXI stream data interfaces.
  • 5 DESIGN AND IMPLEMENTATION: The accelerator executes bias reads, input reads, weight reads, computation, and output writes using an HLS template.The design state machine runs four nested loops after retrieving arguments and computing derived variables.
  • 5 DESIGN AND IMPLEMENTATION: DATAFLOW pipelines operations inside the n loop, while ping-pong buffers overlap data transfer with computation.Input and weight data are read for subsequent iterations during computation, and output feature maps are double-buffered.
  • 5 DESIGN AND IMPLEMENTATION: Transfer functions use maximum-length bursts and partition transfers across multiple ports to reduce port idle time and burst count.Weight transfers prioritize the n dimension over the m dimension because CLP designs have Tn smaller than Tm.
  • 5 DESIGN AND IMPLEMENTATION: The compute pipeline unrolls the Tm and Tn loops and partitions arrays across memory banks for concurrent access.The K × K loops are placed outermost to avoid loop-carry dependencies.

6 EVALUATION

The evaluation compares Single-CLP and Multi-CLP designs across CNNs, FPGA targets, and arithmetic formats, using modeled and implemented results. Multi-CLP generally improves utilization and throughput, while exposing resource and bandwidth trade-offs.

  • Utilization and throughput: Multi-CLP improves dynamic arithmetic unit utilization over Single-CLP in all evaluated cases, with gains from 1.01x for VGGNet-E to 3.8x for AlexNet.The evaluation covers four networks, two Xilinx Virtex-7 FPGAs, and both single-precision floating-point and 16-bit fixed-point arithmetic.
  • AlexNet comparison: 1.31x and 1.54x throughput improvements are obtained on the 485T and 690T FPGAs for AlexNet floating-point designs using equal arithmetic-unit counts.Arithmetic-unit utilization rises from 72.6% to 95.1% on 485T and from 64.0% to 98.9% on 690T.
  • Resource trade-offs: Multi-CLP throughput can be preserved while trading BRAM capacity against off-chip bandwidth, allowing designs to adapt to specific resource constraints.Adding buffers saves bandwidth, whereas reducing buffer sizes lowers BRAM use at the cost of higher bandwidth.
  • SqueezeNet comparison: 2.33x higher throughput is obtained for 16-bit fixed-point SqueezeNet, although Multi-CLP designs require 1.23x and 1.32x more BRAMs than Single-CLP designs.The two Multi-CLP designs also require lower off-chip bandwidth than the corresponding Single-CLP designs.
  • Scaling: 1.3x to 3.3x improvement is observed as available DSP slices increase from 2,240 to 9,600, indicating larger gains on more resource-rich FPGAs.

7 RELATED WORK

Related work spans flexible ASIC scheduling, layer-specific FPGA modules, alternative dataflow and numerical techniques, and automated accelerator-generation frameworks. These approaches address utilization, bandwidth, storage, or implementation flexibility, while Multi-CLP resource partitioning is presented as compatible with several of them.

  • Alternative accelerator architectures: Eyeriss couples a compute grid with a network-on-chip to enable flexible CNN scheduling and limit arithmetic-unit underutilization.The supplied passages also note that incompatibility between layer dimensions and the compute grid can still cause underutilization.
  • Alternative accelerator architectures: An FPGA design with one module per AlexNet layer can achieve high arithmetic-unit utilization but stores all intermediate data on chip, limiting supported network size.The passage further states that one module per layer does not work well for larger networks.
  • Orthogonal techniques: Other accelerator studies explore 2D convolvers, FMA units, zero-skipping, bandwidth and buffer reduction, quantization, matrix decomposition, precision reduction, fused layers, and batch-size optimization.The cited passage groups these techniques by computation, data movement, numerical representation, and layer or batch organization.
  • Orthogonal techniques: C-Brain transforms a stride-S convolution into S stride-1 convolutions, but applies only when the layer stride exceeds one and depends on stride size.This is described as orthogonal to Multi-CLP resource partitioning.
  • Automation and integration: Automated FPGA-accelerator frameworks can generate implementations from CNN specifications, and Multi-CLP can be integrated into such frameworks.A related HLS tool uses different modules for different layer types, but computes all convolutional layers with a single CLP.

8 CONCLUSIONS

The paper concludes that a single CLP suffers from throughput limits when CNN layer dimensions vary, whereas Multi-CLP matches specialized processors more closely to those layers. Under the same resource-budget framing, the resulting designs improve utilization and throughput across several CNNs.

  • 8 CONCLUSIONS: Single-CLP throughput is limited by variation in CNN layer dimensions, which can make fixed CLP parameters a poor fit for individual layers.The conclusion characterizes this as a dynamic resource-utilization problem.
  • 8 CONCLUSIONS: Multi-CLP matches CLP dimensions more closely to CNN layers, resulting in better dynamic resource utilization and higher throughput.The approach partitions resources among multiple specialized CLPs rather than using one processor for all layers.
  • 8 CONCLUSIONS: 3.8x higher throughput was achieved for AlexNet, with speedups of 2.2x for SqueezeNet and 2.0x for GoogLeNet.For AlexNet, dynamic utilization improved from 24% to 91% using 16-bit fixed-point arithmetic on a Virtex-7 690T FPGA.
Loading 1607.00064v2…