Source-linked AI summary
A Reconfigurable Streaming Deep Convolutional Neural Network Accelerator for Internet of Things
Li Du, Yuan Du, Yilei Li, Mau-Chung Frank Chang
TL;DR
IoT CNN inference needs local processing because cloud execution depends on connectivity and can add latency, while conventional local processors have limited compute and high power consumption. The paper proposes a streaming accelerator using data-movement optimization, filter decomposition, parallel pooling, and reconfigurable computation. A TSMC 65nm prototype achieves 152 GOPS peak throughput and 434 GOPS/W energy efficiency, supporting major CNNs.
Problem
IoT CNN inference requires an efficient local accelerator because cloud processing depends on network access and local GPU/DSP processing is computationally limited and power-intensive.
Method
The accelerator combines streaming data flow, filter decomposition, interleaved input-channel reuse, and separate pooling hardware for reconfigurable CNN inference.
Results
152 GOPS peak throughput and 434 GOPS/W energy efficiency are achieved by a TSMC 65nm prototype that supports major CNNs.
Takeaways & Limitations
The accelerator is suitable for integration into IoT devices by reducing unnecessary data movement while supporting arbitrary convolution windows and pooling.
Abstract
from arXiv · showhide
Convolutional neural network (CNN) offers significant accuracy in image detection. To implement image detection using CNN in the internet of things (IoT) devices, a streaming hardware accelerator is proposed. The proposed accelerator optimizes the energy efficiency by avoiding unnecessary data movement. With unique filter decomposition technique, the accelerator can support arbitrary convolution window size. In addition, max pooling function can be computed in parallel with convolution by using separate pooling unit, thus achieving throughput improvement. A prototype accelerator was implemented in TSMC 65nm technology with a core size of 5mm2. The accelerator can support major CNNs and achieve 152GOPS peak throughput and 434GOPS/W energy efficiency at 350mW, making it a promising hardware accelerator for intelligent IoT devices.
I. INTRODUCTION
IoT CNN inference is motivated by cloud connectivity and latency limits, while local GPU/DSP processing is power- and compute-constrained. The paper therefore proposes a streaming accelerator targeting efficient, reconfigurable CNN computation.
- Cloud-based CNN processing requires network connectivity and can introduce latency that is unsuitable for real-time IoT applications.
- Local GPU or DSP processing has limited computation ability and relatively large power consumption for power-limited IoT devices.
- The proposed architecture uses streaming data flow to optimize data movement and minimize energy consumption for CNN inference.
- Filter decomposition supports large or arbitrary convolution windows using smaller computations, while integrated pooling provides one-stop CNN acceleration.
- The accelerator targets CNN convolution and pooling while treating classification-layer computation as suitable for software implementation.
B. Pooling Layer
The pooling layer extracts information from neighboring pixels, with max pooling selecting the largest value in each window. The accelerator includes a max-pooling module for this operation.
- Pooling extracts information from neighboring image pixels within each channel.
- Max pooling selects the maximum image-data value within its pooling window.
- The pooling-window kernel size is K, and I[r][c] denotes input-channel data at position (r,c).
III. SYSTEM OVERVIEW
The accelerator streams channel data and weights through a compute engine, accumulates convolution results in an ACCU buffer, and uses programmable commands to control CNN layers. It repeats this process with updated weights to generate all output features.
- 16-bit fixed-point arithmetic is used because deep networks can tolerate stochastic-rounding conversion with little to no classification-accuracy degradation and fixed-point adders require fewer logic gates.
- The control system uses a 16-bit AXI bus, with commands pre-stored in DRAM and loaded into a 128-depth command FIFO.
- Configuration commands set layer properties such as channel counts and ReLU or max-pooling enablement, while execution commands initiate computation.
- Convolution streams input data to the CU engine, accumulates channel results in the ACCU-buffer scratchpad, and returns the accumulated image after channel scanning.
- After each feature is computed, the accelerator repeats convolution with updated DRAM weights until all output features are generated.
- The streaming architecture uses a CU engine that receives channel-organized input data sequentially and convolution weights from DRAM.
IV. STREAMING AND RECONFIGURABLE FEATURES
The accelerator combines filter decomposition, streaming data flow, and separate pooling support to provide reconfigurable convolution with reduced hardware cost. Decomposition enables large kernels using 3x3 computation units, while zero padding introduces measurable efficiency loss.
- Filter decomposition: The accelerator supports large kernel-sized convolution using only 3x3 computation units through filter decomposition.Filters are padded to a multiple of three, decomposed into 3x3 filters, assigned shift addresses, and recombined.
- Filter decomposition: A 5x5 filter is decomposed into four 3x3 subfilters with shift addresses (0,0), (0,3), (3,0), and (3,3).The subfilter outputs are recombined according to their shift addresses.
- Filter decomposition: A 3Kx3K filter can be decomposed into K2 different 3x3 filter calculations without computation-accuracy loss.The resulting decomposed features are shifted and summed to form the final output feature.
- Efficiency trade-off: The decomposition simplifies the convolution engine to 1x1 and 3x3 filters, but added boundary zero padding wastes computation resources.The paper describes this as a hardware-resource trade-off and computes overall efficiency loss from zero-padding MACs.
- Efficiency trade-off: 16% efficiency loss results from 144 MAC operations used for zero-padding in an 11x11 filter.Table I compares decomposition-related convolution efficiency loss across major CNN networks.
- Efficiency trade-off: AlexNet has the largest efficiency loss, whereas Resnet-18, Resnet-50, and Inception V3 have very small losses and are well suited to the architecture.The difference is attributed to AlexNet’s 11x11 first-layer filter and the smaller filters in the other networks.
B. Streaming Architecture
The streaming architecture reduces convolution data movement by reusing filter weights across an input channel and generating output features through repeated scans of the same input layer.
- Data reuse: Streaming data flow minimizes convolution data movement by exploiting weight and input-layer reuse.Each filter’s weights scan an entire channel image, while each output feature scans the same input layer.
- Data reuse: Each filter-weight set is reused to scan a whole channel image, and each output feature is generated by scanning the same input layer.
1) Filter Weight Reuse:
The accelerator stores fetched weights in the computation engine while input data streams through it, producing partial results for accumulation. Parallel filter windows and buffered rows increase concurrent processing.
- Weight reuse: Filter weights are fetched from DRAM and stored in the computation engine while the input channel image streams through it.The engine produces partial output-feature results that are accumulated in the ACCU buffer until the channel is scanned.
- Weight reuse: During 1x1 convolution, seven of nine multipliers are disabled and two remaining multipliers calculate two output features’ partial sums simultaneously.
- Parallel processing: Sixteen 3x3 filter windows process multiple rows simultaneously, simplifying data flow and module interfaces while reducing hardware design cost.
- Buffering: The buffer bank streams sixteen 16-bit data values per cycle from different rows to the computation engine.The data are divided between odd-numbered and even-numbered input channels.
- Buffering: Two-row FIFO buffers transform eight input rows into ten overlapping output rows, enabling eight 3x3 computation units to run in parallel for each row-data set.
2) Input Channel Reuse:
The accelerator addresses inefficient 1x1 convolution by interleaving channel data to compute two output features in parallel while preserving input bandwidth. Average pooling reuses the convolution engine through an equivalent weighted convolution.
- 1x1 convolution: 1x1 convolution wastes most CU multipliers because each output-feature computation requires only one multiplication per channel.
- 1x1 convolution: Two output features are computed simultaneously during one input-layer scan, but this would otherwise double the CU engine’s output bandwidth.
- 1x1 convolution: Even- and odd-number channel data are multiplied with different feature weights, then partial results are summed to reduce output bandwidth by half.
- Average pooling: Average pooling is implemented by replacing it with a same-window convolution layer whose output-feature count equals the input-channel count.
- Average pooling: Each corresponding channel filter uses weight 1/K^2, while weights for other channels are set to zero.
2) Max Pooling:
The accelerator implements max pooling as a separate block inside the ACCU buffer, supporting pooling windows of size two and three while convolution proceeds in the main engine.
- Max pooling: Max pooling is implemented as a separate ACCU-buffer block that pools output features from the convolution block.
- Max pooling: The pooling block supports window sizes two and three, covering the pooling configurations used by major CNNs cited by the paper.
- Architecture: The accelerator’s architecture includes CU engine, ACCU buffer, and max-pooling modules.
A. CU Engine
The CU engine combines nine processing elements and an adder for convolution, with configurable activation and filter updates; the ACCU buffer uses ping-pong storage to overlap convolution, pooling, and readout.
- CU engine: The CU engine contains nine processing engines and an adder that combines their multiplication outputs for 3x3 convolution.
- CU engine: Processing elements can be disabled according to the EN_Ctrl signal when convolution stride exceeds one, reducing computation power.
- CU engine: Filter weights are fetched from DRAM through DMA, stored in the CU, and synchronously updated for each upcoming input channel.
- CU engine: For 1x1 convolution, only two processing elements operate, the adder is disabled, and two partial results are directly output.
- ACCU buffer: The ACCU buffer combines an accumulator, ping-pong scratchpad, separate max-pooling block, and readout block.
- ACCU buffer: Ping-pong switching lets pooling and readout process one buffer while convolution accumulates into the other, avoiding convolution speed loss.
C. Max Pool
The max-pooling module selects valid scratchpad rows and computes window maxima with comparator feedback, while the prototype demonstrates 152 GOP/s throughput and an area dominated by memory.
- Max-pool implementation: Scratchpad rows share a column address and can be accessed simultaneously, but stride determines which rows contain valid data.
- Max-pool implementation: A MUX selects valid inputs for different stride and pool-size cases, while comparator units and feedback registers accumulate window maxima.
- Max-pool implementation: The comparator recursively compares nearby-row inputs with the prior temporary maximum until the pooling window is fully scanned.
- Prototype results: 152 GOP/s peak throughput is achieved at 500MHz in a TSMC 65nm implementation with a 2mm x 2.5mm core.
- Prototype results: Energy efficiency is defined as peak throughput divided by dynamic power consumption.
- Area breakdown: The CU engine occupies 17% of total area, while the buffer bank and scratchpad occupy most of the accelerator area.
- Scope and limitation: When intermediate data exceeds available SRAM, DMA transfers between DRAM and on-chip SRAM incur large energy consumption.
- Comparison: The accelerator achieves high energy efficiency and comparable performance with low area cost relative to reported work.
VII. CONCLUSION
The proposed streaming CNN accelerator reduces unnecessary data movement, supports arbitrary convolution windows and pooling, and achieves energy-efficient IoT inference.
- 434GOPS/W energy efficiency makes the accelerator suitable for integration with IoT devices.The accelerator was implemented in TSMC 65nm technology with a 5mm2 core and verified using a traffic-sign network on FPGA.
- Filter decomposition enables support for arbitrary window-sized convolution without requiring separate accelerator architectures.
- A separate pooling module supports pooling while the convolution engine performs CNN computation.