PaperlayerView exact revision on arXiv

Source-linked AI summary

NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps

Alessandro Aimar, Hesham Mostafa, Enrico Calabrese, Antonio Rios-Navarro, Ricardo Tapiador-Morales, Iulia-Alexandra Lungu, Moritz B. Milde, Federico Corradi, Alejandro Linares-Barranco, Shih-Chii Liu, Tobi Delbruck

arXiv:1706.01406v2cs.CVcs.NE

TL;DR

CNN inference is computationally expensive for power-constrained platforms. NullHop exploits activation sparsity through zero-skipping and sparse compression, achieving high-throughput, energy-efficient processing across CNNs, including 471.64 GOp/s on VGG19.

  • Problem

    CNN inference requires substantial computation and memory access, creating energy challenges for power-constrained platforms.

  • Method

    NullHop accelerates CNN inference by skipping zero activations and compressing sparse feature maps for direct processing.

  • Results

    471.64 GOp/s and 368.47% efficiency on VGG19 demonstrate NullHop’s high-throughput sparse computation.

  • Takeaways & Limitations

    NullHop’s compressed sparse pipeline reduces I/O power and computing time while maintaining high MAC utilization for embedded CNN applications.

Abstract

from arXiv · show

Convolutional neural networks (CNNs) have become the dominant neural network architecture for solving many state-of-the-art (SOA) visual processing tasks. Even though Graphical Processing Units (GPUs) are most often used in training and deploying CNNs, their power efficiency is less than 10 GOp/s/W for single-frame runtime inference. We propose a flexible and efficient CNN accelerator architecture called NullHop that implements SOA CNNs useful for low-power and low-latency application scenarios. NullHop exploits the sparsity of neuron activations in CNNs to accelerate the computation and reduce memory requirements. The flexible architecture allows high utilization of available computing resources across kernel sizes ranging from 1x1 to 7x7. NullHop can process up to 128 input and 128 output feature maps per layer in a single pass. We implemented the proposed architecture on a Xilinx Zynq FPGA platform and present results showing how our implementation reduces external memory transfers and compute time in five different CNNs ranging from small ones up to the widely known large VGG16 and VGG19 CNNs. Post-synthesis simulations using Mentor Modelsim in a 28nm process with a clock frequency of 500 MHz show that the VGG19 network achieves over 450 GOp/s. By exploiting sparsity, NullHop achieves an efficiency of 368%, maintains over 98% utilization of the MAC units, and achieves a power efficiency of over 3TOp/s/W in a core area of 6.3mm$^2$. As further proof of NullHop's usability, we interfaced its FPGA implementation with a neuromorphic event camera for real time interactive demonstrations.

I. INTRODUCTION · II. CNN PRINCIPLES OF OPERATION

CNNs use successive convolution, non-linearity, and sub-sampling stages to extract high-level image features, but inference requires substantial computation and can be impractical on power-constrained mobile platforms. NullHop addresses this challenge by exploiting activation sparsity through zero-skipping and compressed-representation processing while maintaining configurable efficiency across CNN kernels.

  • I. INTRODUCTION: CNNs are widely used for large-scale machine-vision tasks including classification, localization, and detection because they extract high-level semantic image features.
  • I. INTRODUCTION: CNNs are typically trained with backpropagation on labeled examples using hardware platforms such as GPUs or specialized server architectures.
  • I. INTRODUCTION: Several billion MAC operations per image make state-of-the-art CNN inference computationally expensive and potentially infeasible on power-constrained mobile platforms.
  • I. INTRODUCTION: NullHop exploits activation sparsity by skipping zero inputs without wasted clock cycles or redundant MACs, extending zero-skipping beyond fully connected layers.
  • I. INTRODUCTION: NullHop processes compressed input representations directly, reducing external memory access and storing more input data in accelerator memory than run-length encoding schemes.
  • II. CNN PRINCIPLES OF OPERATION: CNNs extract features through successive convolution, non-linearity, and sub-sampling stages, typically beginning with three input channels corresponding to red, green, and blue.
  • II. CNN PRINCIPLES OF OPERATION: The convolution stage applies filters connecting input feature maps to output feature maps, producing a three-dimensional output array.
  • II. CNN PRINCIPLES OF OPERATION: ReLU, computed as f(x) = max(0,x), is widely used because it is computationally cheap and often improves classification accuracy over sigmoidal nonlinearities.

A. Reduced Precision CNNs · III. ACCELERATOR ARCHITECTURE · A. Sparse Matrix Compression Scheme

NullHop combines reduced-precision CNN training with a configurable accelerator that streams compressed feature maps and decodes them during computation. Its sparse representation uses a binary sparsity map and ordered non-zero value list, achieving compression advantages over prior schemes and run-length coding.

  • A. Reduced Precision CNNs: Reduced-precision weights and activations lower hardware resources, while reduced-bit training can recover accuracy otherwise compromised by rounding.The paper motivates quantization as a way to reduce memory access and computation energy, while noting that training with reduced precision improves accuracy.
  • A. Reduced Precision CNNs: ADaPTION trains networks from scratch or fine-tunes 32-bit floating-point models to specified fixed-point precisions using power2quant.The custom Caffe branch also estimates per-layer decimal-point locations for weights and activations.
  • III. ACCELERATOR ARCHITECTURE: The accelerator processes convolution, ReLU, and max-pooling stages, with ReLU and max-pooling optionally disabled.Its interface provides separate 32-bit input and output buses plus configuration and control interfaces.
  • III. ACCELERATOR ARCHITECTURE: Feature maps stream between the accelerator and external memory in compressed form, remaining encoded until decoded during computation.The Input Decoding Processor generates pixels from compressed inputs, decoding only a small minicolumn at a time before passing them to the Compute Core Module.
  • A. Sparse Matrix Compression Scheme: NullHop’s sparse compression algorithm uses a 3D Sparsity Map and an ordered, variable-length Non-Zero Value List to reconstruct non-zero pixel positions sequentially.The sparsity map has one binary entry per feature-map pixel, while the list stores the corresponding non-zero values.
  • A. Sparse Matrix Compression Scheme: 16 bits requires threshold sparsity 0.0625 for compression, and the algorithm achieves better average compression than the cited run-length method on VGG19.The scheme’s average compression also exceeds the method in, while being easier to decode than the Huffman coding in.

B. Pixel Memory and Decoding: Input Data Processing Unit · 1) Input format: · 2) Decoding the compressed rows:

NullHop streams sparsity-map segments interleaved with non-zero pixels, while the Input Data Processing Unit stores compressed rows and decodes them sequentially for convolution. Its memory banks, row pointers, and kernel-size-specific controllers support overlapping loading and decoding, efficient stripe access, and boundary padding.

  • 1) Input format:: 16-bit word segments are streamed interleaved with corresponding non-zero pixel values, with segment length matched to activation bit precision in this implementation.The segment length is implementation-dependent and chosen to simplify hardware.
  • 1) Input format:: Each sparsity-map segment’s one-count specifies the following pixel count, while one positions encode pixel offsets; the first transmitted word is always a segment.An all-zero segment represents a run of 16 zero pixels and is followed by another segment.
  • 1) Input format:: Compressed rows are streamed consecutively from the top row, with pixels ordered across input feature maps and spatial positions.The supplied passage gives the sequence beginning p(0,0,0), p(1,0,0), through the input maps, then the next spatial position.
  • 1) Input format:: The IDP stores and decodes compressed input rows through multiple SRAM banks, maintaining row-start pointers and decoding pixels sequentially while loading continues.This enables decoding to begin before all input feature maps finish loading.
  • 1) Input format:: Up to kh +1 non-zero pixels can be read per clock cycle within a vertical stripe, while zero padding handles feature-map boundaries without extra data or wasted cycles.The stripe contains the pixels needed to generate a double output-feature-map row for vertical stride 1.
  • 2) Decoding the compressed rows:: The decoder comprises Pixel Memory, Input Tracker, and IDP Manager blocks that respectively store pixels, retain row-start addresses, and control decoding.Pixel Memory uses SRAM banks and arbiters, while Input Tracker stores row starting positions in a small SRAM.
  • 2) Decoding the compressed rows:: The IDP Manager uses FSMs for rows in the active vertical stripe, enabling kh +1 controllers during processing; for kh = 3, four FSMs are enabled.At layer start, FSMs load pointers for the first kh +1 rows and their sparsity maps before issuing Pixel Memory reads.
  • 2) Decoding the compressed rows:: Each FSM computes the first non-zero position from its sparsity map, advances its memory pointer, then reads pixels while forwarding spatial coordinates to the CCM.Subsequent cycles increment pointers and locate the next non-zero entry for continued decoding.

C. Compute Core Module: Pixel Allocator, Controllers, and MAC Blocks

The Compute Core Module (CCM) performs convolution output computation through kernel memory, controllers, pixel allocation, and MAC blocks. Its dataflow supports row-wise processing, accumulator shifting, and workload redistribution across varying output-channel and memory constraints.

  • Core architecture: The CCM computes convolution output Fout and comprises Kernel Memory, MAC blocks, C Controllers, and a Pixel Allocator.Kernel Memory uses M SRAM banks, while the MAC block contains M MACs with multipliers, adders, and 2×kw accumulators.
  • Core architecture: At each clock cycle, the active Controller broadcasts one pixel coordinate to all Kernel Memory banks, so MACs apply different kernel weights for different output channels.Controllers redirect weights and activations to MAC clusters, while the Pixel Allocator routes incoming pixels from the IDP to controllers.
  • Kernel read and accumulation: NullHop computes Fout row-wise, processing 2 output rows simultaneously for on-the-fly 2x2 pooling and accumulating products according to pixel and weight coordinates.Boundary-aware processing omits kernel multiplications that cannot contribute to any output value.
  • Kernel read and accumulation: When the pixel column increases, the Controller shifts completed leftmost accumulator entries into the PRE output buffer.The IDP supplies pixels along vertical stripes, enabling the Controller to identify results belonging to the previous column.
  • Workload redistribution: For channel-count or SRAM-fit mismatches, NullHop redistributes work by splitting output-channel computation across MACs, multiple passes, or memory banks.These mechanisms handle Nout < M, Nout > M, and kernels that exceed individual Kernel Memory Bank capacity.

D. Pooling, ReLU, and Encoding Unit

The PRE module performs final CNN processing by buffering convolution outputs, accumulating partial results, applying ReLU and optional 2 × 2 max-pooling, and encoding outputs for streaming. Its encoder compresses output pixels into SM segments and non-zero pixels, operating on 16 pixels at a time in the 16-bit implementation.

  • Processing pipeline: The PRE module receives convolution results from the MACs, stores them in a buffer, and performs final processing before output streaming.The current buffer size is 2∗M memory entries, where 2 is the pooling dimension.
  • Output encoding: The encoder compresses output pixels according to the specified scheme and streams out SM segments and non-zero pixels.It processes the first B output-buffer pixels, where B equals the chip’s bit precision.
  • Partial-result accumulation: When multiple MAC blocks produce partial results, the PRE buffer accumulates them into Fout values in log(size(cluster)) +1 clock cycles.Each cycle sums two adjacent partial values and stores the result back in the PRE buffer.
  • ReLU and pooling: ReLU is applied during transfer to the output buffer, where max operations implement optional 2 × 2 max-pooling across arriving pixel pairs.Without pooling, one row at a time is transferred from the PRE buffer to the output buffer.
  • Output encoding: In the 16-bit implementation, the encoder processes 16 pixels at a time.This follows from the encoder operating on a number of pixels equal to the chip’s bit precision.

IV. DESIGN IMPLEMENTATION

NullHop was implemented with fixed-point kernels and activations, higher-precision MAC units, dedicated memories, and 128 MACs. Its configuration supports up to 7×7 kernels, 512×512 input images, and 1024 feature maps.

  • Architecture Parameters: 16-bit precision fixed point kernels/activations were paired with 32-bit precision MAC units.The implementation used separate precisions for stored computation data and MAC operations.
  • Memory Configuration: 512 KB of IDP memory and 576 KB of kernel memory were provided for feature-map and kernel storage.These dedicated memories were part of the implementation parameters.
  • Architecture Parameters: 128 MACs and 8 controllers formed the core parallel compute and control resources.The design also included a 32-bit input/output bus.
  • Supported Dimensions: 7 was the maximum supported kernel size, with input images up to 512 rows by 512 columns and 1024 feature maps.These limits define the supported layer dimensions in the implementation.

A. Synthesis, Place and Route · B. FPGA Implementation

NullHop was synthesized in 28 nm technology with a post-place-and-route core size of 6.3 mm^2 and 70% utilization, then validated on a Xilinx Zynq 7100 SoC FPGA. The FPGA implementation uses ARM-controlled DMA between DDR and BRAM, reaches 60 MHz, and includes measured and estimated power results.

  • A. Synthesis, Place and Route: 6.3 mm^2 is the post-place-and-route core size, with 70% core utilization, after synthesis in Globalfoundries 28 nm technology.The implementation selected the smallest available standard-cell library to reduce area and power for embedded deployment.
  • B. FPGA Implementation: The design was implemented on a Xilinx Zynq 7100 SoC FPGA, using AXI4-Stream DMA to connect the FPGA fabric with its ARM processor.Petalinux controls the accelerator; the ARM manages DDR-to-BRAM transfers and computes fully connected layers after the convolutional layers.
  • B. FPGA Implementation: The real-time demonstration interfaces NullHop with an iniLabs DAVIS240C neuromorphic event-based camera through the ARM processor and cAER3 framework.The FPGA architecture uses MM2S and S2MM modules to connect the accelerator to the AXI4-S bus.
  • B. FPGA Implementation: DMA transfers move inputs and outputs between host memory and the accelerator without reformating each layer output, while interrupt-driven execution frees the ARM for other processing.For each layer, the ARM loads configuration and kernels, initiates input and output transfers, and then waits for completion asynchronously.
  • B. FPGA Implementation: The FPGA implementation reaches a maximum clock frequency of 60 MHz after synthesizing and implementing NullHop with the AXI interfaces.The IC implementation targets a clock frequency above 500 MHz, but routing limits the Zynq 7100 implementation to much lower frequencies.
  • B. FPGA Implementation: 1.5 W is the estimated dynamic power consumption for the ARM processor, while NullHop plus AXI4-S logic consumes 0.8 W.Vivado estimates 316 mW static power when the FPGA clock is stopped and the ARM cores are idle, assuming half the logic nodes switch each cycle.
  • B. FPGA Implementation: 63% of the total estimated power is consumed by the ARM, while the Kintex-7 embedded FPGA logic and memory consume the remaining 37%.Within the FPGA blocks, the estimated shares are 27.27% for CCM, 4.14% for IDP, 1.75% for PRE, and 1.15% for AXIstream.
  • B. FPGA Implementation: 8.27 W is the measured system consumption after programming the FPGA while Linux is idle and cAER is not running.The base-board alone consumes 5.1 W, and the 7100 with its fan consumes 6.95 W in reset mode without an FPGA design.

V. APPLICATION EXAMPLE · A. VGG16 and VGG19 networks

The application example evaluates NullHop across CNNs ranging from small custom classifiers to large ImageNet networks. VGG16 and VGG19 require specialized execution arrangements because their feature-map counts, input-memory demands, and kernel sizes can exceed individual accelerator resources.

  • V. APPLICATION EXAMPLE: Several CNNs were studied, spanning small custom classification networks and large ImageNet models with 1000 classes.
  • A. VGG16 and VGG19 networks: 31 GOp/frame and 39 GOp/frame are required by VGG16 and VGG19, respectively, for 224x224x4 input images.Using ADaPTION, the networks reached 68.3% and 71.3% Top-1 accuracy, respectively.
  • A. VGG16 and VGG19 networks: 224x224x4 input images motivated particular hardware arrangements for evaluating NullHop on VGG16 and VGG19.Performance evaluation used randomly chosen images from the ImageNet dataset.
  • A. VGG16 and VGG19 networks: 128 MAC blocks limit each pass to at most 128 output feature maps, so VGG19 layers 5–8 require multiple passes for their 256 output maps.Output maps are divided into subsets, with each subset produced in one pass through the input feature maps.
  • A. VGG16 and VGG19 networks: 512 KB of IDP memory was sufficient to avoid multiple input streaming passes for all tested input images.Compressed input feature maps are more likely to fit into SRAM in zero-skipping mode.
  • A. VGG16 and VGG19 networks: 4k kernel values are supported per MAC block, but VGG19 layers 10–16 require clustering because each output map has 4608 values.Each MAC block provides 8kB of kernel memory, and clustered MAC blocks jointly produce one output feature map.

B. Giga1Net … A. VGG19 and VGG16

The evaluated networks span stress-test, event-camera, and face-detection CNNs, while NullHop delivers high throughput, utilization, and real-time performance on VGG networks despite an output-bandwidth bottleneck in the first convolutional layer.

  • B. Giga1Net: Giga1Net is a 1 GOp/frame CNN designed to expose accelerator inefficiencies across varied kernel sizes and output-feature-map configurations.Its kernels range from 1x1 to 7x7, with 16 to 128 output feature maps.
  • C. RoshamboNet: RoshamboNet is a 5-layer, 20 MOp, 114k-weight CNN that classifies 64x64 DAVIS-camera inputs into rock, scissors, paper, or background.Each input frame is a 2D histogram of 2k DVS events.
  • C. RoshamboNet: Over 99% accuracy and less than 10ms reaction time enabled NullHop’s DAVIS-camera robot-hand demonstration to convincingly appear to outguess human opponents.The system connected the camera, robot hand, and LED display through customized cAER on the Zynq ARM processor.
  • D. Face Detector CNN: The face-detector CNN determines face presence from 36x36 accumulated DAVIS events and requires 1.98 MOp per classified frame.It was trained on 1800k frames from public face datasets and labeled DAVIS frames.
  • VI. RESULTS: The evaluated networks ran on both Mentor QuestaSim HDL simulation and the Xilinx Zynq platform, with results summarized in Tables VII and VIII.This evaluation covers the networks described in Section V.
  • A. VGG19 and VGG16: 471.64 GOp/s and 368.47% efficiency for VGG19, versus 420.83 GOp/s and 328.8% efficiency for VGG16, demonstrate sparsity-driven throughput above the 128 GOp/s ideal maximum.NullHop exceeds the nominal maximum because skipped operations increase effective operations per second.
  • A. VGG19 and VGG16: Above 99% utilization of the 128 MAC blocks outside initial loading was consistent across all layers except the first, despite unpredictable sparsity patterns.Kernels and initial input rows load first, after which input processing and remaining feature-map loading proceed in parallel.
  • A. VGG19 and VGG16: 13 FPS at 500 MHz supported real-time processing, while the FPGA implementation processed VGG16 in about 1.5s; the first layer was limited by output-bus bandwidth.The output bus transmitted at most 2 non-zero pixels per cycle, versus an average computational dispatch capacity of 4.7 output pixels per cycle.

B. Giga1Net · C. RoshamboNet and Face Detector · D. Memory Power Consumption Estimation

Giga1Net demonstrates that NullHop can classify frames at more than 8 FPS despite ARM fully-connected-layer limitations, while small CNNs expose I/O-bandwidth bottlenecks. Memory estimation uses measured data movement and LPDDR3 energy assumptions, with NullHop transferring substantially less data than Eyeriss on VGG16.

  • B. Giga1Net: 3 MAC operations are required for each output pixel in Giga1Net’s first layer, whose sixteen kernels each have size 1x1x3.This low per-pixel workload contributes to underutilization in the first layer.
  • B. Giga1Net: 10% MAC utilization occurs in Giga1Net’s first layer because each 1x1 kernel requires only 3 MAC operations and output writes bottleneck the bus.For 16 output feature maps, NullHop clusters 8 multipliers per output map, leaving 5 MAC units idle during computation.
  • B. Giga1Net: More than 8 FPS is achieved for frame classification at 60 MHz, despite low compute performance from the ARM CPU handling fully-connected layers.The full system reaches almost 100% efficiency.
  • C. RoshamboNet and Face Detector: About 50% of the time, NullHop MACs are idle in RoshamboNet and Face Detector because I/O bandwidth limits kernel and data loading and result streaming.These small CNNs push the pipeline’s flexibility to its limits and represent the lower limit of NullHop efficiency.
  • D. Memory Power Consumption Estimation: 21 pJ/bit is the DRAM memory-access energy used to estimate total ASIC system power including external LPDDR3 memory access.The estimate uses data-movement statistics collected during computation and runs each network as fast as possible.
  • D. Memory Power Consumption Estimation: 42 MB/frame is the average overall memory transfer reported for VGG16, compared with Eyeriss’s 113 MB/frame in batch mode.Eyeriss’s figure comes from 341 MB transferred for a batch of 3 VGG16 images.
  • D. Memory Power Consumption Estimation: Almost 3 times more I/O is transferred by Eyeriss than NullHop for VGG16 in batch mode.The comparison is based on 113 MB/frame for Eyeriss versus 42 MB/frame for NullHop.

VII. DISCUSSION · A. Comparison with Prior Work · B. Conclusion

NullHop combines sparse computation, compressed feature-map processing, and flexible CNN support to achieve high utilization, reduced data movement, and strong simulated efficiency. Compared with prior accelerators, it delivers state-of-the-art performance while remaining suitable for embedded deployment and preserving 16-bit precision.

  • A. Comparison with Prior Work: 368% efficiency was achieved for a large CNN, with NullHop consistently exceeding 100% efficiency in ASIC simulations.The paper attributes this advantage to smaller discrepancies between theoretical peak and effective performance than in most prior architectures.
  • A. Comparison with Prior Work: <350 GOp/s was the performance reported for implementations and, which used more MAC units and 16 mm2 areas than NullHop.Those implementations nevertheless had higher power efficiency because they supported recurrent neural networks and variable bit precision.
  • B. Conclusion: Above 99% MAC utilization was maintained when input or output bandwidth was not limiting, enabling sparse computation comparable to dense pipelines.NullHop’s speedup is directly proportional to CNN sparsity because it skips cycles for zero input pixels.
  • B. Conclusion: 3 TOp/s/W is the projected power efficiency of a dedicated NullHop IC.The architecture uses 16-bit weights and states, supporting acceptable accuracy without the difficult tuning and extended training associated with super-reduced-precision networks.
  • B. Conclusion: Higher compression ratios than prior run-length encoding schemes are achieved by NullHop’s sparsity-map compression, which directly supports compressed feature-map processing.This reduces I/O power consumption and computing time, supporting embedded-system applications.
  • B. Conclusion: Sparse computation enables reductions in power consumption, memory access, and compute time for real-world deployment.NullHop’s data encoding and processing pipeline are optimized for spatially sparse data representations and provide a flexible range of input support.
Loading 1706.01406v2…