PaperlayerView exact revision on arXiv

Source-linked AI summary

SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks

Angshuman Parashar, Minsoo Rhu, Anurag Mukkara, Antonio Puglielli, Rangharajan Venkatesan, Brucek Khailany, Joel Emer, Stephen W. Keckler, William J. Dally

arXiv:1708.04485v1cs.NEcs.ARcs.LG

TL;DR

CNN inference needs high performance and energy efficiency, while pruning and ReLU create exploitable zero-valued weights and activations. SCNN uses a compressed sparse dataflow with extensive operand reuse and a scatter accumulator, achieving 2.7× speedup and 2.3× energy reduction over a comparably provisioned dense architecture on contemporary networks.

  • Problem

    CNN deployments require high performance and energy efficiency, motivating architectures that exploit zero-valued weights from pruning and activations produced by ReLU.

  • Method

    SCNN uses the PT-IS-CP-sparse dataflow to keep weights and activations compressed, deliver only non-zero operands for Cartesian-product multiplication, reuse them, and scatter-accumulate products.

  • Results

    2.7× speedup and 2.3× energy reduction are achieved on a range of networks relative to a comparably provisioned dense architecture.

  • Takeaways & Limitations

    SCNN starts to outperform dense architecture in performance and energy efficiency when weights and activations are each less than 85% dense.

Abstract

from arXiv · show

Convolutional Neural Networks (CNNs) have emerged as a fundamental technology for machine learning. High performance and extreme energy efficiency are critical for deployments of CNNs in a wide range of situations, especially mobile platforms such as autonomous vehicles, cameras, and electronic personal assistants. This paper introduces the Sparse CNN (SCNN) accelerator architecture, which improves performance and energy efficiency by exploiting the zero-valued weights that stem from network pruning during training and zero-valued activations that arise from the common ReLU operator applied during inference. Specifically, SCNN employs a novel dataflow that enables maintaining the sparse weights and activations in a compressed encoding, which eliminates unnecessary data transfers and reduces storage requirements. Furthermore, the SCNN dataflow facilitates efficient delivery of those weights and activations to the multiplier array, where they are extensively reused. In addition, the accumulation of multiplication products are performed in a novel accumulator array. Our results show that on contemporary neural networks, SCNN can improve both performance and energy by a factor of 2.7x and 2.3x, respectively, over a comparably provisioned dense CNN accelerator.

I. INTRODUCTION

SCNN targets CNN inference by exploiting sparsity in both pruned weights and ReLU-produced activations. Its compressed sparse dataflow reduces data movement and improves performance and energy efficiency versus a comparably provisioned dense accelerator.

  • Motivation: 20% to 80% of weights can typically be eliminated across layers without substantively affecting accuracy, creating computational opportunities during inference.Pruning produces zero-valued weights that can be skipped.
  • Motivation: 50–70% of activations are typically clamped to zero by ReLU, further reducing the multiplication work required during inference.The paper states that combining weight and activation sparsity can reduce computation by over an order of magnitude.
  • Contribution: SCNN couples a dataflow that eliminates multiplications with zero operands with compressed representations of both weights and activations through almost the entire computation.This approach extends sparsity exploitation beyond selectively compressing one operand or one part of the dataflow.
  • Contribution: The multiplier array receives reusable vectors of non-zero weights and activations, while a scatter accumulator sums products using their output coordinates.Input vectors are reused in an input-stationary fashion, and coordinate tracking supports accumulation of products that cannot be directly summed in place.
  • Results: 2.7× speedup and 2.3× energy reduction are achieved on a range of networks relative to an equivalently provisioned dense architecture.A 64-PE implementation with 16 multipliers per PE is reported as approximately 7.9mm2, with sparse-dataflow overhead making it somewhat larger than dense hardware.

II. MOTIVATION

CNN sparsity arises from pruning weights during training and from ReLU clamping negative activations during inference. SCNN exploits these zeros through compression and by avoiding zero-operand computation.

  • Sparsity in CNNs: Sparsity is defined as the fraction of zeros in a layer’s weight and input activation matrices.Weight sparsity is created primarily through network pruning, whereas activation sparsity occurs dynamically during inference.
  • Sparsity in CNNs: Pruning sets sufficiently small weights to zero and retrains the remaining network to recover accuracy, producing a smaller network with accuracy extremely close to the original.The process can be repeated to reduce network size while maintaining accuracy.
  • Sparsity in CNNs: ReLU forces negatively valued activations to zero after convolutional layers before those activations are passed to the next layer.Activation sparsity depends strongly on the data being processed.
  • Exploiting sparsity: Sparse data can be compressed to reduce movement through the memory hierarchy and fit larger matrices in storage structures of a given size.Compression reduces the data footprint and can reduce costly memory accesses.
  • Exploiting sparsity: SCNN eliminates multiplications with a zero weight or activation by data gating or by preventing those operands from reaching the multiplier.Its encoding and delivery scheme combines compressed operands with computation elimination.

III. SCNN DATAFLOW

CNN convolution is a multidimensional multiply-accumulate computation whose loop ordering and partitioning determine accelerator dataflow. SCNN selects PT-IS-CP-sparse to exploit reuse patterns in sparse execution.

  • CNN computation: A convolution slides an R×S filter across a W ×H input plane, accumulates across C input channels, and produces K output channels for a batch of N inputs.These parameters describe the computation of a single CNN layer.
  • CNN computation: The complete layer forms a loop nest over seven variables, and associative multiply-add operations permit legal permutations of those loops.The paper assumes batch size 1 for inference in the subsequent discussion.
  • Dataflow: A CNN dataflow orders, partitions, and parallelizes loops to capture reuse patterns and map computation onto accelerator hardware.Dataflow choice substantially affects area and energy efficiency.
  • Dataflow: PT-IS-CP-sparse is a selected sparse dataflow designed around reuse patterns that exploit the structure of sparse CNN computation.The paper presents it as one specific choice rather than an exhaustive enumeration of sparse dataflows.

A. The PT-IS-CP-dense Dataflow

The PT-IS-CP-dense dataflow organizes CNN computation around input-stationary reuse, tiled PE parallelism, and Cartesian-product multiplication, while managing partial sums through accumulation buffers.

  • Temporal dataflow: Input-stationary execution holds an activation while multiplying it by all required filter weights, maximizing activation reuse.The loop structure adds input-channel processing and can be blocked across output-channel groups to improve weight and partial-sum reuse.
  • Buffering and accumulation: The dense dataflow uses weight and input buffers plus an accumulator buffer for output partial sums, whose accesses perform read-add-write updates.A register holding the stationary input over K×R×S iterations filters accesses to the larger input buffer.
  • Intra-PE parallelism: Vectors of F weights and I inputs feed an F×I multiplier array whose Cartesian product produces only useful partial sums.The corresponding accumulation unit updates partial sums at matching output coordinates and requires at least F×I adders for throughput matching.
  • Inter-PE parallelism: Tiling partitions the W×H activation plane into Wt×Ht tiles distributed across PEs, with each PE receiving a full C×Wt×Ht input-activation volume.This spreads work across independently operating PEs while retaining the input-channel dimension within each tile.
  • Inter-PE parallelism: Sliding-window convolution creates crosstile halo dependencies, requiring replicated input halos or communicated partial-sum output halos.The PT-IS-CP-dense design uses output halos, with little efficiency difference between the two approaches.

B. PT-IS-CP-sparse Dataflow

PT-IS-CP-sparse extends the dense dataflow to compressed-sparse weights and activations, delivering nonzero values with coordinates and scattering products into distributed accumulators.

  • Sparse representation: PT-IS-CP-sparse operates on compressed-sparse weights and input activations and produces compressed-sparse output activations.The sparse format is treated as independent of the architecture, provided decoding yields each nonzero value and its coordinate index.
  • Sparse representation: Weights are grouped by output-channel group and activations by input-channel tiles, allowing vectors of nonzeros and their coordinates to reach the multiplier array.Weight blocks contain Kc×R×S values, while activation blocks contain Wt×Ht values.
  • Sparse accumulation: Sparse multiplier outputs are generally discontiguous, so they must be scattered to addresses within the Kc×Wt×Ht output range.Because outputs can occupy any location, the accumulation buffer remains dense until ReLU produces sparse activations.
  • Sparse accumulation: A scatter network distributes F×I partial sums across A accumulator banks, replacing the monolithic dense accumulation buffer with a distributed array.Together, the banks still cover the same Kc×Wt×Ht address range.

IV. SCNN ARCHITECTURE

The SCNN architecture combines multiple sparse processing elements, compressed activation and weight storage, multiplier arrays, scatter-based accumulation, and post-processing. The explored design is evaluated through implementation and modeling, with on-chip capacity and tiled DRAM access shaping its operating scope.

  • Complete architecture: The complete accelerator connects multiple SCNN PEs, nearest-neighbor halo links, a layer sequencer, and a DRAM controller.Each PE receives weights and input activations and delivers output activations.
  • Processing element architecture: Each SCNN PE contains weight storage, input/output activation RAMs, a multiplier array, a scatter crossbar, accumulator banks, and a post-processing unit.The PE processes compressed vectors through the multiplier array and produces an output-channel group of partial sums.
  • Sparse encoding: Run-length encoding stores nonzero data with zero-run counts, and four-bit indices support gaps of up to 15 zeros between elements.Coordinates for accumulation are computed by combining weight and activation index vectors with the current output-space coordinates.
  • Scope and memory: Large networks such as VGGNet require activations to be saved to and restored from DRAM rather than fitting entirely in on-chip activation RAMs.The paper accounts for tiling-related DRAM access energy, while its detailed tiled approach is not presented.
  • Configuration and implementation: The explored design uses an 8×8 PE array, 4×4 multipliers per PE, 32 accumulator banks, and 1MB of activation RAM.Its 1,024 multipliers target real-time HD-video throughput, and the synthesized design reaches 2 Tera-ops at slightly more than 1 GHz.

V. EXPERIMENTAL METHODOLOGY

The evaluation models dense and sparse CNN accelerators using configurable analytical and cycle-level tools, with matched computational resources for comparison across representative networks.

  • Experimental Methodology: The study models SCNN with a parameterizable cycle-level simulator driven by pruned weights and sparse activation maps.The simulator varies PE tiles, RAM capacity, multiplier dimensions, and accumulator buffers, executing network layers individually.
  • Experimental Methodology: TimeLoop analytically explores dense and sparse CNN dataflows across memory hierarchy, ALU, partitioning, and hardware-support configurations.Its inputs include data parameters and architecture parameters for design-space analysis.
  • Experimental Methodology: All accelerator configurations use the same number of multiply ALUs to compare performance with equal computational resources.The comparison includes DCNN, optimized DCNN-opt, and SCNN configurations.
  • Experimental Methodology: The benchmarks use AlexNet and GoogLeNet primarily, plus a synthetic network for varying weight and activation sparsity and VGGNet for broader evaluation.GoogLeNet analysis focuses mainly on convolutional layers within inception modules.

VI. EVALUATION

The evaluation examines SCNN sensitivity to weight and activation sparsity, then compares its performance and energy efficiency with dense CNN accelerators on real-world applications.

  • Evaluation: The evaluation first varies weight and activation sparsity with a synthetic CNN benchmark.It then evaluates SCNN against dense architectures using real-world CNN applications.

A. Sensitivity to CNN Sparsity

SCNN becomes increasingly advantageous as weight and activation density decreases, while sparse-data-structure overheads reduce its benefit at high density.

  • Sensitivity to CNN Sparsity: 24× improvement is reached at 10% weight/activation density, while SCNN achieves about 79% of DCNN performance at 100% density.SCNN begins outperforming DCNN as density decreases to 85%.
  • Sensitivity to CNN Sparsity: DCNN-opt is more energy efficient than DCNN at every density because of zero gating and DRAM traffic compression.These optimizations improve energy without affecting performance comparisons in the performance chart.
  • Sensitivity to CNN Sparsity: SCNN becomes more energy efficient than DCNN at about 83% weight/activation density and more efficient than DCNN-opt at 60% density.The paper expects SCNN to outperform dense architectures on nearly all examined network layers based on measured densities.

B. SCNN Performance and Energy

Across contemporary CNNs, SCNN improves performance and energy efficiency over dense accelerators, while utilization fragmentation, barriers, and dense input layers constrain some gains.

  • Performance: 2.37×, 2.19×, and 3.52× network-wide performance improvements are achieved for AlexNet, GoogLeNet, and VGGNet, respectively.SCNN consistently outperforms the DCNN design across all layers of these networks.
  • Performance: Less than 20% average multiplier utilization occurs in the last two GoogLeNet inception modules because small working sets limit available non-zero operands.Inter-PE barriers also expose load imbalance, although SCNN still averages 2.7× performance over DCNN across three CNNs.
  • Energy-efficiency: 2.3× average energy-efficiency improvement is achieved by SCNN over DCNN, compared with 2.0× for DCNN-opt over DCNN.SCNN’s layer-level improvement ranges from 0.89× to 4.7× over DCNN.

C. PE Granularity

SCNN’s PE-granularity study examines how partitioning a fixed computational throughput affects barriers and multiplier-array fragmentation. Larger networks also create an on-chip-memory sizing trade-off, sometimes requiring tiling and DRAM transfers.

  • C. PE Granularity: The sensitivity study sweeps SCNN from 64 PEs with 16 multipliers each to 4 PEs with 256 multipliers each at fixed 1,024 FLOPS.This isolates the effect of PE granularity while holding chip-wide mathematical throughput constant.
  • C. PE Granularity: Cross-PE global barriers and intra-PE multiplier-array fragmentation can degrade SCNN performance under the PT-IS-CP-sparse dataflow.
  • D. Larger Networks: SCNN is most efficient when all activations fit in its IARAM and OARAM.
  • D. Larger Networks: Tiling is required for 9 of 72 evaluated layers in VGGNet, with DRAM shuttling imposing a 5–62% energy penalty and an 18% mean penalty.The reported penalty reflects moving activation data to and from DRAM for layers that do not fit on chip.

VII. RELATED WORK

Prior CNN accelerators exploit sparsity through gating, compression, or selective operand delivery, but their coverage and placement differ. SCNN combines compressed weights and activations throughout nearly the full processing flow and targets convolutional layers, achieving substantial performance and energy gains over a dense architecture.

  • VII. RELATED WORK: Eyeriss gates multipliers for zero activations and uses run-length encoding for DRAM transfers, but expands data in on-chip buffers.
  • VII. RELATED WORK: Cnvlutin compresses activations and selects non-zero activations for multiplier delivery, whereas Cambricon-X retains only non-zero weights in internal buffers.
  • VII. RELATED WORK: EIE compresses both operands and delivers only non-zero values, but targets fully connected layers rather than convolutional layers.
  • VII. RELATED WORK: SCNN keeps both weights and activations compressed through almost the entire processing flow, maximizing reuse while reducing data movement and on-die storage.
  • VII. RELATED WORK: 2.7× performance and 2.3× energy-efficiency improvements are reported over a dense architecture on AlexNet, GoogLeNet, and VGGNet.
Loading 1708.04485v1…