PaperlayerView exact revision on arXiv

Source-linked AI summary

UCNN: Exploiting Computational Reuse in Deep Neural Networks via Weight Repetition

Kartik Hegde, Jiyong Yu, Rohit Agrawal, Mengjia Yan, Michael Pellauer, Christopher W. Fletcher

arXiv:1804.06508v1cs.NEcs.LG

TL;DR

CNN inference contains repeated weights that can be reused to reduce redundant dot-product work, but exploiting them requires handling irregular repetition and metadata overhead. The paper introduces UCNN, which combines factorized dot products and activation-group reuse in a CNN accelerator. Across three contemporary CNNs, UCNN improves energy efficiency relative to an Eyeriss-style baseline while adding limited processing-element area.

  • Problem

    The paper asks how repeated weights in and across CNN filters can be exploited for efficient inference despite irregular repetition patterns and metadata storage and transport costs.

  • Method

    UCNN uses factorized dot-product dataflow and activation-group reuse to reuse computations, reduce memory traffic, and compress CNN model storage.

  • Results

    UCNN improves throughput-normalized energy consumption by 1.2×–4× relative to a similarly provisioned Eyeriss-style sparsity-optimized baseline, with 17–24% processing-element area overhead.

  • Takeaways & Limitations

    Exploiting repeated nonzero weights can extend sparse-acceleration benefits to broader weight reuse in CNN inference.

Abstract

from arXiv · show

Convolutional Neural Networks (CNNs) have begun to permeate all corners of electronic society (from voice recognition to scene generation) due to their high accuracy and machine efficiency per operation. At their core, CNN computations are made up of multi-dimensional dot products between weight and input vectors. This paper studies how weight repetition ---when the same weight occurs multiple times in or across weight vectors--- can be exploited to save energy and improve performance during CNN inference. This generalizes a popular line of work to improve efficiency from CNN weight sparsity, as reducing computation due to repeated zero weights is a special case of reducing computation due to repeated weights. To exploit weight repetition, this paper proposes a new CNN accelerator called the Unique Weight CNN Accelerator (UCNN). UCNN uses weight repetition to reuse CNN sub-computations (e.g., dot products) and to reduce CNN model size when stored in off-chip DRAM ---both of which save energy. UCNN further improves performance by exploiting sparsity in weights. We evaluate UCNN with an accelerator-level cycle and energy model and with an RTL implementation of the UCNN processing element. On three contemporary CNNs, UCNN improves throughput-normalized energy consumption by 1.2x - 4x, relative to a similarly provisioned baseline accelerator that uses Eyeriss-style sparsity optimizations. At the same time, the UCNN processing element adds only 17-24% area overhead relative to the same baseline.

I. INTRODUCTION

CNN inference is dominated by repeated dot-product work, creating opportunities to reduce computation and memory traffic by reusing repeated weights. UCNN addresses irregular repetition and metadata overhead with factorization, activation-group reuse, and an accelerator architecture evaluated for energy, performance, and area.

  • CNN computation: CNN convolutions compute sliding multi-dimensional dot products whose hardware execution requires input and weight reads plus multiply-accumulate operations.A representative output performs six memory reads, three multiplies, and two additions.
  • Exploiting repetition: Repeated weights enable dot-product factorization and memoization, reducing multiplies and memory reads while preserving the same result.The example saves 33% of multiplies and 16% of memory reads through factorization, while memoization saves 33% of multiplies and memory reads.
  • Motivation: Weight repetition is common because quantized networks use fewer unique weights than filter positions, with 8-bit weights allowing at most 256 unique values.Quantization methods such as INQ and TTQ reduce unique-weight counts further without changing filter size.
  • Motivation: Efficient repetition-aware acceleration is difficult because repetition patterns are irregular and their metadata can impose substantial storage and transport energy costs.These challenges resemble those faced by sparse architectures.
  • UCNN: UCNN combines factorized dot-product dataflow with activation-group reuse to reduce arithmetic and memory traffic while compressing CNN model storage.The architecture also uses techniques intended to amortize irregular-access energy and limit hardware-area overhead.
  • Scope: Weight repetition generalizes beyond CNNs to dot-product-based DNNs, and some techniques such as dot-product factorization apply directly to non-CNN algorithms.The paper focuses on CNNs because of their central role in many workloads.

II. BACKGROUND

CNN inference primarily consists of convolutional layers that transform multi-dimensional inputs with learned filters into output activations. Modern quantization practices reduce unique-weight counts, making repetition widespread and creating an opportunity to exploit nonzero repetitions as well as zeros.

  • CNN background: A convolutional layer applies K R×S×C filters to a W×H×C input, producing a (W−R+1)×(H−S+1)×K output.C and K denote input and output channel counts, respectively.
  • CNN inference: CNN inference evaluates convolutional layers online, with outputs becoming inputs to subsequent layers and batch size assumed to be one.The paper focuses on convolutional layers because they constitute most of the computation.
  • Weight quantization: Quantization reduces model size and arithmetic cost by lowering numerical precision or restricting weights to a small set of unique values.Contemporary networks commonly use reduced precision, including 8-bit weights.
  • Weight repetition: With U unique weights, repetition within and across filters is guaranteed when U<R∗S∗C and U<R∗S∗C∗K, respectively.For 8-bit weights, U≤256, a condition common in contemporary networks such as ResNet-50.
  • Weight repetition: Weight repetition is widespread across networks of different sizes and depths, and nonzero weights are often repeated at least ten times per filter.The combined repetitions of nonzero weights can exceed repetitions of zero weights, indicating untapped reuse potential.

III. EXPLOITING WEIGHT REPETITION

The paper exploits repeated weights by factoring dot products and reusing activation groups, reducing arithmetic, memory accesses, and model-storage overhead. It also treats zero-weight sparsity as a special case of weight repetition.

  • Activation Group Reuse: Activation group reuse generalizes factorization across filters by reusing overlapping activation sub-expressions.The paper identifies this as reuse across the RSCK dimensions, whereas basic factorization operates within one filter’s RSC dimensions.
  • Dot Product Factorization: Dot product factorization groups activations sharing a weight, sums each group, and multiplies each sum once.Each activation group corresponds to one unique weight, with group size equal to that weight’s repetition count.
  • Implementation: Input and weight indirection tables map irregular activation locations and unique weights to activation groups, and can be generated offline.These tables support the factorized traversal of inputs and weights.
  • Dot Product Factorization: Factorization reduces multiplies per filter to its number of unique weights, independent of filter or activation-group size.Reported average multiplication savings range from 5× to 373×.
  • Sparsity and Costs: Zero weights are skipped along with their activation-group summation and multiplication, making sparsity a special case of weight repetition.The paper notes that indirection-table storage and lookup energy create costs that require additional compression and vectorization techniques.

B. Activation Group Reuse

Activation group reuse finds overlaps between activation groups of multiple filters and recursively reuses shared sub-activation groups. The scheme can extend from two filters to G filters through repeated set intersections.

  • B. Activation Group Reuse: Activation group reuse identifies overlapping activation expressions between filters and reuses their intersections across computations.For example, a sub-group such as x+h can be reused within two filters’ overlapping groups.
  • B. Activation Group Reuse: Sub-activation groups for a later filter are formed by taking set intersections with an earlier filter’s activation group.The construction can produce up to U sub-activation groups for the later filter.
  • B. Activation Group Reuse: For G filters, the method recursively forms sub-sub-activation groups between successive filters.G = 1 reduces to ordinary dot product factorization, while the example described uses G = 2.
  • B. Activation Group Reuse: Irregular weight distributions may produce fewer than U overlapping unique weights for a given activation group.The paper defers handling this corner case to its processing-element architecture discussion.

1) Reduced input buffer reads and arithmetic operations:

Activation group reuse reduces repeated buffer reads and arithmetic while compressing the input indirection table. Its applicability depends on cross-filter overlap, which quantized networks can provide.

  • 1) Reduced input buffer reads and arithmetic operations:: Reusing sub-expressions eliminates buffer reads and additions while retaining the multiplication savings of dot product factorization.The paper specifically identifies reused expressions such as x+h as avoiding repeated reads and additions.
  • 1) Reduced input buffer reads and arithmetic operations:: The input indirection table can shrink by an O(G) factor because later filters need not reread reused activation groups.This compression follows from traversing shared activation groups only once.
  • Applicability: Activation group reuse requires overlaps between filters’ activation groups and cannot form reusable compound expressions when no overlaps exist.The paper expects overlaps when filter size is larger than U^G.
  • Applicability: INQ and TTQ retraining enabled G > 1 in experiments, with most ResNet-50 layers satisfying G = 2 to 3 and G = 6 to 7, respectively.The passage also states that these schemes can retain competitive classification accuracy relative to larger-U schemes.
  • C. Partial Product Reuse: The paper does not exploit partial-product reuse across filters within the same input channel because it is incompatible with the other two techniques.This reuse spans the RSK dimensions and is presented as an unexploited opportunity.

A. Baseline Design: DCNN PE

The baseline DCNN processing element computes tiled convolution dot products with conventional elementwise MACs. UCNN retains this structure but adds a factorized two-step datapath, indirection handling, and accumulators for repeated weights.

  • A. Baseline Design: DCNN PE: The DCNN and UCNN processing elements compute dot products between an RSC input region and one or more filters.The accelerator consists of multiple processing elements connected to a shared global buffer.
  • A. Baseline Design: DCNN PE: A DCNN PE multiplies each filter element with its corresponding input and accumulates the results into partial sums across RSC_t-sized tiles.Its input buffer, weight buffer, partial-sum buffer, control logic, and MAC unit support this tiled computation.
  • A. Baseline Design: DCNN PE: The DCNN datapath feeds one weight-input pair per cycle into a fixed-point multiplier and adder, forming a scalar DCNN lane.Vectorization can replicate lanes across output channels, though DCNN and UCNN favor different strategies.
  • UCNN Modification: UCNN breaks each dot product into an inner activation-group sum and an outer weight multiply-accumulate.This modification retains the baseline PE’s basic design and components while changing its dataflow.
  • UCNN Modification: Sorted input and weight indirection tables let UCNN read activation groups sequentially and encode weight-group transitions compactly.The weight indirection table can use one transition bit per entry, while zero weights are sorted last and skipped.
  • Arithmetic bitwidth: UCNN limits activation groups to 16 elements, splitting larger groups to constrain operand widening while retaining up to 16× multiplication reduction in the best case.The resulting multiplier is 4 bits wider on one input under this design assumption.

C. Activation Group Reuse

Activation group reuse shares one input indirection table across multiple filters, reducing model storage and enabling shared sub-computations. Hierarchical sorting and multi-level accumulation extend this reuse while introducing handling requirements for empty groups and sparsity.

  • Activation group reuse: Sharing an input indirection table across G filters reduces model size and can share sub-computations, saving buffer reads and improving PE throughput.G = 1 is the vanilla dot-product-factorization case.
  • Hierarchical sorting: Hierarchical sorting orders entries by nested activation groups using one canonical weight order, allowing a single input-table traversal to serve multiple filters.The weight indirection tables can then use one bit per entry per filter.
  • Example: For G = 2, UCNN completes the illustrated filter computation with 6 multiplies, versus 16 for DCNN with two lanes.The example walks input and weight indirection tables over time while accumulating sub-activation groups.
  • Hierarchical sorting: O(G) indirection-table compression follows from extending the same hierarchical sorting scheme to additional filter levels.The effective table size is (|iiT.entry|+G∗|wiT.entry|)/G = ⌈log2 RSCt⌉/G+1.
  • Datapath: A third accumulator merges lower-level activation-group sums, while boundary-triggered sums are dispatched to the MAC unit.This datapath supports accumulations across different activation-group levels.
  • Edge cases: Empty sub-activation groups require counter extensions or skip entries; skip entries can introduce one cycle bubble per skipped weight.The implementation combines extra counter bits with skip entries for longer skip distances.
  • Sparsity: Weight sparsity limits removable indirection entries because shared filters require the union of their non-zero weights.The architecture still provides an O(G) compression factor for indirection tables, but fewer entries are skippable.

D. Spatial Vectorization

Spatial vectorization addresses UCNN’s irregular indirect input-buffer accesses by reusing indirection-table lookups across spatial lanes. Banking enables parallel reads and conflict-free access, with bounded storage overhead.

  • Spatial vectorization: Spatial vectorization amortizes indirection-table lookups across spatial lanes to offset UCNN’s extra indirect input-buffer access and irregular SRAM reads.The vector size is controlled by VW.
  • Vectorized datapath: Replicating the activation-group-reuse datapath across lanes improves PE throughput to O(G∗VW) relative to the non-vectorized baseline.The input buffer capacity remains O(Ct∗S∗(VW+R)) because successive filter slides overlap.
  • Banking: The input buffer is divided into VW banks and filled using spatial-vector base addresses to support VW activation reads per cycle.The strategy is designed specifically for UCNN’s irregular access pattern.
  • Banking: The banking strategy is conflict-free because varying the vector slot produces different banks for a fixed spatial-vector base address.The bank function yields a different output for each fixed (r,s,c) as v varies.
  • Storage overhead: Input-buffer addressability has a fractional storage overhead below 2×, and some R,VW settings eliminate the overhead entirely.The passage gives VW = 2 for R = 3 as an example with no overhead.

E. UCNN Design Flexibility

UCNN’s broader design uses flexible buffering, stationary dataflow, and support for multiple CNN layer types. Its dataflow prioritizes on-chip inputs, weight reuse, and output-stationary partial sums while accommodating practical layer and capacity constraints.

  • Weight flexibility: UCNN supports networks with varied unique-weight counts by reserving a larger PE weight buffer, including networks not retrained for quantization.Even with large U, redundant weight-buffer accesses can still be removed.
  • Layer support: Nonlinear activations execute at the PE, pooling requires minimal additional logic, and fully connected layers are implemented as convolutions without input-buffer slide reuse.Arithmetic is disabled during pooling.
  • System organization: The architecture connects multiple PEs to shared global input and weight buffers, with multicast buses feeding PE-local buffers.This organization applies to both DCNN and UCNN.
  • Dataflow: Inputs are kept on chip when possible, each PE produces an output column, and partial sums remain in the PE until final outputs are generated.The top-level dataflow seeks to minimize DRAM traffic, especially for weights.
  • Capacity: Most inputs fit on chip with several hundred KB of L2 storage; otherwise, the input is spatially tiled.The cited example states that all but several ResNet-50 layers fit with 256 KB and 8-bit activations.
  • PE dataflow: PE-level dataflow exposes RSC regions of weights and inputs, matching dot-product factorization and reducing partial-sum movement.The RSC regions are placed in the innermost PE loops.
  • Execution: Weights are fetched from DRAM in Kc-filter chunks, while work is distributed across input columns and filters before streaming into PE-local buffers.The final partial sum is stored locally and written to L2 after output generation.

VI. EVALUATION

The evaluation compares UCNN with dense and sparsity-optimized baselines across three CNNs, precision settings, weight densities, and unique-weight configurations. UCNN’s energy benefits arise from activation reuse, reduced indirection-table entries, and repeated-weight computation reuse.

  • Measurement setup: The evaluation uses cycle and energy models plus RTL processing-element implementations under a 32 nm, 1 GHz setup.Arithmetic, SRAM, DRAM, and NoC energy are modeled, while RTL designs are synthesized for hardware validation.
  • Compared designs: DCNN sp skips zero-operand multiplies and compresses DRAM data, while UCNN additionally reuses repeated-weight computations and activation groups.UCNN variants reduce DRAM accesses, memory reads, multiplies, and adds per dot product.
  • Evaluation scope: Three evaluated networks are LeNet, AlexNet, and ResNet-50, with 8-bit and 16-bit weights and 90%, 65%, and 50% weight densities.Input activation density is fixed at 35%, and UCNN is evaluated with U = 3, 17, 64, and 256 unique weights.
  • Energy results: At 16-bit precision, UCNN U3, U17, and U256 reduce ResNet-50 energy by up to 3.7×, 2.6×, and 1.9× relative to DCNN sp at 50% density.The reductions combine activation-group reuse, lower PE arithmetic energy, and fewer indirection-table entries at lower weight density.
  • Cross-network results: Across networks, improvement ranges from 1.2×–4× at 90% density and 1.7×–3.7× at 50% density for 16-bit precision.At 8-bit precision on ResNet-50 with 50% density, improvements for U3, U17, and U256 are 2.6×, 2×, and 1.6× relative to DCNN sp.
  • Layer analysis: Early layers benefit from arithmetic efficiency, whereas later layers benefit more from reduced memory traffic because filter size increases with C and K.The layer analysis concerns ResNet-50 at 50% weight density and 16-bit precision.

C. Performance Analysis

The performance analysis studies idealized sparsity benefits and real INQ data, revealing a trade-off between energy savings and runtime as activation-group size changes. Real implementation overheads reduce the ideal speedup, but UCNN retains a performance-per-watt advantage.

  • Optimistic performance analysis: Larger activation groups reduce energy per inference, while smaller groups such as G = 1 can improve runtime.Grouped indirection tables store the union of non-zero weights across filters, creating an energy–performance trade-off.
  • Performance on real INQ data: On real INQ data, UCNN improves performance by 0.7% with G = 1, versus a modeled best case of 10%.The INQ-trained model has 90% weight density, matching the cited training data.
  • Performance on real INQ data: With DCNN vectorization at VK = 2, DCNN performance improves by 2×, whereas throughput-normalized UCNN G = 2 improves by 1.80×.The gap from the ideal 2× UCNN improvement is largely attributed to indirection-table skip entries.
  • Performance on real INQ data: Despite implementation-dependent performance overheads, UCNN retains a significant performance-per-watt advantage over DCNN configurations.The reported deficit in runtime is dominated by UCNN’s energy savings.

D. Model Size (DRAM storage footprint)

UCNN’s activation-group reuse compresses model storage, often producing smaller models than DCNN sp and competitive sizes relative to TTQ and INQ. Jump-based indirection tables further reduce storage with limited performance penalties in tested cases.

  • UCNN versus DCNN sp: UCNN variants with G > 1 are significantly smaller than DCNN sp across all evaluated weight densities.The G = 1 configuration can exceed DCNN sp model size at higher weight densities because it lacks activation-group reuse.
  • Comparison with quantization schemes: At 50% density, UCNN G = 4 requires approximately 3.3 bits per weight, while at 90% density UCNN G = 2 requires 5–6 bits per weight.The configurations are compared with TTQ- and INQ-style quantization settings.
  • Comparison with quantization schemes: At 30% density, UCNN model size falls below 3 bits per weight, a regime associated with approximately 1% accuracy loss in the cited prior work.The paper reports UCNN sizes as competitive with leading quantization schemes while also supporting on-chip energy reduction.
  • Jump-based indirection tables: Jump tables reduce G = 1 storage from 11 to 8 bits per weight with approximately 2% performance overhead.For G = 2, reducing storage from 6 to 5 bits per weight matches INQ’s model size with negligible performance penalty.
  • Hardware cost: The UCNN processing element adds 17% area with a 17-entry weight buffer and 24% when provisioned for 256 weights.Both UCNN and DCNN RTL designs meet timing at 1 GHz in a 32 nm process.

VII. RELATED WORK

UCNN extends prior CNN-efficiency techniques by exploiting repeated non-zero as well as zero weights, combining computation reuse with model compression. Its design differs from Winograd, sparse accelerators, and weight-sharing methods in supported repetition patterns and hardware trade-offs.

  • Sparsity and repeated weights: Sparse accelerators exploit repeated zero weights, whereas UCNN exploits repetition in both zero and non-zero weights.UCNN therefore generalizes the repetition opportunity targeted by sparsity-based designs.
  • Convolution computation reuse: Winograd factors convolution multiplies through predictable filter slides but does not exploit cross-filter weight repetition and loses effectiveness for non-unit strides.The paper identifies combining Winograd and weight-repetition techniques as future work because their computation regularity and reuse opportunities differ.
  • Lookup-based methods: Lookup-table multiplication can cost more energy than multiplication at the cited SRAM sizes, limiting its net efficiency benefit.The cited 32 nm examples compare 0.1 pJ and 0.4 pJ multiplies with 0.17 pJ and 2.5 pJ lookups for 8-bit and 16-bit cases.
  • Architectures exploiting repeated weights: Deep Compression and EIE reduce weight storage through sharing but do not reuse sub-computations or exploit overlapping repetitions across filters as aggressively.UCNN combines computation reuse with model-size compression.
  • Conclusion: UCNN improves energy efficiency up to 3.7× on three contemporary CNNs relative to an Eyeriss-style accelerator, reaching 4× with dense weights.The conclusion frames UCNN as a step toward exploiting repetition in all weights rather than only zero weights.
Loading 1804.06508v1…