PaperlayerView exact revision on arXiv

Source-linked AI summary

S2TA: Exploiting Structured Sparsity for Energy-Efficient Mobile CNN Acceleration

Zhi-Gang Liu, Paul N. Whatmough, Yuhao Zhu, Matthew Mattina

arXiv:2107.07983v2cs.ARcs.LG

TL;DR

Unstructured sparsity can accelerate CNN inference, but its unpredictable patterns require hardware overhead that is particularly costly for mobile and IoT devices. This paper uses DBB sparsity for weights and activations in S2TA, a systolic-array accelerator. S2TA-AW reports 2.08× lower energy and 2.11× speedup than SA-ZVCG on full DNNs.

  • Problem

    Unstructured sparsity requires complex hardware and buffering, creating energy and area overheads that burden mobile and embedded inference.

    Source: §1 · p. 1

  • Method

    S2TA applies DBB sparsity to weights and activations using pruning and hardware support integrated into a systolic array.

    Source: §1 · p. 1, §5.1 · p. 5

  • Results

    2.08× average energy reduction and 2.11× speedup versus SA-ZVCG are reported for S2TA-AW on full DNNs.

    Source: §8.4 · p. 11

  • Takeaways & Limitations

    S2TA-AW reports 1.4–4.7× lower energy than state-of-the-art accelerators exploiting unstructured sparsity.

    Source: §8.4 · p. 11

    DBB sparsity is lossy, so DNN fine-tuning is required to regain accuracy loss; DAP must also be incorporated during training to prevent inference-time accuracy loss.

    Source: §10 · p. 12

Abstract

from arXiv · show

Exploiting sparsity is a key technique in accelerating quantized convolutional neural network (CNN) inference on mobile devices. Prior sparse CNN accelerators largely exploit un-structured sparsity and achieve significant speedups. Due to the unbounded, largely unpredictable sparsity patterns, however, exploiting unstructured sparsity requires complicated hardware design with significant energy and area overhead, which is particularly detrimental to mobile/IoT inference scenarios where energy and area efficiency are crucial. We propose to exploit structured sparsity, more specifically, Density Bound Block (DBB) sparsity for both weights and activations. DBB block tensors bound the maximum number of non-zeros per block. DBB thus exposes statically predictable sparsity patterns that enable lean sparsity-exploiting hardware. We propose new hardware primitives to implement DBB sparsity for (static) weights and (dynamic) activations, respectively, with very low overheads. Building on top of the primitives, we describe S2TA, a systolic array-based CNN accelerator that exploits joint weight and activation DBB sparsity and new dimensions of data reuse unavailable on the traditional systolic array. S2TA in 16nm achieves more than 2x speedup and energy reduction compared to a strong baseline of a systolic array with zero-value clock gating, over five popular CNN benchmarks. Compared to two recent non-systolic sparse accelerators, Eyeriss v2 (65nm) and SparTen (45nm), S2TA in 65nm uses about 2.2x and 3.1x less energy per inference, respectively.

1. INTRODUCTION

Unstructured sparsity can reduce computation but requires complex buffering and data reordering that burden energy- and area-constrained mobile INT8 accelerators. The paper proposes DBB sparsity for weights and activations, with hardware primitives and a systolic-array design, and reports lower energy than its baseline and unstructured sparse competitors.

1. INTRODUCTION

  • Unstructured sparsity raises hardware overhead: added operand and accumulator buffers increase a baseline systolic array's energy per MAC by 71%.The paper attributes the overhead to operand gathering or result scattering and large buffers for data manipulation.Source: §1 · p. 1
  • DBB bounds the maximum non-zeros per block, fixing the maximum MAC count and limiting output locations to avoid operand buffers and distributed accumulators.This regularity is proposed as a way to keep sparsity-exploiting hardware lean for mobile and embedded inference.Source: §1 · p. 1
  • W-DBB and A-DBB target weight and activation sparsity, respectively; dynamic activation sparsity is handled by pruning co-designed with time-unrolled hardware.DAP compresses activations by 2–3× with negligible impact on test accuracy; it keeps the largest-magnitude elements when a block exceeds its non-zero bound.Source: §1 · pp. 1–2
  • S2TA integrates joint weight and activation DBB sparsity into a systolic array using tensor PEs that consume compressed DBB blocks.The proposed time-unrolled microarchitecture serializes MAC processing within a DBB block.Source: §1 · p. 2
  • The paper reports that unstructured-sparsity operand and accumulator buffers add about 50% energy and 10% area overhead relative to a baseline dense INT8 datapath.The authors note that these overheads can eclipse gains in mobile INT8 designs, unlike floating-point designs where they are amortized.Source: §1 · p. 2
  • Across AlexNet, MobileNetv1, VGG16, and ResNet50v1, S2TA demonstrates 2.08× lower energy than the baseline without DBB; on AlexNet in 65nm technology, it has 2.2× and 3.1× lower energy than SparTen and Eyeriss-v2, respectively.Source: §1 · p. 2

2. MOTIVATION

Unstructured sparsity can reduce computation and bandwidth, but its unpredictable patterns require costly data reordering and buffering. Operand-gather and result-scatter designs illustrate why these overheads challenge efficient mobile INT8 inference.

2. MOTIVATION

  • Sparse GEMM can reduce MAC operations and memory bandwidth by storing only nonzero values and their positions, but irregular patterns require costly data handling.Gather stages reorder operands; scatter stages distribute partial products to output locations.Source: §2.1 · pp. 2–3, §2 · p. 2

2.1 Sparsity-Exploiting Structures Must be Efficient for Mobile DNN Inference

  • Unstructured sparse GEMM disrupts regular computation and requires buffering and data reordering, which can raise energy and area costs in INT8 mobile accelerators.The dense accelerator’s buffers already dominate energy, so added buffering can offset sparsity gains.Source: §2.1 · p. 2

2.2 Overhead 1: Operand Gather Structure

  • Operand gathering stages matching nonzero pairs in FIFOs, and INT8 SA-SMT variants achieve 1.6× and 1.8× speedup but use nearly 50% more power than SA-ZVCG.The variants have roughly the same area as SA-ZVCG; the comparison uses a typical convolution with 50% weight and activation sparsity.Source: §2.2 · p. 3

2.3 Overhead 2: Result Scatter Structure

  • Result scattering requires large read-modify-write accumulator buffers; SCNN uses 1.65 KB per PE and SparTen about 1 KB, versus 2B for operands and 4B for accumulators in a systolic array.The buffer comparison is per PE.Source: §2.3 · pp. 3–4, Table 1 · p. 3

3. MAIN IDEA AND DESIGN OVERVIEW

The paper proposes density bound block (DBB) sparsity to make sparsity patterns more predictable for lean hardware. It applies DBB to weights and activations, then integrates both in a systolic array using tensor processing elements that expose additional data reuse.

3.1 Exploiting Structured Sparsity

  • DBB bounds the maximum nonzeros per tensor block, making maximum workload predictable at design time and reducing load imbalance and the need for distributed accumulators.DBB blocks are processed in order, and results are generated sequentially without data reordering.Source: §3.1 · p. 4
  • A 4/8 DBB block stores nonzero elements and a positional bitmask; blocks with fewer than four nonzeros retain zeros in compressed form, while dense mode remains supported.The example uses blocks of eight along the channel dimension.Source: §3.1 · p. 4, Figure 5 · p. 4

3.2 Design Overview

  • The design combines static weight DBB with dynamic activation DBB, using dynamic pruning and a time-unrolled architecture for activations.Activation sparsity is dynamic, unlike weight sparsity, which is known statically.Source: §3.2 · p. 4
  • Grouping processing elements in a classic systolic array integrates both weight and activation DBB support into tensor PEs, naturally exposing additional data-reuse dimensions unavailable in traditional systolic-array design.Source: §3.2 · p. 4

4. STATIC WEIGHT DBB SPARSITY

Weight DBB bounds nonzeros per block, allowing hardware to be provisioned for a known workload. The proposed 4/8 datapath uses fewer MACs and less weight bandwidth while retaining dense fallback support.

4. STATIC WEIGHT DBB SPARSITY

  • Bounding nonzeros per weight block relaxes load imbalance and lets hardware provision for the maximum block workload.The paper situates weight DBB among prior work and applies the same general DBB principle to activations as well.Source: §4 · p. 5
  • A 4/8 W-DBB datapath uses four MACs instead of eight for the same throughput, reducing MAC count by 50% and weight operand bandwidth by 37.5%.The DP4M8 datapath uses an 8-input mux controlled by the weight block’s positional bitmask.Source: §4 · p. 5
  • The 8:1 mux steers the correct activation into each MAC, and the DP4M8 datapath supports fallback to dense operation for models with density above 50%.The paper describes mux overhead as negligible compared with the hardware required for unstructured sparsity.Source: §4 · p. 5

5. DYNAMIC ACTIVATION DBB SPARSITY

Dynamic activation pruning compresses runtime activations into DBB blocks by retaining the largest-magnitude values. A time-unrolled architecture supports variable activation densities across layers while maintaining datapath utilization and operand bandwidth.

5.1 Dynamic Activation Pruning (DAP)

  • Dynamic Activation Pruning compresses runtime activation blocks to a maximum NNZ by retaining the largest-magnitude elements, but small NNZ/BZ can reduce test accuracy.For MobileNetV1, applying 4/8 DAP to all point-wise layers drops test accuracy from 71% to 56.1%; the paper proposes training extensions to recover accuracy.Source: §5.1 · p. 5

5.2 Architectural Support for A/W-DBB

  • Joint weight and activation DBB compresses both operands and reduces SRAM loading power, using a 4:1 mux before each MAC in the described vector datapath.Matching nonzero positions are identified by comparing the index bitmasks; unused MACs can be clock-gated.Source: §5.2 · p. 5
  • Activation DBB density varies by layer, so fixing it at 4/8 can compromise gains; ResNet50v1’s tuned activation density ranges from 8/8 in early layers to 2/8 toward the end.A naive 4/8 datapath running 2/8 DBB would have about 50% utilization.Source: §5 · p. 5, §5.2 · p. 6
  • Time-unrolling serializes activation-block elements across cycles, allowing density to vary from 1/8 to 8/8 while datapath utilization and operand bandwidth remain constant.A 1/8 block takes one cycle; a 5/8 block takes five cycles.Source: §5.2 · p. 6

6. STRUCTURED SPARSE TENSOR ACCELERATOR (S2TA)

S2TA integrates weight and activation DBB sparsity into a systolic array by replacing scalar processing elements with tensor PEs. Its design supports dynamic activation pruning and reuses tensor operands and accumulators to reduce buffering.

6.1 Overall Architecture

  • To support DBB sparsity in a systolic array, a tensor PE replaces the scalar PE: it accepts a pair of fixed-size operand blocks per cycle, rather than one operand pair.Because DBB weights and activations are naturally blocked, each TPE computes MACs between an activation tensor and a weight tensor.Source: §6.1 · p. 6
  • The time-unrolled S2TA-AW connects scalar datapaths in an outer-product design, while S2TA-W uses a dot-product datapath for weight sparsity alone.The TPE organization also reuses operands within each PE and reuses accumulators across multiple MACs, amortizing operand movement and reducing buffer size.Source: §6.1 · pp. 6–7

6.2 Hardware Support for DAP

  • Dynamic activation pruning selects the largest-magnitude elements from each block using cascaded maxpool stages, with unused stages bypassed.The design uses blocks of 8 elements and caps the stages at 5, supporting A-DBB sparsity from 1/8 to 5/8.Source: §6.2 · p. 7

6.3 Other Design and Implementation Details

  • The accelerator uses separate, double-buffered 0.5MB weight and 2MB activation SRAMs, while four Cortex-M33 MCUs handle non-GEMM operations and control.The SRAM buffers overlap TPE computation and DMA transfers; the MCU cluster is sized so it does not become the performance bottleneck.Source: §6.3 · p. 7

7. METHODOLOGY

The methodology uses a parameterized RTL generator to explore S2TA configurations under a fixed throughput target, then evaluates performance, power, and area through simulation and physical-design flows. Comparisons include systolic-array and non-systolic sparse baselines.

Automatic RTL Generation

  • A parameterized Python RTL generator explores TPE dimensions, array dimensions, and combinations of W-DBB, A-DBB, zero-value clock gating, and time-unrolling.Generated RTL and testbenches are validated in Synopsys VCS, with switching-activity traces used for annotated power simulation.Source: §7 · p. 7

S2TA-A/W Design Point

  • The design-space sweep selects the lowest-power TPE array on the area-versus-power frontier under 4 TOPS peak throughput and fixed clock-frequency constraints.The selected S2TA-AW is an 8×4×4_8×8 outer-product TPE using time-unrolled DP1M4, 4/8 W-DBB, and variable A-DBB.Source: §7 · pp. 7–8

Baselines and Evaluation

  • The evaluation compares the listed baselines; all systolic-array designs have 4 TOPS peak dense throughput and otherwise identical configurations, with performance, power, and area measured using the same EDA flow.SparTen and Eyeriss-v2 PPA metrics are taken directly from their papers.Source: §7 · p. 8

8. EVALUATION RESULTS

S2TA-AW combines weight and activation DBB sparsity, with fine-tuning used to recover accuracy losses. Across full CNN models it reduces energy and improves speed over SA-ZVCG, while comparisons show energy advantages over sparse accelerator baselines.

8.1 Accuracy Results

  • 4/8 W-DBB typically incurs less than 0.5% accuracy loss, while joint A/W-DBB has 0.1%–0.4% more accuracy loss than W-DBB alone.DBB pruning is lossy and requires fine-tuning; the joint-sparsity results include per-layer tuning of activation density.Source: §8.1 · p. 8

8.2 Microbenchmarking Results

  • S2TA-AW achieves up to 9.1× energy reduction versus SA-ZVCG, with speedup increasing from 1× at dense activations to 8× at 12.5% activation density.Unlike S2TA-W's 2× speedup cap, S2TA-AW supports variable activation compression.Source: §8.2 · p. 8, Figure 9 · p. 9
  • SA-SMT's T2Q2 and T2Q4 variants consume 43.0% and 41.2% more energy than SA-ZVCG, respectively, due to operand-staging buffer overhead.S2TA-AW's energy benefits over S2TA-W mainly come from a 3.1× reduction in SRAM energy through activation sparsity exploitation.Source: §8.2 · p. 9, Figure 10 · p. 9

8.3 Full Model Inference Results

  • Across four CNN models, S2TA-AW reduces energy by 1.76–2.79× and achieves 1.67–2.58× speedup versus SA-ZVCG.It also averages 1.84× lower energy and 1.26× speedup versus S2TA-W.Source: §8.3 · pp. 9–10, Figure 11 · p. 10

8.4 Summary of Results

  • 2.08× average energy reduction and 2.11× speedup versus SA-ZVCG were reported for S2TA-AW on full DNNs.The comparison is on full models; the figure reports results normalized to SA-ZVCG.Source: Figure 11 · p. 10, §8.4 · p. 11

8.3 Full Model Inference Results

  • On AlexNet, S2TA-AW in 65nm uses 2.2× less energy per inference than SparTen in 45nm; Eyeriss-v2 consumes 3.1× more total energy than S2TA-AW in the same 65nm process.SparTen performs best on layers with very high sparsity, while Eyeriss-v2 comparisons also report higher S2TA-AW energy efficiency on MobileNet and AlexNet convolution layers.Source: Figure 12 · p. 10, §8.3 · p. 11
  • Fully connected and depthwise layers are memory bound on S2TA when inference does not use batching.These layers are still pruned and included in the full-model results.Source: §8.3 · p. 11

8.4 Summary of Results

  • S2TA-AW achieves 1.4–4.7× lower energy than state-of-the-art accelerators exploiting unstructured sparsity, including SparTen, Eyeriss-v2, and SA-SMT.The summary reports full-accelerator synthesis data with typical weight and activation sparsity.Source: §8.4 · p. 11

9. RELATED WORK

Related work spans indexed unstructured sparsity, DBB weight sparsity, and systolic-array approaches. S2TA is positioned as the first systolic-array architecture to exploit both weight and activation DBB sparsity.

Indexed Unstructured Sparsity

  • Indexed unstructured-sparsity accelerators use CSR or bitmask encodings, while SparTen and Eyeriss v2 support fully sparse inference.The paper compares with SparTen and Eyeriss v2 while focusing on DBB sparsity.Source: §9 · p. 11

DBB Weight Sparsity

  • Kang targets fixed 2/8 W-DBB with a dot-product design and limited data reuse; Nvidia A100 supports fixed 2/4 W-DBB.The cited A100 peak is 3.12 TOPS/W, versus 12.4 TOPS/W for the S2TA-W baseline.Source: §9 · p. 11

Sparsity in Systolic Arrays

  • S2TA is the first architecture described here to exploit both W-DBB and A-DBB in a systolic array, using time-unrolling for up to 8× peak speedup.Earlier systolic-array work cited includes W-DBB support and unstructured-sparsity approaches using staging FIFOs.Source: §9 · p. 11

Bit-wise Sparsity

  • Bit-wise sparsity is presented as orthogonal to DBB sparsity and as a possible avenue for future DBB research.Examples cited include Pragmatic, Tactical, and Laconic.Source: §9 · p. 11

10. CONCLUSION

S2TA-AW combines weight and activation DBB sparsity in a time-unrolled systolic architecture. On pruned INT8 benchmarks, it reduces energy versus two baselines and outperforms cited unstructured sparse accelerators in energy efficiency.

10. CONCLUSION

  • S2TA-AW implements joint weight and activation DBB sparsity in a time-unrolled systolic architecture.Weights are pruned during training, while Dynamic Activation Pruning enforces activation DBB sparsity at runtime.Source: §10 · pp. 11–12
  • S2TA-AW achieves up to 8× speedup in the optimal time-unrolled, variable-DBB design.The design summary identifies S2TA-AW as a low-overhead fully sparse architecture.Source: Table 5 · p. 12
  • Dynamic Activation Pruning is lossy and must be incorporated during training to prevent accuracy loss at inference.Source: §10 · p. 12
  • 2.08× and 1.84× energy reduction on average versus SA-ZVCG and weight-only S2TA-W, respectively, on four pruned INT8 benchmark models.The models are AlexNet, MobileNetv1, VGG16, and ResNet50v1.Source: §10 · p. 12
  • On AlexNet, S2TA-AW has about 2× and 3× the energy efficiency of SparTen and Eyeriss-v2, respectively.Source: §10 · p. 12
Loading 2107.07983v2…