Source-linked AI summary
Accelerating Sparse Deep Neural Networks
Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, Paulius Micikevicius
TL;DR
Large neural networks create pressure to reduce parameters and inference compute, while existing sparsity methods struggle to preserve accuracy and exploit modern matrix hardware. The paper introduces 2:4 structured sparsity, Sparse Tensor Cores, and a retraining workflow; it reports up to 2× sparse GEMM performance and maintained accuracy across varied tasks and architectures. The workflow simplifies deployment preparation, but its retraining stage doubles training time and lacks universally effective shorter schedules.
Problem
Existing pruning methods struggle to maintain model accuracy while gaining inference speed on modern matrix-math hardware.
Method
The paper combines 2:4 structured pruning, compressed sparse storage, Sparse Tensor Cores, and retraining of pruned dense models with fixed sparsity.
Results
The workflow maintains accuracy across a wide range of tasks and neural network architectures, while Sparse Tensor Cores provide up to 2× performance for equivalent dense GEMMs.
Takeaways & Limitations
The workflow prepares accurate models for inference acceleration on Sparse Tensor Cores without changing training hyper-parameters.
Takeaways & Limitations
The retraining workflow doubles training time, and shorter fine-tuning hyper-parameters are highly network- and task-dependent rather than universal.
Abstract
from arXiv · showhide
As neural network model sizes have dramatically increased, so has the interest in various techniques to reduce their parameter counts and accelerate their execution. An active area of research in this field is sparsity - encouraging zero values in parameters that can then be discarded from storage or computations. While most research focuses on high levels of sparsity, there are challenges in universally maintaining model accuracy as well as achieving significant speedups over modern matrix-math hardware. To make sparsity adoption practical, the NVIDIA Ampere GPU architecture introduces sparsity support in its matrix-math units, Tensor Cores. We present the design and behavior of Sparse Tensor Cores, which exploit a 2:4 (50%) sparsity pattern that leads to twice the math throughput of dense matrix units. We also describe a simple workflow for training networks that both satisfy 2:4 sparsity pattern requirements and maintain accuracy, verifying it on a wide range of common tasks and model architectures. This workflow makes it easy to prepare accurate models for efficient deployment on Sparse Tensor Cores.
1 Introduction
The paper addresses the difficulty of preserving accuracy while obtaining inference speedups from sparsity by combining a 2:4 pattern, a pruning workflow, and Sparse Tensor Cores.
- Motivation: Larger neural networks typically improve task accuracy but require hundreds of billions of parameters and trillions of operations per input.Pruning is presented as a way to reduce model size and inference compute requirements.
- Motivation: Existing pruning methods struggle to maintain accuracy and improve inference speed simultaneously.Fine-grained sparsity underutilizes GPU memory and math pipelines, while coarse-grained sparsity loses accuracy beyond moderate sparsity ratios.
- Structured sparsity and workflow: 2:4 sparsity requires at least two zeros in every group of four consecutive values, halving a model’s parameter count.The workflow prunes a densely trained model once, then retrains with the fixed sparsity pattern and the original hyper-parameters.
- Hardware support: Sparse Tensor Cores accelerate 2:4 sparse matrix operations in NVIDIA Ampere GPUs.They target matrix multiplication, which underlies convolutions, linear layers, recurrent cells, and transformer blocks.
- Contributions: The paper contributes structured pruning, compressed storage, sparse matrix-multiplication hardware, and an empirically verified retraining workflow.The workflow is evaluated across popular tasks and architectures without changing hyper-parameters.
2 Related Work
Prior pruning research spans multiple training and sparsity strategies, but the paper identifies an unresolved need for accuracy-preserving inference speedups with a consistent workflow across networks.
- Pruning strategies: Pruning approaches include dense training followed by fine-tuning, gradual pruning during training, and sparse training with predetermined or learned patterns.These categories differ in when sparsity is introduced and how the pattern is selected.
- Fine-tuning methods: Fine-tuning methods vary in pruning criterion, pruning schedule, sparsity structure, and fine-tuning schedule.Reported pruning criteria include magnitude-based and salience-based metrics, while schedules may be single-step or gradual.
- Fixed-pattern training: Some fixed-pattern approaches maintain accuracy at 80%-99% unstructured sparsity, but their runtime efficiency on modern matrix processors is not established.The paper states that unstructured sparsity is unlikely to use modern matrix processors efficiently.
- Gradual sparsification: Gradually sparse training commonly targets 70%-99% unstructured sparsity, often with model-specific changes to training or model hyper-parameters.These models are described as struggling to outperform dense models.
- Open challenge: No single described method simultaneously maintains accuracy and achieves inference speedup across varied networks.The paper frames this as the central unresolved challenge in prior pruning work.
- Open challenge: Existing fine-tuning studies provide inconclusive guidance on pruning schemes, pruning procedures, and fine-tuning schedules.The unresolved choices include magnitude versus heuristic pruning, one-shot versus iterative pruning, and the number of fine-tuning epochs.
3 Sparsity Support in the NVIDIA Ampere Architecture
The NVIDIA Ampere architecture supports 2:4 structured sparsity through compressed storage and Sparse Tensor Cores, targeting efficient memory access and up to 2× matrix-multiplication performance. Speedups depend on GEMM dimensions and arithmetic intensity.
- 2:4 sparsity requires at least two zeros in every group of four values, producing 50% sparsity.
- 3.2 Structured-Sparse GEMM on Tensor Cores: 2× math throughput is available for supported sparse matrix multiplications, while observed speedups approach 2× mainly for larger, higher-arithmetic-intensity GEMMs.Supported input/output formats include FP16, BF16, and 8-bit integer; dimensions must meet library-specific multiples.
- 3.1 2:4 Sparsity and Its Benefits: The compressed format stores only two nonzero values per four-value group and uses separate metadata to encode their positions.Metadata uses 2 bits to encode each nonzero value's position within the group.
- 3.1 2:4 Sparsity and Its Benefits: 2:4 sparsity enables efficient memory accesses by maintaining constant sparsity across sub-blocks and avoiding data-dependent indirection.
- 3.1 2:4 Sparsity and Its Benefits: ∼44% storage-capacity savings result for 16-bit operands, while 8-bit operands save ∼38% memory capacity and bandwidth versus dense tensors.
- 3.2 Structured-Sparse GEMM on Tensor Cores: Sparse Tensor Cores compute sparse matrix × dense matrix = dense matrix operations, skipping multiplications by zero in the sparse first operand.
4 Network Pruning Workflow
The workflow trains a dense model, applies one-shot 2:4 pruning, and repeats training with the same schedule while preserving the fixed sparsity pattern. Channel permutations can reduce pruning impact for parameter-efficient networks, and multi-phase models require choosing which training phases to repeat.
- The deployment-oriented workflow trades higher training cost for a simple, general procedure that avoids hyper-parameter searches.The additional training cost can be amortized over a deployment lifetime of days to months.
- 4.1 The Basic Workflow: The basic procedure trains without sparsity, prunes to a 2:4 pattern, then retrains from the pruned weights using the original optimizer and schedule.The sparsity pattern is maintained throughout retraining, and optimizer state is reset.
- 4.1 The Basic Workflow: Magnitude pruning removes the two smallest weights in each consecutive group of four, producing a sparse matrix from the dense weights.The workflow illustrates this transformation in Figure 5.
- 4.1 The Basic Workflow: Permuting weight columns before pruning distributes large values more evenly, reducing the magnitude lost from enforcing the 2:4 constraint.In the illustrated matrix, total weight magnitude increases from 83.7 without permutation to 102.9 with permutation after pruning.
- 4.1 The Basic Workflow: For parameter-efficient networks, channel permutations can help 2:4 sparsity maintain accuracy when ordinary pruning and fine-tuning still lose accuracy.Permuting corresponding activation-producing weights preserves the GEMM result.
- 4.2 Pruning Details: Pruning targets learnable layers that execute GEMM-like operations, excluding layers whose relevant dimensions are not multiples of 16 or 32 and embedding layers.The dimension constraints correspond to 16-bit floating-point and 8-bit integer formats.
- 4.3 Applying the Workflow to Models Trained in Multiple Phases: Models trained in multiple phases repeat different phases depending on downstream data size: BERT repeats pre-training, whereas detection models repeat only task-specific fine-tuning.Small downstream datasets may not provide enough updates for recovery, while large detection and segmentation datasets can suffice.
- 4.4 Combining Sparsity and Quantization: Quantization calibration, and any required fine-tuning, is recommended after the network has been pruned and retrained for sparsity.The passage describes this ordering for combining sparsity with quantization.
5 Results
Across image classification, detection, segmentation, translation, language modeling, and GAN evaluations, the workflow generally preserves dense-model accuracy under 2:4 sparsity. Some small parameter-efficient networks need column permutations, while aggressive INT8 quantization can cause a small BERT degradation.
- The evaluation spans diverse tasks and architectures, examining both floating-point networks and their INT8 quantized versions.The study uses architecture-specific training hyper-parameters and details from introducing papers or public repositories.
- 5.1 Image Classification Networks: Accuracy is maintained for floating-point and quantized image-classification networks, with differences from dense non-quantized models within run-to-run variation.The reported variation is attributed to random seeds or fine-tuning non-determinism.
- 5.1 Image Classification Networks: Lower-parameter networks including MobileNet v2, SqueezeNet, MNASNet, ShuffleNet v2, and EfficientNet may not fully recover accuracy with the basic workflow.Permuting weights before pruning enables full accuracy recovery for these models.
- 5.2 Object Detection and Image Segmentation: Sparse detection and segmentation models match the accuracy of their dense counterparts when pruning follows full network training and only the second phase is repeated.The evaluation uses COCO 2017 detection and segmentation networks.
- 5.5 Generative Adversarial Networks: The study also evaluates 2:4 sparse GANs using Frechet Inception Distance, where lower scores are better.The cited passage identifies the metric and evaluation scope but does not state the table’s comparative outcome.
- 5.3 Language Translation Networks: Retrained 2:4 sparse language-translation networks achieve accuracy matching their dense originals.The reported evaluation covers two translation architectures on En-De WMT’14 using BLEU scores.
- 5.4 Networks for Natural Language Processing (NLP): For BERT, sparsity matches dense accuracy, while aggressive INT8 quantization can cause a small accuracy degradation.All GEMM layers and residual-connection data operate on INT8 operands in this setting.
6 Conclusions and Future Work
The paper introduces 2:4 structured sparsity, hardware primitives for its acceleration, and a pruning workflow empirically shown to maintain accuracy across many tasks and models. Future work includes reducing retraining costs, extending acceleration to training, and developing general activation-pruning methods.
- Conclusions: 2× math throughput is provided for GEMM-like operations when the first tensor has 2:4 sparsity.Sparse Tensor Cores target convolutions and matrix multiplies in the NVIDIA Ampere GPU architecture.
- Conclusions: The workflow maintained accuracy across a wide range of tasks and neural network models using standard learning-rate schedules.The schedules came from public code repositories.
- Future Work: The workflow doubles training time because it repeats a training session after pruning trained dense-network weights.Shorter fine-tuning schedules maintained accuracy in preliminary experiments, but their hyper-parameters were highly network- and task-dependent.
- Future Work: The workflow targets inference acceleration; accelerating backward training requires 2:4 constraints on transposed weights along both matrix dimensions.Dynamic sparsity masks may also be needed to reduce dense-network updates during training.
- Future Work: A fully general methodology for pruning activations with a 2:4 pattern remains future work.This direction matters for layers such as Transformer multi-head attention that do not involve weights.