PaperlayerView exact revision on arXiv

Source-linked AI summary

Accelerating Deep Convolutional Networks using low-precision and sparsity

Ganesh Venkatesh, Eriko Nurvitadhi, Debbie Marr

arXiv:1610.00324v1cs.LGcs.NE

TL;DR

Deeper convolutional networks carry steep compute and memory costs, motivating methods that improve efficiency without sacrificing accuracy. This paper trains 2-bit weight networks and uses zero-skipping in a deep-learning accelerator. It reports strong ImageNet accuracy, reduced compute relative to a comparable full-precision network, and high accelerator performance density.

  • Problem

    Deeper networks improve accuracy but sharply increase compute and memory requirements, making training longer and deployment harder.

    Source: §1 · p. 1

  • Method

    The paper uses low-precision weights to reduce compute and develops DLAC to exploit sparsity by skipping zero-valued operations.

    Source: §1 · p. 1

  • Results

    76.6% Top-1/93% Top-5 on Imagenet is reported for the low-precision network, and DLAC reaches up to 1 TFlop/mm2 equivalent performance for many Residual-network layers.

    Source: §1 · p. 1

  • Takeaways & Limitations

    A low-precision larger network can achieve higher accuracy with less compute than a regular full-precision network.

    Source: §4.2 · p. 4

Abstract

from arXiv · show

We explore techniques to significantly improve the compute efficiency and performance of Deep Convolution Networks without impacting their accuracy. To improve the compute efficiency, we focus on achieving high accuracy with extremely low-precision (2-bit) weight networks, and to accelerate the execution time, we aggressively skip operations on zero-values. We achieve the highest reported accuracy of 76.6% Top-1/93% Top-5 on the Imagenet object classification challenge with low-precision network\footnote{github release of the source code coming soon} while reducing the compute requirement by ~3x compared to a full-precision network that achieves similar accuracy. Furthermore, to fully exploit the benefits of our low-precision networks, we build a deep learning accelerator core, dLAC, that can achieve up to 1 TFLOP/mm^2 equivalent for single-precision floating-point operations (~2 TFLOP/mm^2 for half-precision).

1. INTRODUCTION

Deeper convolutional networks improve accuracy but demand substantially more compute and memory, making training and deployment harder. The paper combines low-precision weights with zero-skipping and reports strong ImageNet accuracy alongside lower compute and high accelerator performance density.

1. INTRODUCTION

  • The paper targets the steep compute and memory costs of deeper networks, which also make training longer and deployment harder.Multiple-week training times are common even with multiple GPU cards.Source: §1 · p. 1
  • The approach builds on recently proposed low-precision convolution networks [17] to reduce their compute requirements, and demonstrates high accuracy using low-precision weights.Source: §1 · p. 1
  • A low-precision Resnet-34 achieves higher accuracy than vanilla Resnet-18 while requiring ∼3× fewer floating-point operations and a 7× smaller model.The comparison is reported as similar or slightly better accuracy at lower compute complexity.Source: §1 · p. 1
  • 76.6% Top-1/93% Top-5 on Imagenet is the reported accuracy for the 2-bit weight network, within 1.3% of the 2015 Imagenet winner.The authors describe it as the highest reported accuracy with a low-precision network to their knowledge.Source: §1 · p. 1
  • 14nm synthesis evaluation reports up to 1 TFlop/mm2 equivalent performance for many layers in state-of-the-art Residual networks, an order of magnitude above a previously proposed accelerator.DLAC exploits available sparsity by skipping operations on zero values and is applicable to training and inference.Source: §1 · p. 1

2. LOW-PRECISION DEEP CONVOLUTION NETWORK

The section presents ternary low-precision weights and dynamic sparsity as ways to reduce compute while retaining accuracy, then describes the evaluation and training techniques used. It also identifies zero-valued operands as an opportunity for further acceleration.

2.1. Background: Lowering the precision of network weights

  • Ternary weights take values [-1, 0, 1], and the cited approach can provide 16–32× smaller models with very few floating-point multiplications.The paper contrasts this with half precision, which can provide almost 2× savings.Source: §2.1 · p. 2
  • Unlike pruning's clamped zero weights or activations, dynamic sparsity allows those zeros to change across input samples and training phases, achieving >2× more sparsity.The paper says it demonstrates applicability on state-of-the-art networks.Source: §2.1 · p. 2

2.2. Evaluation

  • The evaluation uses Cifar10 with VGG and Imagenet with Resnet; Imagenet contains 1.2 million training and 50,000 test images across 1,000 classes.Accuracy testing uses single-crop testing; the accelerator is synthesized in a 14nm Intel process.Source: §2.2 · p. 2

2.3. Training Low-precision Networks

  • Training starts in full precision for 15 iterations before switching to low precision for about 75 iterations, improving accuracy by almost 2%.The network's first layer remains full precision to limit information loss from the input image, improving Top-1 accuracy by ∼0.5%.Source: §2.3 · p. 2

2.4. Sparsity in Low-precision Networks

  • 16% of forward-pass operations and around 33% of backward-pass operations use non-zero operands in low-precision Resnet-34 training on Imagenet; skipping zeros is reported to improve performance by 3–6×.Figure 1 plots the fraction of operations on non-zero operands for this training setting.Source: §2.4 · p. 2, Figure 1

2.3. Training Low-precision Networks

  • An adaptive learning-rate reduction can improve Top-1 accuracy by more than 1% in some instances; activation regularization reduces noise and induces sparsity.The paper also varies ReLU thresholds to induce higher sparsity and reduce noise; regularization tends to smooth the convergence curve.Source: §2.3 · p. 2

2.4. Sparsity in Low-precision Networks

  • Zero values arise when ReLU zeros activations below threshold 0.01 and when low-precision conversion zeros small-magnitude weights.The paper links these sources of sparsity to zero-skipping during training and inference.Source: §2.4 · p. 2

3. DEEP LEARNING ACCELERATOR

DLAC is a processing-element grid designed to exploit sparsity in low-precision networks by skipping operations on zero values. Its supported operations and training and inference configurations target different workloads.

3. DEEP LEARNING ACCELERATOR

  • DLAC is a two-dimensional processing-element grid with weight and input-feature-map buffers, arithmetic units, output buffers, and control logic for skipping zero-valued operations.Figure 2 depicts the architecture as a 2-D grid of processing elements, each with floating-point units and zero-skipping logic.Source: §3 · p. 2, Figure 2 · p. 3
  • Each processing element has multiple output buffers, trading more buffers for fewer arithmetic units; scheduled updates skip operations on zero values.Source: §3 · p. 3
  • Matrix multiply supports convolutions and fully connected layers, while pointwise operations support arithmetic and ternary control expressions used for ReLU and inference batch normalization.Source: §3 · p. 3
  • Training mode uses 512 single-precision units and 32-bit datapaths; inference mode uses 256 half-precision units, 256 half-precision adders, and 16-bit datapaths for ∼2× performance density.Training can sustain higher effective performance by leveraging dynamic sparsity; inference also uses zero-skipping and can accelerate pruned networks.Source: §3 · p. 3

4. RESULTS

The results report accuracy gains from low-precision networks on Cifar10 and Imagenet, then evaluate DLAC’s ability to exploit sparsity for accelerator performance.

4.1. Accuracy of Low-precision Networks on Cifar10

  • On Cifar10, the low-precision baseline loses ∼3% accuracy, while regularization, gradient updates, ReLU thresholding, and a full-precision first layer improve accuracy by ∼1.6%.The first convolution layer stays full-precision while the remaining layers use the low-precision variant.Source: §4.1 · p. 3

4.2. Accuracy of Low-precision Networks on Imagenet

  • ∼4.8% higher accuracy than previous low-precision work, with ResNet-34 variants also slightly exceeding full-precision ResNet-18.The figure compares aggressive learning-rate reduction and full-precision pre-initialization.Source: §4.2 · p. 4, Figure 4
  • Larger 2-bit ResNet variants provide higher accuracy than corresponding smaller full-precision networks while requiring less compute and model size.The paper reports this depth-scaling pattern for ResNet-34/50/152 against full-precision ResNet-18/34/50, respectively.Source: §4.2 · p. 4, Table 1 · p. 4

4.3. Performance of DLAC on Low-precision Networks

  • DLAC sustains up to 5K FLOP/cycle and averages 2.78K FLOP/cycle on ResNet-34; at 500MHz, these correspond to 2.5 and 1.34 Teraflops/second.Its compute density is 0.6 Teraflop/s/mm2 overall and can exceed 1 Teraflop/s/mm2 for deeper layers.Source: §4.3 · p. 4
  • Skipping zero-valued operations boosts DLAC performance by 1.8–5×, with greater speedups in deeper, sparser ResNet-34 layers.The figure reports layer-by-layer performance and attributes the increase with depth to greater sparsity.Source: Figure 5 · p. 4
  • One DLAC instance provides similar to slightly better performance than one DaDianNao node while being >4× smaller.The paper also says DLAC can provide higher performance as network sparsity increases.Source: §4.3 · p. 4

5. CONCLUSION

The conclusion reports high ImageNet accuracy for 2-bit networks and describes DLAC as an accelerator that exploits their sparsity. It also states that larger low-precision networks can potentially improve accuracy while using less compute than full-precision networks.

5. CONCLUSION

  • 76.6% Top-1/93% Top-5 accuracy on Imagenet is reported for the low-precision networks.The conclusion also says a larger low-precision network can possibly achieve higher performance with less compute than a regular full-precision network.Source: §5 · p. 4
  • DLAC sustains up to 1 Teraflop/mm2 equivalent by skipping operations on zero values.The conclusion describes this as a significant improvement over previous accelerator proposals.Source: §5 · p. 4
  • The paper’s conclusion links low-precision networks and greater dynamic sparsity to reduced compute requirements.It presents DLAC as the means to leverage the networks’ efficiency through zero-skipping.Source: §5 · p. 4
Loading 1610.00324v1…