PaperlayerView exact revision on arXiv

Source-linked AI summary

WRPN: Wide Reduced-Precision Networks

Asit Mishra, Eriko Nurvitadhi, Jeffrey J Cook, Debbie Marr

arXiv:1709.01134v1cs.CVcs.LGcs.NE

TL;DR

Reduced-precision activations can lower memory and computation costs but have typically harmed accuracy. WRPN jointly reduces weight and activation precision while widening layers, matching or surpassing full-precision accuracy and improving efficiency with suitable hardware support.

  • Problem

    Prior low-precision networks often sacrifice full-precision accuracy, while reducing activation precision is especially important because activations occupy substantial memory during training and inference.

  • Method

    WRPN reduces the precision of weights and activations, increases each layer’s filter maps, and uses clipping followed by rounding for hardware-friendly quantization.

  • Results

    Across AlexNet, ResNet-34, and batch-normalized Inception on ILSVRC-12, 2-bit weights with 4-bit activations match baseline accuracy, while binary WRPN reaches 69.85% top-1 on 2x-wide ResNet-34 and 48.04% on 1.3x-wide AlexNet.

  • Takeaways & Limitations

    WRPN can provide better accuracy than previously reported reduced-precision networks while reducing computational cost, buffering requirements, and potentially memory movement with appropriate hardware support.

Abstract

from arXiv · show

For computer vision applications, prior works have shown the efficacy of reducing numeric precision of model parameters (network weights) in deep neural networks. Activation maps, however, occupy a large memory footprint during both the training and inference step when using mini-batches of inputs. One way to reduce this large memory footprint is to reduce the precision of activations. However, past works have shown that reducing the precision of activations hurts model accuracy. We study schemes to train networks from scratch using reduced-precision activations without hurting accuracy. We reduce the precision of activation maps (along with model parameters) and increase the number of filter maps in a layer, and find that this scheme matches or surpasses the accuracy of the baseline full-precision network. As a result, one can significantly improve the execution efficiency (e.g. reduce dynamic memory footprint, memory bandwidth and computational energy) and speed up the training and inference process with appropriate hardware support. We call our scheme WRPN - wide reduced-precision networks. We report results and show that WRPN scheme is better than previously reported accuracies on ILSVRC-12 dataset while being computationally less expensive compared to previously reported reduced-precision networks.

1 Introduction

Low-precision DNNs reduce computation, data movement, and storage, but often sacrifice accuracy, especially when activation precision is reduced. WRPN addresses this by reducing weight and activation precision while widening layers, achieving competitive or improved accuracy with hardware-dependent efficiency gains.

  • Motivation: Low-precision algorithms reduce computation, data movement, and storage, but most prior networks sacrifice accuracy relative to full-precision baselines.Earlier work focused mainly on low-precision weights, which primarily benefits inference at small batch sizes.
  • WRPN approach: WRPN reduces both activation and weight precision while increasing each layer’s filter-map count to preserve or improve baseline accuracy.For FP32 AlexNet converted to 4-bit precision with twice as many filters, operations increase 4x while each operation is 8x more efficient.
  • Results: 4-bit precision is sufficient for training deep and wide models, while 4-bit activations with 2-bit weights achieve accuracy at par with full precision.The reported evaluations cover AlexNet, batch-normalized Inception, and ResNet-34 on ILSVRC-12.
  • Results: 69.85% top-1 accuracy is reported for 2x-wide ResNet-34, and 48.04% top-1 accuracy for 1.3x-wide AlexNet using binary networks.The authors describe these binary-network accuracies, and their 4-bit results, as highest to date to their knowledge.
  • Hardware efficiency: FPGA and ASIC implementations provide 6.5x to 100x efficiency gains over FP32 operations, whereas the evaluated GPU cannot exploit very low-precision operations.The hardware evaluation covers 4-bit through 1-bit operations on Titan X GPU, Arria-10 FPGA, and ASIC.

2 Motivation for reduced-precision activation maps

Activation maps can dominate memory use as mini-batches grow because filters are reused across inputs. Reducing activation and weight precision therefore targets memory footprint, bandwidth, storage, and hardware-support requirements.

  • Figure 1: Figure 1 presents activation and weight memory footprints during training and inference for mini-batch sizes 1 and 32.ACTs denote activations and W denotes weights.
  • Memory footprint: Activation maps occupy a larger memory footprint than weights when mini-batches contain more inputs.Figure 1 compares activation and weight memory for four networks during training and inference as batch size changes.
  • Figure 2: Figure 2 depicts feed-forward convolutional-network memory requirements using weights, activations, and gradient maps.Orange boxes represent weights, blue boxes activations, and green boxes gradient maps.
  • Memory footprint: As batch size increases, activation maps occupy a significantly larger memory fraction than filter weights because filters are reused across input batches.During training, memory also includes weights, activations, and gradient maps; during inference, feature-map buffers are reused across layers.
  • Implications: Reducing activation and weight precision reduces memory footprint, bandwidth, and storage while simplifying hardware requirements.These reductions motivate using lower precision to improve training and inference efficiency.

3 WRPN scheme and studies on AlexNet

WRPN reduces activation and weight precision, then widens filter maps to recover accuracy while controlling compute cost. AlexNet studies show that widening can restore or exceed full-precision accuracy and keep reduced-precision computation below baseline cost.

  • Precision sensitivity: Reducing activation precision harms accuracy more than reducing filter-weight precision in general low-precision AlexNet studies.The study trains networks end-to-end from scratch; binary weights and activations reach 44.2% versus 57.2% for the 32-bit baseline.
  • WRPN scheme: WRPN jointly lowers activation and weight precision while increasing filter-map width to regain model accuracy.The method trades more raw operations for fewer bits per operation and retains the baseline network depth and other design parameters.
  • AlexNet accuracy: With 2x-wide AlexNet, 4-bit weights and 2-bit activations match full-precision accuracy, while 4-bit weights and activations surpass baseline accuracy by 1.44%.Binary weights and activations also improve on XNOR-NET by 4%.
  • Compute cost: With 2x the filters and 4-bit operands, reduced-precision AlexNet costs 49% of the full-precision baseline despite 3.9x more raw compute operations.Compute cost is defined as FMA count multiplied by the sum of activation- and weight-operand widths.
  • Width–precision trade-off: Widening and precision reduction involve a trade-off: 1.3x-wide filters can use 8-bit weights with 4-bit activations at baseline accuracy, whereas 2x widening requires at least 8-bit precision to stay within baseline compute cost.With 1.1x widening, at least 8-bit weights and 16-bit activations are required to match baseline accuracy.

4 Studies on deeper networks

WRPN maintains or improves accuracy in deeper networks by widening layers while reducing activation and weight precision. ResNet-34 and batch-normalized Inception show near-baseline or state-of-the-art results, with lower precision reducing compute cost.

  • ResNet-34: ResNet-34 with doubled filters and 4-bit weights and activations exceeds baseline accuracy by 0.9%.
  • ResNet-34: 4-bit activations with 2-bit ternary weights in ResNet-34 matches baseline accuracy, while 2-bit weights and activations degrade accuracy by only 0.2%.
  • ResNet-34: 69.85% top-1 accuracy is achieved by a binary ResNet-34 with 2x-wide filters at 15% of the full-precision baseline cost.
  • ResNet-34: ResNet-34 binary and ternary top-1 accuracies are reported as state of the art, including comparisons with unpublished technical reports.
  • Batch-normalized Inception: Batch-normalized Inception with doubled filter banks, 4-bit activations, and 2-bit weights loses only 0.02% accuracy versus the single-precision baseline.
  • Batch-normalized Inception: A wider batch-normalized Inception with binary weights and activations remains within 6.6% of the full-precision baseline.

5 Hardware friendly quantization scheme

The paper uses simple clipping-and-rounding quantization with straight-through estimation and evaluates its hardware implications. Low precision can improve efficiency substantially, but realized gains depend on the hardware target.

  • Quantization scheme: The straight-through estimator enables training despite quantization's finite-valued outputs having zero mathematical gradients.
  • Quantization scheme: WRPN quantizes activations and weights while widening layers, using signed weights and unsigned activations for integer convolution operations followed by scaling.
  • Quantization scheme: During training and inference, convolutions multiply k-bit signed weights with k-bit unsigned activations, while backpropagation uses 32-bit gradients with k-bit operands.
  • Quantization scheme: The proposed hardware-friendly quantization clips tensors to fixed ranges and rounds them, avoiding division, tensor-wide maximum computation, and learned quantization parameters.
  • Hardware evaluation: GPU performance improves by at most approximately 4x over FP32 because Titan X provides first-class support for INT8 but not INT4, ternary, or binary precision.
  • Hardware evaluation: ASIC experiments show 2 to 3 orders of magnitude efficiency improvements from lower precision, while FPGA and ASIC provide at least 6.5x efficiency per INT4-or-lower operation over FP32.

6 Related work

Prior reduced-precision methods mainly quantize weights or incur substantial accuracy losses when also quantizing activations. WRPN instead targets training from scratch with simple quantization and wide networks while maintaining competitive accuracy and lower compute cost.

  • Weight and activation quantization: Weight-only quantization methods commonly degrade accuracy, while fine-tuning approaches such as INQ are not applicable to training networks from scratch.For AlexNet on ImageNet, TWN loses 5% top-1 accuracy; INQ reports promising results with 5-bit precision.
  • Weight and activation quantization: Quantizing both weights and activations to 1-bit reduces AlexNet top-1 accuracy by 12% for XNOR-NET and 8% for DoReFa.XNOR-NET also requires re-ordering layers, adding scheme-specific constraints.
  • WRPN: WRPN studies simple quantization in deep and wide networks, achieving baseline-level accuracy at 4-bit activations and 2-bit weights while reporting lower compute cost.The paper also reports state-of-the-art accuracy for wide binarized AlexNet and ResNet.

7 Conclusions

WRPN reduces weight and activation precision while widening layers to preserve accuracy. The scheme targets memory and compute efficiency, supports hardware implementations, and is evaluated across GPU, FPGA, and ASIC platforms.

  • WRPN scheme: WRPN reduces both weight and activation precision while widening layers to compensate for the information capacity lost through quantization.Across AlexNet, ResNet-34, and batch-normalized Inception, 2-bit weights and 4-bit activations match baseline accuracy.
  • Efficiency motivation: Full-precision activations can dominate memory footprint during mini-batch training and cloud-based inference, motivating activation quantization.Reducing both activations and weights is reported to reduce compute complexity to 40% of baseline for 2-bit weights and 4-bit activations.
  • Hardware evaluation: The hardware-friendly WRPN scheme is evaluated on Titan X GPU, Arria-10 FPGA, and ASIC implementations.The evaluation spans low-precision operations from 4 bits to 1 bit.
Loading 1709.01134v1…