PaperlayerView exact revision on arXiv

Source-linked AI summary

Model compression via distillation and quantization

Antonio Polino, Razvan Pascanu, Dan Alistarh

arXiv:1802.05668v1cs.NEcs.LG

TL;DR

The paper addresses efficient execution of accurate deep networks in resource-constrained environments. It jointly uses teacher-student distillation and weight quantization through quantized distillation and differentiable quantization, achieving similar accuracy with up to order-of-magnitude compression and depth-related speedups.

  • Problem

    Deep neural networks are often too compute-, space-, and power-intensive for resource-constrained devices, motivating compression without sacrificing accuracy.

  • Method

    The paper distills larger teachers into shallower students while quantizing student weights, using training-time distillation loss and learned quantization-point locations.

  • Results

    Up to an order of magnitude compression and inference speedup linear in depth were achieved while shallow quantized students reached similar accuracy to full-precision deeper teachers.

  • Takeaways & Limitations

    Distillation can improve quantized-model compression by leveraging large accurate teachers, while shallower students also provide automatic inference-speed improvements.

  • Takeaways & Limitations

    The experiments used manual architecture search for student depth and bit width, which the authors describe as time-consuming and error-prone.

Abstract

from arXiv · show

Deep neural networks (DNNs) continue to make significant advances, solving tasks from image classification to translation or reinforcement learning. One aspect of the field receiving considerable attention is efficiently executing deep models in resource-constrained environments, such as mobile or embedded devices. This paper focuses on this problem, and proposes two new compression methods, which jointly leverage weight quantization and distillation of larger teacher networks into smaller student networks. The first method we propose is called quantized distillation and leverages distillation during the training process, by incorporating distillation loss, expressed with respect to the teacher, into the training of a student network whose weights are quantized to a limited set of levels. The second method, differentiable quantization, optimizes the location of quantization points through stochastic gradient descent, to better fit the behavior of the teacher model. We validate both methods through experiments on convolutional and recurrent architectures. We show that quantized shallow students can reach similar accuracy levels to full-precision teacher models, while providing order of magnitude compression, and inference speedup that is linear in the depth reduction. In sum, our results enable DNNs for resource-constrained environments to leverage architecture and accuracy advances developed on more powerful devices.

1 INTRODUCTION

The paper investigates compressing deep networks by combining quantization with distillation from larger teacher models into smaller students. It proposes two methods and reports similar accuracy with substantial compression and depth-related inference speedups.

  • Motivation: Quantized models can preserve good task solutions while restricting weights to integer levels.
  • Motivation: Existing compression methods mainly optimize weight representations without substantially changing model structure, whereas distillation transfers behavior from larger models.
  • Contribution: The proposed approach compounds depth compression through shallower students with width compression through quantized weights and fewer weights per layer.
  • Contribution: Quantized distillation incorporates teacher-based distillation loss while training a student whose weights use limited levels; differentiable quantization learns quantization-point locations with stochastic gradient descent.
  • Results: Up to an order of magnitude compression and inference speedup linear in depth were obtained while quantized shallow students reached similar accuracy to full-precision deeper teachers across image-classification and translation datasets.
  • Results: A 4-bit quantized ResNet18 matched the ResNet34 teacher's accuracy and exceeded the most accurate model in prior cited work by more than 15% top-1 and 7% top-5 accuracy.

2 PRELIMINARIES

The preliminaries define scaled uniform and non-uniform quantization, including bucketing to reduce magnitude imbalance. They also show that stochastic quantization introduces an approximately Gaussian error into neural-network layer outputs.

  • Quantization process: Quantization scales arbitrary vectors into [0, 1], applies a quantization function there, and maps the result back with the inverse scaling function.
  • Scaling: Linear scaling uses the vector minimum and range to normalize values into [0, 1].
  • Bucketing: Bucketing applies scaling separately to consecutive fixed-size value groups to reduce precision loss from magnitude imbalance.
  • Uniform quantization: Uniform quantization uses s + 1 equally spaced levels, with deterministic nearest-point assignment or stochastic unbiased rounding.
  • Non-uniform quantization: Non-uniform quantization assigns each value to the closest point among a supplied set of quantization points.
  • Stochastic quantization: Stochastic uniform quantization is an unbiased estimator of its input, and its scalar-product error converges toward a standard normal distribution as dimension increases.
  • Stochastic quantization: Quantizing weights is equivalent to adding a zero-mean, asymptotically normally distributed error before each layer's activation, with variance depending on s.

3 QUANTIZED DISTILLATION

Quantized distillation trains a shallower, quantized student to leverage a converged teacher's outputs. It combines distillation loss with projected optimization while accumulating projection errors across updates.

  • Setup: The student is compressed by being shallower than the teacher and representing its weights at limited bit width.
  • Knowledge transfer: The student uses distillation loss, a weighted combination of cross-entropy with teacher soft targets and the correct labels.
  • Optimization: Projected gradient descent takes full-precision gradient steps, projects parameters onto valid quantized values, and accumulates projection error for later updates.
  • Optimization: An equivalent view computes gradients on the quantized model with respect to distillation loss while updating the full-precision model.

4 DIFFERENTIABLE QUANTIZATION

Differentiable quantization learns non-uniform quantization points for quantized neural networks, while related procedures train quantized students using gradient-based updates and distillation loss. The method requires initialization and allocation heuristics, and can converge in fewer iterations despite added computation per iteration.

  • 4.1 GENERAL DESCRIPTION: Differentiable quantization optimizes non-uniform quantization point locations with stochastic gradient descent to improve quantized-network accuracy.The method differentiates the model with respect to quantization points rather than relying on gradients through discrete weight assignments.
  • 4.1 GENERAL DESCRIPTION: Quantized distillation updates full-precision weights using gradients computed from the quantized model and distillation loss, then quantizes the returned weights.The algorithm repeatedly quantizes weights, computes distillation loss, updates the original weights, and finally returns quantized weights.
  • 4.1 GENERAL DESCRIPTION: The discrete assignment of weights to quantization points produces zero gradients with respect to weights, motivating optimization through the quantization points instead.The loss remains continuous with respect to the quantization points, whose gradients can be used for SGD.
  • 4.2 DISCUSSION AND ADDITIONAL HEURISTICS: Quantization-point optimization may be slower per iteration than original-network training, although the authors report requiring an order of magnitude fewer iterations to reach a good solution.Each iteration adds quantization and an additional backward pass for gradients with respect to the points.
  • 4.2 DISCUSSION AND ADDITIONAL HEURISTICS: Initialization by weight quantiles ensures every quantization point is used, while gradient norms guide allocating more points to layers sensitive to quantization.The allocation process can use different numbers of points across layers, with Huffman encoding mitigating some resulting bit-complexity effects.
  • 4.2 DISCUSSION AND ADDITIONAL HEURISTICS: Using distillation loss trains the quantized model to mimic the unquantized model, which the paper says should be easier to learn and provide better results.This replaces optimization against the original task loss with matching the unquantized model's outputs.

5 COMPRESSION

The compression analysis combines low-bit quantization with bucket scaling and optional Huffman encoding to estimate storage savings. The reported gains are substantial but are presented as ballpark figures because implementation costs may reduce practical savings.

  • 5 COMPRESSION: Quantized storage uses bN bits for weights plus scaling-factor overhead for each bucket, whereas full precision uses fN bits.Here f denotes the 32-bit full-precision weight size and N denotes the quantized vector size.
  • 5 COMPRESSION: 14.2× space savings result from 2 bits per component with 256-size buckets, compared with 7.52× using 4 bits.These differentiable-quantization figures are measured relative to full precision.
  • 5 COMPRESSION: 15.05× compression results from 2 bits at 512-size buckets, compared with 7.75× using 4 bits.The passage reports these values as savings relative to full precision.
  • 5 COMPRESSION: Huffman encoding represents quantized values by indices and uses their frequencies to compute an optimal variable-length code.This produces fractional mean bit lengths in some reported size-gain tables.
  • 5 COMPRESSION: The compression figures are ballpark estimates because additional implementation costs may make the savings difficult to translate into practice.The authors explicitly qualify the reported storage gains rather than presenting them as guaranteed deployment savings.

6 EXPERIMENTAL RESULTS

Experiments across image classification and translation evaluate quantized distillation, differentiable quantization, and related baselines on smaller and larger students. Quantized shallow students generally preserve teacher accuracy with substantial compression, while performance depends on bit width, architecture, and training method.

  • Small datasets: Quantized distillation generally achieved the best accuracy across tested bit widths and architectures, while differentiable quantization converged faster.The comparison includes PM quantization with and without bucketing, quantized distillation, and differentiable quantization.
  • Small datasets: At 4-bit precision, CIFAR-100 distilled and differentiably quantized students preserved accuracy within less than 1% of the teacher.At 2-bit precision, accuracy loss was catastrophic, probably because of reduced model capacity; differentiable quantization recovered the most accuracy on this task.
  • Larger datasets: On WMT13, 4-bit quantized distillation produced a higher BLEU score than the teacher with similar perplexity, whereas PM quantization performed poorly even with bucketing.The WMT13 dataset contains 1.7M training sentences and 190K test sentences.
  • Larger datasets: 73.31% validation accuracy was reached by a 4-bit quantized distilled 2xResNet18, versus 69.75% for unquantized ResNet18 and similar accuracy to the ResNet34 teacher.The compressed model was more than 2× smaller than ResNet18, 4× smaller than ResNet34, and about 1.5× faster in inference.
  • Larger datasets: A 4-bit quantized 2xResNet34 student matched almost the same accuracy as its full-precision teacher while being 50% shallower and 2.5× smaller.This experiment transferred from a full-precision ResNet50 teacher.
  • Additional experiments: Distillation loss improved quantized training on CIFAR-10 and OpenNMT relative to normal loss, including 82.40% versus 67.22% at 2 bits on CIFAR-10.On OpenNMT, the 4-bit distilled student reached 25.43 perplexity and 15.73 BLEU, compared with 32.67 perplexity and 15.03 BLEU under normal loss.
  • Additional experiments: Shallower students reduced inference cost approximately linearly with depth: 67.4 seconds for the teacher versus 43.7 seconds for the student in a CIFAR-10 wide-ResNet experiment.The 1.75× depth reduction yielded roughly a 1.5× speedup.

7 DISCUSSION

The paper finds that combining distillation with quantization preserves accuracy while compressing models, and identifies manual student-architecture search as a limitation.

  • 7 DISCUSSION: Quantized distillation and differentiable quantization fit quantized students to larger teacher models through training loss or optimized quantization points.Quantized distillation acts during student training, whereas differentiable quantization optimizes quantization to fit the teacher.
  • 7 DISCUSSION: Up to an order of magnitude in model size compression is achieved on small image-classification and NMT tasks while preserving accuracy.The experiments also report inference-speed improvements from shallower distilled models.
  • 7 DISCUSSION: Naive uniform quantization with bucketing performs well across a wide range of scenarios and may serve as a simple baseline.The analysis suggests bucketing parametrizes Gaussian-like noise induced by quantization.
  • 7 DISCUSSION: Manual search over student depth and bit width is time-consuming and error-prone.Future work proposes reinforcement learning or evolution strategies for architecture discovery under space and latency constraints.

A.1 CIFAR10

The CIFAR10 experiments evaluate convolutional student architectures, quantization settings, model sizes, and a deeper student configuration.

  • A.1 CIFAR10: The CIFAR10 models combine convolutional, dropout, max-pooling, and fully connected layers, with smaller models using 5x5 convolutions.The teacher uses 3x3 convolutional layers, while smaller models use 5x5 layers.
  • A.1 CIFAR10: Distillation training omits dropout layers and computes distillation loss with temperature T = 5.This setting is used for the CIFAR10 distillation experiments.
  • A.1 CIFAR10: The teacher model reaches 89.7% test accuracy with 5.3 million parameters and a 21.3 MB size.The reported tables cover full-precision accuracy, quantized accuracy, and Huffman-encoded model size.
  • A.1 CIFAR10: The experiments also evaluate a deeper student architecture using the same teacher and report its results separately.The deeper student contains three fully connected layers after the convolutional blocks.

A.1.1 CIFAR10 - WIDERESNET ARCHITECTURE

The WideResNet experiments increase filter counts while reducing depth and compare quantized methods on CIFAR10 and CIFAR100.

  • A.1.1 CIFAR10 - WIDERESNET ARCHITECTURE: The WideResNet setup increases the number of filters while reducing model depth.The experiments use the same WideResNet implementation for CIFAR10 and CIFAR100.
  • A.1.1 CIFAR10 - WIDERESNET ARCHITECTURE: The experiments report quantized-model accuracy and Huffman-encoded model sizes for both CIFAR10 and CIFAR100.CIFAR100 experiments focus on one student model and use distillation temperature T = 5.
  • A.1.1 CIFAR10 - WIDERESNET ARCHITECTURE: The CIFAR100 WideResNet models are trained for 200 epochs with an initial learning rate of 0.1.The wide factor controls the number of filters in each layer.

A.3 OPENTNMT INTEGRATION TEST DATASET

The OpenNMT integration experiments modify the codebase to add distillation loss and the proposed quantization methods, using standard training procedures.

  • A.3 OPENTNMT INTEGRATION TEST DATASET: The OpenNMT-py codebase is modified to implement distillation loss and the proposed quantization methods.Models are generally trained for 15 epochs, with learning-rate halving triggered when test-set perplexity stops decreasing.
  • A.3 OPENTNMT INTEGRATION TEST DATASET: The smaller WMT13 model is trained for 5 epochs because it overfits when trained for 15 epochs.The other WMT13 models are trained for 15 epochs.

A.4.1 DISTILLATION VERSUS STANDARD LOSS FOR QUANTIZATION

The section compares quantized models trained with standard loss against models trained with distillation loss under matched architectures and bit budgets. Distillation substantially improves quantized-model accuracy, while differentiable-quantization heuristics matter especially at very low bit widths.

  • Matched architectures and bit budgets isolate the effect of adding distillation loss to quantized-model training.The quantized distilled model equally weights soft cross entropy and normal cross entropy.
  • Distillation loss can significantly improve quantized-model accuracy on the openNMT integration test dataset.The comparison uses models with the Smaller model 1 structure.
  • The CIFAR10 and openNMT integration experiments directly compare distillation loss with normal loss when quantizing.The reported comparisons are organized in Tables 27 and 28.
  • The section concludes that quantization works better when combined with distillation and recommends using the combination when quantizing neural networks.
  • At 4 bits, differentiable quantization is robust across heuristics, whereas at 2 bits, gradient-norm-based bit redistribution is essential.Quantile initialization provides a small improvement at 2 bits, while distillation loss is not crucial in that setting.
Loading 1802.05668v1…