Source-linked AI summary
Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
Asit Mishra, Debbie Marr
TL;DR
Large, accurate DNNs are costly to deploy on resource-constrained inference systems, while low-precision models reduce compute and storage but often lose accuracy. Apprentice combines quantization with knowledge distillation through three training schemes and reports state-of-the-art ternary- and 4-bit-precision results for ResNet models.
Problem
Deploying high-performing DNNs is challenging on systems constrained by memory, compute, and power, while low-precision approaches commonly sacrifice accuracy.
Method
Apprentice applies knowledge distillation between similar-topology full-precision teacher and low-precision apprentice networks through three training schemes.
Results
Apprentice produces state-of-the-art ternary-precision and 4-bit-precision ResNet models, with the third scheme surpassing the first scheme’s accuracy.
Takeaways & Limitations
The resulting low-precision models are intended to simplify inference deployment on resource-constrained and low-latency cloud systems.
Takeaways & Limitations
The paper leaves systematic study of other hyperparameters and loss functions for future work after finding no accuracy improvement from tested alternatives.
Abstract
from arXiv · showhide
Deep learning networks have achieved state-of-the-art accuracies on computer vision workloads like image classification and object detection. The performant systems, however, typically involve big models with numerous parameters. Once trained, a challenging aspect for such top performing models is deployment on resource constrained inference systems - the models (often deep networks or wide networks or both) are compute and memory intensive. Low-precision numerics and model compression using knowledge distillation are popular techniques to lower both the compute requirements and memory footprint of these deployed models. In this paper, we study the combination of these two techniques and show that the performance of low-precision networks can be significantly improved by using knowledge distillation techniques. Our approach, Apprentice, achieves state-of-the-art accuracies using ternary precision and 4-bit precision for variants of ResNet architecture on ImageNet dataset. We present three schemes using which one can apply knowledge distillation techniques to various stages of the train-and-deploy pipeline.
1 INTRODUCTION
High-performing DNNs are difficult to deploy under memory, compute, and power constraints, while low-precision and distillation methods address these costs but can reduce accuracy. Apprentice combines quantization with knowledge distillation through three training schemes for accurate low-precision networks.
- Motivation: Resource-constrained inference systems make deploying compute- and memory-intensive trained networks challenging, especially for real-time edge-device predictions.Edge devices also impose severe power constraints, making ensemble-based accuracy improvements prohibitive.
- Motivation: Low-precision numerics reduce compute, data movement, and storage requirements, but existing approaches commonly sacrifice accuracy relative to full-precision baselines.Knowledge distillation transfers information from a complex network to a smaller network, although prior compression work targeted full-precision models.
- Proposal: Apprentice combines quantization with model compression while keeping teacher and student topologies similar, differing primarily in neuron precision.The teacher operates at full precision and the student uses low-precision neurons.
- Contribution: Three knowledge-distillation schemes produce state-of-the-art ternary-precision and 4-bit-precision models.The schemes apply distillation at different stages of training and deployment preparation.
- Results: Jointly training full- and low-precision networks from scratch establishes new ternary and 4-bit accuracy results for ResNet-18, ResNet-34, and ResNet-50 on ImageNet.The full-precision model also slightly improves and becomes the baseline for the other schemes.
- Results: A trained full-precision network guiding a low-precision network from scratch yields faster convergence, while fine-tuning a precision-lowered apprentice marginally improves accuracy beyond the first scheme.The third scheme sets new state-of-the-art accuracies for ResNet models at ternary and 4-bit precision.
2 MOTIVATION FOR LOW-PRECISION MODEL PARAMETERS
Low-precision parameters target inference memory, compute, and storage constraints by reducing weight-tensor precision. The central trade-off is that low-precision models can have degraded accuracy.
- Inference constraints: Inference memory includes model parameters and activation maps, with dynamic input and output feature-map allocations reused across DNN layers.The total allocation depends on the maximum required feature-map memory across layers together with model parameters.
- Inference constraints: Lowering weight-tensor precision reduces deployment memory requirements, as illustrated for multiple networks and mini-batch sizes in Figure 1.The figure compares activation and weight footprints for mini-batches of 1 and 8.
- Hardware benefits: Low-precision compute can simplify hardware implementation by replacing floating-point multiplication with simpler logic for binary weights and activations.The cited example uses XNOR and popcount circuitry.
- Trade-off: The principal drawback of low-precision models is degraded accuracy relative to full-precision baselines.The paper uses accuracies from prior methods as starting points and comparison baselines.
3 RELATED WORK
Related work addresses efficient DNN inference through low-precision networks, knowledge distillation, and other compression methods. These approaches reduce representation or model size, but quantization can incur substantial accuracy losses.
- Low-precision networks: Low-precision research reduces weight precision through methods including BinaryConnect, ternary-weight networks, and trained ternary quantization.The related work distinguishes approaches targeting weights, activations, or the training pipeline.
- Low-precision networks: Quantizing both weights and activations to 1-bit reduced AlexNet ImageNet Top-1 accuracy by 12% for XNOR-NET and 8% for DoReFa.These figures are reported as accuracy degradation relative to the corresponding baseline.
- Knowledge distillation: Knowledge distillation uses a teacher-student strategy in which a trained large network teaches a shallower student on the same task.Earlier work compressed ensemble information or transferred knowledge to smaller networks.
- Knowledge distillation: Hinton et al. introduced temperature-scaled logits so incorrect-class activations are boosted before Softmax, increasing information passed during back-propagation.FitNets further use intermediate hidden-layer outputs as targets.
- Other compression methods: Pruning, hashing, and weight sharing compress models by removing neurons or aliasing multiple parameters into fewer representations.Pruning produces sparse structures, while hashing and sharing reduce distinct stored weights.
4 KNOWLEDGE DISTILLATION
The paper formulates knowledge distillation as a weighted loss combining teacher, apprentice, and teacher-to-apprentice objectives. A high-precision teacher supplies predictions or logits while the low-precision apprentice learns from the same input.
- Distillation setup: For input x, the teacher and apprentice produce class predictions by applying Softmax to their unnormalized logits.The teacher prediction is denoted pT and the apprentice prediction pA in the formulation.
- Loss formulation: The training loss combines teacher supervision, apprentice supervision, and a teacher-to-apprentice distillation term: L(x; WT, WA) = αH(y, pT) + βH(y, pA) + γH(zT, pA).WT and WA are teacher and apprentice parameters; y is ground truth and α, β, γ weight the loss terms.
- Distillation setup: Knowledge distillation transfers knowledge from a teacher DNN to an apprentice network using the same input image.The teacher is high precision and the apprentice is low precision in the depicted setup.
- Loss formulation: The first loss term trains the teacher, the second trains the apprentice, and the third makes the apprentice mimic knowledge in the teacher network.The paper uses cross-entropy with α = 1, β = 0.5, and γ = 0.5, transferring teacher logits.
5 OUR APPROACH - APPRENTICE NETWORK
Apprentice combines low-precision quantization with knowledge distillation through three training schemes for ResNet students on ImageNet. Across ternary and 4-bit models, distillation improves accuracy, often narrowing the gap to full-precision baselines and surpassing prior low-precision results.
- Approach: Apprentice combines low-precision numerics and teacher-student model compression to improve accuracy while retaining low-precision efficiency.The approach investigates three schemes for producing low-precision student networks.
- Scheme-A: Joint training: Scheme-A jointly trains a full-precision teacher and low-precision student from scratch, with teacher depth chosen equal to or greater than student depth.The scheme uses ResNet topology for both networks and establishes the baseline for later schemes.
- Scheme-B: Distilling knowledge: Scheme-B achieves similar accuracy to the teacher while converging in about 10%-20% fewer epochs than Scheme-A.The reported convergence is around the 80th-85th epoch for Scheme-B versus about 105 epochs for Scheme-A.
6 CONCLUSIONS
The paper presents three knowledge-distillation schemes for improving low-precision network accuracy, targeting smaller models for resource-constrained and low-latency deployment.
- Three knowledge-distillation schemes improve the accuracy of low-precision networks.
7 APPENDIX: ANALYSIS WITH RESNET ON CIFAR-10 DATASET
On CIFAR-10, the appendix evaluates Apprentice across ResNet depths and shows that distillation narrows the accuracy gap between low-precision and full-precision models.
- Experimental setup: CIFAR-10 experiments use ResNet topologies with varying depths, including 6n+2 weight layers.The networks process 32×32 images through convolutional stages with 16, 32, and 64 filters, followed by global average pooling and softmax.
- Precision and depth: 6.19% full-precision Top-1 error and 6.24% ternary Top-1 error are reported for ResNet-110.
- Precision and depth: 0.8% is the Top-1 error gap between full-precision and ternary ResNet-20, at 7.9% versus 8.7%.
- Distillation results: Scheme-A pairs low-precision students with full-precision teachers, with experiments spanning ResNet-20, 32, 44, 56, 110, and 182.
- Distillation results: Apprentice reduces the gap between full-precision and low-precision networks to 0.4%-0.8% for 2-bit weights and 8-bit activations.