PaperlayerView exact revision on arXiv

Source-linked AI summary

Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks

Yoonho Boo, Sungho Shin, Jungwook Choi, Wonyong Sung

arXiv:2009.14502v1cs.LGstat.ML

TL;DR

Low-precision QDNNs can lose accuracy, while conventional KD depends on memory-intensive teacher models that may be unavailable for deployment and adaptation. SPEQ instead shares parameters between student and teacher, stochastically varies activation precision to generate soft labels, and uses cosine-similarity distillation. It outperforms existing quantized training methods across image classification, question answering, and transfer learning without a cumbersome teacher model.

  • Problem

    Low-precision QDNNs suffer accuracy loss, while conventional KD requires large teacher models that may be unavailable for deployed on-device training.

  • Method

    SPEQ shares model parameters between student and teacher, stochastically varies activation precision to produce soft labels, and distills them with cosine similarity.

  • Results

    SPEQ outperforms existing quantized training methods across image classification, question answering, and transfer learning without a cumbersome teacher model.

  • Takeaways & Limitations

    SPEQ provides a low-precision QDNN training approach that remains usable when a larger teacher model is unavailable.

Abstract

from arXiv · show

The quantization of deep neural networks (QDNNs) has been actively studied for deployment in edge devices. Recent studies employ the knowledge distillation (KD) method to improve the performance of quantized networks. In this study, we propose stochastic precision ensemble training for QDNNs (SPEQ). SPEQ is a knowledge distillation training scheme; however, the teacher is formed by sharing the model parameters of the student network. We obtain the soft labels of the teacher by changing the bit precision of the activation stochastically at each layer of the forward-pass computation. The student model is trained with these soft labels to reduce the activation quantization noise. The cosine similarity loss is employed, instead of the KL-divergence, for KD training. As the teacher model changes continuously by random bit-precision assignment, it exploits the effect of stochastic ensemble KD. SPEQ outperforms the existing quantization training methods in various tasks, such as image classification, question-answering, and transfer learning without the need for cumbersome teacher networks.

1 Introduction

SPEQ addresses accuracy loss in low-precision QDNNs while avoiding the memory and training overhead of conventional teacher-based knowledge distillation. It uses shared model parameters, stochastic activation precision, and cosine-similarity distillation across several tasks.

  • Activation quantization mainly adds noise to decision boundaries, so varying activation precision supplies diverse guidance for self-knowledge distillation.The authors motivate SPEQ through contrasting precision effects on weight- and activation-quantized models.
  • SPEQ forms its teacher by sharing quantized weights with the student while randomly selecting activation precision between low and high settings.The shared teacher also uses shared activation clipping levels and stochastically experiences target low-bit quantization.
  • Cosine similarity is presented as an essential loss for distilling stochastic-quantization knowledge to the low-precision student.
  • The shared-parameter design removes auxiliary teacher models and reduces the overhead associated with teacher-model search and hyperparameter tuning.SPEQ can also be combined with conventional KD to further improve target QDNN performance.
  • SPEQ enhances low-precision QDNN accuracy without requiring the large teacher models used by conventional KD.The method is evaluated on image classification, question answering, and transfer learning, with lower training overhead.

2 Related Works

Prior QDNN work reduces precision through quantized training and architectural or optimization techniques, while KD restores accuracy using teacher guidance. SPEQ differs by sharing the student model and injecting stochastic activation precision into the teacher path.

  • 2.1 Quantization of Deep Neural Networks: Quantized DNN research uses low-precision weights and activations, but quantization errors can reduce accuracy and constrain compression-oriented architectures.Prior methods include quantized SGD, stochastic rounding, trainable quantization, and architectural changes such as increased network width.
  • 2.2 Knowledge Distillation for Quantization: Conventional KD transfers representation knowledge from a larger teacher to a student, including methods designed to recover accuracy lost through quantized inference.Examples include Apprentice and quantization-aware KD, which coordinate training between full-precision and quantized models.
  • 2.2 Knowledge Distillation for Quantization: SPEQ's stochastic-precision teacher path uses the same quantized weights while varying activation precision during forward computation.
  • 2.2 Knowledge Distillation for Quantization: SPEQ forms its teacher by sharing the student model and assigning stochastic activation bit precision, unlike prior teacher-based approaches.This design aims to include target-model quantization noise in the teacher information while avoiding pretrained teachers and auxiliary training parameters.

3 Stochastic Precision Ensemble Training for QDNNs

SPEQ trains a quantized student with soft labels from a shared-parameter path whose activation precision varies stochastically. It combines this self-distillation with cosine-similarity guidance to address noisy teacher predictions and activation quantization.

  • 3.1 Stochastic Precision Self-Distillation with Model Sharing: SPEQ computes target-path and stochastic-path logits with shared parameters, using stochastic activation precision to generate soft labels containing the target model’s quantization noise.The target path uses nA-bit activations, while the stochastic path selects between target and higher precision per layer; the shared model avoids auxiliary teachers.
  • 3.2 Stochastic Ensemble Learning: The stochastic path samples each layer’s activation precision from target and high precision, with high precision fixed at 8 bits and probability controlled by u.The total number of layerwise precision combinations is 2^L, making exhaustive search impractical for deep networks.
  • 3.1 Stochastic Precision Self-Distillation with Model Sharing: SPEQ adds cross-entropy and cosine-similarity losses while back-propagating only through the target-precision path, so its extra training computation is stochastic-path forward propagation.The stochastic-path logits are used only to produce soft labels, and no auxiliary model is required.
  • 3.2 Stochastic Ensemble Learning: 88.4% test accuracy is achieved by greedy 2-bit training, exceeding always choosing 8-bit at 88.1% on the 2-bit model.The best-performing stochastic solution selects 2- and 8-bit activation almost uniformly during training.
  • 3.2 Stochastic Ensemble Learning: The stochastic precision path creates diverse soft-label distributions from the same parameters, providing an ensemble-learning effect across activation precisions.Figure 2(b) shows that soft-label distributions vary for one image as activation precision changes.
  • 3.3 Cosine Similarity Learning: Cosine-similarity loss selectively follows the stochastic teacher, whereas KL-divergence can reverse gradient direction when the teacher is less confident than the student.Cosine guidance strengthens with teacher confidence and is neglected when the teacher prediction is ambiguous.

4 Experimental Results

Across image classification and transfer learning, SPEQ improves low-precision models while avoiding the large auxiliary teachers required by conventional KD.

  • 4 Experimental Results: The experiments pretrain a floating-point DNN, retrain the QDNN at target precision, then apply SPEQ from the retrained parameters.
  • 4.1 Results on the CIFAR10 and CIFAR100 Datasets: CIFAR10 performance is best when the stochastic precision probability u lies between 0.4 and 0.6; selecting only 2- and 8-bit precisions beats mixing all precisions by 0.21%.
  • 4.1 Results on the CIFAR10 and CIFAR100 Datasets: 91.4% test accuracy for 2-bit ResNet20 on CIFAR10 improves over 90.7% retraining and exceeds QKD, while 2-bit activation quantization nearly matches full precision.
  • 4.2 Results on the ImageNet Dataset: 59.3% top-1 accuracy for 2-bit AlexNet leaves only a 1.5% drop from full-precision AlexNet, while ResNet18 and ResNet34 each gain approximately 1%.
  • 4.2 Results on the ImageNet Dataset: SPEQ outperforms existing KD methods on 3-bit EfficientNet-b0 without using a large teacher model.
  • 4.2 Results on the ImageNet Dataset: Combining SPEQ with Apprentice KD improves ResNet18 and ResNet34 top-1 accuracy by 0.4% and 0.6%, respectively.
  • 4.3 Results on Transfer Learning: SPEQ improves quantized BERT on SQuAD1.1 and enables 2-bit ResNet18 to outperform the floating-point model on Flowers-102.

5 Concluding Remarks

SPEQ performs self-distillation by sharing teacher and student parameters while stochastically varying activation precision to generate ensembled soft labels.

  • SPEQ uses shared teacher-student parameters and stochastic activation precision to produce soft labels from an ensemble of quantized models.
  • Cosine similarity loss supports distillation when the stochastic teacher is less confident than the student.
  • SPEQ outperforms existing quantized training methods across image classification, question-answering, and transfer learning without requiring a cumbersome teacher model.

A. Implementation details for high-precision inference

The inference experiment increases activation or weight precision while keeping the clipping level and representation range fixed.

  • Increasing precision adds discretization points while keeping the clipping level and representation range consistent.
  • The experiment evaluates weight-only and activation-only 2-bit models using equal or higher inference precision from 2 to 8 bits.

B. Quantization method

The study uses layer-wise uniform symmetric quantization with trainable clipping values for weights and activation signals, extending PACT to both domains.

  • B. Quantization method: Layer-wise uniform symmetric quantization clips and quantizes weights or input signals using a shared trainable scalar clipping value per layer.The method applies the same quantization framework to activation signals and weights.
  • B. Quantization method: PACT quantizes ReLU activations to n bits, using rounding and straight-through-estimator gradients while training the activation clipping value αx.For 2-bit activations, the output levels are {0, αx/3, 2αx/3, αx}.
  • B. Quantization method: L2 regularization decreases αx when all activation outputs remain below the clipping value and its gradient becomes zero.This regularization addresses a condition in which direct gradients cannot further reduce αx.
  • B. Quantization method: The modified PACT procedure quantizes n-bit weights with a trainable clipping value αw and uses four levels for 2-bit weights.The 2-bit levels are {−αw, −αw/3, αw/3, αw}.
  • B. Quantization method: Weight-clipping gradients are nonzero only when weights exceed αx or fall below −αx, and are zero otherwise.The supplied gradient expression describes the clipping-gradient conditions for quantized weights.

C. Gradient analysis of the cosine similarity loss

SPEQ’s stochastic teacher can be more or less confident than the student, making gradient direction a central consideration in choosing the distillation loss.

  • C. Gradient analysis of the cosine similarity loss: Stochastic activation precision produces teacher softmax outputs whose ground-truth confidence may be higher or lower than the student’s target-precision output.Figure 4 contrasts the corresponding student-logit gradients for more-confident and less-confident teachers.
  • C. Gradient analysis of the cosine similarity loss: 90.83% with KL-Loss was similar to 90.73% without KD because less-confident stochastic teachers can reverse the ground-truth gradient direction.KL-Loss produces a positive ground-truth-logit gradient in that case, lowering the student’s correct-class logit.
  • C. Gradient analysis of the cosine similarity loss: Cosine similarity loss preserves the ground-truth gradient direction when teacher confidence is lower than the student’s.The teacher probability acts as a scaling factor, so lower confidence yields a smaller rather than reversed gradient.

D. Experimental details

Experiments evaluate SPEQ across image classification, question answering, and transfer learning settings using varied architectures, datasets, and quantization configurations.

  • D. Experimental details: CIFAR10 and CIFAR100 experiments use VGG16, ResNet20, ResNet32, and MobileNetV2 under shared training settings with augmentation and scheduled learning-rate decay.CIFAR10 uses VGG16 and ResNet20, while CIFAR100 uses ResNet32 and MobileNetV2.
  • D. Experimental details: Figure 4 presents softmax outputs and student-logit gradients for teachers that are more or less confident than the student.The figure supplies paired output and gradient examples for the two confidence conditions.
  • D. Experimental details: ImageNet experiments evaluate AlexNet, ResNet18, and ResNet34 for 90 epochs with random cropping, horizontal flipping, and staged tenfold learning-rate decay.The initial low-precision training learning rate is 0.04, with decay at epochs 30, 50, 60, 70, and 80.
  • D. Experimental details: SQuAD1.1 quantizes BERT weights and hidden signals while fixing attention-score precision to nA bits in the stochastic precision path.The fixed attention-score precision reflects experiments in which stochastic attention-score quantization reduced performance.
  • D. Experimental details: Flowers-102 transfer learning uses frozen ImageNet-pretrained ResNet18 features and trains a new output layer with 5, 10, or 20 images per class.No random cropping or flipping is applied, and the new layer is trained for 50 epochs.
Loading 2009.14502v1…