Source-linked AI summary
Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Christopher De Sa, Zhiru Zhang
TL;DR
Post-training quantization is needed when retraining is unavailable, but outliers mismatch practical linear quantization grids. OCS duplicates outlier-containing channels and halves their values, improving quantization while adding only minor network overhead.
Problem
Post-training quantization addresses scenarios where retraining is unavailable, including black-box, off-the-shelf, or legacy floating-point models without accessible training data.
Method
OCS identifies channels containing outliers, duplicates them, and halves their values, preserving network function while moving outliers toward the distribution center.
Results
OCS on weights outperforms state-of-the-art clipping techniques with minimal overhead on deep CNN and RNN benchmarks, and combining OCS with clipping helps at very low precision.
Takeaways & Limitations
OCS offers a no-retraining quantization method for commodity hardware, trading reduced quantization error for increased network size.
Takeaways & Limitations
OCS can require duplicating entire activation channels and connected weight filters, increasing network size because splitting one weight value requires adding a full matrix row.
Abstract
from arXiv · showhide
Quantization can improve the execution latency and energy efficiency of neural networks on both commodity GPUs and specialized accelerators. The majority of existing literature focuses on training quantized DNNs, while this work examines the less-studied topic of quantizing a floating-point model without (re)training. DNN weights and activations follow a bell-shaped distribution post-training, while practical hardware uses a linear quantization grid. This leads to challenges in dealing with outliers in the distribution. Prior work has addressed this by clipping the outliers or using specialized hardware. In this work, we propose outlier channel splitting (OCS), which duplicates channels containing outliers, then halves the channel values. The network remains functionally identical, but affected outliers are moved toward the center of the distribution. OCS requires no additional training and works on commodity hardware. Experimental evaluation on ImageNet classification and language modeling shows that OCS can outperform state-of-the-art clipping techniques with only minor overhead.
1. Introduction
Post-training quantization addresses the cost of deploying floating-point DNNs when retraining is unavailable. OCS reduces outlier distortion by splitting channels while preserving function, outperforming clipping with minor overhead.
- Quantizing floating-point weights and activations into low-precision numbers reduces model size and MAC complexity, improving throughput and energy efficiency.
- Post-training quantization is important when providers lack client training data or clients lack the expertise, manpower, or data needed for retraining.
- Bell-shaped weight and activation distributions conflict with linear quantization grids because rare outliers receive disproportionate grid range.
- Clipping reduces overall MSE by narrowing dynamic range but greatly distorts outliers, whereas OCS moves them toward the distribution center by splitting channels.
- OCS duplicates outlier-containing channels and halves their values, creating a functionally identical network without retraining or specialized hardware.
- OCS outperforms state-of-the-art clipping techniques on weight quantization while incurring negligible overheads.
2. Related Work
Prior post-training work primarily optimizes clipping thresholds, while OCS instead leverages model expansion to reduce outliers on commodity hardware. Related approaches either use specialized hardware or differ in evaluation scope.
- Clipping methods optimize thresholds using quantization-error minimization, fitted distributions, extrapolated statistics, percentiles, or bitwidth and format tuning.
- TensorRT profiles activations from thousands of user-provided samples and selects thresholds by minimizing KL divergence between original and quantized distributions.
- OCS differs from clipping by leveraging model expansion to improve quantization.
- Outlier-aware quantization uses separate low- and high-precision grids, but requires a specialized outlier-aware accelerator.
- OCS applies the Net2WiderNet transformation to reduce outliers while preserving functional equivalence.
- Cell division is conceptually similar but differs in whether it quantizes activations, uses training data, compares clipping, and evaluates benchmarks.
3. Outlier Channel Splitting
OCS narrows DNN value distributions without changing layer function by duplicating outlier-containing channels and splitting their values. Quantization-aware splitting preserves the original quantized result, while channel selection controls overhead by prioritizing large values.
- Core transformation: OCS reduces outlier magnitudes by duplicating a neuron and halving either its output or outgoing weights, preserving functional equivalence.The transformation narrows weight or activation distributions for linear quantization.
- Practical considerations: OCS targets outliers without retraining, trading reduced quantization error for a larger network usable on commodity hardware.Weight selection uses fixed post-training weights, while activation selection samples distributions from a small number of training images.
- Core transformation: Splitting a channel requires duplicating an entire channel and the associated matrix row or convolutional filters, creating network overhead.For convolutional layers, this includes the full 2D activation channel and connected 2D weight filters.
- Quantization-aware splitting: Naive division by two can double maximum total quantization error because both halves may round in the same direction.For example, w = 3 produces two values of 1.5, which can incur same-direction rounding.
- Quantization-aware splitting: QA splitting forces rounding in different directions near grid midpoints and preserves the original quantization result Q(w).The derivation applies Hermite’s Identity with n = 2.
- Channel selection: OCS selects channels sequentially by largest absolute value and splits ceil(r ∗C) channels, with r controlling approximate tolerable overhead.A knapsack-based allocation method was experimentally no better than the simple expansion-ratio method.
4. Clipping
Clipping is a post-training quantization strategy that chooses a narrower range to reduce error, using several threshold-selection criteria. The section evaluates MSE, ACIQ, and KL-divergence-based approaches.
- Overview: Clipping is presented as the state-of-the-art approach for DNN quantization without training.It can be applied to both weights and activations.
- Mean Squared Error: The MSE method selects the clip threshold minimizing mean squared error between floating-point and quantized values.It builds a histogram and sweeps candidate thresholds between zero and the maximum absolute value.
- ACIQ: ACIQ fits Gaussian and Laplacian distributions and uses the better-fitting model to analytically compute an approximate MSE-minimizing threshold.This avoids sweeping candidate thresholds and permits faster adjustment for activation batches.
- Quantizer setup: The experiments use a symmetric linear quantizer with 2^m −1 grid points, including a zero grid point for signed values.The implementation adjusts formulas from prior work to match this representation.
- KL divergence: The KL-divergence method selects a candidate threshold that approximately minimizes divergence between floating-point and quantized distributions.The implementation smooths the quantized histogram because the two distributions generally have different support.
5. Experimental Evaluation on CNNs
CNN experiments compare post-training clipping and OCS for weights and activations across ImageNet models. Weight OCS generally outperforms clipping with small overhead, whereas activation clipping is stronger than non-oracle OCS.
- Setup: CNN experiments use four pretrained ImageNet models and inference-only evaluation, leaving the first layer unquantized.The models are VGG16, ResNet-50, DenseNet-121, and Inception-V3.
- Weight Quantization: QA splitting is clearly better than naive division by two at 4 bits, where naive splitting causes significant accuracy degradation.The difference between methods is negligible at higher bitwidths.
- Weight Quantization: At 6 bits or fewer, clipping improves accuracy by up to 55% for ResNet-50 and 40% for Inception-V3.The best clipping method shifts from no clipping at high bitwidths, to MSE/ACIQ, then KL at 4 bits.
- Weight Quantization: Weight OCS with r = 0.01 outperforms benchmarked clipping methods at 8-5 bits, while r = 0.02 exceeds clipping by up to 13% for Inception-V3 at 6 and 5 bits.Most gains come from the first channel split, which targets the largest outlier.
- Weight Quantization: At 5 and 4 bits, OCS plus clipping outperforms OCS alone, while higher-precision OCS often removes enough outliers to make additional clipping unnecessary.OCS and clipping both reduce the dynamic range, creating overlap between their effects.
- Activation Quantization: Activation clipping is effective at all tested bitwidths, with MSE generally strongest, while activation OCS improves over linear quantization but performs worse than clipping.Oracle OCS can match or surpass clipping, but requires dynamic runtime channel selection.
6. Experimental Evaluation on RNNs
RNN experiments evaluate weight quantization on a two-layer LSTM language model using WikiText-2. Clipping does not improve perplexity, whereas OCS improves it at sufficiently large expansion ratios and low precision.
- Setup: The RNN experiment uses a two-layer LSTM language model on WikiText-2, keeping activations and hidden states in floating point.Each LSTM layer has hidden size 650, and the vocabulary contains 33,278 words.
- Results: The floating-point baseline achieves perplexity 95.1, and lower perplexity is better.The experiment compares OCS combined with different clipping methods for quantized weights.
- Results: None of the clipping techniques improve perplexity on this model, while OCS achieves substantially better results.OCS begins outperforming the baseline at 6 bits with r = 0.05.
7. Conclusions and Future Work
The paper concludes that OCS enables no-retraining quantization on commodity hardware by reducing outlier magnitude through channel splitting. Weight OCS outperforms clipping with minimal overhead, and combining both methods helps at very low precision.
- Conclusions: OCS improves DNN quantization without retraining and can run on commodity hardware by splitting channels to reduce outlier magnitude.The method trades lower quantization error for network-size overhead.
- Conclusions: Weight OCS outperforms state-of-the-art clipping techniques with minimal overhead on deep CNN and RNN benchmarks.At very low precision, OCS combined with clipping outperforms either method alone.
- Future Work: Future work includes deeper study of channel-selection methods and applying OCS quantization during training.The authors suggest OCS may shape weight distributions during training.