Source-linked AI summary
Loss Aware Post-training Quantization
Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhskii, Ron Banner, Alex M. Bronstein, Avi Mendelson
TL;DR
Post-training quantization remains less accurate at INT4 or lower, although it is useful for deploying DNNs on resource-constrained devices. This paper studies how quantization changes the loss landscape and introduces LAPQ, which jointly optimizes layer parameters. It reports that mild quantization yields flat, separable landscapes, while aggressive quantization produces non-separable steep curvature and LAPQ improves post-training accuracy, including near-baseline results in selected 4-bit settings.
Problem
Current post-training quantization methods provide reasonable accuracy at INT8 or above but fall short at INT4 or lower, while tensor-level metrics and independent layer optimization overlook cross-layer effects.
Method
LAPQ jointly optimizes all layer quantization parameters by minimizing the neural network loss, using loss-landscape analysis and multivariate quadratic optimization.
Results
LAPQ significantly outperforms current post-training methods across two challenging tasks and six DNN architectures, with 0.5% degradation from FP32 on NCF-1B at 8-bit quantization.
Takeaways & Limitations
Mild quantization supports straightforward post-training optimization, whereas aggressive quantization requires joint selection of quantization parameters because the loss becomes highly non-separable with steep curvature.
Takeaways & Limitations
The activation analysis is limited to ReLU networks with quantization ranges of [0, c].
Abstract
from arXiv · showhide
Neural network quantization enables the deployment of large models on resource-constrained devices. Current post-training quantization methods fall short in terms of accuracy for INT4 (or lower) but provide reasonable accuracy for INT8 (or above). In this work, we study the effect of quantization on the structure of the loss landscape. Additionally, we show that the structure is flat and separable for mild quantization, enabling straightforward post-training quantization methods to achieve good results. We show that with more aggressive quantization, the loss landscape becomes highly non-separable with steep curvature, making the selection of quantization parameters more challenging. Armed with this understanding, we design a method that quantizes the layer parameters jointly, enabling significant accuracy improvement over current post-training quantization methods. Reference implementation is available at https://github.com/ynahshan/nn-quantization-pytorch/tree/master/lapq
1 Introduction
Quantization reduces DNN computational requirements for resource-constrained deployment, but post-training methods struggle at low bit widths and often optimize layers independently despite cross-layer dependencies. The paper frames quantization as a joint loss-optimization problem and proposes combining layer-wise initialization with multivariate quadratic optimization.
- Motivation: Quantization uses lower-precision representations to improve DNN computational efficiency and support deployment on resource-constrained edge devices.DNNs’ computational requirements hinder adoption on edge devices, motivating compression methods.
- Loss landscape: Figure 1 visualizes a complex, inseparable cross-entropy loss surface over quantization step sizes for the first two layers.The surface uses one batch of 512 ImageNet images with ResNet18, while colored dots represent solutions from different p-norm objectives.
- Problem: Post-training quantization avoids training and typically needs only a small calibration dataset, but existing methods generally achieve reliable results only around INT8.Training-based approaches can compensate for quantization loss but require substantial resources and training data access.
- Problem: Tensor-level error metrics may not predict network performance, and quantization noise can be amplified across successive layers.These issues make it difficult to select an optimal metric and create dependencies between layer quantization errors.
- Contribution: The proposed method jointly optimizes layer quantization step sizes against network cross-entropy loss instead of optimizing each layer independently.The paper combines layer-by-layer quantization with multivariate quadratic optimization and reports strong interactions among layers.
- Contribution: The method significantly outperforms state-of-the-art post-training quantization methods on two challenging tasks and six DNN architectures.The paper presents this as its final contribution and reports near-baseline behavior in selected low-bit settings.
2 Related work
Prior quantization work spans training-aware and post-training approaches, with post-training methods favored for embedded deployment but generally limited to 8-bit representations. Existing techniques improve quantization through clipping, distribution changes, bias correction, finer granularity, or non-uniform mappings, while usually optimizing layers separately.
- Approach categories: Quantization-aware training can reach 2–4 bits per parameter with baseline-comparable results, whereas post-training quantization avoids weight optimization but has commonly reached only 8 bits.Quantization-aware methods require large labeled datasets and substantial computational power, encouraging post-training deployment solutions.
- Error minimization: Early post-training methods selected thresholds using min-max norms, while later methods optimized clipping through KL divergence, weight distributions, or iterative MSE.These approaches target local quantization distortion and are designed for efficient deployment.
- Distribution adjustment: Other methods improve quantization by equalizing weight ranges, splitting outlier channels, or correcting quantization-induced distribution bias.These techniques modify tensor distributions or compensate for shifts in parameter or activation statistics.
- Deployment trade-offs: Finer-grained quantization and non-uniform mappings can reduce performance degradation but require additional parameters or hardware support.These requirements make them less favorable for real-life or consumer-grade deployment.
- Research gap: Previous work generally did not account for non-separable loss during per-layer or per-channel optimization.Nagel et al. partially addressed this issue by treating pairs of consecutive layers together, while other cited exceptions did not show optimization.
3 Loss landscape of quantized DNNs
The paper analyzes how quantization changes loss-landscape separability and curvature. Mild quantization yields approximately separable, flat behavior, whereas aggressive quantization creates layer interactions and steep curvature that challenge layer-wise parameter selection.
- 3.2 Curvature: At aggressive quantization, the loss becomes highly non-separable with steep curvature, making existing tensor-level error-minimization methods unfavorable.This behavior explains why selecting quantization parameters becomes more difficult at low bitwidths.
- 3.1 Separable optimization: For sufficiently small quantization noise, loss degradation can be approximated as a sum of independent, separable layer-wise processes.This approximation follows by neglecting higher-order terms in the Taylor expansion.
- 3.1 Separable optimization: Larger quantization noise introduces interactions between layers through the quantization interaction term, requiring non-separable optimization.The interaction corresponds to the second-order cross-layer term in the loss expansion.
- 3.2 Curvature: The flat 4-bit loss surface is generic across fine-grained quantization, while coarser quantization generally produces steeper curvature.The conclusion is not attributed to the specific pair of layers visualized in Fig. 2.
- 3.2 Curvature: The Hessian reveals stronger dependencies between adjacent layers than distant layers through larger adjacent off-diagonal terms.These terms correspond to dependencies between clipping parameters of neighboring layers.
- 3.2 Curvature: At 4-bit precision, accuracy is nearly insensitive to small step-size changes, whereas at 2-bit precision the same changes shift accuracy by more than 20%.The best accuracy at 2 bits is obtained by a step minimizing L3.5 rather than the MSE, which corresponds to L2 minimization.
4 Loss Aware Post-training Quantization (LAPQ)
LAPQ addresses the non-separable, steep loss landscape of aggressively quantized networks through a three-stage procedure that initializes and then jointly optimizes layer quantization parameters.
- Method: LAPQ treats quantization as a multivariate optimization problem because the loss landscape is complex and layer parameters are coupled.The method is designed to overcome the difficulty of optimizing quantization steps independently.
- Layer-wise optimization: The method first finds layer-wise quantization steps minimizing Lp quantization error for several p values.Different p values produce candidate step sets that remain optimal under their respective error metrics and lie near the loss optimum.
- Quadratic approximation: LAPQ fits a quadratic function along the candidate trajectory to estimate an optimal quantization step for initialization.The quadratic approximation is motivated by the vanishing first derivative at the local minimum.
- Joint optimization: A gradient-free Powell optimization jointly refines all layer parameters starting from the quadratic-approximation solution.The algorithm iteratively searches along direction vectors and updates the joint quantization-step position.
5 Experimental Results
Experiments evaluate LAPQ on ImageNet vision models and the NCF-1B recommendation task using small calibration sets, with comparisons to existing methods and component analyses.
- Evaluation setup: Experiments cover ImageNet classification and the NCF-1B recommendation system using held-back calibration sets before validation evaluation.The calibration sets contain 512 random images for ImageNet and 50k random user/item pairs for NCF-1B.
- Evaluation setup: Layer-wise quantization is used instead of finer group-wise, channel-wise, pixel-wise, or filter-wise assignments requiring special hardware and additional resources.The paper presents layer-wise quantization as compatible with existing low-precision integer hardware.
- NCF-1B: LAPQ achieves near-baseline NCF-1B accuracy at 8 bits, with 0.5% degradation from FP32 results, while naive MMSE degrades substantially.Table 2 compares LAPQ and MMSE using NCF-1B hit rate.
- Component analysis: The experiments separately assess initialization and bias correction, showing that LAPQ initialization supports joint optimization and bias correction is especially important for MobileNet.Table 3 compares initialization variants for ResNet-18, while Table 4 evaluates bias correction across CNN architectures.
6 Conclusion
The paper concludes that aggressive quantization creates a non-separable, steep loss landscape and introduces LAPQ to jointly optimize quantization parameters for improved post-training accuracy.
- Conclusion: At low precision, the loss function is non-separable and has steep curvature, challenging existing post-training quantization methods.This structure makes tensor-level quantization-error minimization unfavorable in the aggressive-quantization regime.
- Conclusion: LAPQ jointly minimizes the network loss over quantization parameters and outperforms current post-training quantization methods.The method retains layer-wise quantization without requiring channel-wise or filter-wise hardware support.
- Conclusion: In some models, LAPQ nearly reaches full-precision baseline accuracy for 4-bit post-training quantization with layer-wise quantization.The conclusion identifies this as compatible performance in the 4-bit regime for some models.
A Hessian of the loss function
The Hessian of the quantized loss is used to measure dependencies among layer quantization steps. More aggressive 2-bit quantization produces sharper curvature and stronger cross-layer coupling than 4-bit quantization.
- The Hessian contains second-order derivatives of the loss with respect to the vector of quantization steps.Its structure is analyzed to estimate dependencies between clipping parameters of different layers.
- Off-diagonal Hessian magnitudes measure non-separability, because separable functions have diagonal Hessians.Larger off-diagonal values indicate stronger interactions between quantization steps.
- 2-bit quantization has higher diagonal Hessian values than 4-bit quantization, indicating a sharper minimum.In the plotted 15-layer ResNet-18 analysis, closer layers generally show stronger interactions.
B Calibration set size
Calibration-set size balances running time against generalization when applying the proposed method. For ResNet-18, 512 calibration samples are reported as a good balance across tested bitwidths.
- 512 calibration samples provide a good balance between running time and generalization for ResNet-18 across tested bitwidths.The choice was evaluated using various calibration-set sizes and quantization levels.
- The calibration-set experiment measures accuracy for different set sizes at various quantization levels.
C ImageNet additional results
The paper reports additional ImageNet image-classification results for the proposed method and presents a comparison with other methods.
- Additional results evaluate the proposed method on ImageNet for image classification.
- Table C.1 compares the proposed method with other methods.