Source-linked AI summary
Up or Down? Adaptive Rounding for Post-Training Quantization
Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, Tijmen Blankevoort
TL;DR
Post-training quantization needs weight rounding that preserves task performance better than rounding-to-nearest. AdaRound uses a task- and data-aware theoretical formulation, a layer-wise local loss, and soft relaxation; it achieves state-of-the-art results, including 4-bit Resnet18 and Resnet50 within a 1% accuracy drop without fine-tuning.
Problem
Rounding-to-nearest is predominant, but it is not optimal for post-training quantization, especially at lower bit widths.
Method
AdaRound analyzes rounding through task loss, formulates a per-layer QUBO, simplifies it to a local loss, and optimizes it with continuous relaxation.
Results
AdaRound establishes a new state-of-the-art for post-training quantization and quantizes Resnet18 and Resnet50 to 4-bit weights while keeping the accuracy drop within 1%.
Takeaways & Limitations
AdaRound provides a fast post-training rounding method that uses a small amount of unlabeled data and does not require end-to-end fine-tuning.
Takeaways & Limitations
The analysis relies on small perturbations and assumes a converged network so higher-order and gradient terms can be neglected or treated as small.
Abstract
from arXiv · showhide
When quantizing neural networks, assigning each floating-point weight to its nearest fixed-point value is the predominant approach. We find that, perhaps surprisingly, this is not the best we can do. In this paper, we propose AdaRound, a better weight-rounding mechanism for post-training quantization that adapts to the data and the task loss. AdaRound is fast, does not require fine-tuning of the network, and only uses a small amount of unlabelled data. We start by theoretically analyzing the rounding problem for a pre-trained neural network. By approximating the task loss with a Taylor series expansion, the rounding task is posed as a quadratic unconstrained binary optimization problem. We simplify this to a layer-wise local loss and propose to optimize this loss with a soft relaxation. AdaRound not only outperforms rounding-to-nearest by a significant margin but also establishes a new state-of-the-art for post-training quantization on several networks and tasks. Without fine-tuning, we can quantize the weights of Resnet18 and Resnet50 to 4 bits while staying within an accuracy loss of 1%.
1. Introduction
Neural-network quantization improves inference efficiency but introduces performance loss, motivating practical post-training methods. AdaRound formulates data- and task-aware rounding and achieves strong post-training results across networks and tasks.
- Quantization uses low-bit weights and activations to reduce memory-transfer costs, power consumption, and inference time, but can reduce performance.
- Post-training quantization is emphasized because it can be applied during deployment without retraining or extensive hyperparameter tuning.
- Rounding-to-nearest is predominant, yet it is not optimal for post-training quantization.
- AdaRound formulates rounding as a per-layer Quadratic Unconstrained Binary Optimization problem using input-data and task-loss characteristics.
- AdaRound solves the per-layer formulation through continuous relaxation and requires only a small amount of unlabelled data while supporting convolutional and fully-connected layers.
- AdaRound establishes a new state-of-the-art for post-training quantization across several networks and tasks, including Resnet18, Resnet50, MobilenetV2, InceptionV3, and DeeplabV3.
2. Motivation
The paper explains rounding quality through the task loss of a perturbed pretrained network rather than per-weight distance alone. A second-order analysis highlights interactions between perturbations, which can make non-nearest rounding preferable.
- A second-order Taylor expansion approximates the task-loss change caused by perturbing pretrained network weights.
- Ignoring higher-order terms is justified for sufficiently small perturbations, while convergence allows the expected-gradient term to be treated as close to zero.
- The Hessian captures interactions between perturbed weights through their joint impact on task loss.
- Diagonal terms depend only on perturbation magnitudes, whereas off-diagonal terms make perturbation signs relevant.
- Opposite perturbation signs can improve loss, creating a trade-off between diagonal and off-diagonal contributions that rounding-to-nearest ignores.
- In 100 stochastic first-layer rounding trials for Resnet18, 48 choices produced better performance than rounding-to-nearest.
3. Method
AdaRound formulates weight rounding as a task-loss-aware per-layer optimization problem, then makes it tractable with local approximations and a continuous relaxation. The method uses input statistics and a regularized soft rounding process to obtain binary rounding decisions efficiently.
- Task loss based rounding: AdaRound begins with a theoretically motivated per-layer binary rounding optimization derived from the task loss.The formulation uses a second-order Taylor approximation and ignores cross-layer interactions, yielding a QUBO problem.
- Task loss based rounding: The QUBO objective correlates with ImageNet validation accuracy across 100 stochastic rounding choices for 4-bit first-layer quantization of Resnet18.This supports using the approximate cost as a proxy for task-loss optimization.
- From Taylor expansion to local loss: The method replaces the expensive Hessian with a local preactivation reconstruction objective based on mean squared error.Under the stated constant-diagonal Hessian assumption, this objective requires no knowledge of subsequent layers or the task loss.
- AdaRound: AdaRound relaxes the discrete optimization to continuous variables and optimizes the resulting objective with stochastic gradient descent.The relaxed weights use differentiable soft quantization variables and a regularizer that encourages binary solutions.
- AdaRound: A rectified sigmoid and annealed regularization let rounding variables adapt initially and converge toward 0 or 1 later.The rectified sigmoid preserves gradients near the endpoints, while annealing shifts optimization toward binary solutions.
4. Background and related work
The paper situates AdaRound within efficient neural-network inference and the shift from quantization-aware training toward easier post-training quantization. Prior work includes methods for training-time simulation, clipping-range optimization, channel splitting, and layer-specific bit-width selection.
- Quantization for efficient inference: Low-bit quantization reduces memory transfer, power consumption, and inference time but generally introduces performance loss.The related work motivates methods that preserve performance while reducing bit width.
- Quantization-aware training: Quantization-aware training can achieve strong results but requires retraining models and tuning hyperparameters.This practical cost motivates post-training approaches.
- Post-training quantization: Post-training quantization is easier to apply because networks can be quantized during deployment without additional user retraining effort.Recent methods optimize clipping ranges, split channels, or assign different bit widths across layers.
- Post-training quantization: Prior post-training methods improve quantization through clipping-range optimization, channel splitting, or layer-specific bit-width assignment.These approaches use per-layer loss trade-offs, computation increases, or sensitivity information such as SQNR and the Hessian.
5. Experiments
AdaRound is evaluated through ablations, robustness tests, and comparisons with post-training quantization methods. Across these experiments, its design choices improve performance, and it achieves strong ImageNet accuracy with limited unlabelled optimization data.
- Experimental setup: Experiments evaluate AdaRound across approximation choices, optimization design choices, reconstruction objectives, quantization grids, data requirements, and competing methods.The study uses Resnet18 extensively and compares results on several ImageNet models.
- Ablation study: Hessian-based task-loss optimization provides a significant boost over rounding-to-nearest, supporting the Taylor-expansion-based rounding objective.The corresponding experiment reports ImageNet validation accuracy for Resnet18; some alternatives were computationally infeasible.
- Ablation study: Explicit regularization with annealed optimization outperforms the classical temperature-annealing method and produces more stable optimization with lower variance.The rectified sigmoid also provides a consistent small accuracy improvement across models.
- Ablation study: Asymmetric reconstruction MSE produces a noticeable accuracy improvement, while incorporating the activation function provides a small additional gain.These choices are evaluated within the rounding optimization objective.
- Ablation study: AdaRound clearly outperforms straight-through-estimator optimization, which allows more flexible quantized-weight movement but has biased gradients in this setting.The comparison is reported for Resnet18 on ImageNet validation accuracy.
- Ablation study: AdaRound improves over rounding-to-nearest independently of the quantization grid, while both MSE-based grid choices outperform Min-Max.The experiments continue with the selected MSE formulation because neither MSE variant is a clear overall winner.
- Ablation study: With as few as 256 images, AdaRound brings the model within 2% of original FP32 accuracy, and similar-domain unlabelled data remains competitive.Figure 4 studies robustness to image count and dataset source.
- Literature comparison: For 4-bit weight quantization, AdaRound stays within 1% of FP32 accuracy on Resnet18 and Resnet50 and within 2% on InceptionV3 and MobilenetV2.It outperforms competing post-training methods, including comparisons where competitors use more favorable settings.
6. Conclusion
AdaRound is a new rounding method for post-training neural-network weight quantization that significantly improves over rounding-to-nearest. It is fast, uses little unlabeled data, avoids end-to-end fine-tuning, and supports 4-bit weights with limited accuracy loss.
- AdaRound improves significantly over rounding-to-nearest, especially at lower bit widths.
- AdaRound uses a small number of unlabeled data examples and does not require end-to-end fine-tuning.
- AdaRound applies to neural networks with convolutional or fully-connected layers without restrictions.
- AdaRound establishes a new state-of-the-art for post-training weight quantization with significant gains.
- AdaRound can quantize Resnet18 and Resnet50 to 4-bit weights while keeping the accuracy drop within 1%.
A. Comparison among QUBO solvers
The cross-entropy method substantially outperforms the qbsolv QUBO solver for optimizing the task-loss Hessian in the reported Resnet18 comparison.
- Cross-entropy optimization significantly outperforms the qbsolv QUBO solver.The comparison uses the same experimental setup, with only the first Resnet18 layer quantized to 4 bits and ImageNet validation accuracy reported.
- The qbsolv QUBO solver performs worse than rounding-to-nearest in the reported comparison.
B. From Taylor expansion to local loss (conv. layer)
Under stated Hessian and data-independence assumptions, the convolutional-layer optimization becomes output-feature-map MSE optimization that separates by output channel.
- For a convolutional layer, the preactivation is defined as z^(ℓ) = W^(ℓ) ∗ x^(ℓ−1).
- The derivation assumes the relevant Hessian term is diagonal and independent of data samples.
- Under these assumptions, weights affecting different output filters have no interactions in the optimization.
- The Hessian optimization becomes equivalent to MSE optimization for the output feature map.
- The optimization further decomposes into separate problems for each output channel.The decomposition follows because the summed terms involve disjoint variable sets.