Source-linked AI summary
And the Bit Goes Down: Revisiting the Quantization of Neural Networks
Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, Hervé Jégou
TL;DR
The paper addresses reducing the memory footprint of convolutional networks while preserving useful behavior. It introduces activation-focused Product Quantization guided by unlabelled data, achieving compact ImageNet and detection models with efficient CPU inference.
Problem
The paper tackles the problem of reducing convolutional-network memory footprints while preserving network-output quality for in-domain inputs.
Method
The method uses Product Quantization with weighted k-means to minimize reconstruction error on in-domain activations rather than on weights, with unlabelled-data distillation during sequential compression.
Results
The approach achieves a 5 MB ResNet-50 with 76.1% top-1 ImageNet accuracy and compresses Mask R-CNN by 26× while maintaining competitive performance.
Takeaways & Limitations
The resulting models do not require labelled data for compression and use byte-aligned representations that support efficient CPU inference across classification and detection settings.
Abstract
from arXiv · showhide
In this paper, we address the problem of reducing the memory footprint of convolutional network architectures. We introduce a vector quantization method that aims at preserving the quality of the reconstruction of the network outputs rather than its weights. The principle of our approach is that it minimizes the loss reconstruction error for in-domain inputs. Our method only requires a set of unlabelled data at quantization time and allows for efficient inference on CPU by using byte-aligned codebooks to store the compressed weights. We validate our approach by quantizing a high performing ResNet-50 model to a memory size of 5MB (20x compression factor) while preserving a top-1 accuracy of 76.1% on ImageNet object classification and by compressing a Mask R-CNN with a 26x factor.
1 INTRODUCTION
The paper proposes compression for ConvNets that preserves in-domain layer outputs rather than weights, using activation-aware quantization and unlabelled-data distillation. It targets compact, CPU-efficient models and reports strong ImageNet and detection results.
- The method targets memory-efficient compression of high-performing ConvNets for embedded applications such as robotics and virtual or augmented reality.
- The approach uses Product Quantization to exploit convolutional redundancy and stores compressed weights with byte-aligned indexes for efficient CPU inference.
- Rather than minimizing weight error, the method uses weighted k-means to focus on activation accuracy and improve reconstruction for in-domain inputs.
- Sequential compression uses uncompressed-network activations and distillation on unlabelled images to limit error accumulation and fine-tune codewords.
- 5 MB and 76.1% top-1 accuracy on ImageNet were achieved for a compressed ResNet-50, while Mask R-CNN reached a 26× compression factor with competitive performance.
2 RELATED WORK
Related work spans low-precision training, vector and product quantization, pruning, efficient architecture design, and hardware-aware compression. These approaches trade model size, accuracy, and inference efficiency in different ways.
- Network compression includes low-precision weights and activations, including binary or ternary weights and combinations of binary bases.
- Vector and Product Quantization decompose high-dimensional representations into separately quantized subspaces with codebooks.
- Pruning removes connections according to importance criteria and can extend to structured channel- or filter-level sparsity.
- Memory-efficient architectures such as SqueezeNet, ShuffleNet, MobileNets, and EfficientNets use depth-wise or point-wise convolutions and channel shuffling.
- Combining compression methods can yield high compression factors, while hardware-specific quantization targets energy efficiency and latency for deployment platforms.
3 OUR APPROACH
The approach quantizes network weights while directly minimizing output-reconstruction error on in-domain activations. It combines Product Quantization, weighted k-means, spatially structured convolutional quantization, and sequential distillation-based codebook refinement.
- Product Quantization: Product Quantization splits each weight column into m subvectors, assigns each to a shared codeword, and learns the codebook by minimizing a quantization objective.When m=1, PQ becomes vector quantization; when m=Cin, it becomes scalar k-means, while PQ provides an implicit codebook of size k^m.
- Activation-aware quantization: The method targets layer-output reconstruction rather than weight reconstruction, using in-domain activations to learn a weighted k-means codebook.It treats conventional weight approximation as insufficient for preserving outputs on the relevant input domain.
- Activation-aware quantization: The activation-aware objective reconstructs y = xW with ŷ = xĉW after unrolling input activations into subvectors, then adapts the EM procedure.The E-step assigns subvectors using activation-weighted reconstruction error, while the M-step updates codewords through a least-squares solution.
- Convolutional layers: Convolutional filters are quantized spatially with subvectors of size d = K × K to exploit correlation among nearby spatial patches.The reshaping preserves spatial coherence because spatial dimensions are more correlated than output and input channel dimensions.
- Network quantization: The network is compressed sequentially using current layer activations, then codebooks are refined by teacher-student distillation without requiring labelled data.A final global codebook fine-tuning step reduces residual drift across layers and updates BatchNorm statistics.
4 EXPERIMENTS
Experiments evaluate the method across ResNet architectures, compression regimes, size budgets, and ablations, including semi-supervised ImageNet models and Mask R-CNN. Results show strong accuracy–memory trade-offs and benefits from activation-aware quantization and distillation.
- Image classification results: The experiments evaluate vanilla ResNet-18 and ResNet-50 models on ImageNet across multiple compression regimes and compare against established compression methods.The study reports accuracy and compression factors for different operating points, including byte-compatible configurations with k = 256 centroids.
- Image classification results: 2× larger compression ratios are reported against ABC-Net on ResNet-18 and Deep Compression on ResNet-50 at comparable or higher accuracy.For ResNet-18, the comparison uses k = 512 centroids with large blocks; for ResNet-50, it uses k = 256 centroids with large blocks.
- Semi-supervised ResNet-50: 76.12% top-1 accuracy is achieved by a compressed semi-supervised ResNet-50 occupying 5.20 MB, versus 97.5 MB for the uncompressed model.The compressed model uses k = 256 centroids in the low-compression regime and matches the performance of a vanilla, non-compressed ResNet-50.
- Ablation study: Activation-aware quantization with distillation consistently outperforms the ablated alternatives using standard product quantization or label-based fine-tuning.Without activation-aware quantization and fine-tuning, standard product quantization produces top-1 accuracies below 25% across all operating points.
- Image detection: The Mask R-CNN experiment compresses the ResNet-50 FPN backbone and evaluates the method beyond image classification.The compression setup uses k = 256 centroids and layer-wise plus global fine-tuning.
5 CONCLUSION
The paper presents a Product Quantization method that achieves strong compression results on ResNet architectures and generalizes to Mask R-CNN. It uses unlabeled data and byte-aligned models for CPU inference, while identifying broader architecture coverage and nonlinear reconstruction as future directions.
- Conclusion: The proposed Product Quantization method achieves state-of-the-art results on ResNet architectures and generalizes to Mask R-CNN.The conclusion identifies both the primary architecture family and the additional detection architecture.
- Conclusion: The compression scheme requires no labeled data and produces byte-aligned models that support efficient CPU inference.These properties are stated as practical characteristics of the resulting compressed networks.
- Conclusion: The authors identify wider architectural evaluation and incorporating nonlinearity into reconstruction error as directions for further research.They also mention adapting the method to simultaneously compress and transfer ImageNet-trained ResNets to other domains.