Source-linked AI summary
ADMM-NN: An Algorithm-Hardware Co-Design Framework of DNNs Using Alternating Direction Method of Multipliers
Ao Ren, Tianyun Zhang, Shaokai Ye, Jiayu Li, Wenyao Xu, Xuehai Qian, Xue Lin, Yanzhi Wang
TL;DR
DNN compression lacked a systematic framework that jointly exploited weight and bit redundancy while accounting for computation, energy, and hardware overheads. ADMM-NN uses ADMM for joint pruning and quantization, paired with hardware-aware optimization, and reports large pruning, storage, and computation reductions without accuracy loss.
Problem
Prior DNN compression methods lacked a systematic joint framework for weight pruning and quantization and did not adequately account for computation, energy efficiency, and hardware overheads.
Method
ADMM-NN applies ADMM to jointly optimize weight pruning and quantization, then performs hardware-aware optimization accounting for computation, energy, and irregular-sparsity overhead.
Results
Without accuracy loss, ADMM-NN achieves 85× and 24× pruning on LeNet-5 and AlexNet, and 1,910× and 231× reductions in overall model size.
Takeaways & Limitations
The framework provides joint model compression with hardware-level considerations, with promising results also reported for VGGNet and ResNet-50.
Takeaways & Limitations
Actual storage reduction is lower than pruning ratios because pruning requires index storage, and high quantization performance can make indices dominate storage.
Abstract
from arXiv · showhide
To facilitate efficient embedded and hardware implementations of deep neural networks (DNNs), two important categories of DNN model compression techniques: weight pruning and weight quantization are investigated. The former leverages the redundancy in the number of weights, whereas the latter leverages the redundancy in bit representation of weights. However, there lacks a systematic framework of joint weight pruning and quantization of DNNs, thereby limiting the available model compression ratio. Moreover, the computation reduction, energy efficiency improvement, and hardware performance overhead need to be accounted for besides simply model size reduction. To address these limitations, we present ADMM-NN, the first algorithm-hardware co-optimization framework of DNNs using Alternating Direction Method of Multipliers (ADMM), a powerful technique to deal with non-convex optimization problems with possibly combinatorial constraints. The first part of ADMM-NN is a systematic, joint framework of DNN weight pruning and quantization using ADMM. It can be understood as a smart regularization technique with regularization target dynamically updated in each ADMM iteration, thereby resulting in higher performance in model compression than prior work. The second part is hardware-aware DNN optimizations to facilitate hardware-level implementations. Without accuracy loss, we can achieve 85$\times$ and 24$\times$ pruning on LeNet-5 and AlexNet models, respectively, significantly higher than prior work. The improvement becomes more significant when focusing on computation reductions. Combining weight pruning and quantization, we achieve 1,910$\times$ and 231$\times$ reductions in overall model size on these two benchmarks, when focusing on data storage. Highly promising results are also observed on other representative DNNs such as VGGNet and ResNet-50.
1 INTRODUCTION
DNN compression combines weight pruning and quantization, but prior methods lacked a systematic joint framework and often overlooked computation and hardware overheads. ADMM-NN addresses these gaps through ADMM-based joint compression and hardware-aware optimization, achieving substantial pruning, storage, and computation reductions.
- Motivation: Off-chip DRAM accesses can consume 200× more energy than on-chip SRAM, making large DNN models costly for embedded hardware.Limited on-chip SRAM forces frequent DRAM access for large models.
- Motivation: Weight pruning exploits redundancy in the number of weights, while quantization exploits redundancy in their bit representation.Both techniques reduce storage and computational requirements, and binary or ternary quantization can eliminate multiplication operations.
- Limitations of Prior Work: Prior work lacked a systematic joint framework for pruning and quantization and mainly emphasized model-size reduction rather than computation and hardware efficiency.Compression concentrated on fully connected layers, while convolutional layers remained comparatively less compressed and incurred irregular-sparsity overheads.
- ADMM-NN Framework: ADMM-NN uses ADMM to jointly optimize pruning and quantization through a loss-optimization subproblem and an analytically solvable constraint subproblem.The method acts as dynamically updated regularization across ADMM iterations.
- ADMM-NN Framework: ADMM-NN adds hardware-aware optimization that accounts for computation reduction, energy efficiency, and irregular-sparsity overhead.The framework focuses primarily on convolutional layers while compressing fully connected layers accordingly to avoid overfitting and accuracy degradation.
- Results: 85× and 24× weight pruning are achieved on LeNet-5 and AlexNet without accuracy loss, while joint compression reduces stored model size by 1,910× and 231×.The reported computation reduction is 3.6× relative to prior work without accuracy loss.
2 BACKGROUND
Weight pruning and quantization exploit different redundancies and offer complementary compression benefits, but prior combinations were largely heuristic. ADMM decomposes constrained non-convex optimization into tractable subproblems, enabling the paper’s joint formulation.
- Weight Pruning and Quantization: Weight pruning removes redundant weights, while quantization reduces the number of bits used to represent weights.Pruning can provide higher compression potential, whereas quantization is more hardware-friendly.
- Weight Pruning and Quantization: Prior pruning work used iterative heuristics, achieving 9× weight reduction on AlexNet but only 2.7× reduction in its convolutional layers.The convolutional layers are important because they constitute a key computational part of modern DNNs.
- Weight Pruning and Quantization: Weight pruning incurs irregular-sparsity and indexing overhead, whereas binary or ternary quantization can eliminate multiplication operations.These differences create distinct hardware trade-offs between the two compression methods.
- Joint Compression: Because pruning and quantization exploit different redundancies, they can be combined, but prior work lacked a systematic investigation of their joint use.Earlier combinations achieved 27× and later 53× model compression on AlexNet, with the latter lacking a compressed-model specification.
- ADMM Basics: ADMM rewrites a difficult non-convex problem as two subproblems whose solutions can be obtained separately and efficiently.The first subproblem uses a quadratic regularizer and the second can be solved analytically when the constraint function has exploitable structure.
3 ADMM FRAMEWORK FOR JOINT WEIGHT PRUNING AND QUANTIZATION
ADMM reformulates joint pruning and quantization as a constrained optimization problem, then solves it through alternating training and projection steps. The joint procedure prunes first and quantizes remaining nonzero weights, while ADMM dynamically updates the regularization target across iterations.
- Problem formulation: The joint compression problem constrains each layer to retain at most α_i nonzero weights while mapping the remaining weights to quantization levels.The constraint set supports pruning, quantization, or both simultaneously.
- ADMM decomposition: ADMM separates the non-convex constrained problem into an optimization subproblem and a projection subproblem solved iteratively.The first subproblem removes the combinatorial constraints from stochastic-gradient training, while the second projects onto the constraint set.
- ADMM decomposition: The training subproblem uses an L2-regularized DNN loss, with the regularization target dynamically updated by the projection subproblem at each iteration.This dynamic target is identified as the key reason ADMM outperforms conventional regularization for pruning and quantization.
- Projection solutions: For pruning, Euclidean projection keeps the α_i largest-magnitude weights and sets the rest to zero; for quantization, it maps each weight to the nearest quantization value.Both projections are analytical solutions of the corresponding ADMM subproblem.
- Joint procedure: The joint procedure prunes first and then quantizes the remaining nonzero weights because pruning typically exploits more redundancy without accuracy loss than quantization.This ordering avoids the added flexibility and optimization difficulty of deciding simultaneously between zero and a quantization level.
- Convergence: ADMM pruning and quantization for AlexNet require 72 hours and 24 hours, respectively, compared with 173 hours for prior pruning alone.The paper attributes the faster convergence to ADMM's theoretical convergence rate.
- Quantization: Equal-distance quantization uses n bits and M = 2^n levels, with layer-specific interval q_i; zero is excluded because it denotes a pruned weight.Quantization levels are encoded in binary bits for storage and hardware operations, while q_i supports layer-specific scaling.
4 RESULTS AND DISCUSSIONS ON DNN MODEL COMPRESSIONS
ADMM-based compression achieves substantial pruning and joint pruning–quantization gains across representative DNNs, while highlighting the need to account for computation, hardware overhead, and index storage. The framework reports strong accuracy-preserving reductions, including 85× pruning on LeNet-5, 24× on AlexNet, and 1,910× data-representation compression on LeNet-5.
- Weight pruning results: 85× weight reduction on LeNet-5 is achieved without accuracy loss, compared with 12× from prior iterative pruning; 167× is reached at 99.0% accuracy.The reported 85× result is a 7.1× improvement over the cited prior method.
- Weight pruning results: 24× weight reduction on AlexNet and 26× on VGGNet are achieved without accuracy loss, at least twice the state-of-the-art.These results are reported for ImageNet models using the proposed ADMM method.
- Weight pruning results: 7× pruning on ResNet-50 is achieved without accuracy degradation, increasing to 17.4× with accuracy degradation below 1%.The result is reported on ImageNet.
- Joint pruning and quantization: Joint pruning and quantization reaches 167× pruning with 2.78-bit average weights on LeNet-5, yielding 1,910× compression for weight data representation.The compressed LeNet-5 model requires 0.89KB for weight data at 99% accuracy.
- Joint pruning and quantization: AlexNet and VGGNet use 3.7-bit average weight representations and achieve overall weight-data compression ratios close to 200×.The comparison reports substantially higher compression than prior work such as the cited pruning and clustering methods.
- Storage and hardware considerations: Index storage reduces the realizable model-size gain because pruning requires at least one index per weight, with higher pruning ratios increasing index requirements.The paper distinguishes weight-data compression from total model storage and notes that index overhead can exceed the stored weight data.
- Storage and hardware considerations: Model-size-only evaluation favors FC layers, although CONV layers account for 95%–98% of computation in AlexNet and VGGNet and more in ResNet.The paper therefore emphasizes CONV-layer compression when targeting computation reduction and energy efficiency.
- Storage and hardware considerations: Accounting for indices, VGGNet is reduced from 552MB to 8.3MB using 6.9M weights, fitting within a single cited high-end FPGA.The reported size uses the proposed ADMM framework and includes index storage.
5 HARDWARE-AWARE COMPUTATION REDUCTION
The hardware-aware extension of ADMM-NN targets computation-intensive CONV layers while accounting for irregular-sparsity overhead and coordinating FC-layer pruning to preserve accuracy. It uses hardware-specific break-even pruning ratios to select beneficial pruning levels.
- CONV layers account for 98% to 99% of computation in recent VGGNet and ResNet models, making them the primary target for computation and energy reductions.
- Prior work achieved only 2.7× pruning on AlexNet CONV layers, while irregular storage can add index and parallelism overhead.
- The framework moderately prunes FC layers, typically about 3-4×, because CONV-only pruning can cause accuracy degradation.
- The break-even pruning ratio is the minimum layer-specific ratio that avoids hardware performance degradation and is greater than 1 because of irregular-sparsity overhead.
- Break-even ratios are calculated by comparing pruned and unpruned layer delays under the same hardware area and resource constraints.
- The iterative algorithm reduces remaining-weight fractions in proportion to layer computation, restores layers below break-even, and then applies ADMM pruning and quantization.
6 RESULTS AND DISCUSSIONS ON COMPUTATION REDUCTION AND HARDWARE-AWARE OPTIMIZATIONS
Experiments show that ADMM-NN improves computation reduction and synthesized hardware speedup over prior compression methods, while recognizing bottlenecks in the first CONV layer. Greater pruning can yield diminishing hardware-speedup gains.
- 6.1 Computation Reduction Comparisons: The computation study evaluates MAC operations and MACs multiplied by weight bit width as proxies for speed and energy efficiency, respectively.
- 6.1 Computation Reduction Comparisons: ADMM-NN improves computation reduction by nearly 3× on the first metric and 3.6× on the second metric compared with prior work.
- 6.1 Computation Reduction Comparisons: The framework achieves 13× overall weight pruning without accuracy loss because FC layers are pruned alongside the CONV layers.
- 6.2 Synthesized Hardware Speedup Comparisons: Synthesized hardware results show speedup over uncompressed DNNs, whereas baselines suffer speed degradation under irregular pruning overhead.
- 6.2 Synthesized Hardware Speedup Comparisons: The first CONV layer remains unpruned because its attainable ratio is below break-even and it can become a computation bottleneck.
- 6.2 Synthesized Hardware Speedup Comparisons: With 1.5% accuracy loss, further pruning reaches 25.5× overall CONV-layer pruning, but synthesized speedup increases only marginally.
7 CONCLUSION
ADMM-NN combines ADMM-based pruning and quantization with hardware-aware optimization that accounts for computation, energy, and irregular-sparsity overhead. It reports large model-size reductions and improved computation reduction without accuracy loss.
- ADMM-NN jointly applies ADMM-based weight pruning and quantization and adds hardware-aware optimization for efficient implementations.
- 1,910× and 231× reductions in overall model size are achieved on LeNet-5 and AlexNet, respectively, by combining pruning and quantization.
- Without accuracy loss, the framework achieves 3.6× reduction in computation, outperforming prior work.
- Highly promising results are also observed on VGGNet and ResNet models.