Source-linked AI summary
Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon
Xin Dong, Shangyu Chen, Sinno Jialin Pan
TL;DR
Large deep networks are costly to deploy, while existing pruning methods often cannot compress well-trained models substantially or require heavy retraining. The paper introduces layer-wise second-order pruning with a performance-drop bound, and reports substantial compression with little or no retraining. It concludes that L-OBS can preserve performance while reducing retraining, although Hessian computation may be costly for very large layers.
Problem
Existing pruning methods often fail to substantially compress well-trained deep networks or require heavy retraining to restore prediction performance.
Method
L-OBS independently prunes each layer using second-order derivatives of its layer-wise error function and bounds overall performance drop through reconstructed layer errors.
Results
L-OBS achieves substantial compression with little performance loss and can reduce or omit retraining, including 11% of original AlexNet size without accuracy loss.
Takeaways & Limitations
The framework preserves prediction performance with a theoretical guarantee while substantially trimming networks and reducing retraining requirements.
Takeaways & Limitations
For models with sizable layers such as ResNet-101, L-OBS may not be economical because Hessian computation is costly.
Abstract
from arXiv · showhide
How to develop slim and accurate deep neural networks has become crucial for real- world applications, especially for those employed in embedded systems. Though previous work along this research line has shown some promising results, most existing methods either fail to significantly compress a well-trained deep network or require a heavy retraining process for the pruned deep network to re-boost its prediction performance. In this paper, we propose a new layer-wise pruning method for deep neural networks. In our proposed method, parameters of each individual layer are pruned independently based on second order derivatives of a layer-wise error function with respect to the corresponding parameters. We prove that the final prediction performance drop after pruning is bounded by a linear combination of the reconstructed errors caused at each layer. Therefore, there is a guarantee that one only needs to perform a light retraining process on the pruned network to resume its original prediction performance. We conduct extensive experiments on benchmark datasets to demonstrate the effectiveness of our pruning method compared with several state-of-the-art baseline methods.
1 Introduction
Deep networks are increasingly large, making them costly and motivating pruning, but heuristic criteria often cannot preserve prediction performance without heavy retraining. The paper proposes layer-wise pruning with second-order information, error guarantees, and light retraining.
- VGG-16 has 133M parameters versus less than 1M in LeNet-5, increasing deep networks’ memory and computational costs.
- Optimal pruning is NP-hard because the pruning search space grows exponentially with parameter size.
- Heuristic pruning criteria provide no guarantee that prediction performance will survive pruning, so trimmed networks usually require time-consuming retraining.
- Net-Trim bounds overall performance drop through layer-wise reconstructed errors but achieves lower compression because it uses ℓ1-norm sparsity.
- The proposed method prunes each layer using second-order derivatives, bounds overall performance loss by layer-wise reconstructed errors, and requires only light retraining.
- Experiments compare the method extensively with several state-of-the-art approaches to verify its effectiveness.
2 Related Works and Preliminary
Prior pruning methods use magnitude or sparsity-based criteria, but they can damage accuracy, limit compression, or require extensive retraining. OBS offers better pruning through the full Hessian but becomes impractical for deep networks because of its computational cost.
- Classical pruning methods include OBD and OBS, which use second-order derivatives to estimate each parameter’s error increase when pruned.
- OBD simplifies computation with a diagonal Hessian, while OBS uses the full Hessian for better performance at substantially higher computational cost.
- Magnitude-based pruning can remove low-magnitude parameters that are necessary for low error, causing large immediate performance drops and poor pre-retraining robustness.
- Sparsity- or low-rank-based methods may achieve smaller compression ratios or require whole-network retraining to prevent accumulated errors.
- OBS sets one parameter to zero while minimizing the resulting error change, using a constrained optimization problem solved through Lagrange multipliers.
- Computing and storing the full non-diagonal Hessian and its inverse makes OBS impractical for deep models with very many parameters.
3 Layer-wise Optimal Brain Surgeon
L-OBS prunes each layer independently using second-order sensitivity estimates, controls the resulting layer-wise errors, and bounds their propagation to the network output.
- Layer-wise error: L-OBS defines layer-wise error as the mean square difference between pruned and original layer outputs over the training data.The method fixes the layer input to match the well-trained network when pruning.
- Sensitivity-based pruning: Pruning a parameter is approximated through a second-order Taylor expansion of the layer error, rather than evaluating every candidate on the full training set.Directly computing each parameter's reconstructed error would require an expensive pass over all training data.
- Sensitivity-based pruning: The method selects the parameter with the smallest sensitivity, where sensitivity uses the inverse Hessian and captures training-data distribution beyond parameter magnitude.After pruning, the parameter vector is updated by the optimal perturbation derived from the constrained minimization.
- Error propagation: The final network-output error is bounded by a weighted sum of layer-wise errors, with each layer's error scaled by the Frobenius norms of subsequent layers.This bound is linear for feed-forward networks, while computing inverse Hessians can remain costly for very large layers such as those in ResNet-101.
- Computational considerations: The inverse Hessian computation can be reduced by exploiting layer structure, and the fully connected procedure can be extended to convolutional layers with slight modifications.The paper notes that Hessian computation may still be uneconomical for sizable layers.
- Computational considerations: For each layer, exact layer-wise Hessians remain available because they depend only on the corresponding layer inputs, even after earlier pruning operations.The practical control variable is the layer-wise error, which may exhibit a pruning inflection point as more parameters are removed.
4 Experiments
Experiments evaluate L-OBS across multiple architectures and datasets using compression ratio, pre-retraining error, and retraining iterations. The results show substantial compression with preserved accuracy and lighter retraining than comparison methods.
- Experimental setup: L-OBS is evaluated on LeNet, CIFAR-Net2, AlexNet, VGG-16, and ResNet-50 using compression ratio, pre-retraining error, and retraining iterations.The comparisons include random pruning, OBD, LWC, DNS, and Net-Trim.
- LeNet-300-100: 7% overall compression was achieved on LeNet-300-100, slightly better than LWC’s 8%, while L-OBS retained substantially better pre-retraining performance and required lighter retraining.LWC’s performance was described as almost ruined by pruning at comparable compression ratios.
- Retraining efficiency: Around 1:1000 fewer retraining iterations were required by L-OBS than competing methods; iterative pruning matched DNS’s pruning ratio using 643 iterations on LeNet-300-100 and 841 on LeNet-5.The iterative variant alternates pruning and light retraining to obtain higher compression ratios.
- AlexNet and VGG-16: 11% compression ratio was achieved on AlexNet without accuracy loss, using 18.1K retraining iterations after 2.9 hours of Hessian computation and 3.1 hours of retraining.The reported compression ratio is the preserved-parameter ratio relative to the original model size.
- AlexNet and VGG-16: 7.5% overall compression was achieved on VGG-16 without accuracy loss after two pruning-and-retraining iterations, while AlexNet reached 16% of original size with 20.98% pruned top-5 error without retraining.These results illustrate a trade-off between compression ratio and pruning, including retraining, cost.
- ResNet-50: Above 85% ResNet-50 accuracy was maintained when the compression ratio was at least 45%.All layers shared the same compression ratio in these experiments.
5 Conclusion
L-OBS prunes parameters using second-order information from layer-wise error functions while theoretically bounding overall error through reconstructed layer-wise errors. It substantially trims networks with tiny performance loss and can reduce or omit retraining.
- L-OBS prunes parameters using second-order derivatives of layer-wise error functions and provides a theoretical guarantee on overall error through reconstructed errors.
- L-OBS can prune considerable numbers of parameters with a tiny performance drop, reducing or even omitting retraining.
- Compared with previous methods, L-OBS identifies and preserves the network’s important parameters during pruning.
Proof of Theorem 3.2
The proof establishes the overall error bound inductively across layers. It compares accumulated pruned outputs with layer-wise pruned outputs and extends the bound using ReLU properties, the triangle inequality, and layer-wise error definitions.
- Theorem 3.2 is proved by induction, beginning with layer 1 and extending the result from layer l to layer l+1.
- The proof distinguishes accumulated pruned output from layer-wise pruned output, whose input remains fixed to the original network’s input at that layer.
- For ReLU activations, inequalities for individual outputs are combined to obtain a matrix-form bound.
- The layer-wise error definition uses a normalized Frobenius norm between original and accumulated pruned outputs.
- The induction completes the proof for all layers, with Theorem 3.2 obtained as the special case l=L.
Redundancy of Networks
Experiments on LeNet and related networks show substantial parameter redundancy: L-OBS preserves accuracy far into pruning, while sensitivity scores reveal that most parameters have little effect. Retraining can then recover performance after aggressive compression.
- LeNet-300-100 has 267K learnable parameters, while LeNet-5 has 431K and CIFAR-Net uses five convolutional or fully connected layers.
- About 92% of parameters have sensitivity below 0.001, matching the pruning inflection point and indicating a heavy-tailed sensitivity distribution.
- Random pruning preserves prediction accuracy only below a 30% pruning ratio, despite the network’s high redundancy.
- L-OBS resumes LeNet-5’s original performance after 740 iterations at a 7% compression ratio following pruning across layers.
How To Set Tolerable Error Threshold
The accumulated output error is bounded by a weighted sum of layer-wise errors, but it is not directly equivalent to final accuracy. Users can incrementally raise the error threshold to trade compression against performance.
- The accumulated error of the ultimate network output is bounded by the weighted sum of layer-wise reconstructed errors.
- The output error measures the difference before and after pruning but is not strictly inversely proportional to final accuracy.
- Increasing the tolerable error threshold incrementally enables monitoring of model performance while pruning more parameters.
- The threshold controls a practical trade-off between compression ratio and performance drop.
Iterative Layer-wise OBS
Iterative L-OBS alternates pruning with light retraining to progressively compress pretrained models. It achieves very low preserved-parameter ratios across LeNet-300-100, LeNet-5, and VGG-16.
- Iterative workflow: The two-stage workflow alternates pruning and light retraining to progressively reduce model size while rebooting performance.The sequence is pre-training, pruning, retraining, repeated pruning, and light retraining.
- Results: 1.5% compression ratio is achieved on LeNet-300-100 with two-stage iterative L-OBS.CR denotes the ratio of preserved parameters to original parameters, so lower values indicate stronger compression.
- Results: 0.9% compression ratio is achieved on LeNet-5 with two-stage iterative L-OBS.The reported CR measures preserved parameters relative to the original model.
- Results: 7.5% compression ratio is achieved on VGG-16 with two-stage iterative L-OBS.The reported CR measures preserved parameters relative to the original model.