Source-linked AI summary
ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgetting
Xiaohan Ding, Tianxiang Hao, Jianchao Tan, Ji Liu, Jungong Han, Yuchen Guo, Guiguang Ding
TL;DR
Channel pruning seeks to reduce CNN width without losing performance, but traditional penalty-based methods couple remembering with forgetting and face a resistance–prunability trade-off. ResRep separates these roles through re-parameterized compactors and specialized updates, then converts them into an equivalent narrower network. It reports 0.00% top-1 accuracy drop for ResNet-50 at a 54.54% pruning ratio.
Problem
Channel pruning must reduce CNN width without performance drops, but traditional penalty-based pruning couples performance learning and channel elimination, creating a resistance–prunability trade-off.
Method
ResRep re-parameterizes CNNs into remembering and forgetting parts, trains them with separate updates, and equivalently converts the resulting structure into narrower layers.
Results
0.00% top-1 accuracy drop is reported on ResNet-50 at a 54.54% pruning ratio.
Takeaways & Limitations
Separating performance maintenance from pruning enables structured channel removal while preserving the original model architecture in a narrower form.
Abstract
from arXiv · showhide
We propose ResRep, a novel method for lossless channel pruning (a.k.a. filter pruning), which slims down a CNN by reducing the width (number of output channels) of convolutional layers. Inspired by the neurobiology research about the independence of remembering and forgetting, we propose to re-parameterize a CNN into the remembering parts and forgetting parts, where the former learn to maintain the performance and the latter learn to prune. Via training with regular SGD on the former but a novel update rule with penalty gradients on the latter, we realize structured sparsity. Then we equivalently merge the remembering and forgetting parts into the original architecture with narrower layers. In this sense, ResRep can be viewed as a successful application of Structural Re-parameterization. Such a methodology distinguishes ResRep from the traditional learning-based pruning paradigm that applies a penalty on parameters to produce sparsity, which may suppress the parameters essential for the remembering. ResRep slims down a standard ResNet-50 with 76.15% accuracy on ImageNet to a narrower one with only 45% FLOPs and no accuracy drop, which is the first to achieve lossless pruning with such a high compression ratio. The code and models are at https://github.com/DingXiaoH/ResRep.
1. Introduction
ResRep addresses the resistance–prunability trade-off in channel pruning by separating performance maintenance from channel elimination. It re-parameterizes CNN layers with compactors, applies specialized updates, and converts the trained model into an equivalent narrower architecture.
- Motivation: Channel pruning reduces convolutional width, FLOPs, and memory footprint but is difficult because CNN representational capacity depends on layer width.Traditional penalty-based pruning can create structured sparsity, yet reducing width without performance drops remains challenging.
- Motivation: Traditional penalty-based pruning couples performance learning and pruning, creating a resistance–prunability trade-off between training damage and insufficient sparsity.Strong penalties improve sparsity but harm training accuracy, whereas weak penalties preserve performance but leave greater pruning-caused damage.
- Method: ResRep decouples remembering and forgetting by applying regular SGD to performance-maintaining parts and pruning-oriented updates to separate forgetting parts.The design is inspired by evidence that remembering and forgetting are independently controlled biological processes.
- Method: ResRep inserts 1×1 convolutional compactors, applies penalty gradients and gradient resetting only to them, then removes near-zero channels and equivalently converts the model into narrower layers.For conv-BN sequences, batch normalization is first fused into the convolution before the final conversion.
- Properties: ResRep preserves the original objective and training settings for convolutional and batch-normalization parts while automatically finding layer widths for a target global FLOP reduction.The method is presented as resistant, highly prunable, and usable without prior knowledge of each layer’s eventual width.
2. Related Work
Related channel-pruning methods either prune and then finetune based on channel-importance measurements or learn structured sparsity through specialized objectives. ResRep differs by using equivalent structural conversion rather than imperfect pruning followed by finetuning.
- Pruning Taxonomy: Unstructured pruning removes individual parameters but commonly lacks hardware speedup, whereas structured pruning removes whole structures and is more hardware-friendly.Channel pruning is especially practical because it reduces model size, computation, and memory footprint.
- Channel-Pruning Families: Pruning-then-finetuning methods identify unimportant channels from trained models, but pruning can cause significant accuracy drops and trap the resulting model in bad local minima.Some methods repeat pruning and finetuning iteratively to remove channels progressively.
- Learning-Based Methods: PCAS trains attention modules to identify channels but performs imperfect pruning, requires finetuning, and discards the modules instead of using an equivalent transformation.Other learning-based methods use penalties, filter similarity, meta-learning, or adversarial learning to induce pruning.
3. ResRep for Lossless Channel Pruning
ResRep re-parameterizes target convolutional layers with compactors, trains the compactors for structured sparsity while preserving the original layers' performance, then converts the model into the original architecture with narrower layers.
- Convolutional Re-parameterization: 1×1 compactors are appended after target convolutional layers, including after batch normalization when present, to provide channels that can be pruned.The training-time re-parameterized model uses compactors with kernel Q, while the deployed model removes them through equivalent conversion.
- Convolutional Re-parameterization: Conv-BN-compactor sequences are equivalently fused into convolutional kernels and biases, yielding a model with the original architecture but narrower layers.The compactor's cross-channel recombination is merged into the convolution kernel, and its bias contribution is merged into the bias.
- Deployment: The converted weights are saved in a narrower model with the original architecture for testing and deployment.Gradient Resetting can make channels extremely close to zero, so thresholds such as 10^-5 or 10^-9 produce no practical difference in the reported setting.
- Gradient Resetting: Traditional penalty training creates a resistance-prunability trade-off because penalties perturb every channel while weaker penalties leave channels insufficiently close to zero.Strong penalties increase sparsity but cause training damage; weak penalties preserve performance but cause greater pruning damage.
- Gradient Resetting: Gradient Resetting applies penalty gradients selectively to masked compactor channels while retaining the objective gradients for the other channels.The method derives performance gradients normally, applies a binary mask, adds penalty gradients, and uses the result for SGD updates.
- Progressive Channel Selection: Compactor channels are selected by ascending norm-based metrics until the target FLOPs reduction or channel-selection limit is reached.After several epochs, compactor channel norms reflect their competence-based importance, and selected channels receive zero masks.
4. Experiments
Experiments evaluate ResRep across ImageNet and CIFAR-10 models, comparing pruning accuracy, layer-width allocation, baselines, variants, and training dynamics. ResRep achieves lossless high-ratio pruning while its components improve accuracy and preserve performance during training.
- Experimental Setup: 76.15% top-1 accuracy is the ResNet-50 base-model result used for fair comparison on ImageNet.The experiments use the official torchvision base model.
- Pruning Results: ResRep outperforms SASL by 0.82%, HRank by 3.32%, and MetaPruning by 1.77% in top-1 accuracy drop comparisons.The comparisons cover ResNet-50 and MobileNet pruning results.
- Pruning Results: ResRep discovers final layer widths without prior structural knowledge, preserving more higher-level channels in ResNet-50 and MobileNet while pruning ResNet-56’s last blocks aggressively.The reported pattern is associated with ImageNet fitting capacity and CIFAR-10 over-fitting.
- Ablation Studies: The 55.4%-reduced uniformly shrunk ResNet-50 reaches 74.39% accuracy, 1.58% below the 56.1%-pruned ResRep model.The uniformly shrunk model is trained from scratch using the same training settings as the base model.
- Ablation Studies: Res and Rep each outperform the baseline in controlled ResNet-56 experiments, and their combination performs better than either component alone.The comparison uses matched training configurations and FLOPs targets.
5. Conclusion
ResRep suggests decomposing learning-based pruning into performance-oriented and pruning-oriented learning, and demonstrates this through Structural Re-parameterization.
- ResRep suggests separating performance-oriented learning from pruning-oriented learning as a promising direction for learning-based pruning.
- Structural Re-parameterization enables ResRep to construct extra structures that can later be converted back into the original architecture.
- This conversion allows ResRep to use custom techniques, including an update rule applied only to the compactors.