Source-linked AI summary
Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure
Xiaohan Ding, Guiguang Ding, Yuchen Guo, Jungong Han
TL;DR
CNN redundancy motivates slimming, but pruning can be difficult in deep networks with complicated structures and often requires finetuning. The paper proposes C-SGD to make filters identical, enabling their removal while preserving downstream information, and reports effective pruning on CIFAR-10 and ImageNet alongside evidence that redundancy can help convergence.
Problem
Filter pruning must slim redundant CNNs while handling complicated architectures whose layers may need coordinated pruning, and existing zeroing-based methods require finetuning after removal.
Method
C-SGD moves clustered filters toward a common point in parameter space, producing identical filters and shared redundancy patterns across layers that must be pruned together.
Results
The experiments on CIFAR-10 and ImageNet justify C-SGD-based filter pruning, while redundant C-SGD-trained CNNs outperform normally trained counterparts with equivalent width.
Takeaways & Limitations
Identical-filter pruning can remove filters without performance loss or finetuning while preserving more representational capacity than zeroing-out methods.
Takeaways & Limitations
The method cannot ensure convergence of filters and biases when they participate in different computation flows.
Abstract
from arXiv · showhide
The redundancy is widely recognized in Convolutional Neural Networks (CNNs), which enables to remove unimportant filters from convolutional layers so as to slim the network with acceptable performance drop. Inspired by the linear and combinational properties of convolution, we seek to make some filters increasingly close and eventually identical for network slimming. To this end, we propose Centripetal SGD (C-SGD), a novel optimization method, which can train several filters to collapse into a single point in the parameter hyperspace. When the training is completed, the removal of the identical filters can trim the network with NO performance loss, thus no finetuning is needed. By doing so, we have partly solved an open problem of constrained filter pruning on CNNs with complicated structure, where some layers must be pruned following others. Our experimental results on CIFAR-10 and ImageNet have justified the effectiveness of C-SGD-based filter pruning. Moreover, we have provided empirical evidences for the assumption that the redundancy in deep neural networks helps the convergence of training by showing that a redundant CNN trained using C-SGD outperforms a normally trained counterpart with the equivalent width.
1. Introduction
CNNs are computationally intensive, motivating compression methods such as filter pruning, which reduces FLOPs while producing thinner standard networks. C-SGD instead trains filters to become identical, enabling pruning without finetuning and supporting constrained pruning in complicated architectures.
- CNNs’ increasing width and depth raise memory, power, and FLOP requirements, complicating deployment on resource-limited platforms.
- Filter pruning is universal across CNN architectures and platforms, reduces FLOPs, and produces thinner networks without customized structures or extra operations.
- Existing pruning methods commonly remove unimportant filters or reduce their magnitudes, then rely on finetuning to recover performance.
- C-SGD trains multiple filters toward identical parameters, allowing all but one to be discarded while aggregating corresponding next-layer channels with zero performance loss.
- C-SGD addresses constrained filter pruning by producing shared redundancy patterns across layers that must be pruned together.
- The paper presents theoretical and empirical analyses, benchmark evaluations, and evidence that redundancy can help neural-network convergence.
2. Related Work
Filter pruning removes filters to create narrower convolutional layers and reduce computational cost, while other compression methods target connections, parameter representations, precision, or knowledge transfer. Figure 1 contrasts zeroing filters with centripetal merging, which preserves downstream channel information for pruning.
- Removing filters transforms wide convolutional layers into narrower ones, reducing FLOPs and memory more directly than making parameter tensors merely sparse.
- Other CNN compression approaches decompose tensors, use fewer bits per parameter, transfer knowledge, or accelerate convolution through alternative operations.
- Zeroing a conv1 filter makes its feature map and corresponding conv2 input channels useless, so both the filter and downstream channels are removed.
3. Slimming CNNs via Centripetal SGD
C-SGD restructures convolutional filters into clusters and trains filters within each cluster toward a shared parameter point. This enables exact trimming, coordinated pruning across constrained layers, and reduced network width without requiring finetuning.
- Formulation: C-SGD treats a filter as the convolution, normalization, and scaling parameters associated with one output channel.The filter is represented as a five-tuple containing the relevant kernel slice and batch-normalization or scaling parameters.
- Cluster construction: Filters are divided into clusters whose number equals the desired number of surviving filters, with clustering performed evenly or by k-means.K-means uses flattened filter kernels; experiments report only a minor difference between the two clustering strategies.
- Convergence: χ decreases monotonically and exponentially under a proper learning rate, indicating that intra-cluster filter differences progressively vanish.The method’s behavior depends on centripetal strength: excessively large values merge filters instantly, whereas extremely small values make C-SGD resemble normal SGD for a long time.
- Update rule: C-SGD averages objective-function gradients within each cluster while applying weight decay and a centripetal term that removes differences in initial filter values.The update moves filters toward their cluster center in parameter hyperspace, with centripetal strength ϵ controlling the attraction.
- Update rule: The method addresses divergent gradients caused by filters participating in different computation flows by merging their objective-function gradients.This preserves supervision information while enforcing the convergence condition that clustered filters approach one another.
- Trimming and constrained pruning: Identical filters can be trimmed without damage because convolution is linear and combinational, so no finetuning is needed after removal.For constrained architectures, assigning one layer’s cluster set to another forces both layers to learn the same redundancy pattern.
4. Experiments
Experiments on CIFAR-10 and ImageNet evaluate C-SGD for slimming CNNs, while controlled studies compare redundant training, identical-filter pruning, zeroing out, and alternative pruning methods.
- Benchmark evaluation: C-SGD experiments evaluate filter pruning on CIFAR-10 and ImageNet-1K using well-trained base models and simultaneous training of target layers.Comparisons use absolute and relative error increase to assess accuracy changes across base models.
- Slimming very deep and complicated CNNs: 61% FLOPs reduction on the reported aggressive pruning configuration produced no obvious accuracy drop, while DenseNet-40 achieved 60.05% FLOPs reduction with significantly increased accuracy.The DenseNet-40 result used 5, 8, and 10 incremental convolutional layers across its three stages.
- Benchmark evaluation: C-SGD-7/10, C-SGD-6/10, and C-SGD-5/10 shrink the first and second layers in each ResNet-50 residual block to 70%, 60%, and 50% width, respectively.ImageNet experiments use k-means clustering and the ILSVRC2015 training set.
- Redundant training versus normal training: Comparisons indicate that training a redundant network and transforming it to a narrower one may outperform pruning followed by finetuning.The observation is presented as consistent with prior claims that redundancy helps overcome highly non-convex optimization.
- Redundant training versus normal training: Scratch-trained redundant DenseNet-40 and VGG models outperformed normally trained counterparts with equivalent width, supporting a convergence benefit from redundant filters.The authors attribute retained representational capacity partly to unconstrained growth of corresponding input channels.
- Making filters identical versus zeroing out: C-SGD reduced kernel deviation monotonically and exponentially, whereas group-Lasso pruning still caused around 10% accuracy drop when the residual measure reached around 4 × 10^-4.The controlled ResNet-56 study compares identical-filter formation with zeroing-out filters using matched redundancy levels.
- C-SGD versus other pruning methods: C-SGD achieved higher accuracy and better stability than magnitude-, APoZ-, Taylor-expansion-, and group-Lasso-based pruning in controlled DenseNet-40 experiments.Group-Lasso models received additional training and finetuning, making the comparison biased toward Lasso according to the authors.
5. Conclusion
The paper proposes identical-filter redundancy and C-SGD for network slimming, reporting no need for finetuning, preserved representational capacity, strong benchmark results, and evidence that redundancy can aid convergence.
- Conclusion: The paper proposes training filters to become identical so redundant filters can be removed for network slimming.C-SGD moves multiple filters toward a center in filter-parameter hyperspace.
- Conclusion: Removing identical filters requires no finetuning and preserves more representational capacity than zeroing-out methods, according to the paper’s conclusion.The method partly addresses constrained pruning in very deep and complicated CNNs.
- Conclusion: Experiments achieve state-of-the-art results on several common benchmarks and provide empirical evidence that redundancy can help neural-network training convergence.The authors also identify regularization or training applications for C-SGD beyond pruning.