Source-linked AI summary
CondenseNet: An Efficient DenseNet using Learned Group Convolutions
Gao Huang, Shichen Liu, Laurens van der Maaten, Kilian Q. Weinberger
TL;DR
Resource-limited devices need accurate CNNs with lower inference cost. CondenseNet combines dense connectivity with learned group convolutions that prune superfluous feature-reuse connections and convert to standard group convolutions at test time. Experiments report consistent efficiency advantages over alternative architectures, including comparable DenseNet accuracy at one-tenth the computation and a 29% ImageNet top-1 error model using 275 million FLOPs.
Problem
Highly accurate CNNs designed for abundant resources cannot provide real-time inference on low-compute devices.
Method
CondenseNet combines dense connectivity with learned group convolutions that progressively remove unimportant connections and yield regular group-convolution structure for testing.
Results
CondenseNets consistently outperform alternative architectures; compared with DenseNets, they use 1/10 of the computation at comparable accuracy levels.
Takeaways & Limitations
The pruned network can be implemented as standard group convolutions, providing computationally efficient inference in common deep-learning libraries.
Abstract
from arXiv · showhide
Deep neural networks are increasingly used on mobile devices, where computational resources are limited. In this paper we develop CondenseNet, a novel network architecture with unprecedented efficiency. It combines dense connectivity with a novel module called learned group convolution. The dense connectivity facilitates feature re-use in the network, whereas learned group convolutions remove connections between layers for which this feature re-use is superfluous. At test time, our model can be implemented using standard group convolutions, allowing for efficient computation in practice. Our experiments show that CondenseNets are far more efficient than state-of-the-art compact convolutional networks such as MobileNets and ShuffleNets.
1. Introduction
CondenseNet targets real-time CNN inference on resource-limited devices by combining dense feature reuse with learned removal of redundant connections. Experiments report substantially lower computation than comparable architectures while maintaining similar accuracy.
- Motivation: Real-time inference is difficult because highly accurate CNNs were designed for abundant computational resources.The deployment targets include robotics, self-driving cars, and mobile devices.
- Motivation: Efficient CNN research has reduced computation through pruning, quantization, and more efficient architectures.Reported reductions relative to VGG include 5× for ResNets, 10× for DenseNets, and 25× for MobileNets and ShuffleNets.
- Approach: CondenseNet learns which dense connections are redundant, removes less important features per group during training, and produces regular group-convolution connectivity.The incoming feature groups are learned rather than predefined.
- Results: 1/10 computation is sufficient for CondenseNets to reach comparable accuracy levels to DenseNets.This comparison is reported for image-classification experiments.
- Results: 29% top-1 error was achieved on ImageNet by a CondenseNet using 275 million FLOPs, comparable to a MobileNet requiring twice as much compute.FLOPs count multiplication-addition operations throughout the paper.
2. Related Work and Background
The paper builds on model compression, DenseNet feature reuse, and group convolution. CondenseNet combines dense connectivity with learned, regular sparsity to retain useful reuse while enabling efficient group-convolution implementation.
- Related Work: Related approaches improve inference efficiency through weight pruning, quantization, or efficient network architectures.These methods exploit redundant weights or redesign connectivity and computation.
- Related Work: CondenseNet prunes during early training and targets a balance between high sparsity and regular group-convolution structure.This distinguishes it from pruning fully trained networks or using only filter-level pruning.
- Learned Group Convolution: During test time, an index layer rearranges features so learned group convolutions become standard group convolutions.Figure 1 contrasts DenseNet transformations with CondenseNet training-time and test-time transformations.
- DenseNet: DenseNets concatenate feature maps from all preceding layers within a dense block, promoting feature reuse.Each layer uses BN-ReLU transformations, a 1×1 convolution, and a 3×3 convolution to produce k output features.
- Learned Group Convolution: CondenseNet training progressively prunes unimportant filters while requiring filters in the same group to share sparsity patterns.After groupings are fixed, the resulting layer can be implemented with standard group convolution.
- Group Convolution: Group convolution partitions input features into G mutually exclusive groups, reducing computational cost by a factor G.Each group produces its own outputs rather than applying every filter over all input features.
3. CondenseNets
CondenseNet learns structured sparsity in DenseNet connectivity, then deploys the resulting layers as efficient standard group convolutions. It also modifies DenseNet connectivity and growth rates to improve computational efficiency and feature reuse.
- Learned Group Convolution: Learned group convolution automatically selects relevant input features for each filter group, while allowing shared or unused features.This addresses the difficulty of predicting which dense features each later layer needs.
- Learned Group Convolution: The training procedure alternates condensing stages that prune low-magnitude filters with an optimization stage after groupings are fixed.Pruned filters share sparsity patterns within each group, enabling standard group-convolution implementation at test time.
- Learned Group Convolution: Condensation uses group-level sparsity: input-feature columns with small L1-norms are removed, while group-lasso regularization jointly drives columns toward zero.This produces structurally sparse layers whose filters within a group receive the same input-feature pattern.
- Architecture Design: The architecture changes DenseNet by directly connecting layers across feature-map resolutions and doubling growth rates when feature-map size shrinks.The proposed variant therefore generates substantially more features in later dense blocks.
- Architecture Design: Increasing the growth rate with depth strengthens short-range connections and improves computational efficiency, while potentially reducing parameter efficiency.The growth rate is set as k=2^(m−1)k0, with m denoting the dense-block index.
4. Experiments
Experiments on CIFAR-10, CIFAR-100, and ImageNet evaluate CondenseNet against efficient architectures and pruning methods, including ablations of pruning, grouping, condensation, and connectivity stability. CondenseNet achieves comparable accuracy with substantially lower computation, while gradual pruning and learned grouping improve efficiency without large accuracy losses.
- CondenseNet uses approximately 8× fewer parameters and FLOPs than DenseNet-190 at comparable accuracy on CIFAR datasets.
- CondenseNet is about 3× more FLOP-efficient than pruned ResNets or DenseNets, while CondenseNetlight uses half the parameters of the strongest competing baseline at comparable performance.
- 274 million FLOPs yields 29.0% Top-1 error for CondenseNet on ImageNet, comparable to MobileNets and ShuffleNets requiring twice as much compute.
- Ablation Study: Gradual pruning during training outperforms pruning fully trained models, while removing 50% of 1 × 1 convolutional weights causes hardly any accuracy loss.
- Ablation Study: With equal parameter counts, test error tends to decrease as the number of learned-group-convolution groups increases.
- Ablation Study: Across three training runs, filter-group connectivity differs but overall information flow between layers remains similar, with later layers favoring recent features while retaining some early features.
5. Conclusion
CondenseNet combines dense connectivity with learned group convolutions to encourage feature reuse while pruning superfluous connections. The resulting networks can be converted to efficient regular group convolutions and outperform compact alternatives at matched accuracy.
- CondenseNet encourages feature reuse through dense connectivity and prunes filters associated with superfluous reuse using learned group convolutions.
- The pruned network can be converted into regular group convolutions implemented efficiently by most deep-learning libraries.
- CondenseNets outperform MobileNets and ShuffleNets in computational efficiency at the same accuracy level.
- CondenseNet slightly outperforms an architecture discovered by empirically testing tens of thousands of convolutional network architectures while using a much simpler structure.