CondenseNet: An Efficient DenseNet using Learned Group Convolutions

• Because leading CNNs were designed for abundant compute and cannot support real-time inference on low-compute devices, mobile deployment calls for architectures that are compact at test time.The typical setup trains on multi-GPU machines and deploys on resource-limited devices, so fast training parallelization is also desirable.Source: §1 · p. 1
• Prior efficiency efforts prune connections [9] [29] [32], quantize weights [4] [21], or change architectures [5] [12] [16] [19] [22] [47], with ResNets [12] and DenseNets [19] reducing VGG [38] computation 5× and 10×, respectively.MobileNets [16] and ShuffleNets [47] reduce computation by 25× relative to VGG [38].Source: §1 · p. 1
• CondenseNet learns to remove less-important incoming-feature connections per group during training, yielding regular sparse connectivity that can be implemented as group convolutions without predefined feature groups.Dense connectivity reuses features, but can retain redundancies when early features are not needed later.Source: §1 · p. 1
• At comparable accuracy, CondenseNets use 1/10 DenseNet computation; on ImageNet, 275 million FLOPs yields 29% top-1 error, comparable to a MobileNet requiring twice the compute.Throughout the paper, FLOPs means the number of multiplication-addition operations.Source: §1 · p. 1

2.1. Related Work

• Fine-grained pruning can achieve high sparsity but requires many indices and special accelerators, whereas filter-level pruning is less sparse but produces more regular networks.Prior compression work also uses weight quantization [21] alongside pruning [14] [29] [32].Source: §2.1 · p. 2
• CondenseNet prunes early in training and combines greater sparsity than filter pruning with regular group convolutions; MobileNet [16], ShuffleNet [47], and NAS [49] instead use depth-wise separable convolutions.The section notes that depth-wise separable convolutions are not yet efficiently implemented in most deep-learning platforms; knowledge distillation [15] and dynamic inference [7] [8] [17] are not explored.Source: §2.1 · p. 2

2.2. DenseNet

• In DenseNets [19], each layer concatenates all earlier feature maps in its dense block and produces k features, the network’s growth rate.A layer uses BN-ReLU, a 1×1 convolution to reduce channels, then another BN-ReLU and a 3×3 convolution to produce k outputs.Source: §2.2 · p. 2

2.3. Group Convolution

• Group convolution partitions R input features into G mutually exclusive groups, reducing computational cost by a factor of G compared with standard convolution.This sparsely connected convolution was first used in AlexNet and later popularized in ResNeXt [43].Source: §2.3 · pp. 2–3, Figure 2 · p. 2

3. CondenseNets

• Naive 1×1 group convolution sharply reduces DenseNet accuracy because fixed disjoint groups impede reuse of the ordered, diverse concatenated features.Randomly permuting inputs lessens the accuracy loss but does not eliminate it.Source: §3 · p. 3
• Learned group convolution lets each filter group select relevant inputs, share features with other groups, or ignore them while preserving their possible use in later layers.Unlike fixed group convolution, learned groupings adapt during training to the feature reuse patterns of dense layers.Source: §3 · p. 3

3.1. Learned Group Convolution

• Training prunes low-importance inputs under group-level sparsity, progressively retaining 1/C of weights per group before optimizing the fixed groupings.Condensing occupies the first half of training; group-lasso regularization encourages filters in each group to use the same input subset.Source: §3.1 · p. 4
• At test time, an index layer rearranges selected features so learned 1×1 convolutions become standard group convolutions supported by efficient deep-learning libraries.The training procedure uses a cosine-shaped learning-rate schedule.Source: §3.1 · pp. 3–4

3.2. Architecture Design

• Increasing growth rates with depth shifts a larger proportion of features and parameters to later layers, improving computational efficiency but potentially reducing parameter efficiency.The growth rate doubles when feature-map resolution shrinks.Source: §3.2 · pp. 4–5, Figure 5 · p. 5
• Fully dense connectivity links inputs to all subsequent layers, including across dense blocks, using average pooling to downsample higher-resolution inputs.This extends feature reuse beyond the original DenseNet’s within-block connections.Source: §3.2 · p. 5, Figure 5 · p. 5

4.1. Results on CIFAR

• On CIFAR-10, learned group convolution improves FLOP efficiency, increasing growth rate yields further efficiency, and full dense connectivity does not significantly improve efficiency.CondenseNetlight achieves comparable accuracy to DenseNet with half the FLOPs; full connectivity is retained because it substantially improves ImageNet results.Source: Figure 6 · p. 5, §4.1 · p. 6
• On CIFAR, CondenseNet uses approximately 8× fewer parameters and FLOPs than DenseNet-190 at comparable accuracy, while CondenseNetlight performs on par with NASNet-A.Both CondenseNet variants use only 1×1 and 3×3 convolutions, not depth-wise separable convolutions.Source: Table 1 · p. 6, §4.1 · p. 6
• CondenseNet is about 3× more FLOPs-efficient than ResNets or DenseNets pruned by [32]; CondenseNetlight uses half the parameters of the competitive 40-layer DenseNet [32] at comparable performance.The comparison covers state-of-the-art filter-level pruning methods.Source: Table 2 · p. 6, §4.1 · p. 6

4.2. Results on ImageNet

• At 274 million FLOPs, CondenseNet achieves 29.0% Top-1 error, comparable to MobileNets and ShuffleNets at twice the compute; at 529 million FLOPs, it lowers Top-1 error by 3% versus comparable-size models.A 274-million-FLOP CondenseNet also infers images 2× faster than MobileNet without sacrificing accuracy.Source: Table 4 · p. 7, §4.2 · p. 7

4.3. Ablation Study

• Gradual pruning beats post-training pruning on CIFAR-10, halves training from 600 to 300 epochs, and removing 50% of 1×1-convolution weights incurs hardly any accuracy loss.At equal parameter counts, test error tends to decrease as group count rises; condensation factors 2, 4, and 8 have comparable classification error as a function of FLOPs despite removing progressively more weights.Source: §4.3 · pp. 7–8, Figure 7 · p. 8
• Across three independently trained CIFAR-10 models, overall connectivity between layers is similar despite filter-group differences; later layers favor recent features but also use some early features.The models used different random seeds.Source: Figure 8 · p. 8, §4.3 · p. 8

• At the same accuracy, CondenseNets are more computationally efficient than MobileNets and ShuffleNets, and slightly outperform a simpler architecture found by searching tens of thousands of designs.The searched architecture had a more complex structure.Source: §5 · p. 8
• CondenseNet encourages feature reuse through dense connectivity and prunes filters associated with superfluous reuse using learned group convolutions.For inference, the pruned network can be converted to regular group convolutions, which are efficiently implemented in most deep-learning libraries.Source: §5 · p. 8
• The pruning method is simple to implement and adds only limited computational cost during training.Source: §5 · p. 8