Manifold Regularized Dynamic Network Pruning

1. Introduction

• Channel pruning removes redundant filters to make compact networks with lower computational cost and practical hardware speed-up.The introduction motivates such efficiency for deployment on resource-constrained hardware.Source: §1 · p. 1
• Existing dynamic pruning methods select subnetworks independently for each input, overlooking relationships between instances and differences in their complexity.The paper also identifies instance similarity as useful information that existing approaches neglect.Source: §1 · p. 1
• ManiDP uses training-set instance complexity and feature similarity to preserve relationships between inputs in their corresponding subnetworks.It adapts the channel-sparsity penalty to instance complexity and matches similarity between samples in the pruned results.Source: §1 · p. 2, Figure 1 · p. 2
• ManiDP reports higher accuracy and speed-up ratios than state-of-the-art pruning algorithms across several benchmarks.The introduction describes experiments on a series of benchmarks.Source: §1 · p. 2

Static Pruning

• Static pruning removes the same filters for different inputs, and its performance can degrade when pruning rates are large.Prior approaches estimate filter importance or impose structural sparsity to obtain a shared compact network.Source: §2 · p. 2

Dynamic Pruning

• Dynamic pruning selects filter importance from input data and skips unnecessary computation at test time.Prior approaches use collaborative layers, decision maps, or squeeze-excitation modules; irregular skipped regions may require specialized hardware for acceleration.Source: §2 · p. 2
• The paper focuses on training dynamic networks to allocate a suitable subnetwork per instance for a satisfactory accuracy–cost trade-off.This focus complements prior dynamic methods that determine which filters to skip at runtime.Source: §2 · p. 2

Channel Pruning

• Channel pruning removes redundant channels to reduce computational complexity while retaining comparable performance.The conventional formulation balances task loss against channel-wise sparsity on convolution filters.Source: §3 · p. 3

Dynamic Pruning

• Dynamic pruning predicts input-dependent channel saliencies with a controller and thresholds them to mask less-important channels.The controller processes the preceding layer's feature map; lower saliency indicates a less-important channel.Source: §3 · p. 3
• Applying the previous layer's mask lets the network skip calculations for redundant channels and reduce computational complexity.The masked feature map is convolved and passed through ReLU.Source: §3 · p. 3
• The dynamic objective combines task loss with an ℓ1 penalty on channel saliencies, and a larger λ yields sparser, more compact subnetworks.Unlike static pruning, it selects channels separately according to each input instance.Source: §3 · p. 3

4. Manifold Regularized Dynamic Pruning

• The method uses instance complexity and similarity as complementary manifold signals to regularize the relationship between inputs and their subnetworks.Complexity aligns instance difficulty with subnetwork capacity; similarity encourages similar inputs to have similar subnetwork properties.Source: §4 · p. 3, §4.2 · p. 5

4.1. Instance Complexity

• Cross-entropy loss measures instance complexity, while the sparsity of channel saliencies measures subnetwork complexity.A larger loss indicates an instance that is less well fitted; sparser saliencies indicate a more compact subnetwork.Source: §4.1 · pp. 3–4
• The adaptive sparsity penalty is reduced for examples whose loss exceeds threshold C, leaving their subnetworks unsparsified; simpler instances receive a larger penalty.The objective uses learnable β variables to determine whether each subnetwork should be thinned, with the decision depending on loss relative to C.Source: §4.1 · p. 4

4.2. Instance Similarity

• The method aligns input and subnetwork similarity by matching layerwise similarity matrices computed from channel saliencies and intermediate features.The matrices are compared over mini-batches; the final objective combines this consistency loss with adaptive sparsity loss.Source: §4.2 · p. 5
• At inference, a per-layer threshold derived from average channel saliencies skips channels below the cutoff, while input-dependent saliencies yield different pruning rates across instances.The method thereby produces subnetworks with varying computational costs for different inputs.Source: §4.2 · p. 5

5. Experiments

• ImageNet evaluation uses ResNet models of different depths and MobileNetV2, while CIFAR-10 and ImageNet provide the tested classification datasets.The datasets contain 60,000 CIFAR-10 images and 1.28 million ImageNet training images; ImageNet also has 50,000 validation images.Source: §5 · p. 5

5.1. Comparison on ImageNet

• On ImageNet, ManiDP-A reduces ResNet-34 FLOPs by 46.8% with 0.01% performance degradation and achieves better reported cost/error trade-offs than static and dynamic competitors.For ResNet-18, ManiDP-A has 31.12% top-1 error with 51.0% FLOPs reduced, compared with DSA's 31.39% top-1 error and 40.0% FLOPs reduction.Source: §5.1 · p. 6, Table 1 · p. 6
• On ImageNet, ManiDP-A reduces MobileNetV2 FLOPs by 37.2% with 0.38% accuracy loss, versus FBS's 33.6% FLOPs reduction and 0.87% accuracy loss.The authors report a better accuracy–computational-cost trade-off for ManiDP on this lightweight network.Source: Table 2 · p. 6, §5.1 · p. 7
• On ImageNet, measured CPU realistic acceleration is slightly below theoretical acceleration calculated from FLOPs, with I/O, BLAS libraries, and buffer switching among the cited practical factors.The authors note that practical engineering optimization may further reduce these factors' impact.Source: §5.1 · p. 7, Table 3 · p. 7

5.2. Comparison on CIFAR-10

• On CIFAR-10, ManiDP reduces ResNet-56 FLOPs by more than 60% while achieving 6.36% test error, outperforming the cited static and dynamic comparisons.HRank has 6.83% error with 50.0% FLOPs reduction; FBS has 6.48% error with 53.6% FLOPs reduction.Source: §5.2 · p. 7, Table 4 · p. 7

5.3. Ablation Studies

• On ImageNet, fixing the complexity-dependent sparsity coefficient increases pruning error from 0.94% to 1.43%, while disabling similarity loss increases it from 0.98% to 1.43%.The authors attribute the complexity result to mismatching subnet capacity and instance difficulty, and the similarity result to not preserving feature similarity in corresponding sub-networks; using both perspectives yields only a 0.57% error increase.Source: §5.3 · pp. 7–8, Table 5 · p. 7
• Varying sparsity and similarity loss coefficients reveals accuracy trade-offs, while the method works well across λ′ values [0.001,0.01] and γ values around 10.0.Increasing λ′ promotes channel-saliency sparsity but can reduce representation ability; increasing γ improves accuracy unless it is extremely large.Source: §5.3 · p. 8, Figure 3 · p. 8
• At a pruning rate of 0.6%, ResNet-56 retains 93.29% accuracy with 73.87% FLOPs and 67.42% memory reduction, a 0.41% accuracy drop from the original network.The passage describes reduced memory cost for accessing weights because ineffective weights do not participate in inference.Source: §5.3 · p. 8, Figure 4 · p. 8

• ManiDP uses sample complexity and similarity to preserve relationships between instances in their corresponding pruned subnetworks.It adapts the sparsity penalty to instance complexity and matches similarity matrices.Source: §6 · p. 8
• 55.3% FLOPs reduction on ResNet-34 comes with 0.57% top-1 accuracy degradation on ImageNet.Source: §6 · p. 8
• Across several benchmarks, ManiDP achieves better performance with less computational cost than state-of-the-art methods.Source: §6 · p. 8

7.1. Coefficient λ(xi) for different instances

• Complex examples receive a smaller λ(xi)/λ′ and retain more channels, while simple examples usually receive a larger value and become progressively sparser.For simple examples, the sparsity weight can decrease in the final epochs as the subnetwork becomes compact enough to prioritize accuracy.Source: §7.1 · p. 12

7.2. Similarity Matrices

• With similarity loss, feature and channel-saliency similarity matrices become more analogous; without it, they differ substantially.The loss penalizes inconsistency between the matrices to align similarity relationships across the two spaces.Source: §7.2 · p. 12, Figure 7 · p. 12

7.3. Visualization of instances with different complexity

• Subnetwork computational cost increases from the top to the bottom of the ImageNet examples; clear-target images can be handled by compact networks, while vague semantic content calls for larger ones.Source: §7.3 · pp. 12–13, Figure 8 · p. 13