PaperlayerView exact revision on arXiv

Source-linked AI summary

Discrimination-aware Network Pruning for Deep Model Compression

Jing Liu, Bohan Zhuang, Zhuangwei Zhuang, Yong Guo, Junzhou Huang, Jinhui Zhu, Mingkui Tan

arXiv:2001.01050v2cs.CV

TL;DR

Network pruning must reduce redundant channels and kernels while preserving discriminative power, but existing training-from-scratch and reconstruction-based approaches have important limitations. The paper proposes discrimination-aware channel and kernel pruning with adaptive stopping, and reports effectiveness across classification, face recognition, and mobile inference.

  • Problem

    Existing pruning either trains from scratch with difficult optimization or minimizes reconstruction error while overlooking channels’ discriminative power.

  • Method

    DCP and DKP introduce discrimination-aware losses and select channels or kernels using discriminative loss together with feature-map reconstruction error, with adaptive stopping to limit selection.

  • Results

    Extensive experiments on image classification and face recognition demonstrate effectiveness; with 30% channel reduction, ResNet-50 improves baseline Top-1 accuracy by 0.36% on ILSVRC-12.

  • Takeaways & Limitations

    The methods provide a discrimination-aware approach to compressing deep networks at both channel and kernel levels across multiple architectures and tasks.

  • Takeaways & Limitations

    The layer-wise selection strategy may produce suboptimal performance because it considers only a single layer at each step.

Abstract

from arXiv · show

We study network pruning which aims to remove redundant channels/kernels and hence speed up the inference of deep networks. Existing pruning methods either train from scratch with sparsity constraints or minimize the reconstruction error between the feature maps of the pre-trained models and the compressed ones. Both strategies suffer from some limitations: the former kind is computationally expensive and difficult to converge, while the latter kind optimizes the reconstruction error but ignores the discriminative power of channels. In this paper, we propose a simple-yet-effective method called discrimination-aware channel pruning (DCP) to choose the channels that actually contribute to the discriminative power. Note that a channel often consists of a set of kernels. Besides the redundancy in channels, some kernels in a channel may also be redundant and fail to contribute to the discriminative power of the network, resulting in kernel level redundancy. To solve this, we propose a discrimination-aware kernel pruning (DKP) method to further compress deep networks by removing redundant kernels. To prevent DCP/DKP from selecting redundant channels/kernels, we propose a new adaptive stopping condition, which helps to automatically determine the number of selected channels/kernels and often results in more compact models with better performance. Extensive experiments on both image classification and face recognition demonstrate the effectiveness of our methods. For example, on ILSVRC-12, the resultant ResNet-50 model with 30% reduction of channels even outperforms the baseline model by 0.36% in terms of Top-1 accuracy. The pruned MobileNetV1 and MobileNetV2 achieve 1.93x and 1.42x inference acceleration on a mobile device, respectively, with negligible performance degradation. The source code and the pre-trained models are available at https://github.com/SCUT-AILab/DCP.

1 INTRODUCTION

The paper targets redundant channels and kernels in deep networks, where existing pruning methods either train difficultly from scratch or optimize reconstruction while overlooking discriminative power. It introduces DCP/DKP, adaptive stopping, and reports improvements across architectures and deployment settings.

  • Deep networks’ large model size and computational cost hinder deployment on constrained devices, motivating compression methods that reduce redundancy without significant performance degradation.
  • Existing pruning methods either learn sparsity from scratch, which is difficult for deep networks, or minimize feature-map reconstruction error, which can select nondiscriminative channels.
  • DCP adds discrimination-aware losses and selects channels using both discriminative loss and feature-map reconstruction error.
  • DKP further removes redundant kernels because selected channels may contain kernels that do not contribute to the network’s discriminative power.
  • An adaptive stopping condition limits selected channels or kernels to a predefined per-layer maximum, preventing excessive selection during greedy optimization.
  • 0.36%: DCP improves ResNet-50’s baseline Top-1 accuracy on ILSVRC-12 after pruning 30% of channels.
  • The work also extends evaluation to mobile deployment, compact MobileNet models, face recognition, kernel pruning, and computationally reduced training procedures.

2 RELATED WORK

The related work covers quantization, sparse or low-rank compression, and pruning, while motivating discrimination-aware selection beyond reconstruction-based criteria. The paper’s approach combines discrimination-aware losses with reconstruction considerations to select informative structures.

  • Quantization represents weights or activations with low precision, enabling bitwise operations but potentially causing training difficulty and accuracy decreases.
  • Sparse and low-rank compression reduce storage or computation, but sparse methods may require specialized operations and low-rank methods seek compact representations.
  • DCP uses discrimination-aware losses alongside reconstruction loss to identify channels or kernels that contribute to network discriminative power.
  • The method updates the model with additional losses before performing staged, layer-wise channel pruning.
  • Prior pruning work includes reconstruction optimization, dynamic filter recovery, and automatic pruning-rate selection through reinforcement learning, greedy methods, or evolutionary search.

3 PRELIMINARY

The preliminary formulation represents convolutional layers by channel and kernel tensors, then uses a channel-count constraint to remove redundant channels and reduce computation.

  • Layer representation: A convolutional layer contains n×c kernels, with each kernel connecting one input channel to one output filter.The kernel W_j,k has spatial dimensions h_f × z_f.
  • Layer representation: The output feature map is obtained by convolving each input-channel feature map with the corresponding kernel and combining the results.The convolution operator is denoted by ∗.
  • Channel pruning formulation: Channel pruning introduces a variant of the ℓ2,0-norm that counts channels whose kernel-group Frobenius norms are nonzero.The indicator Ω is one for a nonzero group norm and zero otherwise.
  • Channel pruning formulation: The desired number of retained channels can be specified directly or derived from a pruning rate η as κ_l^c = ⌈(1 − η)c⌉.Here c is the original number of channels at layer l.
  • Channel pruning formulation: Removing channels avoids their associated computation, producing models with fewer parameters and lower computational costs.The reduction applies relative to the original models.

4 PROPOSED METHOD

The proposed method addresses reconstruction-based channel selection by combining discrimination-aware losses with feature-map reconstruction, then optimizing channel pruning from a fine-tuned model.

  • Motivation: Reconstruction-only pruning can select channels that reproduce feature maps but are irrelevant to the network’s discriminative power, especially in deeper networks.This motivates selecting channels using more than reconstruction error.
  • Discrimination-aware losses: DCP introduces discrimination-aware losses at intermediate layers so informative channels are associated with discriminative power throughout the network.The losses are inserted at selected intermediate layers rather than necessarily every layer.
  • Discrimination-aware losses: The intermediate loss uses layer outputs followed by batch normalization, ReLU, and average pooling before forming a classification-oriented loss.These operations make the loss computation feasible and accelerate convergence.
  • Optimization: DCP fine-tunes a pre-trained model with intermediate and final losses, then selects channels using both discrimination-aware loss and reconstruction error.The joint objective balances the two terms with λ.
  • Algorithm: Algorithm 1 initializes from a pre-trained model, inserts intermediate losses, fine-tunes the model, and prunes channels layer by layer.The procedure outputs a pruned model after processing the designated layer groups.
  • Optimization: The channel-pruning optimization imposes an ℓ2,0-norm constraint, with sparsity either set by a pruning rate or adjusted by adaptive stopping conditions.The associated joint loss is stated to be convex with respect to one layer’s parameters.

4.4 Discrimination-aware channel pruning

DCP fine-tunes intermediate discrimination-aware losses before performing staged channel pruning, using those losses together with reconstruction objectives during selection.

  • Discrimination-aware channel pruning: DCP inserts additional losses into intermediate layers, learns their parameters, and fine-tunes the network jointly with the final loss.This improves the discriminative power of intermediate layers before pruning.
  • Discrimination-aware channel pruning: After fine-tuning, DCP initializes pruning from the fine-tuned model and performs channel pruning in P + 1 stages.Each stage handles its current layer group independently.
  • Discrimination-aware channel pruning: The method is discrimination-aware because additional and final losses are used during fine-tuning, while additional losses also guide channel selection.Selection is performed from shallower to deeper layers.
  • Channel selection: Algorithm 2 takes training data and pruning parameters as input and returns a selected channel index set together with model parameters.The algorithm iteratively updates the selected set and parameters.

4.5 Greedy algorithm for channel selection

The channel-selection procedure uses a greedy strategy: it ranks unselected channels by gradient magnitude, adds a batch, and optimizes the resulting subproblem.

  • Greedy selection: Because the ℓ2,0-norm makes the direct optimization non-convex, the method solves channel selection with a greedy algorithm.The greedy formulation avoids directly optimizing the non-convex constrained problem.
  • Greedy selection: The procedure begins with all channels removed, computes loss gradients, and selects the B channels with the largest Frobenius gradient norms.These channels are treated as contributing most to decreasing the loss at that iteration.
  • Greedy selection: Selected channel batches are accumulated into A_t, and the optimization explicitly excludes previously selected channels to prevent overlap.This ensures distinct channel batches across iterations.
  • Subproblem optimization: After each batch is selected, the method optimizes the parameters associated with the selected channels while holding the complementary subset defined by A_t.The update uses stochastic gradient descent with learning rate γ.
  • Subproblem optimization: The selected parameters are warm-started from the fine-tuned model, allowing fast optimization with randomly sampled training images.The learning rate must be adjusted carefully for an accurate solution.

4.6 Stopping conditions

The pruning process uses adaptive stopping conditions to determine how many channels to select. A second condition additionally limits selected channels when loss decreases too slowly.

  • Stopping conditions: Stopping condition 1 uses the monotonic decrease of the convex objective to terminate greedy channel selection.The loss L(W_t) decreases with iteration index t.
  • Stopping conditions: Stopping condition 2 adds a per-layer minimum pruning-rate constraint to prevent selecting too many channels.It addresses cases where many informative channels cause the objective to decrease slowly.
  • Stopping conditions: The selected-channel count ||W_t||_2,0 is automatically determined when the stopping condition is reached.Stopping condition 2 can terminate pruning earlier and reduce computational costs.
  • Stopping conditions: Stopping condition 2 is built on stopping condition 1, which often works alone in practice.The paper compares alternative stopping conditions in Section 7.6.

4.7 Techniques for efficient implementations

The implementation reduces DCP’s training overhead through single-round fine-tuning and reuse of unchanged intermediate feature maps. These techniques avoid repeated computation across pruning rates and selection iterations.

  • Single round fine-tuning: Single-round fine-tuning reduces the computational cost of channel pruning.The network is fine-tuned once, and the resulting model can be stored for reuse across pruning rates.
  • Single round fine-tuning: Fine-tuning can be skipped for networks sharing an architecture but using different pruning rates.The same fine-tuned model is reused for those pruning configurations.
  • Feature reusing: Feature reusing avoids recomputing unchanged input feature maps during kernel or channel selection.The feature maps are stored after their first computation and reused in later iterations.
  • Feature reusing: Repeated feature-map computation previously incurred high computational overhead during pruning.The optimization is possible because channel selection does not change these input feature maps.

5 DISCRIMINATION-AWARE KERNEL PRUNING

DKP addresses kernel-level redundancy left by channel pruning by selecting informative kernels within channels. It uses discrimination-aware losses, kernel selection, and an ℓ2,0-norm constraint for finer compression.

  • Motivation: DCP may retain redundant kernels because it assumes all kernels within a channel are equally important.Pruning an entire channel can also discard informative kernels associated with that channel.
  • DKP method: DKP removes redundant kernels to further compress deep networks.The method targets kernel-level redundancy that channel pruning cannot resolve.
  • DKP method: Kernel removal reduces the computational cost associated with the retained kernels.The desired number of kernels is specified for each layer.
  • DKP method: DKP fine-tunes a pre-trained model with additional intermediate-layer losses and the final loss before layer-wise kernel selection.The added losses are intended to improve the discriminative power of intermediate layers.
  • DKP method: DKP formulates kernel pruning with a variant of the ℓ2,0-norm constraint on the weights.This parallels the constraint used for discrimination-aware channel pruning.
  • DKP method: DKP can be applied alone or after DCP, enabling finer compression by selecting informative kernels related to each channel.The sequential configuration is denoted DCP+DKP.

6 EXPERIMENTS

Experiments evaluate DCP and DKP across image classification and face recognition, including compact architectures, adaptive stopping, and mobile deployment. The results generally show improved accuracy or comparable performance alongside substantial reductions in parameters, FLOPs, or inference time.

  • Experimental setup: DCP is evaluated on ResNet, MobileNetV1, and MobileNetV2, while DKP is evaluated on ResNet and VGGNet across image classification and face recognition.The experiments include CIFAR-10, ILSVRC-12, and face-recognition benchmarks.
  • CIFAR-10 classification: DCP improves VGGNet over WM+ by 0.41% on testing accuracy and over Random-DCP by 0.28% in error.These comparisons support the proposed channel-selection strategy and the use of additional losses.
  • CIFAR-10 classification: Adapt-DCP improves VGGNet over DCP while reducing parameters by 91.69% and FLOPs by 69.81%.The Adapt-DCP model lowers testing error by 0.26% compared with DCP.
  • Mobile deployment: Pruned MobileNetV1 achieves nearly 2× mobile acceleration, while pruned MobileNetV2 requires 44.25ms per execution.The evaluations use a mobile CPU without GPU acceleration.
  • Face recognition: Face-recognition pruning preserves comparable performance, including MobileFaceNet with 0.79M parameters and 28.71ms inference time.Pruned LResNet34E-IR remains comparable to its pre-trained model even at a 50% pruning rate.
  • Kernel pruning: DKP Only outperforms DCP for ResNet-18 by 0.18% in Top-1 accuracy with 47.12% parameter and 46.56% FLOPs reductions.For VGGNet, DKP Only with a 90.41% parameter reduction lowers Top-1 error by 0.46% compared with DCP.

7 ABLATION STUDIES

The ablations examine pruning rates, loss weighting, efficient training, stopping conditions, feature maps, and additional losses. Results show trade-offs between compression cost and performance, while discrimination-aware selection and adaptive stopping identify informative channels more effectively.

  • 7.1 Performance with different pruning rates: 0.36% and 0.11% improvements in Top-1 and Top-5 accuracy, respectively, are achieved by pruning 30% of ResNet-50 channels on ILSVRC-12.At a 50% pruning rate, Top-1 error increases 1.02% for ResNet-50 versus 2.28% for ResNet-18.
  • 7.1 Performance with different pruning rates: Increasing the pruning rate worsens pruned-model performance, while larger models such as ResNet-50 degrade less than ResNet-18 at the same rate.The reported comparison attributes this difference to greater redundancy in ResNet-50.
  • 7.2 Effect of the trade-off hyperparameter λ: Performance first improves and then degrades as λ increases, indicating that both reconstruction error and cross-entropy loss contribute to channel selection.A larger λ places more emphasis on reconstruction error, while λ = 1.0 is used by default for simplicity despite not always being optimal.
  • 7.3 Efficient training strategies: Single-round fine-tuning and feature reusing substantially reduce pruning time while maintaining comparable performance to their respective alternatives.The feature-reusing experiment reports comparable performance with greatly reduced channel-pruning time.
  • 7.4 Effect of B: Larger B lowers channel-pruning time but degrades performance; with B = 4, ResNet-56 computational cost decreases by 4.85×.B denotes the number of channels selected at each iteration, so larger B requires fewer iterations.
  • 7.5 Effect of the tolerance ϵ: Smaller ϵ selects more channels and improves pruned-model performance, demonstrating that stopping conditions can automatically determine the pruning rate.The experiment evaluates VGGNet on CIFAR-10 with ϵ ∈ {1 × 10^-4, 3 × 10^-4, 5 × 10^-4}.
  • 7.6 Comparing stopping conditions: Stopping condition 2 achieves comparable performance to condition 1 with significantly lower computational costs, including 69.81% FLOPs reduction for VGGNet.The comparison covers Adapt-DCP applied to VGGNet and ResNet-56.
  • 7.7 Feature-map visualization: DCP removes weakly activated, redundant, or nearly identical channels, whose feature maps are less informative than those of selected channels.One example removes the 49-th channel because it is similar to the 45-th channel.

8 CONCLUSION

The paper concludes that DCP and DKP compress deep networks by selecting discriminative channels and kernels through layer-wise optimization. It reports effectiveness across image classification and face recognition, while identifying layer-wise selection as a direction for improvement.

  • 8 CONCLUSION: DCP adds discrimination-aware losses and performs layer-wise channel selection through a sparsity-induced optimization problem solved by a greedy algorithm.The paper also describes techniques that accelerate channel pruning.
  • 8 CONCLUSION: DKP extends the approach from channels to kernels, and extensive results on image classification and face recognition demonstrate the methods’ effectiveness.The conclusion presents DKP as a further model-compression method based on DCP.
  • 8 CONCLUSION: Layer-wise channel/kernel selection may be suboptimal because each step considers only a single layer.The authors propose considering multiple layers or blocks at each step as future work.
Loading 2001.01050v2…