PaperlayerView exact revision on arXiv

Source-linked AI summary

Discrimination-aware Channel Pruning for Deep Neural Networks

Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, Jinhui Zhu

arXiv:1810.11809v3cs.CV

TL;DR

Existing channel-pruning strategies either train with sparsity constraints or optimize reconstruction error, leaving discriminative channel power insufficiently addressed. DCP adds intermediate discrimination-aware losses, combines them with reconstruction error, and uses iterative greedy selection and optimization. On ILSVRC-12, pruning 30% of ResNet-50 channels improves the original model’s top-1 accuracy by 0.39%.

  • Problem

    Existing pruning methods are computationally difficult to train from scratch or can retain channels that preserve reconstruction but lack discriminative power.

  • Method

    DCP adds discrimination-aware losses to intermediate layers, selects discriminative channels using reconstruction and discrimination objectives, and optimizes selection iteratively with a greedy algorithm.

  • Results

    0.39%: pruning 30% of ResNet-50 channels on ILSVRC-12 improves the original model’s top-1 accuracy.

  • Takeaways & Limitations

    DCP provides an effective way to obtain more compact networks, including improvements after removing redundant channels from compact MobileNet designs.

  • Takeaways & Limitations

    Reconstruction-based pruning depends strongly on pre-trained-model quality and may mistakenly retain nondiscriminative intermediate channels, especially in deeper networks.

Abstract

from arXiv · show

Channel pruning is one of the predominant approaches for deep model compression. Existing pruning methods either train from scratch with sparsity constraints on channels, or minimize the reconstruction error between the pre-trained feature maps and the compressed ones. Both strategies suffer from some limitations: the former kind is computationally expensive and difficult to converge, whilst the latter kind optimizes the reconstruction error but ignores the discriminative power of channels. To overcome these drawbacks, we investigate a simple-yet-effective method, called discrimination-aware channel pruning, to choose those channels that really contribute to discriminative power. To this end, we introduce additional losses into the network to increase the discriminative power of intermediate layers and then select the most discriminative channels for each layer by considering the additional loss and the reconstruction error. Last, we propose a greedy algorithm to conduct channel selection and parameter optimization in an iterative way. Extensive experiments demonstrate the effectiveness of our method. For example, on ILSVRC-12, our pruned ResNet-50 with 30% reduction of channels even outperforms the original model by 0.39% in top-1 accuracy.

1 Introduction

Channel pruning removes redundant channels for efficient inference, but existing training-from-scratch and reconstruction-based approaches have important limitations. DCP addresses these limitations by selecting channels according to discriminative power and reconstruction error, using additional losses and greedy optimization.

  • Deep models require substantial memory and computation, motivating compression methods that reduce redundancy without significant performance degradation.
  • Channel pruning directly removes redundant channels and related filters, reducing model size and inference cost with little additional effort for acceleration.It can also be combined with quantization to obtain more compact models.
  • Existing channel-selection methods either learn sparsity from scratch or minimize feature-map reconstruction error, but each strategy has practical limitations.Training from scratch is difficult for deep networks, while reconstruction can retain channels that are irrelevant to discrimination.
  • DCP introduces discrimination-aware losses into intermediate layers to identify channels with true discriminative power.The method fine-tunes stage-wise using an additional loss together with the final loss.
  • DCP formulates channel selection as an ℓ2,0-norm constrained optimization problem and solves it with a greedy method.The optimization jointly considers channel discrimination power and reconstruction-related objectives.
  • 0.39%: DCP improves original ResNet-50 top-1 accuracy when pruning 30% of channels on ILSVRC-12.At 50% pruning, it outperforms ThiNet by 0.81% in top-1 accuracy and 0.51% in top-5 accuracy.

3 Proposed method

The proposed method adds discrimination-aware losses to intermediate layers, combines discriminative and reconstruction objectives, and prunes channels through staged fine-tuning and greedy selection. Channel sparsity can be predefined or determined automatically during optimization.

  • Motivation: Reconstruction-based pruning minimizes feature-map MSE but can retain channels unrelated to discriminative power and depends strongly on pre-trained-model quality.The latter problem becomes more severe as networks become deeper.
  • Joint objective: The joint objective combines cross-entropy and reconstruction error, with λ balancing the two terms.For a considered layer, the paper states that this joint loss is convex with respect to its parameters.
  • Constrained pruning: DCP formulates channel selection with an ℓ2,0-norm constraint and updates the model stage-wise using additional and final losses.The desired channel count may follow a predefined pruning rate or be automatically adjusted by stopping conditions.
  • Discrimination-aware losses: DCP inserts discrimination-aware losses into intermediate layers and fine-tunes them jointly with the final loss to improve intermediate-layer discriminative power.Losses are placed evenly in the network, with each stage targeting the layers between consecutive loss locations.
  • Greedy channel selection: The greedy selector starts with no channels, activates the channel with the largest gradient Frobenius norm, and optimizes the selected parameters iteratively.The selected parameters are warm-started from the fine-tuned model, while SGD updates them under the channel constraint.
  • Stopping criterion: The algorithm can automatically determine the number of selected channels when the normalized loss change falls below tolerance ϵ.The stopping rule uses |L(W_t−1) − L(W_t)|/L(W_0) ≤ ϵ.

4 Experiments

DCP is evaluated against established pruning and width-multiplier baselines on CIFAR-10, ILSVRC-12, and LFW. Across these experiments, it achieves strong accuracy or error performance while reducing computation, model size, or parameters.

  • Experimental setup: Experiments compare DCP and DCP-Adapt with ThiNet, channel pruning, Slimming, width-multiplier baselines, and Random DCP across three datasets.The evaluation includes CIFAR-10, ILSVRC-12, and LFW, with additional-loss and pruning-rate studies.
  • CIFAR-10: DCP achieves the best performance at the same acceleration rate among compared methods on CIFAR-10.With DCP-Adapt, pruned VGGNet also improves testing error by 0.58% over the pre-trained model and obtains 15.58× reduction in model size.
  • ILSVRC-12: On ILSVRC-12, DCP achieves 2× acceleration on ResNet-50 and outperforms ThiNet by 0.81% in top-1 error and 0.51% in top-5 error.It also improves top-5 error by 0.79% over channel pruning and incurs 1.06% top-1 error degradation versus WM+'s 2.41%.
  • LFW: With 50% pruning on LFW, DCP speeds up SphereNet-4 by 3.66× while improving ten-fold validation accuracy by 0.1%.The resulting model has 45.15M FLOPs and 5.89M parameters; pruning 65% of channels yields 24.16M FLOPs and 98.02% accuracy.

5 Ablation studies

Ablation studies show that DCP's performance depends on pruning rate, the discrimination-aware loss weight, and the stopping tolerance, while selected channels produce more informative feature maps.

  • Performance with different pruning rates: 30% channel pruning lets ResNet-50 outperform its pre-trained model by 0.39% in top-1 error and 0.14% in top-5 error.At a 50% pruning rate, ResNet-50 incurs a 1.06% top-1 error increase versus 2.29% for ResNet-18.
  • Performance with different pruning rates: Increasing the pruning rate generally worsens the performance of pruned models.The study evaluates 30%, 50%, and 70% channel pruning on ResNet-18 and ResNet-50 using top-1 and top-5 error on ILSVRC-12.
  • Effect of the trade-off parameter λ: Increasing λ improves the pruned ResNet-56's performance, with the best result at λ=1.0.A larger λ places more emphasis on the additional discrimination-aware loss; reconstruction error and cross-entropy both contribute to performance.
  • Effect of the stopping condition: Decreasing ϵ selects more channels and improves the pruned VGGNet's performance on CIFAR-10.The experiment uses ϵ ∈ {0.1, 0.01, 0.001} and supports the stopping condition for automatically determining the pruning rate.
  • Feature-map visualization: Feature maps from selected ResNet-18 channels are more informative than feature maps from pruned channels.The visualization concerns the first block, res-2a, and is presented in Figure 2.

6 Conclusion

The paper concludes that DCP compresses deep networks by selecting channels using both reconstruction error and discrimination power, solved through a greedy optimization procedure. Experiments report strong performance across benchmark datasets and compact architectures, while future work targets layer-wise computational cost and additional compression methods.

  • Conclusion: DCP formulates channel pruning as a sparsity-induced optimization problem combining reconstruction error and channel discrimination power.A greedy algorithm solves the resulting optimization problem.
  • Conclusion: Experiments on benchmark datasets show DCP outperforming several state-of-the-art methods at the same pruning rate.The paper also reports performance improvements for compact architectures such as MobileNet v1 and v2.
  • Future work: Future work will incorporate per-layer computational cost and combine DCP with methods such as quantization.The stated goals are further reduction of model size and inference cost.

Supplementary Material: Discrimination-aware Channel

The supplementary material covers theoretical analysis, fine-tuning, block-level pruning, additional-loss counts, sample counts, and sensitivity to pre-trained model quality.

  • Supplementary organization: The supplementary material includes theoretical analysis of the loss function and details of the DCP fine-tuning algorithm.These topics are covered in Sections 7 and 8, respectively.
  • Supplementary organization: Additional studies examine individual-block pruning, the number of additional losses, sample counts, and pre-trained model quality.These investigations are listed among the supplementary sections describing DCP's experimental analysis.

7 Convexity of the loss function

The paper analyzes the joint loss used in DCP and states that it is convex with respect to the parameters of the considered layer under the defined mean-square and cross-entropy losses.

  • Convexity result: The joint loss L(W) is convex with respect to the model parameters W of the considered layer.The proposition assumes the mean-square and cross-entropy losses defined in the paper's equations.
  • Proof: The mean-square loss is convex because the output is linear with respect to W.The proof begins by establishing this property for the mean-square component.
  • Proof: The cross-entropy component is analyzed through its derivatives and Hessian, which is stated to be semi-definite.The proof uses the binary-classification form and notes that it extends to multiclassification.
  • Assumption: The analysis assumes F(p,i) and W are vectors when deriving the loss properties.The text explicitly introduces this assumption before the derivative calculations.

8 Details of fine-tuning algorithm in DCP

DCP fine-tunes the model iteratively by alternating optimization with an intermediate-stage additional loss and the final loss. The procedure uses mini-batches, repeated updates, and learning-rate control.

  • Fine-tuning procedure: DCP fine-tuning computes the additional loss and final loss gradients with respect to W, then updates the model parameters.The procedure applies forward propagation, computes Lp and Lf gradients, and updates W using the specified equations.
  • Fine-tuning procedure: The algorithm repeats fine-tuning for T iterations while using learning rate γ and decay parameter τ.Algorithm 3 lists the inserted-output position, model parameters, iteration count, learning rate, and learning-rate decay as inputs.
  • Fine-tuning procedure: Each fine-tuning iteration randomly selects a mini-batch from the training set before computing the additional-loss gradient.

9 Channel pruning in a single block

The single-block experiment evaluates DCP against ThiNet, APoZ, and weight sum on ILSVRC-12 after pruning 30% of channels. DCP produces lower performance degradation than the compared methods under the same pruning rate.

  • Experimental setup: 30% channel pruning in individual ResNet-18 blocks was evaluated on ILSVRC-12 using DCP, ThiNet, APoZ, and weight sum.
  • Results: DCP significantly outperformed APoZ and weight sum in the single-block pruning experiment.
  • Results: Under the same pruning rate, DCP achieved lower performance degradation than ThiNet.

10 Exploring the number of additional losses

The experiments examine how additional-loss count, channel-selection sample count, and pre-trained model quality affect pruning. More samples reduce degradation but increase computation, while DCP remains relatively insensitive to pre-trained model quality.

  • Additional-loss count: Adding too many additional losses yields little performance gain while significantly increasing computational cost.For ResNet-56 with 50% channel pruning and 2× acceleration on CIFAR-10, losses every 5–10 layers were sufficient as a heuristic trade-off.
  • Selection sample count: More channel-selection samples generally reduce the pruned model’s performance degradation but increase computation cost.The experiment varies the sample count from 10 to 100k for 30% pruning of ResNet-18 on ILSVRC-12.
  • Selection sample count: 10k samples were used for ILSVRC-12 channel selection, while the whole training set was used for CIFAR-10.
  • Pre-trained model quality: DCP showed small sensitivity to pre-trained model quality and steadily outperformed two other methods given models of the same quality.The comparison used ResNet-56 intermediate models from epochs 120, 240, and 400 for 2× acceleration.

13 Pruning MobileNet v1 and MobileNet v2 on ILSVRC-12

DCP was applied to MobileNet models on ILSVRC-12 and compared with ThiNet, while the broader complexity and visualization results describe efficiency and information retention in pruned networks. The reported MobileNet comparisons favor DCP in top-1 error.

  • MobileNet results: DCP outperformed ThiNet in top-1 error by 0.75% for MobileNet v1 and 0.47% for MobileNet v2 on ILSVRC-12.
  • Model complexity: Channel pruning reduces network parameters and FLOPs, resulting in forward and backward acceleration evaluations on CPU and GPU.The complexity measurements use a mini-batch size of 32 on specified Intel Xeon and NVIDIA TitanX hardware.
  • VGGNet results: The pruned VGGNet obtained from DCP-Adapt had lower testing error than the original network, 5.43% versus 6.01%.The pruned structure also had lower layer complexities, especially in deep layers.
  • Feature-map visualization: Visualization results indicate that selected channels contain more information than pruned channels.The feature-map comparison concerns the res-2a block in ResNet-18 across different input images.
Loading 1810.11809v3…