PaperlayerView exact revision on arXiv

Source-linked AI summary

DMCP: Differentiable Markov Channel Pruning for Neural Networks

Shaopeng Guo, Yujie Wang, Quanquan Li, Junjie Yan

arXiv:2005.03354v2cs.CVcs.LG

TL;DR

Existing channel-pruning structure searches are limited by the need to train and evaluate many candidate networks. DMCP models pruning as a differentiable Markov process optimized with task loss and budget regularization, achieving state-of-the-art ImageNet performance across FLOPs settings on ResNet and MobileNetV2.

  • Problem

    Existing channel-pruning methods that search sub-structures require training and evaluating many structures, limiting their scalability.

  • Method

    DMCP models channel pruning as a Markov process and optimizes its transition parameters by gradient descent with task loss and budget regularization.

  • Results

    DMCP achieves state-of-the-art performance on ImageNet with ResNet and MobileNetV2 across various FLOPs settings.

  • Takeaways & Limitations

    The Markov formulation enables efficient differentiable sub-structure search while implicitly selecting the number of retained channels in each layer.

  • Takeaways & Limitations

    Baseline and pruned models use different training settings, and architecture parameters require special handling because their norms can grow very large near deterministic channel-retention probabilities.

Abstract

from arXiv · show

Recent works imply that the channel pruning can be regarded as searching optimal sub-structure from unpruned networks. However, existing works based on this observation require training and evaluating a large number of structures, which limits their application. In this paper, we propose a novel differentiable method for channel pruning, named Differentiable Markov Channel Pruning (DMCP), to efficiently search the optimal sub-structure. Our method is differentiable and can be directly optimized by gradient descent with respect to standard task loss and budget regularization (e.g. FLOPs constraint). In DMCP, we model the channel pruning as a Markov process, in which each state represents for retaining the corresponding channel during pruning, and transitions between states denote the pruning process. In the end, our method is able to implicitly select the proper number of channels in each layer by the Markov process with optimized transitions. To validate the effectiveness of our method, we perform extensive experiments on Imagenet with ResNet and MobilenetV2. Results show our method can achieve consistent improvement than state-of-the-art pruning methods in various FLOPs settings. The code is available at https://github.com/zx55/dmcp

1. Introduction

Channel pruning searches for compact network sub-structures, but existing structure-search methods are costly because they train or evaluate many candidates. DMCP makes this search differentiable through a Markov formulation and achieves strong ImageNet results across FLOPs settings.

  • Motivation: Channel pruning removes unnecessary weights from overparameterized CNNs to obtain more efficient models with limited performance loss.Conventional methods typically pre-train, prune according to a criterion, and then fine-tune the model.
  • Limitations of prior work: Existing structure-search methods use reinforcement learning or evolution to select layer-wise pruning ratios, but require training or evaluating many sampled structures.This scalability limitation remains even when fine-tuning is avoided or weights are predicted by a meta-network.
  • Search-space challenge: Channel pruning differs from differentiable NAS because its search space is the number of channels per layer and channel choices have logical dependencies.Retaining k+1 channels requires retaining at least k channels first.
  • DMCP: DMCP models pruning as a Markov process whose states represent retained channels and whose transitions represent conditional retention probabilities.The marginal probability of retaining each channel is computed from transition probabilities and serves as a scaling coefficient.
  • Results: 0.1% accuracy drop accompanies 30% FLOPs reduction for MobileNetV2, while ResNet-50 achieves 44% FLOPs reduction with only 0.4% drop.These results are reported for ImageNet classification experiments.

2. Related Work

Prior work reduces neural architecture search cost with gradient-based methods, while channel-pruning research includes hard and soft pruning and structure-search approaches. DMCP uses Markov modeling to remove duplicated channel-selection solutions and reduce the effective search space.

  • Neural Architecture Search: DARTS parameterizes candidate-operation probabilities with learnable weights, but searches on a small proxy task before transferring the architecture to a larger target task.ProxylessNAS instead searches directly on large-scale target tasks by sampling two paths.
  • Channel Pruning: Channel-pruning methods are broadly classified as hard pruning, which removes channels iteratively, and soft pruning, which retains pruned channels during training.DMCP is characterized as a soft-pruning method.
  • DMCP: For a layer with C channels, DMCP has an O(C) solution space, whereas combinations-based methods have an O(2^C) space even when retaining the same channel count.The Markov formulation eliminates duplicated solutions among channel combinations.
  • Structure Search: AMC uses reinforcement learning and MetaPruning uses evolutionary search with a meta-network, but both require training or evaluating many structures.These requirements make the methods inefficient for structure search.

3. Method

DMCP makes channel pruning differentiable by modeling each layer’s pruning process as a Markov process with learnable transitions. It optimizes network weights and architecture parameters while using expected channels and FLOPs for budget regularization, then samples a pruned model.

  • 3.1.1 Channel Pruning via Markov Process: DMCP models channel pruning as a Markov process in which states retain successive channels and transitions determine whether the process continues or terminates.Retaining k channels means retaining the first k channels, enforcing the pruning process’s logical ordering.
  • 3.1.1 Channel Pruning via Markov Process: Learnable architecture parameters A parameterize transition probabilities through sigmoid functions, with p1 fixed to retain at least one channel per layer.The marginal probability of retaining each channel is derived from these transitions and can be used to represent pruning.
  • 3.1.2 Differentiable Channel Pruning: Architecture parameters are wrapped into the unpruned network so pruning a channel is represented by setting its marginal retention probability to zero.The pruning process is placed after batch normalization because batch normalization can otherwise scale channel values.
  • 3.1. Definition of Pruning Process: For residual blocks with identity shortcuts, layers requiring equal output channels share architecture parameters, preserving compatibility with element-wise summation.This weight-sharing strategy allows DMCP to address layers that previous methods often leave unpruned.
  • 3.1.3 Budget Regularization: Expected channel counts and expected FLOPs make the pruning budget differentiable, allowing FLOPs optimization by gradient descent with a tolerance margin below the target.The budget regularizer is zero when expected FLOPs lie between γ × FLOPstarget and FLOPstarget.
  • 3.2 Training DMCP: DMCP trains by alternating weight updates and architecture-parameter updates, using sampled sub-structures and accumulated gradients during its two-stage procedure.Stage 1 warms up the unpruned network, after which stages 1 and 2 are iterated to update weights and architecture parameters.

4. Experiments

Experiments evaluate DMCP on ImageNet across model scales, sampling strategies, training components, and comparisons with existing pruning methods. DMCP consistently performs strongly, with ablations supporting warmup, task-loss guidance, weight adaptation, and the variant sandwich rule.

  • Experimental setup: DMCP experiments cover ImageNet classification with MobileNetV2 and ResNet models across multiple target FLOPs settings.MobileNetV2 and ResNet serve as light and heavy model families, respectively.
  • Ablation study: Within 500 iterations, DMCP recovers the pre-trained model’s FLOPs and top-1 training accuracy without a FLOPs constraint.The recoverability test freezes MobileNetV2 weights and optimizes randomly initialized architecture parameters using task loss.
  • Ablation study: DMCP’s performance is not sensitive to the unpruned network scale, although a larger scale can yield slightly better performance.MobileNetV2 1.0x and 1.5x networks are pruned to 59M and 210M FLOPs.
  • Ablation study: Different pruned structures can achieve nearly identical accuracy: the 1.0x model retains more shallow-layer channels, whereas the 1.5x model retains more deep-layer channels.This observation indicates multiple local minima in the channel-pruning search space.
  • Ablation study: Warmup and the variant sandwich rule improve pruning performance, while task loss distinguishes layer importance beyond FLOPs alone.The full training scheme also adapts network weights when architecture parameters change, improving over simpler update schemes.
  • Comparison with state-of-the-art: Under the same settings, DMCP outperforms the compared pruning methods on MobileNetV2, ResNet18, and ResNet50.The compared methods include AMC, MetaPruning, AutoSlim, SFP, and FPGM; slimmable training further boosts DMCP’s performance against AutoSlim.

5. Conclusion

DMCP addresses the cost of searching channel-pruned sub-structures with a differentiable Markov-process formulation optimized by gradient descent. It achieves state-of-the-art performance on ResNet and MobileNetV2 across ImageNet FLOPs settings, though the reported baseline and pruned-model training settings differ.

  • Conclusion: DMCP models channel pruning as a Markov process, making the search differentiable and optimizing it with respect to task loss by gradient descent.The supplied passages identify this as the proposed solution to the need to train and evaluate many sub-structures.
  • Conclusion: Training settings for baseline and pruned models are different, qualifying direct comparisons between them.
  • Conclusion: DMCP achieves state-of-the-art performance with ResNet and MobileNetV2 on ImageNet across various FLOPs settings.

A.1. FLOPs Distribution of the Pruned Model

Sampling 3000 MobileNetV2-210M structures with the trained Markov process produces a FLOPs distribution centered near the desired 210M budget, below the 672M unpruned network.

  • A.1. FLOPs Distribution of the Pruned Model: The mean FLOPs of 3000 sampled MobileNetV2-210M structures lies around 210M, matching the desired budget.The figure describes FLOPs frequency over sampled structures and marks the mean with a red dashed line.
  • A.1. FLOPs Distribution of the Pruned Model: The unpruned MobileNetV2 network has 672M FLOPs, providing the reference for the sampled pruned structures.

A.2. The Channel Distribution of Pruned Layers.

At the 210M FLOPs budget, sampled pruned models usually show unimodal channel-count distributions across layers, while some layers retain all channels.

  • A.2. The Channel Distribution of Pruned Layers: Most sampled layers have a unimodal distribution of retained channel counts under the 210M FLOPs budget.The analysis samples 3000 models whose FLOPs fall within the desired 210M budget.
  • A.2. The Channel Distribution of Pruned Layers: LinearBottleneck6 and LinearBottleneck7 are examples of layers that retain all channels.

B. Comparison between using warm-up and using pre-trained model

Warm-up provides an initialization for iterative DMCP training by imposing greater importance on earlier channel groups, and replacing it with pretrained-model initialization reduces accuracy.

  • B. Comparison between using warm-up and using pre-trained model: Warm-up makes each channel group more important than the immediately following group, providing a suitable initialization for iterative training.
  • B. Comparison between using warm-up and using pre-trained model: 0.6% accuracy drop was observed on DMCP-MBV2 with 210M FLOPs when warm-up was replaced by pretrained-model initialization.

C. Modeling architecture parameters as independent Bernoulli variables

The method compares a linear O(C) channel-selection space with the exponentially larger Bernoulli space, while examining channel distributions across MobileNetV2 layers.

  • The method’s solution space is O(C), whereas Bernoulli architecture parameters yield O(2C) possible channel combinations.
  • The larger Bernoulli solution space makes optimization much harder than the method’s Markov-based formulation.
  • Figure 7 reports channel-frequency distributions for 12 MobileNetV2-210M layers sampled from different blocks, grouped into 15 channel groups per layer.
Loading 2005.03354v2…