PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Optimal Structured CNN Pruning via Generative Adversarial Learning

Shaohui Lin, Rongrong Ji, Chenqian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang, David Doermann

arXiv:1903.09291v1cs.CV

TL;DR

CNN pruning faces computational and memory constraints, while existing methods may require iterative retraining, target specific structures, or depend on labels. GAL jointly learns a sparse soft mask for heterogeneous structures through label-free generative adversarial learning and uses FISTA for removal. Across MNIST, CIFAR-10, and ImageNet, the paper reports effective pruning, including 10.88% Top-5 error and 3.7× speedup for pruned ResNet-50.

  • Problem

    Existing structured-pruning methods may require iterative pruning and retraining, target specific structures, and depend on labels, limiting joint heterogeneous pruning and label-free use.

  • Method

    GAL jointly learns a sparse soft mask for heterogeneous structures with label-free generative adversarial learning, then uses FISTA to remove zero-scaled structures.

  • Results

    10.88% Top-5 error and 3.7× speedup are reported for pruned ResNet-50 on ImageNet ILSVRC 2012.

  • Takeaways & Limitations

    GAL provides an end-to-end structured-pruning approach that jointly handles filters, branches, and blocks across multiple CNN compression experiments.

Abstract

from arXiv · show

Structured pruning of filters or neurons has received increased focus for compressing convolutional neural networks. Most existing methods rely on multi-stage optimizations in a layer-wise manner for iteratively pruning and retraining which may not be optimal and may be computation intensive. Besides, these methods are designed for pruning a specific structure, such as filter or block structures without jointly pruning heterogeneous structures. In this paper, we propose an effective structured pruning approach that jointly prunes filters as well as other structures in an end-to-end manner. To accomplish this, we first introduce a soft mask to scale the output of these structures by defining a new objective function with sparsity regularization to align the output of baseline and network with this mask. We then effectively solve the optimization problem by generative adversarial learning (GAL), which learns a sparse soft mask in a label-free and an end-to-end manner. By forcing more scaling factors in the soft mask to zero, the fast iterative shrinkage-thresholding algorithm (FISTA) can be leveraged to fast and reliably remove the corresponding structures. Extensive experiments demonstrate the effectiveness of GAL on different datasets, including MNIST, CIFAR-10 and ImageNet ILSVRC 2012. For example, on ImageNet ILSVRC 2012, the pruned ResNet-50 achieves 10.88\% Top-5 error and results in a factor of 3.7x speedup. This significantly outperforms state-of-the-art methods.

1. Introduction

GAL targets CNN compression by jointly pruning heterogeneous structures end-to-end, avoiding iterative pruning and retraining. It learns a sparse soft mask without labels and uses adversarial learning plus FISTA to remove structures.

  • Motivation: CNNs deliver strong vision accuracy but consume substantial computation and memory, limiting deployment on resource-constrained devices.Network pruning is one of several proposed CNN-compression techniques and can provide practical compression and acceleration.
  • Motivation: Structured pruning removes units such as filters, while non-structured pruning can create irregular memory access that requires specialized accelerators.The paper focuses on structured pruning to support efficient inference without such irregularity.
  • Challenges: Existing hard-mask and label-dependent approaches make optimization difficult and cannot directly support scenarios without class labels.The cited examples include binary filter masks and sparsity regularization trained with class labels.
  • Proposed approach: GAL jointly prunes channels/filters, branches, and blocks in an end-to-end manner without iterative pruning and retraining.A sparse soft mask scales selected structure outputs to zero, allowing the corresponding structures to be pruned.
  • Proposed approach: GAL treats the pruned network as a generator and uses a discriminator to distinguish baseline outputs from pruned-network outputs, forcing them closer.Adversarial regularization prevents a trivially strong discriminator, while FISTA reliably removes structures whose scaling factors reach zero.
  • Results: 10.88% Top-5 error and 3.7× speedup are reported for the pruned ResNet-50 on ImageNet ILSVRC 2012.The paper presents this as outperforming state-of-the-art methods.

2. Related Work

Prior work spans non-structured and structured pruning, neural architecture search, and knowledge distillation. GAL is positioned as label-free and complementary to other compression techniques.

  • Network Pruning: Non-structured pruning removes individual weights, whereas structured pruning removes organized weights to reduce network size and support fast inference.Non-structured methods can produce irregular memory access, while structured methods target hardware-efficient reductions.
  • Network Pruning: Binary-mask methods guide filter removal, but prior work identifies hard filter pruning as lacking effectiveness and slackness.Examples include NISP and globally learned binary masks.
  • Network Pruning: Prior sparse scaling approaches use supervised training with class-labelled data, while GAL obtains its sparse soft mask from label-free data.The paper states that GAL can transfer to scenarios with unseen labels.
  • Neural Architecture Search: Neural architecture search using reinforcement learning or genetic algorithms explores large spaces but requires significant computational overhead.The related-work discussion also notes that stochastic gradient descent alone cannot reliably remove sparse structures.
  • Knowledge Distillation: GAL is related to knowledge distillation because both involve transferring information from a larger or baseline network to a smaller network.The supplied passage describes knowledge distillation as transferring knowledge from teacher to student using forms such as dark knowledge and attention.
  • Other Compression Methods: GAL is orthogonal to low-rank decomposition and parameter quantization and can be integrated with them for higher compression and speedup rates.This positions GAL as compatible with, rather than a replacement for, those compression approaches.

3. Our Method

GAL learns a sparse soft mask jointly with the pruned network and discriminator, aligning pruned outputs with a fixed baseline while promoting removable structures. FISTA optimizes mask values toward exact zeros so channels, branches, and blocks can be removed reliably.

  • Generative adversarial learning: GAL trains the generator, representing the pruned network, against a discriminator that distinguishes baseline features from pruned-network features.Adversarial regularization updates the discriminator using pruned-network features and extends the two-player game to provide more valuable gradients.
  • Formulation: The pruned network minimizes output-feature discrepancy from the fixed baseline while learning weights and a soft mask.The data loss aligns baseline and pruned outputs, while the discriminator participates in the adversarial optimization.
  • Formulation: An ℓ1 penalty λ∥m∥1 promotes sparse mask values, and mi = 0 makes the corresponding structure removable because its output contributes nothing downstream.Weight decay regularizes WG, while discriminator regularization prevents the discriminator from dominating training.
  • Optimization: FISTA alternates SGD updates of WG with proximal updates of m, addressing slow convergence and non-exact zero scaling factors observed with SGD alone.The resulting exact zeros allow structures to be removed without relying on a threshold that may reduce accuracy relative to the baseline.
  • Structure selection: Soft masks are placed after blocks, branches, and channels, enabling their joint selection across ResNets, GoogLeNet, and DenseNets.After optimization, zero-masked feature maps and associated upper-layer filters are removed; for residual blocks, zeroing the residual mapping does not cut shortcut information flow.

4.1. Experimental Settings

The experiments apply GAL across heterogeneous CNN structures and datasets, using specified pruning selections, hardware, and discriminator architecture. Evaluation tables report pruning rates and classification error with or without fine-tuning.

  • Pruning configurations: GAL uses channel selection for LeNet, VGGNet, and DenseNets; branch selection for GoogLeNet; and block selection for ResNets.ResNets additionally combine channel and block selection to jointly prune heterogeneous structures.
  • Implementation: The optimization runs in PyTorch on two NVIDIA GTX 1080Ti GPUs with 128GB of RAM.Weight decay is 0.0002 and momentum is 0.9.
  • Hyperparameters: The sparsity factor λ is cross-validated in [0.01, 0.1] for channel pruning and [0.1, 1] for branch and block pruning.The stated ranges correspond to the different structure-selection settings.
  • Evaluation: Tables 1 and 2 report pruning results for LeNet on MNIST and VGGNet on CIFAR-10, respectively.The table conventions define Error/+FT, PR, GAL-λ, and M/B.
  • Discriminator architecture: The discriminator has three fully connected layers with ReLU nonlinearity and 128-256-128 neurons.It receives baseline and pruned-network features and predicts whether each input came from the baseline or pruned network.

4.2. Comparison with the State-of-the-art

GAL achieves strong pruning trade-offs across MNIST, CIFAR-10, and ImageNet, with results spanning channels, branches, and residual blocks. Jointly pruning heterogeneous structures improves ImageNet speedup and compression compared with pruning blocks or channels alone.

  • MNIST: 0.1% error increase accompanies 92.6% FLOPs and 93% parameter pruning for GAL on LeNet.This setting uses λ=0.05.
  • CIFAR-10: 6.23% classification error is achieved with 39.6% FLOPs and 77.6% parameter pruning for GAL on VGGNet.GAL outperforms the reported L1 and SSS results on both error and pruning rates.
  • CIFAR-10: 35.3% computational saving is achieved by pruning 3362 of 8904 DenseNet-40 channels, with 5.39% error.GAL has higher computational saving but slightly higher error than the compared Liu et al. result.
  • CIFAR-10: 14 of 36 GoogLeNet branches are removed, yielding 38.2% FLOPs saving, 49.3% parameter saving, and a 0.49% error increase.GAL achieves the best reported trade-off among the compared methods.
  • CIFAR-10: 37.6% FLOPs pruning removes 10 of 27 ResNet-56 residual blocks while decreasing error by 0.12%.With 16 blocks removed, GAL reaches 60.2% FLOPs pruning with 8.42% classification error.
  • ImageNet ILSVRC 2012: 2.22× speedup and 1.32× compression are achieved by jointly pruning blocks and channels in ImageNet ResNet-50.GAL-0.5-joint improves over GAL-0.5, which reports 1.75× speedup and 1.2× compression.

4.3. Ablation Study

The ablation study evaluates discriminator regularizers, optimizers, and GAN variants for GAL on ResNet-56 and DenseNet-40. Adversarial regularization, FISTA, and label-free GAN training each improve pruning outcomes under the reported comparisons.

  • Effect of the Regularizers on Discriminator D: Adversarial regularization achieves the best performance compared with L1-norm and L2-norm discriminator regularizers.The authors attribute this to prolonged generator–discriminator competition that improves generator features toward the baseline output.
  • Effect on the Optimizers: FISTA safely removes redundant structures during training, whereas thresholding SGD outputs causes a significant accuracy drop after pruning.SGD uses a hard threshold of 0.0001 because it does not produce exact zero scaling factors.
  • Effect on the Optimizers: GAL with FISTA achieves better performance than SGD in the optimizer comparison.SGD obtains lower error than random training before thresholding, but pruning near-zero weights can substantially affect the final network output.
  • Effect of the GANs: 8.42% error with a 65.9% parameter pruned rate is achieved by label-free GAL on ResNet-56, versus 9.56% error with a 50.5% parameter pruned rate for label-dependent CGAN.The comparison uses the same regularization and optimizer.
  • Effect of the GANs: Removing GANs significantly increases error, while GAL outperforms CGAN in the reported ResNet-56 comparison.The authors conjecture that adding class labels to CGAN’s discriminator affects generator features’ approximation of baseline outputs.

5. Conclusion

The paper presents GAL for end-to-end structured pruning of heterogeneous CNN structures. It combines sparse soft masks, label-free adversarial learning, and FISTA, with experiments reporting superior performance across architectures and datasets.

  • 5. Conclusion: GAL jointly prunes heterogeneous CNN structures, including channels, filters, branches, and blocks, in an end-to-end manner.The approach avoids iteratively pruning and retraining.
  • 5. Conclusion: A sparse soft mask aligns baseline and pruned-network outputs through an objective with ℓ1-regularization on the mask.The mask scales outputs of specific structures and is learned by label-free generative adversarial learning.
  • 5. Conclusion: FISTA quickly and reliably removes redundant structures by forcing more soft-mask scaling factors to zero.
  • 5. Conclusion: Experiments across state-of-the-art CNN architectures and different datasets demonstrate superior performance gains over state-of-the-art methods.
Loading 1903.09291v1…