PaperlayerView exact revision on arXiv

Source-linked AI summary

Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression

Yawei Li, Shuhang Gu, Christoph Mayer, Luc Van Gool, Radu Timofte

arXiv:2003.08935v1cs.CV

TL;DR

The paper addresses the separate treatment and practical limitations of filter pruning and low-rank decomposition in CNN compression. It unifies both techniques with group-sparsity regularization on a shared matrix, enabling mode switching across layers. The method achieves state-of-the-art performance on VGG, ResNet, ResNeXt, WRN, and DenseNet benchmarks.

  • Problem

    Filter pruning and low-rank decomposition were developed separately, with pruning limited by skip connections and decomposition adding a 1 × 1 convolutional layer.

  • Method

    The paper uses a sparsity-inducing matrix whose column or row group regularization derives filter pruning or decomposition, respectively.

  • Results

    State-of-the-art performance is achieved across VGG, ResNet, ResNeXt, WRN, and DenseNet, with CIFAR10 improvements including 0.41% and 1.51% lower Top-1 error than for VGG and DenseNet.

  • Takeaways & Limitations

    The unified formulation lets compression switch between pruning and decomposition according to layer-specific circumstances, leveraging both methods.

Abstract

from arXiv · show

In this paper, we analyze two popular network compression techniques, i.e. filter pruning and low-rank decomposition, in a unified sense. By simply changing the way the sparsity regularization is enforced, filter pruning and low-rank decomposition can be derived accordingly. This provides another flexible choice for network compression because the techniques complement each other. For example, in popular network architectures with shortcut connections (e.g. ResNet), filter pruning cannot deal with the last convolutional layer in a ResBlock while the low-rank decomposition methods can. In addition, we propose to compress the whole network jointly instead of in a layer-wise manner. Our approach proves its potential as it compares favorably to the state-of-the-art on several benchmarks.

1. Introduction

CNN compression is motivated by deployment costs, while filter pruning and low-rank decomposition offer complementary compact approximations. The paper unifies them through group-sparse matrices that can switch between pruning and decomposition across network layers.

  • CNNs require millions of parameters and substantial computation, limiting deployment on edge and mobile devices.
  • Filter pruning removes weak output-channel connections, whereas low-rank decomposition replaces heavy convolutions with lightweight linear combinations.
  • Filter pruning struggles with ResNet block outputs under skip connections, while decomposition introduces an additional 1 × 1 convolutional layer.
  • Both methods seek compact approximations of network parameters, but they use different approximation strategies.
  • The unified method can flexibly switch between compression modes, uses joint network compression, and achieves state-of-the-art performance across several CNN architectures.
  • Group sparsity on columns or rows of a sparsity-inducing matrix respectively produces filter pruning or decomposition.

2. Related Work

Related work covers pruning-based and decomposition-based approaches for reducing neural-network computation and storage. It distinguishes unstructured and structural pruning, tensor decomposition, quantization, and knowledge distillation.

  • Non-structural pruning removes individual weak connections but produces irregular parameters that hinder implementation-friendly acceleration.
  • Structural pruning zeros structured groups such as convolutional filters to reduce feature-map channels.
  • Tensor decomposition replaces original filters with lightweight factors or projections, reducing parameters and computations.
  • Quantization uses low-bit parameter representations for storage and inference benefits without changing the fully developed network architecture.
  • Knowledge distillation transfers knowledge from a teacher network to a student network, with research focusing on student architecture and loss functions.

3. The proposed method

The method unifies filter pruning and filter decomposition through group sparsity imposed on an introduced matrix A. Its optimization jointly learns compression structure, targets a specified compression ratio, and balances layer behavior.

  • Unified formulation: The proposed framework represents a convolution as X × (W × A), where group sparsity on A compresses the original filters.A is implemented as a 1 × 1 convolution appended to the original layer.
  • Group sparsity: The group-sparsity regularizer applies a function Φ to group ℓ2 norms, with ℓ1 and ℓ1/2 among the principal regularizers used.The paper also studies ℓ1−2 and logsum regularizers in ablation experiments.
  • Unified formulation: Column sparsity in A removes output channels and is equivalent to filter pruning.The resulting compressed filter can be formed by multiplying W and the column-pruned matrix.
  • Unified formulation: Row sparsity in A removes corresponding channels of the preceding convolution while retaining the later convolution, yielding filter decomposition.Keeping the compression matrices separate avoids the loss of computational savings from multiplying them together.
  • Optimization: The proximal-gradient solver updates A with a gradient step followed by a proximal step, while pretrained W uses a smaller learning rate.This keeps the optimization focused on the sparsity matrix rather than substantially modifying pretrained parameters.
  • Compression control: Binary search adjusts the nullifying threshold because compression ratio is monotonic in the threshold but lacks a known explicit inverse.The search halves its step when the target lies between successive compression ratios and stops within tolerance C.
  • Layer balancing: Layer balancing recalibrates regularization using average group ℓ2 norms, while ResNet basic blocks additionally adjust learning rates to reduce unbalanced compression.The first and second sparsity-inducing matrices can otherwise compress unevenly under a shared threshold.

4. Implementation Considerations

For different network blocks, the method places sparsity-inducing matrices where they can manipulate relevant channels. ResNet and ResNeXt bottlenecks require block-specific handling, especially for grouped middle convolutions.

  • ResNet basic block: In a ResNet basic block, both 3 × 3 convolutions receive appended 1 × 1 sparsity-inducing convolutions.The first matrix may regularize rows or columns, while the second matrix uses a restricted configuration described in the implementation.
  • Bottleneck blocks: ResNet and ResNeXt bottleneck blocks use the existing 1 × 1 convolutions as the natural sparsity-inducing matrices.The bottleneck structure is 1 × 1 → 3 × 3 → 1 × 1.
  • Bottleneck blocks: For ResNet bottlenecks, the two matrices select the input and output channels of the middle 3 × 3 convolution.For ResNeXt, sparsity is instead enforced on the already existing groups of the middle group convolution.
  • Initialization: Because bottleneck blocks already contain the needed 1 × 1 convolutions, the original network parameters are used directly and only the newly added matrix A requires initialization.The paper tests pretrained-parameter/identity initialization and an SVD-based initialization.

5. Experimental Results

The method is evaluated across CIFAR10, CIFAR100, and ImageNet2012 using multiple CNN architectures, with results indicating strong compression-performance trade-offs. Experiments also examine regularization choices and annealing.

  • Experimental setup: The method is evaluated on CIFAR10, CIFAR100, and ImageNet2012 across ResNet, ResNeXt, VGG, DenseNet, WRN, and ResNet50 architectures.CIFAR experiments include multiple network depths and architectures, while ImageNet2012 uses ResNet50.
  • CIFAR10 results: At 50% FLOP compression on ResNet56, the proposed method achieves the best performance among the compared state-of-the-art methods.At 24% compression ratio, it is reported as clearly better than KSE.
  • CIFAR10 results: The proposed method obtains lower error rates than SSS for ResNet and ResNeXt networks with 20 and 164 layers.For VGG and DenseNet, Top-1 error is reduced by 0.41 % and 1.51 % compared with.
  • CIFAR100 results: With regularization-factor annealing, the method achieves much better performance on WRN compression than without annealing.The authors attribute this to improved exploration around a local minimum late in compression.
  • CIFAR100 results: On CIFAR100 164-layer networks, the method outperforms SSS without distillation loss and reaches an even lower Top-1 error rate when distillation is used.The corresponding results for 20-layer networks are also reported.
  • ImageNet results: On ImageNet2012 ResNet50, the method achieves state-of-the-art trade-off performance between Top-1 error rate and FLOP compression ratio.Comparisons are made at roughly comparable compression rates because competing methods use different FLOP compression levels.

6. Conclusion

The paper proposes a unified group-sparsity framework that connects filter pruning and decomposition, and validates it experimentally across network architectures and compression settings.

  • Unified formulation: The method hinges filter pruning and decomposition through group sparsity applied to different structured groups of a sparsity-inducing matrix.Column and row constraints correspond to pruning and decomposition, respectively.
  • Unified formulation: The unified formulation lets the algorithm switch flexibly between pruning and decomposition according to network circumstances.Optimization uses proximal gradient methods with learning-rate adjustment, layer balancing, and regularization-factor annealing.
  • Experimental validation: Experimental results validate the proposed method.The method is applied to VGG, ResNet, ResNeXt, WRN, and DenseNet, where it achieves state-of-the-art performance.

Decomposition for Network Compression”

The appendix describes proximal-operator solutions for several regularizers used in the group-sparsity optimization. It also records the experimental handling of non-convex regularizers and regularization factors.

  • Proximal operators: The proximal operator has closed-form solutions for ℓ1, ℓ1/2, ℓ1−2, and logsum regularizers.For ℓ1, the solution is soft-thresholding; for ℓ1/2, it is half-thresholding.
  • Proximal operators: The proximal operator for the sparsity-inducing matrix A is derived by replacing the scalar function with a group-norm regularizer.The solution follows from a theorem for functions of vector norms and a variable substitution.
  • Regularizer handling: For ℓ1/2, ℓ1−2, and logsum regularizers, the paper uses the derived substitutions experimentally even though these regularizers are non-convex.The corresponding main-paper results are reported as competitive.
  • Appendix summaries: The solutions to the four regularizers are summarized in Table 5, while their regularization factors are listed in Table 6.The appendix also specifies conditions and intermediate quantities for some non-convex solutions.

B. Hyper Parameters for Different Regularizers

The appendix specifies learning rates for the sparsity-inducing matrix and network weights during compression optimization on CIFAR and ImageNet.

  • Learning rates: For CIFAR10 and CIFAR100, the learning rate η for A is 0.1, while the learning rate ηs for W is 0.001.The ratio between the learning rates of W and A is set to 0.01.
  • Learning rates: For ImageNet, both η and ηs during optimization are set to 0.001.

C. More Parameter Comparison

The proposed Hinge method achieves lower Top-1 error rates than SSS across the reported operating points, with distillation further improving the result.

  • The proposed method forms a lower error bound than SSS across several parameter operating points.The comparison reports Top-1 error rates for both methods.
  • Without distillation loss, the Hinge method already outperforms SSS in Fig. 9.
  • Adding distillation loss further lowers the proposed method’s Top-1 error rate.

D. Layer-wise Compression Ratio

The paper reports layer-wise or block-wise compression ratios according to each architecture’s convolutional structure. The proposed method tends to compress shallow layers more than deep layers in several networks, while ResNet56 exhibits a sawtooth pattern.

  • Compression granularity by architecture: ResNeXt uses two 1 × 1 convolutions as sparsity-inducing matrices to compress the residual block’s 3 × 3 convolution, so compression is reported block-wise.
  • Figure scope: Figures 7 and 8 show layer-wise or block-wise compression ratios, with Fig. 8 reporting CIFAR100 results.
  • Compression granularity by architecture: WRN and ResNet use one sparsity-inducing matrix for each of their two 3 × 3 convolutions per residual block, so compression is reported layer-wise.
  • Observed compression patterns: For WRN, ResNeXt164, and ResNet164, the approach tends to compress shallow layers more than deep layers.The passage states that this pattern is consistent with former research.
  • Observed compression patterns: For ResNet56, the proposed method produces a sawtooth compression architecture.
  • Figure scope: Figures 9 and 10 compare SSS with the proposed method using Top-1 error rates on CIFAR100 and CIFAR10/CIFAR100, respectively.
Loading 2003.08935v1…