PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Efficient Model Compression via Learned Global Ranking

Ting-Wu Chin, Ruizhou Ding, Cha Zhang, Diana Marculescu

arXiv:1904.12368v2cs.CVcs.LG

TL;DR

Embodied AI applications make it difficult to select a single model complexity because both ConvNet accuracy and speed matter, making trial-and-error pruning time-consuming. LeGR learns a global ranking of filters across layers to generate multiple accuracy/speed trade-offs, and it is reported as faster than prior work with comparable or better performance.

  • Problem

    Selecting a target model complexity for embodied AI is difficult because application performance depends on both ConvNet accuracy and speed.

  • Method

    LeGR learns a global ranking of convolutional filters across layers, then produces architectures with different trade-offs by pruning bottom-ranked filters.

  • Results

    LeGR is faster than prior pruning methods while achieving comparable accuracy with state-of-the-art methods across three datasets and two ConvNet types.

  • Takeaways & Limitations

    Learning the ranking once lets practitioners efficiently obtain ConvNet architectures with different accuracy/speed trade-offs.

  • Takeaways & Limitations

    The global-ranking formulation assumes that top-performing smaller ConvNets are proper subsets of top-performing larger ConvNets.

Abstract

from arXiv · show

Pruning convolutional filters has demonstrated its effectiveness in compressing ConvNets. Prior art in filter pruning requires users to specify a target model complexity (e.g., model size or FLOP count) for the resulting architecture. However, determining a target model complexity can be difficult for optimizing various embodied AI applications such as autonomous robots, drones, and user-facing applications. First, both the accuracy and the speed of ConvNets can affect the performance of the application. Second, the performance of the application can be hard to assess without evaluating ConvNets during inference. As a consequence, finding a sweet-spot between the accuracy and speed via filter pruning, which needs to be done in a trial-and-error fashion, can be time-consuming. This work takes a first step toward making this process more efficient by altering the goal of model compression to producing a set of ConvNets with various accuracy and latency trade-offs instead of producing one ConvNet targeting some pre-defined latency constraint. To this end, we propose to learn a global ranking of the filters across different layers of the ConvNet, which is used to obtain a set of ConvNet architectures that have different accuracy/latency trade-offs by pruning the bottom-ranked filters. Our proposed algorithm, LeGR, is shown to be 2x to 3x faster than prior work while having comparable or better performance when targeting seven pruned ResNet-56 with different accuracy/FLOPs profiles on the CIFAR-100 dataset. Additionally, we have evaluated LeGR on ImageNet and Bird-200 with ResNet-50 and MobileNetV2 to demonstrate its effectiveness. Code available at https://github.com/cmu-enyac/LeGR.

1. Introduction

Filter pruning can trade ConvNet accuracy for inference speed, but embodied AI applications make the desired complexity difficult to choose. LeGR addresses this by learning one global filter ranking that yields multiple accuracy/speed trade-offs for efficient exploration.

  • Embodied AI systems on mobile devices require model compression because compute resources are limited.
  • Filter pruning trades accuracy for inference speed by removing the least important convolutional filters.
  • Choosing a target model complexity is difficult because application performance depends on both ConvNet speed and accuracy.Relevant complexity measures include filter count, FLOP count, model size, and inference latency.
  • LeGR changes pruning from producing one pre-defined-complexity ConvNet to producing a set of ConvNets with different accuracy/speed trade-offs.This reduces repeated pruning overhead when practitioners search for a suitable trade-off.
  • LeGR learns a global ranking of filters across layers, then obtains different architectures by dropping bottom-ranked filters and fine-tuning them.The ranking is learned once and reused to generate architectures with different trade-offs.
  • LeGR is faster than prior pruning methods while achieving comparable accuracy with state-of-the-art methods across three datasets and two ConvNet types.

2. Related Work

Prior filter-pruning methods either rank filters locally or learn architectures under a specified complexity, leaving practitioners to choose that complexity. LeGR instead learns a global ranking to generate pruned ConvNets across complexity values.

  • Pre-defined architecture: Filter-pruning prior art includes weight-norm and output-discrepancy measures for ranking filters within individual layers.
  • Learned architecture: Other methods learn the pruned architecture using controllable complexity parameters, including regularization-based approaches.
  • Specifying a complexity parameter remains difficult in embodied AI applications, where practitioners need to explore different operating points.
  • Some prior methods can produce multiple complexities by repeatedly pruning one filter and fine-tuning before continuing.Examples use normalized Taylor approximation or Fisher information to select filters.
  • LeGR learns one data-driven global filter ranking across layers, enabling architectures with different complexities by removing bottom-ranked filters.

3. Learned Global Ranking

LeGR learns a global ranking of convolutional filters by calibrating layer-wise filter norms, then reuses that ranking to generate pruned ConvNets across FLOP budgets. The formulation relies on subset and norm assumptions and learns the calibration through optimization before pruning and fine-tuning.

  • Learning the Ranking: FLOP count is used to sample ConvNets because the authors find it predictive of latency, supporting exploration of accuracy-versus-speed trade-offs.The method’s workflow learns one ranking at a chosen lowest FLOP count and applies it to other FLOP budgets.
  • Assumptions: LeGR assumes smaller optimal ConvNets contain no more filters in any layer than larger optimal ConvNets.This subset assumption makes one global ranking usable across different FLOP counts.
  • Assumptions: The norm assumption allows ℓ2 norms to rank filters within layers but not compare their importance across layers.The method therefore learns layer-wise transformations to make filter norms globally comparable.
  • Global Ranking: LeGR defines filter importance by applying learnable layer-wise scale and shift parameters to each filter’s ℓ2 norm.The scale and shift parameters are α and κ, indexed by layer; learning them is intended to maximize overall accuracy.
  • Pruning: After learning α and κ, LeGR globally ranks filters, removes the bottom-ranked filters until a target FLOP count is met, and fine-tunes the resulting network.The ranking can produce architectures at multiple FLOP counts without repeating the ranking-learning process or using training data during pruning.
  • Learning the Ranking: LeGR learns α and κ as an optimization problem using validation accuracy from a low-FLOP pruned network as candidate fitness.The search uses a regularized evolutionary algorithm with candidate pools, subset sampling, and random-walk mutations over selected layers.

4. Evaluations

LeGR is evaluated across standard image-classification and transfer-learning benchmarks, comparing its accuracy–speed trade-offs and pruning cost with prior methods. It learns one ranking and reuses it across FLOP counts, producing competitive models while reducing repeated search overhead.

  • 4.1. Datasets and Training Setting: LeGR prunes filters across convolutional layers, jointly handling dependent channels from depth-wise convolutions and residual connections.Importance is measured after learned affine transformations, and grouped channels are pruned together.
  • 4.2. CIFAR-100 Results: On CIFAR-100, LeGR outperforms prior methods especially in the low-FLOP-count regime, while AMC and MorphNet show higher variance under aggressive pruning.FisherPruning performs worse than the other methods in this comparison.
  • 4.2. CIFAR-100 Results: 7× and 5× faster pruning than AMC and MorphNet, respectively, results from learning the α-κ pair once and reusing it across FLOP counts.The reported cost compares obtaining seven pruned ResNet-56 models across FLOP counts, with costs normalized to LeGR.
  • 4.3. Comparison with Prior Art: On CIFAR-10, LeGR outperforms most prior art for ResNet-56 across FLOP count and accuracy dimensions and performs similarly to two cited methods.For VGG-13, the paper reports significantly better results than prior art.
  • 4.3. Comparison with Prior Art: On ImageNet, LeGR is better or comparable to state-of-the-art pruning methods for ResNet-50 and MobileNetV2 while generating models across different FLOP counts.The ranking is learned at one FLOP count for each model and reused for other FLOP counts of interest.
  • 4.4. Transfer Learning: Bird-200: On Bird-200, LeGR outperforms Uniform and AMC when pruning fine-tuned MobileNetV2 and ResNet-50 models.The corresponding unpruned fine-tuned models achieve top-1 accuracies of 80.2% and 79.5%, respectively.

5. Ablation Study

The ablations examine how LeGR behaves across ranking-learning settings, FLOP constraints, and runtime measurements. The results indicate robustness to the minimum FLOP count and approximate linear translation from FLOP reduction to latency reduction.

  • Figures: Figure 6 compares robustness to ˆζl against prior art, while Figure 8 relates latency reduction to FLOP count reduction.The caption states that FLOP count reduction is indicative of latency reduction.
  • Ranking Performance and ˆζl: Rankings learned with different minimum FLOP counts have similar performance, empirically supporting Assumption 1.The experiment targets ResNet-56 on CIFAR-100 with ˆζl ∈ {20%, 40%, 60%, 80%}.
  • Learning α and κ: Learning α and κ is evaluated by varying ˆτ and FLOP count constraints when pruning ResNet-56 on CIFAR-100.The resulting networks are fine-tuned for τ = 21120 gradient steps, corresponding to 60 epochs.
  • FLOP Count and Runtime: Filter pruning produces near-linear acceleration, with a slope of approximately 0.6, without specialized software or hardware support.Wall-clock measurements use ResNet-50 and MobileNetV2 on Intel i7 and ARM A57 CPUs with single-threaded PyTorch 0.4 execution.

6. Conclusion

LeGR is presented as a formulation for efficiently exploring accuracy-versus-speed trade-offs through filter pruning. It learns layer-wise affine transformations of filter norms to construct a global ranking and obtains architectures across FLOP counts with less training cost and comparable state-of-the-art performance.

  • Conclusion: LeGR learns layer-wise affine transformations over filter norms to construct a global filter ranking.The formulation addresses the difficulty of comparing filter norms across layers in a learnable fashion.
  • Conclusion: LeGR enables practitioners to obtain ConvNet architectures with different FLOP counts through one learned ranking.The paper evaluates the formulation with ResNet and MobileNetV2 on CIFAR, Bird-200, and ImageNet.
  • Conclusion: LeGR requires less training cost across different FLOP counts while achieving comparable performance to state-of-the-art pruning methods.The conclusion also provides a theoretical interpretation of the affine-transformation formulation.

A. Optimization Interpretation of LeGR

The optimization interpretation casts LeGR as minimizing a surrogate upper bound on the loss difference between a pruned-and-fine-tuned CNN and its pretrained counterpart. It learns affine-ranking parameters, then thresholds low-scoring filters subject to a FLOP constraint.

  • Optimization Objective: LeGR optimizes binary filter masks to limit the loss difference between pruned-and-fine-tuned and pretrained CNNs.A mask value of zero prunes a filter, while one retains it.
  • Optimization Objective: The loss uses training data, the CNN model, prediction loss, learning rate η, fine-tuning steps τ, gradients, and element-wise multiplication.Fine-tuning updates filter weights with stochastic gradient descent for τ steps.
  • FLOP Constraint: The FLOP constraint is modeled by C(·), with ζ specifying the desired FLOP count.The layer FLOP count depends linearly on the number of filters in its preceding layer.
  • Surrogate Objective: LeGR minimizes a Lagrangian upper bound rather than the original constrained objective.This surrogate is derived from the stated loss and FLOP-count relations.
  • Ranking and Pruning: Filters receive scores s_i = α_l(i) ∥Θ_i∥ + κ_l(i), and the lowest-scoring filters are removed until C(1 − h) ≤ ζ while maximizing ∥h∥_0.LeGR learns α and κ as estimates intended to improve the surrogate solution.

B. LeGR-DDPG

This ablation compares DDPG with evolutionary optimization for learning LeGR’s layer-wise affine transformations. LeGR converges faster to a better solution than AMC and exhibits lower final-network variance with evolutionary optimization than with DDPG.

  • LeGR-DDPG: LeGR converges faster to a better solution than AMC when pruning ResNet-56 to 50% of its original FLOP count on CIFAR-100.Both methods outperform random search during the reported search progress.
  • LeGR-DDPG: Searching layer-wise affine transformations is more efficient and effective than searching layer-wise filter percentages.The comparison evaluates final pruned-network performance as well as search progress.
  • LeGR-DDPG: DDPG incurs larger variance on the final network than the evolutionary algorithm when optimizing LeGR’s affine transformations.LeGR uses two continuous actions per layer, α_l and κ_l, whereas AMC uses one percentage action.

C. ImageNet Result Detail

This section compares LeGR with prior pruning methods on ImageNet and describes a figure analyzing two layer-wise search strategies for ResNet-56.

  • The ImageNet comparison presents LeGR alongside prior pruning methods.
  • Table 2 summarizes ImageNet pruning results, organizing methods by the FLOP count remaining.The table reports accuracy as pretrained → pruned-and-fine-tuned.
  • Figure 9 compares searching layer-wise filter norms with searching layer-wise filter percentages for ResNet-56.Panel (a) shows search progress at 50% FLOP count, while panel (b) compares final performance across constraint levels.
Loading 1904.12368v2…