Source-linked AI summary
AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inference
Jian-Hao Luo, Jianxin Wu
TL;DR
Deep models incur substantial computational and storage costs, while prior filter-pruning pipelines separate channel selection from fine-tuning. AutoPruner makes selection an end-to-end trainable layer that produces binary pruning codes from activations. Experiments report better performance than prior pruning methods and training from scratch, with gradually erased filters supporting smaller accurate models.
Problem
Deep models have high computational and storage overhead, while existing filter-pruning methods treat filter selection and fine-tuning as separate steps.
Method
AutoPruner integrates filter selection into end-to-end model training and generates binary channel codes from previous activation responses to identify removable filters.
Results
AutoPruner outperforms previous state-of-the-art pruning methods and training from scratch on reported datasets and networks.
Takeaways & Limitations
Gradually erasing weak filters during fine-tuning yields a smaller, faster, and still accurate model within the reported experiments.
Abstract
from arXiv · showhide
Channel pruning is an important family of methods to speed up deep model's inference. Previous filter pruning algorithms regard channel pruning and model fine-tuning as two independent steps. This paper argues that combining them into a single end-to-end trainable system will lead to better results. We propose an efficient channel selection layer, namely AutoPruner, to find less important filters automatically in a joint training manner. Our AutoPruner takes previous activation responses as an input and generates a true binary index code for pruning. Hence, all the filters corresponding to zero index values can be removed safely after training. We empirically demonstrate that the gradient information of this channel selection layer is also helpful for the whole model training. By gradually erasing several weak filters, we can prevent an excessive drop in model accuracy. Compared with previous state-of-the-art pruning algorithms (including training from scratch), AutoPruner achieves significantly better performance. Furthermore, ablation experiments show that the proposed novel mini-batch pooling and binarization operations are vital for the success of filter pruning.
1. Introduction
AutoPruner integrates filter selection with fine-tuning in one end-to-end trainable system, automatically identifying removable filters during training. Experiments report stronger compression performance than prior methods and training from scratch.
- Motivation: Deep models impose substantial inference and storage costs, motivating pruning to reduce model size and computational cost.VGG16 has 138.34 million parameters and requires more than 30.94 billion FLOPs for one 224 × 224 image.
- Problem: Typical filter-pruning methods separately evaluate filter importance, remove weak filters, and fine-tune the pruned model.The paper identifies the separation of pruning and model training as a central limitation of the conventional three-stage pipeline.
- Method: AutoPruner integrates filter selection into model fine-tuning through a trainable layer that converts previous activations into a binary code.A zero code value makes the corresponding filter activation always zero, so that filter can be safely removed permanently.
- Results: Experiments on CUB200-2011 and ILSVRC-2012 report better performance than previous state-of-the-art pruning methods at similar or higher compression ratios.AutoPruner also achieves better accuracy than training from scratch, supporting more accurate small models.
- Method: The method gradually erases unimportant filters during fine-tuning, while filter-selection gradients help guide training of preceding convolution layers.The paper presents pruning and fine-tuning as mutually supportive within the integrated framework.
- Properties: AutoPruner supports adaptive compression ratios and simultaneous multi-layer compression within a single trainable model.Its loss encourages a predefined sparsity target while allowing the network to determine actual sparsity using accuracy and compression considerations.
2. Related Work
Prior pruning work includes connection-, filter-, and layer-level approaches, but conventional filter pruning separates channel selection from fine-tuning. AutoPruner instead combines selection with training and produces static pruning decisions.
- Pruning motivation: Pruning reduces model complexity, but redundancy that supports training can make models cumbersome for inference and oversized for smaller datasets.The related-work discussion frames pruning as a way to remove redundancy while retaining a trained model’s useful capacity.
- Pruning categories: Connection-level pruning can create irregular networks that fail to deliver practical inference speedups despite high sparsity.Structured pruning addresses this weakness by removing organized groups such as filters.
- Prior filter pruning: Filter-pruning methods select unimportant filters using criteria such as next-layer statistics, LASSO regression, or channel scaling factors.These approaches represent several strategies for estimating filter or channel importance.
- Research gap: Conventional three-stage algorithms treat channel selection and fine-tuning as separate steps, whereas the proposed approach uses information from uncompressed layers to guide pruning.The paper argues that a single end-to-end system is preferable to this separation.
- Comparison: Dynamic pruning methods use reinforcement learning or sampled network designs, while AutoPruner makes static decisions that remove zero filters permanently.AutoPruner therefore differs from approaches that adapt pruning decisions dynamically.
- AutoPruner framework: AutoPruner generates a binary channel code from activation tensors using pooling, a fully connected projection, and scaled sigmoid binarization.Increasing α makes the approximate code converge toward binary values, after which zero-indexed channels can be pruned.
3. Automatic Filter Pruner
AutoPruner integrates channel selection and fine-tuning in an end-to-end layer that gradually produces binary codes for safely removing filters. Mini-batch pooling, binarization, and sparsity control coordinate layer-consistent pruning toward a predefined compression target.
- Pipeline: AutoPruner takes post-activation responses as input, generates channel index codes, and progressively erases channels whose codes approach zero.Element-wise multiplication applies the codes to activations, enabling filters to be safely pruned after training.
- Pooling: Mini-batch average pooling combines activations from different images so each layer produces a consistent index code rather than an example-specific code.A subsequent 2 × 2 max-pooling step reduces memory and training time, while large spatial pooling can harm accuracy.
- Coding and binarization: Binarization converts approximate codes toward 0-1 values, ensuring that zero-coded filters can be removed without changing the resulting activations.The scaled sigmoid becomes binary as α increases, and gradual binarization lets pruning and fine-tuning promote each other.
- Sparsity control: A predefined preserved-filter ratio r guides compression, but the achieved ratio may vary because the network determines the final binary code.For r = 0.5, the reported actual ratio may be 0.52 or 0.48, reflecting adaptive network compression.
- Sparsity control: The compression objective combines classification loss with a sparsity-control term weighted by λ, which is adjusted according to the current compression ratio.When the ratio is far from its target, λ increases; once the expected code is reached, λ becomes zero so training can focus on classification.
- Implementation: The method can repeat the same operation across multiple layers, while initialization and α must be managed because input magnitudes differ between layers.The coding-layer weights use an initialization with 10× larger standard deviation than the tried MSRA initialization.
4. Experimental Results
Experiments evaluate AutoPruner against other pruning approaches on CUB200-2011 and ImageNet using VGG16 and ResNet-50, with experiments conducted in PyTorch on M40 GPUs.
- Datasets and models: AutoPruner is compared with state-of-the-art pruning approaches on the CUB200-2011 and ImageNet ILSVRC-12 datasets.The evaluation studies filter pruning across a fine-grained and a large-scale image-recognition task.
- Datasets and models: The experiments prune two widely used deep models, VGG16 and ResNet-50.All experiments were conducted using PyTorch on M40 GPUs.
4.1. FLOPs Computation
The paper measures CNN convolutional complexity using FLOPs, with an explicit convention for counting multiplication and addition operations and bias computation.
- FLOPs definition: FLOPs in a convolutional layer are computed as FLOPs = 2HW(CinK^2 + 1)Cout.Here H and W are spatial dimensions, Cin and Cout are input and output channels, K is kernel size, and the added 1 accounts for bias FLOPs.
- Counting convention: The calculation treats one vector multiplication as two FLOPs, counting both multiplication and addition.The authors recalculate FLOPs when other papers use a one-operation convention to support fair comparison.
4.2. CUB200-2011
On the challenging CUB200-2011 fine-grained recognition task, AutoPruner is trained and evaluated as an end-to-end filter-pruning method for VGG16. It is compared with random selection and ThiNet under matched pruning structures and compression settings.
- Dataset and setup: AutoPruner is trained layer by layer from conv1_1 through conv5_3, with its module appended to each current layer’s output.After each stage, zero-index filters and channels are removed, and the added AutoPruner layer is discarded.
- Comparisons: The comparison includes random filter selection and ThiNet, with baseline methods pruning the same number of filters as AutoPruner.This matching procedure is used to generate the same network structure across methods.
- Results: 0.655% higher top-1 accuracy is achieved by AutoPruner than ThiNet when r = 0.5.The paper reports that the advantage expands at r = 0.2, where more filters are discarded, and that both methods outperform random selection.
4.3. Ablation Study
The ablation study examines AutoPruner’s pooling and code-generation choices, its gradient pathway, and the sensitivity of its compression and scaling hyperparameters. Results emphasize the importance of propagating selection gradients and gradually forming binary codes.
- AutoPruner modules: The module ablation compares replacing max pooling with GAP and replacing AutoPruner with trainable scaling factors.The scaling-factor baseline generates binary codes with y = sigmoid(αλ), while other settings remain unchanged.
- AutoPruner modules: Scaling factors cannot propagate gradients back to previous convolution layers, unlike AutoPruner, whose codes are learned from previous-layer activations.The paper connects this difference to the training behavior of preceding layers.
- AutoPruner modules: At r = 0.2, the scaling-factor baseline may fail, leaving model FLOPs at 6.91B.The authors report that gradient information from channel selection helps previous convolution layers focus on preserved filters.
- Hyper-parameters: The compression ratio r is task-dependent and represents a tradeoff between model accuracy and inference speed.The paper treats r as a predefined pruning target rather than a learned parameter.
- Hyper-parameters: AutoPruner is robust to αstart when it is not too large, whereas an initially large value can make it binary immediately and degenerate to random selection.The experiments use αstop = 2; small αstart values gradually erase filters.
4.4. ImageNet ILSVRC-12
On ImageNet, AutoPruner is evaluated against filter-level pruning methods on VGG16 and ResNet-50 under reported FLOPs and validation-accuracy protocols. The results favor AutoPruner across several comparisons, including a block-pruning comparison with SSS.
- Evaluation protocol: ImageNet results compare AutoPruner with state-of-the-art filter-level pruning methods using single-view central-crop accuracy and FLOPs recalculated by Eq. 6.The reported FLOPs can therefore differ from values in the original ResNet and SSS papers.
- VGG16: For VGG16, AutoPruner achieves better results than SSS, RNP, and the three-stage channel-pruning method.The paper attributes the SSS comparison to AutoPruner’s gradient information flowing to previous layers.
- ResNet-50: For ResNet-50, AutoPruner achieves much better accuracy than ThiNet under the same pruning strategy.The paper also reports significantly higher accuracy than Channel Pruning and the same conclusion for SSS.
- Evaluation protocol: Theoretical speed-up ratios are computed from FLOPs and compared for the same model structure.The paper notes that FLOPs-based speed-up is a theoretical value.
- ResNet-50: 2% higher top-1 accuracy is achieved by AutoPruner than SSS with similar FLOPs in block pruning.This comparison is conducted because SSS prunes whole residual blocks on ResNet.
4.5. Consistency of Index Code
The consistency study tests whether AutoPruner produces one layer-level binary index code across different input images. Mini-batch pooling and adaptive binarization are presented as the mechanisms that enforce this behavior.
- Consistency requirement: The generated index code must be determined by a layer rather than by an individual input example.Activations from different images are transformed into a unique code for a specific layer.
- Consistency mechanism: Each index-code channel can be treated as a binary classification task over images.The adaptive sigmoid layer helps project examples into consistent positive or negative labels.
- Consistency mechanism: Removing the adaptive sigmoid causes inconsistent outputs, with a channel useful for some images but useless for others.The reported failure occurs when the network does not generate a unique output across examples.
- Consistency mechanism: Mini-batch pooling combined with binarization ensures consistency of the index code across different images.Batch-wise average pooling mixes information from different images before the subsequent binarization step.
- Binarization ablation: Without increasing α for binarization, the pruned ResNet-50 reaches only 9.154% top-1 validation accuracy before fine-tuning.More than 90% of generated index-code elements are around 0.2 rather than equal to zero, so removing those filters damages accuracy.
4.6. The Value of Network Pruning
Network pruning addresses the redundancy that slows deep-model inference while preserving accuracy. AutoPruner is reported to outperform training-from-scratch and existing pruning approaches in this context.
- Removing redundant parameters can reduce model size and computational cost, addressing the inference slowdown caused by over-parameterization.
- 73.90% top-1 accuracy was achieved by the ThiNet-50 baseline trained from scratch for 180 epochs, versus 74.76% for AutoPruner.
- AutoPruner provides a tool for accelerating model inference while preserving accuracy.
5. Conclusions
The paper concludes that AutoPruner is an end-to-end trainable filter-pruning method for CNN acceleration. It reports improved compression performance over existing pruning methods and training from scratch, while identifying broader vision-task evaluation as future work.
- AutoPruner is an independent layer that can be appended to convolution layers to prune filters automatically.
- AutoPruner significantly improves model compression performance over existing filter-pruning methods.
- AutoPruner outperforms training from scratch and can produce a smaller but still accurate model.
- Future work will evaluate AutoPruner on object detection and semantic segmentation.