PaperlayerView exact revision on arXiv

Source-linked AI summary

DHP: Differentiable Meta Pruning via HyperNetworks

Yawei Li, Shuhang Gu, Kai Zhang, Luc Van Gool, Radu Timofte

arXiv:2003.13683v3cs.CVcs.LGeess.IV

TL;DR

Automatic pruning often relies on nondifferentiable searches, and fixed-size hypernetwork outputs cannot directly support layer-wise configuration search. DHP generates backbone weights from latent vectors and sparsifies them with ℓ1 regularization and proximal gradient; experiments report its potential and improved accuracy with lower computation complexity for pruned α=2 mobile networks versus α=1 models.

  • Problem

    Automatic pruning often relies on nondifferentiable reinforcement-learning or evolutionary search, while fixed-size hypernetwork outputs cannot directly support layer-wise configuration search.

    Source: §1 · pp. 2–3

  • Method

    DHP generates backbone weights from current- and previous-layer latent vectors, then sparsifies channel controls using ℓ1 regularization and proximal-gradient updates.

    Source: §1 · p. 3, §3.1 · p. 6

  • Results

    Experiments reveal the potential of automatic pruning as fine-grained architecture search; α=2 mobile models had higher accuracy but lower computation complexity than α=1 models.

    Source: §1 · p. 4, §5 · p. 14

  • Takeaways & Limitations

    Pruning all layers' latent vectors together automates layer configuration without requiring additional search machinery.

    Source: §1 · p. 3

    Proximal gradient may not be optimal for some problems, though the authors argue it works well here because channel-count search reduces candidate configurations.

    Source: §3.5 · p. 9

Abstract

from arXiv · show

Network pruning has been the driving force for the acceleration of neural networks and the alleviation of model storage/transmission burden. With the advent of AutoML and neural architecture search (NAS), pruning has become topical with automatic mechanism and searching based architecture optimization. Yet, current automatic designs rely on either reinforcement learning or evolutionary algorithm. Due to the non-differentiability of those algorithms, the pruning algorithm needs a long searching stage before reaching the convergence. To circumvent this problem, this paper introduces a differentiable pruning method via hypernetworks for automatic network pruning. The specifically designed hypernetworks take latent vectors as input and generate the weight parameters of the backbone network. The latent vectors control the output channels of the convolutional layers in the backbone network and act as a handle for the pruning of the layers. By enforcing $\ell_1$ sparsity regularization to the latent vectors and utilizing proximal gradient solver, sparse latent vectors can be obtained. Passing the sparsified latent vectors through the hypernetworks, the corresponding slices of the generated weight parameters can be removed, achieving the effect of network pruning. The latent vectors of all the layers are pruned together, resulting in an automatic layer configuration. Extensive experiments are conducted on various networks for image classification, single image super-resolution, and denoising. And the experimental results validate the proposed method.

1 Introduction

Automatic pruning can produce layer-wise configurations, but reinforcement-learning and evolutionary searches raise convergence concerns, and fixed-size hypernetwork outputs do not directly support configuration search. DHP proposes a differentiable hypernetwork approach that prunes through latent vectors.

1 Introduction

  • Reinforcement-learning and evolutionary pruning searches raise convergence concerns, motivating differentiable alternatives for efficient architecture search.The introduction notes reinforcement learning's difficulty with large or middle-sized state spaces and evolutionary search's difficulty selecting among unconverged populations.Source: §1 · p. 2
  • Existing hypernetworks for compression emit fixed-size weights, so they cannot search layer-wise configurations without an additional algorithm.The paper frames input-dependent hypernetwork output size as a way to prune the backbone by operating on latent vectors.Source: §1 · pp. 2–3
  • DHP uses latent vectors to control layer channels and generate backbone weights, then applies ℓ1 regularization and proximal gradient to obtain pruned weights.The method uses the current and previous layers' latent vectors to control the current layer's output and input channels, respectively.Source: §1 · pp. 3–4, Figure 2 · p. 3
  • Automatic pruning can further compress efficient networks into layer-wise distinguishing configurations, extending their architecture search beyond uniform building blocks.The introduction discusses this possibility for efficient networks designed by experts or NAS.Source: §1 · pp. 3–4
  • Experiments on vision tasks and modern CNNs reveal the potential of automatic pruning as fine-grained architecture search.The contribution statement names experiments across various vision tasks and modern convolutional networks.Source: §1 · p. 4

2 Related Works

Prior pruning work includes structured regularization, automated configuration search, and hypernetwork-based methods. The paper positions its approach as a differentiable pruning design built on these lines of work.

2 Related Works

  • Structured pruning reduces feature maps, but its regularization strategies often require hyperparameters tuned to a specific architecture and dataset.Unstructured pruning can reduce parameters but yields irregular weights that limit actual acceleration.Source: §2 · p. 4
  • AutoML pruning explores network configurations using reinforcement learning, genetic algorithms, or hypernetworks, but the cited searches are nondifferentiable.The related-work section says nondifferentiable search does not guarantee convergence.Source: §2 · pp. 4–5
  • NAS has shifted from costly whole-network searches toward cell-based and differentiable methods, with DARTS cited as enabling faster search convergence.The related-work passage contrasts early resource demands with later cell-based and differentiable approaches.Source: §2 · p. 5
  • Hypernetworks predict backbone weights and appear in NAS, multi-task learning, Bayesian networks, and pruning; this paper designs one specifically for differentiable pruning.The paper distinguishes its pruning-oriented design from the broader applications of hypernetworks.Source: §2 · p. 5

3 Methodology

DHP uses latent vectors and hypernetworks to generate convolutional weights whose dimensions track the vectors, so sparsifying the vectors prunes channels. Joint ℓ1 regularization and proximal-gradient updates automate layer configuration; the method also shares vectors across connected residual layers and has a stated boundary on when proximal gradient is suitable.

3.1 Hypernetwork design

  • The hypernetwork’s output dimensions track its latent-vector inputs, so pruning vector elements removes corresponding weight slices and supports automatic channel configuration.It processes current- and previous-layer latent vectors through latent, embedding, and explicit layers; the resulting tensor supplies the convolutional weights.Source: §3.1 · pp. 6–7, Figure 3 · p. 6

3.2 Sparsity regularization and proximal gradient

  • Latent vectors receive joint ℓ1 sparsity regularization and proximal-gradient updates, while SGD updates the hypernetwork parameters.The ℓ1 proximal operator is soft-thresholding; the authors describe the combined procedure as approximately differentiable and faster-converging than reinforcement learning and evolutionary algorithms.Source: §3.2 · pp. 7–8
  • Proximal gradient drives potential latent-vector elements toward zero without manual selection, and optimization stops when the target compression ratio is reached.The algorithm uses two additional hyperparameters: the sparsity regularization factor and mask threshold τ.Source: §3.2 · p. 8

3.3 Network pruning

  • A threshold τ converts sparse latent vectors into masks, which prune the vectors and the corresponding generated weight slices.The mask returns 1 for an element not smaller than τ and 0 otherwise.Source: §3.1 · p. 7, §3.3 · p. 8

3.4 Latent vector sharing

  • For skip-connected layers, sharing one latent vector prunes correlated layers together while leaving their hypernetwork weights and biases unshared.The authors report that using separate vectors with group sparsity produced lower accuracy than latent-vector sharing.Source: §3.4 · pp. 8–9

3.5 Discussion on the convergence property

  • Proximal gradient may not be optimal for every problem; the authors say it works well when searching from scratch makes channel counts more important than filter identity.In that setting, the number of possible candidates is reduced by orders of magnitude.Source: §3.5 · p. 9

3.6 Implementation consideration

  • The implementation expresses hypernetwork operations as compact tensor operations, and applying masks to latent vectors has the same pruning effect as applying them to the output.The pruning analysis omits bias terms because the authors say they have little influence on the hypernetwork output.Source: §3.6 · pp. 9–10

4 Experimental Results

Experiments evaluate DHP on image classification, super-resolution, and denoising. The reported results show competitive accuracy and reductions in practical computation resources across these tasks, with classification outcomes also varying by network and compression setting.

4 Experimental Results

  • Experiments cover ResNet and DenseNet classification, MobileNet classification, SRResNet and EDSR super-resolution, and DnCNN and UNet denoising.Pruning begins from randomly initialized networks, after which training continues under the original network’s training protocol.Source: §4 · p. 10

4.1 Image classification

  • On Tiny-ImageNet, DHP achieves lower Top-1 error rates than MetaPruning under the same FLOPs constraint; ResNet-56 reaches 6.42% Top-1 error at 50% compression.DHP-10 is 1.56% lower in error than MobileNetV2-0.3 with slightly fewer FLOPs and 5.14% fewer parameters; DHP-50 gains 2.59% accuracy over MobileNetV1-0.75 with over 5% fewer FLOPs and 10% fewer parameters.Source: §4.1 · pp. 11–12, Table 1 · p. 11

4.2 Super-resolution

  • DHP reduces practical computation resources while providing inference-efficient and accuracy-preserving super-resolution models.Across operating points, FLOPs and parameter counts change consistently; DHP-20 is inference-efficient and DHP-60 preserves accuracy.Source: Table 2 · p. 12, §4.2 · p. 13

4.3 Denoising

  • DHP reduces computation resources in denoising, while Group fails to reduce actual resource use despite reducing FLOPs and parameters.For DnCNN, Factor is more accurate than the baseline but uses more resources; Factor-SIC3 has five times as many convolutional layers as the baseline.Source: §4.3 · pp. 13–14, Table 3 · p. 13

5 Conclusion and Future Work

The paper concludes that DHP combines hypernetworks with proximal-gradient search and uniformly applied latent-vector sparsity to prune networks automatically. Pruning widened mobile networks produced models with higher accuracy and lower computation complexity than width multiplier α = 1; whether this recurs on other networks remains future work.

5 Conclusion and Future Work

  • Pruning mobile networks with width multiplier α = 2 produced models with higher accuracy and lower computation complexity than those with α = 1.The authors hypothesize that per-layer distinguishing configurations from automatic pruning explain this outcome and leave testing its recurrence on other networks for future work.Source: §5 · p. 14

Supplementary Material for “DHP: Differentiable Meta Pruning via

The supplementary material provides training and testing protocols, a latent-vector sharing strategy, and additional results.

Supplementary Material

  • The supplement is organized around task-specific training and testing protocols, latent-vector sharing, and additional results.It identifies these topics as Sections A, B, and C, respectively.Source: Supplementary Material for “DHP:… · p. 19

A Training and Testing Protocol

The protocol trains and prunes networks from scratch, then removes the hypernetworks and continues training the pruned backbone.

A Training and Testing Protocol

  • Networks are trained and pruned from scratch; after pruning, hypernetworks are removed and backbone training continues under the original network’s protocol.Proximal gradient sparsifies latent vectors, and pruning stops when target and actual FLOPs compression ratios differ by less than 2%.Source: A Training and Testing Protocol · p. 19

A.1 Image Classification

The image-classification protocol specifies CIFAR10 data preparation and training, and also describes a Tiny-ImageNet comparison.

A.1 Image Classification

  • CIFAR10 has 10 classes, with 50,000 training and 10,000 testing images at 32 × 32 resolution.Images are normalized using training-set channel-wise means and standard deviations, with standard data augmentation.Source: CIFAR10 · p. 19
  • Networks are trained for 300 epochs with SGD, momentum 0.9, weight decay 0.0001, and batch size 64.The initial learning rate is 0.1 and is reduced by a factor of 10 after 50% and 75% of training.Source: CIFAR10 · p. 19

CIFAR10

  • The pruning method is also compared on Tiny-ImageNet, which has 200 classes and 500 training images per class.The passage also reports 50 validation images per class.Source: Tiny-ImageNet · p. 19

Tiny-ImageNet

The Tiny-ImageNet protocol specifies dataset preparation, augmentation, and a 220-epoch SGD schedule.

Tiny-ImageNet

  • The images have a resolution of 64 × 64 pixels and are normalized using channel-wise mean and standard deviation; the cited setup reports 50 validation images and uses horizontal-flip augmentation.Source: Tiny-ImageNet · p. 20
  • Horizontal flipping is used to augment the normalized Tiny-ImageNet images.Source: Tiny-ImageNet · p. 20
  • Networks are trained for 220 epochs with SGD, momentum 0.9, weight decay 0.0001, and batch size 64.The initial learning rate is 0.1 and decays by a factor of 10 at epochs 200, 205, 210, and 215.Source: Tiny-ImageNet · p. 20

Training protocol

The super-resolution protocol uses DIV2K for training and evaluates networks on five named test sets plus DIV2K validation.

Training protocol

  • DIV2K provides 800 training, 100 validation, and 100 test images for super-resolution training and evaluation.Source: Training protocol · p. 20
  • Low-resolution training patches are 48 × 48 for EDSR and 24 × 24 for SRResNet, with batch size 16.Source: Training protocol · p. 20
  • Networks use Adam with weight decay 0.0001 for 300 epochs, starting at learning rate 0.0001 and decaying it by 10 after 200 epochs.Testing uses Set5, Set14, B100, Urban100, and the DIV2K validation set.Source: Training protocol · p. 20

Simplified EDSR architecture

The appendix details training settings and architecture-specific latent-vector rules for pruning convolutional networks, including residual, dense, inverted residual, and super-resolution blocks.

Simplified EDSR architecture

A.3 Denoising

  • Denoising uses gray DIV2K patches, Gaussian noise at σ = 70, and 600,000 total training iterations.DnCNN uses 64 × 64 patches and batch size 64; UNet uses 128 × 128 patches and batch size 16.Source: §A.3 · p. 20

B.1 Basic criteria

  • Latent vectors attach to convolutional layers, control pruning, and are shared across consecutive layers; selected vectors, including the input-channel vector, are not sparsified.For depth-wise and group convolution, the vector controls input channels per group; group count reduction depends on the previous layer's latent vector.Source: §B.1 · p. 21

B.2 Residual block

  • A shared latent vector prunes the second convolutions across residual blocks within a stage, reflecting their correlated output channels from skip connections.Source: §B.2 · pp. 21–22

B.3 Dense block

  • A dense-block hypernetwork receives latent vectors from the corresponding earlier dense blocks in the same stage.Dense blocks concatenate their input and output, so each block receives outputs from preceding blocks in that stage.Source: §B.3 · p. 22

B.4 Inverted residual block

  • In an inverted residual block, the first point-wise convolution's latent vector also controls inputs to the depth-wise and second point-wise convolutions.The block's sequence is point-wise, depth-wise, then point-wise convolution.Source: §B.4 · p. 22

B.5 Upsampler of super-resolution networks

  • For ×4 super-resolution, each upsampler doubles spatial resolution, and one shared latent vector links input channels to four corresponding output channels.The vector is repeated and interleaved to control the convolution's output channels before pixel shuffling.Source: §B.5 · p. 22
  • On ResNet56 for CIFAR10, the table reports that latent-vector sharing consistently outperforms non-sharing strategies.The comparison uses ℓ1 regularization with sharing and ℓ2,1 group sparsity without sharing.Source: Table S1 · p. 23

C More Results

Additional results show that DHP retains reasonable accuracy under extreme ResNet compression, while latent-vector sharing generally outperforms non-sharing and ℓ1 regularization converges faster than ℓ2. The ℓ2 results are comparable on ResNet-110 and ResNet-164, but use a larger parameter budget.

C More Results

  • Latent-vector sharing consistently outperforms non-sharing except at λ = 2^-4, τ = 5^-3 and 50% target FLOPs compression, where achieved parameter compression ratios differ.The study develops different sharing rules for easier and better automatic pruning.Source: C More Results · p. 23
  • On ResNet-110 and ResNet-164, ℓ2 regularization gives results comparable to ℓ1 at a larger parameter budget, while ℓ1 leads to faster convergence.The λ used for ℓ2 is 20 times larger than for ℓ1 to obtain reasonable convergence speed; Table S2 compares the regularizers on ResNet.Source: C More Results · p. 23, Table S2 · p. 24
  • About 90% of FLOPs and parameters are pruned at extreme compression; ResNet-164 retains 8.04% of its parameters with a reasonable accuracy drop.The figure plots top-1 error against FLOP and parameter compression ratios for ResNet-164 and ResNet-110.Source: C More Results · p. 24, Figure S1 · p. 24
Loading 2003.13683v3…