PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Sparse Neural Networks through $L_0$ Regularization

Christos Louizos, Max Welling, Diederik P. Kingma

arXiv:1712.01312v2stat.MLcs.LG

TL;DR

Direct L0 regularization is combinatorial and intractable for large neural networks. The paper uses continuous stochastic gates and hard-concrete distributions to optimize sparse models; on CIFAR-10 and CIFAR-100, its wide residual networks improved accuracy over dropout equivalents while offering potential training-time speedup.

  • Problem

    Neural networks can be substantially overparameterized, while direct L0 regularization is combinatorial and intractable for large models.

    Source: §1 · p. 1

  • Method

    The paper smooths expected L0-regularized objectives with continuous gates and a hard-sigmoid, introducing hard-concrete gates by stretching binary-concrete samples.

    Source: §1 · pp. 1–2

  • Results

    On CIFAR-10 and CIFAR-100, L0-regularized wide residual networks improved accuracy over dropout equivalents while allowing potential training-time speedup.

    Source: §4.2 · p. 9

  • Takeaways & Limitations

    Increasing the regularization strength can trade a slight error increase for a larger potential speedup.

    Source: §4.2 · p. 9

    Effective use of learned sparsity for training very large neural networks remains a potential direction for future work.

    Source: §5 · p. 9

Abstract

from arXiv · show

We propose a practical method for $L_0$ norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization. AIC and BIC, well-known model selection criteria, are special cases of $L_0$ regularization. However, since the $L_0$ norm of weights is non-differentiable, we cannot incorporate it directly as a regularization term in the objective function. We propose a solution through the inclusion of a collection of non-negative stochastic gates, which collectively determine which weights to set to zero. We show that, somewhat surprisingly, for certain distributions over the gates, the expected $L_0$ norm of the resulting gated weights is differentiable with respect to the distribution parameters. We further propose the \emph{hard concrete} distribution for the gates, which is obtained by "stretching" a binary concrete distribution and then transforming its samples with a hard-sigmoid. The parameters of the distribution over the gates can then be jointly optimized with the original network parameters. As a result our method allows for straightforward and efficient learning of model structures with stochastic gradient descent and allows for conditional computation in a principled way. We perform various experiments to demonstrate the effectiveness of the resulting approach and regularizer.

1 INTRODUCTION

The introduction motivates sparsification as a way to reduce unnecessary computation and overfitting, then proposes a gradient-based framework for L0 regularization that preserves exact zeros. Its hard concrete gates are formed by stretching binary concrete samples and applying a hard-sigmoid.

1 INTRODUCTION

  • Deep networks can be pruned significantly without accuracy loss, yet may overfit or memorize random patterns without proper regularization (Han et al., 2015; Ullrich et al., 2017; Molchanov et al., 2017; Zhang et al., 2016).The introduction identifies unnecessary computation and resources as a consequence of overparameterization.Source: §1 · p. 1
  • Sparsification removes irrelevant degrees of freedom, reducing unnecessary computation and model complexity to alleviate overfitting.Source: §1 · p. 1
  • L0 regularization directly penalizes parameters for being nonzero without further restrictions, but its combinatorial optimization is intractable for large models.Source: §1 · p. 1
  • The proposed framework smooths the expected L0-regularized objective with continuous distributions, preserving exact zeros while enabling efficient gradient-based optimization.The method transforms continuous random variables with a hard nonlinearity.Source: §1 · p. 1
  • The hard concrete distribution stretches a binary concrete random variable and passes its samples through a hard-sigmoid.The paper reports demonstrating the procedure's effectiveness in various experiments.Source: §1 · p. 2

2 MINIMIZING THE L0 NORM OF PARAMETRIC MODELS

The section formulates L0 regularization as penalizing active parameters and develops continuous stochastic gates that make expected sparsity differentiable while retaining exact zeros.

2 MINIMIZING THE L0 NORM OF PARAMETRIC MODELS

  • L0 regularization penalizes the number of nonzero parameters without shrinking their values; AIC and BIC correspond to specific choices of its weighting factor.The section frames this penalty in regularized empirical risk minimization for parametric models.Source: §2 · p. 2

2.1 A GENERAL RECIPE FOR EFFICIENTLY MINIMIZING L0 NORMS

  • Replacing Bernoulli gates with hard-sigmoid-transformed continuous variables yields a differentiable expected L0 objective while preserving exact zeros for stochastic-gradient optimization.The gate’s probability of being nonzero is computed from the underlying variable’s cumulative distribution function.Source: §2.1 · pp. 3–4

2.2 THE HARD CONCRETE DISTRIBUTION

  • The hard concrete stretches binary concrete samples beyond [0, 1] before hard-sigmoid clipping, placing probability mass at 0 and 1 while retaining reparameterizable gradients.Its temperature controls how closely the binary concrete approximates a Bernoulli variable; at 0 it recovers the Bernoulli but loses differentiability.Source: §2.2 · p. 4

2.3 COMBINING THE L0 NORM WITH OTHER NORMS

  • The framework can combine L0 sparsity with L2 weight decay by making each parameter’s Gaussian prior scale depend on its gate, avoiding extra shrinkage for the gates.The proposed scale is 1 when the gate is zero and equals the gate when it is positive.Source: §2.3 · p. 5

2.4 GROUP SPARSITY UNDER AN L0 NORM

  • Sharing hard-concrete gates across groups enables group sparsity and practical computation savings; experiments use neuron gates per fully connected input neuron and convolutional output feature map.For convolutional layers, each gate is shared across all spatial locations of its feature map.Source: §2.4 · p. 6

3 RELATED WORK

Prior pruning methods generally train dense networks before sparsification, while this approach sparsifies during training; prior continuous relaxations and smoothing methods also differ from hard concrete in how well they represent binary variables.

3 RELATED WORK

  • Weight pruning (Han et al., 2015; Ullrich et al., 2017; Molchanov et al., 2017) is generally inefficient for computation savings, whereas neuron pruning (Wen et al., 2016; Louizos et al., 2017; Neklyudov et al., 2017) can save computation.The cited pruning methods require training the original dense network; sparsifying during training can enable conditional computation and potentially speed training (Bengio et al., 2013; 2015).Source: §3 · p. 6
  • Gaussian and logistic rectifications can represent exact zeros while enabling continuous optimization, but their unimodal distributions cannot capture Bernoulli bimodality needed for the L0 penalty.Prior examples include Gaussian approaches in generative modeling (Salimans, 2016) and logistic approaches to conditional computation (Bengio et al., 2013).Source: §3 · p. 6
  • Unlike Rolfe’s (2016) smoothing transformations, hard concrete’s double rectification represents exact zeros and ones, and its underlying concrete distribution better emulates binary random variables.Rolfe’s method smooths binary latent variables with continuous noise to enable reparameterization gradients.Source: §3 · p. 6

4 EXPERIMENTS

Experiments evaluate MNIST with an MLP and LeNet-5, and CIFAR-10/100 with Wide Residual Networks, assessing sparsification, computation, accuracy, and convergence.

4.1 MNIST CLASSIFICATION AND SPARSIFICATION

  • On MNIST, L0 minimization is competitive with neural-network compression methods and prunes more at layers where gates affect more computation, unlike sparsity-inducing priors that sparsify layers similarly.The more-pruned layers are the MLP input layer and LeNet-5 first fully connected layer; separate layer-specific λ values can shift sparsification toward selected layers.Source: §4.1 · p. 7, Table 1 · p. 7
  • MNIST results show potential FLOP reductions versus original and dropout architectures with minimal or no performance loss; LeNet-5’s λ sep. setting favors speed over parameter compression.Its larger λ values for convolutional layers, which contribute most computation, explain the lower FLOP count than λ = 0.1/N.Source: §4.1 · pp. 7–8, Figure 3 · p. 8

4.2 CIFAR CLASSIFICATION

  • With λ = 0.001/N, L0-regularized WideResNets improve upon dropout-equivalent accuracy on CIFAR-10 and CIFAR-100 while potentially reducing training FLOPs without lagging in convergence.Increasing λ further slightly increases error but can allow larger speedups; reported L0 test errors are medians over five runs after 200 epochs.Source: Table 2 · p. 8, Figure 4 · p. 8, §4.2 · p. 9

5 DISCUSSION

The paper presents hard-concrete gates as a way to optimize L0 regularization with gradients and reports competitive sparsification and improved results over dropout on CIFAR wide residual networks. It identifies conditional computation, Bayesian parameter modeling, and binary latent-variable models as directions for future work.

5 DISCUSSION

  • The method smooths combinatorial L0 optimization with continuous distributions and a hard-sigmoid, using a stretched binary concrete hard-concrete distribution that better mimics Bernoulli gates while enabling gradient-based optimization.The authors describe this as a general recipe for optimizing the L0 norm of parametric models.Source: §5 · p. 9
  • Experiments found neural-network sparsification competitive with current approaches, theoretically allowed training speedup, and improved upon dropout on CIFAR wide residual networks.The authors also describe the process as providing a good inductive bias and regularizer.Source: §5 · p. 9
  • Future work could exploit learned sparsity for efficient large-network conditional computation, add Bayesian weight and bit-precision learning, and test hard-concrete gates in binary latent-variable models.A full Bayesian treatment could enable further speedup and compression through automatic learning of each weight's bit precision; hard-concrete gates could preserve discrete latent variables while retaining reparameterization-gradient optimization.Source: §5 · p. 9

APPENDIX · A RELATION TO VARIATIONAL INFERENCE

Expected L0 minimization is a close surrogate for a variational bound under a spike-and-slab prior, with a fixed coding cost for active parameters. Optimizing the full bound instead also accounts for gate uncertainty and permits prior information about gate behavior.

A RELATION TO VARIATIONAL INFERENCE

  • The spike-and-slab prior, described as a sparsity gold standard, mixes a point mass at zero with a continuous distribution such as a standard normal.The passage specifies Bernoulli gates and a standard normal slab.Source: A RELATION TO VARIATIONAL INFERENCE · p. 11
  • Because the posterior under this prior is intractable, the method uses a factorized spike-and-slab variational approximation over parameters and gates.The approximate posterior factorizes across parameter dimensions.Source: A RELATION TO VARIATIONAL INFERENCE · pp. 11–12
  • The variational free energy includes a gate-distribution KL term and an active-parameter KL term interpreted as the parameter’s code cost.The latter measures information about the data relative to the active slab prior; the inactive-parameter KL is zero.Source: A RELATION TO VARIATIONAL INFERENCE · p. 12
  • Assuming optimization rather than integration over parameters and a fixed active-parameter KL of λ nats yields the coding-cost formulation.An empirical-Bayes interpretation lets the hypothetical active-parameter prior adapt to the approximate posterior; λ measures its flexibility, and λ = 0 permits overfitting.Source: A RELATION TO VARIATIONAL INFERENCE · p. 12
  • Expected L0 minimization matches the variational objective under a spike-and-slab prior when active parameters have a fixed coding cost.For uncertainty over gates, the full bound instead penalizes the entropy of q(z) and can incorporate prior information, such as gates being active 10% of the time on average.Source: A RELATION TO VARIATIONAL INFERENCE · pp. 11–12
  • Setting the negative log-probability equal to the loss makes the resulting variational objective the same as Eq. 3.The full bound is preferable when gate uncertainty matters because it penalizes q(z)'s entropy and can encode prior gate behavior.Source: A RELATION TO VARIATIONAL INFERENCE · p. 12

B THE HARD CONCRETE DISTRIBUTION

The hard concrete modifies the binary concrete by stretching its samples beyond [0, 1] and applying a hard-sigmoid, producing a distribution with point masses at zero and one and a continuous component between them.

B THE HARD CONCRETE DISTRIBUTION

C NEGATIVE KL-DIVERGENCE FOR HARD CONCRETE DISTRIBUTIONS

For a hard-concrete q(z), the prior must share its support, so it can be defined by applying a hard-sigmoid transformation to a continuous distribution. The KL can then be computed using the chain rule for the matching mixture components, with Monte Carlo estimation available when a truncated-distribution integral lacks a closed form.

C NEGATIVE KL-DIVERGENCE FOR HARD CONCRETE DISTRIBUTIONS

  • The prior p(z) must share q(z)'s support and assigns point masses at 0 and 1 plus a continuous component on (0, 1), with weights determined by the CDF of the underlying distribution.Specifically, the masses are P̄s(0) at 0 and 1 − P̄s(1) at 1; the continuous component has weight P̄s(1) − P̄s(0).Source: C NEGATIVE KL-DIVERGENCE FOR HARD… · p. 13
  • Because q(z) and p(z) are mixtures with the same number of components, their KL-divergence can be computed using the chain rule for relative entropy.Source: C NEGATIVE KL-DIVERGENCE FOR HARD… · p. 13
  • If the truncated-distribution integral is unavailable in closed form, it can be estimated by Monte Carlo sampling the truncated pre-rectified variable via inverse transform.The passage gives sampling on a (γ, ζ) interval as an example.Source: C NEGATIVE KL-DIVERGENCE FOR HARD… · p. 13
  • The rectifications are not invertible, so KL(q(z)||p(z)) is not equal to KL(q(s̄)||p(s̄)).Source: C NEGATIVE KL-DIVERGENCE FOR HARD… · p. 13

Related papers

Loading 1712.01312v2…