PaperlayerView exact revision on arXiv

Source-linked AI summary

Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

Yoshua Bengio, Nicholas Léonard, Aaron Courville

arXiv:1308.3432v1cs.LG

TL;DR

The paper asks how to train through stochastic, nondifferentiable neurons when exact gradients are impractical. It compares four estimator families, introduces STS units, and finds that all tested methods support training while conditional computation achieves expected savings.

  • Problem

    The paper addresses how to estimate gradients through stochastic binary decisions when exact averaging over exponentially many configurations is impractical.

  • Method

    The paper compares four gradient-estimation families and introduces STS units, which combine stochastic and smooth operations for ordinary back-propagation.

  • Results

    All tested methods allow training to proceed, and conditional computation achieves the expected computational saving.

  • Takeaways & Limitations

    Stochastic gradient estimators can support sparse gating for conditional computation, including settings without a back-propagated signal into the gater.

  • Takeaways & Limitations

    Inactive units can remain inactive for long periods, causing units to become useless unless their firing rates are actively adjusted.

Abstract

from arXiv · show

Stochastic neurons and hard non-linearities can be useful for a number of reasons in deep learning models, but in many cases they pose a challenging problem: how to estimate the gradient of a loss function with respect to the input of such stochastic or non-smooth neurons? I.e., can we "back-propagate" through these stochastic neurons? We examine this question, existing approaches, and compare four families of solutions, applicable in different settings. One of them is the minimum variance unbiased gradient estimator for stochatic binary neurons (a special case of the REINFORCE algorithm). A second approach, introduced here, decomposes the operation of a binary stochastic neuron into a stochastic binary part and a smooth differentiable part, which approximates the expected effect of the pure stochatic binary neuron to first order. A third approach involves the injection of additive or multiplicative noise in a computational graph that is otherwise differentiable. A fourth approach heuristically copies the gradient with respect to the stochastic output directly as an estimator of the gradient with respect to the sigmoid argument (we call this the straight-through estimator). To explore a context where these estimators are useful, we consider a small-scale version of {\em conditional computation}, where sparse stochastic units form a distributed representation of gaters that can turn off in combinatorially many ways large chunks of the computation performed in the rest of the neural network. In this case, it is important that the gating units produce an actual 0 most of the time. The resulting sparsity can be potentially be exploited to greatly reduce the computational cost of large deep networks for which conditional computation would be useful.

1 Introduction and Background

Gradient-based learning favors smooth computational graphs, yet successful rectifier networks and useful stochastic binary units show that hard or non-smooth operations remain important. The paper motivates gradient estimation for stochastic neurons by their roles in sparse representations, conditional computation, and other discrete decisions.

  • Smoothness and gradient-based learning: Gradient-based learning is practical when the relationship between parameters and the objective is continuous and generally smooth.Piecewise-constant relationships are mostly flat, making gradient-based learning impractical.
  • Smoothness and gradient-based learning: Successful deep networks using rectifiers and other non-smooth nonlinearities challenge smoothness as a necessary condition for exact gradient-based training.These results question the previously common assumption that computational graphs must be smooth.
  • Gradient estimation for stochastic neurons: Stochastic binary neurons output 1 or 0 according to a sigmoid probability, but their exact expected-loss gradient may require averaging over exponentially many configurations.The difficulty arises when averaging the loss over all possible binary samplings in a neural network.
  • Motivations and applications: Binary stochastic outputs support sparse representations, while sparse gating units can activate only a small subset of model parameters for each example.This conditional computation can greatly reduce per-example computation by selecting which model components need to be visited.
  • Prior gradient-estimation approaches: Perturbation-based estimators are more efficient than finite differences, whose gradient cost is O(N^2), but become unbiased only as perturbations approach 0.The text questions whether this limiting assumption suits all-or-none neuronal perturbations.

2 Non-Smooth Stochastic Neurons

The section explains how stochastic neurons can support gradient-based learning when their transformations retain non-zero derivatives, while binary outputs block gradients almost everywhere. It introduces sensitive and insensitive states, analyzes noisy rectifiers, and presents STS units as a trainable stochastic alternative with controlled expected behavior.

  • Gradient flow and non-smoothness: Gradient-based learning can proceed when stochastic-neuron outputs depend differentiably on inputs and injected noise, including additive or multiplicative noise such as Gaussian, dropout, or masking noise.Binary nonlinearities instead have zero derivatives almost everywhere and infinite derivatives at thresholds, preventing gradients from flowing.
  • Sensitive and insensitive states: The proposed intermediate design gives neurons insensitive regions with zero derivatives and sensitive regions with significantly non-zero derivatives.This preserves sparse or shut-off behavior while allowing gradients to flow in selected regions.
  • Noisy rectifiers: For a noisy rectifier with logistic noise, P(hi > 0) = sigm(ai) and E[hi] = s+(ai), where s+(x) = log(1 + exp(x)).The unit is therefore sparsely active while its expected output follows the softplus function.
  • Noisy rectifiers: Even when the basic state is insensitive and zero, occasional gradient signals can draw the unit out of inactivity and provide training information.This mechanism allows gradients to flow intermittently rather than continuously, unlike sigmoid or tanh units.
  • Noisy rectifiers: Inactive units can remain dead for long periods because they rarely receive signals to activate, whereas active units can randomly enter the death zone and become stuck.The section identifies an asymmetry between opportunities to learn inactivity and opportunities to recover from inactivity.
  • STS units: STS units are stochastic units trainable by ordinary back-propagation, with E[hi] = pi and P(hi > 0) = sigm(ai).Their expected behavior is established through the injected noise and the unit definition, with further properties derived for differentiable functions of hi.

3 Unbiased Estimator of Gradient for Stochastic Binary Neurons

For stochastic binary neurons, the gradient of the expected loss can be estimated unbiasedly with (h_i − sigm(a_i))L. Subtracting a unit-specific baseline preserves unbiasedness and can minimize estimator variance.

  • Unbiased gradient estimator: The estimator ĝ_i = (h_i − sigm(a_i)) × L is unbiased for the gradient of the expected loss with respect to a_i.Its expectation over the relevant noise sources equals the desired derivative.
  • Relation to REINFORCE: The estimator is a special case of REINFORCE for Bernoulli stochastic units whose activation probability is given by a sigmoid.REINFORCE expresses the sampled output as a weighted maximum-likelihood target, with weights proportional to reward.
  • Variance reduction: A centered estimator, (h_i − sigm(a_i))(L − L̄_i), remains unbiased for any constant baseline L̄_i.The baseline term has zero expectation because E_z_i[h_i|a_i] = sigm(a_i).
  • Variance reduction: The minimum-variance baseline L̄_i is a unit-specific weighted average of the loss values.Its weights depend on stochastic unit i.
  • Implementation: The lowest-variance estimator can be implemented using two running or moving averages per stochastic neuron, while requiring only broadcasting L and local computation.It does not require back-propagation.

4 Straight-Through Estimator

The straight-through estimator treats a hard threshold as the identity during back-propagation, yielding a biased gradient estimate that has the correct sign for a single neuron layer.

  • Straight-Through Estimator: The straight-through estimator back-propagates through the hard threshold function as if it were the identity.The hard threshold outputs 1 for positive arguments and 0 otherwise.
  • Straight-Through Estimator: Although biased, the estimator has the correct gradient sign for a single neuron layer, but this guarantee can fail across multiple hidden layers.The estimator is explicitly described as biased, with the sign property limited to single-layer back-propagation.
  • Straight-Through Estimator: The estimated gradient is back-propagated further to obtain gradients for parameters influencing a_i.This extends the estimator from the stochastic neuron output to relevant model parameters.

5 Conditional Computation Experiments

The experiments apply stochastic gating to conditional computation, selectively turning off hidden units while enforcing sparse activation. They evaluate this architecture against rectifier and sigmoid baselines under comparable computational constraints.

  • Architecture: Stochastic gating selectively turns off hidden units by multiplying each hidden activation H_i by its gate h_i.The architecture uses gating units to choose which parts of the computational graph are computed for each input.
  • Results: Figure 2 reports MNIST validation learning curves and training, validation, and test classification errors for stochastic and baseline gaters.The figure presents classification error on the validation set and percentage errors for the training criteria and validation/test comparisons.
  • Experimental setup: The experiment uses 400 gater hidden units, 2000 gater outputs, 2000 main-path hidden units, and a 10% average nonzero-activation constraint.Stochastic-model validation and test errors are evaluated after optimizing a deterministic testing threshold.
  • Baselines: The comparison includes a 10%-sparse Baseline Rectifier, an unconstrained 200-output Baseline Sigmoid, and a noisy sigmoid baseline with Gaussian training noise.The sigmoid baselines share the stochastic models’ sigmoid gater and tanh hidden-unit structure, while differing in sparsity, scale, and noise.

6 Conclusion

The paper motivates gradient estimators for highly nonlinear, nondifferentiable functions in stochastic neural networks, including conditional computation. Experiments show that all tested methods support training, while noise injection can improve results and aid parameter exploration and objective fitting.

  • 6 Conclusion: The paper motivates gradient estimators for highly nonlinear, nondifferentiable functions in networks with stochastic neurons.These estimators may support biologically motivated models and engineering goals such as reducing computation or parameter interactions.
  • 6 Conclusion: All tested methods allowed training to proceed.This includes the methods compared for propagating gradients through stochastic or nondifferentiable functions.
  • 6 Conclusion: Noisy rectifiers outperformed non-noisy baseline rectifiers in the gater, while noisy sigmoid baselines outperformed noiseless ones, including on the training objective.These findings suggest that injected noise may help explore good parameters and fit the training objective, beyond serving as a regularizer.

A Details of the Experiments

Experiments use a simple conditional-computation architecture on MNIST, combining a gating subnetwork with an experts subnetwork across four affine transforms. The implementation uses one conditional layer with 2,000 linear expert hidden units and a 10-class softmax output.

  • Architecture: The architecture has four affine transforms, with a gating subnetwork and experts subnetwork receiving the input, and one gating unit paired with each expert unit.The output layer applies softmax over the 10 MNIST classes.
  • Conditional layer: The expert pathway maps vectorized 28x28 MNIST images through an affine transform without a non-linearity into 2,000 expert hidden units.The gater and expert together form a conditional layer whose output is element-wise multiplied.
  • Conditional layer: Experiments use one conditional layer, while stacking conditional layers to build deeper architectures was not attempted.

A.1 Sparsity Constraint

The method enforces sparse gating to reduce computation, targeting 10% average activation so that roughly 200 of 2000 expert units are evaluated. It uses KL-divergence for sigmoid gaters and L1 regularization with adaptive λ control for rectifiers.

  • Sparsity target: A 10% average sparsity target reduces computation for 2000 expert hidden units to approximately 200 active units on average.Efficiency can come from propagating activations only to selected expert units and using those units to compute the output.
  • Sparsity regularization: Sigmoid gaters impose the sparsity constraint with a KL-divergence criterion targeting s = 0.1 across 2000 units.Each unit’s mean activation is estimated within mini-batches of 32 propagations, with λ optimized through cross-validation.
  • Sparsity regularization: Rectifier gaters use an L1-norm criterion, while λ is adjusted when effective sparsity deviates from s = 0.1 by more than 0.01.λ increases when se > s + 0.01 and decreases when se < s − 0.01; this maintained the desired sparsity effectively.

A.2 Beta Noise

Beta-distributed input noise is introduced to make sigmoid gating units decisive while preserving the target sparsity, with β≈40.1 working best for STS.

  • Motivation: KL divergence keeps sigmoids near the target sparsity s = 0.1, producing indecisive gating rather than values that can be rounded at test time.The desired behavior is mean activation 0.1 with sigmoid values above 0.5 approximately 10% of the time.
  • Method: Beta noise is added at the sigmoid input because its skew better supports the imposed sparsity constraint of 0.1 than Gaussian noise.The β parameter is hyper-optimized while α is fixed.

A.3 Test-time Thresholds

At test time, the method replaces training-time noise with deterministic computation to reduce variance. It optimizes the sigmoid threshold so deterministic gating produces the target 10% proportion of ones rather than using 0.5.

  • A.3 Test-time Thresholds: Deterministic test-time computation reduced variance compared with injecting noise during training.This follows the same general spirit as dropout.
  • A.3 Test-time Thresholds: 10% target proportion of ones required optimizing the deterministic sigmoid threshold, because 0.5 produced too few or too many zeros under the sparsity constraint.The target sparsity was 0.1, so simply thresholding sigmoids at 0.5 did not achieve the desired proportion of 0s.

A.4 Hyperparameters

The experiments identify estimator-specific hyperparameters: Gaussian noise standard deviation 1.0, beta noise for STS with its mean used at test time, and a much smaller gater learning rate for SBN. Momentum benefits STS but harms SBN, while maximum incoming-weight norm 2 works best in most cases.

  • Estimator-specific settings: The noisy rectifier works best with Gaussian noise standard deviation 1.0, while STS benefits from beta-distributed noise whose mean replaces sampling at test time.These settings apply to the noisy rectifier and the stochastic half of Smooth Times Stochastic, respectively.
  • Estimator-specific settings: The SBN gater requires learning rate 0.001, 100 times smaller than the main part’s 0.1.The learning-rate difference is specific to the Stochastic Binary Neuron gater and main network components.
  • Estimator-specific settings: The Straight-Through approach works best without multiplying the estimated gradient by t.This specifies the preferred treatment of the estimated gradient for the Straight-Through approach.
  • Optimization and constraints: Momentum helps STS, negatively affects SBN, and has little to no effect on ST and Noisy Rectifier units.The reported momentum effects differ across the four estimator or unit types.
  • Optimization and constraints: A maximum incoming-weight norm of 2 works best in most cases.Hard constraints on the maximum norms of incoming weights were explored for all units.
Loading 1308.3432v1…