PaperlayerView exact revision on arXiv

Source-linked AI summary

Techniques for Learning Binary Stochastic Feedforward Neural Networks

Tapani Raiko, Mathias Berglund, Guillaume Alain, Laurent Dinh

arXiv:1406.2989v3stat.MLcs.LGcs.NE

TL;DR

Training stochastic networks is difficult, particularly when estimating gradients for discrete hidden units. The paper studies fully stochastic networks, proposes two gradient estimators, and tests them across several tasks. The techniques are demonstrated on classification, image completion, and expression-conditioned face generation.

  • Problem

    Estimating gradients for discrete stochastic hidden units is difficult because standard back-propagation is no longer possible.

    Source: §1 · p. 2

  • Method

    The paper compares gradient-estimation methods for fully stochastic networks and proposes a lower-bias approximate back-propagation estimator and a lower-variance modification of Tang & Salakhutdinov (2013).

    Source: §1 · p. 2

  • Results

    The proposed techniques were demonstrated in MNIST classification, MNIST completion, and face-image output conditioned on mean expression.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Using more than one hidden-layer particle significantly enhances modeling capacity, as shown theoretically and experimentally.

    Source: §5 · p. 8

    For structured outputs, finding hidden samples that explain the targets with a reasonably small sample size remains a challenge.

    Source: §4 · p. 7

Abstract

from arXiv · show

Stochastic binary hidden units in a multi-layer perceptron (MLP) network give at least three potential benefits when compared to deterministic MLP networks. (1) They allow to learn one-to-many type of mappings. (2) They can be used in structured prediction problems, where modeling the internal structure of the output is important. (3) Stochasticity has been shown to be an excellent regularizer, which makes generalization performance potentially better in general. However, training stochastic networks is considerably more difficult. We study training using M samples of hidden activations per input. We show that the case M=1 leads to a fundamentally different behavior where the network tries to avoid stochasticity. We propose two new estimators for the training gradient and propose benchmark tests for comparing training algorithms. Our experiments confirm that training stochastic networks is difficult and show that the proposed two estimators perform favorably among all the five known estimators.

1 INTRODUCTION

Stochastic hidden units can represent mappings and output distributions that deterministic networks with simple output distributions cannot, but training discrete stochastic networks presents gradient-estimation challenges. The paper studies those challenges, questions the need for hybrid networks, and proposes new estimators and benchmarks.

1 INTRODUCTION

  • Mixtures over stochastic hidden-unit configurations can model multimodal output distributions, including those needed for one-to-many mappings.With deterministic hidden units, the output conditionals remain in the family of simple distributions used by the network.Source: §1 · p. 1
  • Stochasticity can regularize networks, while binary units also suit hard-decision computations and hardware restricted to binary outputs.The introduction cites dropout as an example of multiplicative binary noise that regularizes networks.Source: §1 · p. 1
  • Discrete hidden units rule out standard back-propagation, and existing gradient estimators trade unbiasedness for high variance against biased back-propagation approximations.Earlier inference approaches also had drawbacks: Gibbs sampling can mix poorly, while mean-field can be inefficient and yield a loose likelihood bound; direct sampling offers independent samples and an unbiased estimate of P(y|x).Source: §1 · pp. 1–2
  • Tang & Salakhutdinov (2013) use more than one sample, which this paper argues is important; the authors alternatively suggest deterministic units may do most of the work around a few stochastic units.Prior applications included mapping faces to varying expressions and silhouettes to color images.Source: §1 · p. 2
  • The paper proposes two gradient estimators—one with less bias than Bengio et al. (2013) and one with less variance than Tang & Salakhutdinov (2013)—and benchmark tests using MNIST and the Toronto Face Database.The tests are proposed as a setting for comparing training algorithms.Source: §1 · p. 2

2 STOCHASTIC FEEDFORWARD NEURAL NETWORKS

The section formulates stochastic binary hidden units and examines how to estimate their training criterion and gradients. It shows that using one hidden-activation sample changes the objective's preference for stochasticity, then presents gradient-estimation approaches for discrete units.

2.1 PROPOSED ESTIMATOR OF THE TRAINING CRITERION

  • The proposed training-criterion estimate averages P(y|h) over M hidden-activation samples drawn from P(h|x), yielding a finite-mixture objective.Source: §2.1 · p. 3
  • With M = 1, the estimated criterion never favors a stochastic hidden-unit output, although the original criterion may; the network can instead drive a stochastic sigmoid toward a deterministic step.The estimated criterion treats M samples as a finite mixture, while the original criterion is maximized when P(y|x) matches the data conditional.Source: §2.1 · p. 3

2.2 GRADIENT FOR TRAINING P(y|h)

  • For the shared output-layer gradient, each sample's contribution is weighted by its normalized likelihood, interpretable as a mixture responsibility.Source: §2.2 · p. 4

2.3 FIRST ESTIMATORS OF THE GRADIENT FOR TRAINING P(h|x)

  • The reviewed gradient estimators for P(h|x) trade off variance: one is unbiased but high-variance, while the back-propagation-based alternative is biased but lower-variance.Source: §2.3 · p. 4

2.4 PROPOSED BIASED ESTIMATOR OF THE GRADIENT FOR TRAINING P(h|x)

  • The proposed biased estimator propagates gradients through discrete hidden units by treating them as continuous variables with additive noise, while ignoring the input-dependence of that noise.The noise is zero-mean for any input signal, which the authors say should help keep the bias relatively small.Source: §2.4 · p. 4

2.5 VARIATIONAL TRAINING

  • The variational approach fixes the proposal R(h|x,y) to P(h|x); its proposed unbiased gradient estimator reduces variance by subtracting a constant c from sample weights without adding bias.The authors use c = 1/M and report it empirically close to the optimum.Source: §2.5 · pp. 5–6, Figure 1 · p. 6

3 EXPERIMENTS

The experiments test stochastic feedforward networks on multimodal structured-prediction tasks and compare gradient estimators with deterministic and hybrid baselines. Network D had the best overall performance, while G3 performed best among the stochastic networks.

3 EXPERIMENTS

  • MNIST predicts lower digit halves from upper halves, while TFD predicts each subject’s facial-expression distribution from that subject’s mean image.TFD used 100 training subjects (1,372 images) and 31 test subjects (427 images); training used M∈{1,20}, with M=100 at test time.Source: §3 · pp. 6–7
  • A preliminary MNIST classification check retained G3–G5 because they performed significantly better than G1 and G2.The viability results are reported in Appendix A.Source: §3 · p. 6
  • Comparison networks included a deterministic baseline and a hybrid with 40 stochastic and 160 deterministic neurons per hidden layer; a second hybrid fixed stochastic probabilities at 0.5.In network B, deterministic-trained hidden weights were tested using stochastic hidden units.Source: §3 · pp. 6–7
  • Among stochastic networks, G3 performed best on both tasks, and its performance rose significantly when training used more than M=1 particles.The objective C_M was estimated accurately with M=20 or M=40.Source: §3 · p. 7
  • Network D performed best across both tasks, outperforming hybrid network C despite using fixed 0.5 activation probabilities for its stochastic units.The authors suggest D may benefit from modeling stochasticity without propagating errors through binary stochastic variables.Source: §3 · p. 7
  • For ambiguous MNIST inputs, stochastic networks generated reconstructions corresponding to different digits, whereas the deterministic network could not model that distribution.The authors note that larger networks and longer training might improve results, particularly on TFD, where deterministic network A outperformed some stochastic networks.Source: §3 · p. 7

4 DISCUSSION

The discussion distinguishes lower gradient-estimate variance from final benchmark performance, and identifies sample selection as a challenge for structured outputs. It also points to stochastic reconstructions of ambiguous digits and notes a TFD evaluation caveat.

4 DISCUSSION

  • Using positive and negative particle weights reduced gradient-estimate variance for G5 versus G4, but the final-performance difference was not substantial.The weights can be interpreted as pulling probability mass toward or pushing it away from sampled hidden states.Source: §4 · p. 7
  • With a reasonably small sample size, finding hidden states that assign structured outputs high probability remains challenging.A proposal distribution R(h|x,y) is suggested as a promising direction for addressing this sampling problem.Source: §4 · p. 7
  • TFD results report the best test error without a separate validation set, based on a single hyperparameter chosen from a sparse grid.MNIST used a separate validation set for learning-rate selection.Source: §4 · p. 7
  • Figure 2 compares original digits, masked inputs, and ten predicted samples for G3, G5, and a deterministic network; stochastic models produce different digits for ambiguous inputs.The columns show G3 at left, G5 in the middle, and the deterministic model at right.Source: Figure 2 · p. 8

5 CONCLUSION

The paper treats stochastic hidden-layer particles as part of the model, not merely a computational trick, and reports theoretical and experimental evidence that using multiple particles enhances modeling capacity. Experiments demonstrate classification, digit completion, and expression-conditioned face generation.

5 CONCLUSION

  • Stochastic hidden-layer particles define the model itself rather than serving only as a computational trick for training deterministic models.Source: §5 · p. 8
  • Multiple particles significantly enhance modeling capacity, as shown theoretically and experimentally.The proposed gradients use hidden-layer particles with normalized weights reflecting how well each particle explains the output targets.Source: §5 · p. 8
  • The techniques were demonstrated on MNIST classification, missing-digit completion, and expression-conditioned face generation.Source: §5 · p. 8
  • The authors suggest that their theory may apply to Dropout and other stochastic-flavored techniques for deterministic models.Source: §5 · p. 8

A CLASSIFICATION EXPERIMENT

MNIST classification serves as a benchmark for comparing training algorithms, although its class-label output does not test stochastic networks’ ability to model complex output distributions. Deterministic networks achieve the best test error; among the proposed estimators, G3 ranks ahead of G5.

A CLASSIFICATION EXPERIMENT

Related papers

Loading 1406.2989v3…