PaperlayerView exact revision on arXiv

Source-linked AI summary

Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches

Yeming Wen, Paul Vicol, Jimmy Ba, Dustin Tran, Roger Grosse

arXiv:1803.04386v2cs.LGstat.ML

TL;DR

Shared weight perturbations make stochastic gradients correlated within mini-batches, limiting the variance reduction available from larger batches. Flipout uses pseudo-independent perturbations to decorrelate those gradients without bias under stated distribution assumptions. Empirically, it achieves ideal variance reduction across investigated architectures and improves the efficiency of LSTM regularization and evolution strategies.

  • Problem

    Shared weight perturbations correlate mini-batch gradients because separate perturbations for every example are expensive, limiting variance reduction from larger batches.

  • Method

    Flipout implicitly applies pseudo-independent, sign-modulated perturbations to examples while preserving each example’s marginal perturbation distribution.

  • Results

    Flipout empirically achieves ideal 1/N variance reduction across investigated architectures, outperforms dropout-based LSTM regularization, and lets one GPU match 40 CPU cores for evolution strategies.

  • Takeaways & Limitations

    Flipout makes weight perturbations practical for large-batch training and enables GPU-vectorized evolution strategies at substantially lower reported cost.

  • Takeaways & Limitations

    Flipout assumes independent perturbations across weights and a perturbation distribution symmetric around zero, and its variance analysis assumes independently sampled examples.

Abstract

from arXiv · show

Stochastic neural net weights are used in a variety of contexts, including regularization, Bayesian neural nets, exploration in reinforcement learning, and evolution strategies. Unfortunately, due to the large number of weights, all the examples in a mini-batch typically share the same weight perturbation, thereby limiting the variance reduction effect of large mini-batches. We introduce flipout, an efficient method for decorrelating the gradients within a mini-batch by implicitly sampling pseudo-independent weight perturbations for each example. Empirically, flipout achieves the ideal linear variance reduction for fully connected networks, convolutional networks, and RNNs. We find significant speedups in training neural networks with multiplicative Gaussian perturbations. We show that flipout is effective at regularizing LSTMs, and outperforms previous methods. Flipout also enables us to vectorize evolution strategies: in our experiments, a single GPU with flipout can handle the same throughput as at least 40 CPU cores using existing methods, equivalent to a factor-of-4 cost reduction on Amazon Web Services.

1 INTRODUCTION

Stochastic weight perturbations support regularization, Bayesian inference, exploration, and evolution strategies, but shared mini-batch perturbations limit variance reduction. Flipout decorrelates example gradients without bias and empirically recovers ideal variance reduction across architectures.

  • Stochastic weight perturbations are used for regularization, Bayesian neural networks, reinforcement-learning exploration, and evolution strategies.
  • Because separate perturbations are expensive, weight-based methods typically use one perturbation per mini-batch, limiting the benefit of larger batches.Activations can be sampled independently for each example, whereas weights are much more numerous than units.
  • Flipout decorrelates gradients between mini-batch examples while preserving unbiased gradient estimates.It targets the uncorrelated-gradient condition needed for ideal 1/N variance reduction.
  • Flipout applies to factorized, zero-symmetric perturbation distributions and architectures including fully connected networks, convolutional networks, and RNNs.Supported examples include DropConnect, multiplicative Gaussian perturbations, evolution strategies, and variational Bayesian neural networks.
  • Flipout empirically achieves ideal 1/N variance reduction, improves training speed, outperforms dropout-based LSTM regularization, and matches 40 CPU cores with one GPU for evolution strategies.The reported evolution-strategy result corresponds to a factor-of-4 AWS cost reduction.

2 BACKGROUND

The background defines stochastic weight perturbation methods and reviews Gaussian, DropConnect, variational Bayesian, evolution-strategy, and local-reparameterization approaches. These methods differ in their perturbation distributions, objectives, and ability to sample independently across mini-batch examples.

  • 2.1 WEIGHT PERTURBATIONS: Weight perturbation methods stochastically sample neural-network weights during training from a distribution qθ and minimize expected data loss.The weights are represented as mean weights plus a stochastic perturbation, W = W + ΔW.
  • 2.1 WEIGHT PERTURBATIONS: Additive Gaussian perturbations use independently sampled Gaussian weight noise, while reparameterization expresses weights as Wij = W̄ij + σijϵij for backpropagation.Here ϵij is sampled from N(0, 1).
  • 2.1 WEIGHT PERTURBATIONS: Multiplicative Gaussian perturbations scale noise by the weights and can be more effective than additive perturbations because weight information is scale-independent.Their form is Wij = W̄ij(1 + σijϵij).
  • 2.1 WEIGHT PERTURBATIONS: DropConnect regularizes by randomly zeroing weights; at a 50% drop rate, it corresponds to mean weights W̄ = W/2 and perturbations uniformly sampled from ±Wij.
  • 2.1 WEIGHT PERTURBATIONS: Variational Bayesian neural networks fit an approximate posterior over weights by maximizing an evidence lower bound rather than fitting a point estimate.Factorial Gaussian posteriors can be interpreted as Gaussian weight perturbations with variance adapted through variational inference.
  • 2.1 WEIGHT PERTURBATIONS: Evolution strategies evaluate many perturbed parameter candidates in parallel using a black-box fitness function.
  • 2.2 LOCAL REPARAMETERIZATION TRICK: The local reparameterization trick samples activations instead of weights to obtain independent mini-batch perturbations in fully connected networks without weight sharing.Variational dropout extends the underlying idea empirically to architectures where the exact trick does not apply.
  • Control variates provide a complementary class of variance-reduction methods that could potentially be combined with flipout.

3 METHODS

Flipout decorrelates per-example gradients by applying pseudo-independent sign transformations to a shared perturbation, preserving unbiasedness while enabling efficient vectorization. Its analysis and experiments show reduced gradient variance, with variance continuing to decrease approximately as 1/N in investigated networks.

  • Flipout: Shared weight perturbations correlate gradients across mini-batch examples, preventing variance from decreasing fully through averaging.This correlation creates an estimation-variance component that may persist as batch size grows.
  • Flipout: Flipout applies when weight perturbations factorize by weight and are symmetric around zero, including Gaussian perturbations, DropConnect, evolution strategies, and variational Bayesian neural nets.The stated assumptions are nontrivial but cover these important use cases.
  • Flipout: Flipout uses one base perturbation and a different rank-one random sign matrix for each example, yielding unbiased gradient estimates with lower mini-batch variance.The perturbation distribution remains unchanged under the sign transformations when weights are independently perturbed and distributions are symmetric around zero.
  • Vectorization: Matrix-multiplication vectorization makes flipout suitable for GPUs and TPUs, while its forward pass requires two matrix multiplications instead of one.The authors describe roughly twice the sequential forward-pass cost and note that the multiplications can be parallelized.
  • Evolution strategies: Flipout enables evolution-strategy workers to evaluate multiple quasi-independent perturbations as a batch, supporting GPU parallelism.The method replicates the starting state by the number of flipout perturbations at each worker.
  • Variance analysis: Empirically, flipout's gradient variance was approximately α/N across the explored batch sizes, matching fully independent perturbations because γ remained small.The analysis reports α ≫ β ≫ γ; shared-perturbation variance instead levels off at β once α/N falls below β.

4 EXPERIMENTS

The experiments evaluate flipout’s variance reduction, regularization, large-batch training, and evolution-strategy applications across diverse neural architectures and tasks. Results show ideal variance scaling, improved LSTM regularization, faster large-batch optimization, and GPU-compatible ES without performance loss.

  • 4.1 VARIANCE REDUCTION: Shared-perturbation variance curves transition between regimes at mini-batch sizes between 100 and 1000, unlike flipout’s linear reduction.The predicted curves have the form a + b/N, with a slope −1 regime, a constant regime, and a smooth transition.
  • 4.1 VARIANCE REDUCTION: Flipout achieves ideal 1/N variance reduction across fully connected, convolutional, and recurrent architectures over the investigated batch-size range.The comparison includes shared perturbations and LRT on MNIST, CIFAR-10, and Penn Treebank models.
  • 4.2 REGULARIZATION FOR LANGUAGE MODELING: Flipout applied to DropConnect outperforms the other evaluated regularizers on both word-level PTB validation and test perplexity.The method is called WD+Flipout and targets hidden-to-hidden recurrent weight matrices.
  • 4.3 LARGE BATCH TRAINING WITH FLIPOUT: Flipout converges in about 3 times fewer iterations than shared perturbations for FC and ConvLe Bayesian networks trained with batch size 8192.Because flipout costs roughly twice as much per update, the reported overall speedup is 1.5x.
  • 4.4 EVOLUTION STRATEGIES: FlipES incurs no performance loss relative to ideal ES and achieves data efficiency comparable with backpropagation on FC and ConvLe networks.Flipout permits batched perturbation evaluation on GPUs, while remaining more expensive than backpropagation per update.

5 CONCLUSIONS

The paper concludes that flipout decorrelates mini-batch weight gradients efficiently and improves the practicality of stochastic weight perturbations. Across experiments, it reduces variance, accelerates large-batch training, regularizes LSTMs, and enables GPU-based evolution strategies.

  • 5 CONCLUSIONS: Flipout decorrelates weight gradients between mini-batch examples while supporting diverse perturbation distributions and neural architectures.The paper reports applicability to DropConnect, multiplicative Gaussian perturbations, evolution strategies, variational Bayesian networks, fully connected nets, convolutional nets, and RNNs.
  • 5 CONCLUSIONS: Experiments demonstrate significant variance reduction, training speedups, and stronger LSTM regularization than dropout-based methods.The reported benefits are concentrated in large-batch training and stochastic recurrent models.
  • 5 CONCLUSIONS: Flipout makes GPU application of evolution strategies practical, increasing throughput for a given computational cost.The conclusion positions the method for large-batch settings favored by modern accelerators.

A PROOF OF THEOREM 2

The proof decomposes mini-batch gradient variance into data, variance, and covariance contributions, then compares independent, shared, and flipout perturbations. The resulting analysis establishes the variance behavior used to explain flipout’s empirical advantage.

  • A PROOF OF THEOREM 2: The proof begins by decomposing gradient variance into data and estimation terms for two examples and their perturbations.The estimation term is further separated into variance and covariance components.
  • A PROOF OF THEOREM 2: The proof separately analyzes fully independent, shared, and flipout perturbations to identify their covariance contributions.Shared perturbations are represented using a sampled base perturbation and shared random sign vectors.
  • A PROOF OF THEOREM 2: Fully independent perturbations eliminate the second variance term, yielding gradient variance α/N.This is the ideal mini-batch variance-reduction scaling.
  • A PROOF OF THEOREM 2: The appendix supplies the theorem proof alongside network configuration descriptions for the FC, ConvLe, ConVGG, and LSTM experiments.These configurations include fully connected, convolutional, and recurrent architectures used in the empirical study.

C VARIANCE REDUCTION EXPERIMENT DETAILS

The variance experiments estimate stochastic gradient variance by repeatedly evaluating gradients on fixed partially trained networks under sampled perturbations. They average weight-level variances by layer and use repeated estimators to construct confidence intervals.

  • C VARIANCE REDUCTION EXPERIMENT DETAILS: Gradient variance is estimated by repeatedly sampling perturbations, computing gradients without updating parameters, and measuring each weight’s variance.The procedure repeats the gradient computation 200 times for each weight in the experiments.
  • C VARIANCE REDUCTION EXPERIMENT DETAILS: Confidence intervals use 50 repeated average variance estimates and 90% t-test intervals for Figure 1.The repeated estimates are denoted f V1 through f V50.
  • C VARIANCE REDUCTION EXPERIMENT DETAILS: Layer-level variance is computed by averaging the estimated variances of weights within that layer.The layer average is denoted by ˜V.
  • C VARIANCE REDUCTION EXPERIMENT DETAILS: For large ConVGG batches, the experiment shares the base perturbation across GPUs to isolate variance-reduction effects from implementation choices.The appendix additionally examines flipout when perturbations are generated independently on different GPUs.
  • C VARIANCE REDUCTION EXPERIMENT DETAILS: Large LSTM mini-batches are split into sub-batches of 64, sharing one base perturbation while sampling independent sign matrices per sub-batch.This procedure is used for batches of size 128 and higher.

D LSTM REGULARIZATION EXPERIMENT DETAILS

The experiments compare flipout-based LSTM regularization with recurrent-dropout, zoneout, and variational-dropout approaches using specified sequence-level settings and hyperparameters.

  • Regularization methods: LSTM regularization methods differ in which recurrent terms they perturb and whether masks are sampled per step or per sequence.Semeniuta et al. drop the cell update vector, while other approaches regularize recurrent connections through hidden-state or cell-state perturbations.
  • Word-level models: Word-level models used gradient clipping threshold 0.25 and method-specific embedding, input, output, hidden-state, cell-update, and zoneout probabilities.The listed settings include 0.35 input/output dropout and 0.2 hidden-state and embedding dropout for variational dropout.
  • Character-level models: Character-level models used orthogonal LSTM initialization, gradient clipping threshold 1, one-hot inputs, and no input or output dropout.Flipout and shared-perturbation LSTMs sampled Gaussian noise with σ = 1 for the hidden-to-hidden weight matrix.
  • Character-level models: The character-level comparisons set recurrent-dropout, zoneout, variational-LSTM, flipout, and shared-perturbation models to distinct regularization configurations.The recurrent-dropout model used 0.25 cell-state dropout, zoneout used 0.5 cell and 0.05 hidden-state probabilities, and variational LSTM used 0.25 hidden-state dropout.

E ADDITIONAL EXPERIMENTS

Figure 3 compares empirical gradient variance for flipout and shared perturbations when training across multiple GPUs. The accompanying experiment reports that flipout retains a variance advantage despite naturally independent noise across sub-batches.

  • Multiple-GPU setting: Flipout achieves lower variance than shared perturbations even when multiple GPUs naturally provide independent noise for separate sub-batches.With mini-batch size 8192, four GPUs provide four independent noise samples for sub-batches of size 2048.
  • Variance comparison: Figure 3 compares empirical gradient variance across multiple GPUs using solid flipout curves and dotted shared-perturbation curves.The key comparison is whether flipout produces lower gradient variance than shared perturbations as the mini-batch setting changes.

E.2 LARGE BATCH TRAINING WITH FLIPOUT

The large-mini-batch experiments evaluate FC and ConvLe networks with mini-batches of 8192 under Bayes by Backprop. Flipout converges faster while retaining the same generalization ability as shared perturbations.

  • Experimental setup: The experiments used Adam with learning rate 0.003 and downscaled the KL term by a factor of 10 to achieve higher accuracy.These settings were applied to both FC and ConvLe large-mini-batch experiments.
  • Results: Flipout converges faster than shared perturbations without changing their generalization ability.The faster convergence does not result in overfitting; training and test errors show the same generalization pattern.
  • Experimental setup: Both FC and ConvLe experiments used mini-batches of size 8192 and compared training and test errors.The figure reports FC results on the left and ConvLe results on the right.

E.3 FLIPES V.S. CPUES

The experiments compare FlipES with 40-core cpuES across increasing fully connected and convolutional model sizes. FlipES scales better because it runs on the GPU, while cpuES becomes increasingly costly as models grow.

  • Scaling result: cpuES computational cost increases with model size, whereas FlipES scales better because it runs on the GPU.The comparison concerns per-update computational cost rather than model accuracy.
  • Fully connected network: FC per-update time rises from 0.12s to 1.86s for FlipES and from 0.51s to 38.0s for cpuES as hidden units increase from 32 to 2048.The table reports intermediate hidden-unit settings of 128 and 512 as well.
  • Convolutional network: Convolutional per-update time rises from 2.3s to 11.77s for FlipES and from 16s to 132s for cpuES as the filter scale increases from 0.25 to 1.5.At filter scale 1.0, FlipES takes 7.12s and cpuES takes 67s.
  • Experimental setup: Figure 5 compares per-update time for FlipES and 40-core cpuES using 5,000 samples as model size increases.The FC network is scaled by hidden units, while the convolutional network is scaled by filter count.

E.4 LARGE BATCH LSTM TRAINING

Flipout applies DropConnect to a large-batch LSTM language model, reducing gradient variance and improving optimization over weight-dropout training. With batch size 8192, WD+Flipout converges faster and reaches lower training perplexity than WD.

  • Method: Flipout implements DropConnect in a large-batch LSTM word-level language model based on Merity et al. (2017).The model uses 400-dimensional word embeddings and three layers with hidden dimension 1150.
  • Variance reduction: Significant gradient-variance reduction appears for mini-batch sizes larger than 256 when comparing WD+Flipout with WD.The comparison concerns first-layer hidden-to-hidden weight gradients.
  • Training results: With batch size 8192, WD+Flipout converges faster than WD and achieves lower training perplexity.The training curves showcase optimization benefits in the large-mini-batch setting.
Loading 1803.04386v2…