PaperlayerView exact revision on arXiv

Source-linked AI summary

Conditional Computation in Neural Networks for faster models

Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, Doina Precup

arXiv:1511.06297v2cs.LG

TL;DR

Deep networks can require substantial computation, motivating conditional activation of only selected units. This paper learns input-dependent sparse activation policies with reinforcement learning and regularization, reporting comparable accuracy with potentially faster execution, while noting hardware and model-size limitations.

  • Problem

    Deep-network training and evaluation can be time-consuming and computationally expensive, motivating conditional computation that activates only selected units.

  • Method

    The paper casts input-dependent activation probabilities as a reinforcement-learning problem, using sigmoid-Bernoulli policies, REINFORCE, and sparsity-oriented regularization.

  • Results

    The experiments show comparable accuracy to standard neural networks and indicate that sparse models can execute faster.

  • Takeaways & Limitations

    Conditional computation can provide similarly accurate models with reduced computation time when sparse execution is effective.

  • Takeaways & Limitations

    On MNIST, models were too small for the specialized sparse implementation to make a substantial speed difference, and reported timings used single-core CPUs.

Abstract

from arXiv · show

Deep learning has become the state-of-art tool in many applications, but the evaluation and training of deep models can be time-consuming and computationally expensive. The conditional computation approach has been proposed to tackle this problem (Bengio et al., 2013; Davis & Arel, 2013). It operates by selectively activating only parts of the network at a time. In this paper, we use reinforcement learning as a tool to optimize conditional computation policies. More specifically, we cast the problem of learning activation-dependent policies for dropping out blocks of units as a reinforcement learning problem. We propose a learning scheme motivated by computation speed, capturing the idea of wanting to have parsimonious activations while maintaining prediction accuracy. We apply a policy gradient algorithm for learning policies that optimize this loss function and propose a regularization mechanism that encourages diversification of the dropout policy. We present encouraging empirical results showing that this approach improves the speed of computation without impacting the quality of the approximation.

1 INTRODUCTION

Conditional computation reduces deep-network computation by activating only input-relevant units, but requires learned gating policies to preserve useful predictions. This paper uses reinforcement learning and policy search to learn sparse, input-dependent activation policies.

  • Motivation: Conditional computation activates only some network units according to the input, making information propagation faster during training and testing.A gating structure learned alongside the original network decides which units are active.
  • Motivation: Using fewer active links and parameters may sharpen gradients and regularize the main network.The approximator must produce outputs using only a small fraction of possible parameters.
  • Approach: The paper learns input-dependent activation probabilities with reinforcement learning while jointly reducing prediction error and participating nodes.The objective targets both prediction accuracy and computational load.
  • Approach: Unlike standard dropout, which is data-independent, the proposed policies can produce specialized computation paths within the network.The distinction is between conditional and unconditional computation.
  • Evaluation: Policy search methods are used to formulate and solve the proposed optimization problem, with preliminary results on standard classification benchmarks.The introduction identifies policy search as the solution approach and reports preliminary benchmark evaluation.

2 PROBLEM FORMULATION

The paper formulates input-dependent activation as a single-step Markov Decision Process. Each layer has a separate Bernoulli policy whose probabilities depend on the previous layer’s activations and whose sampled mask gates the current layer.

  • Model: The model combines a fully connected network with stochastic per-layer policies that activate or deactivate nodes in an input-dependent way during training and testing.The policies are applied at each layer alongside the neural network.
  • MDP formulation: The activation-learning problem is a single-step MDP with continuous states, binary-vector actions, transition dynamics, and neural-network loss as cost.The state is the previous layer’s activation vector, and an action is a mask over the current layer’s units.
  • Policy representation: Each layer uses a separate k-dimensional Bernoulli policy to determine the activation probability of every unit or block.The Bernoulli distributions are parameterized independently of the neural network and depend on the input.
  • Policy representation: Participation probabilities are computed from lower-layer activations through sigmoid parameters, then sampled masks multiply the hidden-layer output element-wise.The policy parameters are θ_l = {Z^(l), d^(l)} and the masked output is h(x) ⊗ u.

3 LEARNING SIGMOID-BERNOULLI POLICIES

The learning procedure alternates neural-network optimization with REINFORCE updates for separate sigmoid-Bernoulli policies. Its regularized objective balances prediction accuracy, sparse computation, policy variance, and parameter penalties, while grouped masks enable sparse matrix multiplication.

  • Policy learning: REINFORCE learns separate policy parameters for each layer because the observation space changes across decision stages.The policy-gradient summation reduces to the layer-specific contribution because each θ_l appears only at its own stage.
  • Policy learning: Mini-batches estimate policy gradients by weighting log action-probability derivatives with each input’s total network cost.The cost vector contains one total cost per example in the mini-batch.
  • Regularization: The sparsity penalties target both expected unit activation rates and the desired per-example activation sparsity.A low target τ can yield a few high-probability activations for some inputs and low probabilities for others.
  • Regularization: The variance penalty maximizes activation variability across examples and discourages policies that learn a uniform distribution.This encourages units to activate with high probability for selected examples and low probability for others.
  • Regularized objective: The regularized loss combines negative log-likelihood, sparsity and variance terms, and L2 penalties for both network and policy parameters.The coefficient λ_s trades prediction accuracy against computational parsimony, while λ_v trades stochasticity against input-dependent saturation.
  • Sparse implementation: Contiguous groups of units are activated together, reducing the policy action space and enabling sparse matrix multiplication over nonzero elements.Grouped masking provides both a smaller policy and a computational advantage in hidden-layer operations.

4 EXPERIMENTS

Experiments across MNIST, CIFAR-10, and SVHN evaluate conditional networks, specialized sparse computation, and regularization effects. The approach can preserve competitive prediction while reducing computation, but benefits depend on model size and hyperparameter choices.

  • MNIST: On MNIST, the conditional model reached around 2.3% test error, between a full network’s 1.9% and a similarly computed network’s 2.8%.The model used 256 hidden units with a target sparsity rate of 6.25%.
  • Limitations: MNIST models were too small for the specialized sparse implementation to make a substantial computation-time difference.This result identifies a condition under which the conditional model is less desirable.
  • CIFAR-10: On CIFAR-10, condnet achieved similar performance to a normal neural network with sensibly reduced computation time.The best-performing model targeted 1/16 sparsity but learned an average sparsity rate of 18%.
  • SVHN: On SVHN, the best condnet reached 7.3% test error and a 10-second validation epoch, compared with 9.1% and 16 seconds for the best standard network.The 10-second runtime was reported with optimization; without it, the condnet took 14 seconds.
  • Regularization: The variance regularization term λv significantly sped training, while only slightly affecting prediction accuracy and learned policies.Increasing λv produced much lower error rates during the first few epochs.
  • Regularization: Increasing λs reduced running time and performance, with λ approximately [300, 400] offering the best CIFAR-10 trade-off in the reported experiment.Other λs values could be selected depending on application requirements.

5 RELATED WORK

The paper relates its reinforcement-learning approach to prior methods that learn input-dependent dropout, sequential filter decisions, layer-wise control, and selective visual attention. These approaches differ in decision timing, granularity, and the parts of computation they selectively process.

  • Prior conditional-computation methods: Standout learns input-dependent dropout at every node, whereas this paper uses layer-wise masks spanning blocks of units.Standout computes a one-shot mask over the entire network; the paper’s method makes layer-wise decisions with block-wise masks.
  • Prior conditional-computation methods: Stochastic Times Smooth neurons provide nonlinear, nondifferentiable gates for conditional computation within deep networks.
  • Sequential policy methods: Stollenga et al. learn a sequential decision process over CNN filters, using direct policy search but staging decisions through forward-propagated activations.
  • Sequential policy methods: DSNN controls layer-wise mappings with REINFORCE, but evaluates datasets where distinct modes make policy discrimination easier.
  • Attention models: Attention models selectively process visual inputs sequentially, whereas this approach targets more general selective computation within the network.

6 CONCLUSION

The paper concludes that reinforcement learning can train conditional-computation policies that preserve standard-model accuracy while exploiting sparse computation. It also identifies algorithmic and hardware-dependent boundaries for realizing these gains.

  • Conclusion: REINFORCE trains layer-activation policies that map activations to Bernoulli masks, combine prediction loss with sparsity regularization, and expose an accuracy–running-time trade-off.
  • Conclusion: The experiments show similarly accurate models may execute faster because their computations are sparse.
  • Future directions: A more efficient policy-search algorithm or more sequential reward formulation could replace REINFORCE, while direct computation-time costs may also help.
  • Deployment scope: Reported running times use single-core CPUs, so adapting sparse computation to multicore and GPU architectures remains ongoing work.

A ALGORITHM

The algorithm samples Bernoulli masks, expands block decisions across units, performs masked forward propagation, and updates network and policy parameters using a regularized loss and REINFORCE.

  • Forward pass: Each forward pass samples Bernoulli variables from activation probabilities and repeats each sampled value across the block when blocksize exceeds one.
  • Forward pass: The masked forward pass produces the prediction, while classification leaves the final softmax layer unmasked.
  • Parameter updates: The loss combines network cost, sparsity and variance terms, and L2 penalties on both neural-network and policy parameters.
  • Parameter updates: Network and policy parameters are interleaved during optimization, with REINFORCE required because the policy cost has zero direct gradient with respect to policy parameters.
  • Minibatch extension: The algorithm extends to minibatches through matrix operations and an R-op for efficient gradient computation.

B REINFORCE

The paper derives REINFORCE as a likelihood-ratio policy-gradient method for trajectory returns. In this application, Markov factorization reduces the gradient to policy terms, and single-step Bernoulli trajectories simplify it further.

  • Policy-gradient formulation: REINFORCE improves a parameterized policy by estimating gradients of expected trajectory return with a likelihood-ratio expression.
  • Policy-gradient formulation: The trajectory gradient requires assumptions for an interchange used in the derivation, and unknown transition probabilities prevent direct trajectory-probability computation.
  • Markov simplification: The Markov property factors trajectory probabilities into initial-state, transition, and policy terms, leaving the policy-log gradient because the other factors do not depend directly on θ.
  • Application to conditional computation: For this model, trajectories have one step and reward equals neural-network cost, so the gradient reduces to the log probability of the sampled Bernoulli mask.
Loading 1511.06297v2…