PaperlayerView exact revision on arXiv

Source-linked AI summary

Randomized Prior Functions for Deep Reinforcement Learning

Ian Osband, John Aslanides, Albin Cassirer

arXiv:1806.03335v2stat.MLcs.AIcs.LG

TL;DR

Deep reinforcement learning needs uncertainty estimates for efficient exploration, but common approaches can miss uncertainty beyond observed data or mishandle sequential dependencies. The paper adds randomized, fixed prior functions to bootstrapped ensemble members, showing exact Bayesian equivalence in Gaussian linear models and successful nonlinear cartpole exploration.

  • Problem

    Deep RL often lacks uncertainty estimates, while bootstrap and distributional approaches do not adequately represent uncertainty beyond observed data for sequential decisions.

  • Method

    Each bootstrapped ensemble member combines a trainable neural network with a randomized, untrainable prior function and optimizes their sum against training loss.

  • Results

    The method is equivalent to exact Bayesian inference for Gaussian linear models, and only bootstrap with prior networks learns a performant cartpole swing-up policy among the compared approaches.

  • Takeaways & Limitations

    Randomized prior functions provide a simple practical alternative for uncertainty-driven exploration in deep reinforcement learning.

  • Takeaways & Limitations

    Independent inference applied to Bellman errors does not correctly propagate uncertainty through the value function because the equation’s two sides are dependent.

Abstract

from arXiv · show

Dealing with uncertainty is essential for efficient reinforcement learning. There is a growing literature on uncertainty estimation for deep learning from fixed datasets, but many of the most popular approaches are poorly-suited to sequential decision problems. Other methods, such as bootstrap sampling, have no mechanism for uncertainty that does not come from the observed data. We highlight why this can be a crucial shortcoming and propose a simple remedy through addition of a randomized untrainable `prior' network to each ensemble member. We prove that this approach is efficient with linear representations, provide simple illustrations of its efficacy with nonlinear representations and show that this approach scales to large-scale problems far better than previous attempts.

1 Introduction

Deep reinforcement learning needs uncertainty-aware decision making, yet many uncertainty methods are poorly suited to sequential problems. The paper proposes randomized prior functions for ensemble models as a practical remedy and supports the approach analytically and experimentally.

  • Deep RL combines neural-network function approximation with sequential decision making under uncertainty.
  • Many state-of-the-art deep RL algorithms lack uncertainty estimates and can fail even on simple problems requiring uncertainty consideration.
  • Popular uncertainty-estimation methods for deep learning can be poor choices for sequential decision problems and may cause arbitrarily poor performance.
  • Bootstrap ensembles approximate uncertainty from perturbed datasets but lack an effective mechanism for uncertainty absent from observed data.
  • Randomized prior functions add a fixed random prior to each ensemble member, yielding exact Bayesian inference for Gaussian linear models and supporting experiments in deep learning.

2 Why do we need a ‘prior’ mechanism for deep RL?

Efficient exploration requires uncertainty about unobserved possibilities, not merely uncertainty or variation derived from observed data. The paper identifies failures in dropout, independent Bellman-error inference, distributional RL, and count-based bonuses that motivate randomized prior functions.

  • Efficient exploration in sparse-reward tasks needs a prior effect that supplies motivation when observed data contain no rewards.
  • Without a prior drive, agents may flounder and require exponentially large amounts of data to learn simple problems.
  • Thompson sampling selects actions according to the probability they are optimal and has theoretical support in domains where other approaches fail.
  • 2.1 Dropout as posterior approximation: Dropout posterior sampling is invariant to duplicated data because its dropout rate does not depend on the data, potentially causing arbitrarily poor decisions.
  • 2.2 Variational inference and Bellman error: Independent inference on Bellman errors fails to propagate uncertainty through the value function, even with exact inference and a correct variational class.
  • 2.3 ‘Distributional reinforcement learning’: Distributional RL models observed-return variation, whereas Bayesian uncertainty concerns beliefs and is the uncertainty relevant for exploration.
  • 2.4 Count-based uncertainty estimates: Count-based state bonuses can be poorly aligned with uncertainty over the unknown optimal action, even in a linear bandit.

3 Randomized prior functions for deep ensembles

The paper reframes posterior sampling as a computational procedure: train on noisy data while incorporating randomized prior information. For nonlinear networks, additive randomized prior functions provide a practical approximation that empirically outperforms alternatives without such priors.

  • Posterior sampling as optimization: Bayesian linear regression motivates generating posterior samples by optimizing on noisy data with randomized regularization.Lemma 3 establishes this procedure for linear models, where it reproduces the Gaussian posterior.
  • Nonlinear extension: For nonlinear function approximators, the procedure is no longer an exact posterior but retains the linear-setting sanity check.The authors present this as a computationally intuitive approximation rather than an analytic extension of Gaussian conjugacy.
  • Empirical comparison: The paper reports that randomized prior functions significantly outperform ensemble methods without priors and outperform weight-space l2 regularization in deep reinforcement learning.The authors attribute a major factor to enforcing prior beliefs in the lower-dimensional output space rather than neural-network weight space.
  • Randomized prior functions: Randomized prior functions add a fixed prior network to each trainable ensemble member, whose summed prediction is optimized against training loss.The prior remains untrainable, while the trainable network fits the observed data around it.

4 Deep reinforcement learning

BootDQN+prior extends bootstrapped value ensembles with distinct fixed randomized prior functions and evaluates them on increasingly complex exploration tasks. The method addresses several uncertainty shortcomings and is the only compared approach reported to learn a performant policy in the modified cartpole task.

  • Algorithm: BootDQN+prior trains K Q-value estimates on perturbed data, each paired with a distinct fixed prior function, and follows one sampled ensemble member per episode.The implementation uses a 50-50 ensemble replay buffer and optimizes each member with its own prior function.
  • Uncertainty properties: The prior mechanism supports posterior concentration, temporally consistent multi-step uncertainty, and separation of epistemic uncertainty from return noise.Each member trains on its own target value, while the mean TD loss avoids fitting return noise as distributional RL does.
  • Exploration: The method explores in task-relevant value space rather than state density and can seek novelty even when all observed rewards are zero.Randomized priors can make unseen-state value estimates positive, encouraging exploratory actions without observed rewards.
  • Chain environments: Only BSP scales gracefully to large chain problems, with log-log results suggesting empirical learning-time scaling Tlearn = ˜O(N 3).The comparison uses N = 5, .., 60, up to 500K episodes, across 5 seeds and ensemble K = 20.
  • Chain environments: Figure 3 compares BS, BSR, and BSP across chain problems indexed by N, using average episodes to reach regret below 0.9.BSP is bootstrap with an additive prior function, whereas BS has no prior and BSR uses l2 regularization toward initial weights.
  • Cartpole swing-up: The modified cartpole task requires nonlinear generalization because reward occurs only when the hanging pole is upright, balanced, and centered.The state uses trigonometric angle coordinates and velocities, making tabular approaches impractical because of dimensionality.
  • Cartpole swing-up: Only BSP learns a performant cartpole policy; DQN with ϵ-greedy, BS, and D4PG never attain positive reward.The result transfers the prior-mechanism intuition from simple domains to a more complex nonlinear setting.

5 Conclusion

The paper argues that uncertainty estimates and effective prior mechanisms are important for efficient exploration in deep RL. It presents randomized prior functions with bootstrapped ensembles as a practical alternative and documents implementation details for BootDQN+Prior.

  • The paper identifies uncertainty estimates, effective prior mechanisms, and efficient exploration as central issues in deep RL.It frames randomized-prior bootstrapped ensembles as a practical alternative to existing methods.
  • The paper supports its claims with linear analysis and experiments, while leaving prior design, meta-learning, and ensemble distillation open.These questions are identified as directions for future work.
  • BootDQN+Prior maintains K parallel perturbed-data buffers and value-function estimates for the ensemble.The implementation may use less than O(K) memory despite the conceptual parallel buffers.
  • Action selection uses approximate Thompson sampling by sampling one ensemble member per episode and following its greedy policy.The selected index is drawn uniformly from 1 through K.

B Reinforcement learning experiments

The experiments use common deep RL implementation choices, including Adam, uniform replay, and tuned exploration baselines. The baseline-specific settings are documented for comparison.

  • Experiments generally use TensorFlow defaults, Adam with learning rate 10^-3, and uniform experience replay with batch size 128.
  • The ϵ-greedy DQN baseline linearly anneals epsilon over 2000 episodes and sweeps the initial epsilon ϵ0.
  • NoisyNet, Dropout, Ensemble, and Bootstrap use greedy policies according to their respective per-episode procedures.

B.1 Chain environments

Chain-environment experiments compare scaling and exploration strategies as problem size increases. BSP is contrasted with dithering, dropout, NoisyNet, count-based exploration, and other baselines.

  • Figure 8 uses a log-log scale to reveal learning-time scaling as problem size N increases.It includes a dithering lower bound Tlearn = 2N and a reference polynomial growth Tlearn = ˜O(N^3).
  • Neither dropout nor NoisyNet scales successfully to large domains after tuning dropout rate and sampling frequency.Both methods were tuned individually up to 50k episodes.
  • Count-based exploration performs much worse than BSP even after sweeping the bonus scale β and using true state visit-counts.The passage attributes the discrepancy in part to inefficient uncertainty propagation over many timesteps.
  • Figure 10 reports the count-based exploration comparison across optimistic bonus scales β.
  • The reported hyperparameter settings include BSP prior scale β = 10, BSR regularizer scale λ = 0.1, and UCB optimistic bonus β = 0.1.

B.2 Sparse cartpole swing-up

In sparse cartpole swing-up, BSP performance depends on the prior scale β. Small values can converge prematurely to a zero-reward stationary policy, while larger values are expected to learn eventually.

  • Small prior scales β prematurely and suboptimally converge to a stationary policy, receiving zero cumulative reward.
  • Larger β values take longer to wash away their prior effect, but are expected to learn a performant policy eventually.
  • The experiments use standard Atari preprocessing and Ape-X hyperparameters, but implementation differences limit direct baseline comparability across all games.
  • The appendix presents the experiments as simple illustrations of potential inadequacies rather than evidence that competing methods fail on every domain.

C.1 Dropout as posterior approximation

The section argues that dropout is a poor Bayesian-posterior approximation for sequential decision problems because its uncertainty does not concentrate appropriately with data. In a simple bandit, Thompson sampling with dropout can therefore incur linear regret.

  • Posterior approximation: The dropout posterior depends on the dataset only through its empirical mean, so datasets sharing that mean produce the same posterior distribution.This makes the approximation insensitive to information in the rest of the observed data.
  • Posterior approximation: Dropout couples posterior mean and standard deviation, preventing uncertainty from vanishing while the mean remains nonzero.The passage contrasts this with Bayesian behavior in which the mean approaches E[Y] while uncertainty converges to zero.
  • Decision consequences: In a two-armed Bernoulli bandit, Thompson sampling with dropout has expected regret Ω(T) for any fixed d, p, and λ.This is contrasted with more reasonable posterior approximations that achieve ˜O(log(T)) regret in the same setting.
  • Decision consequences: The section concludes that dropout can perform poorly even in simple problems where correct uncertainty is essential for action selection.The lower-bound analysis is explicitly described as conservative and illustrative.
  • Dropout behavior: Dropout's posterior variability does not concentrate with increasing data, even for a simple neural-network regression task.The same behavior extends beyond the analyzed linear form to neural networks with nonlinear architectures.

C.2 Variational inference and Bellman error

The section argues that common variational approaches are poorly matched to uncertainty in deep reinforcement learning. Their sample-based losses fit expected targets without matching higher-order uncertainty, and practical performance can remain poor despite tuning.

  • Variational losses: Common variational approximations use losses that are fundamentally ill-suited to the value-function uncertainty problem.The paper reports poor performance for NoisyNet on benchmark reinforcement-learning tasks even after extensive tuning.
  • Variational losses: The sample-based loss matches the expectation of the target distribution but does not match higher moments of uncertainty.The paper suggests parameterized distributions with cross-entropy losses as a possible alternative, leaving that approach for future work.
  • Prior knowledge: Mean-field Gaussian variational models can underestimate uncertainty and may be poor choices for encoding prior knowledge.The passage identifies products of independent Gaussians as a common computationally convenient modeling choice.

C.3 ‘Distributional reinforcement learning’

The section distinguishes distributional reinforcement learning's outcome distributions from Bayesian uncertainty about mean beliefs. Treating the former as posterior uncertainty can produce poor decisions, while the two concepts may need to be modeled separately.

  • Two uncertainties: Distributional reinforcement learning models distributions over outcomes, not epistemic uncertainty in mean beliefs.The paper emphasizes that these are distinct kinds of uncertainty: unknown beliefs versus stochastic outcomes.
  • Two uncertainties: For exploration, Bayesian uncertainty about beliefs should prioritize acquiring new knowledge, whereas distributional reinforcement learning approximates outcome variability.The passage states that both notions can occur in the same decision problem but serve different roles.
  • Decision consequences: Using a distributional value function as a posterior can lead to arbitrarily bad decisions, even when the agent has full information.The example contrasts correct Thompson sampling, which selects the better action with zero regret, against sampling over the distributional value function.
  • Scope and open questions: Distributional value functions can be poor proxies for Bayesian uncertainty, while Bayesian uncertainty can likewise be poor for risk-sensitive outcome modeling.The paper notes that the practical benefit of modeling outcome distributions for expected cumulative reward remains unclear.
  • Count-based comparison: Count-based exploration can require ˜O measurements to cover a linear-bandit space to radius ϵ, whereas efficient exploration requires only ˜O(d) measurements.The passage states that Thompson sampling with a linear model naturally achieves the latter scaling.

C.5 Ensembles without priors

The section shows that ensembles without priors can misrepresent uncertainty outside the observed data. Bootstrapping captures data variability but cannot create uncertainty for unobserved outcomes, motivating randomized prior functions.

  • Naive ensembles: Naive ensembles fit observed data exactly yet can produce almost zero uncertainty across the input space.This is identified as a drawback of ensembles trained only from random initializations.
  • Bootstrap ensembles: Bootstrapped ensembles widen predictive uncertainty where data variability changes, but can remain nearly certain for unobserved zero-target regions.When targets are always zero, bootstrapping reproduces zero targets and fails to generate uncertainty in the opposite tail.
  • Exploration consequences: Without prior uncertainty, an agent that has observed only zero reward cannot simulate positive rewards through bootstrapping or ensembling alone.The paper identifies this as a source of arbitrarily poor decisions.
  • Prior mechanisms: Adding a prior mechanism can remedy the missing uncertainty through l2 regularization toward initial random weights or a fixed additive random prior network.The paper's main contribution is to emphasize the importance of such a prior mechanism in ensemble uncertainty.
  • Prior mechanisms: The proposed combination of bootstrap sampling and prior functions is presented as an effective computational approximation to Bayesian inference in deep reinforcement learning.Training multiple ensemble members is more computationally expensive than training one model, but can be parallelized.
Loading 1806.03335v2…