PaperlayerView exact revision on arXiv

Source-linked AI summary

Bayesian policy gradient and actor-critic algorithms

Mohammad Ghavamzadeh, Yaakov Engel, Michal Valko

arXiv:2604.27563v1cs.LG

TL;DR

Conventional policy-gradient methods face high-variance Monte Carlo estimates, motivating more sample-efficient gradient inference. The paper introduces Gaussian-process Bayesian policy gradients and Bayesian actor-critic methods, deriving posterior gradient updates and evaluating them against classic Monte Carlo methods. The Bayesian policy-gradient method supports non-Markovian settings, while the actor-critic formulation exploits Markov transitions when available.

  • Problem

    Conventional Monte Carlo policy-gradient estimates can have high variance and require many samples, limiting sample efficiency.

  • Method

    The paper models policy gradients and non-parametric action-value critics with Gaussian processes, using Bayesian conditioning and compatible kernels to derive posterior gradient updates.

  • Results

    The proposed Bayesian policy-gradient and actor-critic algorithms provide gradient uncertainty estimates and are experimentally compared with classic Monte Carlo policy-gradient methods across several reinforcement-learning problems.

  • Takeaways & Limitations

    Bayesian policy gradients handle partially observable and non-Markovian systems, while Bayesian actor-critic methods use state-action-reward transitions to exploit the Markov property.

  • Takeaways & Limitations

    Trajectory-based Bayesian policy gradients cannot exploit the Markov property in Markovian systems and may have larger gradient-estimate variance for long trajectories.

Abstract

from arXiv · show

Policy gradient methods are reinforcement learning algorithms that adapt a parameterized policy by following a performance gradient estimate. Conventional policy gradient methods use Monte-Carlo techniques to estimate the gradient, which tend to have high variance, requiring many samples and resulting in slow convergence. We first propose a Bayesian framework for policy gradient, based on modeling the policy gradient as a Gaussian process. This reduces the number of samples needed to obtain accurate gradient estimates. Moreover, estimates of the natural gradient and a measure of the uncertainty in the gradient estimates, namely, the gradient covariance, are provided at little extra cost. Since the proposed framework considers system trajectories as its basic observable unit, it does not require the dynamics within trajectories to be of any particular form, and can be extended to partially observable problems. On the downside, it cannot exploit the Markov property when the system is Markovian. To address this, we supplement our Bayesian policy gradient framework with a new actor-critic learning model in which a Bayesian class of non-parametric critics, based on Gaussian process temporal difference learning, is used. Such critics model the action-value function as a Gaussian process, allowing Bayes rule to be used to compute the posterior distribution over action-value functions, conditioned on the observed data. Appropriate choices of the policy parameterization and of the prior covariance (kernel) between action-values yield closed-form expressions for the posterior of the gradient of the expected return with respect to the policy parameters. We perform detailed experimental comparisons of the proposed Bayesian policy gradient and actor-critic algorithms with classic Monte-Carlo based policy gradient methods, on a number of reinforcement learning problems.

1. Introduction

Policy-gradient methods improve parameterized policies using estimated performance gradients, but Monte Carlo estimates can have high variance and require many samples. The paper proposes Bayesian policy-gradient and actor-critic approaches that use Gaussian processes to improve estimation while representing uncertainty.

  • Motivation: High-variance gradient estimates make conventional policy-gradient methods sample-inefficient and slow to converge.Artificial discounting can introduce bias, while baselines reduce variance without biasing the gradient estimate.
  • Existing approaches: Natural policy gradients make policy updates invariant to bijective reparameterizations of the policy.This replaces the ordinary gradient with the natural policy gradient.
  • Existing approaches: Actor-critic methods combine an actor that updates the policy with a critic that estimates its value function.The critic supplies feedback used by the actor to improve performance.
  • Contributions: The Bayesian actor-critic component uses Gaussian-process temporal-difference critics, extending Bayesian gradient estimation to a setting that can exploit Markov structure.The paper also develops vector-valued Bayesian quadrature, sparse algorithms, additional proofs, and experiments in random walk, Mountain Car, and Ship Steering.
  • Contributions: The paper models policy gradients with Gaussian processes, reducing samples needed for accurate estimates while also providing natural-gradient and gradient-covariance estimates.The framework operates on complete trajectories and therefore can handle partially observable and non-Markovian systems, but cannot exploit Markov structure when it is available.

2. Reinforcement Learning, Policy Gradient, and Actor-Critic Methods

This section formulates reinforcement learning, policy-gradient, and actor-critic methods through policies, trajectories, returns, value functions, and gradient estimates. It also identifies how baselines and compatible critics can reduce gradient-estimation variance.

  • Reinforcement learning: Reinforcement learning studies agents interacting with dynamic, stochastic, incompletely known environments to optimize long-term performance.The interaction is conventionally modeled as a Markov decision process or related partially observable process.
  • Reinforcement learning: A stationary policy maps each state to a probability distribution over actions and induces a Markov chain over state-action pairs.Trajectories, transition probabilities, occupancy densities, cumulative returns, and expected returns are defined from this induced process.
  • Policy-gradient methods: Policy-gradient methods estimate the expected-return gradient with respect to policy parameters and update those parameters in the gradient direction.The policy-gradient formulation assumes continuous differentiability of the policy in its parameters.
  • Policy-gradient methods: Monte Carlo gradient estimates are unbiased and converge with increasing sample count, but conventional methods can require excessive samples because of high variance.The score-function method estimates gradients from independently sampled paths.
  • Actor-critic methods: Actor-critic methods replace the action-value function with a learned critic, whose feedback guides the separately parameterized actor.Compatible approximation conditions allow the learned action-value function to replace the true one in the policy-gradient expressions.
  • Actor-critic methods: When the compatible critic parameter is optimal, choosing the baseline as the value function minimizes variance in the action-value estimator.The compatible approximation uses policy score features in a linear action-value representation.

3. Bayesian Quadrature

Bayesian quadrature evaluates integrals by placing Gaussian-process priors over unknown integrand components and conditioning them on sampled data. The section extends this framework to vector-valued integrals relevant to policy gradients, with analytic posterior moments under structured kernels.

  • Bayesian quadrature: Bayesian quadrature evaluates an integral using samples of its integrand rather than relying solely on conventional Monte Carlo sampling.Monte Carlo typically samples from the target density, whereas Bayesian quadrature permits samples from arbitrary distributions.
  • Bayesian quadrature: A Gaussian-process prior represents uncertainty about an unknown integrand component, and Bayes’ rule produces a posterior over the integral after noisy observations.The kernel incorporates prior knowledge about the integrand’s smoothness.
  • Bayesian quadrature: The posterior integral remains Gaussian because integration is linear, with posterior moments computed from the prior and observed data.The posterior mean and variance are given by closed-form expressions involving the kernel and measurement-noise covariance.
  • Bayesian quadrature: Analytic posterior integration requires choosing the integrand partition and kernel so the resulting integrals can be solved in closed form.The paper provides corresponding analysis for Fisher-kernel constructions used in its policy-gradient models.
  • Vector-valued integrals: Vector-valued integrals arise when either the GP is vector-valued with scalar weighting or the GP is scalar-valued with vector weighting.These alternatives imply different data-generation models and covariance structures.
  • Vector-valued integrals: Modeling cross-component correlations requires matrix-valued kernels and correlated noise specifications; ignoring existing correlations can use data suboptimally.Under simplifying assumptions, the vector integral can instead be treated as independent component-wise integrals.

4. Bayesian Policy Gradient

The paper formulates Bayesian policy-gradient estimation with Gaussian processes, deriving posterior gradient moments and two Fisher-kernel models with closed-form computations. It also gives an evaluation algorithm and notes a kernel-choice limitation.

  • Bayesian formulation: Vector-valued Bayesian quadrature estimates the policy-gradient posterior from observed data, producing posterior means and covariances.The framework models the gradient-related integrand with a Gaussian process and conditions on observed data.
  • Model 1: The first model places a vector-valued GP prior on noisy measurements R(ξ)∇log Pr(ξ; θ), treating each gradient component independently.The model uses a common kernel, noise covariance, and independent component assumptions.
  • Model 2: The second model uses a scalar GP for expected path returns and a vector-valued policy-distribution derivative to obtain gradient posterior moments.Its noisy observations are the actual returns R(ξi), while the Fisher kernel supports closed-form expressions.
  • Kernel choice: Quadratic Fisher and Fisher kernels make the posterior gradient moments analytically tractable, motivating their selection for the two models.The corresponding closed forms use the Fisher score and Fisher information matrix.
  • Limitation: Restricting kernels to Fisher forms enables closed-form moments but introduces a problem-dependent representation error whose magnitude is difficult to quantify.The experiments reported in the paper indicate that this restriction did not significantly harm gradient estimates.
  • Evaluation algorithm: The generic BPG evaluation algorithm samples M paths, computes returns and measurements, updates the kernel matrix, and returns posterior gradient moments.It adds measurement error to the covariance matrix before computing the posterior moments.

5. Extension to Partially Observable Markov Decision Processes

The Bayesian policy-gradient models extend to partially observable Markov decision processes by replacing state inputs with observations. The extension also supports finite observation histories and policies with internal state.

  • POMDP setting: In a POMDP, the policy observes an observation process dependent on the state rather than directly observing the state.The policy maps observations to action distributions.
  • Extension: The Bayesian policy-gradient models and algorithms apply to the partially observable case without changes when observations substitute for states.The Fisher-score expression remains the same as in the observable case under this substitution.
  • Policy inputs: The POMDP Bayesian policy-gradient algorithm can also handle policies based on any finite observation history.This follows the corresponding extension described for GPOMDP.
  • Policy inputs: The algorithm can be extended to policies with internal state, following the analogous GPOMDP extension.The paper states this extension directly for its BPG POMDP algorithm.

6. BPG Experimental Results

Experiments compare Bayesian quadrature and Monte-Carlo gradient estimation, then evaluate Bayesian policy-gradient variants on bandit and LQR problems. Bayesian estimates generally reduce variance, while covariance-aware updates improve policy optimization across tested sample sizes.

  • Gradient Estimation: BQ gradient estimates had lower variance than MC estimates by one order of magnitude at M = 10 and six orders at M = 100.BQ was also more accurate at M = 100 and roughly equally accurate at M = 10.
  • Gradient Estimation: The LQR gradient experiments measured MSE and mean absolute angular error across sample sizes using BQ Models 1 and 2 with sparsification.Results were averaged over 10^4 runs, with angular-error bars showing standard errors of the mean.
  • Gradient Estimation: Adding Gaussian reward noise slightly increased BQ and MC gradient-estimation error, while comparisons remained similar to the noiseless results.The experiments used reward-noise standard deviations σ_r = 0.1 and 1.
  • Policy Optimization: BPG and BPNG were compared with MCPG on LQR optimization using sample sizes M = 5, 10, 20, and 40 over 100 updates.MCPG performed better only at the smallest sample size in the reported comparison.
  • Policy Optimization: BPG-var outperformed BPG and MCPG for all tested sample sizes and exceeded MCPG even at M = 5.BPG-var selects smaller learning rates when gradient variance is large and larger updates when it is small.
  • Policy Optimization: BPG-var converged faster than BPNG with similar final performance, while BPG and BPG-var became increasingly similar as sample size increased.The paper attributes this convergence in behavior to more reliable posterior-mean gradient estimates at larger sample sizes.

7. Bayesian Actor-Critic

The Bayesian actor-critic framework models action-value functions with Gaussian processes and uses transition-level observations to derive posterior policy-gradient moments. Its Fisher-kernel design enables analytic computation, while its transition-based formulation exploits Markov structure but imposes kernel restrictions and approximation trade-offs.

  • Scope and limitations: The trajectory-based formulation supports non-Markovian and partially observable systems, but cannot exploit Markov structure and has higher gradient-estimate variance on long trajectories.Transition-based actor-critic methods use current state, action, and next state observations to take advantage of the Markov property.
  • Bayesian actor-critic framework: Bayesian actor-critic algorithms place a Gaussian-process prior on action-value functions and compute their posterior from observed state-action-reward transitions.The framework is based on Gaussian process temporal-difference learning and applies Bayes’ rule to infer action-value functions.
  • Posterior learning: As more samples are observed, the posterior covariance of the Q-function decreases, reflecting increasing confidence in its estimate.The posterior mean and covariance functions define the posterior Gaussian process over action-value functions.
  • Bayesian actor-critic framework: The posterior policy-gradient moments follow from the posterior moments of the action-value function and the gradient’s linearity in Q.This yields general expressions for the posterior mean and covariance of the policy gradient.
  • Kernel design: Choosing a state kernel plus the invariant Fisher kernel makes the required integrals analytically tractable and supports closed-form gradient-posterior expressions.The Fisher kernel is invariant to policy reparameterization and depends on score vectors and the Fisher information matrix.
  • Scope and limitations: Restricting state-action dependence to the Fisher kernel can cause problem-dependent approximation error, although it was not significant in the reported experiments.The authors report that BAC gradients were generally more accurate than Monte-Carlo estimates with the same number of samples.

8. BAC Experimental Results

The experiments compare Bayesian actor-critic methods with Monte-Carlo and Bayesian policy-gradient approaches across random walk, mountain car, and ship steering tasks. BAC generally provides more accurate, lower-variance gradients and robust policy learning with few episodes.

  • Experimental setup: The evaluation covers a 10-state random walk, continuous-state mountain car, and continuous-state continuous-action ship steering problem.Ship steering uses four continuous state variables and one continuous action; mountain car uses a two-dimensional continuous state and three actions.
  • 8.1 A Random Walk Problem: BAC showed robust policy-learning performance across sample sizes M = 1, 25, 50, and 75, while BPG improved as M increased.BAC never generated a policy whose episode failed to end after 10^6 steps; BPG performed worse than MCPG at M = 1 and 25 and approached BAC at M = 100.
  • 8.2 Mountain Car: BAC outperformed MCPG and had lower variance in mountain car, finding a good policy with only M = 5 episodes.MCPG improved with larger sample sizes and reached similar performance at M = 40, but with a slower rate.
  • 8.3 Ship Steering: In ship steering, BAC converged to a better success ratio than MCPG for every tested sample size and usually showed less performance variance.MCPG was initially slightly better at M = 5, whereas BAC was better from the beginning for M = 10 and 20.

9. Other Advancements in Bayesian Reinforcement Learning

The paper situates Bayesian reinforcement learning across model-free, model-based, inverse, multiagent, multitask, and bandit settings. These approaches use Bayesian distributions over system models, values, policies, rewards, or shared task components.

  • Model-free and model-based Bayesian RL: Bayesian model-free RL maintains posteriors over values or policies without assuming known dynamics or explicitly constructing a system model.The paper distinguishes this class from model-based Bayesian RL, which maintains a posterior over model parameters.
  • Model-free and model-based Bayesian RL: Model-based Bayesian RL uses posterior model uncertainty to balance exploration and exploitation in MDP and POMDP settings.The surveyed methods include both offline and online MDP algorithms.
  • Inverse RL: Bayesian inverse RL places a prior over reward preferences and derives a distribution over reward functions from compatibility with observed expert behavior.The likelihood represents compatibility between the expert policy and the inferred reward function.
  • Multiagent RL: Bayesian multiagent RL additionally maintains uncertainty over the policies of other agents.This extends the posteriors maintained in single-agent Bayesian RL over models, values, or policies.
  • Multitask RL and bandits: Bayesian multitask RL assumes tasks share components drawn from a common generative model, while Bayesian bandit methods target regret minimization.Shared components may include dynamics, reward structure, or value function.

10. Discussion

The discussion presents Bayesian policy-gradient and actor-critic methods as alternatives to high-variance Monte-Carlo gradient estimation. It highlights closed-form posterior updates, uncertainty estimates, empirical accuracy, and remaining scalability questions.

  • Bayesian policy gradients: Bayesian quadrature models the policy gradient as a Gaussian process and computes posterior moments using Bayes’ rule.Appropriate partitioning of the integrand and prior selection yield closed-form posterior moments for the expected-return gradient.
  • Bayesian actor-critic: BAC extends the framework with Gaussian-process critics and closed-form posterior policy-gradient updates compatible with parametric policies.The posterior mean updates the policy, while the posterior covariance gauges update reliability.
  • Bayesian actor-critic: BAC uses individual state-action-reward transitions and therefore exploits the Markov property when system trajectories are Markovian.The paper reports that BAC produced more accurate policy-gradient estimates than both BPG models for the same amount of data.
  • Additional properties and evaluation: Natural-gradient variants of both BPG and BAC require little extra cost, and sparse forms improve time and memory efficiency.The methods were evaluated against Monte-Carlo policy-gradient algorithms and BPG on random walk, mountain car, and ship steering.
  • Limitations and future work: The authors identify larger, more realistic continuous high-dimensional domains as requiring additional evaluation.They also conjecture that posterior second-order statistics could support more efficient algorithms, beyond the posterior mean currently used for updates.

Appendix A. Proof of Proposition 3

The appendix proves Proposition 3 by algebraic manipulation of Fisher-kernel expressions and integral identities. The proof uses the positive-definite eigendecomposition of the Fisher information matrix.

  • Proof construction: The proof begins with an M × 1 vector b and expresses its ith element before applying Fisher-kernel substitutions.The derivation uses algebra, integral replacement by the Fisher information matrix G, and the identity ∇(1) = 0.
  • Proof construction: A parallel derivation treats the scalar b0 by substituting the quadratic Fisher kernel and replacing an integral with the Fisher information matrix G.The stated steps include algebra and the identity involving u(ξ) Pr(ξ; θ).
  • Eigendecomposition: Because G is positive definite and symmetric, the proof writes it as G = VΛV^⊤ using orthonormal eigenvectors and eigenvalues.Subsequent steps use Gv_i = λ_iv_i and eigenvector orthonormality to complete the claim.

Appendix B. Proof of Proposition 4

The proof of Proposition 4 derives the stated matrix identity through substitutions involving the Fisher kernel, policy-gradient identity, and Fisher information matrix.

  • The proof begins by expressing an n × M matrix using the vectors u(ξ1), …, u(ξM).
  • The derivation substitutes the kernel with the Fisher kernel and uses ∇Pr(ξ; θ) = u(ξ) Pr(ξ; θ).
  • Replacing the relevant integral with the Fisher information matrix G completes the algebraic proof.

Appendix C. Proof of Proposition 5

The proof of Proposition 5 applies sparse kernel approximations to the proof for Model 1, preserving b0 while modifying b through the sparse representation.

  • The proof is shown for Model 1, while Model 2 is stated to follow by the same arguments.
  • The kernel matrix K and kernel vector k(·) are replaced by sparse approximations A K̃ A⊤ and A k̃(·).
  • Sparsification leaves b0 equal to n+1 but modifies b.
  • For inducing points ξi ∈ D̃, the sparse coefficients satisfy (b̃)i = 1 + u(ξi)⊤G−1u(ξi).
  • The proposition then follows by replacing b with A b̃ and applying Lemma 1.3.2 from Engel (2005).

Appendix D. Proof of Proposition 6

The proof of Proposition 6 starts from the matrix Bt and reduces its columns through definitions, Fisher-kernel substitutions, Fisher information, and algebraic simplification.

  • The proof begins with the n × (t + 1) matrix Bt and represents its ith column explicitly.
  • Successive lines use the definitions of Bt, g, and k, followed by algebraic rearrangement.
  • The derivation substitutes the policy definition πµ and Fisher kernel kF into the expression.
  • Replacing the integral with the Fisher information matrix G completes the remaining algebra and establishes the claim.
Loading 2604.27563v1…