PaperlayerView exact revision on arXiv

Source-linked AI summary

Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic

Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E. Turner, Sergey Levine

arXiv:1611.02247v3cs.LG

TL;DR

Deep RL needs methods that are both sample-efficient and stable, because policy gradients can be high-variance while off-policy methods can be biased. Q-Prop uses a Taylor expansion of an off-policy critic as a control variate, combining on-policy Monte Carlo gradients with off-policy variance reduction. The paper reports substantial sample-efficiency gains over TRPO and improved stability over DDPG on MuJoCo continuous-control tasks.

  • Problem

    Deep RL faces a trade-off between stable but high-variance policy gradients and sample-efficient but biased off-policy methods.

  • Method

    Q-Prop uses the first-order Taylor expansion of an off-policy critic as a control variate for an on-policy Monte Carlo policy-gradient estimator.

  • Results

    Q-Prop provides large sample-efficiency improvements over TRPO and outperforms actor-critic methods on challenging tasks such as humanoid locomotion.

  • Takeaways & Limitations

    Combining on-policy Monte Carlo gradient estimation with an off-policy critic can produce RL methods that are more stable and efficient.

  • Takeaways & Limitations

    Q-Prop remains limited by the unresolved problem of determining when an off-policy critic is reliable.

Abstract

from arXiv · show

Model-free deep reinforcement learning (RL) methods have been successful in a wide variety of simulated domains. However, a major obstacle facing deep RL in the real world is their high sample complexity. Batch policy gradient methods offer stable learning, but at the cost of high variance, which often requires large batches. TD-style methods, such as off-policy actor-critic and Q-learning, are more sample-efficient but biased, and often require costly hyperparameter sweeps to stabilize. In this work, we aim to develop methods that combine the stability of policy gradients with the efficiency of off-policy RL. We present Q-Prop, a policy gradient method that uses a Taylor expansion of the off-policy critic as a control variate. Q-Prop is both sample efficient and stable, and effectively combines the benefits of on-policy and off-policy methods. We analyze the connection between Q-Prop and existing model-free algorithms, and use control variate theory to derive two variants of Q-Prop with conservative and aggressive adaptation. We show that conservative Q-Prop provides substantial gains in sample efficiency over trust region policy optimization (TRPO) with generalized advantage estimation (GAE), and improves stability over deep deterministic policy gradient (DDPG), the state-of-the-art on-policy and off-policy methods, on OpenAI Gym's MuJoCo continuous control environments.

1 INTRODUCTION

Deep RL is powerful but difficult to deploy because expressive neural methods can be unstable and sample-inefficient. Q-Prop combines on-policy policy-gradient stability with an off-policy critic to improve efficiency without adding bias.

  • Motivation: Deep reinforcement learning often requires extensive hyperparameter sweeps because poor settings can produce unstable or non-convergent learning.Its high sample complexity can also make training impractical on real physical systems.
  • Motivation: Monte Carlo policy gradients provide unbiased or nearly unbiased estimates but often suffer from high variance.Policy constraints and value-based backups are among the techniques proposed to manage this variance and difficult optimization landscapes.
  • Q-Prop: Q-Prop uses an off-policy critic as a Taylor-expansion control variate, combining an analytical critic gradient with a Monte Carlo residual gradient.The method aims to reduce gradient variance without adding bias while retaining off-policy sample efficiency.
  • Q-Prop: Q-Prop combines the advantages of on-policy policy gradients with the efficiency of off-policy learning.The paper positions this combination as a way to develop deep RL methods that are both data efficient and stable.

2 BACKGROUND

Policy-gradient methods offer direct optimization with comparatively reliable gradients but face high variance and on-policy data requirements. Actor-critic methods improve sample efficiency through off-policy critics, while their policy-gradient estimates are generally biased and harder to analyze for stability.

  • Monte Carlo Policy Gradient Methods: Monte Carlo policy gradients directly optimize the RL objective using sampled returns, but their gradient estimates have very high variance.A baseline such as the state-value function is used to reduce this variance.
  • Monte Carlo Policy Gradient Methods: The policy-gradient formulation is unbiased in theory, but practical use of undiscounted state visitation frequencies can make it biased.The paper distinguishes this issue from bias caused by function approximation, off-policy learning, and value backups.
  • Monte Carlo Policy Gradient Methods: On-policy sampling makes policy-gradient optimization sample-intensive, while importance sampling for off-policy trajectories can struggle in high-dimensional action spaces.The difficulty arises from rapidly degenerating importance weights.
  • Policy Gradient with Function Approximation: Actor-critic methods use TD learning to fit a critic and then improve the policy against that critic, potentially gaining sample efficiency from off-policy data.DDPG is presented as a relevant example using a replay buffer and a deterministic policy.
  • Policy Gradient with Function Approximation: DDPG avoids high-variance REINFORCE gradients and trains on off-policy data, but its policy-gradient estimator is generally biased.This bias makes convergence and stability properties difficult to analyze.

3 Q-PROP

Q-Prop uses a Taylor expansion of an off-policy critic as a control variate, combining an analytic critic gradient with a residual Monte Carlo policy-gradient term. Adaptive variants regulate control-variate strength to reduce variance while retaining on-policy stability and incorporating off-policy data.

  • Q-Prop estimator: Q-Prop combines a high-variance, nearly unbiased Monte Carlo gradient with a deterministic, biased critic gradient through a control variate.The resulting estimator uses both gradient signals while preserving the stability of on-policy Monte Carlo policy gradients.
  • Q-Prop estimator: The control variate is formed from a first-order Taylor expansion of a differentiable action-dependent function, with the critic Qw evaluated around the policy’s expected action µθ(st).In the practical construction, f is chosen as Qw and the expansion point as µθ(st).
  • Q-Prop estimator: Q-Prop’s gradient contains an analytic gradient through Qw and a residual REINFORCE gradient, while Qw can be trained with off-policy data.This makes the method resemble an actor-critic algorithm rather than only a Monte Carlo policy-gradient estimator.
  • Q-Prop estimator: If Qw approximates Qπ well, it provides a reliable gradient, reduces estimator variance, and improves convergence rate.The control-variate analysis also indicates that accurate approximation is not the only condition under which Q-Prop can reduce variance.
  • Control-variate analysis and adaptive Q-Prop: The weighting variable η(st) changes control-variate strength without introducing estimator bias, and a surrogate variance measure enables practical adaptation from single-action samples.The surrogate uses Varat(Â(st,at)), while covariance with the Taylor-expanded advantage can be estimated analytically or from one action sample.
  • Control-variate analysis and adaptive Q-Prop: Fully adaptive Q-Prop achieves guaranteed variance reduction when the Taylor-expanded critic advantage correlates with the Monte Carlo advantage, even if Qw is not accurate in absolute terms.Conservative Q-Prop uses η(st)=1 only for positive estimated covariance, whereas aggressive Q-Prop uses its sign.

4 RELATED WORK

Q-Prop combines off-policy critic learning with on-policy policy updates to reduce variance without introducing further bias. It builds on prior work in policy-gradient variance reduction and can be paired with alternative update and critic-training methods.

  • Action-dependent control variates are a relatively recent direction within the broader literature on policy-gradient variance reduction.Most prior work instead focuses on simpler baselining techniques.
  • Model-based Q-Prop remains an alternative direction when the dynamics are learned, where it can correct bias in model-based learning.
  • Q-Prop uses off-policy data to improve on-policy policy-gradient methods without importance weighting or added bias.Prior approaches either introduce bias or suffer from degenerate importance weights and high variance in high-dimensional action spaces.
  • Q-Prop combines on-policy policy updates with off-policy critic learning, allowing it to draw on both research traditions.The authors implement it on top of TRPO-GAE for fair comparison, while identifying other combinations as future work.

5 EXPERIMENTS

Experiments on OpenAI Gym MuJoCo continuous-control domains compare Q-Prop variants with TRPO, VPG, and DDPG. Conservative Q-Prop is the most stable variant, improves sample efficiency over TRPO, and performs especially well on complex domains such as Humanoid.

  • 5.1 ADAPTIVE Q-PROP: Q-Prop variants significantly outperform TRPO in sample efficiency on HalfCheetah-v1, with conservative Q-Prop reaching average reward 4000 using about 10 times fewer samples.Conservative Q-Prop also performs more stably than standard and aggressive variants.
  • 5.1 ADAPTIVE Q-PROP: Q-Prop methods learn with just 1 episode per update, whereas TRPO may require 25000 steps or 25 episodes per update to perform well.The comparison across batch sizes indicates reduced gradient-estimate variance for Q-Prop.
  • 5.1 ADAPTIVE Q-PROP: Conservative Q-Prop is used as the main implementation because stability is important for reliably applying deep RL to real-world tasks.
  • 5.2 EVALUATION ACROSS ALGORITHMS: On HalfCheetah-v1, conservative Q-Prop variants significantly outperform the best TRPO and VPG methods, while Q-Prop with vanilla policy gradient is comparable to TRPO.DDPG shows inconsistent performance and is sensitive to reward scaling and hyperparameter settings; Q-Prop displays more stable, monotonic learning.
  • 5.3 EVALUATION ACROSS DOMAINS: Across multiple domains, Q-Prop consistently improves sample complexity over TRPO and sometimes achieves higher rewards than DDPG in more complex domains.On Humanoid-v1, Q-Prop substantially improves sample efficiency over TRPO while DDPG cannot find a good solution.
  • 5.3 EVALUATION ACROSS DOMAINS: More complex domains may contain narrower regions of stable hyperparameters, making costly hyperparameter sweeps impractical for less stable algorithms.

6 DISCUSSION AND CONCLUSION

Q-Prop combines on-policy policy-gradient estimation with a sample-efficient off-policy critic used as a control variate. It improves sample efficiency over TRPO and outperforms actor-critic methods on challenging humanoid locomotion tasks.

  • Q-Prop combines potentially unbiased on-policy gradient estimation with a sample-efficient off-policy critic acting as a control variate.
  • Q-Prop provides a large improvement in sample efficiency compared with state-of-the-art policy-gradient methods such as TRPO.
  • Q-Prop outperforms state-of-the-art actor-critic methods on challenging tasks such as humanoid locomotion.

A Q-PROP ESTIMATOR DERIVATION

The Q-Prop estimator derives a control variate from a first-order Taylor expansion of a differentiable function around the stochastic policy's mean action.

  • The derivation of the Q-Prop estimator uses a property commonly applied in baseline derivations.
  • A first-order Taylor expansion approximates a differentiable function around the policy mean action.
  • The expansion is formed as f(st, ¯at)+∇a f(st,a)|a= ¯at(at −¯at).

B CONNECTION BETWEEN Q-PROP AND COMPATIBLE FEATURE APPROXIMATION

Compatible-feature actor-critic can be interpreted as a control-variate method, while Q-Prop instead uses a Taylor-expanded critic; variance reduction depends on estimating the Fisher matrix and critic weights effectively.

  • Compatible-feature approximation defines a critic whose features equal the policy score function and fits it to on-policy Monte Carlo returns.
  • Q-Prop uses the Taylor expansion of the critic as its control variate, rather than directly using the compatible critic.
  • Variance reduction depends on effectively computing or estimating Fisher's information matrix and the critic parameter vector w.

C UNIFYING POLICY GRADIENT AND ACTOR-CRITIC

A generalized Q-Prop estimator connects policy-gradient and deterministic actor-critic methods through adjustable variables that control bias and sampling distributions, while allowing flexible critic choices.

  • C UNIFYING POLICY GRADIENT AND ACTOR-CRITIC: The generalized Q-Prop formulation introduces α and ρ_CR to analyze connections between policy-gradient and actor-critic methods.
  • C UNIFYING POLICY GRADIENT AND ACTOR-CRITIC: Bias is generally introduced only when α ≠ 1 or ρ_CR ≠ ρ_π in the generalized estimator.
  • C UNIFYING POLICY GRADIENT AND ACTOR-CRITIC: Standard policy gradient is recovered with η = 0, whereas deterministic actor-critic is recovered with α = 0 and ρ_CR = ρ_β.
  • C UNIFYING POLICY GRADIENT AND ACTOR-CRITIC: The critic Q_w can be any function while the gradient remains almost unbiased, and off-policy temporal-difference learning can estimate it.
  • C UNIFYING POLICY GRADIENT AND ACTOR-CRITIC: Table 2 summarizes implementation options and edge cases of the generalized Q-Prop estimator in Eq. 17.

D EXPERIMENT DETAILS

The experiments use benchmark-based neural architectures for policy and value functions, with distinct stochastic and deterministic policy forms. Training compares specified batch sizes, step sizes, and learning rates through hyperparameter searches.

  • Policy and value function architectures: Policy-gradient methods use a local Gaussian policy with a state-dependent mean and diagonal global covariance.The policy is πθ(at|st) = N(µθ(st),Σθ).
  • Policy and value function architectures: The policy mean network has three hidden layers of sizes 100-50-25, tanh nonlinearities in the first two layers, and a diagonal covariance matrix.
  • Policy and value function architectures: DDPG uses a deterministic policy with the same architecture as the described policy network.
  • Training details: Policy-gradient experiments search batch sizes of 1000, 5000, and 25000 time steps.
  • Training details: The training setup searches trust-region step sizes of 0.1, 0.01, and 0.001, plus Adam base learning rates of 0.001 and 0.0001 for vanilla policy gradients.
Loading 1611.02247v3…