Source-linked AI summary
Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning
Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E. Turner, Bernhard Schölkopf, Sergey Levine
TL;DR
The paper addresses how to combine sample-efficient off-policy learning with more stable on-policy policy gradients. It develops interpolated policy gradients using control variates, analyzes their bias, and evaluates the resulting family empirically. The method unifies prior techniques and often improves sample efficiency and stability over prior state-of-the-art methods.
Problem
Off-policy updates improve data efficiency but can introduce bias and instability, while on-policy gradients are often sample-intensive; the paper studies how to combine their advantages.
Method
IPG uses control variates to mix unbiased, high-variance likelihood-ratio gradients with low-variance, biased deterministic gradients from an off-policy critic.
Results
Interpolated gradients improve sample efficiency and stability in many cases, with ν = 0.2 consistently outperforming Q-Prop, TRPO, and prior actor-critic methods in the reported comparisons.
Takeaways & Limitations
IPG provides a unified family connecting on-policy likelihood-ratio and off-policy deterministic policy gradients, with prior methods appearing as approximate limiting cases.
Takeaways & Limitations
Off-policy actor-critic gradients can have potentially unbounded bias and are generally less stable because of critic estimation error and off-policy state sampling.
Abstract
from arXiv · showhide
Off-policy model-free deep reinforcement learning methods using previously collected data can improve sample efficiency over on-policy policy gradient techniques. On the other hand, on-policy algorithms are often more stable and easier to use. This paper examines, both theoretically and empirically, approaches to merging on- and off-policy updates for deep reinforcement learning. Theoretical results show that off-policy updates with a value function estimator can be interpolated with on-policy policy gradient updates whilst still satisfying performance bounds. Our analysis uses control variate methods to produce a family of policy gradient algorithms, with several recently proposed algorithms being special cases of this family. We then provide an empirical comparison of these techniques with the remaining algorithmic details fixed, and show how different mixing of off-policy gradient estimates with on-policy samples contribute to improvements in empirical performance. The final algorithm provides a generalization and unification of existing deep policy gradient techniques, has theoretical guarantees on the bias introduced by off-policy updates, and improves on the state-of-the-art model-free deep RL methods on a number of OpenAI Gym continuous control benchmarks.
1 Introduction
Deep reinforcement learning faces a trade-off between sample-efficient but potentially unstable off-policy updates and stable but sample-intensive on-policy policy gradients. The paper frames control variates as a way to combine these approaches while connecting several prior methods.
- Off-policy replay improves data efficiency but often reduces stability and ease of use.
- The paper presents IPG as a framework that unifies prior combinations of on-policy and off-policy learning, including Q-Prop, PGQ, and ACER.
- On-policy likelihood-ratio gradients are unbiased but often high-variance and sample-intensive, whereas off-policy actor-critic gradients can be biased and less stable.
- Control variates combine likelihood-ratio gradients with deterministic gradients from an off-policy fitted critic to reduce variance and improve sample efficiency.
- The control-variate gradient estimator combines the likelihood-ratio and deterministic policy-gradient terms while retaining stable gradient estimates.
3 Interpolated Policy Gradient
Interpolated Policy Gradient (IPG) combines unbiased but high-variance likelihood-ratio gradients with biased but low-variance critic gradients. Its parameters and control variates span prior policy-gradient methods while enabling bias analysis and, in some cases, convergence guarantees.
- Core estimator: IPG interpolates likelihood-ratio gradients using on-policy value estimates with deterministic gradients through an off-policy fitted critic.The two components trade unbiasedness and variance against bias and lower variance.
- Core estimator: The critic-based estimator can be biased by off-policy state sampling and critic inaccuracies, but these biases can be bounded.Some cases retain monotonic convergence guarantees.
- Variance reduction: The interpolation parameter ν directly trades bias against variance, while a control variate further reduces likelihood-ratio estimator variance.The control-variate formulation replaces the likelihood-ratio signal with a residual involving on- and off-policy advantage estimates.
- Variance reduction: When the off-policy critic is accurate, the residual likelihood-ratio gradient has low magnitude and reduces overall estimator variance.This residual is formed from the difference between estimated and critic-based advantages.
- Unification: Three choices—β, ν, and control-variate use—interpolate a broad range of prior deep policy-gradient methods, including their limiting cases.The paper presents this relationship in Table 1 and Algorithm 1.
- Special case: The ν = 1 special case corresponds to a deterministic actor-critic method that can use a deterministic policy and learn completely off-policy.Its bias from off-policy sampling increases monotonically with the total variation or KL divergence between β and π.
4 Theoretical Analysis
The theoretical analysis bounds bias in interpolated policy gradients, including errors from off-policy sampling and critic approximation. It also establishes monotonic-convergence guarantees for a related algorithm under specified conditions, while noting practical implementation constraints.
- The analysis applies to general IPG methods and prior algorithms represented as special cases, clarifying how their biases arise.The framework covers IPG with and without control variates.
- Theorem 1 bounds off-policy sampling bias using the KL divergence between the behavior policy β and the actor policy.A separate term captures deviation from the local approximation as the reference policy ˜π moves away from π.
- Algorithm 2 guarantees non-decreasing returns and bounded off-policy sampling, but is often impractical to implement directly.Trust-region IPG with β ≠ π, ν = 1, and a control variate approximates this monotonic algorithm.
- Theorem 2 shows that general IPG bias depends on critic approximation error and the distance between the off-policy sampling policy and actor policy.The parameter ν scales bias from deterministic policy gradients while trading against the variance of likelihood-ratio gradients.
- ν provides a direct bias–variance trade-off: bias from the deterministic gradient grows proportionally with ν, whereas likelihood-ratio variance scales with (1 −ν)^2.
5 Related Work
The paper places IPG within prior approaches that combine on- and off-policy learning, including Q-Prop, PGQ, and ACER. It also distinguishes importance-sampling methods, which preserve unbiasedness at the cost of high variance, and identifies further policy-evaluation analysis as future work.
- IPG provides a common framework connecting Q-Prop, PGQ, and ACER, with intermediate settings relating closely to PGQ and ACER.
- Importance sampling can preserve unbiased policy-gradient estimates, but prior work associates it with considerable variance measured through effective sample size.
- Figure 1 reports faster learning for IPG-β-rand30000 than Q-Prop on HalfCheetah-v1 and stronger Ant-v1 performance for on-policy IPG-ν=1 than DDPG and IPG-ν=1-OU.The Ant comparison contrasts on-policy exploration with heuristic Ornstein–Uhlenbeck process noise.
- The effects of importance sampling on bias and variance are left for future work, while combining IPG with more sophisticated policy evaluation methods is suggested as a possible improvement.
6 Experiments
Experiments compare interpolated policy-gradient variants under shared algorithmic settings, showing benefits from replay sampling, trust-region-constrained off-policy data, and intermediate mixing values.
- Experimental setup: Experiments use common algorithmic structures and evaluate IPG variants on MuJoCo domains in OpenAI Gym.The comparisons hold remaining experimental details fixed, including shared batch-size conventions for relevant methods.
- Replay sampling: Random replay sampling accelerates convergence over Q-Prop, while larger random replay batches improve performance further.Using the 30000 most recent samples instead degrades performance, suggesting that decorrelation can stabilize gradients despite increasing the bias bound.
- Replay sampling: Replay samples from conservative trust-region policies provide greater regularity but less exploration than aggressively exploratory off-policy samples.The paper reports that keeping off-policy samples sufficiently close to the actor can be essential for good performance.
- Trust-region variant: The trust-region on-policy IPG variant improves monotonically on Ant-v1, whereas standard DDPG becomes stuck at a mean reward of 1000.Replacing trust-region exploration with DDPG-style temporally correlated Brownian noise causes large performance degradation.
- General IPG cases: Sweeps over ν found ν = 0.2 consistently better than Q-Prop, TRPO, and prior actor-critic methods, with intermediate ν values usually performing best.The comparison uses matched algorithm designs, and β ≠ π denotes replay-buffer samples matched in batch size to on-policy data.
7 Discussion
The paper presents interpolated policy gradients as a unified family that combines on-policy and off-policy learning while retaining performance bounds. Empirical results report improved sample efficiency and stability in many cases, while the theory explains when methods perform well or poorly.
- Discussion: Interpolated policy gradients mix off-policy and on-policy learning while satisfying performance bounds.The family unifies on-policy likelihood-ratio gradients with off-policy deterministic policy gradients and includes prior methods as approximate limiting cases.
- Discussion: Empirical results show improved sample efficiency and stability over prior state-of-the-art methods in many cases.The theoretical analysis provides intuition for the settings in which different methods perform well or poorly.
8 Proof for Theorem 1
The proof develops a local approximation objective whose first-order agreement with the true return enables tractable bias analysis. This framework supports bounds for the spectrum of interpolated policy-gradient objectives.
- Local approximation objective: The analysis introduces an approximate objective to make policy-gradient bias tractable to study.The approximation replaces the policy-dependent state-sampling distribution with a local reference distribution, following prior policy-optimization analyses.
- Local approximation objective: The expected-return objective and its approximation match in value and gradient when the reference policy equals the current policy.Specifically, J(π) = ˜J(π, ˜π = π) and their gradients agree at that point.
- Proof strategy: This first-order agreement supports monotonic-convergence analysis for a policy-iteration variant.The proof uses the relationship between the true and approximate objectives together with minorization-maximization.
- Bias bound: The proof relies on a lemma bounding the difference between the true return and the approximate objective through advantage terms.The lemma is used to derive bounds for the full spectrum of IPG objectives.
9 Proof for Monotonic Convergence in Algorithm 2
The proof uses a lower-bound construction and minorization-maximization to derive an off-policy policy-iteration algorithm with monotonic improvement guarantees. The guarantee weakens as the behavior policy becomes more off-policy, creating a trade-off between off-policy reuse and update progress.
- Corollary 1: Corollary 1 establishes a lower-bound chain linking the true objective to the local objective used by Algorithm 2.The proof also identifies equality when the current and updated policies coincide.
- Minorization-maximization: Algorithm 2 maximizes the minorizer at each iteration, guaranteeing J(πi+1) ≥ J(πi) regardless of the off-policy distribution or critic performance.This applies minorization-maximization to policy iteration with off-policy samples.
- Off-policy gap: The lower bound differs from the on-policy counterpart by an additional term involving the KL divergence between the updated and behavior policies.The comparison is framed against the monotonic-improvement analysis of Schulman et al.
- Off-policy gap: As the behavior policy becomes more off-policy, the bound gap widens and majorization steps may produce smaller policy changes, slowing learning.This expresses the practical trade-off introduced by off-policy samples.
10 Proof for Theorem 2
Theorem 2 extends the bias analysis to general IPG estimators by combining bounds for likelihood-ratio and off-policy actor-critic components. The ν = 1 case removes the likelihood-ratio term and connects IPG to an off-policy actor-critic method closely related to DDPG.
- Proof strategy: The proof first derives bounds for the local objectives and then combines them with Lemma 3.The same proof pattern is used to establish the required relationships among objectives.
- ν = 1 case: For ν = 1, the likelihood-ratio policy-gradient term is removed, yielding an off-policy actor-critic gradient expression.This isolates the deterministic-gradient component of the interpolated estimator.
- ν = 1 case: The resulting method is closely connected to DDPG but uses a stochastic policy, trust-region updates, and on-policy exploration without heuristic additive noise.It differs from DDPG by not using a target policy network.
- Bias bounds: Theorem 2 attributes general-case bias to δ, the error between the critic-estimated advantage and the true advantage.The theorem is assembled from Corollary 2, Theorem 1, and Lemma 3.
- Bias bounds: Applying a bound that removes ζ's dependence on π preserves the inequality but breaks equality, so the resulting algorithm no longer guarantees monotonic improvement.This marks a limitation of extending the bound in that way.
11 Control Variates for Policy Gradient
The paper describes control variates that combine likelihood-ratio policy gradients with critic-based deterministic gradients across continuous and discrete action spaces. The choices range from Taylor approximations to full critics, reparameterized estimators, and quadratic critics.
- Alternative control variates: The section presents control-variate choices beyond the first-order Taylor expansion used in Q-Prop.These alternatives broaden the available estimators within the IPG framework.
- Continuous actions: For continuous Gaussian policies, the full critic Qw can serve as a control variate, with Monte Carlo or reparameterization methods estimating its policy expectation.Reparameterization is used to reduce correction-term variance.
- Discrete actions: For discrete actions, the action-value vector Qw(st) provides critic values for the k available actions.The policy is represented as a probability vector over those actions.
- Continuous actions: For locally Gaussian continuous policies, the quadratic critic from Normalized Advantage Functions can be used directly as a control variate.This supplies a structured critic for continuous control.
- Deterministic gradients: The deterministic policy-gradient expression supplies another critic-based gradient form for the interpolated estimator.It represents the deterministic-gradient component used by the method.
12 Supplementary Experimental Details
The experiments fix optimization and critic settings across methods to support fair comparisons, while using method-specific control-variate handling for IPG. Additional Humanoid results show stable improvement but are truncated because off-policy critic fitting is computationally expensive.
- Implementation: GAE uses λ = 0.97, TRPO supplies the trust-region optimizer, and the critic uses DDPG-style fitting with replay and a target network.The critic is trained with batch size 64, replay size 1e6, target-network τ = 0.001, and ADAM.
- Hyperparameters: Trust-region step sizes and critic learning rates are fixed by domain after preliminary TRPO and DDPG runs, then held constant for fair comparisons.The trust-region step size is 0.1 for HalfCheetah-v1 and Humanoid-v1, and 0.01 for Ant-v1 and Walker2d-v1.
- Humanoid-v1 experiment: Figure 2 compares IPG-ν = 0.2-π-CV, Q-Prop, and TRPO on Humanoid-v1 over the first 10000 episodes with batch size 10000.The plot displays mean returns as solid lines and all-run variability as background scatter.
- Humanoid-v1 experiment: IPG-ν = 0.2-π-CV out-performs Q-Prop while all compared methods show stable, monotonic policy improvement.The experiment ends at 10000 episodes because fitting the off-policy critic makes Q-Prop and IPG computationally expensive.
- IPG settings: All IPG algorithms use the Q-Prop first-order Taylor control variate, while ν = 1 uses a reparameterized control variate with Monte Carlo sample size m = 1.The Taylor control variate cannot directly train the policy variance term when ν = 1.