PaperlayerView exact revision on arXiv

Source-linked AI summary

Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines

Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M Bayen, Sham Kakade, Igor Mordatch, Pieter Abbeel

arXiv:1803.07246v1cs.LGcs.AIstat.ML

TL;DR

Policy-gradient estimators suffer from high variance, particularly with long horizons and high-dimensional actions. The paper derives bias-free action-dependent baselines that exploit policy structure, and reports faster learning across control, manipulation, synthetic, partially observed, and multi-agent settings.

  • Problem

    Policy-gradient methods need lower-variance gradient estimates because high variance is especially problematic for long-horizon and high-dimensional-action problems.

  • Method

    The paper derives bias-free action-dependent baselines that use factorized policy structure and additional action information without adding assumptions about the MDP.

  • Results

    Action-dependent baselines consistently outperform state-only baselines across standard control, hand manipulation, synthetic high-dimensional, POMDP, and multi-agent tasks, with task-specific learning speed-ups.

  • Takeaways & Limitations

    Using information beyond the local state can reduce variance in factorized policies and extend to partially observed and multi-agent settings.

  • Takeaways & Limitations

    The paper’s main analysis assumes a discrete-time MDP, while general policies without conditional independence require different bias-free baselines.

Abstract

from arXiv · show

Policy gradient methods have enjoyed great success in deep reinforcement learning but suffer from high variance of gradient estimates. The high variance problem is particularly exasperated in problems with long horizons or high-dimensional action spaces. To mitigate this issue, we derive a bias-free action-dependent baseline for variance reduction which fully exploits the structural form of the stochastic policy itself and does not make any additional assumptions about the MDP. We demonstrate and quantify the benefit of the action-dependent baseline through both theoretical analysis as well as numerical results, including an analysis of the suboptimality of the optimal state-dependent baseline. The result is a computationally efficient policy gradient algorithm, which scales to high-dimensional control problems, as demonstrated by a synthetic 2000-dimensional target matching task. Our experimental results indicate that action-dependent baselines allow for faster learning on standard reinforcement learning benchmarks and high-dimensional hand manipulation and synthetic tasks. Finally, we show that the general idea of including additional information in baselines for improved variance reduction can be extended to partially observed and multi-agent tasks.

1 INTRODUCTION

Policy gradient methods face high-variance gradient estimates, especially in long-horizon and high-dimensional-action problems. The paper proposes factorized action-dependent baselines that use other action factors to improve credit assignment and variance reduction.

  • Motivation: High variance in policy-gradient estimators is linked to difficult credit assignment and is worsened by long horizons.Baselines reduce variance by comparing action performance with average performance from the same state.
  • Core idea: Factorized policies allow separate baselines for each action factor while conditioning on all relevant quantities except that factor.Conditional independence given the current state makes these factor-specific baselines computable.
  • Core idea: Using other action factors helps evaluate a specific factor and removes their influence from its reward signal, improving credit assignment.The resulting action-dependent baselines are designed to reduce gradient-estimator variance without changing the policy-gradient target.
  • Applications: Action factors arise naturally from diagonal-covariance Gaussian policies, factorized categorical policies, and decentralized multi-agent actions.These settings cover continuous control, games, and centralized-training/decentralized-execution systems.
  • Evaluation: Experiments compare action-dependent baselines with state-only baselines across standard control, hand manipulation, synthetic high-dimensional, POMDP, and multi-agent tasks.The paper reports consistent improvement and task-specific learning speed-ups.

2 RELATED WORKS

Prior work reduces policy-gradient variance with baselines and uses additional information in several reinforcement-learning settings. The paper differs by exploiting factorization in the policy parameterization itself, a direction it identifies as unexplored.

  • Existing methods: Value-based, policy-based, and actor-critic methods form three major reinforcement-learning method classes, with critics often introducing bias outside compatibility conditions.The cited compatibility conditions are rarely satisfied in practice because of stochastic-gradient methods.
  • Variance reduction: Baselines provide bias-free variance reduction, but prior work had not fully exploited factorization in the policy probability distribution.The paper positions its action-dependent approach as extending this established baseline framework.
  • Additional information: Guided Policy Search and related methods use lower-dimensional training information even when policies act on high-dimensional observations.These approaches differ from using the policy’s own factorization structure.
  • Novelty: Multi-agent work uses additional information during centralized training to accelerate learning, whereas this paper exploits structure in the policy parameterization itself.The related-work discussion identifies this policy-structure use as unexplored.

3 PRELIMINARIES

The paper formulates policy gradients in discounted MDPs, introduces score-function estimation, and reviews how state-dependent baselines reduce variance without bias. The preliminaries also note that the analysis extends from states to observations in POMDPs.

  • MDP setting: The setting is a discrete-time MDP with state space S, action space A, transition function P, bounded rewards, initial distribution ρ0, and discount factor γ.The policy is stochastic and parameterized by θ, with the objective of maximizing expected discounted return.
  • MDP setting: A trajectory samples initial states from ρ0, actions from πθ, and successor states from P, while Q̂ denotes sampled discounted return.Q(at, st) denotes a function approximation of the sampled action return.
  • Partial observability: For POMDPs, observations replace fully observable states in the analysis, with observation set Ω and distribution O added to the model.The fully observable case is recovered when Ω≡S.
  • Score-function estimator: The score-function estimator provides the standard tool for differentiating expectations over θ-dependent distributions and underlies the policy-gradient derivation.Its assumptions include common support and continuity of log pθ(x) in θ.
  • Policy-gradient foundation: The Policy Gradient Theorem expresses the policy-gradient foundation used by the paper.The supplied preliminaries introduce the theorem before rewriting the gradient using state visitation and sampled returns.
  • Baselines: Subtracting a state-dependent quantity from Q̂ reduces gradient-estimator variance without introducing bias.The baseline is justified through the score-function identity and can be optimized for variance reduction.

4 ACTION-DEPENDENT BASELINES

The paper exploits conditional independence among policy factors to construct bias-free action-dependent baselines, allowing each factor's baseline to depend on the other actions. It derives optimal and practical baselines, analyzes their variance reduction, and provides an implementation for factorized policies.

  • 4.1 Baselines for Policies with Conditionally Independent Factors: Factorized policies permit each factor's baseline to depend on the state and all other actions without introducing bias.The resulting gradient estimator contains separate component terms with distinct advantage estimates.
  • 4.1 Baselines for Policies with Conditionally Independent Factors: The method also applies to general policy structures without conditional independence, using different baselines while remaining bias-free.The conditional-independence assumption is therefore specific to the main derivation rather than required by the broader methodology.
  • 4.2 Optimal Action-Dependent Baseline: The optimal action-dependent baseline minimizes policy-gradient-estimator variance and generally lies outside the family of state-dependent baselines.This follows because the optimal baseline differs across action coordinates except in pathological cases.
  • 4.3 Suboptimality of the Optimal State-Dependent Baseline: The variance difference from a state-only baseline is especially large when the Q function is sensitive to actions along directions that most influence the gradient.The paper characterizes this difference using score-weighted marginalized Q quantities.
  • 4.4 Marginalization of the Global Action-Value Function: Practical marginalized-Q baselines reduce computation by learning one Q-function and using it to obtain baselines for each action coordinate.Monte Carlo and mean-marginalized variants trade computational cost against baseline construction practicality.
  • Algorithm: The proposed algorithm updates policy parameters with component advantages while updating the action-value approximation from the current batch.Baseline computation can use either practical technique described in the section.

5 EXPERIMENTS AND RESULTS

Experiments compare action-dependent baselines with state-only baselines across continuous-control and synthetic high-dimensional tasks. The action-dependent approach performs consistently better, while mean-action and Q-function variants perform comparably with different computational trade-offs.

  • Action-dependent baselines perform consistently better than state-only baselines on MuJoCo locomotion and high-dimensional multi-fingered-hand tasks.
  • The experiments use Random Fourier Feature representations for baseline parameterization to enable a fair comparison.The representation uses fixed random features and trainable linear weights.
  • The mean-action and Q-function-sampling variants perform comparably, with the mean-action variant being more computationally efficient.The mean-action variant performs slightly better toward the end of learning.

2. The optimal action is thus to match

The experiments evaluate additional-information baselines in high-dimensional target matching and partially observable or multi-agent tasks. These baselines improve convergence or learning speed while retaining computational scalability.

  • Q-function sampling and mean-action approximation are comparable, with the mean-action approach being more computationally efficient.
  • Action-dependent baselines improve convergence more as target-matching action dimensionality increases.They assess each action dimension using information from the other action dimensions and scale computationally to high-dimensional problems.
  • Around 10% faster convergence occurs at high dimensions with the linear feature action-dependent baseline compared with a linear feature baseline.The target-matching results use five seeds and action spaces from 12 to 2000 dimensions.
  • Additional baseline information enables faster learning in both blind peg insertion and a two-agent communication task.The blind peg-insertion baseline receives the hole location, while the multi-agent baseline incorporates information from the other agent.

6 CONCLUSION

The paper presents action-dependent baselines as a bias-free way to use information beyond the state for variance reduction. It supports the approach with variance analysis, practical baselines, and experiments across continuous-control, high-dimensional, partially observed, and multi-agent settings.

  • Action-dependent baselines use additional signals beyond the state to achieve bias-free variance reduction.
  • The analysis covers variance reduction relative to non-optimal baselines, including the traditional optimal state-dependent baseline.
  • Practical action-dependent baselines perform well across continuous-control tasks and synthetic high-dimensional action problems.
  • Using additional signals beyond the local state extends the approach to partially observed and multi-agent settings.

A DERIVATION OF THE OPTIMAL STATE-DEPENDENT BASELINE

The paper derives variance-minimizing state- and action-dependent baselines for factorized policies, then extends the analysis and algorithm to general action factorizations. The resulting baselines can use all factors that do not influence a given factor while preserving the paper’s variance-reduction objective.

  • The derivation minimizes the trace of the policy-gradient covariance, defined as the sum of component variances.
  • The optimal action-dependent baseline is obtained by writing the variance under an arbitrary baseline and minimizing it component by component.
  • Under the stated parameter-separation assumption, the overall variance is minimized when each component variance is minimized.
  • For a general factorization, factor i’s baseline may depend on the state and all factors that factor i does not influence.
  • The analysis and variance-suboptimality results transfer to general actions, where m individual baselines are fitted from data collected in the previous iteration.

F COMPATIBILITY WITH GAE

The paper combines action-dependent baselines with GAE by using them to form advantage estimates. This retains unbiasedness for the advantage estimate before GAE’s usual bias–variance trade-off is applied.

  • Action-dependent baselines provide an unbiased estimator for the advantage function when used in the temporal-difference error.
  • GAE can then further reduce variance using action-dependent baselines, at the cost of some bias.
  • Figure 4 reports that a slightly biased, lower-variance gradient performs best overall, while high-bias estimates perform poorly.

G HIGH-DIMENSIONAL ACTION SPACES: TRAINING CURVES

In a synthetic target-matching task, action-dependent baselines improve learning in high-dimensional action spaces. For dimensions of 100 or more, they reach the optimum faster than state-only baselines.

  • The action-dependent baseline consistently outperforms the state-only baseline in the high-dimensional training curves.

H EXPERIMENT DETAILS

The experiments use fixed discounting, GAE, and KL targets with two-layer policies, while baseline and task configurations vary across experiments. The synthetic target-matching study uses linear-feature baselines and spans action spaces up to 2000 dimensions.

  • Parameters: The default experiment parameters are γ = 0.995, λ_GAE = 0.97, and kldesired = 0.025.
  • Policies: The policies are two-layer fully connected networks with hidden sizes=(32, 32).
  • Per-experiment configuration: The synthetic m-DimTargetMatching experiments use a linear feature baseline, with task-specific action dimensionalities listed separately.
  • Training curves: The synthetic target-matching learning curves cover 12 to 2000 dimensional action spaces and use 5 seeds.
Loading 1803.07246v1…