PaperlayerView exact revision on arXiv

Source-linked AI summary

Phasic Policy Gradient

Karl Cobbe, Jacob Hilton, Oleg Klimov, John Schulman

arXiv:2009.04416v1cs.LGstat.ML

TL;DR

PPG addresses the tension between sharing representations and avoiding interference between policy and value-function optimization. It alternates policy training with an auxiliary phase that distills value-function features, allowing different sample-reuse levels. Compared with PPO, PPG significantly improves sample efficiency on the Procgen Benchmark.

  • Problem

    Shared policy-value networks risk objective interference and force both objectives to use the same data and sample-reuse level.

  • Method

    PPG alternates PPO-based policy phases with auxiliary phases that distill value-function features into the policy network.

  • Results

    PPG achieves significantly better sample efficiency than PPO in nearly every Procgen environment.

  • Takeaways & Limitations

    PPG preserves shared representations while decoupling policy and value training, enabling more aggressive value-function optimization without significantly interfering with policy training.

Abstract

from arXiv · show

We introduce Phasic Policy Gradient (PPG), a reinforcement learning framework which modifies traditional on-policy actor-critic methods by separating policy and value function training into distinct phases. In prior methods, one must choose between using a shared network or separate networks to represent the policy and value function. Using separate networks avoids interference between objectives, while using a shared network allows useful features to be shared. PPG is able to achieve the best of both worlds by splitting optimization into two phases, one that advances training and one that distills features. PPG also enables the value function to be more aggressively optimized with a higher level of sample reuse. Compared to PPO, we find that PPG significantly improves sample efficiency on the challenging Procgen Benchmark.

1 Introduction

Actor-critic methods must balance policy and value-function optimization while deciding whether to share network parameters. PPG decouples their training into alternating phases, preserving shared representations while reducing interference and enabling different sample-reuse levels.

  • Shared networks let policy and value objectives reuse features, but their competing optimization objectives can interfere with each other.
  • Shared architectures also force policy and value objectives to use the same data and sample-reuse level.
  • PPG preserves feature sharing while decoupling policy and value-function training through alternating optimization phases.
  • Interference between policy and value optimization can negatively affect performance when their parameters are shared.
  • Value-function optimization often tolerates significantly more sample reuse than policy optimization.
  • By reducing interference and using objective-specific sample reuse, PPG significantly improves sample efficiency.

2 Algorithm

PPG separates policy and value optimization into alternating phases while retaining feature sharing through an auxiliary value head. This decoupling also allows different sample-reuse levels for policy and value training.

  • Phasic training: PPG alternates a PPO-based policy phase with an auxiliary phase that distills value-function features into the policy network.The auxiliary phase is PPG’s distinguishing addition relative to PPO.
  • Policy phase: During the policy phase, disjoint policy and value networks reduce interference while optimizing PPO’s clipped policy objective and the value-function objective.The policy also includes an auxiliary value head for representation training.
  • Auxiliary phase: During the auxiliary phase, the policy network optimizes a joint auxiliary and behavioral-cloning objective that limits distortion from the policy active before the phase.The behavioral-cloning coefficient controls the trade-off between auxiliary optimization and preserving the original policy.
  • Auxiliary phase: PPG uses the value-function loss as its auxiliary objective, sharing representations through an auxiliary value head while minimizing policy distortion.The framework can in principle use any auxiliary objective, although this work uses value-function error.
  • Sample reuse: The auxiliary phase reuses fixed policy-phase value targets and performs additional value-network optimization because the value and joint objectives have no shared parameter dependencies.PPG therefore separates optimization while allowing extra value-function sample reuse.
  • Sample reuse: PPG exposes separate controls for policy, value-function, and auxiliary sample reuse rather than requiring policy and value objectives to use identical reuse levels.The relevant controls are Eπ, EV, and Eaux.

3 Experiments

Experiments show that PPG improves sample efficiency over a highly tuned PPO baseline on Procgen while separating policy and value optimization. Performance is strongest with limited policy reuse, more auxiliary reuse, and relatively infrequent auxiliary phases; a single-network variant nearly matches dual-network PPG.

  • Comparison to PPO: PPG achieves significantly better sample efficiency than a highly tuned PPO implementation in nearly every Procgen environment.The Procgen Benchmark is designed to be highly diverse, and the comparison uses PPO with near-optimal sample reuse and loss weighting.
  • Comparison to PPO: Representation sharing is critical in Procgen but often unnecessary in lower-dimensional environments, suggesting PPG is most advantageous for high-dimensional or vision-based observations.The authors conjecture that Procgen’s high-dimensional inputs contribute to the importance of shared representations.
  • Policy Sample Reuse: A single policy epoch is almost always optimal or near-optimal in PPG when hyperparameters are well tuned.The authors attribute PPO’s benefit from greater reuse partly to the additional value-function training that its coupled updates provide.
  • Value Sample Reuse: Additional auxiliary epochs generally improve performance, with gains tapering around 6 epochs.Auxiliary optimization may improve shared features and value-function accuracy, with feature sharing appearing more important in Procgen.
  • Auxiliary Phase Frequency: Performance suffers when auxiliary phases occur too frequently, making relatively infrequent auxiliary phases critical to success.The authors conjecture that each auxiliary phase interferes with policy optimization and that frequent phases exacerbate this effect.
  • Objective and Architecture Variants: A fixed KL penalty performs remarkably similarly to clipping, while a single shared network performs almost as well as PPG’s dual-network architecture.The shared-network variant detaches value-function gradients during the policy phase while retaining full-gradient value training during the auxiliary phase.

4 Related Work

Related work situates PPG among methods addressing representation learning, non-stationarity, replay, policy objectives, and trust-region updates. PPG alternates RL and distillation phases but uses distillation to transfer value-function information into the policy network.

  • Distillation and Auxiliary Objectives: ITER and PPG both alternate a standard RL phase with a distillation phase, but ITER targets generalization whereas PPG transfers value-function information into the policy network.ITER distills policy and value teachers into newly initialized student networks; PPG periodically distills the value-function network into the policy network.
  • Distillation and Auxiliary Objectives: Prior work studies value functions as auxiliary objectives, including Adversarial Value Functions that provide useful auxiliary signals in the four-room domain.The supplied passage also connects distributional RL’s benefits to the richness of its value-related signal.
  • Off-Policy Methods: PPG uses replay-buffer updates during its auxiliary phase but does not improve the policy from off-policy data, unlike replay-based off-policy algorithms such as SAC, DDPG, and ACER.The distinction concerns how replay is used rather than whether a replay buffer exists.
  • Policy Objectives and Trust Regions: PPG is compatible in principle with alternative actor-critic policy objectives, including AWR, V-MPO, TRPO, and ACKTR.TRPO and ACKTR provide trust-region-style alternatives that could be incorporated into the PPG framework.

5 Conclusion

PPG decouples policy and value-function training to accommodate their different sample-reuse needs while retaining shared representations. It improves sample efficiency on Procgen, with hyperparameters generalizing across all 16 environments despite added complexity.

  • 5 Conclusion: PPG enables different sample-reuse levels for policy and value-function optimization by decoupling their training.Conventional shared-network actor-critic methods cannot vary sample reuse between these objectives without interference.
  • 5 Conclusion: Chosen PPG hyperparameter values generalize across all 16 Procgen environments despite introducing additional complexity.
  • 5 Conclusion: PPG mitigates interference between policy and value-function optimization while maintaining the benefits of shared representations.
  • 5 Conclusion: PPG significantly improves sample efficiency on the challenging Procgen Benchmark.

A Hyperparameters

All experiments use the Adam optimizer.

  • A Hyperparameters: All experiments use the Adam optimizer.
  • A Hyperparameters: The optimizer choice is consistent across all experiments: Adam.
  • A Hyperparameters: Adam is the optimization method specified for the experiments.

A.1 PPG-Specific Hyperparameters

The appendix lists the PPG-specific hyperparameters used for policy, value, and auxiliary training.

  • A.1 PPG-Specific Hyperparameters: PPG uses Nπ = 32, Eπ = 1, and EV = 1 for its policy and value training settings.
  • A.1 PPG-Specific Hyperparameters: The auxiliary-training settings are Eaux = 6 and βclone = 1.
  • A.1 PPG-Specific Hyperparameters: Each auxiliary epoch uses 16 minibatches per auxiliary epoch per Nπ.

A.2 Other Hyperparameters

The appendix specifies rollout, optimization, environment, and training settings for the experiments, including 100M total timesteps and no LSTM or frame stacking.

  • A.2 Other Hyperparameters: Each rollout contains 256 timesteps, with 8 minibatches per epoch.
  • A.2 Other Hyperparameters: The experiments use γ = .999, λ = .95, entropy coefficient .01, PPO clip range .2, reward normalization, and learning rate 5 × 10−4.
  • A.2 Other Hyperparameters: The setup uses 4 workers and 64 environments per worker for 100M total timesteps.
  • A.2 Other Hyperparameters: The experiments use neither LSTM nor frame stacking.

B Shared vs Separate Networks

Figure 8 compares PPO implementations that either share features between policy and value networks or train them separately.

  • The baseline shares features between the policy and value networks.
  • The ablation trains separate policy and value networks.

C Auxiliary Phase Value Function Training

PPG can omit value-function optimization during the auxiliary phase because extra value optimization can instead occur during the policy phase.

  • Skipping Lvalue optimization during the auxiliary phase is viable when additional Lvalue optimization occurs during the policy phase.
  • Ljoint remains crucial during the auxiliary phase because it provides the mechanism for sharing features between the value function and policy.

D PPO Sample Reuse

The PPO experiment sweeps sample reuse from 1 to 6 and finds that reuse of 3 is optimal under the stated hyperparameter settings.

  • The sweep compares different levels of sample reuse in PPO.
  • Sample reuse of 3 is optimal for PPO under the experiment’s other hyperparameter settings.
Loading 2009.04416v1…