PaperlayerView exact revision on arXiv

Source-linked AI summary

Maximum a Posteriori Policy Optimisation

Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, Martin Riedmiller

arXiv:1806.06920v1cs.LGcs.AIcs.ITcs.ROstat.ML

TL;DR

Continuous-control reinforcement learning can be sample-intensive, unstable, and difficult to tune. The paper introduces MPO, an off-policy inference-based algorithm with alternating E- and M-steps. Across continuous-control tasks, MPO is reported as highly data efficient, robust to hyperparameter choices, and applicable to complex control problems.

  • Problem

    Continuous-control reinforcement learning can require many samples, exhibit high gradient variance, and be difficult to tune.

  • Method

    MPO casts reinforcement learning as inference and alternates off-policy E-step reweighting of state-action samples with supervised M-step policy updates.

  • Results

    MPO is reported as highly data efficient, robust to hyperparameter choices, and effective across a large set of continuous-control problems.

  • Takeaways & Limitations

    MPO combines off-policy data efficiency with robustness and applicability to complex continuous-control tasks.

  • Takeaways & Limitations

    The trajectory-level KL decomposition depends on assuming that q factors like the policy-induced trajectory distribution.

Abstract

from arXiv · show

We introduce a new algorithm for reinforcement learning called Maximum aposteriori Policy Optimisation (MPO) based on coordinate ascent on a relative entropy objective. We show that several existing methods can directly be related to our derivation. We develop two off-policy algorithms and demonstrate that they are competitive with the state-of-the-art in deep reinforcement learning. In particular, for continuous control, our method outperforms existing methods with respect to sample efficiency, premature convergence and robustness to hyperparameter settings while achieving similar or better final performance.

1 INTRODUCTION

Deep reinforcement learning can acquire sophisticated behaviours, but high sample demands and gradient variance can make learning unstable or slow. MPO is proposed as an off-policy method combining on-policy robustness with off-policy data efficiency, and it achieves rapid continuous-control learning.

  • High sample requirements and gradient variance, especially in continuous action spaces, can cause unstable learning or slow convergence.
  • On-policy methods such as TRPO and PPO offer robustness and moderate tuning demands but use large batches and restrict parameter changes.
  • Off-policy value-gradient methods use replay and learned value functions to achieve much better data efficiency, but can be difficult to tune.
  • MPO is an off-policy algorithm designed to combine on-policy scalability, robustness, and hyperparameter insensitivity with off-policy data efficiency.
  • MPO uses an inference-based EM perspective, alternating a data-based E-step that reweights state-action samples with a supervised neural-network M-step.
  • Across continuous-control problems including a 56 DoF humanoid, MPO used the same optimisation hyperparameters and often solved tasks an order of magnitude faster than the state-of-the-art.

2 BACKGROUND AND NOTATION

The paper situates MPO within reinforcement learning as inference, entropy-based methods, trust-region policy optimisation, and soft Q-learning. Its setting is a discounted continuous-state, continuous-action MDP with trajectories induced by a policy.

  • RL-as-inference methods estimate posterior trajectories consistent with desired outcomes, while maximum-entropy methods encourage robustness and simplicity by penalising determinism.
  • Unlike model-based inference methods, the paper avoids assuming a transition model and avoids on-policy optimisation.
  • Soft Q-learning methods estimate a soft Q-function directly, implicitly inducing a maximum-entropy policy, whereas MPO relies on a parametric policy.
  • Entropy regularisation and KL constraints are used by many policy-gradient methods to stabilise reinforcement-learning objectives.
  • Trust-region methods bound expected KL divergence between new and old policies and can be viewed here as parametric E-steps without an explicit M-step.
  • The formal problem is finding an optimal policy for a discounted MDP with continuous states and actions, transition probabilities, rewards, and discount factor γ.

3 MAXIMUM A POSTERIORI POLICY OPTIMISATION

MPO casts policy search as inference and optimises a KL-regularised lower bound through alternating E- and M-steps. The design uses off-policy samples, trust-region constraints, and flexible variational policies.

  • MAXIMUM A POSTERIORI POLICY OPTIMISATION: RL inference formulations condition a prior trajectory distribution on desired outcomes, with rewards represented through an optimality likelihood.
  • MAXIMUM A POSTERIORI POLICY OPTIMISATION: MPO alternates improving an auxiliary trajectory distribution q in the E-step with updating the parametric policy in the M-step.
  • MAXIMUM A POSTERIORI POLICY OPTIMISATION: The E-step uses off-policy deep RL and value-function approximation to reweight state-action samples, while the M-step fits the policy supervisedly to those targets.
  • MAXIMUM A POSTERIORI POLICY OPTIMISATION: The algorithm targets low-variance returns, low-sample-complexity value estimates, minimal E-step distribution assumptions, supervised policy updates, and hard trust-region constraints.
  • POLICY IMPROVEMENT: The trajectory KL decomposes into state-conditional action KLs because q is assumed to factor like the policy-induced trajectory distribution.
  • POLICY IMPROVEMENT: Optimising q treats π as the default policy, while coordinate ascent alternates partial E-step optimisation of q with M-step optimisation of π.
  • E-STEP: Replacing soft KL regularisation with a hard constraint avoids choosing α when reward and KL terms have arbitrary relative scales.
  • E-STEP: The non-parametric E-step evaluates q from off-policy Q estimates, increasing data efficiency, and the M-step can fit a parametric policy with regularisation.

4 POLICY EVALUATION

MPO evaluates policies off-policy by learning a parametric Q-function with the Retrace policy-evaluation operator. The implementation uses a neural network, target network, truncated returns, and replay-buffer behaviour probabilities.

  • POLICY EVALUATION: MPO relies on a stable policy-evaluation operator to obtain a parametric Q-function in the off-policy setting.
  • POLICY EVALUATION: The Q-function is represented by a neural network and fitted by minimising a squared loss.
  • POLICY EVALUATION: Retrace targets use a target Q-network copied from current parameters after each M-step, truncate the infinite sum after N steps, and use replay-buffer action probabilities as behaviour probabilities.

5 EXPERIMENTS

MPO is evaluated across broad continuous-control benchmarks, including Control Suite tasks, parkour environments, and high-dimensional humanoid and walker problems. Across these settings, it provides stable learning and strong sample efficiency, including an order-of-magnitude advantage over PPO in Walker-2D parkour.

  • 5.1 EVALUATION ON CONTROL SUITE: MPO is evaluated on Control Suite’s 18 continuous-control tasks, spanning classical systems, locomotion, reaching, and manipulation.The experiments also include parkour environments and initial Atari experiments using a categorical policy.
  • 5.1.1 DETAILED ANALYSIS ON WALKER-2D, ACROBOT, HOPPER: MPO gives stable learning on all three detailed Control Suite tasks and is significantly more sample efficient than the on-policy PPO baseline.The detailed comparison covers Acrobot, 2D Walker, and Hopper standing, with PPO, DDPG, and MPO ablations.
  • 5.1.2 COMPLETE RESULTS ON THE CONTROL SUITE: Q-critic methods show a clear data-efficiency advantage over PPO, while Retrace methods outperform DDPG in data efficiency across the tested environments.Entropy-regularized EPG exhibits instabilities, whereas MPO remains stable across environments.
  • 5.2 HIGH-DIMENSIONAL CONTINUOUS CONTROL: MPO quickly finds a stable running policy on Humanoid and outperforms other algorithms in sample efficiency for this high-dimensional control problem.The high-dimensional experiments use a parallel variant with multiple learners collecting trajectories.
  • 5.2 HIGH-DIMENSIONAL CONTINUOUS CONTROL: 70k trajectories, or 60M samples, let MPO find a Walker-2D parkour solution asymptotically no worse than PPO’s, versus approximately 1M trajectories for PPO.This corresponds to an order-of-magnitude improvement; a 22-degree-of-freedom humanoid parkour policy was also learned successfully.

6 CONCLUSION

The paper presents Maximum a-posteriori Policy Optimisation (MPO), a new off-policy reinforcement-learning algorithm. It reports that MPO is data efficient, robust to hyperparameter choices, and applicable to complex continuous-control problems.

  • 6 CONCLUSION: MPO is a new off-policy reinforcement-learning algorithm motivated by the connection between reinforcement learning and inference.Its alternating optimisation scheme has direct relations to several existing algorithms.
  • 6 CONCLUSION: MPO is highly data efficient, robust to hyperparameter choices, and applicable to complex control problems across a large set of continuous-control tasks.

A PROOF OF MONOTONIC IMPROVEMENT FOR THE KL-REGULARIZED

This section establishes a monotonic-improvement guarantee for KL-regularized policy optimization under alternating updates of π and q. It defines the regularized Bellman operators and value functions used in that guarantee.

  • A PROOF OF MONOTONIC IMPROVEMENT FOR THE KL-REGULARIZED: The proof concerns KL-regularized policy optimization with alternating updates on π and q, assuming an uninformative prior on θ.
  • A PROOF OF MONOTONIC IMPROVEMENT FOR THE KL-REGULARIZED: The π-regularized reward for q is defined for policies whose action support contains that of π at every state.
  • A PROOF OF MONOTONIC IMPROVEMENT FOR THE KL-REGULARIZED: The proof uses α > 0 and notes that the relevant inequality holds for every action, not only in expectation under q.
  • A PROOF OF MONOTONIC IMPROVEMENT FOR THE KL-REGULARIZED: The section defines π-regularized and non-regularized Bellman operators for policy q.
  • A PROOF OF MONOTONIC IMPROVEMENT FOR THE KL-REGULARIZED: It defines corresponding π-regularized and non-regularized value functions, together with their optimal value functions and policies.

A.2 REGULARIZED JOINT POLICY GRADIENT

The regularized joint policy-gradient analysis constructs policy and auxiliary-distribution sequences whose objective values do not decrease. The improvement is linked to the gradient norm, yielding convergence to a local maximum under the stated conditions.

  • A.2 REGULARIZED JOINT POLICY GRADIENT: The analysis starts from an initial parameter θ0 and generates a sequence of policies πi = πθi.The policy is parametrized by θi throughout the construction.
  • A.2 REGULARIZED JOINT POLICY GRADIENT: The auxiliary policy qi satisfies the proposition’s stated policy property, with gi denoting the gradient norm minimized by the algorithm.The supplied passage defines gi as the norm of the gradient.
  • A.2 REGULARIZED JOINT POLICY GRADIENT: The values J(θi, qi) are non-decreasing and converge to a local maximum, while improvement is lower-bounded by a constant times the gradient norm.Improvement continues until the gradient vanishes, described as reaching the representation’s capacity limit.
  • A.2 REGULARIZED JOINT POLICY GRADIENT: The derivation uses a monotone operator T πi,qi α whose fixed point is V πi,qi α.This monotonicity supports the subsequent improvement argument.
  • A.2 REGULARIZED JOINT POLICY GRADIENT: The update rule is θi+1 = θi −β∇θf(πi, qi, θi), with the stated guarantee applying for sufficiently small β.The passage explicitly gives the gradient update and its step-size condition.

B ADDITIONAL EXPERIMENT: DISCRETE CONTROL

The paper evaluates MPO on a subset of Arcade Learning Environment games as a robustness proof of concept. It reuses continuous-control hyperparameters and learning rate while changing the network architecture to a DQN-style structure.

  • B ADDITIONAL EXPERIMENT: DISCRETE CONTROL: The experiment tests MPO on a subset of games from the Arcade Learning Environment.The study is presented as a proof of concept for algorithmic and hyperparameter robustness.
  • B ADDITIONAL EXPERIMENT: DISCRETE CONTROL: The KL-constraint hyperparameters and learning rate match those used in the continuous-control experiments.Only the network architecture was changed for this experiment.
  • B ADDITIONAL EXPERIMENT: DISCRETE CONTROL: The experiment uses the standard DQN network architecture and a separate network with the same architecture for predicting policy-distribution parameters.This preserves the architecture structure while assigning the networks different prediction roles.

C EXPERIMENT DETAILS

The experiment-details section specifies hyperparameters for MPO’s non-parametric and parametric variational-distribution variants. Continuous-control experiments primarily use feed-forward networks, with a separate architecture for Parkour-2d.

  • C EXPERIMENT DETAILS: Continuous-control experiments use a feed-forward network except for Parkour-2d, which uses the architecture from Heess et al. (2017).The passage identifies this as an experimental network-architecture distinction.
  • C EXPERIMENT DETAILS: Table 2 reports parameters for the non-parametric variational distribution.The supplied table caption identifies the table’s scope.
  • C EXPERIMENT DETAILS: Table 3 reports parameters for the parametric variational distribution.The supplied table caption identifies the table’s scope.

D DERIVATION OF UPDATE RULES FOR A GAUSSIAN POLICY

The Gaussian-policy derivation explains MPO’s E-step and M-step constructions, including KL-constrained optimization and iterative updates of policy parameters and Lagrange multipliers. It also motivates decoupling mean and covariance constraints empirically.

  • D DERIVATION OF UPDATE RULES FOR A GAUSSIAN POLICY: For continuous control, the policy is Gaussian with full covariance, parameterized by a neural-network mean and Cholesky factor.The covariance is Σ = AAT, with positive diagonal entries enforced through a softplus transform.
  • D DERIVATION OF UPDATE RULES FOR A GAUSSIAN POLICY: The non-parametric E-step derives a variational distribution by replacing expectations with integrals and maximizing a Lagrangian with respect to q.The derivative includes the action-value term and KL-related terms involving q and the policy.
  • D DERIVATION OF UPDATE RULES FOR A GAUSSIAN POLICY: The M-step incorporates a Gaussian prior around the current policy to obtain a KL constraint and an optimization program corresponding to the main-text update.The derivation drops constant terms before matching the resulting program to Equation (11).
  • D DERIVATION OF UPDATE RULES FOR A GAUSSIAN POLICY: Empirically, decoupling the KL constraint into separate mean and covariance terms produced better results.Separate εµ and εΣ values permit different learning rates, with a smaller covariance constraint used to preserve exploration.
  • D DERIVATION OF UPDATE RULES FOR A GAUSSIAN POLICY: Policy parameters and Lagrange multipliers are optimized alternately until both converge.The same inner- and outer-optimization procedure is described for the policy and variational-distribution updates.

E IMPLEMENTATION DETAILS

MPO supports distributed synchronous training through workers that collect data and compute gradients, while a chief averages gradients and updates shared parameters. The paper specifies separate worker procedures for non-parametric and parametric variational distributions.

  • Distributed training: Workers independently collect environment data, compute gradients, and send them to a chief that performs parameter updates by averaging gradients.The scalable variant uses distributed synchronous gradient descent.
  • Distributed training: The chief stores incoming gradients for φ, η, ηµ, ηΣ, and θ, averages them after receiving G gradients, updates parameters, and sends the new parameters to workers.This describes the chief-side synchronization loop.
  • Non-parametric MPO: Non-parametric workers initialize the critic, policy, and variational parameters, update a replay buffer, sample replay minibatches and additional actions, and estimate gradients for policy improvement.The worker procedure repeats gradient updates while interacting with the environment and replay buffer.
  • Parametric MPO: The parametric worker procedure uses the same replay-based gradient workflow, but policy and variational distribution share the same structure, so no separate M-step is performed.The algorithm still computes E-step-related gradients and policy gradients before synchronization.
Loading 1806.06920v1…