PaperlayerView exact revision on arXiv

Source-linked AI summary

Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

Xue Bin Peng, Aviral Kumar, Grace Zhang, Sergey Levine

arXiv:1910.00177v3cs.LGstat.ML

TL;DR

AWR addresses the complexity, instability, sample inefficiency, and limited off-policy reuse of common reinforcement learning methods with a simple supervised-learning-based algorithm. It uses value and policy regression with experience replay, achieving competitive benchmark performance and effective learning from fully off-policy data, while remaining less sample efficient than the most efficient off-policy algorithms.

  • Problem

    Reinforcement learning methods can be complex, unstable, sample inefficient, and typically limited in their ability to reuse off-policy data.

  • Method

    AWR performs value-function and policy updates through supervised regression, using advantage weighting and experience replay to incorporate off-policy data.

  • Results

    AWR achieves competitive performance on standard benchmarks, solves challenging control tasks, and learns effectively from fully off-policy datasets.

  • Takeaways & Limitations

    AWR provides a simple, general approach that can use standard supervised learning methods for continuous or discrete actions and static off-policy data.

  • Takeaways & Limitations

    The comparison with MPO is incomplete because the authors could not reproduce comparable MPO results or achieve similar reported performance.

Abstract

from arXiv · show

In this paper, we aim to develop a simple and scalable reinforcement learning algorithm that uses standard supervised learning methods as subroutines. Our goal is an algorithm that utilizes only simple and convergent maximum likelihood loss functions, while also being able to leverage off-policy data. Our proposed approach, which we refer to as advantage-weighted regression (AWR), consists of two standard supervised learning steps: one to regress onto target values for a value function, and another to regress onto weighted target actions for the policy. The method is simple and general, can accommodate continuous and discrete actions, and can be implemented in just a few lines of code on top of standard supervised learning methods. We provide a theoretical motivation for AWR and analyze its properties when incorporating off-policy data from experience replay. We evaluate AWR on a suite of standard OpenAI Gym benchmark tasks, and show that it achieves competitive performance compared to a number of well-established state-of-the-art RL algorithms. AWR is also able to acquire more effective policies than most off-policy algorithms when learning from purely static datasets with no additional environmental interactions. Furthermore, we demonstrate our algorithm on challenging continuous control tasks with highly complex simulated characters.

1 INTRODUCTION

The paper targets reinforcement learning that is simpler to implement and tune while using off-policy experience. It introduces AWR, which combines supervised regression with replay-based learning and achieves competitive results across established benchmarks and complex simulated-character tasks.

  • 1 INTRODUCTION: Reinforcement learning can acquire sophisticated behaviors, but policy-gradient methods are often unstable, nearly on-policy, and sample-intensive.These limitations motivate an approach that is easier to implement and can reuse off-policy experience.
  • 1 INTRODUCTION: AWR is proposed as a simple off-policy model-free reinforcement learning algorithm built from standard supervised learning methods.Its update uses regression for a value-function baseline and weighted regression for the policy.
  • 1 INTRODUCTION: AWR accommodates continuous and discrete actions and can be implemented in a few lines on top of standard supervised learning methods.The paper also presents a constrained-policy-optimization interpretation and analyzes experience replay theoretically.
  • 1 INTRODUCTION: AWR achieves competitive performance against established on-policy and off-policy algorithms on standard benchmarks and challenging complex-character control tasks.The introduction reports demonstrations involving complex simulated characters, while the figure illustrates a humanoid spinkick and dog canter.

2 PRELIMINARIES

The preliminaries define reinforcement learning through expected discounted return and introduce reward-weighted regression as a supervised policy-search formulation. RWR fits a new policy to on-policy actions weighted by exponentiated returns.

  • 2 PRELIMINARIES: Reinforcement learning seeks a policy that maximizes expected discounted return while mapping observed states to sampled actions.The setup uses states s_t, actions a_t, rewards r_t, and discount factor γ.
  • 2 PRELIMINARIES: The discounted state distribution d^π(s) measures the unnormalized discounted likelihood of visiting state s under policy π.It sums discounted state-occupancy probabilities across timesteps.
  • 2 PRELIMINARIES: AWR builds on reward-weighted regression, an expectation-maximization-based policy-search method that solves a supervised regression problem each iteration.RWR is presented as a predecessor that uses supervised learning for policy updates.
  • 2 PRELIMINARIES: RWR fits a new policy to samples from the current policy by weighting action likelihoods with exponentiated returns and temperature β.The formulation is also interpretable through the dual of a constrained policy-search problem.

3 ADVANTAGE-WEIGHTED REGRESSION

AWR derives an off-policy policy-learning procedure from constrained policy optimization and implements each iteration with supervised regression over replay-buffer data. Its key design choices are advantage-based weighting, a value baseline, and experience replay representing a mixture of past policies.

  • Algorithm overview: Each AWR iteration adds trajectories from the current policy to a FIFO replay buffer, fits a value function to returns, and fits the policy with advantage-weighted regression.The value and policy updates use data stored in the replay buffer.
  • Algorithm overview: AWR weights replay-buffer state-action pairs by exponentiated advantage, with β controlling the weighting temperature.The advantage is formed from return minus the value-function baseline.
  • Derivation: The derivation maximizes expected improvement over a sampling policy and approximates it using that policy’s state distribution.This surrogate objective supports a constrained policy-search formulation whose constraint keeps the new policy close to the data distribution.
  • Derivation: The expected-improvement objective introduces a value baseline into the policy update, unlike related objectives that maximize expected return directly.The paper reports that this baseline is crucial for effective performance and that the resulting weighting difference yields a large empirical improvement.
  • Experience replay and off-policy learning: Experience replay improves sample efficiency by modeling the sampling policy as a mixture of policies from previous iterations rather than only the latest policy.The replay-buffer derivation uses a mixture state-action distribution and an average of the constituent policies’ value functions as the baseline.
  • Practical implementation: For practical implementation, AWR can sample states uniformly from the replay buffer and use TD(λ) instead of Monte Carlo returns to reduce return-estimation variance.Uniform replay-buffer sampling was reported effective, while TD(λ) provides a lower-variance return estimate through bootstrapping.

4 RELATED WORK

Related work separates reinforcement-learning methods by how they use data and highlights the trade-offs of existing off-policy approaches. AWR is most closely related to REPS and MPO but differs in its value learning and replay-buffer treatment.

  • On-policy and off-policy learning: On-policy methods update from data collected by the same policy, whereas off-policy methods improve sample efficiency by reusing data from other sources.Off-policy data can come from different agents or previous algorithm iterations.
  • Off-policy methods: Importance sampling enables off-policy reuse but can introduce optimization instability through high estimator variance.The passage contrasts this with dynamic-programming methods, which can also use off-policy data but are often unstable without stabilization techniques.
  • Off-policy methods: Fully off-policy learning remains difficult for methods that cannot collect additional environmental interactions.This setting is distinct from ordinary reuse of data collected during ongoing interaction.
  • EM-based policy search: RWR casts on-policy policy search as supervised regression within an expectation-maximization framework.AWR builds on this reward-weighted-regression lineage while changing the weighting objective.
  • Closest prior work: REPS and MPO use constrained policy search with supervised-regression policy updates, while AWR optimizes expected improvement and learns its value function by simple supervised regression.REPS instead optimizes expected return and uses a more complex dual-function value-learning procedure.
  • Closest prior work: Unlike the described REPS policy-iteration procedure, AWR incorporates experience replay containing data from previous policies.This distinction directly supports AWR’s off-policy training design.

5 EXPERIMENTS

Experiments evaluate AWR on standard Gym benchmarks, high-dimensional motion imitation, and fully off-policy static datasets. Across these settings, AWR is generally competitive, while ablations show that replay and baseline components materially affect performance and stability.

  • 5 EXPERIMENTS: AWR is evaluated against on-policy and off-policy deep RL algorithms on discrete and continuous OpenAI Gym control tasks.The comparison includes TRPO, PPO, DDPG, TD3, SAC, and RWR.
  • 5.1 BENCHMARKS: AWR significantly outperforms PPO and TRPO in sample efficiency and asymptotic performance, while generally matching their final performance against stronger off-policy methods.AWR is less sample efficient than SAC and TD3, but usually achieves similar asymptotic performance; Humanoid-V2 remains a notable weakness.
  • 5.2 ABLATION EXPERIMENTS: Replay, a value baseline, and TD(λ) are evaluated through component-removal experiments, with the strongest performance when the components are combined.Removing replay or the baseline noticeably degrades performance, whereas Monte Carlo returns remain a viable alternative on some tasks.
  • 5.2 ABLATION EXPERIMENTS: Larger replay buffers make AWR more stable on predominantly off-policy data, but can slow learning by limiting changes in the modeled sampling policy.The experiments compare buffer capacities of 5k, 20k, 50k, 100k, and 500k, with 50k as the default.
  • 5.3 MOTION IMITATION: AWR learns motion-imitation skills for a 34 DoF humanoid and an 82 DoF dog, generally achieving comparable or better performance than PPO.RWR struggles more with the humanoid but performs more strongly on the dog.
  • 5.4 OFF-POLICY LEARNING WITH STATIC DATASETS: On static demonstration datasets, AWR learns policies comparable or better than the original demo policies without collecting additional data.In this fully off-policy setting, AWR is compared with behavioral cloning, BCQ, BEAR, TD3, SAC, and PPO.

6 DISCUSSION AND FUTURE WORK

AWR combines simple supervised-learning policy updates with competitive performance across challenging control settings and fully off-policy datasets, but remains less sample efficient than the most efficient off-policy algorithms.

  • AWR solves challenging control tasks with complex simulated agents and achieves competitive performance on standard benchmarks against established reinforcement-learning algorithms.
  • AWR learns from fully off-policy datasets with performance comparable to state-of-the-art off-policy methods.
  • AWR is not yet as sample efficient as the most efficient off-policy algorithms.
  • The paper identifies improving sample efficiency and fully off-policy learning as opportunities for future deployment in real-world domains.

A AWR DERIVATION

The derivation frames AWR as approximate constrained policy optimization: it maximizes expected improvement using advantages while keeping the new policy close to the sampling policy through a relaxed KL constraint.

  • AWR seeks a policy maximizing expected improvement η(π)=J(π)−J(µ) over a sampling policy µ, expressed using the advantage Aµ(s,a).
  • The state-based objective replaces trajectory expectations with the sampling policy’s discounted state distribution and action distribution.
  • The surrogate objective matches expected improvement to first order and is reliable when the new and sampling policies are close in KL divergence.
  • A pointwise KL constraint is relaxed into an expected soft constraint with coefficient β to make optimization tractable.
  • Lagrangian optimization yields an optimal policy involving a normalized exponential weighting, which can be projected onto a parameterized policy using supervised regression.

B AWR DERIVATION WITH EXPERIENCE REPLAY

With experience replay, AWR models the sampling distribution as a trajectory-level mixture of past policies, constrains updates relative to that mixture, and uses practical approximations for otherwise unavailable rollouts and value baselines.

  • The replay buffer’s sampling distribution is a trajectory-level mixture of k past policies, with corresponding trajectory, state-action, and state marginals.
  • Constraining the new policy against the replay buffer prevents actions far from all past policies and weights similarity by policies’ visited-state densities.
  • The replay derivation produces an optimal policy that is projected onto the parameterized-policy manifold through supervised regression.
  • Computing the expected return in the exponent would require multiple policy rollouts from the same state and a resettable environment, so practice uses a single replay-buffer rollout.
  • The single-sample estimator is biased because mixture-weight expectations occur inside the exponent, although it is reported effective in the experiments.
  • Instead of fitting separate value functions for each past policy, the method fits one mean value function estimating their weighted average.

C EXPERIMENTAL SETUP

The experimental implementation uses neural networks for policy and value estimation, with a separate value network and stochastic-gradient updates.

  • The policy uses a fully connected network with hidden layers of 128 and 64 ReLU units followed by a linear output layer.
  • The value function uses a separate network with a similar architecture and a single linear output unit.
  • Both policy and value networks are updated using stochastic gradient descent with momentum.

D LEARNING CURVES

Across OpenAI Gym benchmarks, AWR is generally competitive with leading methods, while on challenging motion imitation tasks it generally learns faster than PPO and RWR.

  • AWR is generally competitive with the best current methods on OpenAI Gym tasks.
  • The OpenAI Gym learning curves average results over 5 random seeds.
  • On challenging motion imitation tasks, AWR generally learns faster than PPO and RWR.
Loading 1910.00177v3…