PaperlayerView exact revision on arXiv

Source-linked AI summary

Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning

Qingkai Liang, Fanyu Que, Eytan Modiano

arXiv:1802.06480v1cs.AIcs.LGstat.ML

TL;DR

CMDPs require policies to maximize long-term reward while satisfying long-term cost constraints, but existing on-policy dual updates are sample-inefficient and slow. APDO combines an off-policy-trained dual variable with an on-policy primal policy update; on simulated robot locomotion, it reached a reward-and-safety target in half the epochs CPO required.

  • Problem

    Existing CMDP methods use on-policy data for dual updates, which makes them sample-inefficient and slow to converge.

    Source: §1 · p. 2

  • Method

    APDO uses an off-policy-trained dual variable in its dual update while updating the primal policy with an on-policy likelihood-ratio gradient.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    45 epochs versus CPO's 90 to reach average reward 11 while satisfying the safety constraint, a 2x sample-efficiency improvement on this task.

    Source: §4 · p. 3

  • Takeaways & Limitations

    On simulated robot locomotion, APDO achieved better sample efficiency and faster convergence than state-of-the-art CMDP approaches.

    Source: §1 · p. 2

    The paper leaves theoretical guidance for choosing the adjustment epoch and justification of APDO's acceleration effects as future work.

    Source: §5 · p. 4

Abstract

from arXiv · show

Constrained Markov Decision Process (CMDP) is a natural framework for reinforcement learning tasks with safety constraints, where agents learn a policy that maximizes the long-term reward while satisfying the constraints on the long-term cost. A canonical approach for solving CMDPs is the primal-dual method which updates parameters in primal and dual spaces in turn. Existing methods for CMDPs only use on-policy data for dual updates, which results in sample inefficiency and slow convergence. In this paper, we propose a policy search method for CMDPs called Accelerated Primal-Dual Optimization (APDO), which incorporates an off-policy trained dual variable in the dual update procedure while updating the policy in primal space with on-policy likelihood ratio gradient. Experimental results on a simulated robot locomotion task show that APDO achieves better sample efficiency and faster convergence than state-of-the-art approaches for CMDPs.

1 Introduction

CMDPs frame safety-constrained reinforcement learning as maximizing long-term reward while respecting long-term cost limits. APDO addresses the sample inefficiency and slow convergence of on-policy dual updates by adding an off-policy-trained dual variable, and is reported to improve sample efficiency and convergence on simulated robot locomotion.

1 Introduction

  • Existing constrained policy-search methods use only on-policy samples for dual updates, discarding historical data and requiring large batches before incremental updates.These updates are described as sample-inefficient and slow to converge.Source: §1 · p. 2
  • On a simulated robot-locomotion task with safety-motivated constraints, APDO shows better sample efficiency and faster convergence than state-of-the-art CMDP approaches, including PDO and CPO.Source: §1 · p. 2
  • APDO adjusts its dual variable once using an off-policy-trained estimate while retaining an on-policy likelihood-ratio gradient for the primal policy update.The paper reports better sample efficiency and faster convergence than state-of-the-art CMDP approaches.Source: §1 · p. 2
  • APDO's use of off-policy data for dual updates complements methods that combine on- and off-policy policy gradients for unconstrained MDP primal updates.The paper names Q-Prop and IPG as examples of the latter approach.Source: §1 · p. 2

2 Constrained Markov Decision Process

The section defines an MDP through its states, actions, reward, transition probabilities, and initial-state distribution, and describes policies and long-term discounted reward. A CMDP adds long-term cost functions and limits, making the objective to maximize reward subject to those cost constraints.

2 Constrained Markov Decision Process

  • An MDP is specified by states, actions, a reward function, transition probabilities, and an initial-state distribution.A stationary policy maps states to action distributions; this paper searches within a parameterized stationary-policy class.Source: §2 · p. 2
  • The long-term discounted reward is the expected discounted reward accumulated along trajectories generated under a policy.The discount factor is γ, and trajectories follow the initial-state distribution, policy, and transition dynamics.Source: §2 · p. 2
  • A CMDP augments an MDP with long-term discounted costs and limits, seeking a policy that maximizes reward while keeping each cost within its limit.The section specifies m cost functions, with constraints Ci(π) ≤ di for each i.Source: §2 · p. 2

3 Algorithm

APDO mostly follows PDO's dual gradient ascent but, after Kadj iterations, makes a one-time adjustment using an off-policy dual variable trained with historical replay-buffer data, then fine-tunes with new on-policy data. The adjustment epoch trades earlier intervention against a more accurate dual estimate.

3 Algorithm

  • On-policy samples come from the currently used policy, while off-policy samples come from other unknown policies.These sample types distinguish the policy data used in the primal and off-policy dual procedures.Source: §3 · p. 2
  • The CMDP is converted to an unconstrained Lagrangian problem, then solved by alternating policy and dual updates, with an on-policy likelihood-ratio gradient available for the policy step.The policy step also permits an off-policy deterministic policy gradient.Source: §3 · p. 3
  • PDO uses projected dual gradient ascent, whereas CPO solves an optimization problem from scratch at each iteration to derive its dual variable.The PDO update uses constraint-cost deviation and projects the multiplier onto the nonnegative dual space.Source: §3 · p. 3
  • After Kadj iterations, APDO makes a one-time dual adjustment using λOFF trained with historical replay-buffer data, then fine-tunes it with new on-policy data.In most iterations, APDO updates the dual variable using simple dual gradient ascent.Source: §3 · p. 3
  • Off-policy training reuses historical samples, while on-policy updates use each sample once; offline dual optimization also avoids waiting for large batches between updates.These are the stated mechanisms behind APDO's sample-efficiency and acceleration effects.Source: §3 · p. 3
  • A smaller Kadj adjusts earlier but risks an inaccurate λOFF estimate from insufficient data, whereas a larger Kadj delays adjustment while improving estimate accuracy.The paper identifies Kadj as an important APDO parameter.Source: §3 · p. 3

4 Experiments

On the point-gather task, APDO reached the same reward while satisfying the safety constraint in half the epochs CPO required, and its dual variable converged faster than PDO’s. The experiments also compare constraint enforcement and examine APDO’s adjustment epoch.

  • The comparison evaluated APDO, CPO, and PDO on a MuJoCo point-gather task with a safety constraint; APDO used adjustment epoch Kadj = 5.Source: §4 · p. 3
  • APDO took 45 epochs, versus CPO’s 90, to reach average reward 11 while satisfying the safety constraint, yielding a 2x sample-efficiency improvement over CPO in this task.Source: §4 · p. 3
  • PDO failed to enforce the safety constraint during its first 150 epochs; increasing its step size risks over-correction and overly conservative behavior.Source: §4 · p. 4
  • APDO’s dual variable converged to the optimum significantly faster than PDO’s, with a jump after adjustment using the off-policy trained λOFF.Source: §4 · p. 4

5 Future Work

The authors identify theoretical guidance for choosing APDO’s adjustment epoch as future work. They also propose studying theoretical justification for acceleration and a tunable safety parameter.

  • Providing theoretical guidance for setting APDO’s adjustment epoch Kadj remains future work.Source: §5 · p. 4
  • The authors also propose theoretical justification for APDO’s acceleration and a safety parameter to tune the balance between risk-averse constraint enforcement and reward-focused actions.Source: §5 · p. 4

A Primal-Dual DDPG for CMDPs

The appendix presents primal-dual DDPG for CMDPs as an off-policy procedure that uses replay-buffer data to fit APDO’s λOFF. It describes reward and cost critics, target networks, and actor and dual updates.

B Experiment Details

The experiment details specify the point-gather task, shared policy-update settings, and dual-update choices. They also describe the trade-off involved in selecting APDO’s adjustment epoch.

  • The point-gather agent receives reward 10 per apple and cost 1 per bomb, with long-term cost capped at 0.2; two apples and eight bombs spawn per episode.Source: B Experiment Details · p. 7
  • All methods use two-hidden-layer tanh policies and TRPO updates with batch size 50000, KL step size 0.01, discount 0.995, and rollout length 15.Source: B Experiment Details · p. 7
  • PDO’s dual step size is difficult to tune because small steps slow updates while large steps can over-correct; APDO uses the same 0.1 step size for comparison.Source: B Experiment Details · p. 7
  • Smaller Kadj avoids early incremental dual-update delay but can yield a biased λOFF with limited replay data; larger Kadj improves estimate accuracy but delays adjustment.Source: B Experiment Details · p. 7, Figure 2 · p. 7
Loading 1802.06480v1…