Source-linked AI summary
Reward Constrained Policy Optimization
Chen Tessler, Daniel J. Mankowitz, Shie Mannor
TL;DR
RL reward misspecification can produce unwanted behavior, while general constrained optimization lacks a closed-form solution. RCPO introduces a multi-timescale guiding penalty, proves convergence, and empirically obtains stable, sample-efficient constraint satisfaction.
Problem
RL agents can exploit loopholes in misspecified reward signals, while existing constrained optimization lacks a closed-form solution for general constraints.
Method
RCPO incorporates an alternative penalty signal into the reward and uses multi-timescale updates for the critic, policy, and penalty coefficient.
Results
RCPO converges almost surely to a constraint-satisfying solution and empirically shows faster convergence, improved stability, and sample-efficient constraint satisfaction.
Takeaways & Limitations
RCPO handles general constraints, is invariant to scaling of the underlying reward, and does not require prior knowledge.
Takeaways & Limitations
In Hopper-v2, the experiments do not clearly establish whether RCPO or the λ = 0.1 reward-shaping variant is superior.
Abstract
from arXiv · showhide
Solving tasks in Reinforcement Learning is no easy feat. As the goal of the agent is to maximize the accumulated reward, it often learns to exploit loopholes and misspecifications in the reward signal resulting in unwanted behavior. While constraints may solve this issue, there is no closed form solution for general constraints. In this work we present a novel multi-timescale approach for constrained policy optimization, called `Reward Constrained Policy Optimization' (RCPO), which uses an alternative penalty signal to guide the policy towards a constraint satisfying one. We prove the convergence of our approach and provide empirical evidence of its ability to train constraint satisfying policies.
1 Introduction
RL agents optimize accumulated reward, but misspecified or composite reward signals can produce unwanted behavior and require costly, domain-specific tuning. RCPO addresses general constraints with a guiding penalty, proving convergence and reporting stable, sample-efficient constraint satisfaction.
- Reward signals implicitly define agent behavior, but many real applications lack a predefined reward signal.
- Mujoco tasks combine forward-progress rewards with bonuses and penalties for survival, energy use, and foot impact, each weighted by a coefficient.
- Penalty coefficients create a multi-objective problem, require costly tuning, and may transfer poorly across domains, where one setting can cause catastrophic failure.
- Existing constrained methods are limited in supported constraint types and algorithms, including policy parametrization and trajectory-wide violation propagation requirements.
- RCPO incorporates a penalty signal into the reward, proves almost-sure convergence under mild assumptions, and empirically achieves faster convergence and improved stability.The empirical evaluation covers a toy domain and six robotics domains.
- RCPO targets general discounted-sum and mean-value constraints, is invariant to scaling of the underlying reward, and does not require prior knowledge.
2 Preliminaries
The paper formulates constrained decision-making by extending MDPs with penalty signals and general constraints, then considers parametrized policies under assumptions aimed at feasible convergence. General constraints include discounted, average, and other forms.
- Markov Decision Processes: An MDP consists of states, actions, rewards, transition probabilities, an initial-state distribution, and a discount factor, with policies mapping states to action distributions.
- Constrained MDPs: A CMDP extends an MDP with a penalty c(s, a), a constraint C(st), and threshold α, allowing discounted-sum, average-sum, and other general constraints.
- Constrained MDPs: The constrained problem is to optimize the target objective while satisfying the constraint threshold.
- Parametrized Policies: The analysis considers parametrized policies πθ and states assumptions intended to ensure convergence to a constraint-satisfying policy.
- Parametrized Policies: The minimal feasibility requirement supports convergence to a feasible solution, whereas stronger assumptions such as convexity could support optimal convergence but generally fail for non-convex constraints.
3 Constrained Policy Optimization
Constrained policy optimization uses Lagrangian relaxation to balance reward and infeasibility, updating policy parameters faster than the penalty multiplier. The resulting iterates converge almost surely to a feasible fixed point under stated assumptions.
- Lagrangian relaxation converts the constrained problem into an unconstrained objective with a penalty term for infeasibility.
- As λ increases, the penalized solution approaches the constrained solution, motivating faster policy updates and slower multiplier updates toward a feasible saddle point.
- The feasible-solution definition specifies satisfaction of the CMDP constraint.
- The simulation-based algorithm assumes access to samples from simulation rather than direct access to the MDP.
- Projection operators stabilize policy parameters and constrain λ to [0, λmax], while gradients are derived from the Lagrangian using the log-likelihood trick.
- The policy update uses a faster timescale than the penalty coefficient update, enforced by the step-size conditions.
- Theorem 1 establishes almost-sure convergence of the iterates to a fixed point under the stated assumptions and standard stability and bounded-noise conditions.
- Lemma 1 shows that the fixed point from Theorem 1 is a feasible solution under assumptions 1 and 2.
4 Reward Constrained Policy Optimization
RCPO uses an alternative discounted penalty to train the actor and critic for general constraints, while optimizing the original constraint with Monte Carlo sampling. Under stated assumptions, its three-timescale process converges almost surely to a feasible fixed point.
- Actor-critic methods use temporal-difference learning to estimate values from finite samples, but general constraints may not satisfy the recursive property required for critic training.
- RCPO trains the actor and critic with a discounted guiding penalty while optimizing λ from the original constraint using Monte Carlo sampling.
- The algorithm is a three-timescale constrained actor-critic process in which the actor and critic follow the penalized value and λ follows the original constraint update.
- Algorithm 1 specifies penalty and constraint inputs, a threshold α, ordered learning rates, initialized actor and critic parameters, and sampled rewards and penalties.
- RCPO converges almost surely to a feasible fixed point when the local minima associated with the discounted guiding penalty are contained in those associated with the original constraint.
5 Experiments
Experiments evaluate RCPO on a grid-world and six Mujoco tasks, comparing adaptive constraint handling with Lagrange optimization and fixed penalty coefficients. RCPO generally finds feasible solutions, converges faster and more stably in the rover scenarios, and outperforms reward shaping in most robotics domains.
- Experimental setup: RCPO is tested on a grid-world and six Mujoco robotics tasks, with comparisons to Lagrange optimization and reward shaping.The grid-world compares RCPO with a standard Primal-Dual approach, while Mujoco experiments compare it with fixed-λ reward shaping.
- Mars Rover: For both α = 0.01 and α = 0.5, RCPO converges faster and with lower variance than the Lagrange comparison.The rover experiments use A2C and compare RCPO with discounted-penalty optimization of the Lagrange dual form.
- Mujoco torque constraints: In all Mujoco domains, RCPO finds a feasible or near-feasible solution and outperforms constant-λ reward shaping except on Walker2d-v2.The torque constraint limits average motor torque, and RCPO seeks the highest-value policy satisfying that constraint.
- Mujoco torque constraints: The Hopper-v2 results do not clearly establish whether RCPO or the λ = 0.1 reward-shaping variant is superior.This is the stated exception to a clear method comparison within the reported domains.
- Reward-shaping analysis: Selecting a fixed penalty coefficient is difficult because suitable λ values vary across domains and can produce drastically different behavior.Higher-value domains require larger penalties for constraint satisfaction, while the same coefficients can yield severely sub-optimal solutions elsewhere.
6 Discussion
RCPO is a constrained actor-critic method that separates critic, policy, and penalty updates across three timescales. Simulations show stable, sample-efficient convergence to constraint-satisfying policies, while combining RCPO with CPO is proposed as future work.
- Method: RCPO uses three timescales: a TD critic estimates a discounted objective, policy gradients update the policy, and λ ascends on the original constraint.The critic operates fastest, policy learning at an intermediate rate, and penalty learning slowest.
- Empirical validation: Simulations in grid-world and robotics domains show stable and sample-efficient convergence to constraint-satisfying policies.The evaluation covers both grid-world and robotics domains.
- Future work: Combining RCPO with CPO is proposed as an extension that could address complex constraints while retaining feasibility guarantees during training.The proposed combination is motivated by CPO's consideration of the discounted penalty used as RCPO's guiding signal.
- Method: RCPO is presented as a constrained actor-critic approach with additions to the standard Advantage Actor Critic algorithm.The algorithm description distinguishes the original Advantage Actor Critic components from RCPO additions.
B.1 Mars Rover
The Mars Rover setup defines a grid-world MDP with specified rewards, discounting, and constraint parameters. Exploration is supported by linearly decaying random restarts, while the implementation uses shared actor-critic representations and specified evaluation and optimization settings.
- Exploration: Linearly decaying random restarts improve exploration by shifting starts from random non-terminal states toward the tested top-left state.The initial-state distribution increasingly favors the top-left state as training progresses.
- Architecture: The A2C network is non-recurrent, receives the fully observable grid, and shares internal representations between actor and critic.The actor and critic retain separate final projection layers, with ReLU nonlinearities between layers.
- Evaluation: Performance is evaluated every 5120 episodes for 1024 episodes, with λ initialized at 0.6 and learning rate lrλ = 0.000025.The evaluation protocol is intended to reduce noise in risk-sensitive environments and shorten initial convergence time.
- Robotics implementation: Robotics experiments use PPO with domain-dependent input sizes, a DiagGaussian motor-output layer, and Tanh nonlinearities.The Gaussian layer learns per-motor means and standard deviations for torque sampling.
- Evaluation: Online robotics performance is measured over 1M samples, with λ initialized at 0 and lrλ = 5e −7 decaying by κ = (1 −1e −9).The decay is used to avoid oscillations; simulations also use GAE with τ = 0.95 and γ = 0.99.
C Proof of Theorem 1
Theorem 1's proof analyzes RCPO through timescale separation and stochastic-approximation ODEs. It establishes convergence of the policy and penalty recursions toward a local saddle-point structure almost surely.
- Assumptions: The proof assumes a finite λmax initially and separately handles the case where Assumption 2 holds.The finite-bound setup and special case are stated before the convergence steps.
- Step 1: θ-recursion: The proof separates policy and penalty updates by treating the slower λ value as constant while analyzing the θ recursion.This timescale separation yields an ODE governing policy evolution.
- Step 1: θ-recursion: Projection keeps θ within the compact convex set Θ while its recursion tracks an asymptotic ODE for each fixed λ.The projection operator Γθ ensures the ODE remains in Θ.
- Step 2: λ-recursion: The proof first establishes λ-recursion convergence and then convergence of the joint process to a local saddle point of the Lagrangian.This is the stated structure of the second proof step.
- Step 2: λ-recursion: The λ process is represented by an ODE using θ(λk), the limiting point of the θ recursion associated with λk.The ODE analysis identifies the limiting behavior of the coupled process.
- Conclusion: The coupled sequence converges almost surely to internally chain transitive invariant sets, followed by θn → θ∗ and λn → λ(θ∗).These limits complete the convergence argument.
D Proof of Lemma 1
Lemma 1 extends the convergence proof under Assumption 2, using feasibility of local minima and almost-sure convergence of first-order methods to establish a feasible fixed point with an unbounded multiplier.
- Assumption: Assumption 2 requires every local minimum of the relevant objective to satisfy the constraint, namely Jπθ^C ≤ α.This feasibility condition is the key additional assumption in the lemma.
- Result: With λmax = ∞, the process converges to a fixed point (θ∗(λ∗), λ∗) that is a feasible solution.The argument combines Assumption 2 with almost-sure convergence of first-order methods to local minima.
E Proof of Theorem 2
Theorem 2 analyzes RCPO as a three-timescale stochastic approximation scheme and establishes almost-sure convergence to a feasible solution under its assumptions. When key assumptions fail, feasibility or critic-based variance reduction is no longer guaranteed.
- The critic value vk converges to v(λ, θ) because the temporal-difference operator is a contraction.
- The proof uses three timescales: the critic converges first, the policy then converges with the multiplier fixed, and the multiplier-policy-critic process converges almost surely.The critic observes θ and λ as static, after which the policy recursion treats the critic as converged and λ as static.
- Under the stated assumption, letting λmax approach infinity restricts stationary points to a subset of feasible solutions, yielding almost-sure convergence to feasibility.
- Without Assumption 2, gradient descent may reach an infeasible stationary point, so λmax acts only as a bound on constraint regularization.
- Without Assumption 4, gradient estimates may be approximated by Monte Carlo, but the absence of a critic removes reduced-variance and smaller-sample benefits.