PaperlayerView exact revision on arXiv

Source-linked AI summary

Constrained Variational Policy Optimization for Safe Reinforcement Learning

Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Zhiwei Steven Wu, Bo Li, Ding Zhao

arXiv:2201.11927v3cs.LGcs.AIcs.RO

TL;DR

Safe RL seeks policies that meet safety constraints, while prior primal-dual approaches have instability and optimality-guarantee drawbacks. CVPO casts the problem as probabilistic inference and applies an EM-style method with a constrained E-step and trust-region M-step. On continuous robotic tasks, it reports more stable training, better constraint satisfaction, and up to 1000 times better sample efficiency than on-policy baselines.

  • Problem

    Safe RL must learn policies that satisfy constraints, while primal-dual policy optimization has drawbacks including instability and lack of optimality guarantees.

    Source: §1 · pp. 1–2

  • Method

    CVPO uses an EM-style approach that solves an optimal feasible non-parametric variational distribution in a convex E-step, then improves the policy within a trust region in the M-step.

    Source: §1 · p. 2

  • Results

    On continuous control tasks, CVPO reports more stable training, better constraint satisfaction, and up to 1000 times better sample efficiency than on-policy baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CVPO’s decomposition provides off-policy sample efficiency, stability guarantees, and E-step optimality and feasibility guarantees.

    Source: §5 · p. 9

    The method is more computationally expensive because it solves a convex optimization problem in the E-step.

    Source: §5 · p. 9

Abstract

from arXiv · show

Safe reinforcement learning (RL) aims to learn policies that satisfy certain constraints before deploying them to safety-critical applications. Previous primal-dual style approaches suffer from instability issues and lack optimality guarantees. This paper overcomes the issues from the perspective of probabilistic inference. We introduce a novel Expectation-Maximization approach to naturally incorporate constraints during the policy learning: 1) a provable optimal non-parametric variational distribution could be computed in closed form after a convex optimization (E-step); 2) the policy parameter is improved within the trust region based on the optimal variational distribution (M-step). The proposed algorithm decomposes the safe RL problem into a convex optimization phase and a supervised learning phase, which yields a more stable training performance. A wide range of experiments on continuous robotic tasks shows that the proposed method achieves significantly better constraint satisfaction performance and better sample efficiency than baselines. The code is available at https://github.com/liuzuxin/cvpo-safe-rl.

1. Introduction

Safe RL seeks policies that maximize reward while respecting safety constraints, but learning constrained parametrized policies remains challenging. CVPO frames safe RL as probabilistic inference and uses an EM-style algorithm; experiments report more stable training, better constraint satisfaction, and higher sample efficiency than on-policy baselines.

1. Introduction

  • CVPO targets safe policy learning from environment interaction and constraint-violation signals, without requiring prior dynamics or constraint knowledge or an expert oracle.The paper presents this as a more general setting for safe RL.Source: §1 · p. 1
  • The paper identifies a need for constrained RL optimization that is sample efficient, stable, and supported by performance guarantees.It notes instability and missing per-iteration optimality guarantees in primal-dual methods, and practical constraint-satisfaction issues with Taylor approximations.Source: §1 · p. 1
  • The paper formulates safe RL as a probabilistic inference problem to use inference techniques against drawbacks of primal-dual policy optimization, including unstable training and lack of optimality guarantees.The authors describe this as the first such formulation to their knowledge.Source: §1 · p. 2
  • The EM-style algorithm solves an optimal, feasible non-parametric variational distribution in the E-step, then trains the parametrized policy through supervised learning in the M-step.The M-step allows policy improvement with off-policy data.Source: §1 · p. 2
  • The E-step’s closed-form variational distribution has a provable optimality guarantee, while trust-region policy improvement provides a worst-case constraint violation bound and robustness against worst-case training iterations.The authors attribute uniqueness and optimality in most cases to strict convexity of the dual problem.Source: §1 · p. 2
  • CVPO achieves more stable training, better constraint satisfaction, and up to 1000 times better sample efficiency than on-policy baselines on continuous control tasks.The reported experiments evaluate CVPO on a series of continuous control tasks.Source: §1 · p. 2

2. Preliminary and Related Work

This section defines safe RL using constrained Markov decision processes and reviews optimization and inference approaches related to CVPO.

2.1. Constrained Markov Decision Processes

  • A constrained Markov decision process (CMDP) models safe RL with auxiliary safety costs; its objective is to maximize expected cumulative reward while keeping expected costs below a threshold.The CMDP includes states, actions, transitions, rewards, a discount factor, an initial-state distribution, and safety costs; reward and cost are expressed as discounted expected returns.Source: §2.1 · p. 2

2.2. Related Work

  • Primal-dual methods alternate policy and dual-variable updates, but provide little safety guarantee during training; KL-regularized methods offer worst-case bounds but their quadratic approximations can be costly.Primal-dual methods also rely on accurate on-policy constraint-value estimates, complicating off-policy use.Source: §2.2 · p. 2
  • Prior RL-as-inference methods include exact message-passing inference, off-policy EM policy improvement, and variational inference, but the theoretical connection to safe RL had not been explored.MPO is described as achieving comparable performance and better sample efficiency than on-policy methods.Source: §2.2 · p. 3

3. Constrained Variational Policy Optimization (CVPO)

CVPO reframes safe RL as probabilistic inference and alternates a constrained variational-distribution update with policy improvement. Its E-step uses convex optimization, while its M-step improves the policy through supervised learning within a trust region.

3.1. Constrained RL as Inference

  • The inference formulation inserts a feasible variational distribution between reward optimization and policy improvement, replacing direct primal-dual coupling with a supervised-learning update.The paper frames inference as finding feasible actions likely to have produced high reward, rather than directly optimizing actions under constraints.Source: §3.1 · pp. 3–4

3.2. Constrained E-step

  • The E-step maximizes task reward subject to safety feasibility, a KL trust region around the old policy, and valid action-distribution constraints.A non-parametric q makes the objective linear and constraints convex; for continuous actions, sampled particles represent q.Source: §3.2 · p. 4
  • Under Slater’s condition, the E-step’s non-parametric variational distribution has an optimal closed-form solution; its dual is strictly convex and uniquely solvable when the stated conditions hold.The conditions require nonconstant reward and cost Q-functions, no proportionality between them, and a finite safety multiplier.Source: §3.2 · p. 5

3.3. M-step

  • The M-step trains the parameterized policy by weighted maximum a-posteriori learning, using the optimal variational distribution to weight policy samples and a KL constraint to limit updates.The paper uses a Gaussian prior around the old policy parameters; the regularizer is intended to improve robustness and prevent overfitting.Source: §3.3 · p. 6

3.4. Theoretical Analysis

  • CVPO guarantees nondecreasing ELBO under Slater’s condition and provides a worst-case constraint-violation bound linked to the M-step trust-region size and approximation error.When the M-step KL threshold is smaller than the E-step threshold, the next variational distribution is guaranteed feasible and optimal.Source: §3.4 · p. 6

3.5. Practical Implementation

  • CVPO can use replay-buffer samples for off-policy learning; for discrete actions, the implementation replaces action-space integration with summation.The M-step also separates KL constraints on the policy mean and covariance to adjust the exploration–exploitation trade-off.Source: §3.5 · p. 6

4. Experiment

Experiments on five continuous robotic tasks compare CVPO with on- and off-policy baselines. CVPO uses substantially fewer constraint violations for the same reward, while maintaining strong constraint satisfaction; removing the M-step KL constraint can harm safety and reward.

4.1. Tasks

  • Five robotic tasks span circle navigation, static-obstacle goal reaching, and harder dynamic-obstacle button tasks using point and car robots.The study describes five continuous control tasks with different difficulty levels.Source: §4 · p. 6, §4.1 · p. 7

4.2. Baselines

  • Comparisons include PID-Lagrangian SAC, DDPG, and TD3; CPO, TRPO-Lag, and PPO-Lag; and unconstrained TRPO, with matched network sizes.The authors use TRPO as an unconstrained reward-performance reference.Source: §4.2 · p. 7

4.3. Results

  • CVPO outperforms off-policy baselines in constraint satisfaction while maintaining high episodic reward, with more stable training.The training curves average 10 random seeds; shaded areas show standard deviation.Source: §4.3 · pp. 7–8, Figure 2 · p. 8
  • CVPO achieves task performance comparable to unconstrained TRPO while maintaining safety with fewer violations than on-policy safe RL methods.CPO fails to satisfy constraints on Goal and Button, while PPO-Lag shows large variance and low task reward.Source: Figure 3 · p. 8, §4.3 · p. 8
  • 100 ∼ 1000 times less cumulative constraint violations obtain the same task reward with CVPO than with baselines.The comparison uses reward versus cumulative cost, with cost on a log scale.Source: §4.3 · p. 8, Figure 4 · p. 9
  • At convergence, CVPO meets safety requirements with smaller cost variance than most baselines, and its third-quartile cost is below the target threshold.The authors interpret this as state-wise constraint satisfaction rather than satisfaction only in expectation.Source: §4.3 · pp. 8–9, Figure 5 · p. 9

4.4. Ablation study

  • Removing the M-step KL constraint risks constraint failure on Car-Button and lower reward on Car-Circle and Point-Goal through policy overfitting.Without KL constraints, the stated worst-case performance bound tends to be infinite and the robustness guarantee does not hold.Source: Figure 6 · p. 9, §4.4 · p. 9

5. Conclusion

The conclusion presents safe RL as a convex optimization phase over a non-parametric variational distribution followed by supervised policy learning, with theoretical guarantees and empirical gains in constraint satisfaction and sample efficiency. It also notes the E-step’s computational cost; the appendix passage describes an auxiliary trajectory distribution used to derive an ELBO lower bound.

5. Conclusion

  • The method decomposes safe RL into a convex E-step over a non-parametric variational distribution and a supervised learning phase.The conclusion also attributes theoretical optimality and feasibility guarantees to the E-step.Source: §5 · p. 9
  • Experiments show better empirical constraint satisfaction and sample efficiency than previous primal-dual approaches.The conclusion also cites monotonic reward improvement, a worst-case constraint-violation bound, and policy-update robustness as stability properties.Source: §5 · p. 9
  • The E-step requires solving a convex optimization problem, making the method more computationally expensive.Source: §5 · p. 9

A.1. Proof of the evidence lower bound (ELBO) in section 3.1

  • The ELBO proof introduces an auxiliary trajectory distribution and uses Jensen’s inequality to obtain a lower bound on the log-likelihood of optimality.The trajectory probability under policy π is defined in the proof.Source: §A.1 · p. 12

A.2. Proof and discussion of Theorem 1 — the optimal variational distribution and the dual function

The E-step optimizes a variational distribution through a convex constrained problem and its dual, yielding a closed-form distribution with optimality guaranteed under Slater’s condition. The appendix also discusses when the dual is strictly convex, how off-policy batch implementation is used, and conditions under which the analysis’s guarantees apply.

A.2. Proof and discussion of Theorem 1 — the optimal variational distribution and the dual function

  • The E-step formulates a convex optimization over q, then derives its optimal distribution and equivalent dual problem.The optimal q is obtained by setting the Lagrangian derivative with respect to q to zero and determining its normalizer.Source: §A.2 · pp. 12–13
  • Under Slater’s condition, strong duality guarantees the optimality of the closed-form non-parametric distribution.The condition requires a feasible distribution within the old policy’s KL trust region.Source: §A.2 · p. 13
  • Unlike FOCOPS, CVPO optimizes a non-parametric variational distribution, analytically derives a convex dual, and permits off-policy evaluation.The passage contrasts this with FOCOPS’s parametric objective, first-order approximation, gradient-based dual optimization, and on-policy advantage estimates (Zhang et al., 2020).Source: §A.2 · p. 13
  • The batch dual is preferred in off-policy implementation because solving a dual separately for each state is computationally expensive and sensitive to Q-value estimation error.The batch size controls whether the dual is solved over a batch or per state.Source: §A.2 · p. 14
  • The dual is convex, and the per-state variant is strictly convex only under stated non-degeneracy conditions on reward and cost values and finite optimal λ.These conditions require nonconstant reward and cost functions and that reward is not everywhere proportional to cost; strict convexity gives uniqueness in most situations, while equality may allow multiple optima.Source: §A.2 · pp. 14–16
  • When reward is proportional to cost, multiple optimal solutions may exist; if no feasible distribution lies within the trust region, Slater’s condition fails and λ tends toward infinity.The text notes that a larger E-step KL threshold and smaller M-step trust region can help satisfy Slater’s condition.Source: §A.2 · p. 16

A.4. Proof of the M-step — KL regularized policy improvement

The M-step improves policy parameters using the optimal variational distribution, framing the update as a MAP problem with a prior centered on the old policy and a KL-constrained form.

A.4. Proof of the M-step — KL regularized policy improvement

  • Given the optimal variational distribution from the E-step, the M-step optimizes a MAP objective with a Gaussian prior centered on the old policy parameters.The prior uses the Fisher information matrix and a positive coefficient; the objective is then generalized to KL regularization and a hard KL constraint.Source: §A.4 · p. 17

A.5. Proof of Proposition 1 — the ELBO improvement guarantee

The ELBO improvement guarantee is established under Slater’s condition for consecutive policies and feasible variational distributions. With that assumption, the E-step and M-step updates yield monotonic improvement; n-step robustness can maintain the condition for a finite sequence of updates.

A.5. Proof of Proposition 1 — the ELBO improvement guarantee

  • Under the stated Slater assumption, the E-step increases the ELBO with respect to q, and the combined updates yield monotonic ELBO improvement.The proof assumes Slater’s condition for both consecutive policies and a well-optimized M-step.Source: §A.5 · pp. 17–18
  • The guarantee depends on Slater’s condition; with a feasible initial policy and suitable KL thresholds, n-step robustness ensures it for n consecutive updates.If the condition fails, the E-step may focus on reducing cost or restoring feasibility instead of improving reward.Source: §A.5 · p. 18

A.6. Proof of Proposition 2 — worst-case constraint satisfaction bound

The proof relates changes in cost returns to policy divergence, then applies a worst-case bound because the M-step does not impose a cost constraint while updating the policy.

A.6. Proof of Proposition 2 — worst-case constraint satisfaction bound

  • CPO Corollaries 2 and 3 bound Jc(π′)−Jc(π) above by (1/(1−γ))E_{s~ρπ,a~π′}[Aπc(s,a)] + 2γδπ′c/(1−γ)², linking cost-return difference to policy divergence.Here δπ′c = max_s E_{a~π′}[Aθic(s,a)], and Aθic(s,a) denotes the cost advantage function.Source: §A.6 · p. 18
  • The maximum absolute cost advantage, δ, is defined over states and is used in the cost-return bound.The cost advantage is denoted A_c; the passage gives its definition in fragmented notation.Source: §A.6 · p. 18
  • Because the M-step updates θ without a cost constraint, the proof gives a worst-case bound on the next policy’s cost return.The bound includes an expected cost-advantage term and additional terms involving δ and the M-step approximation error.Source: §A.6 · p. 18

A.7. Proof and discussion of Proposition 3 — policy updating robustness

The proposition establishes that the E-step can recover a feasible, optimal variational distribution after policy-update perturbations under trust-region conditions. The discussion extends this guarantee to two steps for Gaussian policies and identifies a robustness–efficiency trade-off for longer horizons.

A.7. Proof and discussion of Proposition 3 — policy updating robustness

  • If the M-step KL threshold ε is below the E-step threshold ε₂, the next E-step distribution is guaranteed feasible and optimal when the current policy is feasible.The proof invokes the Slater condition because the prior feasible policy remains within the E-step trust region.Source: §A.7 · pp. 18–19
  • The guarantee addresses M-step perturbations from function-approximation errors or noisy Q estimates, including worst-case adversarial perturbations.The E-step can still recover a feasible region after such perturbations.Source: §A.7 · p. 19
  • For Gaussian policies, a relaxed KL triangular inequality supports n-step robustness when the M-step threshold is sufficiently small relative to the E-step threshold.The proof develops the two-step case under this condition.Source: §A.7 · pp. 19–20
  • Under the stated small-threshold condition, the variational distribution two iterations later is guaranteed feasible and optimal.The conclusion applies to q* at iteration i+2.Source: §A.7 · pp. 19–20
  • Certifying more recovery steps requires shrinking the M-step trust region faster than the step count, trading robustness for training efficiency.The authors regard one or two steps as sufficient for mild approximation errors in practice.Source: §A.7 · p. 20

B. Implementation Details

The implementation appendix describes CVPO’s replay-buffer training loop, critic and policy updates, environment modifications, and hyperparameter organization.

B.1. Full algorithm

  • The implementation uses neural-network critics and policies, target networks, and Polyak smoothing; critic losses minimize mean-squared Bellman error.The appendix notes other off-policy critic-training methods could also be incorporated.Source: §B.1 · p. 21
  • The Gaussian-policy M-step decouples KL constraints for the mean and covariance, and uses target-policy covariance with current-policy mean to reduce collapse to a local optimum.The resulting KL-constrained policy improvement is trained in a supervised-learning fashion.Source: §B.1 · p. 21
  • The full algorithm collects trajectories, samples replay-buffer transitions, updates reward and cost critics, solves the E-step convex problem, then performs supervised M-step policy updates and Polyak averaging.The algorithm also samples actions for each state and computes the optimal variational distribution before the M-step.Source: §B.1 · p. 22

B.2. Experiment environments

  • Experiments modify SafetyGym and PyBullet environments by increasing simulation time steps and shortening timeouts; the Goal task uses a fixed layout.CVPO also works in original SafetyGym environments but may require different hyperparameters and longer training; its particle sampling and CPU convex optimization make it computationally costly.Source: §B.2 · p. 22

B.3. Hyper-parameters

  • Table 1 organizes hyperparameters into settings shared by all methods and settings specific to CVPO.The appendix directs readers to the code for further hyperparameter details.Source: Table 1 · p. 23, §B.3 · p. 23

C. Complete Experiment Results

The complete-results section presents training curves, reward–cost plots, convergence-cost distributions, and an M-step KL-constraint ablation; the supplied passages describe these visuals but not their outcomes.

C. Complete Experiment Results

  • Off-policy and on-policy baseline training curves are shown by environment, with means and standard deviations across 10 random seeds.Solid lines denote means and shaded regions denote standard deviations.Source: Figure 8 · p. 23, Figure 9 · p. 24
  • Reward is plotted against cumulative cost on a logarithmic cost scale.Source: Figure 10 · p. 24
  • Convergence cost is presented as box plots that distinguish on-policy from off-policy methods.Source: Figure 11 · p. 25
  • The section includes an ablation study of the M-step KL constraint.Source: Figure 12 · p. 25
Loading 2201.11927v3…