PaperlayerView exact revision on arXiv

Source-linked AI summary

BCPPO: Bachelier-Inspired Constrained Proximal Policy Optimization for Tail-Risk-Aware Safe Reinforcement Learning

Dongsheng Hou, Yanqiao Chen, Yuhan Rui

arXiv:2608.30283v1cs.LGcs.AI

TL;DR

Expected-cost constraints may miss rare high-cost events, while high-confidence Monte Carlo CVaR gradients can be noisy and distributional critics add complexity. BCPPO uses masked cost-critic disagreement in a Bachelier-inspired PPO penalty with unchanged TD critic learning and anti-windup mean-cost control. Across 175 runs, no comparator achieves both higher mean return and lower mean CVaR in any task, supporting a practical reward–caution balance with policy-only deployment, but not a safety guarantee.

  • Problem

    Expected-cost control can permit rare high-cost outcomes, while Monte Carlo CVaR gradients may be noisy and distributional critics add complexity.

  • Method

    BCPPO uses masked cost-critic disagreement as a Bachelier-inspired actor penalty, blocks its gradients from critics, and combines it with anti-windup mean-cost control.

  • Results

    Across 175 runs, no comparator achieves both higher mean return and lower mean CVaR than BCPPO in any task.

  • Takeaways & Limitations

    The evidence supports a practical reward–caution balance around variable cost predictions with policy-only deployment.

  • Takeaways & Limitations

    Disagreement is a cautionary training signal, not an error probability, formal safety guarantee, or control for well-covered aleatoric hazards.

Abstract

from arXiv · show

Expected-cost constraints can still permit rare, high-cost events. Monte Carlo conditional value at risk (CVaR) gradients can be noisy at high confidence, whereas critics that model an outcome distribution add complexity. We propose BCPPO (Bachelier-Inspired Constrained Proximal Policy Optimization), a proximal policy optimization (PPO) method. Separately initialized cost-prediction networks (critics), trained with random sample masks, produce disagreement that marks predictions sensitive to which state-action regions occur in the training data and to critic training. A Bachelier formula for the expected amount above a reference level converts this disagreement into a smooth policy-update penalty. Gradients from this penalty do not alter the critics, so temporal-difference (TD) critic learning is unchanged. A saturation-aware controller adjusts the mean-cost penalty and stops accumulated error from growing while that penalty is clipped. Deployment retains only the policy network. The disagreement penalty is neither a tail-event probability nor a guaranteed error bound, and it provides no safety guarantee. Across 175 runs with shared tasks, costs, budgets, training steps, and evaluation seeds, no comparator attains both higher mean return and lower mean CVaR than BCPPO in any task. On Push1, BCPPO has no lower return and no higher CVaR than every comparator, with at least one strict gain. These results support a practical balance among reward, caution around cost predictions that vary across trained critics, and policy-only deployment.

1 Introduction

BCPPO addresses the gap between expected-cost constraints and rare high-cost outcomes by using critic disagreement as a cautionary policy-update signal while retaining mean-cost control. The method is evaluated under shared protocols across tasks and comparators.

  • Motivation: Expected-cost constraints can permit rare, high-cost trajectories when occasional losses remain within the budget.A policy costing zero in 95% of trajectories and L otherwise satisfies the budget whenever 0.05L ≤ Clim.
  • Contribution: BCPPO marks cost predictions sensitive to critic training without interpreting disagreement as a tail probability.Separately initialized critics use independent random sample masks to expose training-data and optimization sensitivity.
  • Method: A Bachelier-inspired expected-excess penalty adds disagreement to the cost branch while blocking its gradients from the critics.The standard reward branch, multiplier, and branch normalization remain part of the full actor update even when omitted from the overview panel.
  • Evaluation: Across 175 runs, no comparator improves both mean reward and CVaR over BCPPO in any task.On Push1, BCPPO improves at least one objective without worsening the other against every comparator.

2 Related Work

Related safe-RL methods constrain expected cost, estimate tail risk from samples or distributions, or use uncertainty estimators. BCPPO instead keeps mean-cost estimates and uses critic disagreement as an actor-side caution signal.

  • Expectation-constrained methods: Expectation-constrained baselines enforce the same mean-cost objective through different policy updates and are not tail-risk algorithms.The listed baselines include CPO, PPO-Lagrangian, and FOCOPS.
  • Sample-based and distributional tail risk: Monte Carlo CVaR gradients identify high-cost trajectories through empirical tail samples but can become noisier as confidence increases.The comparison uses CVaR-weighted PPO as a controlled Monte Carlo upper-tail baseline.
  • Sample-based and distributional tail risk: Distributional methods model return variation explicitly, including quantile cost critics used inside CVaR constraints.QR PPO tests a quantile cost critic in place of uncertainty shaping.
  • Epistemic uncertainty: Epistemic-uncertainty estimators include Bayesian decompositions, deep ensembles, and probabilistic ensembles with trajectory sampling.Safe-RL methods place these ensembles in roles such as transition-model uncertainty, offline conservatism, or exploration.
  • BCPPO’s distinction: BCPPO leaves TD targets and the original cost advantage unchanged, using disagreement only as an actor-side penalty rather than an upper bound or tail guarantee.Its numerical disagreement signal is not an OOD probability or a strict tail-risk guarantee.

3 BCPPO: Method

BCPPO combines masked cost-critic ensembles, a local Gaussian Bachelier expected-excess penalty, normalized PPO actor updates, and anti-windup mean-cost control. The ensemble shapes training but is discarded for policy-only deployment.

  • Critic ensemble: BCPPO separates mean predicted cost from disagreement across independently initialized, bootstrap-masked cost critics.Each critic estimates expected discounted future cost, while members share TD targets but not parameters, initialization, or masks.
  • Critic ensemble: Critic disagreement measures sensitivity to training data and optimization, not environment outcome variance or action safety.Each ensemble member produces one expected-cost prediction rather than a return quantile or environment sample.
  • Bachelier-inspired shaping: BCPPO approximates critic outputs locally as Gaussian and sets a state-action reference level K = µc + c0σc before computing expected excess.The shift c0 = Φ^-1(α) + κ moves the reference farther into the upper tail of the Gaussian approximation when κ increases.
  • Bachelier-inspired shaping: The implemented shaping signal is an analytically scaled disagreement penalty whose mean cancels because the cost advantage already supplies predicted cost.For fixed risk parameters, the coefficient is a positive constant, so equivalent βeff settings produce the same actor coefficient.
  • Critic learning: Detaching RB prevents actor-loss gradients from updating ensemble critics or altering their TD targets.Critics therefore retain standard TD learning independent of the Bachelier-inspired shaping penalty.
  • Training and deployment: BCPPO retains state-value networks for GAE, trains the ensemble during PPO updates, and discards the ensemble at deployment.Deployment uses the same actor-only forward pass as vanilla PPO; training adds O(BM) scalar-critic overhead and O(M) critic storage.
  • Actor update: The actor uses independently normalized reward and cost branches, with disagreement weighted inside the cost branch by β and gated by the controller multiplier λ.When λ = 0, the actor update reduces to reward PPO; when λ > 0, disagreement changes the relative penalty assigned to sampled actions.
  • Mean-cost control: An anti-windup PID controller updates and clips λ while preventing integral accumulation from growing outward at a projection boundary.The multiplier is projected to [0, λmax], and the stored integral is retained or clipped according to the error direction.

4 Experiments

Experiments use a shared rare-event-cost protocol across five tasks, comparing mean-cost and tail-aware baselines with Return, Cost rate, and CVaR@95. Results include cross-task trade-offs, ablations of BCPPO components, and tests of disagreement under missing data coverage and actor updates.

  • Experimental setup: Five tasks use 1M training steps, five training seeds, shared cost limits, and 20 fixed deterministic evaluation seeds.The tasks span Safety-Gymnasium navigation and MuJoCo locomotion under a common rare-event cost wrapper.
  • Main results: BCPPO is non-dominated in all five tasks, dominating 13 of 30 comparator–task means and trading reward against CVaR in the other 17.It dominates all six comparators on Push1, four on Button1, two on Goal1, and WCSAC on Ant.
  • Main results: The statistical claim is limited to non-dominated five-seed mean pairs, not maximal reward, minimal CVaR, or universal superiority.Only three comparator–task pairs have 95% intervals favoring BCPPO on both axes, without corrections for simultaneous testing or multiple comparisons.
  • Metrics and protocol: Return, Cost rate, and CVaR@95 are reported separately because mean-cost compliance can coexist with an excessive observed tail.Across 175 runs, 18 final policies satisfy the mean-cost budget while empirical CVaR exceeds it.
  • Ablation summary: Removing the spread penalty changes the reward–CVaR trade-off by environment, while placing disagreement as a reward bonus raises navigation CVaR most clearly on Push1.The full method favors return on Button1, lower CVaR on Push1, and improves both means on Hopper, with large seed variation.
  • Ablation summary: Removing normalization causes the largest Push1 instability, while disabling anti-windup raises mean CVaR in all three tested environments.Without normalization, Push1 CVaR@95 is 0.934 ± 1.479; disabling anti-windup raises Button1 mean CVaR from 0.414 to 0.668.
  • Mechanism checks: Missing training coverage raises disagreement in controlled tests, but disagreement is not established as a general error score or tail-event probability.AUC gains are 0.168 and 0.099, with matched target-noise-control gains of 0.126 and 0.110.
  • Mechanism checks: A paired one-update test finds larger-disagreement actions receive lower treatment-minus-control log probability, establishing update direction rather than lower final-policy CVaR.For λ > 0, Spearman rank correlation is −0.119 with 95% CI [−0.216, −0.039]; λ = 0 checks show exactly zero parameter difference.

5 Discussion

BCPPO combines critic-disagreement shaping with mean-cost control to balance reward and caution during training, while retaining policy-only deployment. Its evaluation emphasizes reward–CVaR trade-offs and reports limitations on safety interpretation, evaluation scope, and future validation.

  • 5 Discussion: BCPPO separates predicted cost from detached critic-training sensitivity, using Bachelier expected-excess shaping while preserving the original cost branch.The penalty does not alter critic gradients, and deployment retains only the policy network.
  • 5 Discussion: Across 175 runs, no comparator improves both mean reward and CVaR over BCPPO in any task.Figure 2 presents the results as mean reward–CVaR trade-offs with five-seed uncertainty bars and a finite-set empirical frontier.
  • 5 Discussion: Controlled retraining identifies omitted training-data regions through increased disagreement more readily than equal-size controls.Mixed ablations and a rarely positive multiplier caution against attributing every frontier result to disagreement alone.
  • 5 Discussion: The disagreement signal is not a tail-event probability, guaranteed error bound, formal safety guarantee, or control for well-covered aleatoric hazards.Agreement may reflect shared bias, while disagreement may reflect benign instability.
  • 5 Discussion: Evaluation uses a controlled stress test with 20 episodes, five seeds, shared hyperparameters, and configurations whose violations are not jointly retuned or budget-swept.CVaR@95 is the largest observed cost under 20 evaluation episodes.
  • 5 Discussion: Future work should combine disagreement with distributional critics, add a separate constraint for zero mean-cost multipliers, and broaden physical-hazard evaluation.Suggested extensions include more seeds and episodes, wrapper sweeps, shared features or distillation, and downstream-CVaR analysis.

6 Conclusion

BCPPO combines critic-disagreement shaping with anti-windup mean-cost control. Across five tasks, its evaluated trade-offs are non-dominated relative to the tested comparators, while controlled data-removal tests show increased disagreement in omitted regions.

  • 6 Conclusion: Across five tasks, no tested comparator has both higher mean return and lower mean CVaR than BCPPO.The conclusion supports a practical reward–caution balance with policy-only deployment, not universal tail-risk reduction or a safety guarantee.
  • 6 Conclusion: Deliberate data-removal controls show increased disagreement in omitted regions.This supports using disagreement as a cautionary training signal when coverage is limited.

A.1 Proofs of Propositions

The appendix formalizes smoothness and CVaR-excess interpretations of the Bachelier penalty under Gaussian assumptions, while distinguishing the formula’s origin from its reinforcement-learning use. It also records that increasing the reference shift lowers exceedance without implying a safer learned policy.

  • A.1 Proofs of Propositions: For the moving strike K = µ + c0σ, the Bachelier penalty has zero sensitivity to µ and finite sensitivity h(c0) to σ.The result follows by substituting the moving strike into the expected-excess formula.
  • A.1 Proofs of Propositions: Under a local Gaussian surrogate, the Bachelier term represents expected excess beyond a reference level related to CVaR excess.The formula is repurposed from arithmetic-Brownian-motion option pricing to model expected cost excess.
  • A.1 Proofs of Propositions: For κ > 0, the reference strike lies above VaRα, making the expected excess smaller than the VaR-strike excess.This algebra does not imply that the learned policy becomes more conservative as κ increases.
  • A.1 Proofs of Propositions: Branch-normalized advantages are invariant to independent positive rescaling of reward and cost-risk branches, leaving λ as a dimensionless relative priority.The normalization property applies to the branch terms used in the policy update.

A.3 Extended Theoretical Diagnostics

The extended diagnostics establish bounded local sensitivity, anti-windup behavior, and idealized finite-ensemble properties for selected BCPPO components. They explicitly limit these results to analytical diagnostics rather than end-to-end safety, convergence, or reliable error estimation.

  • A.3 Extended Theoretical Diagnostics: The diagnostics are deliberately narrower than policy-safety or end-to-end convergence guarantees.Table 2 frames its tests as diagnostics of stated results, not validation of an end-to-end safety guarantee.
  • A.3 Extended Theoretical Diagnostics: The moving strike K = µc + c0σc admits a distribution-free critic-output interpretation through Cantelli’s inequality.The bound concerns a discrete surrogate over critic outputs rather than realized environmental trajectory costs.
  • A.3 Extended Theoretical Diagnostics: The exact Gaussian exceedance probability is Φ(−c0), but neither surrogate statement bounds environmental cost-limit violations.The Gaussian probability is generally smaller than the Cantelli upper bound.
  • A.3 Extended Theoretical Diagnostics: The idealized spread-surrogate diagnostic has variance contribution bounded without an explicit (1 −α)^−1 factor, but its bias is unquantified.The bias represents mismatch between the Bachelier gradient and the true CVaR gradient.
  • A.3 Extended Theoretical Diagnostics: The displayed trajectory-score comparison does not analyze the implemented clipped, branch-normalized PPO update or its empirical CPPO estimator.The proposition uses population quantiles and an unclipped score estimator.
  • A.3 Extended Theoretical Diagnostics: Increasing κ raises the moving strike and couples α, κ, and β through βeff, weakening raw disagreement shaping when β is fixed.The falling exceedance probability is partly definitional because the event threshold itself moves.
  • A.3 Extended Theoretical Diagnostics: During outward saturation, anti-windup retains the integral state and projection keeps λk+1 within [0, λmax].This proves no integral accumulation over the stated saturated interval, not convergence of the controller or optimization.
  • A.3 Extended Theoretical Diagnostics: Finite-ensemble concentration controls estimation noise only under idealized i.i.d. bounded critic outputs, whereas implemented critics share rollouts and TD targets.The diagnostic does not establish Bayesian uncertainty, OOD detection, or tracking of prediction error or tail-event probability.

B.1 Implementation Details

The implementation standardizes shared training and evaluation protocols while retaining distinct baseline and optimizer configurations. BCPPO trains an ensemble of masked scalar critics, uses PID anti-windup, and discards the ensemble at deployment.

  • Baselines and scope: The comparison includes maintained mean-cost baselines, PPO-based tail-risk methods, and WCSAC, while BCPPO uses disagreement only as a training-time reliability signal.The paper defines frontier membership descriptively and compares finite candidate sets rather than interpolated policies.
  • Ensemble critics: BCPPO trains M = 5 independently initialized scalar critics with shared TD targets and independent 0.8-probability sample masks.The masks perturb each critic’s training data without changing the ordinary scalar TD target.
  • Stability and control: The implementation clips critic-disagreement inputs for numerical stability and freezes the PID integral term when the projected multiplier is saturated.The integral term can unwind after the error reverses.
  • Protocol: The final protocol applies shared per-step cost limits across methods, including 0.65 for Goal1, Button1, and Push1.Other limits are 0.015898692 for Hopper and 3.4 for Ant.
  • Protocol: Main BCPPO settings were fixed before final shared-protocol baseline runs, while later sensitivity sweeps were diagnostic rather than replacement tuning.The paper notes that development included preliminary engineering runs without a registered held-out tuning plan.
  • Computational profile: BCPPO adds ensemble evaluation during training but avoids trajectory sorting, second-order constraint products, and quantile-head losses.Operation counts alone do not establish an elapsed-time advantage.

B.4 Multiplier Activity and Ensemble Diversity Audit

The diagnostics examine when disagreement is active, whether ensemble members are functionally diverse, and whether spread tracks coverage or risk-related signals. Results support a limited coverage-sensitive interpretation but not a general error or safety certificate.

  • Multiplier activity: 14.2% of logged actor updates have positive λ across the 25 final-main runs, with substantial task variation and only 0.1% in Ant.This supports a disagreement penalty conditional on mean-cost control rather than an always-active tail constraint.
  • Ensemble diversity: The 21-checkpoint audit finds mean pairwise critic correlation of 0.917 and normalized effective rank of 0.578 after removing common prediction.The ensembles are therefore not identical copies, although the audit does not establish statistical independence or calibrated epistemic uncertainty.
  • Deployment: Policy-only deployment retains one actor forward pass, whereas policy-plus-ensemble timing is only a conservative training-diagnostic upper bound.The measured policy-only timing differences do not establish that one actor is faster.
  • Diagnostic design: The diagnostic evaluates prediction-error correlation, top-10% error AUC, boundary-hit AUC, and action-distinguishing AUC using per-policy statistics and resampled 95% confidence intervals.Boundary AUC includes only policies containing both event classes.
  • Diagnostic results: Boundary AUC averages 0.868 across seven eligible policies, while mean prediction-error correlation is −0.030 and top-10% error AUC is 0.520.Controlled perturbed actions produce AUC 0.568 with 95% CI [0.528, 0.607].
  • Missing coverage: Structured data removal raises observation-norm OOD AUC from 0.585 to 0.753 and random-projection OOD AUC from 0.571 to 0.670 under the training TD target.The paired gains are 0.168 and 0.099, respectively, while error-correlation gains are null or small.
  • Negative control: At noise multiplier 1.0, structured removal exceeds the noisy control in OOD AUC by 0.126 for observation norm and 0.110 for random projection.The result supports stronger sensitivity to omitted training data than to the tested target-noise model, without a complete epistemic–aleatoric decomposition.

C Training Curves

The training-curve appendix reports periodic return, mean-cost, CVaR, and disagreement diagnostics aggregated across PPO-family runs. Curves use five-seed summaries rather than replacing the fixed final evaluation.

  • Training diagnostics: Figures 3 and 4 group periodic records from 75 internal PPO-family runs using five-seed means and standard-error bands.Figure 4(b) applies a 25-record per-run rolling mean to BCPPO disagreement.

D.1 Component and Sensitivity Results

Component ablations indicate localized, non-uniform effects, while sensitivity analyses show that several parameters alter one effective coefficient and that ensemble size changes the spread signal and variance.

  • Component ablations: Removing anti-windup increases Button1 mean CVaR@95 from 0.41 to 0.67.The ablation evidence is suggestive rather than statistically established and is not uniform across all three tasks.
  • Component ablations: Moving disagreement to the reward branch increases Push1 mean CVaR@95 from 0.52 to 0.84.This is one of three localized design effects reported by the ablations.
  • Component ablations: Removing branch normalization yields Push1 CVaR@95 standard deviation 1.48 across seeds.The reported effects do not establish a uniform component benefit across Button1, Push1, and Hopper.
  • No-penalty comparison: The no-penalty comparison shows reward–risk trade-offs: Button1 gains 1.30 mean return while CVaR rises 0.058, whereas Push1 CVaR falls 0.035 while return falls 0.650.Hopper improves on both means, but seed variation is large.
  • Sensitivity: β, α, and κ all change βeff = βh(Φ−1(α) + κ), so their sweeps test coefficient sensitivity rather than three independent mechanisms.The defaults are β = 0.15, κ = 0.15, α = 0.95, and M = 5.
  • Sensitivity: On Push1, M = 1 leaves spread at its numerical floor and has the largest reported CVaR variance, while M = 10 offers no clear gain over M = 5.The shared default is not shown to be uniquely optimal or best for every task.
  • Scale control: The raw-σc diagnostic is approximately 69.2× stronger at the same β, whereas setting βraw = βh(c0) makes it algebraically identical to the Bachelier penalty.Therefore the unmatched-scale table does not establish empirical superiority.
Loading 2608.30283v1…