Source-linked AI summary
Certifying Lower Bounds for Risk-Sensitive Reinforcement Learning under Adversarial State Perturbations
Tong Li, Saunak Kumar Panda, Yisha Xiang
TL;DR
RL certification has largely focused on risk-neutral objectives, despite vulnerabilities to adversarial state observations. This paper develops risk-sensitive exponential-utility certificates via convex relaxation and dual optimization, finding that risk-averse training generally improves certified lower bounds but becomes harmful when excessive.
Problem
Existing RL robustness certification mainly targets risk-neutral performance, leaving risk-sensitive guarantees under adversarial observations limited.
Method
The paper extends certification to exponential utility under l_p-bounded perturbations using φ-divergence relaxation, convex optimization, and a tractable dual.
Results
Risk-averse training generally yields higher certified lower bounds than risk-neutral training, especially under larger perturbation budgets, across the reported experiments.
Takeaways & Limitations
Training risk aversion should be selected carefully because certification performance is non-monotonic and eventually declines for overly conservative policies.
Takeaways & Limitations
The tractable certification analysis assumes adversarial perturbations occur only at the initial state to control trajectory-distribution divergence.
Abstract
from arXiv · showhide
Reinforcement learning (RL) agents deployed in real-world environments are often vulnerable to adversarial perturbations in state observations, creating risks in safety-critical applications. Certification methods can improve robustness against adversarial perturbations by providing lower bounds on expected cumulative rewards. Existing certification methods, however, mainly focus on risk-neutral objectives. In this paper, we extend certification methods to risk-sensitive objectives by establishing lower bounds on the exponential utility of cumulative rewards under $l_{p}$-norm-bounded state adversarial perturbations ($1\leq p <\infty$). By introducing a $φ$-divergence relaxation of the perturbation set, we formulate the risk-sensitive certification problem as a convex optimization and derive its dual to obtain a tractable approximation of the certified lower bound. We further propose an empirical method that improves certified lower bounds by selecting the training risk-aversion parameter $β$ independently of the risk level used during evaluation. Experiments on both OpenAI Gym environments and a machine replacement problem show that, compared to risk-neutral training, risk-averse training generally yields policies with higher certified lower bounds, particularly under larger perturbation budgets. Moreover, under both risk-neutral and risk-averse evaluation settings, increasing risk aversion during training leads to non-monotonic certification performance, where certified lower bounds initially improve but eventually decrease due to overly conservative policies.
1 Introduction
RL agents can be highly sensitive to small state perturbations, motivating certified guarantees for sequential decisions. This paper extends certification to risk-sensitive objectives and reports improved but non-monotonic certified lower bounds from risk-averse training.
- Small observation perturbations can substantially alter RL behavior, creating reliability concerns in safety-sensitive applications.
- Certification is needed because empirical defenses can face adaptive attacks, while RL certification must handle sequential decision-making.
- Risk-sensitive certification extends robustness guarantees from risk-neutral objectives to exponential utility under bounded adversarial state perturbations.The formulation uses a φ-divergence relaxation, convex optimization, and a tractable dual formulation.
- Risk-averse training generally produces higher certified lower bounds than risk-neutral training, especially under larger perturbation budgets.
- Increasing training-time risk aversion initially improves certification but eventually degrades it because policies become overly conservative.
- Experiments evaluate the proposed approach on OpenAI environments and a practical machine replacement problem.
2 Literature Review
Prior RL robustness work includes empirical defenses and certification methods for adversarial perturbations. The paper positions its contribution as extending certification guarantees to risk-sensitive settings with exponential-utility lower bounds.
- Empirical defenses: Empirical defenses include adversarial training and regularization methods that improve robustness to perturbed state observations.Regularizers include KL-divergence, l2-distance, hinge-loss, and local smoothness penalties.
- Robustness certification: CARRL certifies Q-value lower bounds by propagating interval bounds through neural-network layers.
- Robustness certification: CROP certifies per-state actions and cumulative rewards using smoothed Q-functions, global smoothing, and local adaptive search.
- Robustness certification: Kumar et al. certify expected total reward using a single global smoothing mechanism for cumulative-reward robustness.
- Robustness certification: ReCePS formulates direct cumulative-reward certification through f-divergence-based convex reformulation and dual optimization.
- Contribution: This paper extends certification guarantees to exponential utility and finds that risk-averse training generally improves certificates under stronger perturbations, with degradation at excessive risk aversion.
3 Problem Formulation
The paper models risk-sensitive RL in finite-horizon MDPs with stochastic observations and adversarial state perturbations. It smooths policies with Gaussian noise, formulates worst-case exponential-utility certification, and reformulates the problem through stochastic observations for tractability.
- Problem Formulation: The framework seeks worst-case performance guarantees for policies under bounded adversarial state perturbations in an MDP.
- 3.1 Preliminaries: A finite-horizon MDP is defined by states, actions, transitions, rewards, horizon, and discounting, with policies mapping states to action distributions.
- 3.1 Preliminaries: Risk-sensitive RL evaluates exponential utility of cumulative rewards; negative β represents risk aversion, and larger |β| increases risk sensitivity.
- 3.1 Preliminaries: The smoothed policy injects Gaussian noise into states and selects actions by maximizing expected Q-values over perturbed inputs.
- Risk-sensitive RL with smoothed policy.: Adversarial observations add perturbations δ_t to states, with the full perturbation sequence constrained by an l_p-norm budget for 1 ≤ p < ∞.
- Risk-sensitive RL with smoothed policy.: The certificate minimizes exponential utility over admissible perturbations for the smoothed policy and its induced perturbed trajectory distribution.
- Risk-sensitive RL with stochastic observations.: Gaussian state smoothing is equivalently represented by stochastic observations sampled from N(s_t, σ^2I), with actions selected from those observations.
- Risk-sensitive RL with stochastic observations.: The observation-based formulation represents trajectories through states, observations, actions, transitions, and policy probabilities before convex reformulation.
4 Reformulation of the lower bound certification
The paper replaces the difficult risk-sensitive certification problem with a tractable convex relaxation based on φ-divergence and derives an equivalent finite-dimensional dual program. It then connects norm-bounded state perturbations to suitable divergence thresholds and estimates the certificate with Monte Carlo sampling and confidence bounds.
- The φ-divergence relaxation contains the original perturbation set and converts the potentially non-convex, infinite-dimensional certification problem into a tractable convex formulation.The relaxed set is chosen so Dϵ ⊆ ˜Dϵ′, with ϵ′ determined by the perturbation budget and divergence choice.
- Theorem 1 equivalently reformulates risk-sensitive certification as a finite-dimensional convex program over ξ > 0 and η ∈ R using the conjugate ϕ∗.This dual formulation applies for any choice of ϕ-divergence and replaces optimization over trajectory distributions.
- The relaxation is calibrated to l_p-norm perturbations by selecting divergence functions and thresholds that guarantee Dϵ ⊆ ˜Dϵ′.The construction uses Gaussian-smoothed initial observations and, under an initial-state perturbation assumption, reduces trajectory divergence to observation-distribution divergence.
- For l_1-bounded perturbations, total variation provides a relaxation linked to the perturbation budget, while related constructions cover l_2 and other norm settings.The passages identify TV for the l_1 case and a provably tight relaxation for l_2 perturbations.
- The algorithm approximates the dual expectation with M Monte Carlo trajectories and uses an upper confidence bound to return a computationally tractable certified lower bound.The confidence-bound estimate is used conservatively so the resulting certificate remains a lower bound with high probability.
5 Numerical Study
Experiments evaluate certified lower bounds across OpenAI Gym environments and a machine replacement problem under varied training and testing risk aversion. Risk-averse training generally improves certification, especially at larger perturbation budgets, but excessive aversion produces non-monotonic degradation.
- Lunar Lander: In Lunar Lander, certified lower bounds are non-increasing as the perturbation budget increases under both l2- and l1-norm perturbations.Figure 1 covers risk-neutral and risk-averse testing environments across the two perturbation norms.
- Overall findings: Risk-averse training generally achieves higher certified lower bounds than risk-neutral training, particularly under larger perturbation budgets, across the evaluated environments.The study compares Lunar Lander, CartPole, and a machine replacement problem under risk-neutral and risk-averse testing.
- Training risk aversion: Certified lower bounds initially improve as training risk aversion increases, then decline at βtrain = −0.8 because policies become overly conservative.This non-monotonic pattern appears under both risk-neutral and risk-averse testing conditions.
- CartPole: CartPole exhibits trends similar to Lunar Lander, supporting consistency of the certification findings across the two OpenAI Gym environments.The comparison is presented in Figure 2 under the corresponding testing and perturbation settings.
- Machine replacement problem: In the machine replacement problem, risk-averse training lowers the maintenance threshold, prompting earlier repairs and improving robustness under perturbed observations.The optimal policy repairs when the discretized degradation level reaches a threshold, and risk-averse training tends to reduce that threshold.
6 Conclusion
The paper proposes risk-sensitive certification for RL under adversarial state perturbations and evaluates how training risk aversion affects certified robustness. Across the reported experiments, risk-averse training generally improves certification initially, while excessive aversion degrades it.
- Conclusion: The proposed framework certifies exponential-utility returns under lp-norm-bounded state perturbations using a ϕ-divergence relaxation and a tractable dual formulation.The framework is evaluated on OpenAI Gym environments and a machine replacement task.
- Conclusion: Risk-averse training generally yields higher certified lower bounds than risk-neutral training, especially under larger perturbation budgets.The comparison is reported across the paper’s experimental settings.
- Conclusion: Increasing training-time risk aversion initially improves certified robustness but eventually degrades certification because overly conservative policies emerge.This pattern holds under both risk-neutral and risk-averse evaluation settings.
Appendix A Equivalence Between Trajectory Distribution Divergence and Observation Distribution Divergence
The appendix reduces trajectory-distribution divergence to divergence between initial observation distributions when only the initial state is perturbed. This reduction supports the certification formulation’s tractable observation-level analysis.
- Appendix A: With only the initial state perturbed, trajectory distributions are reduced to the corresponding initial observation distributions.The trajectory includes states, observations, and actions, while the reduction retains only the initial observation.
Appendix B Proof of Proposition 1
The proof analyzes total variation divergence between shifted Gaussian observation distributions by standardizing them and identifying the density-difference half-space. This yields the divergence relationship used for the l1-norm perturbation relaxation.
- Appendix B: The proof shows that TV divergence between shifted Gaussian distributions equals the TV divergence between their standardized versions.Standardization uses the normalized perturbation δ/σ and compares N(δ̃, Id) with N(0, Id).
- Appendix B: The sign of the standardized density difference defines a half-space used to compute the two distributions’ TV divergence.The proof evaluates probabilities of that half-space under the shifted and unshifted standardized Gaussian distributions.