PaperlayerView exact revision on arXiv

Source-linked AI summary

Certifying Lower Bounds for Risk-Sensitive Reinforcement Learning under Adversarial State Perturbations

Tong Li, Saunak Kumar Panda, Yisha Xiang

arXiv:2609.10866v1cs.LGmath.OC

TL;DR

RL certification has largely focused on risk-neutral objectives, despite vulnerabilities to adversarial state observations. This paper develops risk-sensitive exponential-utility certificates via convex relaxation and dual optimization, finding that risk-averse training generally improves certified lower bounds but becomes harmful when excessive.

  • Problem

    Existing RL robustness certification mainly targets risk-neutral performance, leaving risk-sensitive guarantees under adversarial observations limited.

    Source: §1 · p. 2

  • Method

    The paper extends certification to exponential utility under l_p-bounded perturbations using φ-divergence relaxation, convex optimization, and a tractable dual.

    Source: §1 · p. 3, Risk-sensitive RL with smoothed policy · p. 7, Risk-sensitive RL with stochastic… · p. 8

  • Results

    Risk-averse training generally yields higher certified lower bounds than risk-neutral training, especially under larger perturbation budgets, across the reported experiments.

    Source: §1 · p. 3, §5.1 · pp. 17, 19, §6 · p. 20

  • Takeaways & Limitations

    Training risk aversion should be selected carefully because certification performance is non-monotonic and eventually declines for overly conservative policies.

    Source: §1 · p. 3, §6 · p. 22

    The tractable certification analysis assumes adversarial perturbations occur only at the initial state to control trajectory-distribution divergence.

    Source: §4.1 · p. 11

Abstract

from arXiv · show

Reinforcement learning (RL) agents deployed in real-world environments are often vulnerable to adversarial perturbations in state observations, creating risks in safety-critical applications. Certification methods can improve robustness against adversarial perturbations by providing lower bounds on expected cumulative rewards. Existing certification methods, however, mainly focus on risk-neutral objectives. In this paper, we extend certification methods to risk-sensitive objectives by establishing lower bounds on the exponential utility of cumulative rewards under $l_{p}$-norm-bounded state adversarial perturbations ($1\leq p <\infty$). By introducing a $φ$-divergence relaxation of the perturbation set, we formulate the risk-sensitive certification problem as a convex optimization and derive its dual to obtain a tractable approximation of the certified lower bound. We further propose an empirical method that improves certified lower bounds by selecting the training risk-aversion parameter $β$ independently of the risk level used during evaluation. Experiments on both OpenAI Gym environments and a machine replacement problem show that, compared to risk-neutral training, risk-averse training generally yields policies with higher certified lower bounds, particularly under larger perturbation budgets. Moreover, under both risk-neutral and risk-averse evaluation settings, increasing risk aversion during training leads to non-monotonic certification performance, where certified lower bounds initially improve but eventually decrease due to overly conservative policies.

1 Introduction

RL agents can be highly sensitive to small state perturbations, motivating certified guarantees for sequential decisions. This paper extends certification to risk-sensitive objectives and reports improved but non-monotonic certified lower bounds from risk-averse training.

  • Small observation perturbations can substantially alter RL behavior, creating reliability concerns in safety-sensitive applications.Source: §1 · p. 2
  • Certification is needed because empirical defenses can face adaptive attacks, while RL certification must handle sequential decision-making.Source: §1 · p. 2
  • Risk-sensitive certification extends robustness guarantees from risk-neutral objectives to exponential utility under bounded adversarial state perturbations.The formulation uses a φ-divergence relaxation, convex optimization, and a tractable dual formulation.Source: §1 · p. 3
  • Risk-averse training generally produces higher certified lower bounds than risk-neutral training, especially under larger perturbation budgets.Source: §1 · p. 3
  • Increasing training-time risk aversion initially improves certification but eventually degrades it because policies become overly conservative.Source: §1 · p. 3
  • Experiments evaluate the proposed approach on OpenAI environments and a practical machine replacement problem.Source: §1 · p. 3

2 Literature Review

Prior RL robustness work includes empirical defenses and certification methods for adversarial perturbations. The paper positions its contribution as extending certification guarantees to risk-sensitive settings with exponential-utility lower bounds.

Empirical defenses

  • Empirical defenses include adversarial training and regularization methods that improve robustness to perturbed state observations.Regularizers include KL-divergence, l2-distance, hinge-loss, and local smoothness penalties.Source: §2.1 · pp. 3–4

Robustness certification

  • CARRL certifies Q-value lower bounds by propagating interval bounds through neural-network layers.Source: §2.2 · p. 4
  • CROP certifies per-state actions and cumulative rewards using smoothed Q-functions, global smoothing, and local adaptive search.Source: §2.2 · pp. 4–5
  • Kumar et al. certify expected total reward using a single global smoothing mechanism for cumulative-reward robustness.Source: §2.2 · p. 5
  • ReCePS formulates direct cumulative-reward certification through f-divergence-based convex reformulation and dual optimization.Source: §2.2 · p. 5

Contribution

  • This paper extends certification guarantees to exponential utility and finds that risk-averse training generally improves certificates under stronger perturbations, with degradation at excessive risk aversion.Source: §2.2 · p. 5

3 Problem Formulation

The paper models risk-sensitive RL in finite-horizon MDPs with stochastic observations and adversarial state perturbations. It smooths policies with Gaussian noise, formulates worst-case exponential-utility certification, and reformulates the problem through stochastic observations for tractability.

Problem Formulation

3.1 Preliminaries

  • A finite-horizon MDP is defined by states, actions, transitions, rewards, horizon, and discounting, with policies mapping states to action distributions.Source: §3.1 · p. 6
  • Risk-sensitive RL evaluates exponential utility of cumulative rewards; negative β represents risk aversion, and larger |β| increases risk sensitivity.Source: §3.1 · pp. 6–7
  • The smoothed policy injects Gaussian noise into states and selects actions by maximizing expected Q-values over perturbed inputs.Source: §3.1 · p. 7

Risk-sensitive RL with smoothed policy.

Risk-sensitive RL with stochastic observations.

4 Reformulation of the lower bound certification

The paper replaces the difficult risk-sensitive certification problem with a tractable convex relaxation based on φ-divergence and derives an equivalent finite-dimensional dual program. It then connects norm-bounded state perturbations to suitable divergence thresholds and estimates the certificate with Monte Carlo sampling and confidence bounds.

  • The φ-divergence relaxation contains the original perturbation set and converts the potentially non-convex, infinite-dimensional certification problem into a tractable convex formulation.The relaxed set is chosen so Dϵ ⊆ ˜Dϵ′, with ϵ′ determined by the perturbation budget and divergence choice.Source: §4.1 · p. 10
  • Theorem 1 equivalently reformulates risk-sensitive certification as a finite-dimensional convex program over ξ > 0 and η ∈ R using the conjugate ϕ∗.This dual formulation applies for any choice of ϕ-divergence and replaces optimization over trajectory distributions.Source: §4.1 · pp. 10–11
  • The relaxation is calibrated to l_p-norm perturbations by selecting divergence functions and thresholds that guarantee Dϵ ⊆ ˜Dϵ′.The construction uses Gaussian-smoothed initial observations and, under an initial-state perturbation assumption, reduces trajectory divergence to observation-distribution divergence.Source: §4.1 · pp. 11–12
  • For l_1-bounded perturbations, total variation provides a relaxation linked to the perturbation budget, while related constructions cover l_2 and other norm settings.The passages identify TV for the l_1 case and a provably tight relaxation for l_2 perturbations.Source: §4.1 · pp. 10, 12
  • The algorithm approximates the dual expectation with M Monte Carlo trajectories and uses an upper confidence bound to return a computationally tractable certified lower bound.The confidence-bound estimate is used conservatively so the resulting certificate remains a lower bound with high probability.Source: §4.2 · pp. 13–14

5 Numerical Study

Experiments evaluate certified lower bounds across OpenAI Gym environments and a machine replacement problem under varied training and testing risk aversion. Risk-averse training generally improves certification, especially at larger perturbation budgets, but excessive aversion produces non-monotonic degradation.

Lunar Lander

  • In Lunar Lander, certified lower bounds are non-increasing as the perturbation budget increases under both l2- and l1-norm perturbations.Figure 1 covers risk-neutral and risk-averse testing environments across the two perturbation norms.Source: Figure 1 · p. 16, §5.1 · p. 17

Overall findings

  • Risk-averse training generally achieves higher certified lower bounds than risk-neutral training, particularly under larger perturbation budgets, across the evaluated environments.The study compares Lunar Lander, CartPole, and a machine replacement problem under risk-neutral and risk-averse testing.Source: §5.1 · p. 17, §5.2 · p. 20

Training risk aversion

  • Certified lower bounds initially improve as training risk aversion increases, then decline at βtrain = −0.8 because policies become overly conservative.This non-monotonic pattern appears under both risk-neutral and risk-averse testing conditions.Source: §5.1 · p. 17, §5.2 · p. 20

CartPole

  • CartPole exhibits trends similar to Lunar Lander, supporting consistency of the certification findings across the two OpenAI Gym environments.The comparison is presented in Figure 2 under the corresponding testing and perturbation settings.Source: Figure 2 · p. 18, §5.1 · p. 19

Machine replacement problem

  • In the machine replacement problem, risk-averse training lowers the maintenance threshold, prompting earlier repairs and improving robustness under perturbed observations.The optimal policy repairs when the discretized degradation level reaches a threshold, and risk-averse training tends to reduce that threshold.Source: §5.2 · pp. 19–20

6 Conclusion

The paper proposes risk-sensitive certification for RL under adversarial state perturbations and evaluates how training risk aversion affects certified robustness. Across the reported experiments, risk-averse training generally improves certification initially, while excessive aversion degrades it.

Conclusion

  • The proposed framework certifies exponential-utility returns under lp-norm-bounded state perturbations using a ϕ-divergence relaxation and a tractable dual formulation.The framework is evaluated on OpenAI Gym environments and a machine replacement task.Source: §6 · p. 20
  • Risk-averse training generally yields higher certified lower bounds than risk-neutral training, especially under larger perturbation budgets.The comparison is reported across the paper’s experimental settings.Source: §6 · p. 20
  • Increasing training-time risk aversion initially improves certified robustness but eventually degrades certification because overly conservative policies emerge.This pattern holds under both risk-neutral and risk-averse evaluation settings.Source: §6 · pp. 20, 22

Appendix A Equivalence Between Trajectory Distribution Divergence and Observation Distribution Divergence

The appendix reduces trajectory-distribution divergence to divergence between initial observation distributions when only the initial state is perturbed. This reduction supports the certification formulation’s tractable observation-level analysis.

Appendix A

  • With only the initial state perturbed, trajectory distributions are reduced to the corresponding initial observation distributions.The trajectory includes states, observations, and actions, while the reduction retains only the initial observation.Source: Appendix A · p. 22

Appendix B Proof of Proposition 1

The proof analyzes total variation divergence between shifted Gaussian observation distributions by standardizing them and identifying the density-difference half-space. This yields the divergence relationship used for the l1-norm perturbation relaxation.

Appendix B

  • The proof shows that TV divergence between shifted Gaussian distributions equals the TV divergence between their standardized versions.Standardization uses the normalized perturbation δ/σ and compares N(δ̃, Id) with N(0, Id).Source: Appendix B · pp. 22–23
  • The sign of the standardized density difference defines a half-space used to compute the two distributions’ TV divergence.The proof evaluates probabilities of that half-space under the shifted and unshifted standardized Gaussian distributions.Source: Appendix B · p. 23

Related papers

Loading 2609.10866v1…