PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantifying Risk Under Evolving Uncertainty: Belief-Dependent Robustness for Safe Sequential Decision Making

Deep Kumar Ganguly, Jan Kretinsky

arXiv:2608.17574v1cs.AI

TL;DR

Agents learning unknown environments must balance immediate safety against eventual efficiency. RATTL adapts Wasserstein robustness to Bayesian belief entropy, and proves value bracketing and convergence toward best response as beliefs concentrate.

  • Problem

    Agents acting while identifying unknown environments need risk tolerance that decreases as epistemic uncertainty resolves.

  • Method

    RATTL ties a Wasserstein ambiguity radius to Bayesian belief uncertainty, using EVaR duality to parameterize adaptive conservatism.

  • Results

    RATTL brackets value between worst-case and Bayesian-best-response references and converges almost surely to best response under persistent identification.

  • Takeaways & Limitations

    Belief can serve as an interpretable risk dial, selecting a position on the coherent-risk spectrum as the environment becomes identified.

  • Takeaways & Limitations

    The exact algorithm is tabular, lacks large-scale empirical validation, and requires reachability and persistent identification or excitation conditions.

Abstract

from arXiv · show

How cautious should an agent be while it is still learning its environment? We propose RATTL (Risk-Adversarial Total-Reward Learning), which ties caution to epistemic uncertainty: the agent holds a Bayesian posterior over unknown dynamics and plans against a Wasserstein ambiguity set whose radius is a monotone function of that posterior. The radius contracts with evidence, so behaviour interpolates continuously between worst-case robustness and risk-neutral total-reward maximization. The design follows the duality underlying the Entropic Value-at-Risk, which converts the choice of a risk level into the choice of an ambiguity radius. We show the resulting planning problem is well posed under transience and compactness conditions, and prove a Safety Sandwich: the RATTL value lies between the uninformed robust value and the full- knowledge optimum, with a gap that vanishes as the posterior concentrates. In a canonical binary-hazard instance, the induced criterion reduces to Conditional Value-at-Risk at a level set by the posterior entropy. A worked example shows the agent deferring the efficient action until a sharp identification threshold. RATTL targets runtime safety for agents, including LLM-based systems, acting under uncertainty.

1 The Problem: Risk That Changes as You Learn

RATTL addresses how an agent should adjust tolerated risk while identifying an uncertain environment and committing to actions in real time. It formalizes this adaptation through belief-dependent Wasserstein robustness and establishes guarantees for the resulting decision process.

  • The Problem: Risk That Changes as You Learn: An agent must choose actions while updating beliefs about hidden environmental parameters through observation, creating a changing risk-tolerance problem.The problem is framed for autonomous systems acting before uncertainty is fully resolved, including self-driving cars, medical AI, and LLM-based agents.
  • The Problem: Risk That Changes as You Learn: RATTL reduces a partially-observed turn-based stochastic game to a subjective robust MDP whose Wasserstein ambiguity radius equals Bayesian belief entropy.The radius is tied to the Shannon entropy of the agent’s belief.
  • The Problem: Risk That Changes as You Learn: The framework provides contractivity, a Safety Sandwich, and convergence to best response under precisely specified adversarially checked conditions.These guarantees are established in §4.

2 The Risk Spectrum and Why EVaR

EVaR provides a coherent, single-parameter risk spectrum from expectation to worst-case risk and an exact KL-DRO dual. This dual turns the conservatism choice into an ambiguity-radius choice, motivating uncertainty-dependent robustness while RATTL uses Wasserstein sets operationally.

  • Risk spectrum: EVaR orders between CVaR and essential supremum, forming a coherent hierarchy of increasing conservatism.For costs X, E[X] ≤ CVaR_α(X) ≤ EVaR_α(X) ≤ ess sup(X).
  • Risk spectrum: EVaR covers the full spectrum from expectation at α=1 to essential supremum as α→0+.It is a single-parameter coherent family with EVaR_1 = E and EVaR_α → ess sup as α → 0+.
  • DRO interpretation: The EVaR dual makes confidence-level selection equivalent to choosing a KL-ball radius −ln α around the nominal distribution.This equivalence reframes conservatism as ambiguity-set size, which can be tied to epistemic uncertainty.
  • DRO interpretation: EVaR supplies the conceptual bridge, while RATTL uses Wasserstein ambiguity sets for safety reasons developed later.The operational ambiguity sets are not KL-based; the section distinguishes EVaR’s motivating dual from RATTL’s implementation.

3 RATTL: Belief Entropy as a Risk Dial

RATTL models a partially observed stochastic game with Bayesian beliefs over opponent types and converts belief entropy into a Wasserstein ambiguity radius. Its robust Bellman game couples adversarial transition choices with belief updates, under properness, uniform reachability, identifiability, and bounded-reward conditions.

  • Setting: RATTL considers a finite partially observed turn-based stochastic game with terminal states, type-dependent transition kernels, bounded rewards, and an undiscounted total-reward criterion.The agent maintains a Bayesian belief b ∈∆(Z) over opponent types.
  • Entropy-modulated ambiguity: The ambiguity radius ε(b) = βH(b) makes conservatism increase with belief entropy, ranging from effectively worst-case for diffuse beliefs to effectively risk-neutral for sharp beliefs.The ambiguity set uses the 1-Wasserstein distance around the belief-weighted transition kernel.
  • Entropy-modulated ambiguity: As belief entropy decreases, the Wasserstein ambiguity ball shrinks and its center moves, reducing Nature’s ability to push probability mass toward catastrophic states.Figure 2 illustrates ambiguity balls for beliefs with decreasing entropy.
  • Robust Bellman game: The belief update occurs inside the expectation, coupling Nature’s kernel selection with the agent’s future information state in a Max–Min Bellman game.The agent chooses an action while Nature chooses the worst kernel within the current ambiguity budget.
  • Assumptions: The theory assumes almost-sure termination, uniform terminal reachability, distinguishable type dynamics, and bounded rewards.Uniform reachability strengthens almost-sure reachability by requiring terminal attainment within m steps with probability at least η.

4 Theoretical Guarantees

Under transience, compactness, and identification conditions, RATTL planning is contractive, uniquely solvable, and geometrically convergent. Its value is sandwiched between maximally robust and belief-averaged known-type values, converging almost surely to the realized-type optimum as posterior uncertainty vanishes.

  • Contractivity: The Bellman operator is a contraction with modulus ρ = 1 − 1/W < 1, yielding a unique fixed point and geometric value-iteration convergence.Here 1 ≤ w(s) ≤ W := m/η < ∞ under the stated transience assumptions.
  • Safety Sandwich: The Safety Sandwich places RATTL’s value between the frozen-radius maximin value and the belief-averaged known-type optimal value.The lower bound follows from nested Wasserstein ambiguity sets, while the upper bound follows because the nominal kernel is feasible and the mixture-process trajectory laws coincide.
  • Safety Sandwich: A stronger ceiling based on the realized true type is generally false because the true kernel can lie outside the belief-dependent ambiguity set; both bounds coincide as b → δz∗.As the posterior becomes a point mass, the ambiguity radius vanishes and the belief-averaged benchmark converges to VBR(s, z∗).
  • Convergence to best response: Under persistent identification, bt → δz∗ almost surely, the ambiguity set collapses to the true kernel, and V∗(s, bt) → VBR(s, z∗) almost surely.Persistent exploration supplies the excitation needed for Bayesian consistency; additional likelihood separation yields a rate result.
  • Tractability: The inner adversarial optimization is a finite linear program, enabling belief-grid value iteration with |S||G||A| LPs of size O(|S|) per sweep.Particle or variational posteriors preserve tractability, while the continuous-state Lipschitz-critic extension remains open.

5 From KL to Wasserstein: A Coherent-Risk Reading

RATTL uses Wasserstein ambiguity rather than KL because Wasserstein can reach off-support catastrophes at finite distance cost. The resulting risk is coherent and Lipschitz-regularized, reduces exactly to CVaR in the two-state safety instance, and becomes an entropy-controlled transport-CVaR more generally.

  • Motivation: Wasserstein ambiguity can assign finite-cost mass to off-support catastrophes, whereas KL ambiguity cannot, making Wasserstein better suited to tail-risk modeling under concentrated beliefs.In the five-state cliff example, Wasserstein reaches the catastrophe and reports a worst case two orders of magnitude lower than KL.
  • Coherence and Lipschitz-regularization: The Wasserstein worst-case functional is a coherent risk measure over a convex compact scenario set.It satisfies monotonicity, translation equivariance, positive homogeneity, and subadditivity.
  • Coherence and Lipschitz-regularization: Wasserstein risk equals a Lipschitz-regularized expectation obtained through inf-convolution with fλ(s) = mins′(f(s′) + λd(s′, s)).Finite-state transport linear-program duality yields an optimal λ⋆≤Lipd(f).
  • Two-point Wasserstein risk: In the two-state safety instance, the robust loss induced by a Wasserstein ball is exactly a Conditional Value-at-Risk, not an Entropic Value-at-Risk.The matched EVaR level depends on ε without a closed form, while CVaR provides the exact characterization.
  • Entropy-controlled risk: As ε = βH(b), the CVaR tail level θ(b) = q/(q + βH(b)/D) decreases as belief entropy grows.Thus sharper beliefs induce higher tail levels toward risk-neutrality, while greater entropy produces deeper-tail caution.
  • Limitations: For |S| ≥3, the worst case yields a distance-weighted transport-CVaR rather than ordinary CVaR, and no single ε-uniform EVaR-level identity holds.The general transport-CVaR↔EVaR comparison remains an open problem beyond the canonical case.

6 Worked Example: The Ambiguous Bridge

In the Ambiguous Bridge diagnostic, RATTL keeps the agent on the safe CRAWL action until posterior confidence in benign conditions reaches approximately 98%, then switches to SPRINT. The threshold is α*=1081/1100 and well-posedness requires β < D/ln |Z| ≈1.443.

  • Environment: The diagnostic has bridge, goal, and fall states, with SPRINT succeeding only under benign dynamics while CRAWL always reaches the goal.Rewards are −1 for SPRINT, −20 for CRAWL, +100 at the goal, and −1000 at the fall state.
  • Conditions: Well-posedness requires ε(b)≤D, giving β < D/ln |Z|≈1.443.Here D=1 and β=1; β shifts the threshold only when safe and risky actions face different ambiguity.
  • Safety switch: The safety switch occurs at α*=1081/1100≈0.983, where the risky action’s worst-case value overtakes the safe action.The threshold follows from 1100α=1081 after the ambiguity terms cancel.
  • Safety switch: The agent CRAWLs while uncertain and SPRINTs after reaching roughly 98% confidence that conditions are benign.In this symmetric environment, the threshold is independent of β because both actions incur the same per-unit transport penalty.

7 Related Work

RATTL connects robust MDPs, risk-sensitive reinforcement learning, and Bayesian opponent modeling through a belief-dependent, non-stationary Wasserstein radius. Its stated novelty is coupling that radius to Shannon belief entropy and deriving the Safety Sandwich and a two-point CVaR identity.

  • Robust MDPs and risk-sensitive RL: RATTL extends robust MDPs with a belief-dependent, non-stationary ambiguity radius, contrasting with static rectangular uncertainty sets and the RMDP↔game equivalence.Robust MDPs with static rectangular uncertainty sets are identified as RATTL’s ancestors, while RATTL proceeds in the opposite direction to the cited equivalence.
  • Robust MDPs and risk-sensitive RL: RATTL’s risk-sensitive foundation draws on EVaR and CVaR as its coherent-risk backbone.The passage identifies Entropic Value-at-Risk and Conditional Value-at-Risk as the relevant prior risk frameworks.
  • Safety Sandwich: −682.5 and 99 are constant reference bounds, with RATTL’s value remaining between the worst-case floor and full-information ceiling at every belief.As belief sharpens, RATTL’s value rises from Vmaximin toward VBR while remaining no less safe than maximin and no more reckless than the informed optimum.
  • Novelty: The paper claims novelty in coupling a Wasserstein radius to Shannon belief entropy rather than posterior credible-set width.Choi and Li are cited as an example of work using posterior credible-set width instead.
  • Novelty: The paper also claims to derive the Safety Sandwich and a two-point CVaR identity.These results are presented as consequences not previously established in the cited related-work discussion.

8 Limitations, Future Work, and Conclusion

RATTL is currently limited to small tabular belief spaces and requires strong reachability and identification conditions. The paper concludes with value-level safety guarantees while identifying broader risk comparisons, alternative ambiguity constructions, and real LLM-agent validation as future work.

  • Limitations: The exact algorithm discretizes the belief simplex and is feasible only for small |Z|, with no large-scale empirical validation yet.The authors provide a scalable roadmap in §4 but do not report large-scale experiments.
  • Limitations: The guarantees require uniform reachability for contraction and persistent identification or excitation for convergence and its rate, excluding degenerate exploration.These conditions go beyond bare properness.
  • Future Work: Future work includes general transport-CVaR↔EVaR comparisons for |S| ≥3, hybrid Wasserstein+KL ambiguity, and Rényi or Tsallis entropy as alternative information dials.The proposed alternatives test whether the Safety Sandwich extends beyond the current construction.
  • Conclusion: The Safety Sandwich provides a value-level bracket between a worst-case floor and a Bayesian-best-response ceiling, rather than a runtime behavioral guarantee.Turning this bracket into an actionable runtime certificate and validating it on real LLM-agent pipelines is identified as the main empirical goal.
  • Conclusion: The conclusion ties risk attitude to belief entropy through a Wasserstein ambiguity radius and reports contraction, Safety Sandwich, and almost-sure best-response convergence results.The stated results are accompanied by explicit, adversarially checked conditions.

A Full Proof of Theorem 1

The proof establishes that the Bellman operator is a contraction on a complete weighted affine space of bounded value functions. Uniform transience yields a finite weight bound, enabling Banach’s fixed-point theorem to complete the argument.

  • Weighted function space: The proof works on the affine space B_T of bounded functions matching terminal rewards, where the weighted norm separates points and is equivalent to the sup norm off T.B_T is complete, and the Bellman operator maps B_T into itself.
  • Transience bound: W = m/η uniformly bounds the worst-case expected hitting time through the geometric tail bound Pr(τ_T > km) ≤ (1 − η)^k.The argument also notes w(s) ≥ 1 for every non-terminal state.
  • Operator bound: The robust expectation operator is Lipschitz in the weighted norm because infima over ambiguity sets are non-expansive under uniform function perturbations.The proof bounds value differences using |V − V′|(s′) ≤ ∥V − V′∥_w w(s′), with terminal differences equal to zero.
  • Contraction and conclusion: The drift inequality yields ∥TV − TV′∥_w ≤ (1 − 1/W)∥V − V′∥_w, so Banach’s theorem completes the proof.The contraction follows after dividing the drift bound by w(s) ∈ [1, W].

B Full Proof of Theorem 2

The proof establishes the Safety Sandwich by comparing the adaptive and frozen-radius Bellman operators, then bounding every policy’s robust value by its Bayesian mixture and type-specific robust values.

  • Operator properties: Both adaptive and frozen-radius operators are monotone self-maps with unique fixed points, and value iteration converges for each.Theorem 1 applies to both operators because ε_max is a valid radius.
  • Lower bound: Posterior entropy bounds the adaptive radius by ε_max, so the adaptive ambiguity set is contained in the frozen-radius set and V_maximin ≤ V*.The common center yields U(b) ⊆ U_max(b), implying T_maxV ≤ TV and the lower bound by induction and limits.
  • Upper bound: The nominal kernel belongs to the adaptive ambiguity set via zero transport, so each policy’s robust value is bounded above by its nominal value under Bayesian posterior updates.The nominal and mixture processes induce the same trajectory law while maintaining b_t = Pr_mix(z | h_t).
  • Upper bound: For every policy, the Bayesian mixture value is bounded by the type-specific robust values because the policy is feasible in each type-z MDP.Chaining this inequality and taking the supremum over policies yields the upper side of the sandwich using proper minimax SSP theory.

C Statement and Proof of Theorem 3 · D Details for §5

Under identifying visitation and persistent excitation, the posterior concentrates on the true type and the robustness price vanishes at O(log t/t). The §5 ambiguity functional is convex and coherent, admits a finite-LP dual representation, and obeys a one-sided EVaR bound via CVaR ordering.

  • C Statement and Proof of Theorem 3: Under (PI), identifying state-action pairs are visited infinitely often, so the posterior converges almost surely to the true type.The realized log-likelihood ratio against every wrong type diverges to +∞ almost surely.
  • C Statement and Proof of Theorem 3: Under persistent excitation, posterior mass on each wrong type decays geometrically with identifying visits, yielding E[H(b_t)] = O(|Z| log t/t).The result assumes Θ(t) identifying visits.
  • C Statement and Proof of Theorem 3: O(log t/t) is the price of robustness, because both the floor and averaged-ceiling gaps vanish with residual mass on wrong types.The bound follows from |VBR(s, z∗) − V∗(s, b_t)| ≤ C times posterior mass on wrong types.
  • D Details for §5: The ambiguity functional U_ε is convex because convex combinations of optimal couplings preserve the reference marginal and satisfy the Wasserstein triangle bound.For mixtures, W1(θQ0 + (1 − θ)Q1, P̄) ≤ θW1(Q0, P̄) + (1 − θ)W1(Q1, P̄).
  • D Details for §5: U_ε satisfies coherence axioms because it is independent of X: monotonicity and positive homogeneity are termwise, while subadditivity follows from decoupled suprema.Translation-equivariance uses ⟨Q, 1⟩ = 1.
  • D Details for §5: Finite-LP strong duality yields the boxed representation with marginal multipliers u(s) and budget multiplier λ ≥ 0, after optimizing u(s) to obtain f_λ(s).The dual argument uses u(s) = min_s′(f(s′) − λd(s, s′)) = f_λ(s), and CVaR ≤ EVaR gives a one-sided EVaR bound.
Loading 2608.17574v1…