Policy Gradient for Coherent Risk Measures

Aviv Tamar; Yinlam Chow; Mohammad Ghavamzadeh; Shie Mannor

Source-linked AI summary

Policy Gradient for Coherent Risk Measures

Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, Shie Mannor

arXiv:1502.03919v2Feb 13, 2015cs.AIcs.LGstat.ML

TL;DR

Risk-sensitive reinforcement learning needs methods that handle coherent risk measures beyond specific choices such as variance or CVaR. This paper develops unified policy-gradient algorithms for static and time-consistent dynamic coherent risk, using sampling with convex programming and value-function-based methods. The approach generalizes previous results and provides consistent static gradient estimation under stated conditions.

Problem
Existing risk-sensitive reinforcement-learning methods focus on particular risk measures, while the paper seeks a unified treatment of the broader coherent-risk class for static and dynamic settings.
Method
The paper combines sampling with convex programming for static coherent-risk gradients and uses a value-function-based policy-gradient theorem for dynamic Markov coherent risk.
Results
The framework extends risk-sensitive reinforcement learning to the whole class of coherent risk measures and generalizes several studies focused on specific measures.
Takeaways & Limitations
Coherent-risk policy gradients provide a unified approach spanning static and time-consistent dynamic risk-sensitive reinforcement learning.
Takeaways & Limitations
The paper identifies improving gradient-estimate convergence with importance sampling as future work, especially for rare-event-sensitive criteria such as CVaR.

Abstract

from arXiv · show

Several authors have recently developed risk-sensitive policy gradient methods that augment the standard expected cost minimization problem with a measure of variability in cost. These studies have focused on specific risk-measures, such as the variance or conditional value at risk (CVaR). In this work, we extend the policy gradient method to the whole class of coherent risk measures, which is widely accepted in finance and operations research, among other fields. We consider both static and time-consistent dynamic risk measures. For static risk measures, our approach is in the spirit of policy gradient algorithms and combines a standard sampling approach with convex programming. For dynamic risk measures, our approach is actor-critic style and involves explicit approximation of value function. Most importantly, our contribution presents a unified approach to risk-sensitive reinforcement learning that generalizes and extends previous results.

1 Introduction

The paper develops a unified policy-gradient framework for coherent risk measures in reinforcement learning, covering both static and time-consistent dynamic settings. It generalizes methods built for individual risk measures through sampling, convex programming, and value-function approximation.

Motivation: Coherent risk measures provide a broad framework for managing cost variability in reinforcement learning, beyond expected-cost objectives and specific measures such as variance or CVaR.Their preference depends on the cost distribution, rare-event sensitivity, estimation, and computational tractability; coherent measures have widespread acceptance in finance and related fields.
Motivation: Time consistency requires that a policy optimal over n stages remains optimal from stage t onward, motivating dynamic Markov coherent risk measures for sequential decisions.Dynamic Markov coherent risk measures satisfy both coherence and time consistency.
Contributions: The paper formulates policy-gradient algorithms for the whole class of coherent risks, including static risk of total discounted return and time-consistent dynamic Markov risk.This extends and unifies previous approaches focused on individual risk measures.
Contributions: For static coherent risk, the method combines sampling with convex programming and provides a gradient formula, a general estimation algorithm, and a consistency result.The result supports estimating gradients when the risk-envelope saddle point is not analytically available.
Contributions: For dynamic Markov coherent risk, the paper derives a policy-gradient theorem involving a suitable value function and develops a corresponding actor-critic algorithm.The approach is intended to support policy and value-function approximation in large or continuous MDPs.

2 Preliminaries

The preliminaries define coherent risk through risk envelopes and distinguish static risk from dynamic Markov coherent risk. The framework assumes convex, compact risk envelopes with smooth constraints and applies them to discounted MDP costs.

MDP Setting: The MDP consists of state and action spaces, bounded costs, transition probabilities, a discount factor, an initial state, and a parameterized stationary Markov policy.The paper notes that dynamic Markov risk admits stationary Markov optimal policies, unlike static risk in general.
Coherent Risk Measures: A coherent risk measure is represented as an expectation under a worst-case density selected from a convex risk envelope.The risk envelope uniquely represents the coherent risk measure and supports the paper’s gradient-based approach.
Coherent Risk Measures: The paper assumes each risk envelope has affine equality constraints, convex inequality constraints, and sufficient smoothness in the policy parameter.These conditions cover popular coherent measures including CVaR, mean-semideviation, and spectral risk measures.
Dynamic Risk Measures: Static risk measures evaluate uncertain outcomes without temporal structure, whereas dynamic risk measures account for the temporal evolution of trajectory outcomes.Ignoring temporal structure in multi-period decisions can produce time-inconsistent behavior.
Dynamic Risk Measures: Markov coherent risk measures apply a static coherent risk at each state using the policy-induced transition distribution and define an infinite-horizon risk through a discounted limit.The construction assumes bounded costs and γ < 1, with each statewise risk evaluation independent of the whole past.

3 Problem Formulation

The paper studies static and dynamic risk-sensitive optimization when objective gradients are analytically unavailable or state spaces are too large for exact computation. It estimates gradients from samples and uses them in stochastic gradient descent to seek locally optimal policies.

Problem Setting: The static problem optimizes a coherent risk of a random variable, while the dynamic problem optimizes a Markov coherent risk for an MDP.In reinforcement learning, the static random variable can be the cumulative discounted trajectory cost.
Problem Setting: The central problem is estimating gradients of static and dynamic coherent-risk objectives when their dependence on policy parameters is complex and non-convex.The goal is local optimization rather than solving the generally intractable problems globally.
Sampling Approach: The sampling assumptions provide i.i.d. samples of the static random variable or next states for each MDP state-action pair.These samples enable estimation when analytical gradients or tractable dynamic programming are unavailable.
Optimization: Estimated gradients are inserted into standard stochastic gradient descent to learn locally optimal solutions for both static and dynamic risk problems.The dynamic estimator is motivated by first estimating the gradient for static risk and extending the idea to dynamic risk.

4 Gradient Formula for Static Risk

The static-risk section derives sampling-based policy-gradient estimators for coherent risk measures, using risk-envelope structure and convex programming. It recovers specialized formulas for CVaR and mean-semideviation and establishes consistency for the general estimator.

4 Gradient Formula for Static Risk: The method assumes bounded, well-defined likelihood-ratio gradients for policy probabilities, a standard requirement in policy-gradient applications.The paper notes this requirement is satisfied in applications including queueing, inventory management, and financial engineering.
4 Gradient Formula for Static Risk: Coherent-risk structure represents risk through a convex risk envelope, making the risk optimization a convex program with saddle points.This structure is the basis for the static gradient formula and its sampling implementation.
4 Gradient Formula for Static Risk: Theorem 4.2 provides a likelihood-ratio gradient formula at a Lagrangian saddle point, enabling sampling-based estimation for static coherent risk.The derivation uses the envelope theorem and the likelihood-ratio trick.
4.1 Example 1: CVaR: For CVaR, the resulting formula remains valid for discrete distributions and without the additional smoothness assumptions used in earlier work.The paper also gives a simpler proof than the cited prior result.
4.2 Example 2: Mean-Semideviation: Mean-semideviation measures variability above the mean and yields a gradient combining the expected-cost gradient with a semideviation correction.Replacing expectations by sample averages gives a sampling-based estimator, illustrated later numerically.
4.3 General Gradient Estimation Algorithm: The general algorithm uses N i.i.d. samples, constructs an empirical risk envelope, and solves a convex problem with O(N) variables and constraints.KKT multipliers from the convex program are then used in the gradient estimator.
4.3 General Gradient Estimation Algorithm: Sampling plus convex programming yields a consistent estimator of both the coherent risk and its policy gradient under stated regularity conditions.The estimator converges almost surely, and the conditions cover CVaR, mean-semideviation, and spectral risk measures.

5 Gradient Formula for Dynamic Risk

The dynamic-risk section extends policy-gradient reasoning to Markov coherent risk through a value-function gradient and an actor-critic procedure. The critic uses function approximation, while the actor estimates risk-weighted transitions and gradients by two-phase sampling.

5 Gradient Formula for Dynamic Risk: The dynamic coherent-risk gradient is derived by combining the static gradient formula with a dynamic-programming decomposition.This produces a formula for ∇θVθ(x), extending the policy-gradient theorem to Markov coherent risk.
5 Gradient Formula for Dynamic Risk: The actor-critic algorithm alternates between estimating the risk-sensitive value function and updating the policy with the resulting gradient estimate.The critic computes Vθ, and the actor uses Theorem 5.2 to estimate ∇θρ∞(M).
5 Gradient Formula for Dynamic Risk: The paper reports that the full actor-critic algorithm, its convergence analysis, and function-approximation gradient error are provided in supplementary material.The main text presents only key ideas and results because of space limitations.
5 Gradient Formula for Dynamic Risk: For large state spaces, the critic estimates Vθ with function approximation by exploiting its equivalence to a robust-MDP value function.This avoids applying dynamic programming directly under the curse of dimensionality.
5 Gradient Formula for Dynamic Risk: The actor uses a two-phase sampling procedure: critic-derived risk-weighted transitions are followed by multiple next-state samples for estimating hθ(x,a).The procedure addresses the need to sample from transitions weighted by ξ*θ.

6 Numerical Illustration

The numerical illustration shows that different risk objectives select different assets from the same return distributions. Risk-sensitive policies favor assets according to downside and tail behavior rather than mean return alone.

6 Numerical Illustration: The experiment is designed to demonstrate why risk criteria should remain flexible to user preferences and problem-specific return distributions.The paper contrasts a risk-neutral objective with mean-semideviation and mean-standard-deviation objectives.
6 Numerical Illustration: The illustration uses three assets with two normally distributed returns and one Pareto-distributed return whose variance is infinite.A1 has distribution N(1,1), A2 has N(4,6), and A3 has mean 3 with Pareto tail parameter α=1.5.
6 Numerical Illustration: Risk-averse policy π2 chooses heavy-tailed asset A3 for its lower downside, whereas policy π3 chooses A1 because A3 has a heavy upper tail.Although A2 has the highest mean return, the two risk-averse objectives produce different selections; A3 stochastically dominates A1.

7 Conclusion

The paper extends risk-sensitive reinforcement learning to coherent risk measures, while highlighting flexibility in risk preference and future challenges in estimation and risk shaping.

7 Conclusion: The approach estimates gradients for both static and dynamic coherent risk measures, extending risk-sensitive RL beyond methods focused on specific measures.The algorithms combine sampling with convex programming and generalize several recent studies.
7 Conclusion: Importance-sampling methods are identified as a future direction for improving gradient-estimate convergence, especially for rare-event-sensitive criteria such as CVaR.
7 Conclusion: The coherent-risk framework gives decision makers flexibility to design risk preferences for problem-specific cost-variability management.The paper states that its numerical example demonstrates the importance of selecting appropriate problem-specific risk measures.
7 Conclusion: Coherent-risk representation connects risk with model uncertainty, but principled risk shaping for handling model misspecification remains beyond the paper’s scope.The paper distinguishes uncertainty from stochastic dynamics and uncertainty from modeling errors.

B Gradient Results for Static Mean-Semideviation

This section derives a sampling-friendly gradient for the static mean-semideviation risk measure and uses it to define the GMSD estimator.

B Gradient Results for Static Mean-Semideviation: An optimal auxiliary variable can be characterized as a contact point of the positive deviation and used in the gradient representation.The contact-point characterization constrains the auxiliary solution by an L2 norm.
B Gradient Results for Static Mean-Semideviation: The resulting gradient formula expresses the terms as expectations, making sampling-based estimation possible.
B Gradient Results for Static Mean-Semideviation: The mean-semideviation gradient combines the expected-cost gradient with a deviation-weighted likelihood-ratio expectation.The formula includes c SD(Z) and the positive deviation (Z − E[Z])+ alongside policy-log-probability terms.
B Gradient Results for Static Mean-Semideviation: The derivation relies on the envelope theorem because the optimization constraints do not depend on the policy parameter.
B Gradient Results for Static Mean-Semideviation: The paper names the resulting sampling-based gradient estimator GMSD and describes it in Algorithm 1.

C Consistency Proof

The consistency proof establishes that sample-average saddle-point problems converge to the true problem under compactness, continuity, boundedness, and law-of-large-numbers assumptions.

C Consistency Proof: Consistency requires a nonempty bounded true saddle-point set, continuous finite-valued constraint functions, and eventually nonempty bounded sample saddle-point sets.
C Consistency Proof: The proof uses compactness, convexity, continuity, pointwise S.L.L.N., and epi-convergence to obtain uniform convergence of the sampled Lagrangian.
C Consistency Proof: The sample-average saddle-point sets converge to the true saddle-point set, with set deviation tending to zero almost surely.The proof establishes convergence of limit points and the deviation D{S_N,S} → 0 as N → ∞.
C Consistency Proof: The consistency argument transfers convergence from sampled saddle points to the gradient estimator through the envelope theorem and bounded derivatives.
C Consistency Proof: For the dynamic result, the discounted terminal gradient contribution vanishes as t → ∞, enabling the infinite-horizon conclusion by bounded convergence.

E Gradient Formula for Dynamic Risk - Full Results

The section develops a dynamic coherent-risk policy-gradient formula involving a risk-sensitive value function, then constructs actor-critic and sampling-based approximations with consistency guarantees.

E Gradient Formula for Dynamic Risk - Full Results: The dynamic policy-gradient formula extends the standard policy-gradient theorem to Markov-coherent risk measures through the risk-sensitive value function.The gradient is expressed using a value function for the risk objective and underlies the proposed actor-critic method.
E Gradient Formula for Dynamic Risk - Full Results: The actor-critic procedure estimates the gradient by using a critic to calculate the risk-sensitive value function and an actor to sample the gradient.The critic evaluates the risk-sensitive value function, while the actor uses that estimate for gradient estimation.
E.2 Risk-Sensitive Bellman Equation: The risk-sensitive Bellman operator adds policy-induced stage cost to a discounted worst-case expectation over the risk envelope, with its fixed point equal to the risk objective.Exact Bellman computation becomes intractable for large state spaces because of the curse of dimensionality.
E.3 Value Function Approximation: The value function is approximated linearly with state-dependent features, and projected risk-sensitive value iteration uses simulated trajectories to estimate its fixed point.The approximation assumes the feature mapping has full column rank and projects onto a low-dimensional function subspace.
E.3 Value Function Approximation: The dynamic risk Bellman equation is equivalent to a robust Bellman equation, enabling robust temporal-difference methods under a contraction assumption.The contraction yields a unique fixed point for the projected risk-sensitive Bellman operator.
E.3 Value Function Approximation: The empirical approximation replaces the inner robust optimization with a sample-average problem, with kernel density estimation suggested for huge or continuous spaces.The inner optimization can be computationally expensive or intractable when the state space is large.
E.3 Value Function Approximation: The ℓ2-regularization term ensures convergence of empirical optimizers and KKT multipliers, which supports the subsequent policy-gradient analysis.The empirical value parameters converge almost surely as sample size and iteration count increase.
E.4 Gradient Estimation: The two-phase sampling estimator is consistent: as samples increase and value-function error decreases, the sampled gradient converges to the true dynamic-risk gradient.The gradient can also be interpreted through a risk-neutral value function under a risk-adjusted transition model and stage-wise cost.

F Convergence Analysis of Empirical PRSVI

The convergence analysis shows that empirical projected risk-sensitive value iteration approaches the unique projected fixed point under sampling and contraction conditions.

F Convergence Analysis of Empirical PRSVI: The uncertainty-set construction remains bounded under empirical transition probabilities, allowing the convergence argument to control the sampled risk operators.The analysis uses bounded convex uncertainty sets and compares the true and empirical transition models.
F Convergence Analysis of Empirical PRSVI: As both sample size N and iteration count k increase, the sampling-based PRSVI parameters converge almost surely to the unique projected risk-sensitive fixed point.The fixed point solves the projected risk-sensitive Bellman equation.
F Convergence Analysis of Empirical PRSVI: The empirical visiting distributions and transition probabilities converge almost surely to their statistical limits under the strong law for Markov processes.This convergence supports replacing empirical quantities with their population counterparts in the analysis.
F Convergence Analysis of Empirical PRSVI: Because the projected risk-sensitive Bellman operator is a contraction, the empirical projected value-iteration sequence converges to its unique fixed point.The contraction argument is combined with vanishing empirical approximation error.

G Technical Results

The technical results establish sensitivity and approximation bounds for the empirical robust optimization underlying gradient estimation, including explicit errors for stage costs and occupancy measures.

G Technical Results: The empirical robust optimization problem is analyzed under positivity, differentiability, constraint-qualification, convexity, and second-order conditions.These assumptions support KKT-based sensitivity analysis of empirical optimizers and multipliers.
G Technical Results: The analysis uses compact convex uncertainty sets, strict complementarity, and second-order sufficient conditions to ensure stable empirical optimization.These properties support convergence of the empirical saddle-point quantities used by the gradient estimator.
G Technical Results: The empirical and population optimizers and KKT multipliers differ by bounded quantities controlled through a nonsingular sensitivity system.The result follows from an implicit-function-based sensitivity theorem.
G Technical Results: The stage-wise cost approximation error is bounded by M_h∆, where ∆ is the value-function approximation error.The bound holds uniformly over states and actions.
G Technical Results: The convergence proof relies on empirical transition probabilities approaching the true transition probabilities almost surely as sample size increases.Bounded uncertainty sets and optimizer convergence control the remaining approximation terms.
G Technical Results: The empirical occupancy measure satisfies an ℓ1 error bound M_π∆ relative to its limiting value.This connects value-function approximation error to the accuracy of the estimated occupancy measure.