Source-linked AI summary
Policy Gradients with Variance Related Risk Criteria
Dotan Di Castro, Aviv Tamar, Shie Mannor
TL;DR
Variance-related risk criteria matter in reinforcement learning applications such as finance and process control, but optimizing them is computationally difficult. The paper derives a new episodic cost-to-go variance formula and uses it to build model-based and model-free policy-gradient algorithms. These algorithms are proved to converge to locally optimal points and are demonstrated in risk-sensitive portfolio management.
Problem
Variance-related reinforcement-learning criteria, including Sharpe-Ratio and mean-variance objectives, are important but difficult to optimize, with some problems computationally intractable.
Method
The paper derives an episodic trajectory-variance result and develops model-based and model-free policy-gradient algorithms using a penalty method for constraints.
Results
The algorithms are proved to converge almost surely to locally optimal points of their corresponding objective functions.
Takeaways & Limitations
Gradient-based local optimization provides an algorithmic approach for reinforcement learning with variance-related risk criteria, demonstrated in a risk-sensitive portfolio problem.
Takeaways & Limitations
The framework assumes state-dependent rewards; extending the results to state-and-action rewards is described as straightforward rather than developed here.
Abstract
from arXiv · showhide
Managing risk in dynamic decision problems is of cardinal importance in many fields such as finance and process control. The most common approach to defining risk is through various variance related criteria such as the Sharpe Ratio or the standard deviation adjusted reward. It is known that optimizing many of the variance related risk criteria is NP-hard. In this paper we devise a framework for local policy gradient style algorithms for reinforcement learning for variance related criteria. Our starting point is a new formula for the variance of the cost-to-go in episodic tasks. Using this formula we develop policy gradient algorithms for criteria that involve both the expected cost and the variance of the cost. We prove the convergence of these algorithms to local minima and demonstrate their applicability in a portfolio planning problem.
1. Introduction
Risk-aware decision making extends reinforcement-learning objectives beyond expected reward to include statistics such as variance, but many variance-related criteria are computationally hard. This paper develops locally optimizing policy-gradient algorithms for episodic variance-related criteria, with model-based and model-free methods that converge and are tested in portfolio management.
- Motivation: Risk criteria augment expected reward with statistics such as variance or Value at Risk, addressing inherent and parametric uncertainty.The paper focuses on inherent uncertainty and uses learning to mitigate parametric uncertainty.
- Motivation: The Sharpe Ratio measures expected profit relative to standard deviation and is widely used in financial decision making.The paper identifies finance and process control as prominent applications of risk-aware decision making.
- Challenge: Variance-based criteria are computationally demanding: some mean-variance optimization problems are intractable or not even approximable.Traditional dynamic-programming methods do not solve these problems in general.
- Contribution: The paper addresses this difficulty with policy-gradient algorithms that optimize variance-related objectives locally, starting from a new episodic variance result.The episodic setting enables derivation of policy-gradient-style algorithms.
- Contribution: Model-based and model-free algorithms are presented, convergence is proved, and applicability is demonstrated numerically in risk-sensitive portfolio management.The framework also extends immediately to other listed performance criteria.
2. Framework and Background
The framework models episodic reinforcement learning in a finite MDP with parameterized policies, accumulated reward, and trajectory-level mean and variance. It formalizes variance-related optimization problems and explains their difficulty through nonconvex achievable reward-variance pairs.
- MDP and policy: The agent operates in a finite-state, finite-action MDP whose policy induces a Markov transition function and an ergodic stationary distribution.The policy is probabilistic and differentiable in its parameter vector.
- Episodic formulation: An episode ends at the first return to a recurrent state x*, and B denotes the accumulated reward along that trajectory.The framework focuses on the finite-horizon, or episodic, case.
- Episodic formulation: J(x) is the expected accumulated reward from state x, while V(x) is the corresponding trajectory variance.Both quantities depend on the policy parameter even when that dependence is suppressed in notation.
- Optimization criteria: The paper studies stochastic optimization problems involving constrained trajectory variance and maximization of the Sharpe Ratio.The setup assumes bounded J(x*) and V(x*), and the Sharpe-Ratio problem additionally requires V(x*) > ϵ for some ϵ > 0.
- Optimization difficulty: Mean-variance optimization cannot generally use policy iteration and is hard because the achievable (J(x*), V(x*)) region need not be convex.In the symmetric example, deterministic policies achieve J(x*) values −2, 0, and 2 with zero variance, while intermediate nonzero values require randomization and variance.
- Optimization difficulty: Figure 1 pairs the example MDP diagram with a phase plane showing the nonconvex structure of achievable reward-variance pairs.The right pane depicts the achievable (J(x*), V(x*)) pairs.
3. Formulae for the Trajectory Variance and its Gradient
The paper derives solvable formulae for the mean and trajectory variance of cumulative reward between visits to a recurrent state. These formulae support gradient expressions for policy optimization and extend related discounted-horizon results to the episodic setting.
- Variance formulae: The episodic analysis treats a transition to the recurrent state as terminating cumulative reward accumulation, yielding a Bellman-like equation for the value function.The recurrent state contributes no further rewards after termination.
- Variance formulae: Proposition 3.1 defines a transition matrix with the recurrent-state column zeroed and uses it to obtain solvable expressions for J and V.The mean satisfies J = (I − P′)^−1r in vector form.
- Variance formulae: The derivation rewrites the second-moment recursion to relate the cumulative-reward variance to reward terms, transition probabilities, and the value function.The displayed algebra establishes the intermediate quantity used for the variance equation.
- Gradient expressions: Proposition 3.1 also yields gradient expressions with respect to policy parameters for both the mean and variance.The gradients are presented using element-wise vector products.
- Relation to prior work: The episodic equations differ from Sobel’s discounted-horizon equations because the discounted invertibility argument does not immediately extend to the undiscounted case.Sobel replaces I − P′ with I − βP for β < 1.
4. Gradient Based Algorithms
The paper turns the mean–variance objectives into gradient-based algorithms, using penalty methods for constraints and exact or simulation-based gradient estimates. Under stated smoothness, recurrence, and step-size conditions, the algorithms converge to locally optimal points.
- Algorithm design: The framework derives both exact and simulation-based gradient algorithms for the constrained mean–variance problems.Exact gradients suit small problems, while simulation is used when the state space is large or the model is unknown.
- Penalty methods: Penalty methods transform the constrained optimization problem into an unconstrained objective by penalizing infeasible variance values.As λ increases, the penalized solution converges to the constrained solution, motivating successive penalty updates.
- Exact gradient algorithm: With known transitions, gradient ascent applies the analytical gradients to the penalized objective using positive step sizes.The update uses the gradient of J together with the penalty derivative and the gradient of V.
- Exact gradient algorithm: Under bounded second derivatives, countably many local optima, and suitable step sizes, the exact gradient algorithm converges to a locally optimal point.A differentiable policy such as softmax can satisfy the required smoothness conditions.
- Simulation-based optimization: Likelihood-ratio estimates from trajectories between recurrent-state visits provide unbiased estimates of the mean and variance gradients for simulation-based optimization.The algorithm updates policy parameters episodically and uses separate estimates of J and V.
- Simulation-based optimization: A two-timescale scheme updates estimates of J and V faster than policy parameters because J(x*)∇J(x*) cannot be unbiasedly estimated from one trajectory.The fast estimates approach stationary limits while the policy evolves more slowly.
- Simulation-based optimization: The simulation-based algorithm converges almost surely to a locally optimal point of its objective under the stated assumptions and step-size conditions.The proof identifies stable fixed points of the limiting ODE with local optima and establishes convergence to that set.
5. Experiments
The experiments apply simulation-based policy-gradient algorithms to a simplified portfolio problem with liquid and non-liquid assets, comparing average reward, variance-constrained reward, and Sharpe Ratio objectives.
- Portfolio model: The portfolio contains liquid assets sellable at every step and non-liquid assets with time-dependent interest rates, maturity constraints, and default risk.The model represents investment holdings and the non-liquid asset’s interest-rate deviation in the state.
- Portfolio model: The investment dynamics are illustrated in Figure 2.
- Compared criteria: The algorithms optimize average reward, variance-constrained reward with V(x*) ≤ b, and the Sharpe Ratio.The three criteria are evaluated using simulation-based algorithms from Section 4.
- Results: The average-reward policy was riskier and produced higher gain than the variance-constrained policy.Figure 3 presents the corresponding accumulated-reward distributions.
- Results: Maximizing the Sharpe Ratio produced a very conservative policy that almost never invested in the non-liquid asset.
6. Conclusion
The paper presents a gradient-based framework for reinforcement learning with variance-related risk criteria in settings where globally optimal solutions are difficult to obtain. It establishes local optimality and highlights further questions about risk measures, convergence, and Sharpe Ratio behavior.
- Contribution: The work introduces an algorithmic approach for reinforcement learning with variance-related risk criteria, which pose significant algorithmic challenges.
- Experimental comparison: Figure 3 compares accumulated-reward distributions for total-reward maximization, variance-constrained reward maximization, and Sharpe Ratio maximization.
- Contribution: The gradient-based approach targets local optimality because obtaining an optimal solution appears difficult even when the model is known.
- Open questions: Further investigation is proposed for percentile criteria, variance-reduction methods, temporal-difference algorithms, and the risk-averse behavior observed under Sharpe Ratio maximization.The paper describes these as open issues rather than settled results.