Source-linked AI summary
Risk-Averse Trust Region Optimization for Reward-Volatility Reduction
Lorenzo Bisi, Luca Sabbioni, Edoardo Vittori, Matteo Papini, Marcello Restelli
TL;DR
Return-variance risk measures do not capture every short-term reward fluctuation that matters in settings such as trading. The paper defines reward volatility and develops mean-volatility policy-gradient and trust-region methods; it shows that reward volatility bounds return variance and derives updates with monotonic improvement guarantees.
Problem
Return-focused risk measures can miss short-term variation in individual rewards, although that variation can matter when interim outcomes such as daily P&L are monitored.
Source: §1 · p. 1Method
The paper defines reward volatility under the state-occupancy measure, derives a mean-volatility policy gradient, and adapts trust-region optimization to that objective.
Source: §1 · pp. 1–2, §4 · p. 3Results
Takeaways & Limitations
Reward volatility offers a way to target smoother reward trajectories and constrain return variance in risk-averse settings.
Source: §1 · p. 1, §8 · p. 8Obtaining an unbiased estimator for X requires triple sampling, which may be restrictive.
Source: §4.1 · p. 4
Abstract
from arXiv · showhide
In real-world decision-making problems, for instance in the fields of finance, robotics or autonomous driving, keeping uncertainty under control is as important as maximizing expected returns. Risk aversion has been addressed in the reinforcement learning literature through risk measures related to the variance of returns. However, in many cases, the risk is measured not only on a long-term perspective, but also on the step-wise rewards (e.g., in trading, to ensure the stability of the investment bank, it is essential to monitor the risk of portfolio positions on a daily basis). In this paper, we define a novel measure of risk, which we call reward volatility, consisting of the variance of the rewards under the state-occupancy measure. We show that the reward volatility bounds the return variance so that reducing the former also constrains the latter. We derive a policy gradient theorem with a new objective function that exploits the mean-volatility relationship, and develop an actor-only algorithm. Furthermore, thanks to the linearity of the Bellman equations defined under the new objective function, it is possible to adapt the well-known policy gradient algorithms with monotonic improvement guarantees such as TRPO in a risk-averse manner. Finally, we test the proposed approach in two simulated financial environments.
1 Introduction
The introduction argues that maximizing expected return alone is insufficient when decision-makers also need to control risk, especially short-term reward variation. It proposes reward volatility and mean-volatility optimization, then extends the approach to trust-region updates.
1 Introduction
- Reward-volatility control targets single-step reward variation across visited states, a risk dimension that can matter when interim financial results are monitored.The introduction contrasts this with return variance, which measures variation in accumulated rewards across trajectories.Source: §1 · p. 1
- The paper introduces reward volatility as a risk measure and shows that it bounds return variance, up to a normalization factor.The measure avoids the inter-temporal correlation terms that complicate return variance.Source: §1 · p. 1
- The authors derive a policy gradient theorem for a mean-volatility objective that trades off expected-return maximization against risk minimization.They attribute the analysis's tractability to reward volatility's simpler relationship to rewards over time.Source: §1 · p. 1
- The proposed TRVO algorithm adapts TRPO-style updates to mean-volatility optimization and is designed to guarantee continuing improvement.The paper tests its algorithms in two simulated financial environments.Source: §1 · p. 2
2 Preliminaries
The preliminaries define the discounted MDP setting and the policy-performance quantities used later. They then introduce parametric policies and the policy gradient framework for optimizing expected return.
2 Preliminaries
- The paper studies infinite-horizon, discounted MDPs with state and action spaces, transition model, bounded rewards, discount factor, and initial-state distribution.Returns are defined as discounted cumulative rewards along a trajectory.Source: §2 · p. 2
- The discounted state-occupancy measure describes the distribution of states reached over time under a policy.It is defined in terms of the probability of reaching a state after t steps from the initial state.Source: §2 · p. 2
2.1 Actor-only policy gradient
- Expected return can be expressed using either transition probabilities or the state-occupancy measure.The section presents these as two formulations of the normalized policy return.Source: §2.1 · p. 2
- For parametric policies, the policy gradient theorem provides a gradient for optimizing expected performance over policy parameters.The policy is parameterized by θ, and the objective is to find parameters that maximize performance.Source: §2.1 · p. 2
3 Risk Measures
The section distinguishes reward volatility from return variance and defines mean-volatility as a trade-off between expected return and short-term risk. It relates the two risk measures and illustrates how they can favor different policies.
3 Risk Measures
- Reward volatility is expressed as an expected sum over trajectories, unlike return variance, which is not defined in the same state-occupancy form.The discussion emphasizes that reward volatility concerns single-step rewards rather than accumulated trajectory returns.Source: §3 · p. 3
- Mean-volatility combines expected return with a risk penalty, with λ ≥ 0 controlling the trade-off between return maximization and volatility minimization.The section contrasts this objective with mean-variance optimization.Source: §3 · p. 3
- Reward volatility bounds return variance up to a normalization factor, while omitting the inter-temporal reward correlations included in return variance.The paper presents this bound as a reason reward-volatility analysis can help constrain return variance.Source: §3 · p. 3, §A.1 · p. 10
- In the deterministic MDP example, mean-variance can prefer action b for its higher expected return despite a −10 interim reward, whereas mean-volatility may choose the other path depending on λ.The return variance remains zero in the example, so it does not capture that short-term reward risk.Source: §3 · p. 3, Figure 1 · p. 3
4 Risk-Averse Policy Gradient
The paper derives a policy gradient theorem for reward volatility using action- and state-volatility functions, then develops an actor-only estimator and VOLA-PG. Its estimator can be unbiased with triple sampling, though the paper notes this may be restrictive.
4 Risk-Averse Policy Gradient
- The reward-volatility policy gradient theorem uses action- and state-volatility functions, with linear Bellman equations enabling the mean-volatility objective.The paper presents linear Bellman equations and the resulting policy gradient theorem as theoretical advantages over return variance.Source: §4 · pp. 3–4
4.1 Estimating the risk-averse policy gradient
- VOLA-PG estimates the risk-averse gradient from sampled trajectories and updates policy parameters by stochastic gradient ascent.The estimator adapts a policy-gradient estimator by using a reward term involving squared deviations from the estimated return; it can be turned into a GPOMDP estimator with variance-minimizing baselines.Source: §4 · p. 4, §4.1 · p. 4
- An unbiased action-volatility estimator uses three separate trajectory sets, and the paper notes this sampling requirement may be restrictive.The sets provide separate return estimates and reward samples.Source: §4.1 · p. 4
5 Trust Region Volatility Optimization
The paper extends trust-region policy optimization to the mean-volatility objective, aiming to guarantee monotonic improvement at each policy update. It derives TRVO using the linear Bellman equations and adapts TRPO-style results, while noting that policy-dependent rewards require proper assumptions.
5 Trust Region Volatility Optimization
- TRVO adapts a trust-region method to mean-volatility optimization, with the aim of guaranteeing monotonic improvement at each policy update.The paper attributes this adaptation to the linear Bellman equations available for its approach.Source: §5 · pp. 4–5
- The paper states that theoretical results leading to TRPO also hold for the mean-volatility objective.Its performance-difference analysis includes a squared difference between policies’ expected returns; neglecting that term yields a bound.Source: §5 · p. 5
- The policy-dependent reward Rλ(s,a) = R(s,a) − λ(R(s,a) − Jπ)^2 requires proper assumptions for the policy gradient theorem and performance bound to hold.The paper notes these results do not generally hold for policy-dependent rewards without such assumptions.Source: §5 · p. 5
- The trust-region surrogate incorporates KL divergence between policies to provide monotonic improvement guarantees.The paper describes this as an approximation of the surrogate function.Source: §5 · p. 5
6 Related Works
The related work connects risk-averse objectives with safe policy updates, which address environmental and model uncertainty. It contrasts reward volatility’s linear Bellman structure with limitations of return-variance and other risk measures.
6 Related Works
- The paper combines risk-averse objectives and safe policy updates, which address inherent risk from environmental stochasticity and model risk from imperfect parameter knowledge.The authors describe this combination as new to the best of their knowledge.Source: §6 · p. 5
- Unlike reward volatility’s Bellman equation, Sobel’s return-variance equation lacks dynamic-programming monotonicity, motivating complex return-variance minimization approaches.The cited approaches include Di Castro, Tamar, and Mannor 2012; Tamar and Mannor 2013; and Prashanth and Ghavamzadeh.Source: §6 · p. 5
- The paper states that nonlinear Bellman relationships for other risk measures prevent extending safe guarantees and TRPO to them, while prior safe updates addressed risk-neutral objectives.A safe VOLA-PG alternative appears in Appendix B as more rigorous but less practical than TRVO.Source: §6 · p. 6
7 Experiments
Experiments compare risk-aware algorithms in a portfolio-management task and an S&P 500 trading framework. Results distinguish portfolio risk trade-offs from trading performance and convergence behavior.
7 Experiments
- The experiments compare VOLA-PG, TRVO, a discount-adjusted mean-variance policy-gradient approach, and risk-averse TRPO-exp across portfolio management and trading tasks.The portfolio task is simplified; the trading setting uses S&P 500 daily prices and 50-day episodes.Source: §7 · p. 7, §7.1 · p. 7, §7.2 · p. 7
7.2 Trading Framework
- TRVO’s trading frontier dominates TRPO-exp, while TRPO-exp becomes unstable at high risk-aversion parameter c.VOLA-PG’s frontier is dominated at the same iteration count because it had not converged.Source: Figure 3 · p. 7, §7.2 · pp. 7–8
7.1 Portfolio Management
- As risk aversion parameter λ changes, Algorithm 1’s mean-variance frontier almost coincides with the Di Castro, Tamar, and Mannor (2012) frontier; return variance is equally reduced by mean-volatility or mean-variance optimization, while VOLA-PG better optimizes reward-volatility.Source: §7.1 · p. 7
8 Conclusions
The conclusion presents reward volatility and a mean-volatility policy gradient with a trust-region update for risk-aware reinforcement learning. It reports trading results and identifies real online financial evaluation as future work.
8 Conclusions
- Reward volatility measures reward variability between steps, and reducing it produces smoother trajectories that avoid shocks.The conclusion states that this measure bounds return variance.Source: §8 · p. 8
- A mean-volatility policy-gradient algorithm and trust-region update use the linearity of the Bellman equations to ensure monotonic objective improvement at each gradient update.The conclusion frames this guarantee as important for online algorithms in real-world contexts.Source: §8 · p. 8
- In the tested financial environments, TRVO outperformed the naive approach on the Trading Framework and converged faster than its policy-gradient counterpart.The conclusion describes tests in two financial environments.Source: §8 · p. 8
- Evaluation in a real online financial setting with new data points outside the training distribution remains future work.Source: §8 · p. 8
A Proofs
The proofs introduce state-occupancy densities and develop the variance inequality, policy-gradient and trust-region results, and an unbiased estimator. The supplied proof passages state the estimator’s sampling conditions but provide limited detail on several derivations.
A Proofs
- The state-occupancy densities represent discounted probabilities of visiting states, either from another state or from the start.Source: A Proofs · p. 10
A.1 Variance inequality
- The variance-inequality proof expands both sides and applies Cauchy–Schwarz in its derivation.Source: §A.1 · pp. 10–11
A.2 Risk-averse policy gradient theorem
- The appendix states a reward-volatility policy-gradient theorem using action-variance and state-variance functions.The passage names the theorem but does not provide its full statement.Source: §A.2 · p. 11
A.3 Trust Region Volatility Optimization
- The trust-region derivation separates the mean-volatility advantage from the standard advantage when expressing performance difference.Source: §A.3 · pp. 12–13
A.4 Gradient Estimator
- The action-volatility estimator is unbiased when return estimates use two trajectory sets and reward samples use a third set.Source: §A.4 · p. 13
B Safe Volatility Optimization
The section adapts safe policy-gradient analysis to mean-volatility optimization under smoothing-policy assumptions, and separately links exponential utility on rewards to the mean-volatility objective.
B Safe Volatility Optimization
- Smoothing policies require a convex parameter space and bounds on score-function and observed-information norms.The analysis is restricted to twice-differentiable policies satisfying these conditions.Source: B Safe Volatility Optimization · p. 14
- Theorem 8 adapts safe policy-gradient analysis to the mean-volatility objective using a policy-Hessian norm bound.The paper describes this as an adaptation of Papini, Pirotta, and Restelli (2019).Source: B Safe Volatility Optimization · p. 16
- For stochastic gradient updates, a confidence bound on gradient-estimation error supports choosing safe step sizes and batch sizes.The text states that these meta-parameters maximize guaranteed performance improvement.Source: B Safe Volatility Optimization · pp. 14, 16
- Under a Gaussian gradient assumption, an F-distribution ellipsoid gives a confidence region for the unknown mean.The region is stated to contain the mean with probability at least 1−δ.Source: B Safe Volatility Optimization · p. 17
C Exponential Utility applied on the reward
- The paper presents exponential utility applied to rewards as a loose approximation to the mean-volatility objective.The argument uses a second-order Taylor expansion of the reward exponential utility.Source: C Exponential Utility applied on the… · p. 18
D Experiments
The experiments describe a portfolio-optimization setup and an S&P trading framework, including their task settings, actions, and policy architectures.
D.1 Portfolio Optimization
- Portfolio optimization uses nl=1.1, M=10, prisk=0.05, pswitch=0.1, α=0.2, and a two-hidden-layer policy with 10 neurons per layer.The task parameters are listed for the portfolio environment.Source: §D.1 · p. 19
D.2 S&P Trading Framework
- The S&P trading framework uses short, flat, and long actions, charges fees of 7/100000, and uses a two-hidden-layer policy with 64 neurons per layer.The actions are −1, 0, and 1, respectively.Source: §D.2 · p. 19