Source-linked AI summary
Risk-sensitive Reinforcement Learning
Yun Shen, Michael J. Tobia, Tobias Sommer, Klaus Obermayer
TL;DR
The paper addresses how reinforcement-learning agents can represent human risk preferences when rewards and transition probabilities are uncertain and unknown. It develops utility-based risk-sensitive Q-learning, proves convergence, and applies the model to behavioral and fMRI data, where it fits behavior better and aligns with risk-related neural signals.
Problem
Human risk preferences can differ between gains and losses, while behavioral models requiring true probabilities are unsuitable when those probabilities are unknown during learning.
Method
The paper applies utility functions to TD errors, derives risk-sensitive Q-learning, proves convergence, and models sequential investment behavior with gain–loss risk parameters.
Results
The risk-sensitive model significantly outperformed standard Q-learning and expected utility behaviorally, while its TD errors and Q-values correlated with fMRI activity in several risk-related regions.
Takeaways & Limitations
Utility-based risk-sensitive Q-learning provides a candidate model of human sequential decision-making with prospect-theory-consistent risk preferences and associated neural signals.
Takeaways & Limitations
The convergence analysis assumes sufficiently broad exploration, and some utility functions require truncation or approximation to satisfy the global Lipschitz condition.
Abstract
from arXiv · showhide
We derive a family of risk-sensitive reinforcement learning methods for agents, who face sequential decision-making tasks in uncertain environments. By applying a utility function to the temporal difference (TD) error, nonlinear transformations are effectively applied not only to the received rewards but also to the true transition probabilities of the underlying Markov decision process. When appropriate utility functions are chosen, the agents' behaviors express key features of human behavior as predicted by prospect theory (Kahneman and Tversky, 1979), for example different risk-preferences for gains and losses as well as the shape of subjective probability curves. We derive a risk-sensitive Q-learning algorithm, which is necessary for modeling human behavior when transition probabilities are unknown, and prove its convergence. As a proof of principle for the applicability of the new framework we apply it to quantify human behavior in a sequential investment task. We find, that the risk-sensitive variant provides a significantly better fit to the behavioral data and that it leads to an interpretation of the subject's responses which is indeed consistent with prospect theory. The analysis of simultaneously measured fMRI signals show a significant correlation of the risk-sensitive TD error with BOLD signal change in the ventral striatum. In addition we find a significant correlation of the risk-sensitive Q-values with neural activity in the striatum, cingulate cortex and insula, which is not present if standard Q-values are used.
1 Introduction
Standard reinforcement learning selects actions to maximize expected cumulative reward, but human choices can shift between risk aversion and risk seeking. Existing risk-sensitive approaches either assume economic rationality or require true transition probabilities that may be unavailable during learning.
- RL models sequential decisions in Markov environments by choosing actions that maximize expected cumulative rewards.
- Human agents can be risk-averse in some situations and risk-seeking in others, unlike consistently risk-neutral or economically rational policies.
- Risk-sensitive MDP research transforms rewards, transition probabilities, or both to represent differing risk preferences across agents.
- Behavioral and neuroscience models can allow economic irrationality but often require knowledge of true probabilities at the start of learning.
- The paper develops valuation functions, risk-sensitive Q-learning with a convergence proof, and an application to human behavior in a sequential investment task.
2 Valuation Functions and Risk Sensitivities
The paper defines valuation functions for uncertain outcomes and uses their curvature to characterize risk preferences. Utility-based shortfalls provide a broad framework that includes standard expected reward, entropic valuation, and prospect-theory-like gain–loss asymmetries.
- A valuation function ρ ranks outcome-distribution pairs, while monotonicity and translation invariance formalize higher outcomes and sure outcomes.
- Centralization subtracts ρ(0, µ), making the valuation a subjective mean that can vary by reference point and include the objective mean as a special case.
- Concave valuation functions induce risk aversion, whereas convex valuation functions induce risk seeking because diversification is respectively preferred or disfavored.
- Utility-based shortfall: Utility-based shortfalls apply a utility function to outcomes relative to an acceptance level and generalize valuation functions used across several fields.
- Special cases: The framework includes standard expected reward for linear utility and entropic valuation for exponential utility.
- Prospect theory: Mixed convex–concave utilities model distinct gain and loss preferences, while subjective probability weighting can underestimate or overestimate event probabilities.
3 Risk-sensitive Reinforcement Learning
The paper formulates risk-sensitive reinforcement learning for MDPs with random rewards and unknown transition models, then derives valuation-based Q-learning with convergence guarantees. Utility applied to the TD error transforms rewards and transition probabilities, enabling prospect-theory-like behavior.
- An MDP comprises states, admissible actions, transition probabilities, and distributed random rewards, with finite state, action, and noise spaces assumed.
- Risk-sensitive objectives replace standard expected cumulative rewards with valuation maps over future reward distributions and state-action pairs.
- The optimal value function satisfies a risk-sensitive Bellman equation, and an optimal deterministic policy selects the action maximizing the corresponding state-action valuation.
- Because reward and transition models may be unknown, the paper derives generalized Q-learning that estimates Q-values while the agent explores the environment.
- Under Assumption 3.1 and suitable learning-rate conditions, Q_t(s, a) converges to Q*(s, a) for every admissible state-action pair with probability 1.
- Convergence requires every state-action pair to be visited infinitely often, so learning policies must sufficiently explore rather than become too greedy prematurely.
- Applying a nonlinear utility to the TD error transforms both rewards and transition probabilities, while S-shaped utilities reproduce key prospect-theory effects without an explicit probability-weighting function.
- The global Lipschitz requirement excludes some important utilities, including exponential and polynomial forms, unless they are truncated or approximated.
4 Modeling Human Risk-sensitive Decision Making
The sequential investment game exposes distinct human risk preferences that standard Q-learning fails to capture, while RSQL fits behavior better and links risk-sensitive values to fMRI activity.
- 4.1 Experiment: The game models sequential investment as a 7-state MDP with four investment actions, making higher investment both riskier and potentially more rewarding.Path 1 has EV = 90 and std = 14.9, whereas Path 4 has an average loss of -9.75 and std = 6.9; Path 3 offers EV = 52.25 with std = 12.3.
- 4.2 Risk-sensitive Model of Human Behavior: Thirty subjects displayed heterogeneous strategies: 17 mainly chose high-reward Path 1, six chose low-reward Path 4, and seven showed no significant path preference.Because more than 40% did not follow the expected-reward-maximizing policy, standard Q-learning failed to explain their behavior.
- 4.2 Risk-sensitive Model of Human Behavior: RSQL significantly outperformed expected utility and standard Q-learning in explaining behavioral data.The relative BIC difference was ∆B = −500.14 for RSQL and ∆B = −23.10 for EU; more negative scores indicate better fit.
- 4.2 Risk-sensitive Model of Human Behavior: Most subjects were risk-averse for positive TD errors and risk-seeking for negative TD errors, consistent with behavioral-economics findings.The RSQL utility parameters quantify risk preferences separately for wins and losses.
- 4.2 Risk-sensitive Model of Human Behavior: Subjects in the Path 4 and random groups had normalized subjective probabilities averaging 10% below risk-neutral levels, explaining their conservative policies.Path 1 subjects had small |∆p| values and behavior resembling risk-neutral agents.
- 4.3 fMRI Results: RSQL-derived TD errors and Q-values significantly modulated fMRI activity, whereas standard Q-values failed to predict brain activity at p < .01 uncorrected.TD errors modulated the subcallosal gyrus extending into ventral striatum, while Q-values modulated the left anterior insula and bilateral ventral striatum.
5 Summary
The paper derives and validates a risk-sensitive Q-learning framework for sequential decision-making, linking mixed risk preferences to prospect theory and neural signals.
- The utility-based shortfall framework yields a risk-sensitive Q-learning algorithm whose optimal policy corresponds to the risk-sensitive objective.The algorithm models mixed risk preferences and is proved to converge.
- S-shaped utility functions replicate key behavioral features predicted by prospect theory.These include different risk preferences for gains and losses.
- In a sequential investment game, the risk-sensitive algorithm fit human behavior significantly better than standard Q-learning and the expected utility model.
- Risk-sensitive TD errors correlated with ventral-striatum BOLD changes, while risk-sensitive Q-values correlated with neural activity in the striatum, cingulate cortex, and insula.The corresponding correlation was not present when standard Q-values were used.
- The framework is limited in its current form to finite-state MDPs, although continuous-state extensions using function approximation are possible.Extensions to SARSA, TD(λ), and average-case reinforcement learning are also identified.
A Mathematical Proofs
The appendix establishes valuation-function properties through translation invariance and expected-utility arguments, including a characterization involving the utility-based shortfall.
- The sup-norm measures the maximum absolute component of a finite-dimensional vector.
- The centered valuation function subtracts ρ(0, µ), allowing valuation functions to be centralized without changing the relevant structure.
- Strict monotonicity and continuity of u characterize the shortfall threshold through the condition Eu(X − m*) = x0.
A.1 Proofs for Risk-sensitive Q-learning
The proofs show that the risk-sensitive Q-learning update satisfies the conditions needed for almost-sure convergence by establishing contraction and controlled noise properties.
- The risk-sensitive update is reformulated using an operator H applied to Q-values and a transformed temporal-difference error.The reformulation uses ũ(x) = u(x) − x0 and dt = rt + γ maxa qt(st+1, a) − qt(s, a).
- Under the proposition’s assumptions, the Q-value sequence converges with probability one to the unique fixed point Hq* = q*.
- The operator H is a contraction under the sup-norm when the utility function satisfies the stated Lipschitz assumptions.Lemma A.2 supplies a contraction factor below one.
- The stochastic update is analyzed with a history filtration, martingale-difference noise, and learning-rate conditions.
A.2 Truncated Algorithms with Weaker Assumptions
The appendix extends convergence guarantees to truncated utility functions under weaker practical assumptions, while specifying the resulting algorithm and its exploration requirements.
- Exponential and power utility functions may violate the required global Lipschitz condition, motivating local assumptions and truncation.
- Bounded rewards provide finite bounds on the solution Q*, enabling utility truncation outside a sufficiently large interval.
- The truncated and original utility functions yield identical unique Q* solutions when they agree on the relevant bounded interval.
- The truncated risk-sensitive Q-learning algorithm initializes Q-values, selects actions under a proper policy, updates with αt = 1/N(st, at), and truncates Q-values.
A.3 Heuristics for Polynomial Utility Functions
The appendix addresses utility functions that violate local Lipschitz assumptions near zero by proposing two approximation schemes. These approximations support the polynomial utilities used in the analysis.
- The utility u(x) = x^p with p ∈ (0, 1) is not Lipschitz near zero.
- The first scheme replaces u with u_ϕ(x) = (x + ϕ)^p − ϕ^p for positive ϕ.
- The second scheme approximates u near zero with a linear function.
- In both schemes, ϕ should be chosen very close to zero.
- For p > 1, the strictly positive lower-bound requirement also creates a problem near zero, which small-ϕ approximations address.The second scheme is used in Section 4 for both p > 1 and p ∈ (0, 1).
A.4 Softmax Policy
This section establishes conditions under which the softmax policy is proper. Under those conditions, every state is visited infinitely often.
- A proper policy visits every state infinitely often.
- The transition kernel under a policy combines the MDP transition probabilities P(y|x, a) with action probabilities π(a|x).
- If every state pair can be connected with positive probability under a suitable deterministic policy, the softmax policy is proper.
- Uniformly bounded Q-values imply a positive lower bound ϵ₀ on every softmax action probability.
- This lower bound transfers the deterministic-policy reachability condition to the softmax policy, ensuring infinite state visitation.
- In the behavioral experiment's MDP, each state can reach every other state with probability 1 within at most four steps under some deterministic policy.
B Magnetic Resonance Protocol and Data Processing
The study acquired structural and functional MRI data using a 3T system and processed them in SPM8. The functional analysis modeled stimulus events alongside task and computational variables.
- Structural MRI used a T1-weighted MPRAGE sequence with 1 × 1 × 1 mm³ voxel resolution.Images were acquired on a 3T whole-body Siemens Magnetom TIM Trio using a 32-channel head coil.
- Structural and functional images were processed in SPM8, including anatomical normalization to Montreal Neurological Institute standard space.The anatomical workflow also used segmentation, DARTEL, slice-timing correction, realignment, and distortion correction.
- The first-level GLM included stimulus-event regressors and parametric modulators for trial-specific outcomes and computational-model parameters.The modeled task variables included investment choice value and total reward value.