Source-linked AI summary
Equivalence Between Continuous-Time Risk-Sensitive Control and R\'enyi Divergence Minimization
Shinji Kataoka, Kaoru Teranishi, Yasumasa Fujisaki
TL;DR
Continuous-time nonlinear stochastic control lacks a unified probabilistic formulation for risk-averse and risk-seeking policies, while standard KL control is risk-neutral and minimax KL extensions remain risk-averse. This paper establishes an equivalence between risk-sensitive control and Rényi divergence minimization over path measures. The resulting framework relates divergence order to control risk attitudes and supports both kinds of policies.
Problem
Standard KL control is risk-neutral, minimax KL control is restricted to risk-averse settings, and prior Rényi control-as-inference equivalence did not cover continuous-time systems.
Source: §I · p. 1Method
The paper uses Girsanov-based path-measure change to formulate continuous-time risk-sensitive control as Rényi divergence minimization.
Source: D. R´enyi divergence minimization · p. 3, §IV · p. 6Results
The study establishes equivalence between risk-sensitive control and Rényi divergence minimization for continuous-time nonlinear stochastic systems, with divergence order mapping to risk attitudes.
Source: §I · p. 1, §III · p. 6, §IV · p. 6Takeaways & Limitations
The equivalence provides a unified probabilistic inference framework for risk-averse and risk-seeking control problems.
Source: §IV · p. 6
Abstract
from arXiv · showhide
In this study, we show that a continuous-time risk-sensitive control problem is equivalent to a R\'enyi divergence minimization problem over trajectory path measures. Reformulating stochastic optimal control as probabilistic inference via Kullback-Leibler (KL) divergence minimization avoids the computational intractability of the Hamilton-Jacobi-Bellman equation. However, standard KL control is inherently risk-neutral, and recent minimax extensions remain restricted to risk-averse settings. Our equivalence result resolves this limitation by offering a unified probabilistic framework for arbitrary risk attitudes in continuous-time nonlinear systems. Based on Girsanov theorem, we explicitly map the risk sensitivity to the R\'enyi divergence order, deriving a noise-dependent control penalty scaled by risk preference. This formulation seamlessly modulates tail-weighting behaviors, interpolating between zero-forcing for risk-averse policies and mass-covering for risk-seeking policies. These findings bridge stochastic control and information-theoretic inference, providing a foundation for sampling-based control algorithms.
I. INTRODUCTION
The introduction motivates sampling-based control as an alternative to intractable HJB equations, then identifies the need for a framework spanning risk-averse and risk-seeking behavior. It presents an equivalence between continuous-time risk-sensitive control and Rényi divergence minimization as that framework.
I. INTRODUCTION
- HJB equations characterize optimal control, but their nonlinear partial differential form is often analytically intractable, motivating sampling-based alternatives.KL control uses trajectory-measure divergence and a path-integral representation that enables Monte Carlo computation.Source: §I · p. 1
- Standard KL control is inherently risk-neutral, while risk-sensitive control is relevant to tasks requiring either avoidance or acceptance of high-cost risks.The introduction cites safety-critical control as risk-averse and exploration or financial decision-making as potentially risk-seeking.Source: §I · p. 1
- Existing minimax KL control is limited to risk-averse scenarios, and prior Rényi control-as-inference equivalence covered discrete-time rather than continuous-time systems.Rényi divergence offers a tunable emphasis on distribution regions, but its continuous-time extension remained open.Source: §I · p. 1
- The study establishes equivalence between risk-sensitive control and Rényi divergence minimization for continuous-time nonlinear stochastic systems.Its tunable parameter accommodates both risk-averse and risk-seeking policies in one probabilistic formulation.Source: §I · p. 1
- The paper connects Rényi variational-inference geometry to control risk attitudes, providing a control-theoretic perspective on zero-forcing and mass-covering behavior.The introduction presents this connection as a contribution of the study.Source: §I · p. 2
A. Notation
The notation section defines measure-theoretic conventions and the KL divergence used in the paper.
A. Notation
- The section defines absolute continuity, equivalent probability measures, the Radon–Nikodym derivative, expectation under a measure, and KL divergence.KL divergence is the expectation under P of the log Radon–Nikodym derivative dP/dQ when it exists, and is infinite otherwise.Source: A. Notation · p. 2
B. System Description and Assumptions
This section defines the controlled and uncontrolled path measures and states conditions that justify changing between them. The assumptions align control and noise channels and ensure the Girsanov measure change is valid.
B. System Description and Assumptions
- The system has state x_t, control u_t, Brownian motion w_t, drift f_t, input matrix g_t, and diffusion matrix h_t over a finite horizon.The state trajectory has controlled path measure P; the uncontrolled dynamics set u_t = 0 and have path measure Q.Source: B. System Description and Assumptions · p. 2
- The input and diffusion matrices must have identical column spaces, and the input matrix must have full column rank.This lets any control input be represented as a drift in the underlying noise process, so control and noise act through identical channels.Source: B. System Description and Assumptions · p. 2
- Novikov’s exponential integrability condition on the induced noise drift justifies the Radon–Nikodym derivative and Girsanov measure change.Under the stated integrability condition, P and Q are equivalent; controls violating it can lose absolute continuity and invalidate probabilistic evaluation via Girsanov.Source: B. System Description and Assumptions · p. 2
C. Risk-sensitive control
The section formulates risk-sensitive control by minimizing the logarithm of an expected exponential cumulative cost. Its parameter controls sensitivity to cost variation and determines whether the policy is risk-averse or risk-seeking.
C. Risk-sensitive control
- Risk-sensitive control minimizes the logarithm of the expected exponential cumulative running cost rather than the risk-neutral expected cumulative cost.The section notes that linear cost evaluation can underestimate rare, extreme trajectories.Source: C. Risk-sensitive control · p. 2
- As θ approaches zero, the risk-sensitive formulation recovers standard risk-neutral optimal control.Source: C. Risk-sensitive control · p. 3
- θ > 0 penalizes cost variance and yields risk-averse behavior, whereas θ < 0 yields risk-seeking behavior that tolerates worst-case scenarios for potentially lower expected costs.The logarithmic and exponential transformations introduce sensitivity to higher-order cost moments and a trade-off between expectation and variance.Source: C. Risk-sensitive control · p. 3
D. R´enyi divergence minimization
The section formulates risk-sensitive control as minimizing Rényi divergence between a controlled path measure and a cost-weighted reference path measure, with the divergence order governing tail weighting and risk attitude.
D. Rényi divergence minimization
- Across evaluated α values, Dα(P ∥ Q) is lower than Dα(P ∥ R), and the difference increases as α increases.The table reports Rényi divergences for the Gaussian measures across several α values.Source: Table I · p. 3, D. R´enyi divergence minimization · p. 3
- For the Gaussian measures, α = 0.5 emphasizes overlap between P and Q; increasing α shifts emphasis toward P’s left tail, where Q lacks probability mass.This makes the divergence more sensitive to low-probability tail deviations as α increases.Source: D. R´enyi divergence minimization · p. 3
- The reference measure Q* reweights the uncontrolled path measure Q by the exponential of negative total state cost, normalized by Z.Because the total state cost is nonnegative, Z is finite and Q* is equivalent to Q.Source: D. R´enyi divergence minimization · pp. 3–4
- The proposed problem minimizes the Rényi divergence of the controlled path measure P from the reference path measure Q*.Attaining Dα(P ∥ Q*) = 0 means P coincides with Q*.Source: D. R´enyi divergence minimization · pp. 3–4
- Orders α > 1 yield risk-averse control, while 0 < α < 1 yields risk-seeking control, linking the divergence order to risk sensitivity θ.The order parameter controls tail weighting relative to Q*.Source: D. R´enyi divergence minimization · p. 4
III. MAIN RESULTS
The section establishes an equivalence between risk-sensitive control and Rényi divergence minimization for continuous-time stochastic systems, under stated assumptions and a matching condition. The divergence order is linked to risk sensitivity, covering both risk-averse and risk-seeking control.
III. MAIN RESULTS
- Under Assumptions 1 and 2 and the matching condition, the risk-sensitive control and Rényi divergence minimization problems are equivalent, with the divergence order chosen from the control parameters.The theorem also gives a relation for recovering the optimal control input between the two formulations.Source: §III · pp. 4–5
- Girsanov-based changes of path measure make the Rényi minimization objective coincide with the risk-sensitive control objective.The proof derives the path-measure relationship from the system dynamics and then changes measure to establish the objective identity.Source: §III · pp. 4–5
- As α approaches 1, Rényi divergence converges to KL divergence and the formulation recovers KL control; θ approaching 0 gives risk-neutral control.The equivalence between KL control and risk-neutral control is cited as established prior work, [15].Source: §III · p. 5
- The control-cost matrix is noise-dependent and scaled by 1/α, reducing the penalty for α > 1 and increasing it for 0 < α < 1.The text connects the lower penalty to applying larger inputs to suppress deviations, and the higher penalty to tolerating disturbances rather than rejecting them.Source: §III · pp. 5–6
- Rényi order links approximation behavior to risk attitude: α > 1 strengthens zero-forcing for risk-averse control, while 0 < α < 1 is more mass-covering for risk-seeking control.The framework covers both attitudes, unlike the cited minimax KL framework, which is restricted to risk-averse settings [15].Source: §III · p. 6
IV. CONCLUSION
The conclusion states that the paper establishes an equivalence between risk-sensitive control and Rényi divergence minimization for continuous-time nonlinear stochastic systems. It identifies a unified inference framework for risk-averse and risk-seeking control and points to methods for solving the minimization problem as future work.
IV. CONCLUSION
- The paper establishes equivalence between risk-sensitive control and Rényi divergence minimization for continuous-time nonlinear stochastic systems using Girsanov's theorem and a change of path measure.The derivation gives a control-cost matrix and an explicit relationship between risk sensitivity θ and Rényi order α.Source: §IV · p. 6
- The resulting probabilistic inference framework covers both risk-averse and risk-seeking control and bridges information-theoretic control with risk-sensitive optimal control.Source: §IV · p. 6
- Future work will develop solution methods for Rényi divergence minimization and clarify its relationships to other optimal control formulations.Source: §IV · p. 6