Source-linked AI summary
Open-Loop and Closed-Loop Solvabilities for Stochastic Linear Quadratic Optimal Control Problems
Jingrui Sun, Xun Li, Jiongmin Yong
TL;DR
The paper asks how open-loop and closed-loop solvability differ in stochastic LQ control and how each relates to Riccati equations and cost properties. It distinguishes the notions, characterizes closed-loop solvability through Riccati equations and convexity, and connects finiteness with Riccati convergence and minimizing sequences. Its results include examples showing that open-loop solvability can hold without closed-loop solvability and that some prior claims are incorrect.
Problem
The paper investigates the distinct solvability notions for stochastic LQ problems and their relationships with Riccati equations, convexity, and finiteness.
Method
The paper uses Riccati equations, quadratic cost-functional representations, convexity analysis, and minimizing sequences to characterize solvability and finiteness.
Results
Open-loop optimal controls can exist without a closed-loop optimal strategy, while under stated conditions finiteness, unique closed-loop solvability, and uniform convexity are equivalent.
Takeaways & Limitations
Open-loop and closed-loop solvability must be distinguished, and Riccati-based conclusions require the paper’s stated conditions.
Takeaways & Limitations
The paper shows that a previously stated result is incorrect, with both its sufficiency and necessity parts failing in the discussed settings.
Abstract
from arXiv · showhide
This paper is concerned with a stochastic linear quadratic (LQ, for short) optimal control problem. The notions of open-loop and closed-loop solvabilities are introduced. A simple example shows that these two solvabilities are different. Closed-loop solvability is established by means of solvability of the corresponding Riccati equation, which is implied by the uniform convexity of the quadratic cost functional. Conditions ensuring the convexity of the cost functional are discussed, including the issue that how negative the control weighting matrix-valued function R(s) can be. Finiteness of the LQ problem is characterized by the convergence of the solutions to a family of Riccati equations. Then, a minimizing sequence, whose convergence is equivalent to the open-loop solvability of the problem, is constructed. Finally, an illustrative example is presented.
1 Introduction
The paper formulates a stochastic linear-quadratic control problem and distinguishes open-loop from closed-loop solvability. It studies their links with Riccati equations, cost convexity, finiteness, and minimizing sequences.
- Problem formulation: The stochastic LQ problem seeks an admissible control minimizing a quadratic cost for each initial pair.The value function is the infimum of the cost functional over admissible controls.
- Open-loop and closed-loop solvability: Open-loop optimal controls may depend on the initial state, whereas closed-loop strategies must work for all initial states.The paper explicitly separates these notions because their solvability properties differ.
- Solvability characterizations: Open-loop solvability is linked to an optimality FBSDE, while closed-loop solvability is linked to a regular solution of the Riccati equation.The paper develops these equivalences as a central organizing principle.
- Finiteness and minimizing sequences: Finiteness is characterized through convergence of a family of Riccati-equation solutions, with convergent minimizing sequences leading to open-loop solvability.This provides a constructive route from finite value to optimal controls.
- Main facts: The paper reports that the value function need not be continuous in time, while under stated conditions finiteness, unique closed-loop solvability, and uniform convexity are equivalent.The equivalence in Fact 3 assumes D(·)=0 and uniformly positive-definite R(·), without requiring nonnegative Q(·) or G.
2 Preliminaries
This section establishes the stochastic LQ setting and formalizes finiteness, open-loop optimal controls, and closed-loop optimal strategies. A counterexample demonstrates that open-loop optimal controls can exist without a closed-loop strategy.
- Finiteness: Finiteness means that the value is greater than −∞, defined at an initial pair, time, or globally according to the quantifiers over x and t.The coefficients need not impose positive-definiteness or non-negativity on G, Q(·), and R(·).
- Solvability definitions: An open-loop optimal control minimizes the cost for one initial pair, whereas a closed-loop strategy uses Θ∗(·)X∗(·)+v∗(·) and must be optimal for every initial state.Closed-loop optimality therefore imposes a stronger state-independent requirement on the strategy.
- Counterexample: The example admits continuous open-loop optimal controls but is not closed-loop solvable on any [t,1] with t∈[0,1).This demonstrates that open-loop solvability does not imply closed-loop solvability.
- Analytical tools: The cost representation and Lyapunov-equation tools support analysis of convexity and solvability.The section also introduces standard coefficient assumptions and operator-based estimates used later.
3 Representation of the Cost Functional
The cost functional is represented as a quadratic functional of the control using bounded operators and forward-backward stochastic equations. This representation yields derivative, stationarity, and convexity conditions.
- Quadratic representation: The cost functional can be written as a quadratic expression involving operators M2(t), M1(t), M0(t), and affine terms.The representation separates control-control, state-control, state-state, linear, and constant contributions.
- FBSDE representation: The operator M2(t) is linked to an adapted FBSDE through B(s)⊤Y(s)+D(s)⊤Z(s)+S(s)X0(s)+R(s)u(s).This identifies the control-side quadratic operator with the stochastic state and adjoint variables.
- Lyapunov component: The associated Lyapunov equation determines M0(t), with its solution represented through the fundamental stochastic state equation.The construction uses the homogeneous state dynamics and corresponding matrix-valued stochastic evolution.
- Optimality system: For an open-loop optimum, the Fréchet derivative yields a stationarity condition that, together with the FBSDE, forms the optimality system.The stationarity relation introduces the coupling needed to characterize open-loop optimal controls.
- Convexity: Under the standard conditions (1.4), the zero-initial-state cost functional is uniformly convex.Uniform convexity is used later in studying finiteness and open-loop or closed-loop solvability.
4 Solvabilities of Problem (SLQ), Uniform Convexity of the Cost Functional, and the Riccati Equation
The section distinguishes open-loop from closed-loop solvability and links closed-loop solvability to regular solutions of the Riccati equation. Uniform convexity of the cost functional is shown equivalent to strong regular solvability, while counterexamples demonstrate that open-loop solvability alone is weaker.
- Under uniform convexity, the problem is uniquely open-loop solvable, and the optimal control can be represented in state-feedback form through the Riccati and adapted BSDE solutions.
- Open-loop solvability does not imply closed-loop solvability: continuous open-loop controls may exist for every initial pair while the Riccati equation lacks a regular solution.
- Closed-loop solvability is equivalent to existence of a regular Riccati solution together with an adapted BSDE solution.
- A strongly regular Riccati solution implies unique closed-loop solvability and therefore unique open-loop solvability.
- Uniform convexity of the zero-initial-state cost functional is equivalent to strong regular solvability of the Riccati equation.
- The value function for the homogeneous problem is represented quadratically as V^0(t, x) = ⟨P(t)x, x⟩.
5 Finiteness of Problem (SLQ) and Convexity of Cost Functional
The paper relates finiteness and cost convexity to Riccati-equation behavior, while showing that convexity alone need not ensure finiteness. Under additional conditions, finiteness, convexity, and solvability become equivalent, including cases where R is not positive semidefinite.
- Finiteness: Convexity of u(·) ↦ J0(t, x; u(·)) is not sufficient for finiteness, as an example has V0(0, x) = −∞ for every x ≠ 0.The example demonstrates divergence of the optimization problem despite convexity.
- Finiteness: Finiteness is characterized by boundedness from below of the family’s initial Riccati solutions Pε(0), with their limit yielding the value-function representation.This characterization applies under the stated convexity condition and also provides the limiting Riccati object.
- Indefinite weighting: R(·) need not be positive semidefinite; if it is negative definite, convexity requires injective D(·) and sufficient positivity from G or Q(·) to compensate.Thus, negative control weighting can be compatible with convexity under compensating conditions.
- Solvability and convexity: Under condition (5.32), finiteness implies closed-loop solvability, and the theorem’s equivalent statements include unique open-loop and closed-loop solvability.The equivalence is established together with the relevant convexity and Riccati properties.
6 Minimizing Sequences and Open-Loop Solvabilities
Without uniform convexity, the paper constructs minimizing sequences from regularized problems and characterizes open-loop solvability through their convergence. Open-loop solvability is equivalent to boundedness of the regularized controls in the control Hilbert space.
- Motivation: The paper studies open-loop solvability without assuming uniform convexity of the cost functional.This extends the analysis beyond the setting where uniform convexity already yields solvability.
- Minimizing sequences: For finite problems, the paper constructs a minimizing sequence from the unique optimal controls of ε-regularized problems.The regularized controls are associated with strongly regular solutions of the perturbed Riccati equations.
- Minimizing sequences: The regularized controls form a minimizing sequence because their costs converge to the original infimum.The convergence is expressed as lim ε→0 J(t, x; uε(·)) = inf J(t, x; u(·)) = V(t, x).
- Open-loop solvability: Under convexity, open-loop solvability is equivalent to convergence properties of the constructed sequence, and every weakly or strongly convergent subsequence has an open-loop optimal limit.The result is stated for the sequence defined by the regularized problems.
- Open-loop solvability: Open-loop solvability is also equivalent to L2-boundedness of the regularized controls {uε(·)}ε>0.For the homogeneous problem, this becomes L2-boundedness of {Θε(·)Xε(·)}ε>0.
- Consequences: A minimizing sequence whose control-state products are L2-bounded yields a regular Riccati solution and, consequently, open-loop solvability and finiteness.The resulting regular solution satisfies the Riccati equation’s range-related conditions.
7 An example
The example exhibits open-loop solvability without closed-loop solvability. Its value function is finite but discontinuous in time, while the Riccati solution fails the regularity range condition.
- Open-loop behavior: The example admits continuous open-loop optimal controls for every initial pair, so it is open-loop solvable.The optimal controls are continuous in the control-time variable.
- Closed-loop behavior: The corresponding Riccati equation has a unique solution that is not regular because its range condition fails.The failure of regularity prevents closed-loop solvability on any [t, 1].
- Value function: The value function is V0(t, x) = 0 for 0 ≤ t < 1 and V0(1, x) = x2.Thus the finite value changes discontinuously at the terminal time.