Source-linked AI summary
Discrete Time Mean-Field Stochastic Linear-Quadratic Optimal Control Problems
Robert. J Elliott, Xun Li, Yuan-Hua Ni
TL;DR
The paper addresses solvability of discrete-time stochastic mean-field LQ control problems, in which expectations enter the dynamics and cost. It combines Hilbert-space optimization, operator LQ reformulation, matrix dynamic optimization, Riccati equations, and completing the square to obtain and validate linear state feedback. The resulting control is characterized through feedback gains under the paper’s stated assumptions.
Problem
The paper studies solvability of discrete-time stochastic mean-field LQ problems whose dynamics and cost depend on expectations of states and controls.
Method
The problem is treated through Hilbert-space quadratic optimization, operator LQ reformulation, matrix dynamic optimization, Riccati equations, and completing the square.
Results
The paper obtains necessary and sufficient solvability conditions and a linear state-feedback optimal control with feedback gains derived through the reformulated optimization problem.
Takeaways & Limitations
The discrete-time mean-field LQ problem can be represented in progressively more explicit forms leading to an optimal feedback control and an optimality validation.
Takeaways & Limitations
The formulation assumes martingale-difference disturbances, deterministic coefficient matrices, and independence between the initial value and disturbances.
Abstract
from arXiv · showhide
This paper first presents necessary and sufficient conditions for the solvability of discrete time, mean-field, stochastic linear-quadratic optimal control problems. Then, by introducing several sequences of bounded linear operators, the problem becomes an operator stochastic LQ problem, in which the optimal control is a linear state feedback. Furthermore, from the form of the optimal control, the problem changes to a matrix dynamic optimization problem. Solving this optimization problem, we obtain the optimal feedback gain and thus the optimal control. Finally, by completing the square, the optimality of the above control is validated.
1 Introduction
The paper studies discrete-time stochastic mean-field LQ control, where expectations of states and controls enter the dynamics and cost. It develops solvability conditions and multiple routes to an implementable linear state-feedback control.
- Problem formulation: The problem seeks an admissible control minimizing a quadratic cost for a discrete-time stochastic mean-field system.The state dynamics include both state/control terms and their expectations, while the system is driven by stochastic disturbances.
- Motivation: Mean-field terms represent collective interactions through expectations, extending the classical stochastic LQ formulation.The paper relates expectations to limiting interactions among many agents and motivates variance-related cost terms through mean-field applications such as finance.
- Relation to prior work: The discrete-time setting is distinguished from continuous-time mean-field LQ control because systems may be sampled or obtained by discretizing continuous-time dynamics.The paper contrasts its formulation with the continuous-time approach in.
- Solution strategy: The paper first converts the control problem into a quadratic optimization problem in Hilbert space to obtain necessary and sufficient solvability conditions.This initial optimal control has an abstract form that may be viewed as open loop.
- Solution strategy: It then uses operator LQ, matrix dynamic optimization, Riccati equations, and completing-the-square arguments to derive and validate linear state-feedback controls.The sequence moves from operator feedback representations to feedback gains and a final optimality verification.
2 Preliminaries and abstract considerations
The section formulates the mean-field LQ problem as a Hilbert-space quadratic optimization problem and derives solvability conditions. It then reformulates the problem as an operator stochastic LQ problem, yielding a linear state-feedback control under stated positivity assumptions.
- Problem (MF-LQ) is defined as finite when its cost is finite for every initial value, and solvable when an optimal control exists for each initial value.
- The problem is treated through two methods: Hilbert-space quadratic optimization for solvability conditions and operator LQ reformulation for linear state feedback.
- Problem (MF-LQ) is solvable if and only if Θ1 ≥0 and the associated optimality equation has a solution; strict positivity gives a unique minimizer.
- The assumptions Qk, Qk + ¯Qk ≥0, Rk, Rk + ¯Rk > 0, and GN, GN + ¯GN ≥0 imply Θ1 > 0.
- Although the initial optimal-control representation may appear open-loop, the operator reformulation produces a closed-loop linear feedback of the current state.
3 Solution by Riccati equations
The paper develops implementable solutions for the discrete-time mean-field LQ problem by converting it from operator feedback to matrix dynamic optimization and deriving feedback gains through Riccati equations.
- Operator formulation: The operator formulation establishes that the optimal control is a linear state feedback, providing the basis for an implementable representation.The preceding abstract results are described as mathematically pleasing but not directly implementable because they reference operator expressions.
- Matrix dynamic optimization: Problem (MF-LQ) is reformulated as a matrix dynamic optimization problem in the feedback gains and associated state matrices.The reformulation uses the state covariance-related matrices generated by the closed-loop dynamics.
- Riccati equations: The matrix minimum principle yields first-order conditions for the optimal feedback gains under the stated positivity conditions on Qk, Rk, G_N, and their mean-field counterparts.The conditions include Qk, Qk + ¯Qk ≥ 0, Rk, Rk + ¯Rk > 0, and G_N, G_N + ¯G_N ≥ 0.
- Riccati equations: The resulting gain equations are coupled with Riccati difference equations for the matrix sequences Pk and ¯Pk.The proof derives the Riccati relations and establishes their positive semidefiniteness by induction.
- Verification: Completing the square independently in the mean and fluctuation coordinates validates the optimal control and shows its equivalence to the Riccati-based solution.The control can be written as uk = (Lk + ¯Lk)Exk + Lk(xk − Exk), separating mean and deviation components.
4 Numerical results
A 4-period numerical example applies the Riccati equations to a three-dimensional state system and obtains the corresponding Riccati solutions and optimal control.
- Numerical example: The numerical study considers a 4-period mean-field stochastic LQ example with x0 ∈ R3 and periods k = 0, 1, 2, 3.The system includes deterministic matrices, stochastic disturbances, and mean-field terms in both drift and diffusion.
- Numerical example: The example specifies diagonal state, control, and terminal weighting matrices Qk, ¯Qk, Rk, ¯Rk, G4, and ¯G4.These matrices define the costs used in the example.
- Computed solutions: Applying Theorem 3.2 produces the optimal control for the numerical example.The control follows from the feedback-gain characterization established by the theorem.
5 Conclusion
The paper presents four complementary methods for the discrete-time mean-field LQ problem and obtains an optimal control expressed as linear state feedback using two Riccati equations.
- Conclusion: The four methods are quadratic optimization in Hilbert space, operator LQ analysis, matrix dynamic optimization, and completing the square.Together, these methods address solvability, feedback representation, gain computation, and optimality validation.
- Conclusion: The resulting optimal control is a linear state feedback determined using two Riccati equations.The paper identifies infinite-horizon mean-field LQ control and system stability as future research topics.