Source-linked AI summary
Deep Learning Approximation for Stochastic Control Problems
Jiequn Han, Weinan E
TL;DR
High-dimensional stochastic control is difficult because conventional methods face the curse of dimensionality. The paper directly approximates finite-horizon, time-dependent controls with stacked feedforward subnetworks trained through model dynamics, and reports near-optimal solutions across trading and energy-storage examples while extending to high dimensions.
Problem
High-dimensional stochastic control problems face the curse of dimensionality, limiting the conventional dynamic-programming approach.
Method
The method stacks feedforward subnetworks approximating time-dependent controls through model dynamics and uses the control objective as the deep-network loss.
Results
The numerical examples suggest near-optimal solutions for optimal trading and constrained energy storage, including multidimensional storage cases.
Takeaways & Limitations
The approach appears applicable to high-dimensional stochastic control and to resource allocation, dynamic games, and wealth-management problems.
Takeaways & Limitations
As the time horizon increases, learning variance becomes larger at fixed batch size and more iteration steps are required; multiple constraints may increase policy nonlinearity.
Abstract
from arXiv · showhide
Many real world stochastic control problems suffer from the "curse of dimensionality". To overcome this difficulty, we develop a deep learning approach that directly solves high-dimensional stochastic control problems based on Monte-Carlo sampling. We approximate the time-dependent controls as feedforward neural networks and stack these networks together through model dynamics. The objective function for the control problem plays the role of the loss function for the deep neural network. We test this approach using examples from the areas of optimal trading and energy storage. Our results suggest that the algorithm presented here achieves satisfactory accuracy and at the same time, can handle rather high dimensional problems.
1 Introduction
The paper targets the curse of dimensionality in high-dimensional stochastic control by approximating time-dependent controls with deep neural networks. Its finite-horizon, control-focused formulation differs from reinforcement learning and approximate dynamic programming, with numerical examples suggesting near-optimality and high-dimensional applicability.
- Dynamic programming is mathematically elegant but encounters the curse of dimensionality in high-dimensional stochastic control problems.
- Deep learning results on difficult machine-learning problems motivate neural-network approximations for settings where the curse of dimensionality is the essential obstacle.
- The paper approximates each finite-horizon, time-dependent control with a feedforward subnetwork and stacks subnetworks into one trainable deep network.
- Unlike related reinforcement-learning work, the framework is finite-horizon and directly approximates controls rather than value functions.
- Numerical examples suggest near-optimality while handling high-dimensional problems with relative ease.
2 Mathematical formulation
The paper formulates a finite-horizon, model-based stochastic control problem with state-dependent equality and inequality constraints. The objective combines intermediate and terminal costs and is minimized over admissible state-dependent control functions.
- The model uses a finite horizon T, filtered probability space, state st ∈ St ⊂ R^m, and control at ∈ R^n.
- System evolution is model-based, with deterministic drift and incoming random information represented by ξt+1.
- Controls may be subject to state-dependent equality constraints gi(st, at) = 0 and inequality constraints hj(st, at) ≥ 0.
- Assuming the current state fully characterizes the model, admissible controls are represented as functions of st satisfying the constraints for every state.
- The minimization objective uses intermediate costs ct(st, at), terminal cost cT(sT), and total cost CT.
3 An neural network approximation algorithm
The algorithm represents each time-dependent control as a feedforward neural subnetwork and connects these subnetworks through stochastic dynamics and accumulated costs. Monte-Carlo samples train the resulting deep network with the control objective as loss, while penalties and projection address constraints.
- The control-state dependence is approximated by a multilayer feedforward neural network with parameters θt at each time.
- For a sampled stochastic process, total cost CT becomes the output of a deep network formed by connecting time-indexed subnetworks.
- The architecture combines control subnetworks, cost contributions, and dynamics-based shortcuts linking successive time blocks.
- With N hidden layers per subnetwork, the complete network contains (N + 2)T layers.
- Equality and inequality constraints enter through penalty functions, and learned controls are projected onto the admissible set during testing.
- The network is trained using backpropagation with standard SGD-type optimization, batch normalization, TensorFlow, and Adam.
- The subnetworks use four layers with two hidden ReLU layers, normally initialized without pre-training; deterministic initial states cause batch normalization to be skipped at t = 1.
4 Numerical results and discussion
The method is evaluated on optimal trading and constrained energy-storage problems, including a multidimensional setting. It achieves near-optimal results against analytical or dynamic-programming benchmarks while remaining applicable as dimensionality increases.
- Optimal trading: The trading experiment minimizes expected execution cost for multiple stocks over a fixed horizon using a control space R^23 → R^10.The model uses n = 10 stocks and m = 3 market-condition variables.
- Optimal trading: 1.001, 1.002, and 1.009 are the average relative trading costs to the exact solution for T = 20, 25, and 30.The computed strategy also approximates the exact optimal strategy well.
- Energy storage and allocation benchmark: The energy-storage benchmark optimizes renewable-storage revenues under stochastic demand while imposing multiple constraints.The state includes storage, wind production, electricity price, and demand; controls represent energy transfers among these components.
- Energy storage and allocation benchmark: 1.002 and 0.995 are the average relative rewards for T = 10 and 15 against the optimal lookup-table benchmark.For T = 10, the neural-network policy gives higher expected reward than the lookup-table policy; relaxing discretization improves reward in both cases.
- Energy storage and allocation benchmark: As the energy-storage time horizon increases, variance becomes larger with the same batch size and more iterations are required.The learning curves are rougher than in the trading example, possibly because multiple constraints make the optimal policy more nonlinear.
5 Conclusion
The paper presents a deep-learning approach that directly approximates controls for high-dimensional finite-horizon stochastic control problems. Numerical results suggest near-optimal solutions under multiple constraints, with potential applicability across several large-scale problem domains.
- The method approximates time-dependent controls with feedforward neural networks and stacks them through model dynamics.The control objective serves as the deep-learning loss function.
- Numerical results suggest near-optimal solutions across different constrained stochastic control problems.The reported scope includes cases with multiple constraints.
- The approach should apply to dynamic resource allocation, dynamic games, and wealth management with large portfolios.The paper contrasts this broader setting with prior treatments based on separability or mean-field assumptions.