Source-linked AI summary
Reinforcement Learning for Batch Bioprocess Optimization
Panagiotis Petsagkourakis, Ilya Orson Sandoval, Eric Bradford, Dongda Zhang, Ehecatl Antonio del Rio Chanona
TL;DR
Bioprocesses are difficult to optimize because their stochastic behavior, complexity, and plant-model mismatch limit accurate modeling. The paper uses a recurrent-neural-network Policy Gradient policy, first trained with an approximate model and then updated from true-plant measurements. Across case studies, the approach produced near-optimal policies, outperformed NMPC in the reported comparisons, and required only simple online policy evaluations.
Problem
Bioprocess optimization must address stochastic dynamics and plant-model mismatch when accurate true-system models and process structure are unavailable.
Method
A recurrent-neural-network control policy is pretrained with an approximate model and adapted batch-to-batch using Reinforce Policy Gradients and true-system measurements.
Results
The proposed methodology produced near-optimal policies for stochastic systems and outperformed NMPC in the reported case studies, with only simple policy evaluations needed online.
Takeaways & Limitations
The two-stage framework reduces costly true-system evaluations while providing an efficient closed-loop stochastic control policy.
Takeaways & Limitations
Robust constraint satisfaction remains unresolved, including safety concerns associated with reinforcement learning.
Abstract
from arXiv · showhide
Bioprocesses have received a lot of attention to produce clean and sustainable alternatives to fossil-based materials. However, they are generally difficult to optimize due to their unsteady-state operation modes and stochastic behaviours. Furthermore, biological systems are highly complex, therefore plant-model mismatch is often present. To address the aforementioned challenges we propose a Reinforcement learning based optimization strategy for batch processes. In this work, we applied the Policy Gradient method from batch-to-batch to update a control policy parametrized by a recurrent neural network. We assume that a preliminary process model is available, which is exploited to obtain a preliminary optimal control policy. Subsequently, this policy is updatedbased on measurements from thetrueplant. The capabilities of our proposed approach were tested on three case studies (one of which is nonsmooth) using a more complex process model for thetruesystemembedded with adequate process disturbance. Lastly, we discussed the advantages and disadvantages of this strategy compared against current existing approaches such as nonlinear model predictive control.
1. Introduction
Bioprocess optimization is challenging because biological complexity, stochastic behavior, plant-model mismatch, and uncertain disturbances limit accurate modeling and control. The paper motivates Policy Gradient reinforcement learning as a data-driven alternative for batch-to-batch optimization, including nonsmooth systems and comparison with NMPC.
- Bioprocesses combine intricate metabolic and fluid-dynamic relationships, making accurate physics-based models difficult to construct for large-scale biosystems.
- Stochastic macro-scale behavior and plant-model mismatch make bioprocess control and optimization an open challenge.
- Reinforcement learning, particularly Policy Gradients, is proposed to handle nonlinear, stochastic optimal-control problems with process disturbances and model mismatch.
- Approximate Dynamic Programming relies on the Hamilton-Jacobi-Bellman equation, whose solution becomes intractable for nonlinear systems with continuous states and controls.
- Policy Gradients directly estimate a control policy without requiring a process model or solving the Hamilton-Jacobi-Bellman equation, supporting continuous state and control spaces.
- The proposed strategy extends batch-to-batch Policy Gradient optimization with faster transfer-learning adaptation, additional complex and nonsmooth case studies, and comparison against NMPC.
2. Methodology
The methodology formulates batch optimization under stochastic dynamics and plant-model mismatch as policy learning, using a recurrent stochastic policy updated through sampled trajectories and transfer learning. The approach combines a preliminary policy with true-system measurements to adapt the controller.
- 2.1. Problem Statement: The batch process is modeled as a stochastic nonlinear state-space system with states, inputs, and process disturbances, and the policy seeks to maximize an economic criterion.The optimization does not assume a specific disturbance model, and the resulting problem may remain intractable even when dynamics are known.
- 2.1. Problem Statement: A parametrized policy maps system states to control actions while accounting for probabilistic state transitions in a closed-loop interaction with the physical system.The policy πθ is optimized against the expected performance index J.
- 2.3. Recurrent Neural Network: The policy is a recurrent neural network that uses past states and actions, outputs probability-distribution parameters, and samples the next control action.The recurrent structure supplies memory for sequential data, while stochastic outputs represent the exploration–exploitation trade-off.
- 2.2. Policy Gradient Methods: Policy gradients maximize expected performance directly and are particularly suited to continuous state and control spaces.The parameter update uses gradient ascent with a learning rate and an unbiased estimator derived through the Policy Gradient Theorem.
- 2.4. Reinforce Algorithm: The Reinforce algorithm estimates policy gradients from sampled trajectories without requiring a dynamic process model, with Monte Carlo estimation used for the expectation.The trajectory-density derivation removes terms independent of the policy parameters, leaving a model-free policy update.
- 2.5. Reinforcement Learning for Bioprocess Optimization under Uncertainty: The proposed uncertainty-handling strategy first trains a preliminary policy, then adapts it on the true system using transfer learning and recurrent-policy parameters.The stated methodology targets plant-model mismatch and disturbances; the preliminary network outputs statistical parameters such as a mean and standard deviation for stochastic actions.
3. Computational Case Studies
Three fed-batch bioreactor studies test batch-to-batch Policy Gradient adaptation using preliminary-model training and increasingly challenging true-plant dynamics. Across ordinary, stochastic, and nonsmooth cases, the approach is compared with NMPC and adapts policies through plant measurements.
- Three fed-batch bioreactors maximize final target-product concentration using light and inflow rate as manipulated variables.
- 3.1. Case Study 1 - Ordinary Differential Equations: The first case study models plant disturbances as Gaussian noise with mean 0 and standard deviation 0.02 on states y1 and y2.The normalized horizon is divided into 10 intervals with one constant control input per interval.
- The preliminary deterministic model is trained offline to construct an RNN control policy before true-plant adaptation.Training used 100 epochs and 800 episodes in the first case study.
- 3.1. Case Study 1 - Ordinary Differential Equations: After 25 real-plant batches, four adaptation epochs achieved reward 0.591 versus 0.583 for the stochastic-free optimal solution of the complex plant model.The policy reward initially had large variance, which became undetectable by the end of preliminary training.
- 3.2. Case Study 2 - Stochastic Differential Equations: In the stochastic differential-equation case, the same offline policy adapted to the new environment and was reported superior to NMPC in 100 Monte-Carlo simulations.The authors attribute faster adaptation to systematic transfer learning and shift computationally intensive training offline.
- 3.3. Case Study 3 - Nonsmooth Model: In the nonsmooth case, the proposed method was reported superior to NMPC, whose control actions had larger variance under nonsmoothness and uncertainty.The policy’s online action computation requires only RNN evaluation, while the OCP becomes difficult with integer variables and lacks a global-optimality guarantee.
4. Conclusions and Future Work
The proposed batch-to-batch reinforcement-learning framework adapts policies to uncertain bioprocesses using transfer learning, achieving strong performance with limited true-system evaluations. Its main outstanding challenge is robust constraint satisfaction and associated safety concerns.
- The two-stage framework trains a policy on an approximate model, then adapts it on the true system to reduce costly and time-consuming evaluations.The approach targets uncertain and complex bioprocesses where true dynamics are unknown.
- Transfer learning enables systematic adaptation to a new environment using relatively few episodes, and the resulting policy outperforms NMPC with simple policy evaluations.The method was validated on two case studies involving different stochastic processes.
- 3 hours of offline CPU time versus 0.002 secs online shifts computational complexity offline and makes policy deployment efficient.The resulting stochastic policy provides a closed-loop solution for a nonlinear stochastic dynamical system.
- 100 total batches were implemented across four epochs and 25 batches, yet the policy performed better from the beginning of online implementation.The authors therefore state that fewer batches could still outperform NMPC.
- Robust constraint satisfaction remains an unresolved limitation, with implications for reinforcement-learning safety and a stated focus for future work.The considered systems include stochasticity, plant-model mismatch, and no available process structure.