Source-linked AI summary
From Learning to Control: Data-Driven Multi-Agent Reinforcement Learning for Multivariable Control in a Microalgae Bioprocess
Juan D. Gil, Ehecatl Antonio Del Rio Chanona, José Luis Guzmán, Manuel Berenguel
TL;DR
Microalgae PBRs require stable pH and dissolved oxygen control despite coupled, variable dynamics. This paper uses offline-trained, online-fine-tuned MARL agents to regulate both variables, reporting performance comparable to PID control with greater adaptability under changing conditions.
Problem
Microalgae cultivation requires stable pH and dissolved oxygen, but their coupled dynamics and sensitivity to environmental and operational fluctuations make joint control challenging.
Source: §1 · pp. 1–2Method
Results
0.12 [-] for pH and 44 [%] for DO were the largest reference deviations over four days, comparable to PID-based control.
Source: §4.3 · p. 8Takeaways & Limitations
The four-day trial reports that MARL maintained stable control under disturbances and varying operating conditions in the open PBR.
Source: §4.3 · p. 8
Abstract
from arXiv · showhide
Effective control of bioprocesses is particularly challenging due to the intrinsic nonlinearity and dynamic variability of living-cell systems. In microalgae-based photobioreactors (PBRs), maintaining stable pH and dissolved oxygen (DO) levels is critical for optimal growth and productivity, yet their strong coupling and sensitivity to environmental fluctuations make multivariable control difficult. This study proposes a novel hybrid offline-online Multi-Agent Reinforcement Learning (MARL) framework for simultaneous pH and DO regulation, leveraging Deep Deterministic Policy Gradient (DDPG) agents to achieve a fully data-driven and model-free control solution. The agents are trained using historical data generated by an expert system, eliminating the need for direct experimentation with the environment. After deployment, the agents operate autonomously, continuously fine-tuning their policies daily to adapt to evolving process dynamics and reject fast transient disturbances. Experimental validation in an open, industrial-scale PBR at the University of Almeria demonstrated the framework's capability to maintain stable operation under realistic conditions. The results confirm that model-free MARL control provides a robust and adaptive alternative for complex bioprocess environments.
1 Introduction
Microalgae bioprocess control must manage coupled pH and dissolved oxygen despite nonlinear, changing conditions. The paper introduces a model-free MARL controller trained from expert data and validated in an open industrial-scale PBR.
1 Introduction
- Stable pH and dissolved oxygen (DO) are critical to microalgal growth, but environmental fluctuations and coupled process dynamics make their joint control challenging.CO2 injection supplies carbon and buffers pH, while air injection regulates DO.Source: §1 · pp. 1–2
- Existing control methods rely on accurate, continually refined process models, while simultaneous pH–DO regulation in open PBRs remains underexplored.Prior joint-control work used selective control, while many implementations rely on ON/OFF logic or PID controllers.Source: §1 · p. 2
- The study trains DDPG agents on historical expert PID data, then fine-tunes them daily during autonomous operation in an open industrial-scale PBR over four days.The authors describe this as the first validation of an MARL-based control strategy in an industrial-scale bioprocess.Source: §1 · p. 2
2 Material and methods
The methods describe the open raceway PBR and its control challenge, then introduce the partial-observation and actor–critic concepts used to explain DDPG. Control is achieved by manipulating CO2 and air flows to regulate pH and DO.
2.1 System overview and control problem description
- The 80 m2 raceway PBR uses CO2 and air injection to control pH and DO, while irradiance and irregular dilution flows affect process dynamics.Measurements focus on the channel end farthest from the injection zones, the most challenging control location.Source: §2.1 · pp. 2–3, Figure 1 · p. 3
2.2 Reinforcement learning background
- A POMDP represents settings where agents receive partial observations rather than direct access to the full state, and the actor–critic architecture selects and evaluates actions from those observations.The paper describes the state, observation and action spaces, transition dynamics, reward function, and discount factor as components of the POMDP.Source: §2.2 · p. 3
2.3 Deep Deterministic Policy Gradient
- DDPG uses a deterministic actor and a critic, with replay-buffer minibatches for training and target networks to stabilize critic updates.The critic minimizes mean-squared error between temporal-difference targets and online predictions; the actor ascends the expected cumulative reward gradient.Source: §2.3 · pp. 3–4
3 Proposed MARL methodology
The proposed MARL framework assigns independent DDPG agents to pH and DO control. It trains them offline on expert-controller data and then fine-tunes their policies online as the PBR dynamics change.
3 Proposed MARL methodology
- Independent DDPG agents regulate pH and DO by manipulating CO2 injection and air flow, respectively.Offline pretraining, gradual online updates, and soft target-network updates are used to mitigate decentralized learning instability.Source: §3 · p. 4
3.1 Offline training
- Offline training updates each agent using historical data from the expert PID controller without direct interaction with the physical PBR.Agents learn independently while shared process observations capture coupled dynamics and support emergent coordination.Source: §3.1 · p. 4
3.2 Online fine-tuning
- Online fine-tuning adapts policies to time-varying dynamics and disturbances, preloading the replay buffer with historical data before adding recent operating experience.Fine-tuning may use fewer epochs to reduce overfitting or destabilization.Source: §3.2 · pp. 4–5
T end end
Online fine-tuning initializes from offline-trained policies and historical experience, then incorporates new operating data. The buffer retains recent experiences for continued agent updates.
T end end
- Online fine-tuning initializes each agent’s actor and critic from the offline-trained parameters and loads its buffer with historical data.Source: T end end · p. 5
- During PBR operation, each agent collects new transitions containing its observation, action, reward, and next observation.Source: T end end · p. 5
- The agents’ buffers are updated to retain recent experiences, and fine-tuning can use fewer epochs to limit overfitting or destabilization.Source: T end end · p. 5
4 Results
The section describes PID-generated training data, the MARL controller’s implementation, and its performance in the real PBR. Across four operating days, MARL maintained pH and DO regulation comparable to PID control while adapting to changing conditions.
4.1 Collection of historical experiences
- PID-generated data covered three days; two independent PID controllers regulated pH and DO by manipulating CO2 injection and air flow, respectively.The PID loops omitted derivative terms, decouplers, and feedforward controllers; their reference points were pH 8 and DO 220%.Source: §4.1 · pp. 5–6
4.2 MARL computational implementation
- The agents used historical PID data, with observation inputs reflecting process coupling and disturbances; offline training ran up to 3000 epochs, followed by 50-epoch daily online fine-tuning.The pH agent received pH, irradiance, dilution flow, and air-injection events; the DO agent received DO and irradiance.Source: §4.2 · pp. 7–8
4.3 Results of the MARL approach in the real PBR system
- After daily fine-tuning, the controller compensated for a dilution-related pH drop on the final day, regulating pH and CO2 concentrations accurately under similar conditions.Earlier operation included lower temperature and irradiance than during training, alongside a midday pH deviation.Source: §4.3 · p. 8, Figure 6 · p. 9
- Across four days of operation, the largest reference deviations were 0.12 [-] for pH and 44 [%] for DO; these were comparable to PID-based control, while the MARL controller showed superior adaptability under varying operational conditions.Source: §4.3 · p. 8
5 Conclusion
Over four consecutive days in an industrial-scale open PBR, decentralized DDPG agents maintained stable pH and DO regulation under realistic disturbances. The authors identify year-long seasonal evaluation and shared-critic or cooperative architectures as future work.
5 Conclusion
- Over four consecutive days, the MARL controller stably regulated pH and DO in an industrial-scale open PBR amid irradiance fluctuations and dilution-flow injections.Daily fine-tuning improved adaptability, helping the agents correct deviations and compensate for variable coupling effects.Source: §5 · p. 9
- Two decentralized DDPG agents were trained on historical expert PID-control data and then fine-tuned online with real process data, one agent per variable.The agents regulated pH and DO independently.Source: §5 · p. 9
- Full-year evaluation under seasonal variability and shared-critic or cooperative multi-agent architectures remain future research directions.The proposed methodology's long-term adaptability has yet to be assessed over a full year.Source: §5 · p. 9