PaperlayerView exact revision on arXiv

Source-linked AI summary

Two-Timescale Reinforcement Learning for Real-Time Optimization and Economic NMPC: Experimental Validation

Saket Adhau, Jose Matias, Sebastien Gros, Sigurd Skogestad

arXiv:2609.35141v1eess.SYcs.SY

TL;DR

Process optimization and control depend on models that can be inaccurate, limiting economic performance under plant–model mismatch. The paper uses two-timescale Q-learning to tune RTO and ENMPC from plant data; in laboratory validation, the proposed scheme achieved 8.6% higher profit than nominal ENMPC while retaining the reported feasibility and path-constraint properties.

  • Problem

    RTO and ENMPC performance degrades under plant–model mismatch, while correcting structural errors through parameter estimation alone can be insufficient.

    Source: §1 · p. 1

  • Method

    A two-timescale Q-learning framework tunes parameterized RTO and ENMPC layers, with slow steady-state updates and fast transient-control updates, without explicit plant-gradient estimates.

    Source: §1 · p. 2, §6 · p. 11

  • Results

    8.6% higher profit than nominal ENMPC, with deployed-policy input feasibility preserved and empirical path-constraint satisfaction under disturbances and measurement noise.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Learned model parameters moved toward independently identified reference plant values along directions most influential to the economic objective, while weakly coupled directions remained partially identified.

    Source: §6 · p. 11

    Offline training data cover limited operating conditions and disturbance patterns, leaving generalization to unseen scenarios open.

    Source: §6 · p. 11

Abstract

from arXiv · show

We propose a reinforcement learning (RL) framework that tunes both Real-Time Optimization (RTO) and Economic Nonlinear Model Predictive Control (ENMPC) to address plant--model mismatch in process systems. Drawing on modifier-adaptation concepts, the method parameterizes the dynamic model, stage costs, constraints, and RTO modifiers, and uses Q-learning to adjust these parameters at two timescales: a fast update for the ENMPC layer and a slow update for the RTO layer. The framework is experimentally validated on a laboratory rig emulating a three-well subsea oil-production network. Using plant measurement data, the proposed RTO-RLMPC scheme achieves 8.6% higher economic profit than nominal ENMPC, preserves input feasibility of the deployed ENMPC policy and empirically satisfies the path constraints under disturbances and measurement noise, and drives the learned model parameters toward reference plant values, independently identified from experimental data, along the directions that most affect the economic objective. This work provides one of the first experimental demonstrations of RL-tuned ENMPC integrated with RTO.

1. Introduction

Plant–model mismatch can undermine the economic and feasibility benefits of hierarchical optimization and control. The paper proposes a two-timescale RL framework combining RTO and ENMPC, and reports higher profit than nominal ENMPC in laboratory experiments.

1. Introduction

  • Both RTO and MPC rely on process models, so model inaccuracy degrades performance as plants face fluctuating markets and changing environments.The conventional hierarchy uses a steady-state RTO model for setpoints and faster dynamic MPC for tracking and disturbance rejection.Source: §1 · p. 1
  • ENMPC integrates economic optimization with dynamic control, but model-structure errors can persist despite parameter estimation, while fully data-driven control forfeits model-based structural guarantees.These limitations motivate correcting model-based optimization directly from plant measurements.Source: §1 · p. 1
  • Modifier adaptation corrects RTO objectives and constraints from process data, but increasingly depends on plant-gradient estimates as nominal-model quality deteriorates.RL offers a route to optimize closed-loop performance from plant data without explicit gradient estimates.Source: §1 · p. 1
  • Q-learning tunes RTO parameters slowly for steady-state economics and ENMPC parameters quickly for transient dynamics, without requiring plant gradient estimates.The framework parameterizes the dynamic model, stage costs, constraints, and RTO modifiers.Source: §1 · p. 2
  • 8.6% higher profit than nominal ENMPC was achieved while preserving deployed-policy input feasibility and empirically satisfying path constraints under disturbances and measurement noise.The result comes from laboratory validation on a rig emulating a three-well subsea oil-production network.Source: §1 · p. 2

2. Background

The background reviews model-based approaches to plant–model mismatch and the use of RL to tune structured MPC policies. It contrasts their capabilities and limitations across RTO, modifier adaptation, ENMPC, and RL-based MPC.

2.1. RTO and Modifier Adaptation

  • Model parameter adaptation can fail under structural mismatch; modifier adaptation adds constraint and gradient corrections but requires costly plant-gradient estimates.Safety margins used to ensure feasibility can further reduce achievable economic benefit.Source: §2.1 · p. 2

2.2. Economic Nonlinear MPC

  • ENMPC optimizes an economic objective over a dynamic horizon, but its fixed nominal model can lose performance under plant–model mismatch.Feedback alone is generally insufficient to recover optimality.Source: §2.2 · pp. 2–3

2.3. RL-Based MPC

  • Parameterized ENMPC serves as a structured RL function approximator, tuning dynamics, costs, terminal cost, and constraints from closed-loop data despite model mismatch.The formulation embeds process knowledge and MPC theory into RL.Source: §2.3 · p. 3

3. Proposed RTO–RLMPC Framework

The framework couples RTO and ENMPC by parameterizing both layers and using reinforcement learning to adapt them on different timescales. The RTO targets the plant optimum, while ENMPC parameters address transient control behavior.

3. Proposed RTO–RLMPC Framework

  • RTO modifiers and parameterized ENMPC are tuned together, with ENMPC parameters updated at each control step and RTO modifiers once per episode.The RTO equilibrium is passed to ENMPC as a reference in its stage-cost tracking term.Source: §3 · p. 3, §3.1 · p. 4
  • Independent RL tuning of RTO and ENMPC is described as failing to ensure consistency between layers, while the joint design adapts both within one framework.A parameterized RTO with conventional tracking MPC forgoes transient economic optimization; a parameterized ENMPC with standard RTO leaves RTO unable to adapt to mismatch.Source: §3 · pp. 3–4

3.1. Parameterized RTO with Modifier Terms

  • RL updates RTO modifier terms from observed input–output trajectories, correcting plant–model mismatch without explicit plant-gradient estimation.The modifiers correct the RTO objective and constraints; their corrections vanish when the corresponding true cost or constraints are exactly known functions of the inputs.Source: §3.1 · p. 4
  • ENMPC parameters affect the action value both directly through the controller and indirectly through the RTO setpoint, while RTO-only parameters enter through the setpoint.The episode length is selected ad hoc from process knowledge of settling time: too short can mix transient data into episode averages, while too long slows RTO adaptation and wastes data.Source: §3.1 · p. 5
  • The classical two-timescale convergence conditions are not fully met: both layers use constant step sizes, and formal guarantees for this NLP-based setting remain future work.The update-frequency separation only heuristically mimics the required timescale ratio; empirically, the TD error reaches zero within approximately 150 episodes and learned parameters stabilize.Source: §3.1 · p. 5

3.3. Sensitivities for the RTO–RLMPC Update

  • Implicit differentiation of the RTO KKT residual supplies the optimizer sensitivity needed for the update, under standard regularity conditions for the ENMPC sensitivity.The RTO optimizer sensitivity is taken with respect to parameters entering the RTO; terminal-cost and stage-cost/constraint parameters do not enter that problem.Source: §3.3 · p. 5

4. Case Study: Subsea Gas-Lift Oil-Well Network

The case study uses a laboratory rig emulating a three-well subsea gas-lift network, where limited gas is allocated among wells to maximize production. The section describes its state estimation, parameterization, training data, economic formulation, and offline learning procedure.

4. Case Study: Subsea Gas-Lift Oil-Well Network

  • The optimization allocates limited lift gas among three wells to maximize oil production, with excessive injection increasing frictional losses and reducing output.The laboratory rig emulates a subsea gas-lift network; air and water substitute for natural gas and crude oil.Source: §4 · p. 6, §4.1 · p. 6

4.1. Rig Description

  • The rig comprises reservoir, well, and riser sections, and uses air and water as dynamics-preserving substitutes for natural gas and crude oil.Figure 2 identifies gas injection rates as manipulated inputs, pressure and liquid-flow measurements as outputs, and reservoir valve openings as disturbances.Source: §4.1 · p. 6, Figure 2 · p. 6

4.2. State Estimation

  • An Extended Kalman Filter estimates the full state vector from measured wellhead pressures and liquid flow rates for the ENMPC and RTO solvers.Its noise covariance matrices were calibrated from rig instrumentation noise characteristics.Source: §4.2 · pp. 6–7

4.3. Process Modeling and Parameterization

  • Adjustable parameters target reservoir-valve disturbances, while ENMPC parameterization covers economic stage costs, terminal costs, and state bounds.The state-bound offsets can widen or narrow the effective box; cost-weight matrices are projected onto the positive-semidefinite cone.Source: §4.3 · p. 7

4.4. Dataset

  • Training data comprise simulated 1200-step closed-loop episodes sampled every 10 seconds under varying gas rates and reservoir-valve settings.Each episode represents 200 minutes of simulated closed-loop data.Source: §4.4 · p. 7

4.5. Problem Formulation

  • The economic formulation maximizes oil revenue subject to gas-injection bounds and a total gas-availability limit of 7.5 sL/min.Each well’s gas injection is bounded between 1 and 5 sL/min; RL updates use negative instantaneous profit plus penalties for state-bound violations.Source: §4.5 · p. 7

4.6. Offline Learning

  • Offline learning compares standard RLMPC with RTO–RLMPC, then applies the trained policies to the experimental rig for online validation.Exploration perturbations decay across episodes, and parameter updates are projected to maintain physically meaningful ranges and well-posed ENMPC costs.Source: §4.6 · p. 8

5. Results

The experimental evaluation compares nominal ENMPC, RLMPC, and RTO–RLMPC under declining-production disturbances, then examines learning convergence and the contribution of model learning. RTO–RLMPC improves profit over nominal ENMPC, while the learning analyses show both selective parameter identification and a cost of disabling model updates.

5.1. Experimental Results

  • RTO–RLMPC tracks RTO steady-state flow targets more closely than nominal ENMPC during later disturbance phases, with the strongest improvement in well 3.Well 3 carries the highest economic weight; gas allocation shifts toward it as valve openings change.Source: §5.1 · pp. 8–9, Figure 4 · p. 9
  • RTO–RLMPC has the smallest mean absolute deviation from the 7.5 sL/min total-gas limit (0.028 sL/min) and the smallest largest exceedance (0.100 sL/min).Measured totals occasionally exceed the bound during disturbance transitions, by amounts comparable to the flowmeter noise floor; commanded totals satisfy the limit by construction.Source: §5.1 · p. 9, Figure 5 · p. 9

5.2. Learning Convergence

  • The TD error converges to zero within approximately 150 episodes, indicating consistency between the Q-function, value function, observed stage costs, and transitions.The episode-averaged error is shown over 200 episodes of offline training.Source: §5.2 · p. 9, Figure 7 · p. 10

5.1. Experimental Results

  • 8.6% higher profit than nominal ENMPC is achieved by RTO–RLMPC; RLMPC improves profit by approximately 4.7%.Profit signals use a 60 s moving average, and shaded bands represent local variability across three repeated experiments.Source: §5.1 · p. 9, Figure 6 · p. 10

5.2. Learning Convergence

  • Learned model parameters move toward independently calibrated reference values along directions that most affect the closed-loop economic objective; weakly coupled directions remain partially identified.Reservoir-valve parameters change more than top-valve parameters, while the movement penalty decreases substantially and allows more responsive disturbance rejection.Source: §5.2 · pp. 10–11, Figure 8 · p. 10, Table 2 · p. 10
  • After model parameters stabilize, the RTO modifier can continue growing without bound along a sensitivity-flat direction under the active total-gas constraint, although the optimal input remains well-behaved.The RTO parameters are unconstrained; the authors identify regularization or projection as a possible safeguard for future work.Source: §5.2 · p. 10

5.3. Ablation: Role of Model Learning

  • Without model learning, the economic objective degrades to 436 × 10^3 versus approximately 447 × 10^3 for the proposed method, and the mean TD error is 0.015 versus 0.001.The model-frozen variant settles on more conservative cost weights and trades some economic performance for improved constraint satisfaction.Source: §5.3 · p. 11

6. Conclusion

The framework integrates RTO and ENMPC through modifier-adaptation concepts, using parameterized terms to tune both layers without explicit steady-state gradient estimation. On a laboratory-scale gas-lifted oil-well network, it improved average profit over nominal ENMPC, while model parameters moved toward independently identified plant values along economically influential directions.

6. Conclusion

  • The method integrates RTO and ENMPC through modifier-adaptation concepts, with RTO modifier terms providing a steady-state optimum reference in the parameterized ENMPC stage cost.This removes the need for explicit steady-state gradient estimation typically required in modifier adaptation.Source: §6 · p. 11
  • Approximately 8.6% average profit improvement over nominal ENMPC was achieved on a laboratory-scale gas-lifted oil-well network.The framework also tracked economically optimal steady states more closely, particularly for the highest-value well.Source: §6 · p. 11
  • The temporal-difference error converged within roughly 150 episodes, and learned model parameters moved toward independently identified plant values along directions that most influence the economic objective.Directions weakly coupled to closed-loop cost remained only partially identified.Source: §6 · p. 11
  • For any fixed parameter vector, the standard dissipativity-based ENMPC stability argument applies under usual terminal-cost assumptions, but transferring certificates to online learning remains an open theoretical question.The work addresses this pragmatically with fast ENMPC updates per step and slow RTO updates per episode, empirically verified on the rig.Source: §6 · p. 11
  • Performance is bounded by parameterization richness, and training data cover limited operating conditions and disturbance patterns, leaving generalization to unseen scenarios open.The cubic scaling of NLP sensitivity computation with decision-variable count also requires decomposition or approximate sensitivities for larger networks; unbounded RTO-parameter growth motivates future regularization or projection.Source: §6 · p. 11

A. Model Equations

The dynamic model comprises mass balances and pressure/flow relations for three identical wells.

A. Model Equations

  • The model, adapted from Marchetti et al., comprises mass balances and pressure/flow relations for three identical wells.The passage introduces the mass-balance equations but does not state their full form.Source: A. Model Equations · p. 11
  • The mass-balance notation identifies holdups, injection and inflow, and outlet flows.The passage labels these quantities as m_g,l, w_g,l, and w_g,l,out, respectively.Source: A. Model Equations · p. 11
  • The pressure before injection is modeled using Darcy–Weisbach.Source: A. Model Equations · p. 11
  • Mixture density is given by total gas and liquid holdup divided by total volume, and the model includes total outlet flow.The stated relation is ρ_mix = (m_g + m_l)/V_total.Source: A. Model Equations · p. 11

CRediT authorship contribution statement

The authors divide conceptualization and methodology across the team, with supervision and project administration assigned to senior contributors.

CRediT authorship contribution statement

Loading 2609.35141v1…