PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning for Process Control: A Primer for Beginners

Steven Spielberg, Aditya Tulsyan, Nathan P. Lawrence, Philip D Loewen, R. Bhushan Gopaluni

arXiv:2004.05490v1eess.SYcs.SY

TL;DR

Industrial controllers can degrade as processes change, and model re-identification is costly and complicated. The paper develops a model-free DRL controller that learns through process interaction and demonstrates set-point tracking across several process types.

  • Problem

    Industrial controllers can lose performance as process characteristics drift or set points change, while maintaining them and re-identifying models is costly and complex.

    Source: §1 · p. 1

  • Method

    The paper develops an adaptive, model-free DRL controller that learns a set-point-tracking policy in real time through process interaction.

    Source: §10 · p. 18

  • Results

    The controller's efficacy is demonstrated in set-point tracking on SISO, MIMO, and nonlinear systems with external disturbances.

    Source: §10 · p. 18

  • Takeaways & Limitations

    The authors conclude that DRL shows promise as an alternative to traditional model-based industrial controllers, while practical and theoretical challenges remain.

    Source: §10 · p. 18

    The controller has no optimality guarantees and is scoped to discrete-time dynamics and full-state observation, both identified as challenges for further development.

    Source: §9 · p. 18

Abstract

from arXiv · show

Advanced model-based controllers are well established in process industries. However, such controllers require regular maintenance to maintain acceptable performance. It is a common practice to monitor controller performance continuously and to initiate a remedial model re-identification procedure in the event of performance degradation. Such procedures are typically complicated and resource-intensive, and they often cause costly interruptions to normal operations. In this paper, we exploit recent developments in reinforcement learning and deep learning to develop a novel adaptive, model-free controller for general discrete-time processes. The DRL controller we propose is a data-based controller that learns the control policy in real time by merely interacting with the process. The effectiveness and benefits of the DRL controller are demonstrated through many simulations.

1. Introduction

Industrial controllers can lose performance as processes drift, while model re-identification is costly and complicated. The paper motivates a model-free DRL approach that learns a control policy through process interaction.

1. Introduction

  • Performance can erode as process characteristics drift or set points change, making long-term maintenance essential and typically complicated and expensive.Source: §1 · p. 1
  • Model-based controllers need high-quality process models; re-identification can take two weeks or more, require external excitation, interrupt operations, and complicate multivariable processes.Source: §1 · p. 1

1.1. Reinforcement Learning and Process Control

  • Online nonlinear adaptation is difficult amid hidden states and high dimensionality; for infinite MDPs, Q-learning may converge suboptimally or make online action optimization impractical.The cited discussion also identifies computational complexity as a contributing challenge.Source: §1 · p. 1, §1.1 · p. 3
  • Prior RL methods established feasibility for nonlinear process control but often assumed process models, limiting use where accurate models are unavailable.Source: §1.1 · p. 2

1.2. Deep Reinforcement Learning (DRL)

  • The paper proposes a model-free, off-policy actor-critic DRL controller using deep networks to generalize across continuous states and actions, with a tutorial-style presentation for newcomers.Source: §1.2 · p. 3

2. The Reinforcement Learning (RL) Problem

The RL problem models an agent interacting with an environment to learn a policy that maximizes expected cumulative reward. The section introduces MDPs, value functions, and the assumptions that limit exact solutions in process control.

2. The Reinforcement Learning (RL) Problem

  • An RL agent observes a state, chooses an action, then receives a new state and scalar reward from the environment, repeating across steps or until an episode ends.Source: §2 · pp. 3–4

2.1. Markov Decision Process (MDP)

  • An MDP specifies state and action spaces, an initial-state distribution, transitions, and rewards; the Markov property makes the next-state distribution depend on the current state and action.Source: §2.1 · p. 4
  • The discount factor γ controls how strongly future rewards count: γ = 0 gives weight only to immediate reward, while values approaching 1 emphasize future rewards.Source: §2.1 · p. 4

2.2. Value Function and Optimal Policy

  • The state-value function evaluates expected future reward from a state under a policy, while the action-value function supports recovering an optimal policy through greedy action choice.Source: §2.2 · p. 4
  • Exact Bellman solutions require known environment dynamics and sufficient computation, assumptions that hinder process control when models are uncertain or changing and spaces are continuous.Source: §2.2 · pp. 4–5

3. Q-learning

The section presents Q-learning as a way to learn the optimal action-value function directly, then extends it with function approximation for continuous state and action spaces. The extension enables generalization but lacks general convergence guarantees and can be unstable.

3. Q-learning

  • Q-learning learns Q⋆ directly through successive approximations, rather than first estimating a policy-specific Q-function and then deriving the optimum.It uses the Bellman identity for Q⋆ as its basis.Source: §3 · p. 5
  • TD learning estimates Q⋆ from sampled experience tuples, replacing the Bellman expectation with a sample-based update.The update uses a learning rate and TD error; policy evaluation and improvement then produce a greedy policy.Source: §3 · p. 5
  • Algorithm 1 is model-free, online, and off-policy; its ϵ-greedy behavior policy supports exploration, and its Q estimates converge to Q⋆ with probability 1.It updates from the latest experience tuple while learning a policy distinct from the behavior policy.Source: §3 · p. 5

3.1. Q-learning with Function Approximation

  • Function approximation represents Q(s, a) with parameters and a quadratic loss, addressing the storage burden of Q-values across infinite state and action spaces.Discretization may be inefficient for complex industrial dynamics.Source: §3.1 · p. 5
  • With function approximation, the unknown Q⋆ target can be approximated by a bootstrap target and the loss minimized by stochastic gradient descent.The TD target depends on function-approximation parameters; deriving the update assumes the target is independent of those parameters.Source: §3.1 · p. 6
  • Off-policy learning, bootstrapping, and function approximation form the “deadly triad”: together they can cause divergence, though each supports practical learning.They respectively enable decoupling behavior from target policies, computational and data efficiency, and scalability and generalization.Source: §3.1 · p. 6

4. Policy Gradient

Policy-gradient methods optimize a parameterized policy directly, using a gradient theorem to estimate updates without knowing the process’s state distribution. REINFORCE handles continuous actions but requires episode returns and can have high-variance estimates that slow convergence.

4. Policy Gradient

  • Q-learning with function approximation lacks general convergence guarantees and requires Q-function maximization that is impractical for large unconstrained approximators and continuous actions.The stated exception to the convergence-guarantee limitation is linear function approximation.Source: §4 · p. 6
  • Policy-gradient methods optimize policy parameters directly instead of estimating Q⋆ and using it to select actions.The section describes a parameterized policy that maximizes the value of starting from a given state.Source: §4 · pp. 6–7
  • The policy-gradient theorem gives a closed-form gradient without requiring the usually unknown state distribution, and expresses it using the Q-function and policy gradient.The stated form assumes a stochastic policy and is on-policy.Source: §4 · p. 7
  • REINFORCE estimates the policy gradient using sampled returns, avoids continuous-action optimization, and supports deterministic or stochastic policies.Its estimate is guaranteed to converge to a local optimum when the gradient estimate is unbiased.Source: §4 · p. 7
  • REINFORCE is not amenable to online implementation because it needs the episode’s total return, and return-based gradient estimates have high variance that slows convergence.A low-variance bootstrapped Q-function estimate is offered as an actor-critic alternative.Source: §4 · p. 7

5. Actor-Critic Architecture

The actor-critic architecture pairs a policy-updating actor with a critic that estimates the policy’s Q-function. Its deterministic variant uses state-based gradients and can be implemented off-policy to support exploration.

5. Actor-Critic Architecture

  • The actor-critic combines policy gradient with Q-learning: the actor adjusts policy parameters, while the critic approximates the Q-function and evaluates the actor’s policy.Under certain conditions, the actor update with the critic’s approximation guarantees convergence to a local optimal policy.Source: §5 · p. 7

5.1. Deterministic Actor-Critic Method

  • Deterministic policy gradients use a deterministic policy that maps each state to an action, avoiding the stochastic-policy gradient’s integration over states and actions.The deterministic gradient involves an expectation over states generated under the policy, making its computation more efficient than the stochastic policy gradient.Source: §5.1 · pp. 7–8
  • Off-policy deterministic policy gradients use states generated by a stochastic behavior policy, allowing exploration while learning a deterministic policy.The actor updates using the off-policy gradient, and the critic recursively estimates the Q-function; Algorithm 4 gives pseudocode.Source: §5.1 · p. 8

6. Deep Reinforcement Learning (DRL) Controller

The proposed model-free DRL controller uses an actor–critic architecture with deep networks to learn a control policy through interaction with a process. The section specifies its state and reward design, learning mechanisms, implementation choices, and tuning guidance for set-point tracking.

6.1. States, Actions and Rewards

  • The RL state combines current and past outputs, past actions, and current set-point deviation, while the two proposed rewards score tracking error or its monotonic reduction.The discussion assumes outputs fully represent the process state and that the required history lengths are known a priori.Source: §6.1 · pp. 8–9

6.2. Policy and Q-function Approximations

  • Deep neural networks approximate the policy and Q-function over continuous state and action spaces.The policy network maps a state to an action, while the Q-network evaluates a state–action pair.Source: §6.2 · p. 9

6.3. Learning Control Policies

  • The controller learns through a deterministic off-policy actor–critic: the actor proposes actions, the critic estimates their Q-values, and process interaction supplies training experience.After training, the actor computes actions for given states.Source: §6.3 · p. 9
  • Replay memory and slowly updated target networks address correlated training samples and unstable target estimates during actor–critic learning.The method uniformly samples stored tuples for batch updates; target-network parameters track the critic parameters gradually to make target values change more slowly.Source: §6.3 · pp. 9–10
  • Gradient clipping keeps controller actions within their feasible interval by scaling or reversing gradients near or beyond action bounds.The section assumes interval-valued action spaces and notes that tight input limits can speed convergence.Source: §6.3 · p. 10

6.4. Network Structure and Implementation

  • The implementation uses two-hidden-layer actor and critic networks, batch normalization, and Adam, while hyperparameters are process-specific and require careful selection.The paper states that optimal hyperparameter selection is beyond its scope.Source: §6.4 · p. 11

6.5. DRL Controller Tuning

  • Tuning guidance covers episode length, rewards, state design, replay memory, learning, and exploration; it recommends stopping updates after tracking completes and stopping exploration once networks are trained.For processes with large time constants, episodes should run longer; the paper also notes that tighter input bounds generally lead to faster convergence.Source: §6.5 · p. 12

7. Simulation Results

Four simulations evaluate the DRL controller on paper-machine, distillation-column, and HVAC processes, as well as its response to an abrupt process change. The reported results include set-point tracking and real-time policy relearning.

7.1. Example 1: Paper Machine

  • In the paper-machine simulation, basic tracking was learned in under 100 episodes, followed by generalization to sinusoidal and random set-points, including an action outside [0, 100].The reported results include tracking random set-points both inside and outside the training output space.Source: §7.1 · p. 13, Figure 5 · p. 13, Figure 6 · p. 14

7.2. Example 2: High-purity Distillation Column

  • From some initial conditions, the DRL controller tracked both distillate and bottom-product set-points within 75 minutes on an ill-conditioned MIMO column.To speed learning, the implementation restricted set-point pairs to values in [0, 5] differing by no more than 0.5.Source: §7.2 · p. 14, Figure 8 · p. 15, Figure 9 · p. 16

7.3. Example 3: HVAC System

  • With polar reward (28), HVAC tracking was smoother and offset-free; discharge-air temperature followed changing set-points in under 50 seconds with small offset.Using the ℓ1 reward (27) left a persistent offset despite increasing episode length.Source: §7.3 · p. 16, Figure 10 · p. 16

7.4. Example 4: Robustness to Process Changes

  • About 400 seconds after a process-gain doubling, the DRL controller learned a new policy for the same set-point.The learned policy had become invalid after the change, and the system began to deviate before relearning.Source: §7.4 · p. 17

8. Comparison with Model Predictive Control

The DRL controller contrasts with MPC by learning from process interaction rather than requiring a process model, while handling constraints, future rewards, and adaptation differently. Its discussion assumes full state availability and flags safe operation as an open issue.

8. Comparison with Model Predictive Control

  • The DRL controller needs no a priori process model and updates actor and critic parameters at each sampling time using recent experiences.MPC instead uses model-based predictions and optimization; its response to model-plant mismatch can involve costly model re-identification.Source: §8 · pp. 17–18
  • DRL constraints can be embedded in rewards or discouraged through gradient clipping, but violations may occur during training and safe operation remains a priority.MPC explicitly incorporates constraints into its optimization at each step.Source: §8 · p. 17
  • DRL's discount factor γ determines the present value of future rewards, corresponding to MPC's user-specified prediction horizon.Source: §8 · p. 17
  • This paper considers DRL when the full system state is available for policy improvement; filtering or observers are left to future work.Source: §8 · p. 17
  • Figure 12 presents Example 4 set-point tracking under abrupt process changes and the DRL controller's action.Source: Figure 12 · p. 18

9. Limitations

The DRL controller lacks the established optimality guarantees available for several MPC formulations. Its stated scope assumes discrete-time dynamics and full state observation, and several practical and theoretical issues remain open.

9. Limitations

  • Unlike several established MPC formulations, the DRL controller currently has no optimality guarantees.Source: §9 · p. 18
  • Discrete-time dynamics and full state observation are explicit challenges for further investigation and development.Source: §9 · p. 18
  • Other open issues include data requirements, computational complexity, non-convexity, fitting, performance improvement, and hyper-parameter initialization.Source: §9 · p. 18
  • For continuous-time systems, the stated temporal-difference error formulation requires complete knowledge of system dynamics.Source: §9 · p. 18

10. Conclusions

The paper develops an adaptive, model-free DRL controller for set-point tracking in discrete-time nonlinear processes. Simulations demonstrate efficacy across SISO, MIMO, and nonlinear systems with external disturbances, though practical and theoretical challenges remain.

10. Conclusions

  • The controller is developed for set-point tracking in discrete-time nonlinear processes using reinforcement learning and deep learning.Source: §10 · p. 18
  • Its efficacy is demonstrated in set-point tracking problems on SISO, MIMO, and nonlinear systems with external disturbances.Source: §10 · p. 18
  • The authors describe the controller as promising as an alternative to traditional model-based industrial controllers, while noting unresolved practical and theoretical challenges.Source: §10 · p. 18
  • With increasing computing power and data volumes, DRL controllers have the potential to become an important process-control tool.Source: §10 · p. 18

Basics of Neural Networks

A neural network maps inputs to outputs through layered relations shaped by weight matrices and activation functions. Training selects weights to reduce discrepancies between network-produced pairs and examples the network is expected to learn.

Basics of Neural Networks

  • A neural network is a mapping from an input space to an output space, with internal structure organized in layers.Source: Basics of Neural Networks · p. 18
  • Each layer uses a weight matrix to map inputs to outputs and an elementwise activation function, typically nonlinear.Source: Basics of Neural Networks · p. 18
  • Layer-specific weights and activation functions define the network, while weight sparsity patterns determine connections between successive layers.Source: Basics of Neural Networks · p. 18
  • Training optimizes weight matrices to reduce discrepancies from expected input-output pairs; gradient descent is described as the practical iterative approach.Source: Basics of Neural Networks · p. 19
Loading 2004.05490v1…