PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning with Shallow Controllers: An Experimental Application to PID Tuning

Nathan P. Lawrence, Michael G. Forbes, Philip D. Loewen, Daniel G. McClement, Johan U. Backstrom, R. Bhushan Gopaluni

arXiv:2111.07171v1eess.SYcs.LGcs.SY

TL;DR

Real-world implementation of deep RL for process control remains challenging despite reported simulation results. This paper trains a PID controller as the RL policy on a physical system and reports improved tracking against existing tuning baselines, with training reaching peak performance in under 40 minutes in the described experiments.

  • Problem

    Real-world process-control applications of RL remain sparse, while implementation raises software-integration, stability, and sample-efficiency challenges.

    Source: §1 · p. 2, §7 · p. 25

  • Method

    The paper directly parameterizes the RL policy as a PID controller and uses a model-free deep RL algorithm to tune it on a physical system.

    Source: §1 · p. 2, §7 · p. 25

  • Results

    All three experiments outperformed Accutune III across the reported metrics except settling time in the constrained RL experiment, and peak performance was reached in under 40 minutes of operation.

    Source: §6.3 · pp. 21–22

  • Takeaways & Limitations

    A PID policy uses substantially fewer parameters than neural-network policies and can use existing controller hardware rather than requiring new hardware for the RL policy.

    Source: §7 · p. 25

    The paper identifies case-by-case configuration of RL agents for novel environments, particularly after learning failure, as an unresolved challenge.

    Source: §7.1 · p. 26

Abstract

from arXiv · show

Deep reinforcement learning (RL) is an optimization-driven framework for producing control strategies for general dynamical systems without explicit reliance on process models. Good results have been reported in simulation. Here we demonstrate the challenges in implementing a state of the art deep RL algorithm on a real physical system. Aspects include the interplay between software and existing hardware; experiment design and sample efficiency; training subject to input constraints; and interpretability of the algorithm and control law. At the core of our approach is the use of a PID controller as the trainable RL policy. In addition to its simplicity, this approach has several appealing features: No additional hardware needs to be added to the control system, since a PID controller can easily be implemented through a standard programmable logic controller; the control law can easily be initialized in a "safe'' region of the parameter space; and the final product -- a well-tuned PID controller -- has a form that practitioners can reason about and deploy with confidence.

1. Introduction

Real-world deep RL process-control applications remain sparse, with integration, stability, and sample efficiency among the challenges. This work evaluates model-free deep RL as an industrially oriented PID auto-tuning strategy and details its physical implementation.

1. Introduction

  • Real-world RL process control faces formidable software-integration and algorithmic challenges, including closed-loop stability and sample efficiency.The introduction notes that applications in the field remain sparse.Source: §1 · p. 2
  • The paper extends prior work to a physical system by treating a PID controller as the RL policy and evaluates the approach against auto-tuning criteria and baselines.It aims to provide a guide to implementing RL in real-world process control.Source: §1 · p. 2

1.1. Related work

  • Earlier process-control RL studies included model-based applications, while later work explored deep RL, transfer learning, and RL methods that tune or select PID gains.The cited examples include simulated or pre-trained approaches and methods assuming an identified model.Source: §1.1 · p. 3
  • Only a handful of studies on RL for process control contain real-world validation, and this work uses neither a simulation model nor offline pre-training.It also outlines evaluation criteria and provides hardware, software, and algorithmic implementation details.Source: §1.1 · pp. 3–4

2. Methodology

The methodology frames PID gains as trainable policy parameters in an actor-critic RL approach. It applies this formulation to discrete-time PID control, with the PID update represented as a shallow network.

2.1. Reinforcement learning

  • The RL formulation seeks a policy maximizing expected long-term reward, and the paper focuses on deterministic policies mapping states to actions.Deep RL uses neural networks as function approximators for continuous state and action spaces.Source: §2.1 · pp. 5–6
  • The actor-critic approach estimates the state-action value function with a critic network and uses its actor loss for policy updates; the implementation uses modified TD3.TD3 is described as a refinement of DDPG.Source: §2.1 · pp. 6–7

2.2. PID in the RL framework

  • The PID formulation uses tracking error and signal differences, including filtered output differences for derivative information and an anti-windup term based on the discrepancy between proposed and saturated input.The controller’s proposed output is clipped to produce the physically admissible input sent to the plant.Source: §2.2 · pp. 7–8
  • The method treats PID gains as trainable policy parameters and casts the PID controller as a shallow neural network.The PID update is differentiated through the actor and critic losses using the chain rule and can be computed automatically.Source: §2.1 · p. 6, §2.2 · p. 9

3. Scorecard for RL algorithms in process control

The scorecard evaluates RL tuning beyond final tracking performance, including training behavior, system perturbation, and practical implementation. It uses established tracking and input-variation metrics to quantify outcomes.

Scorecard criteria

  • The proposed evaluation criteria cover nominal performance, stability during and after training, system perturbation, initialization sensitivity, hyperparameters, training duration, and practicality.These assess both the path to the final controller and the requirements for using it.Source: §3 · p. 9
  • The scorecard also asks how many episodes are needed for satisfactory performance and what hardware and user expertise deployment requires.These criteria address training duration and practicality or specialization.Source: §3 · pp. 9–10

Quantitative measures

  • The quantitative measures include integral absolute error, integral squared error, total variation, overshoot, and settling time.The paper notes that many criteria have counterparts in existing tuning methods for comparison.Source: §3 · p. 10

Evaluation scope

  • The evaluation considers the RL agent’s path to its final form, its perturbations to the process, usability, and software and hardware requirements.Total variation of the input is denoted TVu, and percent overshoot is scaled by the initial setpoint change.Source: §3 · p. 10

4. Lab setup and software structure

The paper describes a physical two-tank level-control setup and a modular software arrangement for running RL with standard hardware. The HMI manages control and data collection, while the RL code processes data and updates PID parameters.

4.1. Description of the two-tank system

  • The laboratory system controls the level in an upper tank supplied by a pump, with a lower tank serving as the reservoir.The level controller generates a flow reference for a separate flow controller; the level PID is tunable while the flow PID is fixed.Source: §4.1 · pp. 10–12

4.2. Human-machine interface

  • The HMI adjusts PID parameters, runs experiments, saves process data, and embeds a simulator that supports switching between simulation and reality.The embedded simulator was not used to pre-train the RL agent.Source: §4.2 · p. 13

4.3. Software and hardware for training RL agents

  • The software is modular: the RL agent does not directly actuate the system, and the HMI and instrumentation interface with the physical process.The actor representation is implemented in PyTorch, while a hardware PLC implements PID control; trained parameters are communicated to it.Source: §4.3 · p. 13
  • The implementation adapts the actor to the linear PID form and constrains gains to be positive, with the anti-windup parameter constrained to (0, 1).Softplus and Sigmoid transformations respectively enforce these parameter ranges.Source: §4.3 · pp. 13, 15
  • The RL code processes newly saved tank data in batches, updates its PID representation, and returns parameters for the HMI to read.The agent can be trained on a lab desktop, and the modular setup also permits cloud-based training.Source: §4.3 · pp. 15–16
  • The HMI retains online experiment-design settings, while the RL code processes new data in the background.The HMI can change design parameters while the RL code is running.Source: §4.3 · p. 15

5. Experiment design and case studies

The experimental design uses predefined setpoint cycles, measured-data updates to PID parameters, and a stabilizing fallback. The section also describes reward choices, actor updates under input constraints, and baseline tuning methods.

5.1. Experiments for learning

  • Learning experiments cycle through predefined setpoints, update PID parameters from measured data, and retain a stabilizing fallback when tracking is poor.A fixed timer bounds the time before switching to safe PID parameters unless the system reaches steady state sooner; this ensures the next step change starts from steady state.Source: §5.1 · p. 16

5.2. Reward function selection

  • The cost penalizes tracking error and changes in the control input, with fixed powers p and q and a penalty weight λ; the experiments use 0.1(∆u_t)^2 for the input penalty.The suggested initial weight normalizes the input-change penalty using an approximation of the maximum observed change in the operating region.Source: §5.2 · p. 17
  • The hybrid cost combines absolute-error behavior near the origin with a smooth transition to a parabola, offering a choice between slower, less oscillatory and faster responses.The text contrasts absolute error, which emphasizes small errors, with squared error, which weights large errors more heavily.Source: §5.2 · pp. 17–18

5.3. Tuning subject to input constraints

  • Because physical systems impose input constraints, the actor update is modified to steer actions into a predefined range and reduce infeasible proposals near the limits.The alternative of bounding actions with an actor activation can diminish gradients at the constraints; the update scheme affects training, while physical control still obeys the limits.Source: §5.3 · p. 18

5.4. Standard tuning methods

  • The comparison uses SIMC and Accutune III as performance and robustness baselines, evaluated alongside RL with the paper’s scorecard rather than a shared objective.SIMC derives PI parameters from a first-order-plus-dead-time plant model and uses the closed-loop time constant as its tuning parameter; Accutune III is relay autotuning with fast or slow settings.Source: §5.4 · p. 19

6. Lab results

The experiments evaluate RL-based PID tuning directly on the physical two-tank system, without simulation or offline pretraining. Across the reported operating conditions, RL tuning achieved strong tracking results, with training durations and practical trade-offs assessed against conventional tuning methods.

6.1. Experimental setup

  • Experiments ran on the physical two-tank system without pretraining; the unconstrained runs began from different PID parameters, while the constrained run trained with input constraints.The constrained setup included setpoint changes both with and without input constraints.Source: §6 · p. 19, §6.1 · pp. 19–20

6.2. Evaluation procedure

  • Evaluation cycled through 60, 65, 60, and 63 cm setpoints for four minutes each, with input constraints inactive to compare PID parameter quality across methods.Performance measures included normalized IAE, ISE, TV, and TVu, along with overshoot and settling time.Source: §6.2 · p. 20

6.3. Summary of results

  • All three RL experiments beat Accutune III across metrics except settling time for constrained RL; RL and SIMC had similar TV, TVu, and overshoot, while RL improved IAE, ISE, and settling time.The table reports average normalized performance across experiments; the absolute-value reward was judged the overall best option.Source: §6.3 · pp. 20–22, Table 1 · p. 21, experimental robustness · p. 23
  • Unconstrained RL performance plateaued at around 10 episodes, and all experiments reached peak performance in less than 40 minutes of operation.The episodes were roughly four minutes each; training heatmaps showed reduced overshoot but slightly longer rise time.Source: §6.3 · p. 22
  • The constrained and unconstrained runs starting at kp = 4.0 had similar evaluation performance, except unconstrained RL achieved roughly 40% better settling time.The constrained run was trained under more realistic conditions and showed excellent qualitative tracking with input constraints in place.Source: §6.3 · p. 22, Figure 5 · p. 23

6.4. Overall evaluation

  • Deep RL achieved excellent nominal performance across initializations, reward functions, and input constraints, but took 30–50 minutes versus roughly 10 minutes for Accutune III.The authors consider closed-loop training more practical than open-loop tuning when an acceptable setpoint range can be provided.Source: §6.4 · pp. 24–25

7. Discussion and conclusion

The discussion argues for hybrid RL approaches that build on established PID or MPC architectures, and presents a PID controller as a practical RL policy. The experiments show monotonic improvement in IAE and ISE within 30–50 minutes, while systematic configuration for novel environments remains an open challenge.

7. Discussion and conclusion

  • Given DNN policies’ difficulty for rigorous behavioral explanation and the process industries’ investment in MPC and PID, the authors favor hybrid RL approaches.They connect DNN nonlinear structure in closed loop with challenges in sample efficiency, interpretability, and stability.Source: §7 · p. 25
  • A PID policy required orders of magnitude fewer tunable parameters than approaches using a DNN or value-based policy to output PID parameters.Because PID controllers are standard in universal digital controllers, deployment required sending new parameters rather than installing additional hardware.Source: §7 · p. 25
  • IAE and ISE improved monotonically, yielding a well-tuned PID controller within 30–50 minutes.The critic was trained from scratch, without pre-training from simulation or historical process data.Source: §7 · p. 25
  • The algorithm ran in closed loop on the system, using a local desktop computer and no specialized hardware.The authors describe these deployment results as encouraging.Source: §7 · p. 26

7.1. Opportunities in deep RL

  • Systematically configuring an agent for a novel environment after failed learning remains an open question, since training is case-by-case.The authors describe offline RL on historical data and meta-RL across similar systems as promising directions.Source: §7.1 · p. 26
  • Offline training is framed as a safety precaution, while learning across systems could reduce calibration costs and increase scalability.The authors identify reliable hyperparameter choice after poor training performance as an outstanding issue.Source: §7.1 · p. 26

Appendix A. Further algorithmic and implementation details

The training procedure separates physical process interaction through a PLC from Python-side learning updates. A recurrent critic is trained from replay-memory transitions, while the PID gains form the actor policy.

Appendix A. Further algorithmic and implementation details

  • The pseudocode initializes target actor and critic parameters from their corresponding live parameters before running episodes.Episodes observe the environment state, execute the PLC policy, and observe the resulting next state.Source: Appendix A · p. 32
  • The PLC executes the actor’s control action and returns the next state, while process measurements are stored for later training updates.When updating, measurements are converted into structured transition tuples and placed in replay memory.Source: Appendix A · pp. 32–34
  • The algorithm samples transition batches from replay memory, computes critic targets, updates critic and policy weights, and soft-updates target parameters.Training continues until the next step change or the system reaches steady state.Source: Appendix A · p. 32
  • The PID controller is the actor policy, while the critic approximates the Q-function using replay-memory training data.The critic combines a feedforward network with a GRU that processes state information before the hidden state and action enter a DNN.Source: Appendix A · pp. 32–33
  • The recommended hyperparameter settings are listed in Table A.3, and the main design choice the authors modified was the critic network.They also adjusted actor and critic learning rates and policy update delay; other hyperparameters followed Achiam’s recommended defaults.Source: Table A.3 · p. 33, Appendix A · p. 33

Appendix B. Further experiments and discussion

The appendix compares final tuning experiments across reward functions and controller baselines using performance statistics and heatmaps. It reports better and more consistent performance for absolute-value rewards than squared-error rewards, with hybrid rewards sometimes matching or exceeding the absolute-value results.

Appendix B. Further experiments and discussion

  • Table B.4 identifies experiment labels and specifications, including the five experiments discussed in Section 6.The appendix reports performance statistics in Table B.5 and visualizes lab results in Figure B.6.Source: Table B.4 · p. 34, Appendix B · p. 34
  • The end-of-training comparison evaluates four reward functions against the metrics defined in Equation (13).The appendix includes Equation (27) and the hybrid cost function from Equation (24) among the reward functions considered.Source: Appendix B · p. 34
  • The absolute-value reward performed better and more consistently than squared-error rewards, while the hybrid reward had similar and sometimes superior nominal performance.Figure B.6 visualizes final process-variable performance across experiments; darker shades indicate more time spent at the corresponding y-axis value, with setpoints at 60 cm and 65 cm.Source: Appendix B · p. 35, Figure B.6 · p. 35

Appendix B.1. Experimental robustness

The robustness appendix evaluates final tuning parameters under two independent changes to the two-tank system and aggregates those results with nominal performance. Across comparisons, the authors favor the reward in Equation (27) overall for performance, generalization, and robustness.

Appendix B.1. Experimental robustness

  • Robustness tests repeated the step sequence after independently restricting top-tank outflow to 50% or halving the flow-controller proportional gain.Table B.5 averages each statistic across four step changes and across nominal performance plus the two robustness experiments, reporting standard deviations.Source: Appendix B.1 · p. 36, Table B.5 · p. 36
  • RL-2 and RL-3 were superior to RL-1 and RL-4, respectively, across categories in average and standard deviation, except for ISE between RL-1 and RL-2.Both were among the most promising results among RL-1 through RL-4.Source: Appendix B.1 · pp. 36–37
  • Experiments based on Equation (27) were judged the overall best option for performance, generalization between training experiments, and robustness.For constrained experiments, RL-8 and RL-10 had similar TV, TVu, and %OS scores, while RL-10 was slightly better in IAE, ISE, and ST.Source: Appendix B.1 · p. 37
  • RL-6 achieved the best IAE and ISE among hybrid-reward experiments, but had worse overshoot than RL-2.RL-7 showed a wider detrimental change in IAE and ISE from its initial tuning than RL-2 and RL-3.Source: Appendix B.1 · p. 37

Related papers

Loading 2111.07171v1…