Source-linked AI summary
A Benchmark Environment Motivated by Industrial Control Problems
Daniel Hein, Stefan Depeweg, Michel Tokic, Steffen Udluft, Alexander Hentschel, Thomas A. Runkler, Volkmar Sterzing
TL;DR
Industrial control lacks a test bed combining high dimensionality, complex stochastic behavior, and relevant batch-RL settings. The paper presents the experience-driven Industrial Benchmark and prototype setups; it argues that the IB's combination of properties makes it a useful addition to RL benchmarks.
Problem
Industrial control lacks a test bed combining high dimensionality and complex heteroscedastic stochastic behavior, while batch-RL settings are also relevant.
Source: §I · p. 1Method
The paper presents an open-source industrial benchmark with motivated dynamics and prototype experimental setups for industrial-control scenarios.
Source: §I · pp. 1–2Results
The authors argue the IB is a useful addition because its combination of high dimensionality and complex stochasticity appears novel among existing benchmarks.
Source: §II · p. 2, §V · p. 7Takeaways & Limitations
The IB and its prototype setups provide a test bed for studying scenarios common in real-world industrial control.
Source: §I · p. 2, §V · p. 7The IB is not designed to approximate any real system; it aims to pose comparable hardness and complexity.
Source: §I · p. 1
Abstract
from arXiv · showhide
In the research area of reinforcement learning (RL), frequently novel and promising methods are developed and introduced to the RL community. However, although many researchers are keen to apply their methods on real-world problems, implementing such methods in real industry environments often is a frustrating and tedious process. Generally, academic research groups have only limited access to real industrial data and applications. For this reason, new methods are usually developed, evaluated and compared by using artificial software benchmarks. On one hand, these benchmarks are designed to provide interpretable RL training scenarios and detailed insight into the learning process of the method on hand. On the other hand, they usually do not share much similarity with industrial real-world applications. For this reason we used our industry experience to design a benchmark which bridges the gap between freely available, documented, and motivated artificial benchmarks and properties of real industrial problems. The resulting industrial benchmark (IB) has been made publicly available to the RL community by publishing its Java and Python code, including an OpenAI Gym wrapper, on Github. In this paper we motivate and describe in detail the IB's dynamics and identify prototypic experimental settings that capture common situations in real-world industry control problems.
I. INTRODUCTION
Industrial control lacks a test bed combining properties such as high dimensionality and complex stochastic behavior. The paper presents the Industrial Benchmark (IB), describes its place among existing benchmarks, and outlines prototype setups relevant to industrial control.
I. INTRODUCTION
- Industrial RL could target greater energy efficiency and lower emissions, but experiments on complex, expensive systems entail risk and cost.Source: §I · p. 1
- Industrial control lacks a test bed combining high dimensionality with complex heteroscedastic stochasticity, and relevant settings include batch RL rather than exploration.Source: §I · p. 1
- The open-source IB is experience-driven, models high-dimensional continuous and partially latent control with heteroscedastic noise and delayed multi-objective reward, but does not approximate a real system.The benchmark is intended to pose comparable hardness and complexity to tasks such as finding optimal turbine settings.Source: §I · pp. 1–2
- The paper presents detailed benchmark dynamics and prototype experimental setups relevant to industrial control.The authors aim to encourage study of scenarios common in real-world situations.Source: §I · p. 2
II. PLACEMENT OF THE INDUSTRIAL BENCHMARK IN THE RL BENCHMARK DOMAIN
- The IB combines high dimensionality with complex stochasticity, a combination the authors present as novel among existing continuous-control benchmarks.The comparison considers both stochastic observations and stochastic transitions on hidden variables.Source: §II · p. 2, Figure 1 · p. 2
III. DETAILED DESCRIPTION
The IB exposes continuous steering actions and a partial observation of its Markovian state. Its setpoint shapes the dynamics, while reward is a known function of the successor state.
III. DETAILED DESCRIPTION
- Each 3D action in [-1, 1]^3 proposes changes to velocity, gain, and shift, whose values are each bounded by [0, 100].Source: §III · p. 2
- The unchangeable setpoint affects dynamics and stochasticity; holding it constant makes it a hyperparameter, while varying it produces highly non-stationary dynamics.Source: §III · p. 3
- The IB computes reward deterministically from the successor state and assumes the reward function and all variables influencing it are known and observable.Source: §III · p. 3
- Observations expose steerings, setpoint, consumption, and fatigue; the agent may use observation and action history to estimate the partially observed Markovian state.Source: §III · p. 3
- The benchmark dynamics comprise operational cost, mis-calibration, and fatigue sub-dynamics.Source: §III · p. 3
A. Dynamics of operational cost
The benchmark couples a delayed, blurred operational-cost signal with mis-calibration dynamics designed to reward periodic shift oscillation around a setpoint-defined operating point.
A. Dynamics of operational cost
- Operational cost depends on setpoint, velocity, and gain, and its observed signal is delayed and blurred by convolution.The paper relates this behavior to nonlinear, multi-influence industrial processes such as heating during combustion.Source: A. Dynamics of operational cost · p. 3
B. Dynamics of mis-calibration
- Mis-calibration dynamics use setpoint and shift to drive latent domain, response, and direction states toward periodic shift oscillation around a setpoint-defined operating point.The direction index cycles within −6 to 6; returning to the safe zone as the cycle completes resets the dynamics.Source: B. Dynamics of mis-calibration · pp. 3–4
- A sine-shaped shift trajectory is the optimal mis-calibration policy, while leaving the safe zone can initiate a positive- or negative-direction cycle.The policy may choose when to leave the safe zone; if it reaches the direction boundary, the response reverses and the direction moves back toward zero.Source: B. Dynamics of mis-calibration · pp. 3–4
- Mis-calibration adds to convoluted operational cost before heteroscedastic observation noise produces visible consumption.The noise is Gaussian with zero mean and standard deviation σ = 1 + 0.02 ĉ_t.Source: B. Dynamics of mis-calibration · p. 4
C. Dynamics of fatigue
The fatigue dynamics create a competing reward objective: reducing operational cost through velocity and gain changes increases fatigue. Fatigue can also rise sharply through latent-variable amplification and bifurcation.
C. Dynamics of fatigue
- Reducing operational cost by changing velocity and gain increases fatigue, creating opposing dependencies in the multi-criteria task.Fatigue is one of the two components of the final negative reward shown in the rollout.Source: C. Dynamics of fatigue · pp. 4–5, Figure 4 · p. 6
- Fatigue is computed from basic fatigue using an amplification that depends on latent variables, effective velocity and gain, and noise.Setpoint-dependent transformations determine effective velocity and gain, which also affect the noise and latent variables.Source: C. Dynamics of fatigue · p. 5
- When either latent variable μ_v or μ_g reaches 1.2, amplification can bifurcate, increasing fatigue and negatively affecting reward.In the illustrated rollout, the latent variable μ_g self-amplifies before fatigue shifts rapidly to higher, less noisy regions.Source: C. Dynamics of fatigue · pp. 5–6
- In the example fatigue dynamics, binomial and exponential noise components combine from t = 0 to t = 80 to produce heterogeneous spike-like behavior.From t = 80 to t = 100, µg self-amplifies; around t = 100, fatigue dynamics rapidly shift toward higher, less noisy regions.Source: C. Dynamics of fatigue · p. 6
D. Setpoint dynamics
The setpoint may be fixed or vary over time; in the variable setting, it changes at a constant rate over sampled intervals, with the rate flipped at bounds.
D. Setpoint dynamics
- In the variable setting, the setpoint changes by a constant value over a fixed period, after which a new sequence length and change rate are determined.Sequence length is sampled uniformly from U{1, 100}; the rate is drawn from a mixture of U(0, 1) and a zero-change delta distribution, weighted 0.9 and 0.1.Source: D. Setpoint dynamics · p. 6
- When the setpoint reaches 0 or 100, the change rate is flipped with probability 50%, producing piecewise-linear trajectories with constant change.Source: D. Setpoint dynamics · p. 6
- Figure 5 shows four example trajectories of the setpoint in the variable setting.Source: Figure 5 · p. 7
IV. EXPERIMENTAL PROTOTYPES
The benchmark aims to reflect aspects considered vital in industrial applications through experimental settings that prototype key industrial control situations.
IV. EXPERIMENTAL PROTOTYPES
- The benchmark aims to include aspects considered vital in industrial applications, using prototype experimental settings that capture key industrial control situations.Source: §IV · p. 7
A. Batch Reinforcement Learning
Batch reinforcement learning starts from data collected on an already-running system and seeks a better policy using only that batch, reflecting industry settings where exploration is restricted to avoid potential damage.
A. Batch Reinforcement Learning
- The learner must return a deployable policy based solely on an initial batch from an already-running system.Source: A. Batch Reinforcement Learning · p. 7
- These scenarios are common in industry, where exploration is usually restricted to avoid possible system damage.Source: A. Batch Reinforcement Learning · p. 7
Random exploration:
Random exploration generates batches of transitions with a behavior policy, and evaluation lets the learned policy drive the system autonomously from specified starting states.
Random exploration:
- The batch is generated by applying a random behavior policy, such as sampling action proposals uniformly.Source: Random exploration · p. 7
- In cited examples, ten setpoints each receive 1,000 behavior-policy time steps, yielding 10,000 state transitions; batch sizes can also be varied.Source: Random exploration · p. 7
- Evaluation initializes the system either at its start settings or at a random state-space location, then lets the policy control it autonomously.Source: Random exploration · p. 7
Safe behavior policy:
The paper describes safe-behavior batch learning and transfer learning as prototype industrial-control settings, and presents the IB as a useful RL benchmark with industrially relevant properties.
Safe behavior policy:
- Safe-behavior policy learning improves a safe but suboptimal policy from batch data, where large parts of the state space may be unavailable.The batch may be concentrated in specific state-space areas, with little information elsewhere.Source: Safe behavior policy · p. 7
B. Transfer Learning
- Transfer learning uses data from related systems or contexts to improve learning for a target system with limited data.The IB varies setpoint p to define stochastic systems whose dissimilarity grows with the distance between setpoints.Source: B. Transfer Learning · p. 7
V. CONCLUSION
- The IB combines properties other benchmarks do not provide and offers prototype experimental setups relevant to industrial control.The authors describe it as a step toward studying RL in realistic industrial settings.Source: §V · p. 7
APPENDIX A STATE DESCRIPTION
The appendix distinguishes the observable vector from the preferred minimal Markovian state, which contains 20 variables including latent sub-dynamics variables.
APPENDIX A STATE DESCRIPTION
- The observation vector contains velocity, gain, shift, setpoint, consumption, and fatigue, but does not fulfill the Markov property.The paper also calls this vector the observable state.Source: Appendix A · p. 8
- The preferred minimal Markovian state has 20 values, combining the observation vector with latent variables from the sub-dynamics.The current operational cost is omitted because it can be calculated from velocity, gain, and setpoint.Source: Appendix A · p. 8
- The latent variables include nine previous operational costs, three mis-calibration variables, and two fatigue variables.The current operational cost is not included in the state definition.Source: Appendix A · p. 8
APPENDIX B GOLDSTONE POTENTIAL BASED EQUATIONS
The appendix describes a linearly biased Goldstone potential for computing the mis-calibration reward penalty, using equations to calculate a function of effective shift and direction angle.
APPENDIX B GOLDSTONE POTENTIAL BASED EQUATIONS
- The mis-calibration reward component's penalty is computed using a linearly biased Goldstone potential.Constants are predefined for the subsequent penalty computation.Source: Appendix B · p. 8
- The appendix links the equation-defined function to the penalty computation by specifying the potential and its inputs.The stated inputs are effective shift and the sine of direction angle.Source: Appendix B · pp. 8–9
- The function ω(ρs, he) is computed from effective shift he and the sine of direction angle ρs.The text introduces a set of equations for this computation.Source: Appendix B · pp. 8–9