PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning-Based Production Scheduling in an Industry-Based Coating Scenario Using the Digital Model Playground

Arne Kröger, Ralf Buschermöhle, Wilhelm Hasselbring, Henrik Wilbers

arXiv:2608.14122v1cs.AI

TL;DR

Production-scheduling RL research often lacks validation in realistic industrial scenarios. This paper models an industry-derived coating process in the open-source DMPG, benchmarks DQN and PPO against dispatching rules, and finds PPO delivers the most robust balanced performance.

  • Problem

    Most production-scheduling studies lack validation in realistic industrial or real-world production scenarios, limiting evidence of industrial applicability.

  • Method

    The paper models an industry-derived coating process with sequence-dependent setups, breakdowns, stochastic disturbances, and variable utilization, then benchmarks DQN and PPO against dispatching rules.

  • Results

    PPO achieves the best overall balance across performance criteria, outperforming nearly all alternatives while avoiding severe degradation in any single measure.

  • Takeaways & Limitations

    The open-source DMPG provides a reusable, transparent testbed for comparing scheduling algorithms in realistic industrial settings.

  • Takeaways & Limitations

    Because hyperparameter optimization was not conducted, the selected parameters may be suboptimal and DQN could potentially outperform PPO under alternative configurations.

Abstract

from arXiv · show

Production scheduling in complex manufacturing environments is challenging when sequence-dependent setup times, stochastic disturbances, and due-date constraints must be addressed simultaneously. While reinforcement learning (RL) methods have shown promising results in research, most studies rely on simplified benchmark processes, limiting their industrial relevance. This paper demonstrates the applicability of RL-based scheduling in an industry-inspired coating process that reflects practical complexities such as sequence-dependent setup times, machine breakdowns, and variable utilization. The open-source Digital Model Playground (DMPG), a discrete event simulation framework, is used to model the scenario and to train RL agents. Two standard algorithms, Deep Q-Networks and Proximal Policy Optimization, are benchmarked against conventional dispatching rules to illustrate feasibility and to provide a transparent testbed for further research. Results indicate that RL-based scheduling achieves balanced improvements across key performance indicators, with PPO delivering the most robust performance. The main contribution of this work is to bridge the gap between academic research and industrial practice by validating RL-based scheduling in a realistic, shareable scenario and by providing a reusable open-source framework for future studies.

I. INTRODUCTION

Digital twins and reinforcement learning are rapidly expanding areas in production scheduling, yet both remain insufficiently validated in real industrial or real-world scenarios. This work aims to combine these techniques by creating digital twins from production facilities and using simulation for scheduling research.

  • Motivation: Tripling from 2019 to 2023, publications on digital twins for production scheduling reflect strong industrial and academic interest.Ouahabi et al. identified this increase while noting that most studies lack validation in real industrial environments.
  • Motivation: About 20 publications in 2018 grew to 200 in 2023 for reinforcement learning in production scheduling.Modrak et al. reported this steep increase, while Panzer et al. identified a lack of validation in real-world production-scheduling scenarios.
  • Research goal: The research goal is to combine digital twins and reinforcement learning by creating digital twins from production facilities and using simulation.The supplied passage states that the study intends to use these techniques together for production scheduling.
  • Paper structure: The paper proceeds from related work to experimental setup, agent training, results, comparison with dispatching rules, and contextual discussion.This sequence is stated in the paper’s organizational overview.

II. RELATED WORKS

Prior RL scheduling studies generally adaptively select dispatching rules in simplified benchmark environments, often omitting industrial complexities such as sequence-dependent setups and stochastic disturbances. This work addresses that gap by using the open-source DMPG framework to validate RL scheduling in an industrial coating scenario and benchmark it against dispatching rules.

  • Prior RL scheduling research: Most prior RL scheduling approaches select dispatching rules rather than making direct scheduling decisions and evaluate simplified benchmark scenarios with limited industrial relevance.These studies primarily target reducing tardiness or setup times in parallel or hybrid flow shop problems.
  • Prior RL scheduling research: Prior studies report strong results from PPO, A2C, DQN, and multi-agent DQN variants, typically by adaptively assigning or selecting dispatching rules.Reported comparisons include dispatching rules, metaheuristics, EDD, and static rule application across parallel, hybrid, and flexible flow shops.
  • Research gap: Despite demonstrated DRL superiority over dispatching rules and metaheuristics, prior work often neglects sequence-dependent setup times, stochastic disturbances, and machine breakdowns.The literature summary identifies generic benchmark problems and algorithmic refinements of PPO or DQN as dominant patterns.
  • Contribution: This work builds on the open-source Digital Model Playground, a discrete event simulation framework, to address limitations in prior RL scheduling research.The framework is used as the basis for modeling a scenario intended to address the identified shortcomings.
  • Contribution: An actual industrial coating process provides a realistic setting for validating RL scheduling and benchmarking it against dispatching rules.The stated aim is to combine methodological rigor with practical relevance.

III. EXPERIMENTAL SETUP

The experimental setup presents the production scenario, introduces the Digital Model Playground structure and defines observation and action spaces, then explains how experiments are conducted.

  • The setup first presents the production scenario.
  • It then describes the DMPG structure and defines the observation and action spaces.
  • Finally, it explains how the experiments are performed.

A. Production Scenario

The paper models a two-step CED and powder-coating process as a discrete-event simulation, capturing sequence-dependent color-change setups and practical production constraints. Scheduling balances setup-related costs, weighted due-date deviations, and unfinished products under variable utilization.

  • Production process: The simulated process combines CED coating and powder coating, with CED cleaning, coating, and curing followed by grinding, buffering, powder application, and curing.Products use one of two CED ovens, one of three grinding stations, two powder cabins, and two powder-coating ovens.
  • Production process: Color changes in powder stations create sequence-dependent setup times because cleaning effort and material losses vary by color transition.Switching between similar yellow shades is less demanding than changing from blue to yellow.
  • Scheduling objectives: Scheduling minimizes setup times and associated costs while reducing deviations from planned finishing times.Color sequencing is optimized because powder-booth changes require thorough cleaning and cause material losses.
  • Scheduling objectives: Due-date deviations penalize both earliness and lateness, with delays weighted more heavily because they create greater disruption to subsequent assembly processes.Reducing deviations also limits the need for buffer storage.
  • Performance evaluation: The evaluation uses Setup Time, Weighted Deviation, and Number Not Finished as key performance indicators.Weighted Deviation fully weights delays and assigns half weight to early completions; unfinished products prevent avoiding production altogether.
  • Simulation configuration: The discrete-event simulation generates arrivals from four sources with random interarrival times, assigns random due dates, and varies utilization using a factor randomly selected between 0.7 and 1.3.Interarrival times are scaled by the utilization factor at the beginning of each evaluation.

B. Reinforcement Learning with the Digital Model Playground

The Digital Model Playground is an open-source discrete-event simulation framework integrating deep reinforcement learning for production-process optimization. Its setup defines RL-specific state and reward components, supports DQN/PPO training with parallel experience collection, and benchmarks agents against EDD and MST dispatching rules.

  • Framework: DMPG is a Python discrete-event simulation framework introduced in December 2024 as an open-source alternative to commercial simulation tools, with distributed execution, visualization, and DRL integration.Its DES performance is described as comparable to commercial software.
  • Framework: Users implement the simulation model and Connector, then define only the reward and state representation while configuration files specify hyperparameters.The framework automatically instantiates DQN or PPO agents using TF-Agents, Reverb, and a trainer–worker architecture.
  • State and actions: The agent state comprises buffered products, neighboring-station statuses, active powder-station colors, and system utilization, while five actions select four colors or idling.Selecting a color schedules the earliest-due-date product of that color; idling waits for a new buffer entry.
  • Reward design: The reward penalizes accumulated delay beyond due dates plus 3,000 time steps, combining normalized delay, setup-time, and weighted-deviation measures.The heuristic weights are 5 for delay, 0.6 for setup time, and 0.4 for weighted deviation.
  • Benchmarking: RL agents are benchmarked against earliest due date (EDD) and minimum setup time (MST), with MST decisions triggered only at buffer lengths of 0, 5, 10, 20, or 30 products.If fewer products are present than the predefined buffer length, no scheduling decision is taken.

IV. RESULTS

Across varying process loads, PPO generally provides the strongest overall scheduling performance, while DQN and MST rules show metric-specific advantages with trade-offs in other criteria. The evaluation uses repeated simulations to compare RL agents with conventional dispatching rules.

  • Evaluation setup: The comparison varies utilization factors from 0.7 to 1.3, corresponding to approximately 75%–140% utilization, with 100 simulation runs per level.Average results are reported in Fig. 3, Fig. 4, and Fig. 5.
  • Setup time: PPO consistently achieves the lowest setup times, particularly at high utilization, whereas EDD performs poorly and MST-0 and DQN are competitive mainly at high utilization.MST-0 and DQN setup times increase as utilization decreases.
  • Due-date deviation: DQN achieves the lowest weighted due-date deviation at high utilization, but this advantage disappears at lower utilization levels.MST-20 and MST-30 perform substantially worse across this criterion.
  • Overall performance: PPO achieves the best overall balance across criteria, while DQN has the lowest average weighted deviation but substantially more unfinished products.Larger MST minimum buffer lengths reduce setup times but increase deviations and unfinished products; MST-5 has slightly lower deviation than PPO.

V. DISCUSSION

The study demonstrates that RL-based scheduling can be applied to a realistic coating process, while emphasizing limitations involving human supervision, scenario factors, hyperparameters, and comparisons with alternative methods. RL avoids the setup-time versus due-date trade-off observed for MST dispatching rules.

  • Industrial realism: The modeled scenario combines CED and powder coating with downstream steps, machine breakdowns, and varying utilization to reflect real-world production factors.The process is derived from a production facility and includes selected downstream treatments analogous to labeling.
  • Performance: DQN and PPO were benchmarked against conventional dispatching rules, with RL-based scheduling generally achieving superior performance.The comparison supports the applicability of RL-based scheduling to real-world processes, despite model simplifications.
  • Limitations: Industrial evaluation should compare RL with the combined decision-making of dispatching rules and human operators, who can adapt proactively to machine failures.Dispatching rules cannot dynamically prevent production of affected product types in the described failure scenario.
  • Limitations: RL performance may vary with heterogeneous processing times, different numbers of parallel stations, and alternative scenarios.The study did not investigate heterogeneous processing times or different numbers of parallel stations, so RL may perform better or worse relative to dispatching rules.
  • Limitations: Hyperparameters critically affect trained-agent performance, and alternative configurations could allow DQN to outperform PPO.No hyperparameter optimization study was conducted, so the selected parameters may not be optimal.
  • Performance: MST rules trade lower setup times for higher due-date deviations and more unfinished products as buffer thresholds increase.Larger buffers enable more efficient color sequencing but increase deviations and unfinished products, whereas RL-based scheduling does not exhibit this trade-off.

VI. CONCLUSION

The study applies reinforcement learning to production scheduling in an industry-derived coating scenario with realistic operational complexities. It provides a transparent, reusable testbed while identifying validation, scalability, and human-expertise challenges for future work.

  • Contributions: The study models production scheduling in an industry-derived coating process incorporating sequence-dependent setup times, stochastic disturbances, machine breakdowns, and utilization variability.PPO and DQN are benchmarked against classical dispatching rules, and RL agents achieve competitive performance.
  • Contributions: The open-source Digital Model Playground provides a reusable, transparent testbed for reproducible comparative studies of scheduling algorithms in realistic industrial settings.The framework is intended for both researchers and practitioners.
  • Limitations: RL scheduling performance is sensitive to hyperparameter selection, while scalability to other industrial processes and interaction with human expertise remain unvalidated.Human expertise is described as a key factor in real-world operations.
  • Future work: Future work will compare RL scheduling with human decision-making in real production environments and extend DMPG with advanced RL algorithms or alternative optimization techniques.The proposed alternatives include evolutionary methods and metaheuristics.
Loading 2608.14122v1…