Source-linked AI summary
Distributed Optimization of Modular Production Systems using Model-based Reinforcement Learning with Inverse Models
Andreas Schwung, Steve Yuwono, Sofiene Lassoued, Dorothea Schwung
TL;DR
Flexible modular manufacturing requires self-learning control that adapts to changing production goals while limiting real-world training effort. The paper embeds lightweight approximate inverse models into MBRL policy networks so learning occurs primarily in task space. On a laboratory modular production testbed, inverse models improve performance and reduce training cycles, particularly for off-policy algorithms.
Problem
Flexible, individualized manufacturing requires self-learning and rapidly adaptable modular systems, while industrial RL is limited by extensive training and data requirements.
Method
The paper integrates lightweight feedforward approximate inverse process models into MBRL policy networks, separating actuation-dynamics learning from task-space learning.
Results
Inverse models consistently reduce overflow and required training cycles across six TD3, SAC, and DDPG settings, with algorithm-dependent effects on overall reward.
Takeaways & Limitations
Inverse models improve learning efficiency and performance for modular manufacturing control, with pronounced benefits reported for off-policy algorithms.
Abstract
from arXiv · showhide
This paper presents a novel approach for data-driven self-learning control of highly flexible, modular manufacturing systems. Specifically, we employ a novel framework for model-based reinforcement learning which introduces approximate inverse process models within the training of reinforcement policies. This approach disentangles the learning of actuation dynamics and the dynamics in state space, resulting in RL-based training solely within the task space. We propose a lightweight feedforward architecture for approximate inverse models and integrate them within the policy network of standard RL algorithms. We apply the approach to a laboratory modular production testbed with heterogeneous production modules. The results underline the efficiency improvements for modular manufacturing units in terms of both performance and training speed, particularly for off-policy algorithms.
I. INTRODUCTION
Modern manufacturing needs modular, self-learning systems that can adapt quickly to individualized production, but RL remains costly to train with real industrial data. The paper addresses this gap by embedding approximate inverse models in policy networks so RL learns primarily in task space, reporting improved performance and learning speed on a modular testbed.
- RL adoption in industry is constrained by extensive training effort, large data requirements, limited operating time, cost, and production-interruption risks.
- Model-based RL reduces real-environment interactions by training a system model from a comparatively small random-interaction dataset, but inverse-model integration remains rarely explored.
- RL primarily trains desired state trajectories rather than more complex action trajectories through a partly pretrained policy network.
- The proposed MBRL approach embeds approximate inverse models in policy networks, disentangling actuation dynamics from task-space learning.
- A lightweight feedforward architecture is proposed for approximate model inversion.
- The approach is applied to a modular production system, with encouraging improvements in performance and learning speed.
II. RELATED WORK
Prior work addresses distributed manufacturing through conventional distributed control, games, and MARL, while MBRL reduces real-world interactions. The paper identifies a gap in combining forward and inverse models for distributed self-learning control.
- Conventional distributed control methods include optimal control, distributed MPC, consensus-based control, and fuzzy control, but typically lack self-learning capabilities.
- MARL and game-theoretic approaches have been applied to distributed manufacturing, yet the reviewed methods do not exploit inverse models.
- Distributed MARL is difficult because independently learning agents face partially observable, non-Markovian environments and centralized critics require extensive communication.
- MBRL reduces real-world interactions through learned world models or planning, but prior approaches do not combine forward and inverse models.
III. DISTRIBUTED MBRL WITH INVERSE MODELS
The proposed framework targets modular distributed systems whose interchangeable subsystems have local controllers and production goals. It formulates optimal production control over an episode while accommodating diverse module-level objectives.
- A. Problem Statement: The framework considers distributed systems with subsystems that can be omitted, included, or interchanged, each equipped with local control and neighboring communication.
- A. Problem Statement: The optimization problem seeks production control for a distributed system over an episode t = 0, . . . ,T using local actuation and module production goals.
- A. Problem Statement: Production goals can represent product concentrations, mass flows, or processing times, allowing objectives to vary across modules and applications.
B. Framework Overview
The framework trains decentralized agents with learned forward models and approximate inverse models, shifting policy learning toward desired state trajectories. Lightweight feedforward models are trained during ramp-up and then fixed for model-based RL.
- Framework setup: Each module agent uses neighboring and local states to learn forward dynamics and train within its corresponding world model.The forward model predicts the next state from local, neighboring, and action inputs.
- Inverse-policy design: The policy splits into a state governor that proposes the next state and an inverse model that derives the action needed to reach it.The inverse model uses current, next, and neighboring states; a corrective action is added to form the final action.
- Model architecture: For the distributed setup, small per-actuator MLPs with 4 →128 →128 →1 architectures provide training stability and sufficient accuracy.Both models use SiLU activations, linear outputs clamped to [0, 1], and MSE regression.
- Model training: The inverse model and forward world model are trained from the same data during ramp-up, then kept fixed during subsequent RL training.The supplied passages also note that the approach can use MLPs, recurrent networks, or transformers depending on system complexity.
D. Model Based RL with inverse models
The proposed policy network combines a fixed inverse model with a learnable state governor and can be trained using standard RL algorithms. The framework is intended to address action-space difficulties while retaining model-based training.
- Policy construction: The policy consists of an inverse model and a state governor, with the governor producing the next state and a corrective action component.The inverse model converts the proposed next state into an action, while the corrective component is added to obtain the final action.
- Training assumption: The inverse model is fixed so that only the state-governor parameters remain learnable during policy optimization.This definition makes the policy depend on the learnable governor parameters rather than inverse-model parameters.
- Algorithm compatibility: The architecture supports arbitrary online or offline RL algorithms, including value-based and actor-critic approaches.The reported application compares TD3, SAC, and DDPG, focusing on off-policy actor-critic methods.
- Procedure: Algorithm 1 summarizes the implementation and learning procedure for model-based RL with inverse models.The supplied passage identifies the algorithm as pseudocode for the overall procedure.
IV. EXPERIMENTS AND RESULTS
The experiments apply the method to the Bulk Good Laboratory Plant, a distributed system with four operational modules. The supplied experimental passages describe the training setup and simulation-based implementation but do not report outcome comparisons.
- Testing environment: The evaluation uses the Bulk Good Laboratory Plant, whose loading, storage, weighing, and filling modules transport bulk goods through actuators and reservoirs.The plant is a distributed manufacturing system used to test the proposed methodology.
- Actuation: The described control parameters include conveyor and rotary-feeder motor speed, binary vibratory-conveyor operation, and vacuum-pump turn-on duration.The passages provide examples of heterogeneous actuator controls across the plant modules.
- Learning procedure: Algorithm 1 trains forward and inverse models before replacing the real environment with world models for RL-agent training.The pseudocode includes episode collection, state and reward calculation, action generation, and experience-buffer storage.
- Training setup: The simulation is useful for training the proposed model-based RL framework before deployment on the real system.The supplied passage places the simulation within the BGLP implementation setup.
B. Training Setup on the BGLP
The BGLP setup defines distributed actuator objectives around buffer levels, production-demand fulfilment, and power consumption, combined through weighted rewards.
- Each BGLP player represents an actuator and observes the fill levels of preceding and subsequent reservoirs.
- The objectives maintain reservoir levels to prevent bottlenecks and overflow, minimize power consumption, and satisfy production demand for the final player.
- The reward combines the objectives using predefined weights ωv = 1.0, ωp = 0.001, and ωd = 4.0.
- Production demand is considered fulfilled when the last hopper’s fill level exceeds the specified demand.
C. Experimental Results
The experiments use identical BGLP settings across training and testing, varying production targets while evaluating the learned policies with the defined reward.
- Experiments use 100 training episodes of 10,000 seconds, targeting production rates of 0.125 and 0.15 L/s, followed by one testing episode.
1) Inverse model training:
Inverse-model training is stable and accurate: losses decrease consistently, and predicted actuator actions closely match ground truth, including on unseen data.
- Decreasing loss curves indicate stable inverse-model training with sufficient accuracy for each actuator.
- Predicted and ground-truth actuator actions show very good agreement on unseen data, with actuator 3 following a discrete step-function reference.
2) Distributed MBRL with inverse models:
Across TD3, SAC, and DDPG at two production rates, inverse models consistently reduce overflow and training cycles, while reward effects depend on the algorithm.
- Inverse models reduce overflow and required training cycles across all six algorithm–production-rate settings.For TD3 at 0.15 L/s, overflow falls from 0.137 L to 0.009 L; for DDPG, training cycles fall from 100k to 30k.
- TD3 and DDPG fail without inverse models but meet demand with them, increasing reward from 8.0-9.0 to approximately 10.1.
- SAC already solves the task without inverse models, so its reward remains comparable while overflow and power consumption decrease with inverse models.
- The inverse models add negligible overhead because each small network is trained once, frozen thereafter, and contributes one additional forward pass per step.Each actuator network has roughly 17k parameters.
- Freezing the inverse model outperforms keeping it trainable, consistent with robustness to moderate inverse-model inaccuracies.Output clamping to [0,1] additionally limits the effect of actuation errors.
V. CONCLUSION AND FUTURE WORK
The framework integrates inverse process models into model-based reinforcement learning for modular manufacturing, separating actuation dynamics from task-space dynamics. Laboratory experiments report improved learning efficiency and performance, while future work targets robustness, scalability, and statistical validation.
- The framework separates actuation dynamics from task-space dynamics, enabling reinforcement learning to focus on task-space optimization.
- Integrating inverse models improved learning efficiency and performance compared with conventional RL, particularly for off-policy methods.
- Future work will examine sequence-to-sequence inverse models and robustness under changing operating conditions and model uncertainties.
- Further evaluation will address scalability on larger modular plants, deployment on the real testbed, and confidence intervals across multiple random seeds.