Model-predictive control and reinforcement learning in multi-energy system case studies

• Integrated control strategies enable efficient energy use at lower costs.Source: Highlights · p. 2
• MPC and RL have common mathematical ground.Source: Highlights · p. 2
• RL-based energy management does not require a priori models.Source: Highlights · p. 2
• RL can outperform MPC after training.Source: Highlights · p. 2
• Safety and fast convergence remain challenges for RL.Source: Highlights · p. 2

1. Introduction

• Coupled energy carriers and flexible resources make integrated control relevant to improving multi-energy-system efficiency and performance.Examples include combined heat and power linking gas, heat, and electricity, and heat pumps linking electricity and heat.Source: §1 · p. 2
• MPC optimizes operation while accounting for system dynamics, constraints, and forecast uncertainty, but depends on an adequate model that can be inaccurate or fail to adapt to system changes.The introduction notes that finite-horizon MPC also makes long-term constraints and objectives challenging to represent adequately.Source: §1 · pp. 2–3
• Model-free RL learns system dynamics and a state-to-action policy through environment interaction or historical data without prior system knowledge.Source: §1 · p. 3

1.1. Contribution and outline

• The paper identifies simultaneous benchmarking of MPC and multi-objective RL in multi-energy management as a contribution for demonstrating their mathematical synergy and motivating future work.Source: §1.1 · p. 4

2.1. Discussion

• Prior energy-hub work developed models for multi-carrier dispatch, power flow, and design, with later frameworks incorporating storage and time dependence.Source: §2.1 · p. 4
• MPC studies address forecasted load and price changes, but longer prediction horizons increase computational effort; distributed MPC has also been studied for feasible intermediate solutions and prediction uncertainty.Source: §2.1 · pp. 4–5
• Model-free RL has been proposed for multi-energy management and related energy-hub design and demand-side-management problems.Source: §2.1 · p. 5

2.2. Conclusion of state-of-the-art

• Studies of multi-energy systems span design, distribution, and operation across building-to-district scales, with objectives including ownership costs and operational running costs.Operational objectives are only seldom considered simultaneously across optimization problems.Source: §2.2 · p. 5
• The state-of-the-art treats linear and integer programming as mature for presumed system models, while RL promises adaptability and lower human modeling effort given sufficient training.Source: §2.2 · p. 6
• RL's immature stability, feasibility, robustness, and constraint-handling theory, together with exploratory action randomness, limits confidence in safe real-world application.Source: §2.2 · p. 6

3. Preliminary - Optimal control problem

• The formulation starts from a general optimal-control problem to compare MPC and RL, with a discrete-time assumption for continuous energy systems.Source: §3 · p. 9

3.1. Continuous system

• For continuous stochastic systems, the control signal is chosen to minimize expected stage and terminal losses over stochastic trajectories.The state is x_t, the control is u_t, and W_t represents stochastic noise.Source: §3.1 · p. 9

3.2. Discrete-time approximation

• The formulation extends finite-endpoint objectives to discounted infinite-horizon versions, including a discrete-time approximation in which disturbances can enter the dynamics in any form.Source: §3.1 · p. 9, §3.2 · pp. 9–10

4.1. Model-predictive control

• MPC optimizes over a receding horizon using an a priori system model and state and action constraints.Its horizon solution is intentionally suboptimal for the underlying finite- or infinite-horizon system to manage computational complexity.Source: §4.1 · p. 11

4.2. Reinforcement learning

• A fully observable MDP represents single-agent RL through states, actions, transition probabilities, and immediate rewards.The transition probability depends on the current state and action, and the reward is unknown to the agent before the transition.Source: §4.2 · p. 11
• RL maximizes expected discounted rewards, equivalent to stochastic optimal control when reward is the negative loss, without an explicit dynamics model or MPC-style constraint set.The MDP definition is required, but the transition probability matrix need not be explicitly defined.Source: §4.2 · pp. 11–12

5. Methodology

• The tool-chain exports a Modelica model as a co-simulation FMU, wraps it in a Python Gym environment, and connects controllers to environments through a general interface.Using the FMU's do_step() method speeds simulation compared with simulate(), but requires proper initialization handling.Source: §5 · p. 12

5.1.1. Simulation model

• The simple system is a 2,430-DAE model with wind, photovoltaic, boiler, CHP, and battery assets, including CHP operating limits and state-of-charge-dependent battery rates.The renewable generation plants are non-controllable, and the asset inputs, outputs, capacities, and safe limits are described in Table 3.Source: §5.1.1 · p. 13

5.1.2. Model-predictive controller

• The simple-case LMPC uses a MILP with a 72-hour prediction horizon and 24-hour control horizon, tested with perfect foresight and Gaussian-noise forecasts.Its linearized, simplified model can deviate slightly from the optimum; a 15-minute control horizon showed no significant performance difference.Source: §5.1.2 · pp. 13–15

5.1.3. Reinforcement learning agent

• For the simple case, PPO2 receives energy-cost-based rewards and controls CHP and battery set-points, while a separate controller sets boiler output to meet remaining heat demand.The state includes demand, renewable generation, prices, cost, and storage information; grid connections are assumed sufficiently large to meet electrical and gas demand.Source: §5.1.3 · p. 16

5.2. Case study II - complex

• The complex case couples nine energy assets across carriers; its MPC enforces per-step carrier balance, and the study uses discrete-action PPO2 and continuous-action TD3 agents.The simulation has 2,548 differential-algebraic equations and uses Calliope for the MPC formulation.Source: §5.2.1 · pp. 17–18, §5.2.2 · p. 18, §5.2.3 · p. 19

6. Results and discussion

• LMPC loses 2% in case study I and 10% in case study II from inaccurate prediction and modelling errors, while RL can outperform perfect-foresight LMPC in the simple configuration.The comparison concerns energy-cost minimization subject to comfort fulfilment.Source: §6 · pp. 19–20, Table 7 · p. 20
• RL agents use function approximations and show difficulty operating storage when charging incurs an immediate penalty for a possible mid-term reward.The authors leave open whether the storage difficulty stems from hyper-parameters, PPO’s surrogate objective, the algorithm, or RL itself.Source: §6 · p. 20
• The random agent’s relatively high case-study-I performance partly reflects its separate boiler controller, which keeps thermal discomfort close to zero despite random set-points elsewhere.This reduces the thermal-discomfort term in Equation 16e.Source: §6 · p. 20
• The reported unconstrained RL results involve many unsafe environment interactions and a training cost, unlike LMPC’s initial and ongoing modelling cost.The evaluation covers energy-cost minimization and demand fulfilment, including thermal discomfort when the thermal system is not grid-connected.Source: §6 · pp. 20–21
• In case study I, both RL algorithms outperform perfect-foresight LMPC after 125,000 training steps (3.56 years) and realistic LMPC after ∼80,000 and ∼105,000 steps.The passage associates the realistic-LMPC thresholds with 2.28 and 3 years, respectively.Source: §6 · p. 21, Figure 5 · p. 21
• In case study I, PPO and TD3 reach 70% of their best performance after 2.04 and 2.17 years, and 90% after 2.94 and 3.57 years, respectively.The passage warns that the long training time could be impractical in a real setup.Source: §6 · p. 21
• After ∼2 years, the case-study-II TD3 agent outperforms realistic LMPC, reaching ∼70%, ∼90%, and ∼95% of its maximum performance after 3,000, 10,000, and 12,500 time steps, respectively.These milestones correspond to ∼4.5, ∼14.9, and ∼18.6 weeks of data using quarterly interactions.Source: §6 · p. 22, Figure 6 · p. 22

7. Conclusions and future work

• The study benchmarks state-of-the-art on- and off-policy multi-objective RL against linear MPC in dynamically simulated, day-ahead multi-energy systems.The comparison is framed around their mathematical synergy and differences.Source: §7 · p. 22
• RL can be an adequate multi-energy optimal-control technique that outperforms practically achievable LMPC given sufficient time and memory for training.The authors note that this performance comes with unsafe environment interactions and a training cost, while avoiding MPC’s modelling cost.Source: §7 · p. 22
• Performance depends on adequate a priori unknown hyper-parameters, which the authors say can be mitigated with offline pre-studies.Their cross-configuration results suggest hyper-parameters may be task-specific rather than wholly environment-specific, but they call for further research.Source: §7 · pp. 22–23
• Future work proposes safe exploration using stability theory, shielding for strict safety constraints, and planning steps to reduce training budgets.Other proposed directions include demand-side management, industrial production, battery-storage root-cause analysis, and robustness to faulty observations.Source: §7 · p. 23

CRediT authorship contribution statement

• The contribution statement assigns conceptualization and methodology to Glenn Ceusters, with software, analysis, data, visualization, supervision, and writing roles distributed across the listed contributors.The supplied statement names Ceusters, Román Cantú Rodríguez, Alberte Bouso García, Rüdiger Franke, Geert Deconinck, Lieve Helsen, Ann Nowé, Maarten Messagie, and Luis Ramirez Camargo.Source: CRediT authorship contribution statement · p. 23

Appendix A.1 Coefficient of determination (R2-score)

• The R2-score passage defines the metric using predicted and ground-truth values for n samples.The supplied passage introduces the estimated R2 definition but does not include its displayed formula.Source: Appendix A.1 · p. 24

Appendix A.2 Mean absolute percentage error (MAPE)

• The MAPE passage defines the metric using predicted and ground-truth values and specifies a strictly positive ϵ to avoid undefined results when ground truth is zero.The definition is described over n samples.Source: Appendix A.2 · p. 24

Appendix B Hyper-parameter optimisation

• Manual hyper-parameter tuning requires repeated training and evaluation trials, making it computationally and time intensive; automated tuning is presented as a way to reduce that burden.The appendix also notes potential benefits for human effort, error, reproducibility, and fairness of comparisons.Source: Appendix B · p. 25
• The studies used Optuna with a Tree-structured Parzen Estimator to search user-defined hyper-parameter spaces for objectives in both case studies.The reported best agents were trained with parameters found through these optimization studies.Source: Appendix B · p. 25
• The hyper-parameter studies comprised approximately 50–60 trials each, with case-study-I runtimes of 17.54 hours for PPO and 24 hours for TD3.Case-study-II runtimes were approximately 49 days for PPO and 51 days for TD3.Source: Appendix B · pp. 25–26
• Optimization-history plots show each trial’s objective value as blue points and the best objective so far as a red curve; the selected parameters are from the first trial attaining the lowest red-curve value.Hyper-parameter-importance plots indicate which parameters had greater impact during the study.Source: Figure A1 · p. 26, Appendix B · pp. 27–28, Figure A2 · p. 27

Appendix C Simulations visualization

• LMPC uses battery storage across all three periods, while TD3 avoids it, possibly because its hyperparameters led the policy to a local optimum.The passage identifies time-step and discount-factor choices during policy updates as possible contributors.Source: Appendix C · p. 28
• Both controllers slightly overproduce thermal energy, but LMPC’s model energy balance and TD3’s reward penalty limit the excess.The passage attributes overproduction to model and forecast inaccuracy for LMPC, and to the missing operational constraint for TD3.Source: Appendix C · p. 28
• Thermal discomfort is close to zero in winter and slightly higher in summer; TD3 has lower discomfort than LMPC in midseason but also struggles in summer.The passages attribute LMPC’s mismatch to linear-model inaccuracy and explain that TD3 reduces discomfort through a reward term rather than an explicit constraint.Source: Appendix C · pp. 32, 34

Appendix D Pseudocode of PPO and TD3

• PPO collects environment trajectories, computes rewards-to-go and advantage estimates, then updates its policy and value function; TD3 selects actions, stores transitions, and updates its networks.The pseudocode also shows TD3 sampling batches from a replay buffer and updating target networks.Source: Appendix D · pp. 36–37

Appendix E Run-time statistics

• LMPC has significantly longer runtime than RL agents because it solves a mixed-integer linear program at each receding control horizon.RL agents execute the learned policy and update their function-approximation algorithms.Source: Appendix E · p. 39, Table A3 · p. 39, Table A4 · p. 39
• For online training, mean runtime per time-step is 0.0135 s for PPO and 0.0147 s for TD3 in case study 1, and 0.0062 s for PPO and 0.0114 s for TD3 in case study 2.These means include the function-approximation update step.Source: Appendix E · p. 39, Table A3 · p. 39, Table A4 · p. 39
• With offline pre-training, the total training runtime for 250,000 time-steps is required before policy execution.The passage says this training runtime is added to the RL agents’ total policy-execution runtime in the tables.Source: Appendix E · p. 39, Table A3 · p. 39, Table A4 · p. 39