PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Sustainable Hydrogen Systems: Supply Chain Optimization with Model Predictive Control and Reinforcement Learning

Mahammad Valiyev

arXiv:2609.11933v1math.OCcs.LG

TL;DR

Hydrogen supply-chain operation must manage renewable variability, uncertain prices and demand, and engineering constraints, while the conditions under which forecast-augmented RL can outperform alternatives remain an open question. This paper compares four controllers in a unified framework and reports the strongest economic performance for MPC, competitive robust behavior for RL-NF, and no consistent advantage for RL-F over RL-NF.

  • Problem

    It remains open under what conditions forecast-augmented RL can outperform reactive RL or MPC in hydrogen supply-chain operations amid renewable, price, and demand uncertainty.

    Source: §1 · pp. 2–3

  • Method

    The study compares RBC, MPC, RL-NF, and RL-F in a unified renewable-powered hydrogen supply chain under common disturbances, constraints, and economic assumptions.

    Source: §1 · p. 3, §2.6 · p. 9

  • Results

    MPC achieves the highest economic performance, RL-NF is robust and competitive without forecasts, and RL-F does not consistently outperform RL-NF.

    Source: §1 · p. 3, §4 · p. 20

  • Takeaways & Limitations

    The findings motivate further investigation into robust and hybrid control strategies as hydrogen infrastructure scales.

    Source: §1 · p. 3

    The analysis uses stylized disturbances, a simplified market, and merchant operation without contractual delivery penalties; RL outcomes depend on the chosen modeling, training, and forecast configurations.

    Source: §4 · p. 20

Abstract

from arXiv · show

Hydrogen supply chains are expected to play a central role in future low-carbon energy systems by enabling renewable energy integration, long-duration storage, and decarbonization of industrial and transportation sectors. However, their operation is challenged by renewable generation variability, electricity price fluctuations, uncertain hydrogen demand, and engineering constraints associated with electrolyzers, energy storage, and grid interaction. As hydrogen infrastructure expands toward commercial deployment, operational strategies must balance economic performance, reliability, and sustainability under dynamic and uncertain conditions. This paper investigates and compares four control approaches for a renewable-powered hydrogen supply chain: a rule-based controller (RBC), model predictive control (MPC), reinforcement learning without forecasts (RL-NF), and reinforcement learning with forecast-augmented observations (RL-F). All methods are evaluated within a unified, physically realistic framework incorporating electrolyzer minimum-load and ramp-rate constraints, battery and hydrogen storage dynamics, grid import limits, and consistent economic assumptions, enabling a fair comparison under identical operating conditions. Simulation results show that MPC achieves the highest economic performance by exploiting short-term forecasts to coordinate storage, reduce grid dependence, and improve efficiency. RL-NF demonstrates robust and competitive performance without future information, highlighting the capability of learning-based methods to discover effective policies from experience. RL-F does not consistently outperform its no-forecast counterpart, suggesting that forecast uncertainty and increased state complexity can limit forecast-augmented learning. The results provide guidance for selecting operational control strategies in future hydrogen energy systems.

1 Introduction

Hydrogen can support renewable integration and decarbonization, but operating its supply chain requires managing uncertainty and physical constraints. This study compares four control strategies under common conditions and finds that MPC leads economically, while RL-NF remains competitive and RL-F does not consistently improve on it.

1 Introduction

  • Hydrogen produced with renewable electricity is presented as a potential contributor to long-duration storage, industrial decarbonization, and clean fuel supply.The passage also identifies potential emissions reductions and system flexibility as benefits of renewable hydrogen.Source: §1 · p. 2
  • Hydrogen supply-chain operation must contend with uncertain renewable availability, electricity prices, and demand alongside electrolyzer, storage, and grid constraints.The passage identifies possible consequences of suboptimal operation, including excessive grid dependence, renewable curtailment, increased costs, and underused assets.Source: §1 · p. 2
  • Rule-based control is simple and interpretable but reactive, whereas MPC incorporates forecasts, constraints, and economic objectives through receding-horizon optimization.Prior work cited in the passage reports MPC benefits when short-term forecasts are sufficiently reliable, alongside computational burden and sensitivity to forecast errors.Source: §1 · p. 2
  • Reinforcement learning can learn adaptive policies under uncertainty, but constraint satisfaction, training stability, and interpretability remain challenges.Forecast information does not necessarily improve RL performance when forecasts are noisy or weakly informative, leaving its comparative value an open question.Source: §1 · pp. 2–3
  • The study compares RBC, MPC, RL-NF, and RL-F in a unified renewable-powered hydrogen supply chain under identical disturbances, constraints, and economic assumptions.The modeled system includes variable renewables, storage, an electrolyzer with minimum-load and ramping constraints, grid interaction, and uncertain demand.Source: §1 · p. 3
  • MPC achieves the highest economic performance by exploiting short-term forecasts, while RL-NF is robust and competitive and RL-F does not consistently outperform RL-NF.The authors attribute these differences to forecast-based coordination for MPC and challenges learning from noisy forecast information for RL-F.Source: §1 · p. 3

2.1 System Description and Problem Formulation

The paper models a stylized renewable-powered hydrogen supply chain with coupled electricity, battery, electrolyzer, hydrogen storage, and sales operations. Its one-week hourly simulation exposes the system to changing solar availability, electricity prices, and hydrogen demand.

2.1.1 Hydrogen Supply Chain Process Model

  • Solar photovoltaic generation supplies variable, low-cost electricity with diurnal patterns and stochastic variability.Solar availability is time-dependent.Source: §2.1.1 · p. 3
  • The battery stores excess solar electricity for later discharge when renewable availability is insufficient, subject to finite power and energy capacities and charging and discharging efficiencies.Its stated role is temporary electricity storage.Source: §2.1.1 · p. 3
  • The modeled system links solar generation, battery storage, grid interaction, electrolyzer operation, hydrogen storage, and sales through energy balances that define feasible operation.Figure 1 presents the process model, mass and energy balances, feasibility constraints, and operational limits.Source: §2.1.1 · p. 3, Figure 1 · p. 4
  • The electrolyzer converts electricity from solar, battery discharge, or the grid into hydrogen at fixed efficiency, subject to rated-power, minimum-load, and ramping constraints.These operating limits constrain its feasible power input.Source: §2.1.1 · p. 4
  • Produced hydrogen can be sold immediately or stored for later withdrawal to meet demand shortfalls, within tank-capacity and throughput limits.Hydrogen demand is exogenous.Source: §2.1.1 · p. 4
  • The system is simulated in hourly steps over one week with time-varying solar availability, electricity prices, and hydrogen demand.The setup combines variable renewable input with changing market and demand conditions.Source: §2.1.1 · p. 4

2.2 State Variables, Control Variables, and System Dynamics

The formulation tracks battery charge and stored hydrogen, chooses electricity and hydrogen flows, and evolves storage and production through system dynamics. MPC optimizes control variables explicitly, while RL actions pass through a feasibility layer.

2.2.1 State Variables

  • The two primary state variables are battery state of charge and hydrogen storage level, both bounded by their respective capacities.These states evolve through energy and mass balances driven by control actions and system efficiencies.Source: §2.2.1 · p. 4

2.2.2 Control Variables

  • Electricity allocation decisions cover solar charging, battery discharge, grid supply, direct solar supply, and total electrolyzer power.The controller determines these energy allocation decisions at each time step.Source: §2.2.2 · p. 5
  • Hydrogen allocation decisions distinguish direct sales, production sent to storage, and sales from storage; RL actions are mapped to feasible values.MPC explicitly optimizes the control variables, while the RL feasibility layer ensures consistent system behavior and constraint satisfaction.Source: §2.2.2 · p. 5

2.2.3 System Dynamics

  • System dynamics specify the evolution of battery charge, hydrogen storage, hydrogen production, and total hydrogen sales.The passage introduces these four relationships without specifying their algebraic forms.Source: §2.2.3 · p. 5

2.3 Objective Function

The objective maximizes net profit over the simulation horizon, equivalently minimizing operating costs minus hydrogen sales revenue. The same economic objective is used for MPC and as the RL reward.

2.3 Objective Function

  • The objective maximizes horizon-wide net profit by accounting for grid electricity, battery cycling, electrolyzer operation, hydrogen storage, and sales revenue.The per-step cost includes a carbon adder on grid electricity, and hydrogen sales generate revenue.Source: §2.3 · p. 5
  • The objective is used directly in MPC and as the reward signal for RL, enabling consistent economic comparison across methods.The stated equivalence is minimizing total operating cost minus hydrogen sales revenue.Source: §2.3 · p. 5

2.4 Operational Constraints

The section introduces operating constraints and explains how controllers enforce them. Unmet hydrogen demand is allowed without an explicit penalty beyond foregone sales revenue.

2.4 Operational Constraints

  • MPC enforces the system constraints explicitly, while RL controllers enforce them through a feasibility layer.The layer ensures physically realistic operation; unmet demand is permitted and carries no explicit penalty beyond foregone sales revenue.Source: §2.4 · p. 6
  • Unmet hydrogen demand is permitted and incurs no explicit penalty beyond foregone sales revenue.Source: §2.4 · p. 6

2.5 Disturbance and Forecast Models

The section models hourly solar availability, electricity prices, and hydrogen demand as time-varying disturbances, then specifies forecast access and uncertainty for the controllers.

2.5 Disturbance and Forecast Models

  • All controllers are evaluated hourly under identical disturbance realizations to ensure a fair comparison.Source: §2.5 · p. 6

2.5.1 Solar Availability

  • Solar availability combines a deterministic half-sine diurnal shape with slow cloud-induced and short-term irradiance variability.Figure 2 presents realized solar disturbance and representative short-horizon stochastic forecasts.Source: §2.5.1 · p. 6, Figure 2 · p. 7

2.5.2 Electricity Prices

  • Electricity prices follow a time-of-use tariff with occasional stochastic spikes, and Figure 3 shows realized prices and representative forecasts.Source: §2.5.2 · p. 7, Figure 3 · p. 8

2.5.3 Hydrogen Demand

  • Hydrogen demand is time-varying and scaled to a 300 kW electrolyzer, whose maximum production is approximately 6 kg/h at 50 kWh/kg.Figure 4 presents realized demand disturbance and representative forecasts.Source: §2.5.3 · p. 7, Figure 4 · p. 8

2.5.4 Forecast Availability and Uncertainty

  • MPC and RL-F receive short-horizon stochastic forecasts, whereas RL-NF operates under partial observability; the prediction horizon is 12 hours and uncertainty grows with lead time.Signal-specific base uncertainty levels are 10 kW for solar, 0.02 $/kWh for price, and 0.5 kg/h for demand; identical disturbance models and random seeds support a fair comparison.Source: §2.5.4 · p. 9

2.6 Control Strategies

The four strategies share a system model, constraints, cost structure, and disturbance realizations; they differ in how decisions use forecasts, optimization, and learning.

2.6.1 Rule-Based Control (RBC)

  • RBC is a myopic, forecast-free heuristic that prioritizes solar for electrolysis, then battery discharge and grid imports, while balancing hydrogen production and storage against demand.Grid imports meet remaining electrolyzer needs regardless of electricity price; RBC serves as the baseline benchmark.Source: §2.6.1 · p. 9

2.6.2 Model Predictive Control (MPC)

  • MPC optimizes cumulative economic cost over a rolling horizon using disturbance forecasts and explicit physical and operational constraints.It implements only the first action, then shifts the horizon, updates forecasts, and resolves the problem.Source: §2.6.2 · p. 10

2.6.3 Reinforcement Learning Framework

  • RL learns policies through environment interaction without runtime optimization, using PPO and a feasibility layer to map preferences to admissible controls.Its per-step economic-profit reward matches the MPC objective; the feasibility layer clips targets and maintains energy and mass balance.Source: §2.6.3 · pp. 10–11
  • RL-NF observes current system information only, while RL-F adds short-horizon noisy forecasts; neither RL variant performs MPC-style multi-step optimization.RL-NF is partially observable, and RL constraint coupling is implicit rather than explicit.Source: §2.6.3 · pp. 10–11, Figure 5 · p. 11

3 Results

MPC coordinates dispatch across renewable availability, prices, and demand while achieving near-complete hydrogen utilization. RL-NF remains competitive without forecasts, while RL-F improves decision timing without increasing production scale.

3.1 Rule-Based Control: Baseline Performance

  • RBC sends solar to electrolysis, charges the battery with excess, and meets shortfalls with price-insensitive grid imports; hydrogen utilization is about 80%.It establishes a lower-bound benchmark, with grid-dominated costs constraining profitability.Source: §3.1 · pp. 11, 13, Figure 6 · p. 12, Figure 7 · p. 13

3.2 Model Predictive Control

  • MPC sustains high electrolyzer utilization, aligns hydrogen production with sales near-completely, and coordinates grid and storage use against renewable supply, prices, and demand.Revenue remains dominant relative to operating cost, while storage-related costs are negligible.Source: §3.2 · pp. 13–14

3.3 Reinforcement Learning without Forecasts (RL-NF)

  • RL-NF achieves near-complete hydrogen utilization with strong direct solar use and moderate grid supplementation, but its production scale is lower than MPC's.Its dispatch remains reactive, with limited battery engagement and little intertemporal structure.Source: §3.3 · pp. 14, 16, Figure 11 · p. 17

3.4 Reinforcement Learning with Forecasts (RL-F)

  • RL-F sustains operation during favorable periods and manages hydrogen storage more smoothly; production is similar to RL-NF, with improvement mainly in decision timing.Its aggregated results show balanced grid support and near-complete hydrogen utilization.Source: §3.4 · p. 16, Figure 12 · p. 18, Figure 13 · p. 19

4 Conclusion

Under identical disturbances, constraints, and economic assumptions, MPC has the strongest economic performance, while RL is competitive but forecasts do not consistently help. The authors flag simplifying assumptions and outline hybrid, uncertainty-aware control and multi-agent RL as future directions.

4 Conclusion

  • All four controllers were compared under identical disturbances, constraints, and economic assumptions.The study describes this setup as a transparent and fair comparison.Source: §4 · p. 20
  • RBC remains profitable but grid-dependent with ∼80% hydrogen utilization; MPC leads economically with near-perfect utilization, while RL-NF is competitive and noisy forecasts do not consistently improve RL.The conclusion identifies forecast-based intertemporal planning as MPC’s strength and says RL-F performance is condition-dependent.Source: §4 · p. 20
  • Stylized disturbances, a simplified market, merchant operation without delivery penalties, and configuration-specific RL outcomes limit generalization; broader validation is needed.The authors call for more realistic system representations and multiple uncertainty realizations.Source: §4 · p. 20

Future Work

  • Hybrid MPC–RL designs could combine forecast-based optimization with RL adaptability, including RL-tuned MPC, MPC-guided training, or supervisory switching.The proposed aim is to preserve constraint satisfaction and economic efficiency while improving robustness to model mismatch and changing conditions.Source: Future Work · p. 20
  • Future MPC work could use scenario-based, chance-constrained, or distributionally robust formulations to reduce forecast-error sensitivity and improve reliability.The proposed setting includes renewable and demand variability.Source: Future Work · p. 20
  • Multi-agent RL could coordinate distributed electrolyzers, storage units, and grid interfaces as hydrogen systems scale and decentralize.The authors identify stability, coordination, and safety constraints as open challenges.Source: Future Work · p. 20
Loading 2609.11933v1…