Differentiable Hybrid-Action Neural Feedback Control for District Heating Networks

I. INTRODUCTION

• Hybrid DHN control must jointly choose active units and setpoints while balancing economic, reliability, and environmental objectives under nonlinear dynamics and mixed-integer constraints.These requirements motivate methods that handle network dynamics and mixed-integer decisions while remaining computationally efficient for deployment.Source: §I · p. 1

A. Related work

• Online MPC for hybrid systems can require computationally demanding MINLP solves, while DHN approaches often separate detailed dynamics from integer decisions.Repeatedly solving complex MINLPs and reliance on commercial solvers are identified as barriers to real-time and industrial deployment.Source: A. Related work · pp. 1–2
• Existing RL approaches include hybrid action policies, but model-free training can require extensive interactions and standard policies do not inherently guarantee complex actuator constraints.Applications to DHNs have mostly focused on fully continuous or fully discrete control, with hybrid formulations less explored.Source: A. Related work · p. 2
• Differentiable control methods face long-horizon gradient difficulties and commonly focus on continuous inputs, while prior hybrid policies use short training horizons or forecasts.The cited prior-work comparison also notes that the effect of Gumbel noise on deployed decisions and robustness to plant-model mismatch were unexamined.Source: A. Related work · p. 2

B. Main contributions

• HANC jointly generates categorical actions and continuous setpoints, supports BPTT over full closed-loop trajectories, and enforces complex actuator constraints by design.It deploys as a causal feedback policy using current measurements and internal memory, without forecasts.Source: B. Main contributions · p. 3
• 30% lower operating cost than an industrial rule-based baseline despite model mismatch was achieved on a high-fidelity simulation of a district heating network.The controller was trained exclusively on a physics-informed surrogate model.Source: B. Main contributions · p. 3

A. Problem formulation

• The system has nonlinear, time-varying dynamics, with outputs determined by state, control inputs, and exogenous signals.The policy can use past outputs and context; context may also affect performance through the loss, as with dynamic prices or meteorological data.Source: A. Problem formulation · p. 3
• The feasible action set combines real-valued commands with finite categorical choices and may couple them into disconnected feasible regions.No convexity or connectedness assumption is imposed on the action set.Source: A. Problem formulation · p. 3
• The problem is to minimize finite-horizon loss by optimizing a finite-dimensional parameterization of a causal policy that jointly produces continuous and discrete actions.The loss can depend on outputs, controls, disturbances, and context, and the formulation allows objectives to include output constraints.Source: A. Problem formulation · p. 3

B. Controller architecture

• HANC combines continuous and categorical neural branches with a differentiable assembly layer to produce actuator-feasible inputs.The parameterized policy is designed for low deployment cost.Source: B. Controller architecture · p. 4
• For continuous actuators, a sigmoid output transformation places internal controls within their specified lower and upper bounds.The continuous branch generates differentiable real-valued internal variables.Source: Continuous branch · p. 4
• The discrete branch produces logits for admissible categories and selects a valid category using a hard one-hot representation.The one-hot selection has exactly one active entry; taking the argmax of logits selects the most likely category.Source: Discrete branch · p. 4
• For coupled actuators, the assembly layer uses the selected mode to choose among feasible continuous setpoints, so the assembled input satisfies the action constraints.The same layer can encode logical dependencies, including an AND gate that enforces a constraint in the forward pass.Source: Assembly Layer · p. 4

C. Differentiable training of the controller

• The argmax selection is piecewise constant with zero gradient almost everywhere, preventing BPTT from training the full hybrid policy.The Gumbel-Softmax reparameterization provides a gradient path through discrete actions.Source: C. Differentiable training of the… · p. 5

1) Gumbel reparameterization trick:

• Gumbel-Softmax adds parameter-free Gumbel noise to logits and replaces argmax with softmax, producing a differentiable relaxation that approaches a one-hot sample as τ tends to zero.The relaxation trades fidelity to discrete sampling for differentiability.Source: 1) Gumbel reparameterization trick · p. 5

C. Differentiable training of the controller

• The straight-through estimator uses a hard one-hot sample in the forward pass and the soft relaxation in the backward pass, allowing gradients to train controller parameters with BPTT.Figure 2 depicts Gumbel noise and hard sampling in the forward path, with gradients flowing through the relaxation in the backward path.Source: Figure 2 · p. 5, 1) Gumbel reparameterization trick · p. 5

2) Straight-Through Gumbel estimator:

• The straight-through estimator avoids using fractional operating modes in the plant while restoring a gradient path through categorical decisions.Directly using the real-valued relaxation would produce actuator modes that cannot be realized.Source: 2) Straight-Through Gumbel estimator · p. 5
• Hard one-hot actions drive the forward rollout, while relaxed actions carry gradients backward through the selection layer for BPTT.The relaxed weights also distribute learning signals across candidate setpoints, even when only one mode executes.Source: 2) Straight-Through Gumbel estimator · p. 5

3) Annealing:

• Annealing lowers the temperature from a high initial value, preserving smoother early gradients while moving training toward the discrete regime.A positive minimum temperature preserves gradient flow, and categorical input constraints remain guaranteed in the forward pass and at deployment.Source: 3) Annealing · pp. 5–6

4) Stochastic exploration and decision margins:

• Training-time Gumbel noise perturbs discrete decisions to promote exploration and encourage logits to remain robust away from switching boundaries.At deployment, noise is disabled and selection uses deterministic arg max.Source: 4) Stochastic exploration and decision… · p. 6
• The architecture enforces hybrid input constraints by construction, leaving an unconstrained optimization over controller parameters and requiring only one neural-network forward pass at deployment.The computationally intensive work is incurred during offline training.Source: 4) Stochastic exploration and decision… · p. 6

III. NEURAL NETWORK CONTROL OF A DHN

• HANC is applied to a district heating network as an example of mixed-action control involving interacting components, thermal dynamics, constraints, and discrete operating decisions.The section outlines the system, a differentiable discrete-time surrogate, and closed-loop rollout training.Source: §III · p. 6

A. System considered

• The RSE S.p.A. benchmark plant has a gas boiler, electric boiler, stratified thermal storage, and four thermal loads connected by a distribution network.The storage’s layered temperature state provides thermal memory for shifting heat production over time.Source: A. System considered · pp. 6–7

B. System modeling

• The topology-guided surrogate combines GRU submodels for loads and pipes with a stratified TES model, analytical boiler relations, and conservation-based network interconnections.Its architecture enforces mass and energy conservation while remaining differentiable.Source: B. System modeling · p. 7
• The identification data come from 20 days of five-minute simulator runs with pseudorandom input excitation and synthetic demand profiles, split chronologically into training, validation, and test sets.The split is 70%, 15%, and 15%, respectively.Source: B. System modeling · p. 7
• The identified surrogate achieves a normalized test RMSE of 3.75 × 10^-2 and FIT index of 87.94% (R2 = 0.886).It has 820 trainable parameters and is used as the differentiable system model for controller training.Source: B. System modeling · p. 7

C. Control design and training

• Controller training samples scenarios and unrolls the controller-surrogate loop over the full horizon, then applies BPTT to the batch loss.Gradient clipping, temperature annealing, and gated recurrent units stabilize full-horizon training without truncation.Source: C. Control design and training · p. 8, Switching regularization · p. 9
• The normalized, regret-like economic loss is anchored to demand-based cost envelopes, making storage arbitrage visible and avoiding gradients dominated by tariff scale or incompressible demand cost.Shifting heat with storage can produce a cost loss below zero.Source: Economic cost · p. 8
• Physical-consistency penalties discourage surrogate-exploiting trajectories, including under-delivery, storage stratification inversion, and energy-balance violations.The one-sided squared softplus penalty grows with violation depth; the energy-balance term accounts for distribution losses and changes in stored energy.Source: Physical-consistency penalties · pp. 8–9
• Operational penalties target supply temperatures below 65°C, while second-difference regularization of relaxed mode weights discourages chattering without taxing each sustained mode transition.A first-difference penalty would bias the policy against using discrete actuators.Source: Operative constraints · p. 9, Switching regularization · p. 9
• A tariff curriculum begins with exaggerated price contrast, then interpolates to realistic tariffs and selects the saved controller under the target distribution.This schedule is intended to identify strategies before specializing to realistic prices.Source: Switching regularization · p. 9

IV. SIMULATION RESULTS

• Performance is assessed in closed loop on an independent high-fidelity Modelica plant driven by real market prices.The evaluation concerns the district heating system described earlier in the paper.Source: §IV · pp. 9–10
• The evaluation uses dynamic electricity pricing and the paper’s operating-cost objective for heat generation.The objective is to select cost-effective heat sources while exploiting storage to shift production in time.Source: §IV · pp. 9–10
• The controller is evaluated against the requirement to meet thermal demand while respecting equipment constraints.The supplied passages describe the evaluation setup but report no comparative outcome or numerical result.Source: §IV · p. 10

• The section introduces the signals supplied to controller Kθ and the control actions it generates.Source: A. Controller signals · p. 10

• Online inputs include heating-station supply and return temperatures and massflows, plus temperatures from the stratified tank’s sensor string.Source: 1) Measurements and context · p. 10
• Current tariffs and thermal demand from four loads are also supplied; measurable demand is folded into the policy context signal.Source: 1) Measurements and context · p. 10

2) Control actions and input constraints:

• The gas-boiler supply temperature is constrained to 65–80 °C, and each executed actuator action remains physically feasible during training and deployment.Actions are sampled every five minutes.Source: 2) Control actions and input constraints · p. 10
• TES flow uses discharge, hold, or charge selection with a continuous magnitude in the active interval; electric-boiler power is gated by active flow.The TES flow intervals are [−0.5, −0.25], {0}, and [0.25, 0.5] kg/s; electric-boiler power is limited to {0} or [20, 140] kW.Source: 2) Control actions and input constraints · p. 10

B. Dataset and training setup

• Training scenarios use synthetic residential demand profiles and two-level electricity tariffs; the dataset contains 360 scenarios spanning 576 five-minute steps.Initial storage temperatures are drawn uniformly from 60–80 °C, and the gas price is drawn between each scenario’s two electricity-price levels.Source: B. Dataset and training setup · p. 10
• Validation uses a high-fidelity Modelica network calibrated against real-network measurements, with real Italian day-ahead energy prices.Source: B. Dataset and training setup · p. 10

1) Qualitative behavior of the HANC:

• During cheap-electricity windows, HANC uses the EB to serve loads and charge the TES, then discharges storage and uses the GB during expensive periods.The rollout shows the TES cycling in step with price regimes, with storage temperatures spanning nearly the full tank range.Source: 1) Qualitative behavior of the HANC · pp. 10–11, Figure 4 · p. 11
• The policy adapts to price windows with different timing and extends its learned behavior beyond the two-day training horizon in a seven-day rollout.It is a reactive mapping from internal state and current measurements to actions, rather than a fixed-window plan.Source: 1) Qualitative behavior of the HANC · p. 11, Figure 5 · p. 11
• Across the four loads, supply temperature stays above 65°C while return temperature remains around 60°C, despite shifts between high-temperature/low-flow and low-temperature/high-flow modes.These rollout results indicate that training accounted for the stated operating constraints.Source: 1) Qualitative behavior of the HANC · p. 11, Figure 6 · p. 12

2) Quantitative benchmark against rule-based strategies:

• The proposed controller reduced mean operating cost by 30% versus the rule-based baseline and was strictly cheaper in all 400 paired rollouts, with per-rollout savings of 28.4%–31.7%.Table I evaluates closed-loop true primary-energy cost on the independent simulator over 400 rollouts with randomized loads and initial TES temperature.Source: 2) Quantitative benchmark against… · p. 12, Table I · p. 12

3) Gumbel versus deterministic straight-through:

• EB gate switching has a median of 51.5 per two-day rollout with deterministic straight-through training versus four with Gumbel training, while operating costs are statistically indistinguishable.The Gumbel policy follows the four-transition pattern required by the two cheap-price windows on almost every sample; even the deterministic controller’s best rollout has 24 switches.Source: 3) Gumbel versus deterministic… · pp. 12–13, Table III · p. 13
• Increasing the deterministic controller’s switching regularization penalty by two orders of magnitude still left excessive jitter, which the authors describe as pathological within this setup.The comparison indicates that the switching behavior persisted despite the larger penalty.Source: 3) Gumbel versus deterministic… · p. 13
• For EB gates, 48% of deterministic-policy logit mass lies within |l| < 1 and its median margin is 1.085; Gumbel-trained gate medians are 3.95 when on and 12.2 when off.The TES margins show a similar pattern: deterministic margins are almost all below 3, while HANC margins are concentrated almost entirely above 10.Source: Figure 8 · p. 13, 3) Gumbel versus deterministic… · p. 13

V. CONCLUSION

• HANC jointly generates categorical actions and continuous setpoints, enabling full-trajectory backpropagation while enforcing complex actuator constraints by design.The Gumbel straight-through trick preserves discrete actions while allowing training with standard backpropagation-through-time approaches.Source: §V · pp. 13–14
• On a more accurate simulator, the controller exploited price-based heat-storage arbitrage and significantly outperformed industry-standard baselines.It was trained exclusively on a physics-informed surrogate model; noise injection also made it more robust to deployment-condition changes.Source: §V · p. 14
• Evaluating HANC on the real plant against which the simulator was calibrated remains a direction for future work.The reported performance is in simulation.Source: §V · p. 14