Source-linked AI summary
Generalizing HVAC Control With Domain Randomized Reinforcement Learning
Pablo Boitel, Kun Zhang
TL;DR
Scalable HVAC control needs robust transfer across heterogeneous zones without accurate per-site models or retuning. NOMAD-RL combines recurrent thermostat-level control with physics-informed adaptive domain randomization, and it outperforms PID and non-randomized RL while approaching MPC. The study is preliminary, using limited training and hyperparameter tuning.
Problem
Scalable HVAC control lacks robust transfer across heterogeneous thermal zones without accurate building models or per-site retuning.
Method
NOMAD-RL combines a recurrent thermostat-interface policy with physics-informed neural-flow domain randomization over correlated, multimodal thermal-zone parameters.
Results
NOMAD-RL consistently outperforms PID and non-randomized RL baselines while approaching well-tuned MPC in single-zone and multi-zone settings.
Takeaways & Limitations
Adaptive, physics-informed domain randomization supports transferable HVAC control without accurate models, with the strongest reported margin occurring in multi-zone settings.
Takeaways & Limitations
Training was limited to a preliminary 6e6-step run on a single CPU/GPU with limited hyperparameter tuning.
Abstract
from arXiv · showhide
Deploying advanced HVAC (Heating, Ventilation and Air Conditioning) controllers at scale remains difficult because performance often depends on accurate building models or per-site retuning. We propose NOMAD-RL (Neural Online Meta-Adaptation for Dynamics), a general-purpose Reinforcement Learning (RL) controller designed to transfer across heterogeneous thermal zones through a universal, non-invasive thermostat interface. The controller acts on temperature setpoints from zone measurements and forecasts, while a recurrent policy supports online adaptation under partial observability. Our main contribution is an adaptive domain randomization scheme based on physics-informed normalizing flows, which models correlated and multimodal distributions of thermal-zone parameters while maintaining physical plausibility and controllability. This produces a realistic and progressively adaptive training curriculum that improves transfer across buildings. We evaluate NOMAD-RL against a constant-setpoint PID controller, RL without domain randomization, and MPC in single- and multi-zone settings. NOMAD-RL consistently outperforms the PID and non-randomized RL baselines, and approaches the performance of a well-tuned MPC, especially in the more challenging multi-zone case. These results highlight the potential of adaptive, physics-informed domain randomization for robust and transferable HVAC control.
1 Introduction
NOMAD-RL targets scalable HVAC control across heterogeneous thermal zones without relying on per-site modeling or retuning. It combines recurrent online adaptation with physics-informed adaptive domain randomization and outperforms simpler baselines while approaching well-tuned MPC.
- Buildings use 30% of global final energy and produce 26% of energy-related CO2 emissions, making HVAC control a promising cost- and emissions-reduction lever.
- MPC balances comfort, energy, and constraints but is difficult to deploy broadly because accurate models require substantial derivation, calibration, and maintenance.RL can also depend implicitly on calibrated simulators through its training environments.
- NOMAD-RL uses a universal thermostat interface and recurrent policies to transfer across heterogeneous zones under partial observability and online adaptation.
- Its adaptive domain randomization models correlated and multimodal thermal-zone parameters with neural spline flows and physics-informed regularization.The regularization preserves physical plausibility and controllability while the training distribution remains diverse.
- Across single-zone and multi-zone experiments, NOMAD-RL outperforms constant-setpoint and non-randomized RL baselines while approaching well-tuned MPC.
2 Method
The method treats HVAC control as partially observable supervisory setpoint control and trains a recurrent policy across randomized thermal-zone dynamics. An adaptive flow-based curriculum selects realistic, informative environments while limiting training to a preliminary 6e6-step run with limited tuning.
- The controller adjusts thermostat setpoints rather than directly actuating the heating plant, leaving low-level actuation to an existing PID loop.This supervisory interface supports safety, interpretability, and deployment across buildings.
- HVAC control is formulated as a POMDP, so a recurrent policy uses limited measurements, delayed effects, and interaction history to infer hidden dynamics.
- Domain randomization: Each episode samples 25 simulator parameters from p_phi(xi), optimizing expected return across a family of environments rather than one nominal model.The simulator includes building, internal-mass, heat-pump, and PID parameters.
- Domain randomization: Adaptive domain randomization addresses the trade-off between overfitting from narrow distributions and instability from broad ones using neural spline flows.The flow represents correlated, multimodal, high-dimensional parameter distributions and is fitted to a Boltzmann target by weighted maximum likelihood.
- Domain randomization: Candidate environments are uniformly sampled, locally refined by gradient steps, and scored using policy return, physical plausibility cost, and an exploration bonus.The resulting utility assigns higher relevance to more useful parameter vectors.
- Domain randomization: The learned distribution is updated with Boltzmann weighting and a KL trust region, then expanded gradually as policy performance improves.
- Training configuration: Training used a preliminary 6e6-step run on a single CPU/GPU with limited hyperparameter tuning.
3 Results and evaluation of NOMAD-RL
NOMAD-RL is evaluated against PID, recurrent PPO without domain randomization, and MPC in single- and multi-zone HVAC settings. It outperforms PID and recurrent PPO on shifted and coupled zones while remaining closer to MPC in the more challenging multi-zone case.
- Evaluation setup: NOMAD-RL is compared with constant-setpoint PID, recurrent PPO without domain randomization, and MPC using cost-based HVAC evaluation.Costs incorporate energy consumption, comfort, and actuator saturation; MPC uses the exact thermal-zone model in the sampled-zone comparison.
- Single-zone setting: 70.4% lower cost than PID was achieved by NOMAD-RL on the nominal thermal zone, although recurrent PPO was 8.7% less costly.Both RL controllers outperformed the constant-setpoint PID baseline.
- Single-zone setting: 60.1% lower cost than PID and 51.4% lower cost than recurrent PPO were achieved by NOMAD-RL on a sampled zone with shifted dynamics.NOMAD-RL was 46.6% more costly than MPC, which had access to the exact model.
- Multi-zone setting: 67.9% lower cost than constant-setpoint PID was achieved by NOMAD-RL in the coupled 3-zone building.The multi-zone evaluation aggregates building-level cumulative cost across three thermally coupled zones.
- Multi-zone setting: 14.5% more costly than MPC was NOMAD-RL in the coupled 3-zone setting, narrowing the gap to MPC as problem complexity increased.The reported result indicates effectiveness in the more challenging multi-zone environment.
4 Conclusion
The conclusion presents NOMAD-RL as a transferable HVAC controller using a universal thermostat interface and physics-informed adaptive domain randomization. Reported results show strong gains over PID and non-randomized RL, while future work targets real-building validation and improved readiness.
- Controller design: NOMAD-RL uses a universal thermostat interface intended to support safety, interpretability, practical deployability, and lightweight operation.The supervisory approach avoids direct low-level plant control and is described as suitable for embedded devices or connected thermostats.
- Adaptive domain randomization: Physics-informed normalizing flows model correlated and multimodal thermal-zone parameters while enforcing physical plausibility and controllability.The resulting curriculum progressively expands as the recurrent policy improves and supports online adaptation from observations and forecasts.
- Results: 67.9% lower cost than PID and a 14.5% gap to MPC were reported for NOMAD-RL in the multi-zone setting.In single-zone settings, NOMAD-RL also outperformed PID and non-randomized RL while remaining within 46.6% of MPC.
- Future work: Future work includes further training, hyperparameter tuning, simulator and dataset expansion, and validation on real buildings.The authors also plan HVAC benchmark evaluation and exploration of larger architectures, transformers, and flow matching.