PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Agent Deep Reinforcement Learning for HVAC Control in Commercial Buildings

Liang Yu, Yi Sun, Zhanbo Xu, Chao Shen, Dong Yue, Tao Jiang, Xiaohong Guan

arXiv:2006.14156v2eess.SYcs.LG

TL;DR

The paper addresses HVAC energy-cost minimization in multi-zone commercial buildings under dynamic pricing, uncertain conditions, and coupled comfort constraints. It reformulates the task as a Markov game and solves it with attention-based multi-agent deep reinforcement learning. Simulations using real-world traces report effective, robust, and scalable control, including substantial cost reductions for 30 zones.

  • Problem

    The paper seeks to minimize HVAC energy cost while maintaining temperature and CO2 comfort despite unknown thermal dynamics, uncertain parameters, coupled constraints, and a large discrete solution space.

  • Method

    The optimization problem is reformulated as a Markov game and solved with a scalable, model-free multi-agent deep reinforcement learning algorithm using an attention mechanism.

  • Results

    When 30 zones are considered, the proposed algorithm reduces average energy cost by 56.50%-75.25% compared with other baselines while maintaining occupant comfort.

  • Takeaways & Limitations

    Simulation results based on real-world traces show effectiveness, robustness, and scalability of the proposed HVAC control algorithm.

Abstract

from arXiv · show

In commercial buildings, about 40%-50% of the total electricity consumption is attributed to Heating, Ventilation, and Air Conditioning (HVAC) systems, which places an economic burden on building operators. In this paper, we intend to minimize the energy cost of an HVAC system in a multi-zone commercial building under dynamic pricing with the consideration of random zone occupancy, thermal comfort, and indoor air quality comfort. Due to the existence of unknown thermal dynamics models, parameter uncertainties (e.g., outdoor temperature, electricity price, and number of occupants), spatially and temporally coupled constraints associated with indoor temperature and CO2 concentration, a large discrete solution space, and a non-convex and non-separable objective function, it is very challenging to achieve the above aim. To this end, the above energy cost minimization problem is reformulated as a Markov game. Then, an HVAC control algorithm is proposed to solve the Markov game based on multi-agent deep reinforcement learning with attention mechanism. The proposed algorithm does not require any prior knowledge of uncertain parameters and can operate without knowing building thermal dynamics models. Simulation results based on real-world traces show the effectiveness, robustness and scalability of the proposed algorithm.

I. INTRODUCTION

The paper targets model-free HVAC energy-cost minimization in multi-zone commercial buildings while accounting for occupancy, thermal comfort, and indoor air quality. It formulates the problem as a Markov game and proposes scalable multi-agent deep reinforcement learning with attention.

  • Motivation: Commercial HVAC systems account for about 40%-50% of electricity consumption in commercial buildings, creating an economic burden for operators.Building thermal inertia also makes HVAC systems candidates for demand side management.
  • Motivation: Existing optimization approaches require building thermal dynamics knowledge and face limitations in complex HVAC control settings.The paper motivates learning-based methods because they can learn policies from environment interactions without explicit thermal models.
  • Problem: The objective is to minimize long-term HVAC energy cost under dynamic prices while maintaining comfortable temperature and CO2 concentration ranges with random occupancy.The controlled variables are each zone’s air supply rate and the AHU damper position.
  • Approach: The paper formulates the optimization problem as a Markov game incorporating random occupancy, temperature comfort, and CO2 comfort.The formulation designs the environment state, actions, and reward function without requiring building thermal dynamics models.
  • Approach: A scalable multi-agent deep reinforcement learning controller with an attention mechanism is proposed as a model-free solution requiring no prior knowledge of uncertain parameters.The paper reports effectiveness, robustness, and scalability in simulations using real-world traces.
  • System model: The HVAC model includes an AHU and zone-level VAV boxes, with discrete air-supply rates and AHU damper positions.Indoor temperature depends on neighboring zones, outdoor temperature, air supply, and disturbances, while CO2 depends on occupancy and ventilation-related variables.

B. HVAC Energy Cost Model

The HVAC energy-cost model combines supply-fan and cooling-coil energy costs, both weighted by the time-varying electricity price.

  • Energy-cost components: HVAC energy cost consists of supply-fan and cooling-coil components.The supply-fan power is approximated by a cubic function of total air supply rate.
  • Supply fan: The supply-fan energy cost is determined by electricity price and the cubic total air-supply term.The cubic dependence makes the supply-fan cost coupled across zones.
  • Cooling coil: Cooling-coil power depends on air supply rates and the AHU damper position through mixed-air and supply-air temperatures.The model uses air specific heat, coil efficiency, and chiller coefficient of performance.
  • Cooling coil: The cooling-coil energy cost is obtained by multiplying cooling-coil power by electricity price over the time slot.The formulation expresses the cost using the cooling-coil power model and the supply-air temperature.

C. Energy Cost Minimization Problem

The paper formulates a stochastic long-term HVAC cost-minimization problem whose difficulty comes from unknown dynamics, coupled constraints, discrete actions, non-separable costs, and uncertain parameters.

  • Formulation: The stochastic program minimizes expected long-term HVAC energy cost, with zone air-supply rates and AHU damper position as decision variables.The expectation covers random electricity prices, outdoor temperatures, and occupant numbers.
  • Challenges: Unknown thermal dynamics and spatially and temporally coupled temperature and CO2 constraints make the optimization difficult.Future zone temperature depends on prior temperatures, while CO2 concentration couples information across zones.
  • Challenges: The objective is non-convex and non-separable across zones, while electricity prices, outdoor temperatures, and occupancy remain uncertain.The paper also notes that prior DDPG and DQN HVAC methods do not jointly address the problem’s discrete-action or scalability requirements.

D. Energy Cost Problem Reformulation

The optimization is reformulated as a Markov game with one agent per zone plus one AHU agent. Local observations, coordinated actions, and distributed reward components represent cost and comfort objectives.

  • Game formulation: The Markov game uses N + 1 agents: one for each zone and one for the AHU damper position.Each agent selects actions from local observations while optimizing expected return.
  • State: The global state is formed from local observations containing temperatures, neighboring-zone information, electricity price, time index, occupancy, and CO2 levels.The AHU agent additionally observes occupancy and CO2 information from all zones.
  • Action: Zone agents choose air-supply rates, while the AHU agent chooses the damper position.The joint action is (m1,t, m2,t, · · ·, mN,t, σt).
  • Reward: The reward function combines energy-cost penalties with temperature-deviation and CO2-violation penalties.Positive weights α and β scale the reward components associated with cost and CO2 comfort.
  • Reward: The supply-fan penalty is imposed on zone agents because total fan energy depends non-separably on their air-supply rates.The cooling-coil penalty is distributed across all agents because coil energy depends on both zone air supply and AHU damper position.
  • Coordination: Agents exchange state and reward information before acting, then observe the next state and calculate rewards after the transition.This coordination supports the coupled multi-zone control problem.

III. MADRL-BASED HVAC CONTROL ALGORITHM

The proposed controller uses a multi-agent actor-critic framework with soft actor-critic exploration and attention-based critics to solve the HVAC Markov game.

  • Basic Principle of MAAC Approach: MAAC learns approximations of both policy and action-value functions in an actor-critic framework.The action-value function represents expected return and is learned through off-policy temporal-difference learning.
  • Critic Training: Critic networks are jointly trained with a regression loss using target values computed from rewards, discounted future values, and replay-buffer transitions.The target value function and experience replay buffer support off-policy learning.
  • Policy Training: Policy parameters are updated with policy gradients based on the learned action-value function.The actor network parameter is denoted by θ, while ψ denotes the critic-network weight parameter.
  • Soft Actor-Critic: Soft actor-critic training adds an entropy term to balance exploration against reward maximization and avoid non-optimal deterministic policies.The temperature parameter ϕ controls this balance, while b(s) is a state-dependent baseline.
  • Attention Mechanism: The critic uses attention to selectively incorporate information from other agents when calculating each agent’s action-value function.Shared key, query, and value transformations produce attention weights over other agents’ contributions.

3 Initialize the weights of target networks Q

The training and execution procedures initialize parallel agents, collect transitions, update critics and policies from replay data, and softly update target networks.

  • Network Updates: Training samples mini-batches of transitions from the replay buffer before updating the critic network with the joint regression loss.The critic update is specified as minimizing loss function (23).
  • Network Updates: The policies are updated using equation (24), followed by soft updates of target critic and actor networks.The target parameters combine current and previous parameters using ξ.
  • Execution Algorithm: The execution algorithm initializes local observations and has all agents select and execute actions in parallel.Each agent uses the learned policy πθ(·|oi,t) at the beginning of each time slot.
  • Experience Collection: Each agent receives a next observation after acting, while transitions are sent to environments and stored in the replay buffer.Stored transitions contain observations, actions, next observations, and rewards.

B. The Proposed MADRL-based HVAC Control Algorithm

The proposed HVAC controller separates model training from policy execution: agents learn through environment interaction, then act using fixed actor-network weights.

  • Training Algorithm: During training, each agent interacts with the multi-zone building and stores transition tuples in experience replay buffer D.Mini-batches are sampled once the buffer length exceeds the batch size, enabling network updates from past transitions.
  • Execution Algorithm: After training, actor-network weights remain fixed while the learned policies determine actions from current local observations.The execution procedure repeats observation, action, and next-observation steps until testing ends.
  • Execution Algorithm: Because decisions use only the current observation, the trained controller performs online HVAC actions without updating actor weights during testing.

A. Experimental Setup

Simulations combine real-world electricity-price, outdoor-temperature, and occupancy traces with baseline HVAC schemes and training configurations.

  • Data Sources: Simulations use Beijing commercial Time-of-Use prices, Austin hourly outdoor temperatures, and zone occupancy traces from prior data sources.The traces cover June 1 to August 31, 2018, and occupancy data are adapted because only three traces are available.
  • Sensitivity Analysis: Figure 5 examines how varying α and β changes energy cost, CO2 concentration deviation, and temperature-related performance.The caption states that 95% confidence intervals are computed across 10 runs.
  • Training Configuration: Training configurations include actor and critic learning rates, hidden-layer sizes, and experience-replay capacity.A larger replay buffer is selected for larger numbers of agents because it affects convergence.
  • Baselines: The experiment evaluates the proposed method against rule-based and heuristic HVAC control schemes.The rule-based scheme uses an ON/OFF temperature-control policy, while the heuristic scheme uses building thermal dynamics models.

C. Comfort-Related Performance Metrics

The paper uses Average Temperature Deviation (ATD) and Average CO2 Concentration Deviation (ACD) to summarize thermal and indoor-air-quality discomfort across occupied zones and time slots.

  • Average Temperature Deviation (ATD) measures thermal discomfort across occupied zones and time slots.
  • Average CO2 Concentration Deviation (ACD) measures indoor air quality discomfort across occupied zones and time slots.
  • The metric definitions account for the total number of occupied slots associated with each zone.

D. Algorithmic Performance under Varying α and β

The proposed algorithm balances energy cost against temperature and CO2 comfort through tunable weights, achieves favorable performance across tested conditions, and remains robust under disturbances and larger zone counts.

  • D. Algorithmic Performance under Varying α and β: Increasing α and decreasing β generally reduce mean total energy cost while increasing average CO2 concentration deviation.α weights energy cost relative to temperature violation, whereas β weights CO2 concentration deviation relative to temperature violation.
  • D. Algorithmic Performance under Varying α and β: Increasing either α or β increases the mean average temperature deviation, enabling a flexible energy-cost and comfort tradeoff.The paper states that practitioners can select α and β according to tolerable ATD and ACD.
  • E. Algorithmic Effectiveness: The proposed algorithm satisfies the given conditions and achieves the lowest total energy cost among the three schemes.
  • E. Algorithmic Effectiveness: Coordinating zones allows the algorithm to reduce air supply when electricity prices are high and increase it when prices are low.It also selects a larger σ_t while maintaining indoor air quality comfort, which helps save energy.
  • F. Algorithmic Robustness: The proposed algorithm achieves the best performance under three random-disturbance scenarios, supporting robustness to such disturbances.The scenarios use υ_i values of 1°C, 2°C, and 3°C.

G. Algorithmic Scalability

The proposed algorithm remains effective as the number of zones grows, achieving substantial energy-cost reductions while maintaining comfort. However, imposing a practical upper limit on total air supply would reduce these improvements.

  • Scalability: A larger experience replay buffer is required to facilitate convergence when the number of agents increases.The attention mechanism supports scalability to more agents and zones, but larger experiments require greater replay-buffer capacity.
  • Convergence: The proposed algorithm’s episode rewards generally increase and become more stable during training.With 20,000 episodes, the average training time is about 13 hours; fluctuations reflect exploration and varying system parameters.
  • Performance: The reductions arise because the algorithm lowers total air supply across zones when electricity prices are high.This behavior is depicted in Fig. 8(c).
  • Practical boundary: Imposing an upper limit on total air supply in practice would decrease the achieved performance improvements.The impact of this limit is identified as a topic for future investigation.
Loading 2006.14156v2…