PaperlayerView exact revision on arXiv

Source-linked AI summary

A Review of Deep Reinforcement Learning for Smart Building Energy Management

Liang Yu, Shuqi Qin, Meng Zhang, Chao Shen, Tao Jiang, Xiaohong Guan

arXiv:2008.05074v3eess.SYeess.SP

TL;DR

Buildings account for substantial energy use and emissions, while SBEM must handle difficult thermal modeling, uncertainty, coupled constraints, large optimization spaces, and restrictive premises. The paper comprehensively reviews DRL for SBEM by system scale, surveys its methods and applications, and identifies unresolved issues and future directions. It concludes that model-based DRL is more practical than model-free DRL, while DRL has potential to improve building performance metrics.

  • Problem

    SBEM lacks a systematic overview of DRL methods despite significant building energy and carbon impacts and multiple control challenges.

  • Method

    The paper comprehensively reviews DRL applications and methods for SBEM from the perspective of single subsystems, multiple subsystems, and building microgrids.

  • Results

    Model-based DRL approaches are more practical than model-free approaches because they can generate training data and reduce interactions with real environments.

  • Takeaways & Limitations

    DRL-based energy management has potential to improve energy cost, peak load, and occupant dissatisfaction metrics compared with some traditional methods.

Abstract

from arXiv · show

Global buildings account for about 30% of the total energy consumption and carbon emission, raising severe energy and environmental concerns. Therefore, it is significant and urgent to develop novel smart building energy management (SBEM) technologies for the advance of energy-efficient and green buildings. However, it is a nontrivial task due to the following challenges. Firstly, it is generally difficult to develop an explicit building thermal dynamics model that is both accurate and efficient enough for building control. Secondly, there are many uncertain system parameters (e.g., renewable generation output, outdoor temperature, and the number of occupants). Thirdly, there are many spatially and temporally coupled operational constraints. Fourthly, building energy optimization problems can not be solved in real-time by traditional methods when they have extremely large solution spaces. Fifthly, traditional building energy management methods have respective applicable premises, which means that they have low versatility when confronted with varying building environments. With the rapid development of Internet of Things technology and computation capability, artificial intelligence technology find its significant competence in control and optimization. As a general artificial intelligence technology, deep reinforcement learning (DRL) is promising to address the above challenges. Notably, the recent years have seen the surge of DRL for SBEM. However, there lacks a systematic overview of different DRL methods for SBEM. To fill the gap, this paper provides a comprehensive review of DRL for SBEM from the perspective of system scale. In particular, we identify the existing unresolved issues and point out possible future research directions.

I. INTRODUCTION

The paper reviews DRL for smart building energy management (SBEM) because building energy use is substantial and SBEM faces modeling, uncertainty, constraint, real-time optimization, and versatility challenges. It organizes the literature by building system scale, surveys DRL methods, and identifies unresolved issues and future directions.

  • Global buildings consumed 30% of total energy and generated 28% of total carbon emissions in 2019.
  • SBEM is challenged by inaccurate thermal models, uncertain parameters, coupled operational constraints, large solution spaces, and restrictive method premises.Examples of uncertainty include renewable generation, electricity price, temperatures, CO2 concentration, and occupancy.
  • DRL can learn control policies from environmental interaction without explicit thermal models or forecasts and can coordinate coupled constraints through reward design.Multi-agent DRL addresses spatial coordination, while action and reward design can address temporal constraints.
  • A trained DRL agent can generate control actions through neural-network forward propagation in a few milliseconds, supporting real-time operation.
  • DRL methods use simulated or real data, avoid rigorous mathematical premises, and may adapt to varying environments through online learning.
  • The review classifies SBEM applications by system scale: single subsystems, multiple building energy subsystems, and building microgrids.It systematically surveys DRL methods and highlights unresolved issues and future research directions.

II. AN OVERVIEW OF DEEP REINFORCEMENT LEARNING

The paper introduces deep reinforcement learning as the combination of deep learning and reinforcement learning, contrasting it with other machine-learning feedback paradigms. It then frames DRL methods within Markov decision processes, where agents select actions from observed states to maximize future returns under uncertainty.

  • Machine learning is classified by feedback into supervised learning, unsupervised learning, and reinforcement learning.Supervised learning uses labeled prediction errors, unsupervised learning lacks labeled feedback, and reinforcement learning receives delayed feedback through environmental interaction.
  • DRL combines deep learning with reinforcement learning by using deep neural networks to approximate value functions or policies.This combination provides representation and decision-making capabilities under uncertainty.
  • A. MDP: An MDP is defined by (S, A, P, R, γ), representing states, actions, transitions, rewards, and discounting for multi-stage decisions under uncertainty.At each time slot, an agent observes a state, acts, receives a reward, and transitions to a new state while maximizing expected future return.

B. RL

Reinforcement learning learns policies through environment interaction, while Q-learning estimates action values and DRL stabilizes this approach for higher-dimensional states. DRL methods are classified by whether they require environment models and by whether they learn values or policies.

  • RL agents learn policies by repeatedly observing states, taking actions, receiving rewards, and updating policies until convergence.
  • Q-learning learns an action-value function rather than the policy directly and approximates the optimal action-value function without requiring transition probabilities.
  • Neural-network function approximation supports high-dimensional state spaces but can make RL unstable or divergent.
  • DQN stabilizes Q-learning using experience replay and target networks, with replay storing and randomly sampling experience transitions for training.
  • Model-free DRL learns directly from unknown environments, whereas model-based DRL constructs environment models; model-free methods include value-based and policy-based approaches.

III. DRL-BASED SMART BUILDING ENERGY MANAGEMENT

SBEM addresses the operation of diverse building energy equipment through optimization problems that incorporate economic, environmental, and social impacts. HVAC control provides a representative comfort-aware formulation with uncertain parameters and zone-coupled dynamics.

  • SBEM covers equipment including PVs, wind turbines, generators, storage systems, HVAC, lighting, blinds, windows, water heaters, vehicles, and appliances.
  • HVAC systems are used to illustrate SBEM because they consume substantial power and can be adjusted flexibly while preserving occupant comfort.
  • The representative HVAC problem minimizes expected energy cost while accounting for thermal comfort and building-zone dynamics.
  • Decision variables include each zone’s air supply rate and the air-handling unit’s damper position.

B. Procedure of Solving SBEM Problems using DRL

Solving SBEM with DRL involves reformulating the optimization problem, selecting a suitable algorithm, analyzing training and testing complexity, and evaluating multiple performance dimensions.

  • The original SBEM problem is reformulated as an MDP or Markov game, with multiple agents used to improve scalability when solution spaces grow with zone number.
  • An appropriate DRL algorithm is designed for the reformulated problem, such as attention-based multi-agent DRL for scalable HVAC control.
  • Testing requires only neural-network forward propagation, enabling real-time decisions within several milliseconds even for high-dimensional states.
  • Training complexity depends on network architecture, convergence episodes, weight-update frequency, and batch size.
  • The designed algorithm is evaluated for convergence, effectiveness, scalability, and robustness.

C. Representative DRL Algorithms for SBEM

Representative DRL algorithms for SBEM are organized into model-free and model-based approaches, each with distinct data, modeling, and action-space trade-offs. The review also groups applications by system scale, from single subsystems to multiple subsystems and microgrid environments.

  • DRL algorithms for SBEM are divided into model-free and model-based categories.
  • Model-free methods avoid explicit building models but require sufficient experience, causing potentially long and costly exploration.
  • Model-free algorithms differ in action-space support: DQN handles discrete actions, DDPG continuous actions, and PPO and A2C/A3C both types.
  • Model-based methods simulate building environments to generate training episodes, improving sample complexity but requiring accurate environment models.
  • Applications are reviewed across single subsystems, multiple energy subsystems, and building energy systems in microgrid environments.

A. Model-free DRL Methods for HVAC Control

DRL-based HVAC control methods target energy reduction and thermal comfort while addressing the cost and duration of real-environment exploration. Model-based frameworks reduce real interactions by training agents with calibrated or data-driven simulators before deployment.

  • Multi-agent DDQN agents coordinate building and chiller set-points to reduce HVAC energy consumption without sacrificing user comfort.
  • Real-building training is constrained because exhaustive exploration can incur unacceptably high costs and require long learning times.
  • BEM-based DRL uses building-energy modeling, calibration, offline A3C training, and real-time deployment through a building automation system.
  • 16.7% reduction in heating demand was achieved compared with the rule-based control strategy.
  • Historical HVAC operational data can create training environments and pre-train differentiable MPC policies before PPO-based improvement.

C. DRL Methods for EWH Control

DRL methods for EWH control address sparse observations, sample-efficiency requirements, and complex system dynamics through recurrent, model-based, or model-informed approaches. Reviewed work includes sequence-based state representations and energy optimization in residential deployments.

  • LSTM-based fitted Q-iteration stores past observation and action sequences in the state vector for sparsely observed EWH control.
  • Model-based DRL is preferred when interacting with actual EWH environments would require many samples.
  • About 20% energy-consumption reduction was reported for Deep PILCO hot-water optimization across 32 houses in the Netherlands.
  • Deep PILCO alternates experience collection, dynamics-model training, trajectory generation, policy updates, and subsequent policy execution.
  • DRL can obtain EWH operating policies from complex known dynamics models by using model input-output relationships.
  • Most reviewed single-subsystem work targets HVAC, with energy cost reductions of 4%-71.2% and energy-consumption decreases of 12.4%-34.5% without sacrificing thermal comfort.

V. APPLICATIONS OF DRL IN MULTIPLE ENERGY SUBSYSTEMS OF BUILDINGS

DRL applications for multiple building energy subsystems coordinate residential and commercial equipment to reduce energy costs while considering comfort and other operational objectives. The review reports substantial potential but notes that most methods remain simulation-evaluated.

  • Residential applications coordinate HVAC, ESSs, EVs, WMs, PVs, and EWHs, while commercial applications coordinate HVAC, lighting, blinds, windows, and personal electric devices.
  • Joint scheduling of HVAC systems and ESSs reduced residential energy cost by 8.1%-15.21% in simulation results.
  • Residential multi-subsystem optimization also considers peak demand, transformer-capacity violations, and revenue from excess renewable energy.
  • Commercial-building studies increasingly consider interactions among subsystems affecting energy consumption, thermal comfort, air quality, and illumination.
  • Occupant-behavior shaping can be incorporated through recommenders that learn energy-saving actions from occupant feedback.
  • Up to 59% relative energy-cost reduction was reported while maintaining occupant comfort, and most methods were evaluated by simulations.

VI. APPLICATIONS OF DRL IN BUILDING MICROGRIDS

This section reviews DRL-based energy optimization for building microgrids, covering systems with uncontrollable and flexible building loads. Existing methods mainly target economic benefits, while practical implementation and building-load control remain limited.

  • Organization: The section organizes prior work into microgrids with uncontrollable building loads and microgrids that consider building-load flexibility.The review summarizes these works in Table VII and provides insights at the section’s conclusion.
  • Microgrids with uncontrollable building loads: DRL-based methods have been proposed for residential microgrids involving distributed generation, storage devices, responsive loads, load shedding, and peer-to-peer energy trading.Examples include DQN-based control with battery and hydrogen storage, penalties for non-served demand, and energy trading among multiple microgrids.
  • Microgrids with uncontrollable building loads: DDPG-based methods support continuous-action microgrid management, while model-based MuZero addresses online residential-microgrid scheduling under uncertainties.The cited methods optimize generation cost, power imbalance, or scheduling performance with different model assumptions and action types.
  • Microgrids with flexible building loads: Flexible scheduling of building loads can reduce microgrid energy costs by controlling workload execution time and served workload quantity.For a data-center microgrid with renewable sources, simulation results reported a 6.42% energy-cost reduction.
  • Summary: The reviewed literature mainly focuses on the economic impacts of building microgrids and reports economic benefits for microgrid operators.The section’s summary identifies limited attention to building-load control and a lack of practical implementations.

VII. OPEN ISSUES AND FUTURE RESEARCH DIRECTIONS

The paper identifies unresolved SBEM issues involving practical deployment, multiple timescales, and conflicting objectives. It proposes offline or model-based learning, hierarchical DRL, and improved reward design as directions for future work.

  • Data-efficient building energy optimization: Most model-free DRL methods for SBEM are not implemented in practice because training requires lengthy real-environment interaction and costly trial-and-error exploration.Random HVAC temperature-setpoint actions can cause thermal discomfort and high energy consumption.
  • Data-efficient building energy optimization: Offline training from sensed building-operation data, meta reinforcement learning, and learned environment models could reduce data requirements and exploration costs.The proposed approaches respectively use collected data, few data and training episodes, or model-based DRL to limit real-environment interactions.
  • Multi-timescale building energy optimization: Existing DRL methods generally address single-timescale optimization, although building HVAC decisions may operate at different update intervals.Supply-air temperature and re-use-air ratio may be adjusted hourly, whereas zone supply-air rates can change every 10–15 minutes.
  • Multi-timescale building energy optimization: Hierarchical DRL is proposed for multi-timescale problems by assigning long-timescale actions to an upper level and short-timescale actions to a lower level.The lower level selects actions using the system state and the upper-level choices, coordinating decisions with delayed rewards.
  • Multiple optimization objectives: SBEM objectives such as energy cost, energy consumption, carbon emissions, and comfort can conflict, making weighted synthetic rewards difficult to configure.Different objective weights may have different units or scales, and the resulting policies may not support flexible operating modes.

D. Multi-zone building energy optimization

For multi-zone and multi-building SBEM, existing DRL methods face scalability and training-data challenges. The review highlights coordination, transfer learning, and broader system-scale integration as continuing research needs.

  • Multi-zone building energy optimization: Multi-zone HVAC DRL methods have been demonstrated for up to 30 zones, but scalability remains difficult as practical commercial buildings may contain more zones.Separate zone training lacks coordination, MADDPG critics grow with concatenated agent information, and more scalable multi-agent algorithms are needed.
  • Multi-zone building energy optimization: Existing HVAC DRL control methods mainly focus on single-zone buildings, leaving multi-zone coordination and scalability insufficiently addressed.The cited approaches include separately trained agents, MADDPG, and MAAC-based control for increasingly larger zone configurations.
  • Efficient training of DRL agents in multi-building energy optimization: Model-based DRL is sample-efficient, but learning building thermal-dynamics models requires substantial historical data that may be unavailable for brand-new buildings.Limited historical data makes accelerating DRL-agent training a challenging problem.
  • Efficient training of DRL agents in multi-building energy optimization: Transfer learning could pre-train thermal-dynamics models or control policies, but large source–target MDP gaps make mapping and knowledge selection difficult.The difficulty increases when state or action dimensions differ and in multi-agent SBEM settings.
  • Conclusions and lessons learned: The review concludes that DRL may improve energy cost, peak load, and occupant dissatisfaction degree simultaneously compared with some traditional methods.It also identifies continuing challenges involving data efficiency, multiple timescales, objectives, zones, buildings, and microgrids.
Loading 2008.05074v3…