Source-linked AI summary
On-line Building Energy Optimization using Deep Reinforcement Learning
Elena Mocanu, Decebal Constantin Mocanu, Phuong H. Nguyen, Antonio Liotta, Michael E. Webber, Madeleine Gibescu, J. G. Slootweg
TL;DR
Building energy scheduling requires online optimization amid complex consumption, renewable-generation, and user-behavior data, while existing approaches can be time consuming for large real databases. The paper applies deep reinforcement learning through DQN and DPG, extending scheduling to simultaneous actions and evaluating both methods on PecanStreet data. DPG is reported as more suited than DQN for online scheduling, with online decisions made in a few milliseconds after learning.
Problem
Existing building-energy optimization methods do not consider online solutions effectively for large-scale real databases because they repeatedly compute possible solutions.
Method
The paper adapts deep reinforcement learning for online building-energy scheduling, investigating DQN and DPG with simultaneous device actions.
Results
DPG is more suited than DQN for online energy-resource scheduling at both building and aggregated levels.
Takeaways & Limitations
After learning, DRL can make control decisions in a few milliseconds, supporting online scheduling of electricity-consuming devices.
Abstract
from arXiv · showhide
Unprecedented high volumes of data are becoming available with the growth of the advanced metering infrastructure. These are expected to benefit planning and operation of the future power system, and to help the customers transition from a passive to an active role. In this paper, we explore for the first time in the smart grid context the benefits of using Deep Reinforcement Learning, a hybrid type of methods that combines Reinforcement Learning with Deep Learning, to perform on-line optimization of schedules for building energy management systems. The learning procedure was explored using two methods, Deep Q-learning and Deep Policy Gradient, both of them being extended to perform multiple actions simultaneously. The proposed approach was validated on the large-scale Pecan Street Inc. database. This highly-dimensional database includes information about photovoltaic power generation, electric vehicles as well as buildings appliances. Moreover, these on-line energy scheduling strategies could be used to provide real-time feedback to consumers to encourage more efficient use of electricity.
I. INTRODUCTION
The paper targets online building-energy optimization using deep reinforcement learning, motivated by complex smart-grid conditions and large-scale metering data. It evaluates DQN and DPG, extending them for simultaneous device actions and testing them on building and aggregated data.
- Large-scale AMI data can support understanding individual consumption behavior to optimize building energy use.
- Existing building-energy optimization methods are time consuming because they repeatedly compute possible solutions and have limited online applicability to large real databases.
- The study proposes online DRL methods to learn electrical patterns and optimize building energy consumption or cost.
- DQN and DPG are investigated, with DQN enhanced to handle multiple simultaneous device actions.
- The methods are evaluated on the PecanStreet database at both building and aggregated levels under renewable-generation and electricity-use variability.
- The formulation addresses both load-peak reduction and energy-cost minimization for flexible electrical devices, including air conditioning and electric vehicles.
III. BACKGROUND AND PRELIMINARIES
This section introduces reinforcement learning, Markov decision processes, and deep neural networks as background concepts.
- The section provides a brief overview of reinforcement learning, Markov decision formalism, and deep neural networks.
A. Reinforcement Learning
The paper frames building energy management as reinforcement learning in a Markov decision process, where an agent learns actions that maximize future rewards.
- An RL agent learns to act within a partially observable Markov Decision Process defined by states, actions, transitions, and rewards.
- The transition function gives the probability of reaching a next state after selecting an action in the current state.
- The reward function specifies the immediate reward after a transition, while the policy determines action selection.
- In the building environment, states include energy consumption and price, while actions depend on electrical-device constraints.
- The value function Qπ(s, a) is the expected total reward from state s using action a under policy π.
B. Deep Neural Networks
Deep neural networks provide the layered function approximators used in deep reinforcement learning, mapping MDP states to actions. The paper combines reinforcement learning with DNNs and introduces DQN and DPG as its DRL methods.
- DNN structure: DNNs use multiple neuron layers, with hidden layers applying nonlinear transformations to outputs from preceding layers.The architecture can be represented using composite functions across stacked hidden layers.
- Activation functions: ReLU applies max(0, x_i) to each input, while the paper also uses a relaxed form with a nonzero negative slope.The relaxed activation becomes ReLU when η = 0.
- DRL mapping: In DRL, the DNN receives MDP states as inputs and produces MDP actions as outputs.This state-to-action mapping is the interface between the neural network and the reinforcement-learning environment.
- Proposed DRL architecture: The proposed DRL method performs on-line building resource allocation using a combination of reinforcement learning and DNNs.The approach is intended to learn patterns in electricity consumption and operate at different aggregation levels.
- DRL methods: DQN is value-based, whereas DPG directly parameterizes the policy and updates its parameters by gradient ascent on expected long-term reward.The two methods provide alternative DRL formulations for the building-energy problem.
A. Deep Q-learning (DQN)
DQN replaces the standard reinforcement-learning value function with a deep Q-network and trains it using stochastic-gradient methods. Experience replay addresses sequential-data instability while the action representation supports building-device scheduling.
- Q-value approximation: DQN approximates the reinforcement-learning value function Qπ(s, a) with a deep Q-network Q(s, a, θ).The network parameters θ consist of the DNN weights and biases, and training uses a mean-squared-error objective in Q-values.
- Learning update: Stochastic gradient descent updates the DQN parameters during learning.The deep Q-network is trained as a variant of the Q-learning algorithm.
- Experience replay: Experience replay randomly samples historical transitions to smooth the training distribution and address sequential-data correlation.The sampled transitions contain states, actions, rewards, and next states.
- Building control: The binary action vector is incorporated into the Q-learning target so flexible building-device loads can be controlled.The approach specifically treats device actions as part of the maximization over next-state Q-values.
B. Deep Policy Gradient (DPG)
DPG directly models action probabilities with a parameterized policy and optimizes expected reward through policy gradients. Its sampling formulation collects trajectories and uses their rewards to estimate parameter updates.
- Policy representation: DPG estimates the probability p(a|s_t, θ) of taking an action in a given state instead of estimating Q-values for every action.This output representation allows multiple actions to be sampled and executed simultaneously.
- Optimization objective: The policy-gradient objective is to maximize total expected reward under a parameterized policy.The formulation applies to the building optimization problems defined in the paper.
- DPG model: In DPG, the DNN acts as a probability density function over game states and yields the corresponding optimization problem.The paper denotes this density function as f(x).
- Gradient estimation: The gradient estimator samples x_i from p(x|θ) and uses the score-function term ▽θ log p(x_i|θ).The derivative operator ▽θ denotes the first-order partial derivative with respect to the output data.
- Trajectory learning: Trajectory-level gradients use sampled states, actions, and rewards collected over an entire game.The trajectory is represented as τ = (s_0, a_0, r_0, …, s_T−1, a_T−1, r_T−1).
V. IMPLEMENTATION DETAILS
The implementation compares DQN and DPG using shared network settings, device-specific action representations, and joint rewards for multiple scheduling objectives. Experiments use Pecan Street building data and evaluate optimized daily peaks.
- Network architecture: Both models use similar DNN architectures, with each state encoded as a two-step time window.For peak reduction, the input includes time, base load, PV, AC, EV, and dishwasher variables at t−1 and t.
- Network architecture: DQN uses 8 outputs for combined actions, whereas DPG uses 3 outputs for separate device-action probabilities.The devices are the air conditioner, electric vehicle, and dishwasher.
- Network architecture: DPG scales linearly with the number of flexible devices, while DQN’s output-layer size grows exponentially with that number.This difference follows from combined-action outputs in DQN versus device-specific outputs in DPG.
- Training setup: The experiments train models for 5000 episodes, each containing 20 randomly chosen days, and update weights after every two episodes.The final policy is retained as the learning output.
- Reward design: Joint rewards combine multiple components for flexible consumption, total energy consumption, and total cost objectives.The selected reward components differ between the optimization problems represented by Eq. 7 and Eq. 1.
- Reward design: The reward formulation can generalize to an arbitrary number of tasks, but interval ranges and coefficients depend on the application.The ranges may be enlarged when comfort limits are relaxed.
- Algorithm procedure: The DPG algorithm samples device actions, collects states and rewards, computes discounted rewards, estimates gradients, and updates network parameters at episode end.The implementation resets the environment using a random day when the current day ends.
- Evaluation: Table I compares daily building-level peak values averaged over one year with optimized peaks from DQN and DPG.The comparison uses 15-minute resolution for the building-level evaluation.
VI. RESULTS AND DISCUSSION
The proposed methods are validated on the Pecan Street database for peak reduction and cost minimization across different numbers of buildings.
- The evaluation uses a large real-world Pecan Street database to analyze peak reduction and cost minimization for various numbers of buildings.
1) Buildings pattern:
The study evaluates building-level optimization using year-long, 15-minute-resolution data that capture varied consumption, solar-generation uncertainty, customer behavior, and time-of-use pricing. Results cover peak reduction and cost minimization with DQN and DPG.
- Building data and tariffs: The Pecan Street data contain up to 90 million daily electricity-consumption records used to construct device-specific patterns.Patterns are averaged over 2015 at 15-minute resolution.
- Building data and tariffs: Three building patterns differ in solar-generation uncertainty and occupant behavior, including whether electric vehicles are used frequently.
- Building data and tariffs: The tariff includes time-of-use on-peak, mid-peak, and off-peak rates, with separate winter, weekend, and working-day components.Self-generating customers also receive compensation for solar generation.
- Peak reduction: Peak-reduction results are reported for three buildings over one year using 15-minute-resolution data.
- Cost minimization: Cost-minimization results are summarized in Table II, with differences across buildings linked to their average electrical patterns and conservative-behavior capabilities.
- Cost minimization: DPG obtains the best results for both peak reduction and cost reduction in building BIII.The cost-minimization solution also has a secondary impact on peak reduction.
D. Scalability and learning capabilities of DRL
The study examines scalability across building aggregations and DPG learning behavior. It reports scalable optimization, heterogeneous savings across 48 buildings, convergence after approximately 1000 episodes, and rapid post-learning decisions.
- Scalability: The methods are evaluated with DQN and DPG for aggregations of 10, 20, and 48 buildings, and the approach is reported as scalable for both optimization problems.Peak-reduction and cost-minimization results are presented separately in Tables III and IV.
- Scalability: Building savings vary substantially: DPG can halve yearly cost in some cases but reduces it by only a few percentage points in others.
- Learning capabilities: DPG reward increases rapidly at first and then much more slowly after about 1000 episodes, when average and optimized average peak values converge.
- Scalability: For 48 buildings, DPG compares unoptimized and optimized annualized energy costs for each building.
- Learning capabilities: The long-term reward expectation continues increasing until approximately 2500 episodes.
- Computational time: After learning, DRL can make control decisions in a few milliseconds, whereas PSO reruns costly optimization for every decision.
- Overall findings: Across building and aggregated levels, the paper reports DPG as more suited than DQN for online energy-resource scheduling while both methods minimize energy cost or flatten the net-energy profile.