Source-linked AI summary
Deep Reinforcement Learning for Smart Home Energy Management
Liang Yu, Weiwei Xie, Di Xie, Yulong Zou, Dengyin Zhang, Zhixin Sun, Linghua Zhang, Yue Zhang, Tao Jiang
TL;DR
The paper tackles smart-home energy-cost minimization with HVAC and ESS under comfort requirements, uncertain parameters, and no building thermal model. It reformulates the problem as an MDP and applies DDPG, with real-world-trace simulations reporting 8.10%-15.21% cost savings without sacrificing thermal comfort.
Problem
The paper studies smart-home energy-cost minimization with HVAC, ESS, comfort constraints, uncertain parameters, temporally coupled operations, and no building thermal dynamics model.
Method
The approach reformulates sequential energy management as an MDP and uses a DDPG-based algorithm to jointly schedule ESS and HVAC from the current environment state.
Results
8.10%-15.21% energy-cost savings were achieved without sacrificing thermal comfort compared with two baselines in simulations using real-world traces.
Takeaways & Limitations
The proposed strategy provides an effective and robust model-free approach to smart-home energy management without prior knowledge of uncertain parameters or building thermal dynamics.
Abstract
from arXiv · showhide
In this paper, we investigate an energy cost minimization problem for a smart home in the absence of a building thermal dynamics model with the consideration of a comfortable temperature range. Due to the existence of model uncertainty, parameter uncertainty (e.g., renewable generation output, non-shiftable power demand, outdoor temperature, and electricity price) and temporally-coupled operational constraints, it is very challenging to determine the optimal energy management strategy for scheduling Heating, Ventilation, and Air Conditioning (HVAC) systems and energy storage systems in the smart home. To address the challenge, we first formulate the above problem as a Markov decision process, and then propose an energy management strategy based on Deep Deterministic Policy Gradients (DDPG). It is worth mentioning that the proposed strategy does not require the prior knowledge of uncertain parameters and building thermal dynamics model. Simulation results based on real-world traces demonstrate the effectiveness and robustness of the proposed strategy.
I. INTRODUCTION
The paper addresses smart-home energy-cost minimization with HVAC, storage, renewable energy, and comfort constraints without assuming a building thermal model. It formulates the problem as an MDP and proposes a model-free DDPG strategy, whose simulations report lower cost while preserving comfort.
- Motivation: Smart homes combine internal networks, intelligent controls, and home automation, creating opportunities to reduce energy costs through coordinated energy management.Dynamic electricity prices can guide ESS and thermostatically controllable-load scheduling; HVAC systems consume about 40% of total energy.
- Problem: The optimization problem minimizes smart-home energy cost over a time horizon while maintaining a comfortable indoor temperature range without a building thermal dynamics model.The setting includes renewable energy, ESS, HVAC systems, and non-shiftable loads.
- Problem formulation: The problem is reformulated as an MDP by designing environment states, actions, and rewards for energy exchange, ESS operation, HVAC adjustment, and parameter uncertainty.The formulation includes temporally coupled operational constraints and does not assume known uncertain parameters.
- Method: The proposed DDPG algorithm jointly schedules ESS and HVAC systems using the current environment state, avoiding prior knowledge of uncertain parameters and building thermal dynamics.DDPG is presented as a model-free energy-management approach, contrasting with model-based methods that require thermal-environment information.
- Results: 8.10%-15.21% energy-cost savings were reported without sacrificing thermal comfort compared with two baselines.The simulations used real-world traces, and robustness testing indicated a practical tradeoff between comfort maintenance and energy-cost reduction.
III. SYSTEM MODEL AND PROBLEM FORMULATION
The system model represents a smart home with distributed generation, ESS, HVAC, and multiple load types, while modeling storage constraints and comfort through a temperature range. ESS operation is bounded by energy, rate, and non-simultaneous charging/discharging constraints; HVAC power is continuously adjustable.
- Smart-home components: The smart home includes distributed generators, ESS, non-shiftable and other loads, HVAC, and a home energy management system.Distributed generators may be solar panels or wind generators, while non-shiftable loads must be served completely without delay.
- A. ESS Model: ESS storage dynamics use charging and discharging powers with separate charging and discharging efficiency coefficients.Charging power is nonnegative and discharging power is assigned a different sign, supporting the action design.
- A. ESS Model: ESS stored energy must remain between its maximum capacity and minimum energy level.These bounds prevent charging above Bmax or discharging below Bmin.
- A. ESS Model: ESS charging and discharging powers are limited by maximum charging and discharging rates.The model denotes these limits as cmax and dmax.
- A. ESS Model: The ESS model prohibits simultaneous charging and discharging.This operational restriction is expressed by the product constraint ct · dt = 0.
- B. HVAC Model: Thermal comfort is represented by a comfortable indoor-temperature range rather than a full multidimensional comfort model.The paper notes that comfort depends on air temperature, radiant temperature, humidity, air speed, clothing, and metabolic rate.
- B. HVAC Model: The inverter HVAC system can continuously adjust its input power within its rated-power limit.This continuous adjustment supports HVAC control for maintaining occupant thermal comfort.
C. Power Balancing
The power-balancing model equates supplied and served power while allowing grid purchases or sales, renewable generation, ESS operation, and HVAC demand. Energy cost combines grid transactions with ESS depreciation associated with cycling.
- Power balancing: Smart-home power balance requires aggregated supply to equal served demand at every time slot.The balance includes grid power, renewable output, ESS discharging, non-shiftable demand, HVAC input, and ESS charging.
- Power balancing: The grid-power sign determines whether the smart home purchases energy from or sells energy to the utility grid.Negative gt denotes energy sales, whereas nonnegative gt denotes energy purchases.
- Energy cost: Buying and selling prices determine the grid energy cost at each time slot.A single grid-power variable represents either purchasing or selling behavior.
- ESS depreciation: ESS depreciation cost is included because frequent charging or discharging harms storage-system lifetime.The model uses a depreciation coefficient ψ measured in $/kW.
E. Total Energy Cost Minimization Problem
The paper formulates expected total energy-cost minimization under uncertain system parameters and random control actions. Because thermal dynamics, parameter distributions, and temporally coupled constraints are difficult to model, it uses an approximate MDP and DDPG-based solution.
- Uncertainty: The objective expectation covers randomness in renewable output, non-shiftable demand, outdoor temperature, and buying or selling electricity prices.These are system parameters whose uncertainty enters the cost-minimization problem.
- Uncertainty: The expectation also covers potentially random grid exchange, ESS charging or discharging, and HVAC input-power actions.These controls are selected at each time slot.
- Challenges: Accurate indoor-temperature dynamics are difficult to obtain because they depend on building structure, surroundings, and internal heat gains.The paper identifies these dependencies as a primary challenge in solving the optimization problem.
- Challenges: Unknown joint distributions of random system parameters and temporally coupled ESS and HVAC constraints further complicate optimization.Current actions can affect future decisions, while dynamic programming faces the curse of dimensionality.
- Solution: The paper reformulates the sequential decision problem as an MDP and develops a DDPG-based energy-management algorithm without requiring thermal dynamics or prior random-parameter knowledge.This approach targets the time-coupling property without relying on the unavailable models and distributions.
- MDP formulation: The MDP is an approximation because renewable generation and electricity prices may be non-Markovian in practice.The paper states that reinforcement learning can still solve the corresponding problem empirically, as supported by its simulations.
- MDP formulation: The MDP interaction has an HEMS agent observe state st, take action at, and receive reward Rt+1 after the environment transitions to st+1.The environment includes generators, loads, ESS, HVAC, the utility grid, and indoor or outdoor temperature.
1) Environment State:
The environment state combines renewable generation, demand, storage, temperatures, electricity price, and daily time position. The action controls ESS operation and HVAC input, while rewards penalize energy use, storage depreciation, and temperature deviation.
- The state includes renewable output, non-shiftable demand, ESS energy, outdoor and indoor temperatures, buying price, and the daily time-slot index.
- The MDP action controls ESS charging or discharging and HVAC input power; grid exchange follows from these decisions.
- A single variable f_t represents ESS charging when nonnegative and discharging when nonpositive, simplifying simultaneous charge/discharge control.
- The reward penalizes HVAC energy consumption, ESS depreciation, and indoor-temperature deviation from the comfortable range.
4) Action-Value Function:
The paper uses discounted returns and action-value functions to formulate sequential energy management, then applies DDPG to continuous smart-home states and actions.
- The HEMS agent maximizes expected future return, defined as the sum of discounted rewards from jointly controlling ESS and HVAC systems.
- The optimal action-value function is defined by maximizing the policy-dependent action-value function and satisfies a recursive Bellman optimality equation.
- Unknown state-transition probabilities motivate Q-learning with a neural-network function approximator for continuous system states.
- A. Algorithmic Design: DDPG uses actor and critic networks: the actor maps environment states to actions, while the critic estimates Q(s_t, a).
- A. Algorithmic Design: During deployment, the actor selects ESS and HVAC actions from the current state, executes them, and receives the next state and reward.
- A. Algorithmic Design: Training stores transitions in replay memory, samples mini-batches, and updates critic, actor, and target networks.
B. Algorithmic Computational Complexity
The proposed testing procedure scales linearly with the number of testing slots and is reported to support real-time implementation under practical slot durations.
- B. Algorithmic Computational Complexity: The testing algorithm has computational complexity O(H_test), where H_test is the number of testing time slots.
- B. Algorithmic Computational Complexity: Shorter time slots increase H_test, while slot duration also affects control opportunities and DRL training convergence.
- B. Algorithmic Computational Complexity: The authors report that typical slot durations exceed the algorithm’s per-slot computation time, enabling real-time implementation.
V. PERFORMANCE EVALUATION
The performance evaluation describes simulations assessing convergence, parameter effects, effectiveness, and scalability of the proposed energy management algorithm.
- The evaluation uses a simulation setup, baseline comparisons, and analyses of convergence, β variation, effectiveness, and scalability.
A. Simulation setup
The simulations use real-world traces for key household energy variables and consider residential HVAC cooling, with parameter settings provided separately.
- Simulation data: Real-world traces for solar generation, non-shiftable power demand, outdoor temperature, and electricity price come from the Pecan Street database.The data concern home energy consumption and solar generation in Austin’s Mueller neighborhood.
- HVAC setting: The simulation considers the cooling mode of a residential HVAC system.
- Parameter settings: Table I reports the main parameter settings used in the simulations.
B. Baselines
The evaluation compares the proposed method with three baselines, then examines convergence, β sensitivity, and performance against practical and reference strategies.
- Baselines: Baseline1 uses an ON/OFF HVAC policy without an energy storage system.It turns HVAC on above the maximum temperature and off below the minimum temperature.
- Baselines: Baseline2 applies the paper’s DDPG-based HVAC control without ESS, isolating savings associated with ESS use and DDPG control.
- Baselines: Baseline3 minimizes testing-period cumulative cost with known uncertainty parameters and indoor-temperature dynamics, providing a performance lower bound for reference.The paper states that this optimal solution is not achievable in practice because of parameter and model uncertainties.
- Algorithmic convergence process: The average reward generally increases and becomes more stable during training, despite fluctuations across episodes.The fluctuations reflect a minimum exploration probability of 0.1 and varying system parameters.
- β sensitivity: As β increases, mean total energy cost generally decreases while mean total temperature deviation increases; β = 1 is considered proper below 1°C deviation.The reported means use 95% confidence intervals across 40 runs.
- Algorithmic effectiveness: The proposed algorithm reduces mean total energy cost by 15.21% versus Baseline1 and 8.10% versus Baseline2 while maintaining smaller mean temperature deviation.Baseline3 performs best but requires prior knowledge of uncertain parameters and the thermal dynamics model; the remaining gap motivates future work.
4) Algorithmic robustness:
The proposed DDPG-based algorithm is evaluated under thermal disturbances that the simulation model cannot capture, demonstrating robustness while trading energy cost against temperature violations.
- The simulation model cannot capture practical thermal disturbances from solar irradiance, lighting systems, and computers.
- Under three disturbance magnitudes, the proposed algorithm outperforms Baseline1 and saves total energy cost by up to 10% versus Baseline3.The comparison with Baseline3 involves a small increase in total temperature violation.
- Unlike Baseline3, the proposed algorithm requires no prior knowledge of uncertain parameters or the thermal dynamics model.
- Figure 7 compares three schemes at β = 0.6 using 95% confidence intervals across 40 runs.
- The paper reports simulation results associated with ESS and HVAC systems and separately presents the proposed algorithm’s robustness.