Optimal Demand Response Using Device Based Reinforcement Learning

Zheng Wen; Daniel O'Neill; Hamid Reza Maei

Source-linked AI summary

Optimal Demand Response Using Device Based Reinforcement Learning

Zheng Wen, Daniel O'Neill, Hamid Reza Maei

arXiv:1401.1549v2Jan 8, 2014cs.LGcs.AIeess.SY

TL;DR

Residential and small commercial demand response needs automated EMS decisions, but prior approaches depend on difficult dissatisfaction models and do not scale well. The paper formulates rescheduling as device-based reinforcement learning, approximately decomposes it over device clusters, and demonstrates Q-learning on a representative example. The formulation learns dissatisfaction from user evaluations, supports EMS-initiated and more flexible user jobs, and has linear complexity under its assumptions.

Problem
Residential and small commercial demand response requires a fully automated EMS, while prior formulations depend on difficult, costly, user-specific disutility functions and face exponential device scaling.
Method
The paper formulates EMS rescheduling as a reinforcement-learning problem and approximately decomposes it into device-cluster problems solved through interaction with users and grid signals.
Results
The formulation learns dissatisfaction from completed or canceled-job evaluations, enables EMS-initiated and flexible user jobs, and has computational complexity linear in device clusters; Q-learning is demonstrated on a representative example.
Takeaways & Limitations
Device-based RL provides a scalable EMS formulation that can adapt scheduling decisions without explicitly specifying the consumer’s dissatisfaction function.
Takeaways & Limitations
The paper evaluates only classical Q-learning on a representative synthetic example and leaves its assumptions for validation with real-world data.

Abstract

from arXiv · show

Demand response (DR) for residential and small commercial buildings is estimated to account for as much as 65% of the total energy savings potential of DR, and previous work shows that a fully automated Energy Management System (EMS) is a necessary prerequisite to DR in these areas. In this paper, we propose a novel EMS formulation for DR problems in these sectors. Specifically, we formulate a fully automated EMS's rescheduling problem as a reinforcement learning (RL) problem, and argue that this RL problem can be approximately solved by decomposing it over device clusters. Compared with existing formulations, our new formulation (1) does not require explicitly modeling the user's dissatisfaction on job rescheduling, (2) enables the EMS to self-initiate jobs, (3) allows the user to initiate more flexible requests and (4) has a computational complexity linear in the number of devices. We also demonstrate the simulation results of applying Q-learning, one of the most popular and classical RL algorithms, to a representative example.

1 Introduction

Residential and small commercial building demand response requires a fully automated EMS that can learn user- and grid-dependent scheduling decisions. The paper proposes a device-based RL formulation designed to address dissatisfaction modeling, job initiation, request flexibility, and scalability.

65% of demand response’s total energy savings potential is estimated to come from residential and small commercial buildings.DR can shift consumption away from peak periods, improving efficiency and reducing costs, emissions, and outage risk.
A fully automated EMS is necessary because users repeatedly weigh current energy prices against delaying device use under uncertain future prices.The EMS must learn decisions consistent with each user’s cost-delay trade-offs rather than rely on continual manual intervention.
Existing RL-based EMS formulations require difficult, costly, and residence-specific disutility functions to represent dissatisfaction with rescheduling.The paper identifies these functions as idiosyncratic and insufficiently specified in prior work.
The proposed formulation learns dissatisfaction from evaluations of completed or canceled jobs instead of requiring a pre-specified disutility function.This removes the impractical assumption that delay dissatisfaction can be captured by a known function.
The EMS can self-initiate jobs and support flexible user requests with distinct target times, priorities, and cancellation options.EMS-initiated jobs use probing and feedback to anticipate future energy usage.
Under reasonable assumptions, computational complexity grows linearly with the number of device clusters, enabling RL methods for many devices.The formulation approximately decomposes the problem over device clusters, and the paper also proposes performance metrics relative to current and prescient-optimal behavior.

2 Description of fully-automated EMS

A fully automated EMS observes grid signals, handles user and EMS-initiated jobs, and schedules device operation while learning decisions consistent with user cost-delay preferences. The formulation assumes exogenous energy prices and supports flexible, prioritized requests and cancellations.

EMS operation: A fully automated EMS observes grid signals, receives user requests and evaluations, and schedules jobs across managed devices.Grid signals are delivered through a communication network; the EMS treats energy prices as exogenous because an individual residential or small-commercial consumer has negligible market influence.
User interaction: User-initiated jobs let consumers request device operation, cancel uncompleted requests, and later evaluate completed or canceled requests.Each request identifies a device, request time, target time, and priority.
EMS operation: The EMS can self-initiate jobs for idle devices without an explicit user request, while users evaluate some resulting jobs.An example is pre-cooling a commercial building before tenant arrival to capture lower energy costs or mask cooling latency.
Model assumptions: The model uses discrete time and assumes each device job completes in one time step and consumes constant energy Cn, though operating periods can be generalized.There are N smart devices, and Cn depends only on device type.
User interaction: Target time may differ from request time, but must satisfy τr ≤ τg ≤ τr + Wn under the paper’s timing assumption.The target time represents when the user prefers the job completed, while priority indicates the strength of that preference.

3 Device Based Reinforcement Learning

The paper formulates automated demand-response scheduling as device-based reinforcement learning and approximately decomposes it across clusters under additive dissatisfaction and weakly dependent requests. This yields a near-optimal solution with complexity linear in the number of device clusters while allowing learning from unknown user and grid models.

3.1 Dissatisfaction Function and Cost Function: The paper defines dissatisfaction over rescheduling histories and assumes it is approximately additive across devices to avoid an intractable global formulation.Assumption 1 represents total dissatisfaction as device-level terms based on each device’s history; the motivation is that rescheduling preferences are weak relative to other life preferences.
3.1 Dissatisfaction Function and Cost Function: The instantaneous cost combines electricity expenditure with user dissatisfaction, weighted by γ, and the EMS minimizes expected infinite-horizon discounted cost with discount factor α.Electricity cost at time t is Pt multiplied by the energy consumed by devices operating at t.
3.2 Decomposition of the Problem: Under exogenous Markov grid signals and weak conditional dependence of requests across device clusters, the optimal scheduling problem approximately decomposes over device clusters.Each cluster problem uses device dissatisfaction and energy consumption, grid-signal processes, and a statistical model of requests and cancellations.
3.2 Decomposition of the Problem: A near-optimal solution comes from solving the decomposed device-based MDPs separately, with computational complexity linear in the number of device clusters.The device-based MDP is obtained by properly specifying statistical models for user requests and cancellations.
3.3 Reinforcement Learning Formulation: Because dissatisfaction and statistical models are initially unknown in practice, the EMS learns scheduling decisions through reinforcement learning while interacting with users and grid signals.The EMS is the RL agent, and user evaluations can support learning the dissatisfaction function.
3.3 Reinforcement Learning Formulation: The paper permits any RL algorithm for the device-based problem and uses a no-rescheduling policy as the baseline for evaluating potential cost reduction.Under this baseline, the EMS never self-initiates a job, according to the supplied passage.

4 A Simplified MDP Model

The paper develops a simplified device-based MDP for EMS scheduling, using assumptions that enable approximate decomposition over device clusters. It defines episode dynamics, learns dissatisfaction from user evaluations, and evaluates RL policies through relative improvement and demand-response potential.

Model scope: The device-based RL problem approximately decomposes over device clusters under the section’s simplifying assumptions.The assumptions simplify exposition but are described as nonessential when the preceding conditions for cluster decomposition hold.
Model assumptions: The simplified model assumes one device per cluster, an exogenous finite-state ergodic price Markov chain, and price-independent request and cancellation processes.The model also assumes time-invariant statistical models for energy prices and user behavior.
Dissatisfaction model: User dissatisfaction is learned from evaluations of completed or canceled jobs rather than requiring a pre-specified dissatisfaction function.Assumption 4 equates dissatisfaction with the user’s evaluation, while the paper notes that behavioral noise could be added in practice.
Episode dynamics: Episodes terminate when a job completes or an unsatisfied request is canceled, after which the device state regenerates from a fixed distribution and a new episode begins.The user-EMS interaction is organized over time periods, while episodes may span multiple time steps.
Performance metric: Relative improvement measures normalized expected user cost reduction, while relative demand-response potential upper-bounds it; both demand-response potential measures are non-increasing in γ.The baseline value does not depend on γ, and for sufficiently large γ the baseline becomes optimal, causing DRP to approach zero.
State transitions: The device portion of the MDP state is [s_t, g_t]^T, with transitions determined by on/off actions and regeneration after episode termination.The transition structure includes multiple possible request types and can be extended to support stacked uncompleted requests by representing target times as a vector.

5 Experiment Results

The paper evaluates Q-learning on a representative device-based EMS example, using relative DR potential and relative improvement to assess performance across the trade-off parameter γ.

5.1 The Q-Learning Algorithm: Q-learning is implemented as an off-policy, online, incremental algorithm that can learn from real-time data while following an exploratory policy.The behavioral policy combines exploration with action selection based on current Q-function estimates, using Boltzmann exploration in this implementation.
5.1 The Q-Learning Algorithm: The algorithm updates action values from observed state, action, instantaneous-cost, and next-state data using temporal-difference learning.The Q-function estimate is updated for the experienced state-action pair, with β_t denoting the step-size.
5.2 Experiment Setup: The representative example uses price states {10, 12, 15, 20}, high and normal request priorities, W = 4, Ŵ = 5, α = 0.9995, C = 1, and |S| = 96.The transition model includes request arrival and cancellation probabilities that increase with elapsed time, while the ten request types are assumed equally likely.
5.2 Experiment Setup: RDRP is non-increasing in γ, equals 1 when γ = 0, and approaches 0 as γ approaches infinity.These properties follow for the example because it satisfies the conditions of Theorem 1.
5.3 Performance: The simulations run Q-learning for 4,000 episodes and repeat each trade-off setting 200 times to estimate expected performance relative to the baseline.RI is computed using the analytically derived baseline value and the sample mean of the learned policy’s value; for γ ≥ 4.4, RI is negative.
5.3 Performance: RI decreases with γ, indicating that Q-learning achieves smaller improvement over the no-rescheduling baseline as user preference for avoiding rescheduling increases.The conclusion applies to this representative example with the other parameters fixed.

6 Conclusion and Future Work

The paper concludes that its device-based RL formulation supports scalable, flexible, and automated demand response, while identifying validation on real-world data and broader algorithmic evaluation as future work.

6 Conclusion and Future Work: The proposed device-based RL formulation removes the need for a pre-specified dissatisfaction function and has computational complexity linear in the number of device clusters.It also enables EMS self-initiation of jobs and allows users to initiate more flexible requests.
6 Conclusion and Future Work: Simulation results suggest that Q-learning outperforms the no-rescheduling baseline for a broad range of trade-off parameter γ.This result is reported for the paper’s representative example.
6 Conclusion and Future Work: The paper’s assumptions have not yet been tested with real-world data and statistical methods.The authors identify this validation as possible future work.
6 Conclusion and Future Work: The evaluation applies only Q-learning to a representative synthetic example, with state-of-the-art algorithms and additional synthetic or real-world examples left for future work.This limits the demonstrated evaluation scope.

Appendix A Bellman Equation

The appendix derives the Bellman equation for the device-based MDP and explains how state transitions and actions enter the optimal Q-function formulation.

Appendix A Bellman Equation: When the device-MDP transition model and dissatisfaction function are known, dynamic programming can solve the device-based MDP through its Bellman equation.The Bellman equation provides the basis for computing the optimal Q-function.
Appendix A Bellman Equation: The state is xt = [Pt, st, gt]T and the action space is A = {off, on}.Pt denotes the price component, while st and gt belong to the device portion of the state.
Appendix A Bellman Equation: For a device without a current request, the Bellman formulation evaluates transitions over the next price, elapsed-time, and request-priority states.The next price follows the price Markov chain, while the device portion follows the regeneration distribution π0.
Appendix A Bellman Equation: The request-arrival probability depends on the current elapsed time, with the elapsed-time state increasing until it reaches Ŵ.The transition description identifies the price transition as Markovian and defines the elapsed-time update explicitly.
Appendix A Bellman Equation: For a device that already has a request, the Bellman formulation similarly takes expectations over the next price and regenerated device state after the relevant action.The next device portion is drawn from π0 when the device regenerates its state.
Appendix A Bellman Equation: The optimal Q-function is the unique solution of the Bellman equation, and value iteration or policy iteration can compute it.These dynamic-programming methods are tractable when the device-based state and action spaces are usually small.

Appendix B Proof for Theorem 1

The proof establishes how the demand-response potential changes with the dissatisfaction tradeoff parameter γ: it is non-increasing and becomes zero above a finite threshold.

The policy value decomposes into discounted electricity cost Aµ plus γ times discounted rescheduling dissatisfaction Bµ.
The baseline policy has zero rescheduling dissatisfaction because it never initiates jobs and user jobs complete or are canceled before their target times.
A positive gap ΔB separates the baseline from the second-best deterministic policy with nonzero dissatisfaction.
For sufficiently large γ, the dissatisfaction penalty outweighs any possible electricity-cost reduction, making the baseline policy optimal.The proof bounds the maximum cost reduction using the energy-price range and discount factor, then compares it with ΔB.
DRP is non-increasing in γ, and DRP = 0 for every γ > γ*, implying DRP → 0 as γ → ∞.
When γ = 0, the optimal value is zero under a policy that neither initiates jobs nor responds to user requests, so the relative DR potential reaches one.