Source-linked AI summary
Reinforcement Learning Applied to an Electric Water Heater: From Theory to Practice
Frederik Ruelens, Bert Claessens, Salman Quaiyum, Bart De Schutter, Robert Babuska, Ronnie Belmans
TL;DR
Electric water heaters offer energy storage for demand response, but learning from high-dimensional sensor states can require more observations. The paper combines an auto-encoder representation with fitted Q-iteration; in a 40-day lab experiment, it reduced energy cost by 15% compared with a thermostat controller.
Problem
Learning from high-dimensional sensor states requires more observations to estimate the Q-function and cover the state-action space.
Source: §IV · p. 3Method
The paper uses an auto-encoder to create a compact state representation and applies fitted Q-iteration to learn a water-heater control policy.
Source: §I · p. 2, §IV · p. 4Results
15% lower energy cost than the default thermostat controller after 40 days of lab experimentation.
Source: B. Evaluation · p. 7Takeaways & Limitations
Compact features can improve control-policy quality with relatively few observations, while the full state vector is recommended when only 1–10 temperature sensors are available.
Source: §VIII · p. 7
Abstract
from arXiv · showhide
Electric water heaters have the ability to store energy in their water buffer without impacting the comfort of the end user. This feature makes them a prime candidate for residential demand response. However, the stochastic and nonlinear dynamics of electric water heaters, makes it challenging to harness their flexibility. Driven by this challenge, this paper formulates the underlying sequential decision-making problem as a Markov decision process and uses techniques from reinforcement learning. Specifically, we apply an auto-encoder network to find a compact feature representation of the sensor measurements, which helps to mitigate the curse of dimensionality. A wellknown batch reinforcement learning technique, fitted Q-iteration, is used to find a control policy, given this feature representation. In a simulation-based experiment using an electric water heater with 50 temperature sensors, the proposed method was able to achieve good policies much faster than when using the full state information. In a lab experiment, we apply fitted Q-iteration to an electric water heater with eight temperature sensors. Further reducing the state vector did not improve the results of fitted Q-iteration. The results of the lab experiment, spanning 40 days, indicate that compared to a thermostat controller, the presented approach was able to reduce the total cost of energy consumption of the electric water heater by 15%.
I. INTRODUCTION
Water heaters can shift energy use through stored hot water, making them candidates for residential demand response. The paper motivates reinforcement learning and compact state representations as ways to learn control policies when system knowledge is limited.
I. INTRODUCTION
- Thermal storage lets electric water heaters adapt consumption without reducing user comfort, making them candidates for residential demand response.A household using 100 liters of hot water daily may consume about 2 MWh per year.Source: §I · p. 1
- Model predictive control typically requires system identification, state estimation, and forecasts, while model errors can affect controller stability.The paper presents these implementation demands as costs to balance against possible financial gains.Source: §I · p. 1
- Reinforcement learning can learn policies through interaction without system modeling, useful when expert knowledge or forecasts are unavailable or too costly.The cited discussion frames the MPC–RL trade-off around the quality of system-dynamics knowledge available for system identification.Source: §I · p. 1
- The paper combines an auto-encoder state representation with fitted Q-iteration and evaluates state representations and batch sizes in simulation before applying an RL agent in the lab.The agent seeks to minimize the water heater’s energy-consumption cost.Source: §I · pp. 1–2
II. REINFORCEMENT LEARNING
- Batch RL reuses stored interactions, and fitted Q-iteration estimates a Q-function from a fixed batch of past interactions.The overview contrasts this data reuse with standard Q-learning, which discards each observation after updating its Q-function.Source: §II · p. 2
III. PROBLEM FORMULATION
The paper formulates electric-water-heater control as a finite-horizon Markov decision process and defines the objective through expected return and its associated Q-functions.
III. PROBLEM FORMULATION
- The control objective is to minimize an electric water heater’s energy cost given a daily price profile, despite unknown system dynamics, hot-water demand, and backup-controller settings.The backup controller can override agent actions when safety or comfort constraints are violated.Source: §III · p. 2
A. Markov decision process framework
- The agent’s sequential decision problem is represented as a finite-horizon Markov decision process with state and action spaces, stochastic transitions, and a cost function.At each of T steps, the state evolves according to the transition function.Source: A. Markov decision process framework · p. 2
- Each transition includes a random disturbance drawn conditionally on the current state, a control action, and a resulting state.The transition is associated with a cost.Source: A. Markov decision process framework · p. 2
- The agent seeks a policy mapping states to actions that minimizes expected return over the finite horizon.The return is evaluated from an initial state while following the policy.Source: A. Markov decision process framework · p. 3
- A Q-function characterizes cumulative return from a state and action, followed by policy h thereafter.The optimal Q-function is the best Q-function attainable by any policy.Source: A. Markov decision process framework · p. 3
- The optimal policy is calculated from the optimal Q-function, which satisfies the Bellman optimality equation.The paper follows the notation introduced for this problem formulation in [28].Source: A. Markov decision process framework · p. 3
B. Observable state vector
The observable state combines time information with locally measured temperatures along the water-buffer tank.
B. Observable state vector
- The state includes day of the week and quarter of the day to represent timing relevant to system dynamics and hot-water demand patterns.The paper describes demand as having diurnal and weekly patterns.Source: B. Observable state vector · p. 3
- The physical component contains temperature measurements from sensors connected along the storage tank’s hull.Each sensor measurement is part of the observable state at time step k.Source: B. Observable state vector · p. 3
C. Control action
The agent issues binary heating commands, while a backup mechanism can enforce user comfort and safety constraints.
C. Control action
- The agent’s binary action switches the heating element off or on, but the backup mechanism may override it to enforce comfort and safety constraints.The logic of that backup controller is assumed unknown to the learning agent.Source: C. Control action · p. 3
- The learning agent can observe the physical action enforced by the backup controller, which is needed to calculate cost.The physical action may differ from the agent’s requested action.Source: C. Control action · p. 3
- The backup controller determines the physical action from the state and user-defined safety and comfort settings.Its control logic is based on state-of-charge bounds and can select heater power or zero power.Source: C. Control action · p. 3
D. Cost function
The section links time-varying electricity prices to the cost signal used in learning and describes compact sensor-state features for fitted Q-iteration. It also notes an assumption that may exclude useful low-variance input components.
D. Cost function
- At each optimization period, the agent receives a price vector for the next T steps and a cost at each time step.Source: D. Cost function · p. 3
- λ_k denotes the electricity price at step k, while Δt is the length of one control period.Source: D. Cost function · p. 3
IV. BATCH OF FOUR-TUPLES
- Using all sensor measurements creates a high-dimensional state that requires more observations to estimate Q and cover the state-action space.Source: §IV · p. 3
- An auto-encoder reduces sensory-state dimensionality by encoding measurements through a narrower middle layer.Source: §IV · p. 4
- The encoder maps observed state x_l to feature vector z_l, and conjugate-gradient descent trains its weights.Source: §IV · p. 4
- The feature-learning setup assumes all input data are equally important, risking omission of useful low-variance components.Source: §IV · p. 4
V. FITTED Q-ITERATION
This section describes the learning algorithm and exploration strategy based on the batch of feature vectors introduced earlier.
V. FITTED Q-ITERATION
- The section presents a learning algorithm and exploration strategy based on the batch of feature vectors R.Source: §V · p. 4
A. Fitted Q-iteration
Fitted Q-iteration builds a regression dataset from state-action pairs and iteratively updates Q-value targets. It reuses past experiences while recalculating costs using the next day's price vector.
A. Fitted Q-iteration
- Fitted Q-iteration builds a training set from state-action pairs in R, with targets based on the previous iteration's Q-function approximation.Source: A. Fitted Q-iteration · p. 4
- The first iteration approximates expected cost; later iterations update Q-values from the previous Q-function until T iterations incorporate future costs.Source: A. Fitted Q-iteration · p. 4
- Costs are recalculated using the price vector supplied at the start of each day, allowing past experiences to be reused for the next day's policy.Source: A. Fitted Q-iteration · p. 4
- Extremely randomized trees estimate the Q-function, although other regression algorithms, including neural networks, can also be used.Source: A. Fitted Q-iteration · p. 4
B. Boltzmann exploration
Boltzmann exploration selects actions according to a probability involving the Q-function and current feature vector. Lowering its temperature shifts selection toward higher-valued actions.
B. Boltzmann exploration
- The agent selects actions using a Boltzmann probability based on the Q-function and current feature vector.Source: B. Boltzmann exploration · p. 4
- As temperature τ_d approaches zero, exploration decreases and the policy becomes greedier; starting with high temperature makes exploration initially random.Source: B. Boltzmann exploration · p. 4
- The Q-function is scaled to [0, 100], and τ_1 starts at 100 for equal action probabilities; lowering τ_d increases selection probability for higher-valued actions.Source: B. Boltzmann exploration · p. 5
VI. SIMULATION-BASED RESULTS
The simulations tested how state representation affects fitted Q-iteration and compared the resulting policy with a thermostat. Low-dimensional features helped with limited batches, while the mature agent reduced energy costs under both price profiles.
VI. SIMULATION-BASED RESULTS
- The simulation used a 50-layer tank model, a 2.36 kW heater, a 200-liter buffer, and mean daily hot-water use of 120 liters.Experiments used Belgian day-ahead and balancing-market prices.Source: §VI · p. 5
B. Step 2: evaluation
- 24% for day-head prices and 34% for imbalance prices: fitted Q-iteration reduced total energy cost versus the default strategy.The comparison uses the day-ahead and imbalance price profiles, respectively.Source: Figure 4 · p. 5, B. Step 2: evaluation · p. 6
A. Step 1: feature selection
- Feature performance depended on batch size: AE 3 cost less than AE 15 after 10 days, but AE 15 cost less after 75 days.AE 1 produced a relatively bad policy regardless of batch size.Source: A. Step 1: feature selection · p. 5
- With a limited batch, low-dimensional features outperformed the full state of 50 temperature measurements.The full-state results improved as batch size increased; selecting an appropriate representation during learning was left to future work.Source: A. Step 1: feature selection · p. 5
B. Step 2: evaluation
- The evaluation added each day's observations to the batch, retrained the auto-encoder daily, and used 10 days of Boltzmann exploration.The thermostat heated from below its minimum state-of-charge threshold until reaching 100% and did not respond to prices.Source: B. Step 2: evaluation · p. 6
- The mature agent shifted consumption toward low-price periods in the day-ahead and imbalance-price profiles.The simulation figures show temperature profiles and power consumption alongside the relevant price profiles.Source: B. Step 2: evaluation · p. 6, Figure 5 · p. 6
VII. LAB RESULTS
The lab experiment applied fitted Q-iteration to a real electric water heater and compared it with a thermostat. Over 40 days, the agent reduced energy cost, while the best lab performance used all eight temperature measurements.
A. Lab setup
- The heater was equipped with eight temperature sensors and a controllable power relay; a controllable outlet valve simulated household hot-water demand.The simulated demand averaged 120 liters per day.Source: A. Lab setup · p. 6
- The agent assumed a deterministic price profile for the following day and used 10 days of Boltzmann exploration.Both day-ahead and imbalance prices were used for comparison with the simulations.Source: A. Lab setup · p. 6
B. Evaluation
- Among the tested feature vectors, the best lab performance used all eight temperature measurements.Thus, further reducing the state vector did not improve the lab results.Source: B. Evaluation · p. 6
- With identical price profiles and tap-water demands, the agent reduced cost by consuming during low-price moments.Fig. 7 shows sensor measurements and power alongside imbalance prices.Source: B. Evaluation · p. 6, Figure 7 · p. 7
- 15%: over 40 days, fitted Q-iteration reduced energy cost versus the default thermostat; excluding the first 10 exploration days, the reduction was 28%.Fig. 9 reports cumulative and daily energy costs.Source: B. Evaluation · p. 7
VIII. CONCLUSIONS AND FUTURE WORK
Fitted Q-iteration reduced lab energy costs relative to a thermostat, while simulation results showed faster convergence with compact features. The conclusions make the preferred state representation dependent on the amount of data and sensor availability, and propose selecting representations during learning as future work.
VIII. CONCLUSIONS AND FUTURE WORK
- In simulation, compact features reached good policies much faster than full-state learning, with energy-cost reductions of 24% using day-ahead prices and 34% using imbalance prices versus a default thermostat.The experiment used an electric water heater with 50 temperature sensors.Source: §VIII · p. 7
- With relatively few observations (25 days), compact features can improve policy quality; as observations increase, higher-dimensional representations are advisable, and 1–10 sensors favor the full state vector.The lab results also report that reducing the state vector did not improve fitted Q-iteration.Source: §VIII · p. 7
- 15% lower total energy cost than the thermostat within 40 days in the lab experiment.Fig. 9 compares cumulative and daily energy costs for the learning agent and default thermostat.Source: Figure 9 · p. 7, §VIII · p. 7
- In the real-world experiment, fitted Q-iteration obtained good control policies within 20 days of operation.Source: §VIII · p. 7
- Future work proposes choosing a state representation during learning by comparing experts that pair learning algorithms with different feature representations.The proposed selection would use a performance metric to identify the highest-performing expert.Source: §VIII · p. 7