PaperlayerView exact revision on arXiv

Source-linked AI summary

Experimental analysis of data-driven control for a building heating system

Giuseppe Tommaso Costanzo, Sandro Iacovella, Frederik Ruelens, T. Leurs, Bert Claessens

arXiv:1507.03638v2cs.AI

TL;DR

Building climate control offers demand-response flexibility, but MPC faces the challenge of accurate thermal and actuation models. This paper applies model-assisted batch reinforcement learning with virtual data and policy shaping to dynamic pricing. Simulation reaches within 90% of a mathematical optimum in about 20 days, while the living-lab assessment reports policy generalization across outside-temperature forecasts.

  • Problem

    MPC for building heating requires accurate thermal-dynamics and actuation models, motivating practical data-driven approaches for climate-control flexibility and demand response.

    Source: §1 · p. 1

  • Method

    The approach uses Fitted Q-Iteration with model-derived virtual data and domain-knowledge policy shaping for HVAC under dynamic pricing.

    Source: §2.2 · p. 2

  • Results

    Within approximately 20 days, simulation performance reached within 90% of a mathematical optimum; after approximately 16 days, living-lab policies generalized across outside-temperature forecasts.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The living-lab assessment qualitatively confirmed that learned policies could be used across different outside-temperature forecasts.

    Source: §6 · p. 9

    Testing other demand-response objectives and gray-box models for adding virtual tuples remains future work.

    Source: §6 · p. 9

Abstract

from arXiv · show

Driven by the opportunity to harvest the flexibility related to building climate control for demand response applications, this work presents a data-driven control approach building upon recent advancements in reinforcement learning. More specifically, model assisted batch reinforcement learning is applied to the setting of building climate control subjected to a dynamic pricing. The underlying sequential decision making problem is cast on a markov decision problem, after which the control algorithm is detailed. In this work, fitted Q-iteration is used to construct a policy from a batch of experimental tuples. In those regions of the state space where the experimental sample density is low, virtual support samples are added using an artificial neural network. Finally, the resulting policy is shaped using domain knowledge. The control approach has been evaluated quantitatively using a simulation and qualitatively in a living lab. From the quantitative analysis it has been found that the control approach converges in approximately 20 days to obtain a control policy with a performance within 90% of the mathematical optimum. The experimental analysis confirms that within 10 to 20 days sensible policies are obtained that can be used for different outside temperature regimes.

1. Introduction

Building climate control offers thermal-storage flexibility for demand response, but MPC depends on accurate thermal and actuation models. This paper positions offline BRL with domain knowledge as a data-driven route for dynamic-pricing operation.

1. Introduction

  • MPC can exploit building thermal inertia or direct heat storage for demand response, but accurate thermal-dynamics and actuation models are its major implementation challenge.The flexibility can support services such as peak shaving, valley filling, ancillary services, or energy arbitrage.Source: §1 · p. 1
  • Completely data-driven approaches are considered practical but may sacrifice performance; they can pair data-driven models with optimization or learn policies directly from interactions.The paper contrasts online RL policy updates with BRL policies calculated offline from historical data.Source: §1 · pp. 1–2
  • The paper applies offline BRL with prior knowledge to building climate control for demand response applications.BRL calculates policies offline using a batch of historical data, unlike online RL.Source: §1 · p. 2
  • Fitted Q-Iteration learns a policy from batch data, enhanced by model-derived virtual data; this approach is called MABRL.Source: §1 · p. 2

2. Related Work

Related work contrasts model-based predictive control with reinforcement-learning approaches, then frames this work's combination of FQI, virtual trajectories, and policy shaping for dynamic pricing.

2.1. Model Predictive Control

  • MPC optimizes control using building-dynamics, comfort, and exogenous-input models; implementations often identify LTI models or use gray-box models such as thermal circuits.Source: §2.1 · p. 2

2.2. Reinforcement Learning

  • RL learns policies through environment interactions and can mitigate model bias; prior BRL/FQI studies report 20–40 days to convergence at increased computational complexity.Source: §2.2 · p. 2
  • This work applies FQI, virtual trajectories, and policy shaping to HVAC under dynamic pricing, with quantitative and qualitative assessment in simulation and an experiment.Source: §2.2 · p. 2

3. Problem Formulation

The section formulates building climate control as a cost-minimizing Markov decision problem, then specifies the state information, control actions, backup controller, and dynamic-pricing reward.

3. Problem Formulation

  • The MDP defines state transitions under actions and random disturbances, with the objective of finding a policy that minimizes the T-stage energy-price cost.An optimal policy satisfies the Bellman optimality equation; the paper uses MABRL, related to value iteration.Source: §3 · pp. 2–3

3.1. State description

  • The state space is divided into time-dependent, controllable physical, and uncontrollable exogenous information.The state description follows the approach presented by Ruelens et al. [25].Source: §3.1 · p. 3

3.1.1. Timing

  • Time is represented by the quarter hour within a day to identify behavioral daily patterns; adding day of week is outside this study's scope.The paper notes that a day-of-week extension could be added at little extra cost.Source: §3.1.1 · p. 3

3.1.2. Physical representation

  • The controllable physical state is indoor air temperature, represented between lower and upper bounds set by the end consumer.Source: §3.1.2 · p. 3

3.1.3. Exogenous Information

  • Outside temperature and solar radiance form the exogenous state; they affect indoor temperature but are invariant to control actions, and forecasts are assumed available for policy construction.Source: §3.1.3 · p. 3

3.1.4. Control action

  • The HVAC action is binary ON/OFF, and the previous action is included in the state to help avoid too-frequent switching.The state contains only part of the system's actual state; adding previous state tuples can increase its dimension.Source: §3.1.4 · p. 3

3.2. Backup controller and physical realisation

  • A backup controller filters the policy's requested action by mapping it to the physical action.Its function is specified using system-specific information.Source: §3.2 · p. 3

3.3. Reward model

  • For dynamic pricing, the reward responds to an external price vector; the equation uses average air-conditioner power over the interval ∆t.Source: §3.3 · p. 3

4. Model-Assisted Batch Reinforcement Learning

The controller uses fitted Q-iteration to approximate a state-action value function from experimental and virtual tuples, then shapes the resulting policy with domain knowledge. An artificial neural network supplies virtual support samples, while policy dispatch balances exploration and exploitation.

4. Model-Assisted Batch Reinforcement Learning

  • Fitted Q-iteration approximates the state-action value function from a batch of experimental and virtual tuples, and the resulting values determine control actions.The experimental and virtual tuples form F_L; the policy action is obtained from the resulting approximate Q-function.Source: §4 · p. 4

4.1. The support model: Artificial Neural Network

  • The support model uses an ensemble of 40 single-output ELMs to predict internal-temperature change, trading regression performance for fast training that ensembling partially mitigates.The experimental trials use regularization C = 100.Source: §4.1 · pp. 4–5

4.2. Fitted Q-Iteration

  • FQI replaces observed external temperature and solar radiance with forecasts, biasing the approximate Q-function toward those forecasts.It generates at most n virtual tuples from random state-action pairs, accepting a sample only when its nearest experimental tuple lies outside radius r; a computational budget H limits the procedure.Source: §4.2 · p. 5
  • Extremely randomized trees estimate the Q-function in FQI, although other regressors such as neural networks or support vector machines are possible.The radius for virtual-sample acceptance could instead use local inter-tuple distances, which the authors leave for future work.Source: §4.2 · p. 5

4.3. Policy Shaping

  • Policy shaping fits a weighted combination of triangular membership functions to approximate the original policy and uses expert knowledge to enforce monotonicity.The membership-function centers lie on an equidistant grid across each state-space dimension.Source: §4.3 · p. 5

4.4. Policy dispatch

  • Policy dispatch selects actions through exploration or exploits the learned policy, with ε reduced by half every four days during performance assessment.The initial exploration probability is ε0 = 0.4.Source: §4.4 · pp. 5–6

5. Performance assessment

The assessment combines simulated controller benchmarking with living-lab policy analyses, examining forecast integration, policy shaping, virtual tuples, and operation under different outside-temperature regimes.

5.1. Policy representation

  • Policy maps show quarter-hour time and indoor temperature mapped to ON/OFF actions, allowing offline inspection of learned behavior and feature effects.The maps are conditioned on forecasted outside temperature and solar radiation.Source: §5.1 · p. 6

5.2. Simulated environment

  • Approximately 20 days of learning produced near-optimal policies whose performance relative to the mathematical optimum then stabilized.Adding virtual tuples made only a limited contribution, with a slight economic advantage for MABRL over BRL.Source: §5.2 · p. 6
  • The simulated benchmark compares BRL, MABRL, optimal MPC, and default hysteresis control using cumulative electricity cost and daily economic performance.The model-based simulation was used because living-lab external conditions could not be reproduced exactly from day to day.Source: Figure 2 · p. 6, §5.2 · p. 6

5.3. Experimental environment

  • The living-lab setup collected room and outdoor temperatures, roof solar radiation, and air-conditioner power every five minutes; forecasts and policy updates arrived three times daily.Forecasts covered eight hours at 15-minute granularity, and the policy was dispatched every five minutes.Source: §5.3 · p. 6, Figure 3 · p. 9

5.3.1. Ability to integrate forecasts

  • With more varied observed temperatures, policies became more dependent on the outside-temperature forecast; colder forecasts yielded less low-price responsiveness than warmer ones.At colder outside temperatures, stored room heat is lost faster, limiting the ability to avoid high energy prices.Source: §5.3.1 · pp. 6–7, Figure 4 · p. 7

5.3.2. Policy shaping

  • Policy shaping filled gaps in the original policies, which were attributed to nonuniform batch sample density and the extra-trees regressor’s randomness.The shaping used triangular membership functions and enforced a strictly decreasing policy as indoor temperature increased, based on physical understanding of heat loss.Source: §5.3.2 · p. 7, Figure 5 · p. 8

5.3.3. E

  • With two days of experimental data, the model-assisted policy had 94% overlap with the 20-day regime policy, versus 90% for the early policy.The model-assisted policy used two days of virtual tuples from a support model trained on the two-day experimental batch.Source: §5.3.3 · pp. 7–8, Figure 6 · p. 8

5.3.4. Power profiles

  • After 12 and 16 days, the controller typically switched on at the start of low-price periods and largely avoided later high prices in two outside-temperature regimes.The reported power-profile analysis was based on the HVAC power, energy price, internal temperature, and control policy traces.Source: §5.3.4 · pp. 8–9

6. Conclusions

Model-assisted batch reinforcement learning reached performance within 90% of a mathematical optimum in simulation and produced policies for different outside-temperature forecasts in the living lab. Policy shaping enforced a decreasing relationship between indoor temperature and policy, while virtual tuples offered only limited performance gains.

6. Conclusions

  • Within approximately 20 days, the controller reached performance within 90% of a mathematical optimum in simulation.The living-lab analysis also found that after approximately 16 days, policies generalized across different outside-temperature forecasts.Source: §6 · p. 9
  • Policy shaping enforced policies to be strictly decreasing as indoor temperature increased.Source: §6 · p. 9
  • Adding virtual tuples could improve policy quality when experimental batches were small, but increased performance only to a limited extent.Source: §6 · p. 9
  • Future work will evaluate the algorithm for other demand-response objectives and use other models, including gray-box models, to add virtual tuples.Source: §6 · p. 9

Appendix A. ETP model

The simulated experiments use an Equivalent Thermal Parameter model fitted to living-lab experimental data. Its heat-flow description accounts for exchanges involving indoor air, the building envelope, solar gains, and air-conditioner gains.

Appendix A. ETP model

  • The simulated-experiment data batch comes from an Equivalent Thermal Parameter model fitted to living-lab experimental data.Source: Appendix A · p. 9
  • The model describes heat flow into indoor air using indoor–envelope exchange, solar gains, and air-conditioner gains, and heat flow to the envelope using indoor-air exchange and solar gains.Source: Appendix A · p. 9
  • Figure A.9 sketches the equivalent thermal parameter model.Source: Figure A.9 · p. 9

Appendix B. Benchmark

The benchmark is an optimal HVAC schedule found by minimizing electricity cost with known prices. It uses a mixed-integer linear programming solver and assumes knowledge of the thermal model, backup-controller settings, and future disturbances.

Appendix B. Benchmark

  • Figure 7 plots outside temperature, HVAC power and electricity price, and the MABRL policy with resulting indoor air temperature.Source: Figure 7 · p. 10
  • The HVAC objective is to minimize electricity cost using known prices, and the optimal solution is found with mixed-integer linear programming.The solver used is Gurobi.Source: Appendix B · pp. 9–10
  • The benchmark optimal solution uses the thermal model, backup-controller settings, and future disturbances as known inputs.Source: Appendix B · p. 10
Loading 1507.03638v2…