PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolutional Neural Networks For Automatic State-Time Feature Extraction in Reinforcement Learning Applied to Residential Load Control

Bert J. Claessens, Peter Vrancx, Frederik Ruelens

arXiv:1604.08382v2cs.LGcs.SYeess.SY

TL;DR

Direct load control of heterogeneous residential flexibility sources faces high-dimensional states and partial observability. The paper combines convolutional feature extraction with fitted Q-iteration, using past observations to learn state-time features. In simulation, scaled performance reached 0.95 after about 30 days and then remained stable.

  • Problem

    Direct control of heterogeneous residential flexibility clusters involves high-dimensional states and relevant information that is not directly observed, while accurate model acquisition and maintenance can outweigh its financial benefits.

    Source: §I · p. 1

  • Method

    The approach uses a CNN and multilayer perceptron in fitted Q-iteration, presenting sequences of state-bin distributions to extract state-time features.

    Source: C. High-Dimensional Feature Extraction · p. 3

  • Results

    After about 30 days, scaled performance reaches 0.95 and remains stable, with a scaled performance of one corresponding to the theoretical benchmark.

    Source: D. Results · p. 8

  • Takeaways & Limitations

    In a qualitative simulation of heterogeneous TCLs with partial observability, the approach extracted state-time features relevant to learning near-optimal policies.

    Source: C. High-Dimensional Feature Extraction · p. 3

Abstract

from arXiv · show

Direct load control of a heterogeneous cluster of residential demand flexibility sources is a high-dimensional control problem with partial observability. This work proposes a novel approach that uses a convolutional neural network to extract hidden state-time features to mitigate the curse of partial observability. More specific, a convolutional neural network is used as a function approximator to estimate the state-action value function or Q-function in the supervised learning step of fitted Q-iteration. The approach is evaluated in a qualitative simulation, comprising a cluster of thermostatically controlled loads that only share their air temperature, whilst their envelope temperature remains hidden. The simulation results show that the presented approach is able to capture the underlying hidden features and successfully reduce the electricity cost the cluster.

I. INTRODUCTION

Residential-cluster load control combines sequential decisions with high-dimensional, heterogeneous states and partial observability. This work uses convolutional feature extraction within fitted Q-iteration to learn state-time features from past observations.

I. INTRODUCTION

  • Residential flexibility control involves sequential decisions, high-dimensional heterogeneous states, and relevant states that may not be directly observed.Model-based control also depends on model fidelity, while obtaining and maintaining an accurate model can outweigh its financial benefits.Source: §I · p. 1

A. Aggregate and dispatch

  • Aggregate-and-dispatch uses a reduced-dimensional bulk model for centralized optimization and dispatches set-points to devices with a heuristic, but still requires centralized system identification.The reduced dimensionality makes centralized MPC tractable.Source: A. Aggregate and dispatch · p. 2

B. Reinforcement Learning for demand response

  • Earlier reinforcement-learning work on aggregate-and-dispatch used small, handcrafted state descriptions and assumed full observability, motivating richer states and automatic high-dimensional feature extraction.Including previous observations can support extraction of state-time features representative of non-observable state information.Source: B. Reinforcement Learning for demand… · p. 3

C. High-Dimensional Feature Extraction

  • A CNN and multilayer perceptron are used as the regression algorithm in fitted Q-iteration, with state-bin histories supplying features relevant to near-optimal policies.The resulting strategy is evaluated in a simplified qualitative scenario of heterogeneous TCLs with partial observability and time-varying prices.Source: C. High-Dimensional Feature Extraction · p. 3

II. RELATED WORK AND CONTRIBUTION

  • The contribution combines convolutional and multilayer-perceptron components for batch reinforcement learning and evaluates the strategy in a qualitative residential-load-control scenario.The related-work overview covers heterogeneous TCL control, batch RL for load control, and automated feature extraction.Source: §II · p. 2, C. High-Dimensional Feature Extraction · p. 3

III. BACKGROUND AND MOTIVATION

Residential TCL control is partially observable because the building-mass temperature is hidden while operational air temperature is available. The paper frames control as an MDP and seeks a policy minimizing finite-horizon cost using state-action values.

III. BACKGROUND AND MOTIVATION

  • TCL dynamics span fast air-temperature and slow building-mass time scales, but typically only operational air temperature is available.A model-based Kalman filter requires a calibrated model; model-free estimation from recent observations expands the state space.Source: §III · p. 3
  • The paper combines batch reinforcement learning with a convolutional neural network to extract information about non-observable states under partial observability.The network is motivated by avoiding the severe state-space extension associated with extracting hidden-state information from observation histories.Source: §III · p. 3

IV. PROBLEM FORMULATION

  • The control problem is formulated as an MDP with state space, action space, transition function, and transition cost.Transitions evolve from x_k to x_{k+1} under action u_k and random process w_k.Source: §IV · pp. 3–4
  • The objective is to find a policy minimizing the T-stage cost, represented through a Q-function as cumulative return from a state-action pair followed by that policy.The optimal Q-function is the best obtainable across policies, and the optimal policy is calculated from it.Source: §IV · p. 4

A. State description

The state representation combines time-dependent, controllable, and exogenous information. It includes hour-of-day timing, TCL operating temperatures, and outside temperature with an available forecast.

A. State description

  • The state space comprises time-dependent, controllable physical, and exogenous uncontrollable information.The observable state vector is introduced, though its passage does not provide further readable detail.Source: A. State description · p. 4
  • The time-dependent component contains hour of day to represent timing in the non-stationary dynamics.The stated rationale is that consumer behavior tends to be repetitive and diurnal.Source: A. State description · p. 4
  • The controllable physical state includes each TCL’s operational temperature and user-set lower and upper temperature bounds.Source: A. State description · p. 4
  • The exogenous state includes outside temperature, which affects physical dynamics but cannot be influenced by the control action; its forecast is assumed available for Q-function calculation.Source: A. State description · p. 4

B. Backup controller and physical realisation

Each TCL receives a binary requested ON/OFF action, which a backup controller may override according to indoor-air-temperature comfort bounds.

B. Backup controller and physical realisation

C. Cost function

The cost function addresses energy arbitrage against an external electricity-price vector. The implementation follows a three-step sequence: aggregation, optimization, and real-time control.

C. Cost function

  • Energy arbitrage is the control objective, with TCLs responding to an external price vector λ.The cost definition uses each TCL’s average power consumption during an interval and the electricity price at a time step.Source: C. Cost function · p. 4

V. IMPLEMENTATION

  • The implementation uses aggregation, optimization, and real-time control, while the indicated models provide room temperature T but not building-mass temperature Tm.Source: §V · p. 5, Figure 1 · p. 5

A. Step 1: Aggregation

The aggregation step represents historical cluster observations as binned states across time. Batch reinforcement learning then uses tuples to approximate the state-action value function.

A. Step 1: Aggregation

  • Historical observations are binned by each TCL's state of charge, then concatenated across time steps into the aggregate state representation.The bins use equidistant support points spanning the cluster's minimum and maximum state of charge.Source: A. Step 1: Aggregation · p. 5

B. Step 2: Batch Reinforcement Learning

1) Fitted Q-Iteration:

Fitted Q-iteration uses forecasted exogenous information and a neural network to estimate Q-values. The proposed regression network combines convolutional processing of binned state histories with a multilayer perceptron for other inputs.

1) Fitted Q-Iteration:

  • Fitted Q-iteration substitutes forecasted outside temperature for observed successor-state exogenous information, evaluates candidate actions, and trains the network on state-action inputs paired with Q-values.The network is evaluated |U| times to determine the minimum value of the current Q-function approximation.Source: 1) Fitted Q-Iteration · p. 5
  • The proposed Q-function approximator uses a convolutional component because the high-dimensional binned state history is expected to contain strongly locally correlated state-time features.The network receives the state-action pair and returns an approximated Q-value.Source: 1) Fitted Q-Iteration · p. 5, 2) Regression Algorithm · p. 5

2) Regression Algorithm:

  • The regression network processes the binned state history with two convolutional layers and sends time-dependent state, exogenous information, and action through a separate dense network before merging their features.Convolutions operate across both bin and time dimensions to identify local structure and events spanning multiple time steps; fully connected layers produce the predicted Q-value.Source: Figure 2 · p. 6, 2) Regression Algorithm · p. 6

C. Step 3: Real-time control

The controller selects a cluster-level action with an ε-greedy strategy, then dispatches it to individual thermostatically controlled loads through an auction-based system.

C. Step 3: Real-time control

  • The controller selects the cluster action using an ε-greedy strategy whose exploration probability decreases daily according to a harmonic sequence.The chosen action is dispatched across thermostatically controlled loads through an auction-based multi-agent system.Source: C. Step 3: Real-time control · p. 6
  • Each TCL's bid function specifies consumed power as a function of heuristic priority, with its corner priority indicating the urgency of scheduling.Scheduling priority increases as the TCL's state of charge approaches zero.Source: C. Step 3: Real-time control · p. 7
  • The cluster-level clearing process matches the aggregate action to an aggregate bid function and returns a clearing priority to the TCLs.The returned clearing priority is sent to the different TCLs.Source: C. Step 3: Real-time control · p. 7

VI. RESULTS

The qualitative simulation evaluates a controller on a cluster of thermostatically controlled loads, with a theoretical benchmark providing a perfect-information cost reference. Ignoring the hidden building-mass temperature increases benchmark cost.

VI. RESULTS

  • The simulation models a cluster of 400 TCLs exposed to dynamic energy prices, with a 24-hour optimization horizon and comfort constraints safeguarded by a backup controller.The control period is one hour, and the policy is updated every 24 hours using a new price profile and outside-temperature forecast.Source: §VI · p. 7

A. Simulation model

  • The simulated second-order building model treats indoor air temperature as observable and building-mass temperature as hidden, while TCL heterogeneity produces dephasing and loss of power-consumption coherence after about 30 hours.The figure depicts the TCL temperature distribution, average power, and average observable and non-observable temperatures over time.Source: A. Simulation model · p. 7, Figure 3 · p. 7

B. Theoretical benchmark

  • Ignoring the non-observable building-mass temperature Tm increased theoretical benchmark cost by 2.5%.The benchmark has perfect model information and access to building-mass temperature, so its result is a lower limit on cost.Source: B. Theoretical benchmark · pp. 7–8

C. Deep regression architecture

A convolutional regression architecture uses past observations to extract state-time features, and simulations show near-benchmark control with lower costs when observation history is included. The section also reports the learning trajectory and the study’s partial-ob observability setting.

C. Deep regression architecture

  • The CNN processes a 28 × 28 grid of past observations through two convolutional layers, while a feedforward branch handles current state and action inputs.The convolutional layers use four 7 × 7 filters and eight 5 × 5 filters; their features are combined with the feedforward branch to estimate one output.Source: C. Deep regression architecture · p. 8

E. State-time features

  • Adding observation history reduces average cost by approximately 1.2% compared with omitting history, with a two-sample t-test indicating nearly 97% probability of different means.The comparison holds the network architecture unchanged and normalizes cumulative cost over the final 30 days; Table I identifies the conditions as A without and B with state-time features.Source: Table I · p. 8, E. State-time features · p. 8

D. Results

  • Scaled performance reaches 0.95 after about 30 days and remains stable, with 1 corresponding to the theoretical benchmark solution.The measure is daily cost using the proposed approach scaled by daily cost from the theoretical benchmark, averaged over six runs.Source: D. Results · p. 8

VII. CONCLUSIONS

  • In a qualitative simulation of heterogeneous thermostatically controlled loads, the CNN extracts state-time features from past observations and achieves near-optimal results despite hidden envelope temperatures.The loads share operational air temperature, while envelope temperature remains hidden.Source: §VII · p. 8

D. Results

  • After 60 and 70 days, the approach’s power profiles are close to the benchmark across different outside temperatures.Figure 4 contrasts learned profiles in black with benchmark profiles in red, alongside price profiles and outside temperatures.Source: D. Results · p. 8, Figure 4 · p. 9
  • Across six runs, Figure 5 shows average cluster power with a 95% envelope; its final two subplots use a purely greedy policy.The dashed line gives daily prices, while the black line and gray envelope show mean power and the 95% range of profiles.Source: D. Results · p. 8, Figure 5 · p. 9

APPENDIX: CONVOLUTIONAL NEURAL NETWORK

The appendix provides code for the simulated neural network, including its convolutional branch and dense layers. Figure 6 compares scaled performance with and without state-time features and also shows average outside temperature.

APPENDIX: CONVOLUTIONAL NEURAL NETWORK

  • Figure 6 depicts average scaled performance over six runs with and without state-time features, alongside average outside temperature.The caption identifies the comparison but does not state which condition performs better.Source: Figure 6 · p. 9
  • The code reshapes the input into a 28 × 28 grid and applies convolutional layers with four filters followed by eight filters.The second convolution uses 5 × 5 filters; the first uses a width parameter supplied in the code.Source: APPENDIX: CONVOLUTIONAL NEURAL NETWORK · p. 9
  • After flattening, CNN adds a dense layer with 8*scaledGraph*scaledGraph inputs and 32 units; model then adds a Dense(48,24) layer and a Dense(24,1) layer.Source: APPENDIX: CONVOLUTIONAL NEURAL NETWORK · p. 9
Loading 1604.08382v2…