PaperlayerView exact revision on arXiv

Source-linked AI summary

Definition and evaluation of model-free coordination of electrical vehicle charging with reinforcement learning

Nasrin Sadeghianpourhamami, Johannes Deleu, Chris Develder

arXiv:1809.10679v2cs.LGcs.AIstat.ML

TL;DR

Model-based demand response depends on accurate models despite uncertainty in EV behavior, while scalable joint RL coordination remains limited. The paper formulates an MDP with a station-count-independent aggregate state and actions over EV charging fractions. In real-world-data simulations, the learned policy reduces normalized cost relative to business-as-usual, while remaining more costly than an all-knowing optimum.

  • Problem

    Model-based demand response needs accurate models despite uncertain, heterogeneous EV-user behavior, and scalable RL coordination of multiple charging stations remains limited.

    Source: §I · pp. 1–2

  • Method

    The paper formulates an MDP with a station-count-independent aggregate state and actions that charge selected fractions of EVs on feasible flexibility diagonals.

    Source: B. Action Space · p. 4, §VII · p. 10

  • Results

    39% (10 stations) and 30.4% (50 stations) normalized-cost reductions versus BAU; learned policies cost 13% and 15.6% more than optimum, respectively.

    Source: §VII · p. 10

  • Takeaways & Limitations

    A policy learned from a smaller station group may coordinate a larger group when EV arrival, departure, and energy-demand distributions are similar.

    Source: §VII · p. 10

    Generalization to larger groups is supported when EV arrival, departure, and energy-demand distributions are similar.

    Source: §VII · p. 10

Abstract

from arXiv · show

Initial DR studies mainly adopt model predictive control and thus require accurate models of the control problem (e.g., a customer behavior model), which are to a large extent uncertain for the EV scenario. Hence, model-free approaches, especially based on reinforcement learning (RL) are an attractive alternative. In this paper, we propose a new Markov decision process (MDP) formulation in the RL framework, to jointly coordinate a set of EV charging stations. State-of-the-art algorithms either focus on a single EV, or perform the control of an aggregate of EVs in multiple steps (e.g., aggregate load decisions in one step, then a step translating the aggregate decision to individual connected EVs). On the contrary, we propose an RL approach to jointly control the whole set of EVs at once. We contribute a new MDP formulation, with a scalable state representation that is independent of the number of EV charging stations. Further, we use a batch reinforcement learning algorithm, i.e., an instance of fitted Q-iteration, to learn the optimal charging policy. We analyze its performance using simulation experiments based on a real-world EV charging data. More specifically, we (i) explore the various settings in training the RL policy (e.g., duration of the period with training data), (ii) compare its performance to an oracle all-knowing benchmark (which provides an upper bound for performance, relying on information that is not available or at least imperfect in practice), (iii) analyze performance over time, over the course of a full year to evaluate possible performance fluctuations (e.g, across different seasons), and (iv) demonstrate the generalization capacity of a learned control policy to larger sets of charging stations.

NOMENCLATURE

The introduction motivates model-free EV demand response by the difficulty of modeling customer behavior and the limited scalability of existing RL approaches. It presents a scalable MDP and fitted Q-iteration, then frames experiments on real-world data around training settings, oracle performance, variation over time, and generalization.

NOMENCLATURE

  • The nomenclature defines state, next state, time-to-departure, time-to-charge, flexibility, connected-EV counts, aggregate demand, action sets, and normalized policy costs.Source: NOMENCLATURE · p. 1

I. INTRODUCTION

  • Model-predictive demand-response methods require accurate models, while heterogeneous loads and uncertain user behavior make modeling and transfer across groups challenging.The introduction positions reinforcement learning as a model-free alternative that does not require a priori models of user flexibility or energy pricing.Source: §I · pp. 1–2
  • Existing RL demand-response methods face state- and action-space dimensionality challenges, and prior model-free solutions for jointly coordinating multiple EV stations are limited.Existing approaches either focus on an individual EV or use a heuristic to derive aggregate load decisions.Source: §I · p. 2
  • The proposed MDP uses compact state and action representations that do not scale linearly with station count and can generalize across group sizes and charging rates.The paper describes the formulation as a refinement of an earlier state and action representation in.Source: §I · p. 2
  • The paper uses batch reinforcement learning, specifically fitted Q-iteration, with function approximation to learn an EV charging policy.Source: §I · p. 2
  • Real-world-data simulations cover 10 and 50 charging stations and examine training-data settings, comparison with an all-knowing oracle, month-to-month performance, and generalization to different group sizes.The training questions include the duration of the data span and the number of sampled decision-tree trajectories.Source: §I · p. 2

II. RELATED WORK

Prior RL work addressed single-EV charging or fleet control through staged aggregate decisions and heuristic dispatch. This paper instead frames load flattening as a joint EV-group MDP, with a binned state based on connected-session timing and charging needs.

II. RELATED WORK

  • Shi et al. and Chis et al. study single-EV charging; includes V2G frequency regulation, while reports 10%–50% savings using real-world pricing data.Frequency regulation is described as instantaneous adjustment to balance generation and demand and keep grid frequency within tight bounds.Source: §II · p. 2
  • Claessens et al. and Vandael et al. use batch RL in fleet-control approaches that separate aggregate planning from individual-EV control or heuristic dispatch.The approach has aggregation, optimization, and real-time-control steps; uses day-ahead and intra-day decisions.Source: §II · pp. 2–3
  • Unlike and, the proposed approach jointly chooses actions for individual EVs using an aggregate state and does not require a heuristic to translate aggregate decisions.It learns aggregate load while seeking a policy to flatten the load curve.Source: §II · p. 3

III. MARKOV DECISION PROCESS

  • The paper targets load flattening by minimizing peak-to-average aggregate load through a sum-of-squared-consumption cost, with unknown transition probabilities.More advanced demand-response objectives are left for future work.Source: §III · p. 3

A. State Space

  • Each EV session is characterized by departure time and charging completion time; the current state omits arrival time because future arrivals are not known a priori.The formulation translates requested energy into charging time while implicitly assuming equal charging rates within a group.Source: A. State Space · p. 3

1 Initialize xs with zeros

The state representation bins connected EV sessions by remaining departure and charging times, then normalizes the resulting grid by the maximum station count. Its grid size depends on the connection horizon rather than the number of stations.

1 Initialize xs with zeros

  • The aggregate state pairs the timeslot with a matrix whose bins count EVs by remaining departure and charging times; the matrix dimensions depend on maximum connection time.The algorithm initializes the matrix with zeros and counts EVs into their corresponding bins.Source: §1 · p. 3
  • Normalizing the matrix by Nmax makes the aggregate state scale-free with respect to absolute group size.Source: §1 · p. 3
  • The formulation assumes common charging rates within a group, with a proposed extension that adds charging rate as a third binning dimension.Source: §1 · p. 3
  • The illustrative figure shows a 3×3 state grid and action states for two charging stations, followed by the full decision tree over three slots.The example bins sessions by departure and charging times; shaded cells indicate enough time to finish charging.Source: §1 · p. 3, Figure 1 · p. 4
  • The grid's diagonals encode charging flexibility; lower-diagonal EVs cannot meet requested demand by departure and are handled with a penalty in the cost function.The flexibility value is defined from the difference between departure and charging times.Source: §1 · p. 4
  • Because state size depends on Smax rather than Nmax, the representation scales to groups with different numbers of charging stations.Source: §1 · p. 4

B. Action Space

The action space groups EVs by flexibility and chooses what fraction of each eligible group to charge. The cost combines a quadratic demand term with a penalty for unfinished charging.

B. Action Space

  • Each action is a length-Smax vector whose entries specify the fraction of EVs on each main or upper diagonal to charge at full power.Lower diagonals are excluded to keep the action space relatively small; the cost is designed to prevent cars from ending up there.Source: B. Action Space · p. 4

C. Cost function

  • The demand cost is quadratic in total power consumption per decision slot, which is proportional to the number of EVs charged under the equal-rate assumption.The cost depends on aggregate demand rather than the timeslot, although time remains part of the state.Source: C. Cost function · pp. 4–5
  • A second cost term penalizes EVs whose charging cannot be completed by departure, with penalty factor M set above 2Nmax.The paper assumes only feasible charging requests, for which charging time does not exceed time until departure.Source: C. Cost function · p. 4
  • The transition cost sums the demand and unfinished-charging terms, making even one incomplete EV costlier than charging all EVs simultaneously.Source: C. Cost function · pp. 4–5

D. System Dynamics

EV arrivals and charging demands are stochastic, so transition probabilities are unknown. Perfect knowledge would permit dynamic programming on a decision tree; without it, transitions must be learned through interaction.

D. System Dynamics

  • Transition probabilities are unknown because EV arrivals and charging demands are stochastic.Source: D. System Dynamics · p. 5
  • With perfect knowledge of arrivals and demands over the control horizon, the problem becomes a decision tree whose action costs can be computed recursively by dynamic programming.Source: D. System Dynamics · p. 5
  • Without that knowledge, the transition probabilities must be estimated by taking actions in the environment and observing resulting transitions and costs.Source: D. System Dynamics · p. 5

E. Learning Objective: State-Action Value Function

The objective is to minimize expected T-step return, represented by an optimal state-action value function. Because transition probabilities are unknown, the paper learns an approximation to that function for control.

E. Learning Objective: State-Action Value Function

IV. BATCH REINFORCEMENT LEARNING

The paper learns a charging policy from stored EV-transition experiences using batch reinforcement learning and fitted Q-iteration. Although the state and action representations do not scale with fleet size, the state-action space grows exponentially, so only sampled trajectories are used.

IV. BATCH REINFORCEMENT LEARNING

  • Batch reinforcement learning separates data collection from optimization, using historical EV arrivals, departures, and demands plus a random policy to collect transition-cost tuples.Source: §IV · p. 5

A. Fitted Q-iteration

  • Fitted Q-iteration uses past transition-cost tuples to iteratively estimate the optimal action-value function through regression on labeled data.The action-value function starts at zero, and a fully connected neural network is used as the function approximator.Source: A. Fitted Q-iteration · p. 5

B. The size of state-action space

  • State and action representations are independent of the number of stations, but the state-action space still grows exponentially with the control horizon.For 50 occupied stations and a 10-slot horizon, the described state has at least 51 possible actions when EVs share the same flexibility.Source: B. The size of state-action space · p. 6
  • Because the full state-action space is infeasible to include in FQI, the method samples decision-tree trajectories, raising the question of how many suffice across group sizes.Source: B. The size of state-action space · p. 6

V. EXPERIMENT SETUP

The experiments use ElaadNL charging transactions to construct daily EV-charging episodes and train a fitted Q-iteration policy. The setup fixes a tractable state-action representation and samples trajectories from each day.

A. Data Preparation

  • The dataset contains over 2 million charging sessions from more than 2,500 public stations in the Netherlands, with arrival, departure, requested-energy, and charging-rate records.The privately owned vehicles represent a mixture of car types that is unknown in advance.Source: A. Data Preparation · p. 6
  • Sessions are represented with a 24-hour maximum connection duration because more than 98% of ElaadNL sessions last less than 24 hours.This duration sets the maximum horizon for representing charging demand.Source: A. Data Preparation · p. 6
  • A 2-hour decision slot yields 12 slots and a 12 × 12 demand matrix, while limiting the action branching factor to support training within available computation time.Each action has 12 decisions, one per flexibility window.Source: A. Data Preparation · p. 6
  • Each episode is a day beginning at 7 am, with the system assumed empty between days and a zero-demand terminal state.The terminal state gives trajectories zero action-value and is intended to stabilize FQI learning with a neural-network function approximator.Source: A. Data Preparation · p. 6

B. Algorithm Settings

  • The experiments compare the 10 and 50 busiest charging stations, and FQI uses 12 iterations to estimate the 12-step return.Station subsets are chosen by recorded transaction count.Source: A. Data Preparation · p. 6, B. Algorithm Settings · p. 6
  • Training data are generated by randomly sampling trajectories from each day, recording state, action, next state, and transition cost until the terminal state.Each sample is a full trajectory from the day's initial to terminal state.Source: 1) Creating set F · p. 6

1) Creating set F:

The setup builds F from sampled daily trajectories, fits an ANN to state-action pairs, and evaluates learned policies against optimum and business-as-usual costs. Training spans and test days are specified to assess performance across different data windows.

1) Creating set F:

  • Each sample is a full trajectory from initial to terminal state; charging demands are adjusted so requested charging can be fulfilled within 24 hours.The experiments use an Intel Xeon E5645 processor with 2.4 GHz and 290 GB RAM.Source: 1) Creating set F · pp. 6–7

2) Neural network architecture:

  • The ANN has two ReLU hidden layers with 128 and 64 neurons, takes a 145-value state and 12-value action, and uses Huber loss to stabilize learning.Its single-neuron output layer uses linear activation.Source: 2) Neural network architecture · p. 7

C. Performance Evaluation Measure

  • The test set is the last 92 days of 2015, while training spans range from 1 to 9 months and use five randomly selected contiguous periods for spans shorter than 9 months.The 9-month span covers the full training-data range from January 1 through September 30.Source: C. Performance Evaluation Measure · p. 7
  • Policy cost is normalized against the optimum, where a learned optimum policy scores 1; performance is also compared with BAU charging that starts immediately on arrival.The optimum cost is obtained by quadratic optimization, and daily policy costs sum instantaneous transition costs.Source: C. Performance Evaluation Measure · p. 7

VI. EXPERIMENTAL RESULTS

The experiments compare learned charging coordination with BAU and optimal costs, examine how training and test periods affect performance, and test a policy on larger station groups.

A. Learning the Charging Coordination (Q1–Q2)

B. Variance of performance over time (Q3)

VI. EXPERIMENTAL RESULTS

  • The experiments also test the normalized cost of a policy trained on 10 stations when applied to larger groups.Source: §VI · p. 7, Figure 5 · p. 9

B. Variance of performance over time (Q3)

  • Across test months, greater charging flexibility brought larger cost reductions versus BAU but made it harder for RL to approach the optimum, partly because crucial high-delay days were scarce in training.Source: B. Variance of performance over time… · p. 9

C. Generalization to Larger Scales (Q4)

A policy trained on 10 stations generalizes to larger groups when EV arrival, departure, and energy-demand distributions are similar. The section tests this transfer by scaling up a duplicated test set and reports how normalized cost changes.

C. Generalization to Larger Scales (Q4)

  • Training on the largest set—nine months with 5K sampled trajectories per day—takes approximately 3 hours for 10 stations and 48 hours for 50.The reported runtimes were measured on an Intel Xeon E5645 processor.Source: C. Generalization to Larger Scales (Q4) · p. 9
  • The test set duplicates EV sessions from the last quarter of 2015, while the policy is trained on nine months of data from 10 stations.This isolates the effect of increasing Nmax without changing other system characteristics.Source: C. Generalization to Larger Scales (Q4) · p. 9
  • Duplicating the EV group can change the optimum: the example shifts from charging 50% of cars in two slots to 25% in each slot.The example uses a four-slot control horizon.Source: C. Generalization to Larger Scales (Q4) · p. 9
  • After the 2× scale increase, further increases in Nmax cause only marginal normalized-cost increases across 5K–20K sampled trajectories.The largest jumps in normalized cost occur when the group size doubles.Source: Figure 6 · p. 10, C. Generalization to Larger Scales (Q4) · p. 10

VII. CONCLUSION

The conclusion reports the learned policy’s cost reductions against BAU and an all-knowing optimum, alongside month-to-month variation and transfer to larger station groups. It also identifies training-data sampling and algorithmic alternatives for future study.

VII. CONCLUSION

  • The paper formulates an MDP with a scalable aggregate state that accounts for individual EV arrival time, charging duration, and connection duration.The formulation is independent of the number of stations and charging rates.Source: §VII · p. 10
  • For training spans longer than three months, about 5K sampled trajectories per day achieve performance similar to larger sample counts.The state and action representations are group-size independent, but the state-action space remains too large to feed in full to FQI.Source: §VII · p. 10
  • The learned policy reduces normalized cost versus BAU by 39% for 10 stations and 30.4% for 50, while costing 13% and 15.6% more than the optimum, respectively.The optimum assumes perfect knowledge of future EV charging demand.Source: §VII · p. 10
  • Across months, greater flexibility corresponds to larger cost reductions versus BAU, but also a larger gap from the optimum.The authors attribute the larger gap to scarce high-flexibility training days and random sampling that may miss rare, crucial state-action pairs.Source: §VII · p. 10
  • A policy trained on 10 stations generalizes to groups with up to 10× more arrivals when EV arrival, departure, and energy-demand distributions are similar.The conclusion presents this as evidence that the MDP formulation generalizes across group sizes.Source: §VII · p. 10
  • Future work will examine informed exploration, convolutional function approximation, and policy iteration as alternatives to random exploration, fully connected networks, and value iteration.The current approach uses random state-action sampling, a fully connected network, and value iteration.Source: §VII · pp. 10–11

Related papers

Loading 1809.10679v2…