Learning to Operate an Electric Vehicle Charging Station Considering Vehicle-grid Integration

I. INTRODUCTION

• Model-based scheduling can be time-consuming at scale, while multi-EV reinforcement learning must handle state and action dimensions that vary with stochastic arrivals and departures.Existing approaches described in the passage use heuristics, manually designed features, fixed-dimensional grids, or future-condition predictions, each with stated limitations.Source: §I · p. 3
• CADE combines centralized EV assignment with decentralized reinforcement-learning charger control, using shared operational experience to learn policies under stochastic arrivals and charging needs.An online algorithm assigns the station-wide demand-charge penalty to individual chargers.Source: §I · pp. 3–4

II. PROBLEM FORMULATION

• The station has charging and waiting spots, and its scheduling objective is to maximize profit while meeting customer charging needs.Profit comprises net charging and discharging revenue, penalties for unmet charging needs, and demand charges.Source: §II · p. 4

A. Net Revenue from Charging and Discharging EVs

• Charging and discharging generate station revenue through EV-owner payments and transactions with the electric utility.The formulation assumes the payment to customers for discharging exceeds the EV charging cost to cover battery degradation.Source: A. Net Revenue from Charging and… · p. 4

B. Penalty of Failing to Satisfy Customers’ Charging Needs

• An EV customer specifies a departure time and target energy level upon arrival, and the station incurs a penalty if the EV departs below that target.The penalty reflects the gap between final and target energy levels.Source: B. Penalty of Failing to Satisfy… · p. 4
• The paper uses a linear function for each EV’s shortfall penalty, with the total station penalty equal to the sum across EVs.The passage identifies this linear form as suggested by.Source: B. Penalty of Failing to Satisfy… · pp. 4–5
• Requested target energy is restricted to a threshold physically achievable given the battery capacity and maximum charging power.The passage also specifies that the penalty activates only for a shortfall.Source: B. Penalty of Failing to Satisfy… · pp. 4–5

C. Demand Charge

• Demand charges are based on maximum load during the billing period, and differing on-peak and off-peak rates incentivize shifting charging toward off-peak periods.The passage also notes that discharging during on-peak periods is incentivized.Source: C. Demand Charge · p. 5
• The demand-charge calculation uses the recorded peak load for each time-of-use period, with T/TB scaling the result when the simulation and billing horizons differ.The recorded peak is the maximum summed charger power within the period; negative peak demand is not compensated.Source: C. Demand Charge · p. 5

D. Summary of the Charging Station Optimization Problem

• The station optimization maximizes profit subject to operational constraints grouped into three categories.One stated constraint sets an EV’s charging power to zero when it is not connected to a charger.Source: D. Summary of the Charging Station… · p. 5

1) Power constraints:

• The feasible charging-power range is constrained by charger and battery limits, and disconnected EVs must receive zero power.The power constraints define the permitted range for each EV’s charging power.Source: 1) Power constraints · p. 5

2) Space constraints:

• Charger occupancy and waiting-area capacity are limited, and an EV in the station must be either connected to a charger or in a waiting spot.An EV outside the station cannot be connected to a charger.Source: 2) Space constraints · p. 5

3) Energy constraints:

• Battery energy must remain within [emin, emax] while an EV is in the charging station.Source: 3) Energy constraints · p. 5
• MPC effectiveness depends heavily on accurate predictions of future EV arrivals and does not scale well as problem size grows.These limitations motivate model-free reinforcement-learning methods for real-time decisions in complex environments.Source: 3) Energy constraints · pp. 5–6

III. TECHNICAL METHODS

• CADE assigns arriving EVs centrally to chargers or waiting spots, then has each charger make local charging or discharging decisions as an RL agent.Experiences from all chargers are collected to update shared action-value networks, whose parameters are broadcast to the chargers.Source: §III · p. 6

A. Overview of Markov Decision Process

• In an MDP, an agent observes a state, chooses an action, receives a reward, and transitions to a new state; its objective is to maximize expected discounted rewards.The discount factor balances the relative importance of near-term and future rewards.Source: A. Overview of Markov Decision Process · p. 6

• Each charger is an agent whose environment consists of its connected EV and the charging station.Source: B. Formulate the Charger Control… · p. 6
• The charger state includes connection status, global time, remaining dwell time and energy needs, waiting-area demand, time-of-use period, and recorded peak load.The state also represents the connected EV’s current energy and the remaining energy required to reach its target.Source: B. Formulate the Charger Control… · p. 6
• Each charger’s action is its output power, selected from a discrete, uniformly spaced set bounded by physical charger limits and the EV battery’s feasible charging or discharging rate.Source: B. Formulate the Charger Control… · p. 7
• The reward aligns charger control with station net revenue while accounting for demand charges and penalties for unmet customer charging needs.The connected-EV penalty applies when an EV leaves below its desired energy level; waiting-area shortfalls are also penalized, with their aggregate penalty split among chargers.Source: B. Formulate the Charger Control… · p. 7
• The demand-charge component penalizes charger output when aggregate station power exceeds the recorded peak for the current time-of-use period.The increase is scaled when the study horizon differs from the electricity billing period.Source: B. Formulate the Charger Control… · p. 7

• Each charger stores state, action, next-state, and reward experiences in a shared replay memory used to learn charger action-value functions with DQN.Source: C. Decentralized Execution at the… · p. 7
• DQN uses a target network alongside the Q-network to stabilize learning by reducing correlations between the function approximator and its target.The Q-network parameters are updated by stochastic gradient descent, and target-network parameters are periodically replaced by them.Source: C. Decentralized Execution at the… · p. 7
• The Q-network takes a charger’s state and action as input and outputs one Q(s, a) value, matching the ordinal charging-power decisions.Source: C. Decentralized Execution at the… · pp. 7–8

• The scheduler uses the learned charger action-value function to choose whether each EV is connected to a charger or assigned to a waiting spot.A connected EV’s allocation value is the best Q value over actions; waiting is represented by the Q value for zero power.Source: D. Centralized Allocation at the… · p. 8
• The waiting-area energy demand used in evaluating candidate EV assignments is approximated by its value from the previous time step.Source: D. Centralized Allocation at the… · p. 8
• The allocation maximizes the sum of EV connection values while satisfying the charging-area and waiting-space limits.When the number of arriving EVs does not exceed charger capacity, all EVs are allocated to the charging area.Source: D. Centralized Allocation at the… · p. 8

E. Summary of CADE Framework

• CADE allocates arriving EVs to chargers and the waiting area by solving an optimization problem that uses Q(s, a; θ) to generate qc.The centralized allocation is followed by decentralized execution by individual chargers.Source: E. Summary of CADE Framework · p. 8
• Each charger obtains its own state and executes charging or discharging actions in the decentralized training loop.The framework description identifies the decentralized execution steps as lines 11, 17, and 21 during deployment.Source: E. Summary of CADE Framework · p. 8
• Experiences from homogeneous chargers are stored in a shared replay memory, and all chargers share parameter vector θ.A mini-batch is sampled from the memory to minimize the loss function.Source: E. Summary of CADE Framework · p. 8

IV. NUMERICAL STUDY

• The study compares proposed and baseline algorithms in terms of optimality and scalability, and examines learned action-value functions across operating scenarios.The supplied study description does not report comparison outcomes in this section.Source: §IV · p. 8
• CADE training uses an Nvidia GeForce RTX 2080 Ti GPU.Source: §IV · p. 9
• The MPC-based baseline runs on an AMD Ryzen 7 8-core CPU with the Gurobi solver.Source: §IV · p. 9

A. Setup of the Numerical Study

• EV arrivals are modeled as a Poisson process across office, residential, highway, and retail locations, without prior knowledge of their specific patterns.Residential arrivals may be higher in the afternoon, while highway and retail patterns have different daily peaks.Source: A. Setup of the Numerical Study · p. 9, Figure 2 · p. 9
• The study assumes on-peak, mid-peak, and off-peak periods, with constant customer charging and discharging prices and pd greater than pc.The passage says the higher discharging price compensates customers for battery degradation.Source: A. Setup of the Numerical Study · p. 9
• The action-value network has three hidden layers with 256, 128, and 64 neurons, and uses SGD with an initial learning rate of 0.01.The learning rate decreases during training, γ is 1, and the target network updates every 25 episodes.Source: A. Setup of the Numerical Study · p. 9

B. Learning Performance of the Proposed CADE Framework

• Around 1,000 training episodes, charging-station profit stabilizes as its components improve; the penalty term falls to zero, while demand-charge reduction is limited.The paper attributes the limited demand-charge reduction to difficulty predicting future EV charging demands.Source: B. Learning Performance of the… · p. 10
• Chargers learn to charge mostly during off-peak and mid-peak hours when pc − pe > 0 and demand-charge price pl is relatively low.Source: B. Learning Performance of the… · p. 10
• Chargers learn to discharge during on-peak hours when pd − pe > 0 and demand-charge price pl is high.Source: B. Learning Performance of the… · p. 10

D. Scalability Analysis

• $7.8 to $8.4 average daily profit per charger is maintained as station size increases from 10 to 100 chargers.Waiting spots are set to half the number of chargers, and the office-zone arrival pattern is scaled with station size.Source: Figure 6 · p. 10, D. Scalability Analysis · p. 10

C. Profit Comparison with Baseline Methods

• CADE outperforms all baseline methods in profit except MPC-ideal, which has the unfair advantage of perfect knowledge of future EV arrivals.Source: C. Profit Comparison with Baseline… · p. 10

D. Scalability Analysis

• CADE's computation time is much shorter than MPC's and grows linearly with charger count rather than exponentially.Source: D. Scalability Analysis · p. 11, Table III · p. 11

E. Interpretation of the Learned State-Action Values

• The learned Q values favor off-peak charging, on-peak discharging, higher charging power near departure or with more waiting EVs, and higher power after a higher historical peak load.Source: E. Interpretation of the Learned… · p. 11, Figure 7 · p. 11

V. CONCLUSION

• The conclusion frames CADE as maximizing station profit from charging and discharging revenue while accounting for demand charges and penalties.Source: §V · p. 11