Source-linked AI summary
Intelligent Electric Vehicle Charging Recommendation Based on Multi-Agent Reinforcement Learning
Weijia Zhang, Hao Liu, Fan Wang, Tong Xu, Haoran Xin, Dejing Dou, Hui Xiong
TL;DR
EV drivers can struggle to find charging because infrastructure is limited and demand is uneven, while one-step recommendations miss long-term effects. Master models stations as agents, coordinating them and incorporating future competition during training. On two real-world datasets, it achieves the best comprehensive performance against nine baselines.
Problem
Limited charging infrastructure and spatiotemporally unbalanced demand make it difficult for EV drivers to find charging, while greedy single-objective recommendations overlook long-term capacity-demand interactions.
Source: §1 · p. 1Method
Master formulates station-based charging recommendation as multi-objective MARL, using a centralized attentive critic, delayed access to future-competition information during training, and multi-critics with dynamic gradient re-weighting.
Source: §1 · p. 2Results
Master achieves the best comprehensive performance against nine baselines on two real-world datasets.
Source: §1 · p. 2Takeaways & Limitations
Decentralized execution enables immediate online recommendations without requiring future charging-competition information.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Electric Vehicle (EV) has become a preferable choice in the modern transportation system due to its environmental and energy sustainability. However, in many large cities, EV drivers often fail to find the proper spots for charging, because of the limited charging infrastructures and the spatiotemporally unbalanced charging demands. Indeed, the recent emergence of deep reinforcement learning provides great potential to improve the charging experience from various aspects over a long-term horizon. In this paper, we propose a framework, named Multi-Agent Spatio-Temporal Reinforcement Learning (Master), for intelligently recommending public accessible charging stations by jointly considering various long-term spatiotemporal factors. Specifically, by regarding each charging station as an individual agent, we formulate this problem as a multi-objective multi-agent reinforcement learning task. We first develop a multi-agent actor-critic framework with the centralized attentive critic to coordinate the recommendation between geo-distributed agents. Moreover, to quantify the influence of future potential charging competition, we introduce a delayed access strategy to exploit the knowledge of future charging competition during training. After that, to effectively optimize multiple learning objectives, we extend the centralized attentive critic to multi-critics and develop a dynamic gradient re-weighting strategy to adaptively guide the optimization direction. Finally, extensive experiments on two real-world datasets demonstrate that Master achieves the best comprehensive performance compared with nine baseline approaches.
1 INTRODUCTION
The introduction frames EV charging recommendations as a long-term, multi-objective problem shaped by limited capacity and uneven demand. Master treats stations as agents and combines coordination, future-competition information, and adaptive objective optimization.
1 INTRODUCTION
- Limited charging capacity and spatiotemporally unbalanced demand can leave drivers facing occupied stations and long waits, raising cost and inefficiency and potentially increasing range anxiety.The introduction identifies reducing CWT, CP, and CFR as recommendation goals.Source: §1 · p. 1
- Greedy recommendations optimize a single current-step objective and overlook long-term capacity-demand interactions, producing suboptimal global recommendations such as longer CWT and higher CFR.The passage cites prior greedy strategies as.Source: §1 · p. 1
- Assigning each charging station an agent gives quantity-independent state and action spaces, addressing centralized scalability while avoiding dedicated policies for ad-hoc, non-repetitive drivers.The framework is also described as more robust to potential failures of other agents.Source: §1 · p. 2
- A centralized attentive critic coordinates agents so stations can cooperate, for example by balancing requests away from heavily occupied stations.The introduction frames a charging request as ultimately being served by one station.Source: §1 · p. 2
- Future EVs can occupy the last available spot and cause extra CWT or charging failure; delayed access uses future-competition information in training while execution needs no future information.This supports immediate online recommendations through centralized training and decentralized execution.Source: §1 · p. 2, Figure 1 · p. 2
- Because long-term CWT, average CP, and CFR objectives may differ in scale and structure, Master uses multi-critics and dynamic gradient re-weighting to focus on poorly optimized objectives.The strategy is intended to guide optimization away from local optima induced by dominating objectives.Source: §1 · p. 2
- Master achieves the best comprehensive performance against nine baselines on two real-world datasets.The datasets were collected from Beijing and Shanghai.Source: §1 · p. 2
2 PROBLEM DEFINITION
The problem is to assign each daily charging request to a suitable station while minimizing overall wait time, average price, and failure rate. The section defines the three measures and when a request counts as finished.
2 PROBLEM DEFINITION
- A request is finished once it charges successfully or gives up, which is treated as charging failure.Source: §2 · p. 3
- CWT is the sum of travel time to the target station and queuing time there until the request is finished.Source: §2 · p. 3
- CP is the unit price per kilowatt hour, generally combining electricity cost and service fee.Source: §2 · p. 3
- CFR is the share of accepting requests that fail to charge, calculated over all requests accepting the recommendation.Source: §2 · p. 3
- For requests in a day, the recommendation task is to select a charging station while jointly minimizing overall CWT, average CP, and CFR among requests accepting recommendations.The problem statement defines the task over a set of charging requests Q and stations C.Source: §2 · p. 3
3 METHODOLOGY
Master formulates public charging recommendation as station-agent bidding over sequential requests, then combines centralized coordination, delayed access to future competition, and adaptive multi-objective optimization.
3.1 MARL Formulation
- Each charging station is an agent that bids for each request; the highest bid wins, while request-completion rewards encode CWT, CP, and failure penalties.Rewards are returned when a request finishes; exceeding the CWT threshold is treated as failure and receives a smaller reward.Source: §3.1 · p. 3
3.2 Centralized Training Decentralized Execution
- Master uses a centralized attentive critic to coordinate nearby active agents during training, while decentralized actors execute from local observations and highest bids.Attention integrates active-agent information despite different active sets across requests.Source: §3.2 · p. 4, §3.2.1 · p. 4, §3.2.3 · p. 5
- Delayed access postpones training access to a request's transition until subsequent station availability is observed, giving critics future competition information without changing execution.The strategy records available spots at intervals after the request time, excluding the request's own influence; negative counts represent queued EVs.Source: §3.2.2 · pp. 4–5
3.3 Multiple Objectives Optimization
- Two objective-specific attentive critics and gap-based dynamic gradient weights adapt actor updates between CWT and CP objectives instead of fixed reward averaging.A larger gap ratio signals a poorly optimized objective and calls for a larger update weight.Source: §3.3 · pp. 5–6
4 EXPERIMENTS
Experiments evaluate Master on Beijing and Shanghai charging data using four charging-related metrics and comparisons with nine baselines. The setup describes the datasets, training procedure, evaluation measures, and comparison methods.
4.1 Experimental setup
- The Beijing and Shanghai datasets cover May 18, 2019–July 1, 2019, with station data from a public app and charging requests from Baidu Maps.Charging-station supplies, prices, and powers were collected from the app.Source: §4.1.1 · p. 6
4.1.1 Data description.
- Each station’s future demand used data from its own grid and eight neighboring grids; the first 28 consecutive days formed training, the following three validation, and the remaining 14 testing.Table 1 summarizes statistics for the two datasets.Source: §4.1.1 · p. 6, Table 1 · p. 6
4.1.2 Implementation details.
- All RL algorithms recommended the top 50 nearest stations for 60 iterations, with the best validation iteration selected for testing.Master used a 30-minute competition-modeling interval, temperature 0.2, discount factor 0.99, and Adam with learning rate 5e−4.Source: §4.1.2 · p. 6
4.1.3 Evaluation metrics.
- Evaluation used MCWT, MCP, TSF, and CFR to measure waiting time, charging price, daily savings against ground truth, and charging failures.MCP is calculated over successful charging requests, while TSF can be negative when recommendations overspend relative to ground-truth actions.Source: §4.1.3 · pp. 6–7
4.1.4 Baselines.
- Comparisons included nine baselines and MASTER-ATT, a Master variant without charging-competition information or the multi-critic architecture.Baselines included centralized methods CDQN and CPPO, independent IDDPG, cooperative MADDPG, and several rule-based recommendations.Source: §4.1.4 · p. 7
4.2 Overall Performance
Across Beijing and Shanghai, Master achieves the most well-rounded performance among the compared baselines. It improves waiting time, price, savings, and failure metrics relative to multi-agent baselines, while also reducing waiting time, price, and failures relative to ground-truth actions.
4.2 Overall Performance
- 51.4%, 13.6%, and 96.5% reductions in MCWT, MCP, and CFR on Beijing, and 38.9%, 16.2%, and 91.0% on Shanghai, were reported against ground-truth charging actions.Recommendations also saved 16,219 fees per day on Beijing and 12,497 on Shanghai.Source: §4.2 · p. 7
- Master-ATT improved all four metrics over MADDPG, especially MCWT and CFR, with the attentive critic reported to avoid the full-state and joint-action burden.The comparison concerns two MARL algorithms using centralized training and decentralized execution.Source: §4.2 · p. 8
- Centralized RL methods CDQN and CPPO performed worse than the multi-agent approaches, while Greedy-P-10 led MCP but performed badly in MCWT and CFR.Greedy-P-10 selects the least expensive stations without considering the other goals.Source: §4.2 · p. 8
4.3 Ablation Study
The ablation study finds that centralized attention and future charging-competition information matter for MCWT, while the multi-critic architecture affects MCP and TSF. Varying the candidate count trades off MCWT against MCP and TSF, with reported endpoint values remaining in a small range.
4.3 Ablation Study
- The ablation evaluates Master and three variants on Beijing and Shanghai across four metrics.The variants remove centralized attention (noATT), future competition information (noCP), or the multi-critic architecture (noMC).Source: §4.3 · p. 8
4.4 Effect of Recommendation Candidates
- At k=20 and k=∗, (MCWT, MCP) are (9.56, 1.603) and (11.73, 1.437); relaxing k raises MCWT but lowers MCP and TSF.Across candidate counts, the authors describe performance as varying within a small range and remaining acceptable for online recommendation.Source: Figure 7 · p. 8, §4.4 · p. 8
4.3 Ablation Study
- Removing centralized attention or charging-competition information more strongly affects MCWT, while removing multi-critics degrades MCP and TSF.The authors attribute these effects to cooperation and competition for the first two components, and convergence problems for the single-critic variant.Source: §4.3 · p. 8
4.5 Convergence Analysis
Dynamic gradient re-weighting adjusts emphasis between CWT and CP as their convergence rates change, reducing asynchronous optimization. The case study relates critic attention and station actions to charging conditions and demand.
4.5 Convergence Analysis
- When CWT lags CP, dynamic re-weighting gives CWT more weight and adjusts the objectives toward synchronous convergence; static average weights do not address this asynchronism.The reported convergence curves use moving averages over every 500 successive training steps.Source: §4.5 · pp. 8–9
4.6 Case Study
- Actions are associated with supply, future supply and demand, ETA, and CP; low ETA and CP with available spots correspond to high actions, while scarce spots and high future demand correspond to low actions.The authors interpret these patterns as mediating the tension between constrained charging capacity and spatiotemporally unbalanced demand.Source: §4.6 · p. 9
- Figure 9 uses rows for factors, columns for the top-50 nearest stations, and darker colors for greater factor values; the critics attend more to high-action stations.The authors describe these stations as highly competitive bidding participants.Source: Figure 9 · p. 9, §4.6 · p. 9
5 RELATED WORK
Prior charging-recommendation work addresses travel time, queuing, fairness, or commercial benefits, while the paper frames its task around multiple divergent objectives and ad-hoc requests. Related MARL methods offer coordination strategies, but the cited transportation applications are not directly transferable to recommendation.
Charging Station Recommendation
- Most cited charging-location recommenders target time, while other work considers commercial benefits; these approaches mainly focus on a single objective, unlike the paper’s multi-objective task.Examples include minimizing travel and queue time, reducing idle time under fairness constraints, and charging strategies for commercial needs.Source: §5 · p. 9
Multi-Agent Reinforcement Learning
- Independent MARL agents may fail to coordinate, while learned communication can incur high information-transfer overhead; centralized training with decentralized execution supports lightweight, fault-tolerant execution.The passage presents this architecture as an alternative for large-scale agent systems.Source: §5 · pp. 9–10
Multi-Agent Transportation Systems
- Cited MARL transportation applications include traffic-signal control, ride-hailing, shared-bike repositioning, and delivery scheduling, but the authors argue they cannot be directly adopted for recommendation.Source: §5 · p. 10
6 CONCLUSION
The paper formulates EV charging recommendation as multi-objective MARL, with charging stations as agents and centralized attentive critics, delayed access, and adaptive multi-critic optimization. Experiments on two real-world datasets report effectiveness against nine baselines.
6 CONCLUSION
- Experiments on two real-world datasets demonstrate Master’s effectiveness compared with nine baselines.Source: §6 · p. 10
- Master treats each charging station as an agent and uses centralized attentive critics to coordinate cooperative policies, with delayed access incorporating future charging competition during training.Source: §6 · p. 10
- Multi-critics and dynamic gradient re-weighting adapt optimization across divergent objectives: overall CWT, average CP, and CFR.Source: §6 · p. 10
A SIMULATOR DESIGN
The simulator uses historical charging-system and road data to model a day of operation and collect reinforcement-learning transitions. It represents charging arrivals, recommendations, station capacity, and queues.
A SIMULATOR DESIGN
- The simulator combines historical station availability, prices, charging powers, EV charging, and road-network data to simulate a day.It loads station and request data minute by minute.Source: A SIMULATOR DESIGN · p. 11
- A Baidu API computes request-to-station ETA, while an MLP trained on historical requests predicts future demand during evaluation.The learned model is used to predict future demand in the evaluation phase.Source: A SIMULATOR DESIGN · p. 11
- Accepted recommendations can yield negative charging-wait-time and charging-price rewards; full stations queue vehicles until a spot opens or waiting exceeds 45 minutes.Failed requests receive penalties of −60 for waiting time and −2.8 for price.Source: A SIMULATOR DESIGN · p. 11
- When requests occur, the simulator stores charging-station transitions in replay memory for reinforcement-learning algorithms [27].The simulator also updates spot availability as vehicles depart and arrive; queued vehicles may make available-spot counts negative.Source: A SIMULATOR DESIGN · p. 11
B DATASET STATISTICAL ANALYSIS
The Beijing analysis finds that charging-request quantities vary across the day and that ETA and charging-price distributions differ substantially.
B DATASET STATISTICAL ANALYSIS
- Charging-request quantity changes notably across the day, indicating unbalanced demand.Figure 10 presents the distributions of request quantity, ETA, charging power, and charging price for Beijing.Source: B DATASET STATISTICAL ANALYSIS · p. 11
- The ETA distribution differs from the charging-price distribution, with charging price much more concentrated.The comparison indicates substantial differences between the two objectives.Source: B DATASET STATISTICAL ANALYSIS · p. 11
- The observed distribution contrast indicates large differences between the ETA and charging-price objectives.Source: B DATASET STATISTICAL ANALYSIS · p. 11
C PROPOSITION
The proposition considers two different reward functions and states that their policy-gradient updates are equivalent.
C PROPOSITION
- The proposition defines r1 (for example, r_cwt) and r2 (for example, r_cp) as different reward functions.Source: C PROPOSITION · p. 11
- The proposition states that the policy gradients for updating policy b are equivalent.Source: C PROPOSITION · pp. 11–12
- Figure 10 presents Beijing distributions for charging-request quantity, ETA, charging powers, and charging price.Source: Figure 10 · p. 12