Source-linked AI summary
Emergent Charging Coordination in Electric Delivery Fleets
Javier Vales-Alonso, Juan J. Alcaraz
TL;DR
Electric delivery fleets must coordinate when, where, and how much to charge despite shared-station congestion and limited shift time. The paper uses shared learned policies that act locally on vehicle state and public station occupancies, then tests them zero-shot across real-city networks. PPO-joint completes 98.6% of shifts on time across 20 cities, near the 99.5% Oracle reference.
Problem
Shared charging stations couple vehicles’ choices about when to charge, which station to use, and how much energy to add, while prior approaches may require central coordination or reservations.
Source: §1 · pp. 2–3Method
Each vehicle runs a learned controller that uses its own state and public station congestion to decide charging actions, without reservations, messaging, or a central controller.
Source: §1 · p. 3Results
98.6% of delivery shifts finish on time with PPO-joint across the 20-city zero-shot benchmark, compared with the omniscient Oracle reference of 99.5%.
Source: §1 · p. 4Takeaways & Limitations
The results show that local reactions to public congestion can coordinate a fleet without central control or vehicle messaging.
Source: §1 · p. 3, §8 · p. 34The congestion model assumes stationary arrivals within a shift, and charging costs and battery degradation are not modeled.
Source: §8 · p. 34
Abstract
from arXiv · showhide
In electric delivery fleets, mid-shift charging is non-trivial: each vehicle must decide when, where and how much to charge to finish on time with battery above a safety floor. The choices are coupled: queues build where too many vehicles pick the same station. Prior work resolves this coupling with central dispatching, precomputed schedules or reservations, machinery that charging infrastructure rarely supports. Instead, we use a family of learning agents under purely local control: every vehicle runs the same policy, deciding alone from its time budgets and broadcast station occupancies, leading to emergent coordination without central control or messaging. We validate this paradigm in simulation on real OpenStreetMap networks of twenty cities, each with a frozen scenario calibrated by an omniscient Oracle (99.5% of shifts completed on time), whereas a naive greedy rule (nearest station on low battery) completes just 73%. Agents trained with neuroevolution (NEAT) and policy gradients (PPO) on four cities and deployed zero-shot across all twenty, sixteen never seen in training, complete 96.8% and 98.6% of shifts, with the policy-gradient controllers proving more robust when demand or vehicle characteristics drift beyond the trained regime. In contrast, tuned threshold heuristics that read vehicle urgency alone fall short in contended cities (~80%). Through training, these learning agents rediscover partial charging and short opportunistic sessions, and route around busy stations, cutting per-session queue waits from about 45 minutes to under 2. In summary, this coordination paradigm balances local urgency against public occupancy, reaching near-Oracle performance at minimal implementation cost.
1. Introduction
Mid-shift charging couples decisions about timing, station choice, and charge target, while shared-station congestion makes individually reasonable choices affect the fleet. The paper proposes learned, local control using vehicle state and public station congestion, and evaluates it on real-city networks.
1. Introduction
- Each vehicle must decide when to charge, which station to use, and how much to charge, with shared stations coupling vehicles’ choices.A full charge can consume too much shift time, while a partial charge may require another stop; a farther, less congested station can save time overall.Source: §1 · p. 2, Figure 1 · p. 3, Figure 2 · p. 4
- The proposed emergent coordination has each vehicle use its own state and public station congestion, with no reservations, messages, or central controller.Every vehicle runs the same learned controller, which balances congestion against its own charging urgency.Source: §1 · p. 3
- Controllers use NEAT or PPO in per-candidate or joint architectures, and are evaluated in a simulator built on real-city networks with modeled charging physics and station congestion.Training uses four cities; sixteen additional cities are held out for benchmarking against an omniscient Oracle.Source: §1 · pp. 3–4
- PPO-joint, the strongest generalist controller, completes 98.6% of delivery shifts on time across the 20-city benchmark zero-shot, statistically close to the omniscient Oracle reference at 99.5%.Source: §1 · p. 4
- Table 1 contrasts coordination approaches by decision timing, rule source, and coordination information; this work addresses all three charging decisions using a public signal on real road networks.The table describes most prior approaches as targeting cost, grid load, or travel time, while this work targets on-time service with batteries above a safety floor.Source: Table 1 · p. 5
2. Related Work
Related work spans offline optimization and online rules, controllers, and learned policies, with differing assumptions about what decisions they make and what information they use. The paper positions its approach as online learned control using a public signal without central control or vehicle-to-vehicle messaging.
Offline optimization
- Offline EVRP optimization jointly plans routes and charging, whereas this work treats delivery routes as fixed and charging stops as detours.The paper motivates this setting by unpredictable queues, delays, traffic, and station occupancy.Source: §2 · pp. 5–6
Online non-learned methods
- Online non-learned rules include local SoC thresholds, while MPC and ADP centrally optimize from fleet-wide state and re-plan for the fleet.The paper keeps a hand-designed threshold heuristic as a baseline but removes the central optimizer from its own setting.Source: §2 · p. 6
Online reinforcement learning
- Prior reinforcement-learning studies often separate timing, station selection, and energy targets, and evaluate simplified settings or metrics other than fleet on-time service.Some pin charging to a home charger or single station; distributed multi-agent controllers may instead rely on vehicles exchanging messages.Source: §2 · p. 6
Signal-only coordination
- The paper places its approach in the online, learned, signal-only category and, to its authors’ knowledge, identifies it as the first of its kind.It relates this approach to decentralized agents reacting to shared signals without central control or vehicle messaging.Source: §2 · p. 7
3. The Operational Problem
The operational objective is to complete delivery shifts without violating the battery safety floor or the shift time budget. Vehicles make local charging decisions on fixed routes, while charging physics and shared-station queues shape the choices available to them.
3. The Operational Problem
- A shift succeeds when a vehicle completes its route within the shift and returns to the depot without its battery falling below the safety floor.The objective is to maximize the fraction of vehicles that succeed; decisions occur at route stops before departure to the next stop.Source: §3 · p. 7
3.1. Charging physics
- Charging power is limited by both station power and the vehicle’s battery-management cap, then tapers above 80% SoC under the CC/CV profile.The slow CV phase makes the marginal time cost of charging rise near full charge, so target SoC affects the time budget.Source: §3.1 · p. 8
3.2. Station congestion
- Stations may be shared with outside vehicles, and each vehicle observes only public connector occupancy, not the queue behind busy connectors.When all connectors are busy, an arriving vehicle must wait; external demand and queue dynamics vary by scenario.Source: §3.2 · p. 8
3.3. Formal model
- The problem is a Dec-POMDP: vehicles are coupled through station queues, and each vehicle acts from its own local state and nearby public occupancies.Actions include whether to charge, which station to choose, and the target SoC; the team reward operationalizes the fleet success-ratio objective.Source: §3.3 · pp. 8–9
4. Agent Design
The agent uses a shared policy to choose whether and where to charge and, when charging, how much. Its signals encode vehicle urgency alongside station detours, charging opportunities, and public occupancy.
4. Agent Design
- The shared policy decomposes charging into station selection and target-SoC choice, matching two coupled decisions with separate preference-scoring and charging-amount components.The action pairs a candidate station or no-charge option with a target SoC in 0.05 steps from 0 to 100%.Source: §4 · pp. 9–10
4.1. Decision Signals
- For each candidate, the four-signal block combines vehicle margins with detour and expected SoC gain, which reflects nonlinear charging and anticipated queueing.The queue estimate uses station EVSE count and publicly broadcast occupancy; the gain signal can value a free slower station above a congested fast one.Source: §4.1 · pp. 10–11
- Decision-oriented signals encode time and energy margins so the policy can learn station selection without rediscovering established queueing and charging dynamics.The time margin measures time to deadline; the energy margin measures driving time before reaching the safety floor.Source: §4.1 · pp. 10, 12
4.2. Per-candidate policy
- The per-candidate policy scores each station and a synthetic no-charge option, then assigns a target SoC if a real station wins.Because no-charge competes directly with stations, the policy learns when charging is preferable based on urgency, detour, and congestion rather than a hand-tuned threshold.Source: §4.2 · p. 12
4.3. Joint policy
- The joint policy scores candidates together in one network pass, with a dedicated null output; comparing it with per-candidate scoring tests the effect of parameter sharing and permutation invariance.The joint form outputs |Ci| + 1 station scores and a target-SoC output.Source: §4.3 · pp. 12–13
4.4. Reward
- The team reward averages vehicle outcomes per episode: route completion earns +1, while battery-floor failure or timeout earns −1 with a small progress credit.Training and evaluation average the reward over J independent episodes.Source: §4.4 · p. 13
5. Learning Algorithms and Reference Controllers
This section compares NEAT and PPO policy learners with a congestion-blind threshold controller, an offline Oracle, and simple rule-based baselines. PPO uses a specialized training setup and is reported as the strongest controller.
5.1. NEAT (Neuroevolution)
- NEAT evolves neural networks that map charging signals to candidate scores and target SoC, with either fixed-size per-candidate or candidate-scaled joint policies.Networks grow through mutations, while selection and speciation preserve fitter networks and protect new structures; episode reward supplies one fitness value without decision-level credit assignment.Source: §5.1 · p. 13
5.2. PPO (Proximal Policy Optimization)
- PPO updates policies in small clipped steps, using generalized advantage estimation from a learned value function.It uses the same observation and action space as NEAT, with station choice and target SoC drawn from categorical distributions.Source: §5.2 · p. 14
- PPO’s shared fleet policy is trained through one ego-learner per episode, while the other vehicles follow frozen past-policy checkpoints.This yields a clean single-agent trajectory against a stationary but varied fleet; behavioral cloning from the greedy rule is also needed for PPO to train reliably.Source: §5.2 · pp. 14–15
- Despite its heavier training setup, PPO proves the strongest controller among those evaluated.Its setup requires a single-agent reduction, behavioral-cloning warm-start, and historical policy pool; NEAT runs the same policy search without them.Source: §5.2 · p. 15
5.3. A congestion-blind control: the SA-tuned rule
- The SA-tuned rule charges by thresholds on vehicle urgency and SoC, then takes the nearest station without using occupancy or detour signals.Dropping those signals bounds their combined value; the gap to learned controllers opens where congestion bites.Source: §5.3 · pp. 15–16
5.4. The Oracle
- The Oracle searches offline for a charging plan using full episode information, providing a planning reference rather than a deployable policy.Its plans specify each charge’s route leg, station, and target SoC; simulated annealing begins from the best reactive controller and polishes the plan.Source: §5.4 · p. 16
5.5. Rule-based baselines
- The greedy baseline charges to 100% at the nearest station below 20% SoC, while the null baseline never charges.Charging badly can make every vehicle time out, which scores the minimum reward rather than the null policy’s outcome.Source: §5.5 · pp. 16–17
6. The Evaluation Environment
The evaluation environment simulates delivery shifts on real urban road networks, with shared charging stations, external demand, and routes generated under sampled time budgets. A discrete-event engine executes those shifts, while a calibrated statistical surrogate makes routing queries practical at training scale.
6. The Evaluation Environment
- Each city’s drivable OpenStreetMap graph hosts delivery points, stations, and depots; fresh fleet and demand draws vary the scenario between shifts.Delivery points are sampled over graph nodes, concentrating them where the road network is denser; benchmark evaluation freezes one draw per city.Source: §6 · p. 17, §6.1 · p. 17
6.1. Scenario, Fleet and External Demand
- Stations share connectors with external users, whose arrivals, service durations, and FIFO queues create congestion that fleet vehicles experience.The simulator tracks station occupancy and its rolling mean as the congestion input to the controller.Source: §6.1 · pp. 17–18
6.2. Route Generation
- Routes grow from a home station by alternating local and longer-range delivery stops until the estimated route, service, return, and reserve time exceeds the shift budget.Time budget, long-range jump probability and radius, and service-time mean are resampled per route.Source: §6.2 · pp. 18–19
6.3. Discrete-Event Simulation Engine
- A discrete-event engine tracks vehicle and station state, advancing shifts through route-leg, charging-arrival, and periodic station-update events.A shift ends in on-time completion, battery depletion, or timeout, and the vehicle reward contributes to the episode reward.Source: §6.3 · p. 19
6.4. Statistical Routing Model
- The per-city routing surrogate replaces costly graph queries with deterministic distance-and-time estimates from distance-binned circuity and speed averages.It answers queries in about 1 µs versus roughly 90 ms for one Los Angeles shortest-path query; across a full route, mean distance and time errors are 7.6% and 9.3%, respectively.Source: §6.4 · pp. 19–20, Table 4 · p. 20, Figure 4 · p. 21
- Because bin means need not satisfy the triangle inequality, the detour signal clips negative values to zero.The same routing surrogate is used by controllers and the Oracle under common random numbers, so its residual error is shared in their comparison.Source: §6.4 · p. 20
7. Experiments and Results
The experiments train controllers on four cities and test them zero-shot across twenty, including sixteen cities not used for training. They also examine transfer to a high-capacity, larger-van regime beyond the training conditions.
7. Experiments and Results
- Controllers are trained on four cities and evaluated zero-shot on those cities plus sixteen additional cities across five continents.Training uses randomized base-scenario cities; evaluation freezes one scenario per city.Source: §7 · p. 21
- The same cities are re-evaluated in a high-capacity, larger-battery-van regime with rising fleet-to-charger density.This is an out-of-distribution but realistic configuration used to test transfer beyond trained conditions.Source: §7 · p. 21
- The evaluation asks what learned controllers add when only some vehicles need charging, but those vehicles may face sharp charger contention..Source: §7 · p. 21
7.1. The base scenario
The base benchmark uses commercially grounded delivery scenarios with varied energy deficits, charger contention, and time reserves. Controllers are trained through domain randomization and evaluated on frozen, Oracle-solvable city scenarios.
7.1.1. Scenario setup
- 50.7% of vehicles must charge in the base regime, compared with 29.7% on the same routes in the high-capacity regime.Scenarios also vary in charger power per EV and time reserve, producing different forms of difficulty.Source: §7.1.1 · p. 22, Figure 5 · p. 23
7.1.2. Benchmark scenarios
- Benchmark scenarios are frozen per city and selected only when the Oracle completes 99–100% of shifts.The 20-city set varies in station count, fleet size, and reserve fraction; the scenarios are held out from training.Source: §7.1.2 · p. 23
7.1.3. Controllers training
- Four learned controllers are trained from scratch by domain randomization on four base-scenario cities, with a comparable budget of about 300 CPU-hours each.PPO reaches its level using a single-agent reduction, behavioral-cloning warm start, and historical policy pool; the SA-tuned rule is fit separately.Source: §7.1.3 · pp. 23–24
7.1.4. Scenario evaluation
- Evaluation uses up to 50 episodes per frozen city configuration, with routes and initial charge varying between episodes while controllers face identical conditions.The Oracle is limited to 20 episodes per city because its per-episode planning is computationally heavy.Source: §7.1.4 · p. 24
7.2. Results on the base benchmark
Across the base benchmark, PPO reaches near-Oracle service and outperforms NEAT on average. Reading station occupancy improves reward and reduces queue waits, while learned controllers generally charge below a full battery and less often than the Oracle.
7.2. Results on the base benchmark
- +0.968 average reward for PPO across both structures, versus NEAT’s +0.860; PPO-joint leads on average with a 0.019 gap to the Oracle.PPO is the best reactive controller in 16 of 20 cities; PPO-per-candidate wins the most individual cities, 11 of 20.Source: §7.2 · p. 25
- +0.139 reward from reading occupancy over the congestion-blind SA-tuned rule, with per-session queue waits of about 2 minutes versus 9.5 for that rule and 45 for greedy.The reward gain is positive in all 20 cities and reaches +0.324 in Chongqing; the best reactive controller in each city completes at least 98.6% of shifts on time, compared with the Oracle’s 99.5%.Source: §7.2 · p. 25
- PPO-joint averages 0.97 charging sessions per vehicle, below the Oracle’s 1.93, with learned-controller exit SoC between 57% and 88%.Learned controllers cover about 156–161 km per vehicle-day, but that distance is biased by completion because controllers that fail to finish drive less.Source: §7.2 · p. 25
7.3. Policy interpretability
The controllers use urgency and station occupancy to learn when to charge and which stations to prefer. Their behavior includes rerouting around congestion, subdividing charges, and charging partially, though joint coordination ties the policy to a fixed number of candidate stations.
7.3. Policy interpretability
- The learned policies respond to low energy margins, penalize detours, and prefer freer stations when their occupancy-related gain is higher.These behaviors are not explicitly enforced.Source: §7.3 · p. 25
- Joint scoring lowers a station’s score when a rival has free capacity, letting the controller bypass congestion for a farther free station.The effect steepens from about −3 on average to −6 to −12 in contended states.Source: §7.3 · pp. 25–26
- In Manila, PPO-per-candidate and NEAT-per-candidate average 6.7 and 19.6 charging sessions per vehicle, versus 1.2 for PPO-joint.NEAT-per-candidate’s narrow scores lose discrimination outside its training range, while NEAT-joint’s cross-candidate coupling collapses to zero.Source: §7.3 · p. 26
- The joint policy is tied to a fixed number of candidate stations and must be rebuilt if that number changes.Per-candidate scoring is invariant to the candidate-set size.Source: §7.3 · p. 26
- In Cairo, PPO-joint and the planner subdivide charging into short top-ups, while greedy and SA-tuned each take one large charge and miss the shift.In Manila, PPO-joint, SA-tuned, and the Oracle stop between 0.45 and 0.85 SoC; greedy charges fully and overruns the shift.Source: §7.3 · p. 27
- When a free Jakarta station broadcasts saturated occupancy, its charging share falls from 33% to zero and its load shifts to neighboring stations.Fleet reward changes from 1.00 to 0.98.Source: §7.3 · p. 27
7.4. Beyond the training regime: capacity and density
The study tests zero-shot transfer to a high-capacity regime and sweeps demand density beyond the training range. PPO transfers with the least loss, while the learned controllers’ advantage over the tuned rule does not hold in every off-distribution city.
7.4. Beyond the training regime: capacity and density
- Across the density sweep, PPO, NEAT-joint, and the SA-tuned rule hold in both capacity regimes, while NEAT-per-candidate collapses and turns negative in the base regime.The sweep spans 5 to 50 EVs per station; densities above 15 are extrapolation without retraining.Source: Figure 11 · p. 29
- At density 25 in the high-capacity regime, PPO transfers with the least loss, SA stays close behind, NEAT-joint holds, and NEAT-per-candidate loses the most.The high-capacity regime uses large vans and is never seen during training; PPO-joint charges about 0.51 sessions per vehicle there.Source: §7.4 · pp. 27, 29, Table 8 · p. 33
- In the high-capacity regime, only NEAT-per-candidate collapses outside training, charging on nearly every decision and causing timeouts or depletions.Its failure is specific to its small genome, not to per-candidate scoring generally.Source: §7.4 · p. 29
- The learned controllers’ advantage over SA is limited to the base scenario: at density 25, unchanged learners fall behind SA in Cairo, Paris, Tokyo, and Toronto.There, the tuned threshold rule is the best reactive controller.Source: §7.4 · pp. 29–30
8. Conclusions
The paper concludes that shared local controllers can coordinate fleet charging from vehicle state and public station occupancy, without reservations, messaging, or a dispatcher. PPO-joint approaches the Oracle across the benchmark, while the model leaves several operational factors unrepresented.
8. Conclusions
- PPO-joint comes within a mean reward gap of 0.019 of the omniscient Oracle across twenty cities, transfers zero-shot to sixteen held-out cities, and charges about half as often as the Oracle.Neuroevolution and the tuned threshold rule perform at a lower level; the advantage over fixed thresholds appears in congested cities.Source: §8 · p. 34
- The shared local interface reads each vehicle’s state and public per-station occupancy, requiring no reservations, messaging, or central dispatcher.Per-connector charger status is already available through OCPP and can be broadcast through MQTT; decisions need only a few microseconds on a CPU.Source: §8 · p. 34
- The congestion model assumes stationary within-shift arrivals and does not model charging costs or battery degradation.The authors identify online adaptation and driver rest breaks as possible extensions.Source: §8 · p. 34