Source-linked AI summary
Learning to Incentivise: Using Reinforcement Learning for Sustainable Urban Mobility
Germán Pardo-González, Shaghayegh Vosough, Katerina Papadaki, Claudio Roncoli
TL;DR
Incentive schemes can encourage drivers to choose socially beneficial routes, but budgets constrain their allocation. This paper uses multi-agent reinforcement learning to assign dynamic incentives and jointly consider travel time and emissions; it reports a trade-off between the objectives and identifies weights that can outperform the dynamic equilibrium baseline on both.
Problem
Incentive schemes must allocate limited funds efficiently to encourage routes that may be longer for drivers but offer greater environmental and system-wide benefits.
Source: §I · p. 1Method
The study formulates dynamic incentive allocation as an MDP and uses Independent Q-learning with each origin-destination traveller as an agent.
Source: §II · p. 2Results
Minimising emissions lowers emissions but can raise TTT, while suitable objective weights can outperform the dynamic UE baseline on both emissions and TTT.
Source: B. Numerical results · pp. 5–6Takeaways & Limitations
Dynamic incentives provide a proposed alternative to road pricing that encourages drivers to reroute for broader social benefits.
Source: §I · p. 1, §IV · p. 6The study does not model whether drivers comply with incentivised routes, though a compliance rate could represent this behavior.
Source: §IV · p. 6
Abstract
from arXiv · showhide
Traffic management has traditionally focused on toll-based and road pricing solutions. However, road pricing often raises concerns about accessibility and public dissatisfaction, leading to its prohibition in some places, such as Finland. This study optimises the dynamic allocation of incentives to drivers, encouraging them to reroute onto alternative (potentially longer) paths to achieve greater societal benefit, reduced total travel time (TTT) and total emissions in the transportation network, contributing to sustainable urban mobility. We employ a multi-agent reinforcement learning approach to dynamically assign incentives to drivers to reduce both TTT and emissions, with travel times estimated using traffic simulation software. We demonstrate that, with an unlimited budget and an objective of minimising travel time, the incentive scheme reduces TTT by 16%, compared to the dynamic User-Equilibrium (UE) with a budget equivalent to about 11% of the UE total time. When the goal is to minimise emissions, a 9% reduction in CO2 emissions is observed under an unlimited budget. We demonstrate a critical trade-off: minimising TTT leads to an increase in emissions, while prioritising emission reductions raises TTT. However, with the right combination of weights in the multi-objective function, both TTT and total emissions are improved beyond the baseline.
I. INTRODUCTION
The study extends incentive allocation to dynamic traffic and jointly considers travel time and emissions. It evaluates incentive schemes against a dynamic equilibrium baseline and examines how budgets and objective weights affect network outcomes.
I. INTRODUCTION
- Incentive-based approaches promote voluntary travel changes as an alternative to congestion pricing, which can face fairness concerns and regulatory barriers.Road transport accounted for nearly 26% of the UK’s total emissions in 2021.Source: §I · p. 1
- Financial constraints make efficient incentive allocation important because real-world schemes must balance environmental benefits with smooth traffic movement.Drivers may accept longer, lower-emission routes when compensated appropriately.Source: §I · p. 1
- Prior work examined incentive effects under static traffic conditions, motivating mechanisms that account for dynamic traffic when reducing emissions.The study extends static link and path incentive work on total travel time to a dynamic setting and adds emissions analysis.Source: §I · pp. 1–2
- Earlier emissions-oriented toll approaches include categorical toll strategies and MARL tolls balancing route travel time with imposed tolls.The paper notes that categorical toll strategies may yield suboptimal solutions.Source: §I · p. 2
- The study combines dynamic incentives, SUMO-based emissions estimates, and a multi-objective reward to examine travel-time and emissions trade-offs against a dynamic equilibrium baseline.It uses the Helsinki Kamppi network and explores objective weights and different incentive budgets.Source: §I · p. 2
II. METHODOLOGY
The methodology formulates dynamic incentive allocation as an MDP and solves it using Independent Q-learning. Each traveller, defined by an origin-destination pair, acts as an agent, and emissions estimates are incorporated into a multi-objective reward.
II. METHODOLOGY
- The study formulates incentive allocation as an MDP using a universal modelling framework and extends prior work with emissions in a multi-objective reward.The model aims to determine each driver’s optimal incentive under a limited budget.Source: §II · p. 2
- Independent Q-learning solves the MDP, with each traveller defined by an origin-destination pair acting as an agent with its own Q-function.The algorithm is adapted for multi-agent environments.Source: §II · p. 2
- The methodology estimates CO2 emissions and incorporates them into the multi-objective reward alongside congestion.The approach combines MARL with microscopic traffic simulation.Source: §II · p. 2
A. Markov Decision Process Formulation
The formulation treats each traveller’s route choice as a one-step decision and uses incentive choices and weighted rewards to balance individual and network-wide travel time and emissions.
A. Markov Decision Process Formulation
- A traveller’s origin and destination define the one-step MDP state; after choosing a route, the traveller reaches the destination with no further decisions.The formulation therefore does not store a changing state along the trip.Source: A. Markov Decision Process Formulation · p. 2
- Each agent can incentivise at most one available path or none, with Yw marking incentivised paths and Xw recording the path actually taken.When no path is incentivised, the shortest path is used.Source: A. Markov Decision Process Formulation · p. 2
- The incentive reduces the selected path’s travel time by the difference from the minimum path time plus ϕ, making it ϕ units faster than the minimum.The path with minimum adjusted cost is then selected.Source: A. Markov Decision Process Formulation · pp. 2–3
- The authority tracks available budget B and spent budget b in time units, applying an incentive only when δ + b ≤ B.If funds are insufficient, travel times are unchanged and the original shortest path is selected.Source: A. Markov Decision Process Formulation · p. 3
- The reward combines weighted individual and network travel-time terms with weighted individual and total-emissions terms.Individual emissions are estimated from vehicle acceleration and velocity, and total emissions sum emissions across agents.Source: A. Markov Decision Process Formulation · p. 3
- Each agent has its own action-value function, which is iteratively updated across episodes using a learning rate that decreases toward convergence.Because the state remains unchanged, the Q-function represents action value without an explicit state.Source: A. Markov Decision Process Formulation · pp. 3–4
B. Solution Algorithm.
The algorithm initializes candidate routes and action values, then runs episodes that select routes under the incentive budget and update learned values from SUMO outcomes.
B. Solution Algorithm.
- The algorithm initializes k shortest paths and their free-flow travel times, sets each agent’s Q-values to zero, and chooses a small ϕ.These values seed the subsequent episodes.Source: B. Solution Algorithm · p. 4
- In each episode, agents sample actions, apply incentives only when δ + b ≤ B, and select the path with minimum adjusted travel time.If the budget is insufficient, the algorithm does not apply incentives.Source: B. Solution Algorithm · p. 4
- After all trips, SUMO supplies network TTT and agent travel times, which the algorithm uses to compute rewards and update each Q-function.The reward and Q-update are specified by Eqs. (1) and (6).Source: B. Solution Algorithm · p. 4
A. Case study
The study applies its methodology to the Kamppi area of Helsinki, a centrally located area known for peak-hour congestion.
A. Case study
- The case study uses the Kamppi area of Helsinki, Finland, with its network depicted in Fig. 1.The area is centrally located and known for congestion during peak hours.Source: A. Case study · p. 4, Figure 1 · p. 4
- Fig. 2 presents CO2 emissions and TTT under the TTT-minimising objective across different incentive budgets.Source: Figure 2 · p. 4
- The simulation network contains 235 edges, 152 nodes, and 1100 O-D pairs.The trips, departure times, and network definition are obtained from.Source: A. Case study · p. 4
B. Numerical results
The results show a trade-off between travel time and emissions, with performance affected by incentive budget and objective weights. The Pareto analysis indicates that suitable weights can improve both metrics over the dynamic UE baseline.
B. Numerical results
- When minimizing travel time, TTT outperforms the SUMO baseline; emissions are reduced with an unlimited budget and outperform baseline at every budget.Source: B. Numerical results · pp. 4–5
- With emissions-only optimization, emissions are lowest under an unlimited budget and outperform dynamic UE at every budget, while TTT may rise above baseline.The passage notes that TTT can increase even beyond the baseline when emissions are minimized.Source: B. Numerical results · p. 5, Figure 3 · p. 5
- With equal weights, emissions are significantly below dynamic UE but not as low as under travel-time optimization, while including emissions appears to degrade TTT.With an unlimited budget, the algorithm converges to a lower TTT; at a 50e3 budget, its final TTT appears slightly worse.Source: B. Numerical results · pp. 4–5, Figure 4 · p. 5
- For a 50e3 budget, equal weighting appears slightly worse for TTT, and the authors suggest favoring travel-time weights at small budgets to improve both metrics over baseline.The suggested weighting gives priority to travel time while retaining improvements in emissions and TTT relative to baseline.Source: B. Numerical results · p. 5
- Suitable objective weights can improve both emissions and TTT over dynamic UE, while the Pareto solutions deteriorate as incentive budgets decrease.The Pareto fronts show the trade-off between the objectives and indicate that TTT performance requires some budget allowance.Source: Figure 5 · p. 5, B. Numerical results · p. 6
IV. CONCLUSIONS
The study uses dynamic multi-agent reinforcement learning incentives to reduce emissions and congestion, and reports that prioritising objectives affects performance relative to the dynamic UE baseline. It does not model whether drivers comply with route incentives.
IV. CONCLUSIONS
- The dynamic incentive scheme targets both emissions and overall congestion, including through objectives that minimise either measure or combine them.The authors present it as a potentially more publicly acceptable alternative to road pricing.Source: §IV · p. 6
- The approach extends prior work by incorporating emissions into a multi-objective MARL algorithm and using SUMO to estimate travel times dynamically in simulated Helsinki traffic.The reported results indicate that the algorithms are reliable for an unlimited budget and identify objective priorities for outperforming the dynamic UE baseline.Source: §IV · p. 6
- The study does not model whether drivers follow incentivised routes, although a compliance rate could represent this behaviour.The authors also note that they used trip incentives rather than comparing path incentives.Source: §IV · p. 6