PaperlayerView exact revision on arXiv

Source-linked AI summary

Sharing the Control Authority Between Deep Reinforcement Learning and Model Predictive Control: Application to Multi-Class Transportation Networks

Giray Onur, Azita Dabiri, Bart De Schutter

arXiv:2608.20858v1eess.SYcs.LG

TL;DR

Large-scale multi-class transportation networks challenge DRL through limited learning capacity and MPC through computation and model dependence, complicating real-time control. The paper proposes a hierarchical DRL-MPC framework that assigns low-frequency inputs to MPC and high-frequency inputs to DRL. On a multi-class freeway network with model mismatch and noisy demands, it reports stronger constraint handling than the state-feedback-MPC baseline and much faster online computation than hierarchical MPC.

  • Problem

    Large-scale multi-class transportation networks create DRL learning limitations, MPC computation costs, and dependence on accurate models, challenging real-time control.

  • Method

    The hierarchical DRL-MPC framework assigns low-frequency control inputs to MPC and high-frequency control inputs to DRL.

  • Results

    The framework offers a favorable trade-off across efficiency, constraint handling, smoothness, and computation, and is more resilient to prediction-model mismatch than hierarchical MPC and state-feedback-MPC frameworks.

  • Takeaways & Limitations

    Dividing control authority combines MPC’s optimization and constraint handling with DRL’s model independence and fast online computation for multi-class freeway control.

  • Takeaways & Limitations

    Multi-frequency control remains constrained by the need to compute high-frequency inputs sufficiently quickly for real-time operation, especially when using computationally demanding MPC.

Abstract

from arXiv · show

Transportation networks, in particular multi-class transportation networks (i.e., networks with mixed vehicle types), are complex systems that are challenging to control. Recently, Deep Reinforcement Learning (DRL), which learns control policies from interactions with the environment, and Model Predictive Control (MPC), which uses a system model to optimize control inputs, have been increasingly utilized for transportation network control. However, nonlinear system dynamics and high-dimensional state spaces in large-scale networks limit DRL's learning capacity under time-constrained training and increase MPC's computation time, hindering real-time implementation with limited computational resources. Moreover, MPC depends on an accurate network model, which is often unavailable for complex systems such as multi-class transportation networks. This paper proposes a novel DRL-MPC framework for multi-class transportation networks that divides control authority between DRL and MPC, combining DRL's fast online computation and model independence with MPC's built-in optimization and constraint-handling capabilities. In the hierarchical framework, MPC operates at the higher level and determines low-frequency control inputs whose slower update rate accommodates its high computation time, while DRL operates at the lower level and determines high-frequency control inputs using its fast online deployment. The framework is evaluated on a multi-class freeway network against a hierarchical MPC controller and a hybrid state-feedback-MPC controller, including scenarios with model mismatch and noisy traffic demands. Results show that the proposed framework outperforms the hybrid state-feedback-MPC controller, substantially reduces online computation time compared with the hierarchical MPC controller, and provides more effective constraint enforcement under model mismatch.

1. Introduction

Transportation-network control increasingly combines MPC’s optimization and constraint handling with DRL’s model-free learning, but computational demands and multi-frequency real-time requirements remain challenging. The paper proposes a hierarchical DRL-MPC framework that assigns low-frequency inputs to MPC and high-frequency inputs to DRL, evaluating it on a multi-class freeway network under model mismatch and noisy demands.

  • Motivation: Traffic congestion harms mobility and creates environmental, social, and economic costs, motivating advanced transportation-network control methods.Reported impacts include pollution, fuel consumption, noise, accidents, and longer travel times.
  • Existing approaches: MPC optimizes long-term transportation objectives while explicitly handling system constraints, but computational demands can increase with complex optimization procedures.MPC predicts future system evolution, solves a finite-horizon optimization problem, and applies the first computed control input.
  • Existing approaches: DRL learns policies directly from environmental interactions without an explicit system model, making it appealing when model knowledge is limited or dynamics are uncertain.The paper cites applications including ramp metering and variable speed-limit control.
  • Hybrid control: Existing hybrid DRL-MPC methods commonly use MPC as a safety filter or DRL to modify MPC models, inputs, or parameters.These strategies seek to combine model-free learning with MPC-based constraint satisfaction or compensate for prediction-model inaccuracies.
  • Research gap: High-frequency inputs must be computed quickly for real-time control, which is difficult for computationally demanding MPC in large-scale transportation networks.Prior multi-frequency frameworks assign different transportation-control inputs to different update frequencies to limit recomputation burden.
  • Contribution: The proposed framework assigns low-frequency control measures to MPC and high-frequency measures to DRL, and evaluates this division on a multi-class freeway network under noisy demands and model mismatch.The reported evaluation compares the framework with hierarchical MPC and examines constraint handling and online computation.

2. Related work

Prior transportation-control research establishes MPC and DRL as useful but complementary approaches, while hybrid methods largely adapt MPC through DRL in single-class networks. The paper identifies multi-class, multi-frequency DRL-MPC control as an underexplored direction amid persistent computational, modeling, training, and constraint-handling challenges.

  • MPC for transportation network control: MPC has been applied to freeway and urban networks to optimize traffic objectives while accounting for system dynamics and constraints.Applications include variable speed limits, ramp metering, route guidance, travel delays, and emissions.
  • MPC for transportation network control: Centralized MPC can become computationally demanding in large-scale networks, while model simplification trades computational efficiency against prediction accuracy.Distributed MPC reduces computational burden but remains dependent on accurate prediction models.
  • DRL for transportation network control: DRL learns transportation-control policies from experience without explicit environment models and has been applied to coordinate multiple freeway control measures.The cited work includes actor-critic DRL for ramp metering and variable speed-limit coordination.
  • DRL for transportation network control: DRL-based approaches face long training times, sample inefficiency, and limited mechanisms for guaranteeing state-constraint satisfaction.The paper frames data efficiency, constraint awareness, and near-optimal performance as continuing research needs.
  • Combined DRL-MPC approaches: Existing hybrid frameworks use DRL to adjust MPC outputs, prediction models, objectives, constraints, control laws, or optimization settings.Examples include hierarchical frameworks in which MPC supplies low-frequency inputs and DRL modifies control at high frequency.
  • Research gap: Hybrid DRL-MPC research has focused largely on single-class networks, leaving multi-class and different-frequency control settings underexplored.The paper identifies this scope as an opportunity for further research.

3. The DRL-MPC framework

The hierarchical DRL-MPC framework assigns high-frequency control inputs to a low-level DRL agent and low-frequency inputs to a high-level MPC controller. This division lets DRL deploy quickly while giving MPC time to optimize predictions and constraints.

  • Framework structure: The framework assigns high-frequency control inputs to the low-level DRL agent and low-frequency inputs to the high-level MPC controller.The hierarchy divides control authority between the two controllers according to their update frequencies.
  • Framework structure: The MPC controller uses the trained DRL policy to predict future network states even though DRL determines the lower-level inputs.This incorporates the fixed DRL policy into MPC’s prediction over its horizon.
  • Low-level DRL agent: The DRL agent receives network states and normalized traffic demands, then computes high-frequency inputs through its policy at intervals of length T_l.The state variables include normalized network quantities such as densities, outflows, velocities, and current traffic demands.
  • Low-level DRL agent: Each DRL control input is held over its low-level control interval before the next high-frequency update.The applied input remains constant across the corresponding network sampling steps.
  • High-level MPC controller: The MPC controller computes low-frequency inputs over a prediction horizon using a receding-horizon optimization strategy.It predicts network states at network sampling steps while updating control inputs only at high-level steps, then applies the first input and repeats.
  • High-level MPC controller: Choosing m_h as an integer multiple of m_l synchronizes MPC and DRL control updates, while m_h = m_l removes the hierarchical frequency distinction.The high-level interval satisfies T_h = m_hT and the low-level and high-level counters align through this integer relationship.
  • High-level MPC controller: The MPC formulation is nonlinear and nonconvex for the multi-class METANET model, requiring a nonlinear optimization solver.This computational structure motivates assigning MPC to lower-frequency control updates.

4. Training the DRL-MPC framework

The DRL component is trained within the hierarchical DRL-MPC architecture using experience collected while both controllers operate together. DDPG and SAC use replay-based off-policy learning and a reward aligned with MPC’s objective while penalizing state-constraint violations.

  • Training procedure: The DRL agent is trained on samples collected while it operates with the high-level MPC controller, so its learned policy accounts for MPC’s effects.Collected samples update the DRL policy within the integrated architecture rather than in isolation.
  • Training algorithms: DDPG and SAC are used as off-policy actor-critic algorithms, with DDPG learning a deterministic policy and SAC learning a stochastic policy.Their shared off-policy structure permits reuse of past experience tuples stored in a replay buffer.
  • Training algorithms: Replay-buffer sampling improves sample efficiency by reusing past experience, which is useful because MPC computation increases the time required to collect each training sample.Mini-batches are sampled from stored transitions for policy and value updates.
  • Reward design: The shared reward combines TTS, low-level control-input fluctuation penalties, and positive penalties for violating state constraints.The reward is the negative of these objective and penalty terms, aligning DRL reward maximization with MPC cost minimization.
  • Training procedure: The overall training procedure is summarized in Algorithm 1, with detailed DDPG and SAC formulations provided in Appendix B.

5. Case study

The case study evaluates shared DRL-MPC control on a benchmark multi-class freeway under noisy demands and prediction-model mismatch. MPC controls low-frequency vehicle splitting, while DRL controls high-frequency ramp metering, with comparisons against hierarchical MPC, state-feedback-MPC, and no control.

  • Experimental setup: The benchmark network contains two vehicle classes and uses one low-frequency vehicle splitting rate plus two high-frequency ramp metering rates.The vehicle splitting rate is assigned to MPC, while ramp metering rates are assigned to the DRL agent.
  • Experimental setup: Noisy demands are generated by adding zero-mean Gaussian noise with origin- and class-specific standard deviations, then applying a third-order low-pass Butterworth filter.The filter uses a normalized cutoff frequency of 0.1.
  • Experimental setup: Four scenarios combine nominal or noisy demands with original or perturbed MPC prediction-model parameters.The scenarios cover demand variability, model mismatch, and their combination.
  • Controllers and evaluation: All controllers use the same network state measurements and identical control-input bounds, while evaluations compare DRL-MPC, hierarchical MPC, state-feedback-MPC, and no control.The proposed framework is evaluated under the same freeway-network conditions as the comparison controllers.
  • Controller implementation: The MPC controller uses a five-initialization multi-start SQP strategy with optimality, step, and constraint tolerances set to 10^-2.This setup balances control performance and online computation time.
  • Controller implementation: During deployment, MPC enforces queue-length limits as hard constraints, whereas training uses a soft violation penalty to avoid early infeasibility.The soft penalty accommodates early training stages when the DRL policy may not yet regulate ramp metering effectively.

5.3. Training results

Across the four scenarios, the trained agents converged within the training budget, while deployment results show DDPG-MPC offered a favorable balance of performance, constraint handling, and computation time.

  • Training results: All DDPG and SAC agents converged within the training budget across all four scenarios, with narrowing across-seed min–max ranges.The plotted training returns were smoothed and retained the reward scaling used during training; they do not represent converged-policy deployment performance.
  • Deployment results: Under accurate prediction models, hierarchical MPC achieved the lowest median TTS, while DDPG-MPC, SAC-MPC, and SF-MPC reduced TTS by about 5% relative to no control.The hierarchical MPC reduction was about 10% in these scenarios.
  • Deployment results: Under model mismatch, hierarchical MPC’s median TTS reduction shrank to about 4%, while DDPG-MPC and SAC-MPC remained competitive in median TTS.SAC-MPC trailed DDPG-MPC by a small margin, and DDPG-MPC’s TTS spread widened noticeably in Scenario 4.
  • Deployment results: Under model mismatch, DDPG-MPC kept median total and maximum queue length constraint violations small, unlike hierarchical MPC and SF-MPC.SAC-MPC incurred larger median total and maximum violations than DDPG-MPC in both mismatch scenarios.
  • Deployment results: DDPG-MPC reduced online computation time by approximately thirtyfold relative to hierarchical MPC, with both DRL-MPC variants requiring computation times in the order of a minute.Hierarchical MPC required around 28 minutes across the four scenarios, whereas SF-MPC was fastest.
  • Deployment results: DDPG-MPC provided the most favorable balance among the proposed DRL-MPC variants and considered baselines across TTS, constraint violations, and computation time.SAC-MPC produced smoother control trajectories and lower training-seed variability but incurred larger queue violations and higher SOC values than DDPG-MPC.

6. Conclusions

The proposed DRL-MPC framework divides control inputs across hierarchical MPC and DRL layers to combine their complementary strengths in multi-class transportation networks. It achieves favorable performance across efficiency, constraints, smoothness, computation, and model-mismatch resilience, while its broader applicability remains to be evaluated.

  • Low-frequency MPC inputs and high-frequency DRL inputs combine MPC’s optimization and constraint handling with DRL’s fast computation and model independence.The framework applies MPC at the higher level and DRL at the lower level.
  • Across scenarios with prediction-model mismatch and noisy demands, the framework offers a favorable trade-off among traffic efficiency, constraint handling, input smoothness, and online computation time.
  • The framework is more resilient to prediction-model mismatch than hierarchical MPC and state-feedback-MPC frameworks.
  • The framework could be adapted to robotic motion control and power-electronic converter control, but these applications remain future evaluation areas.

Appendix A. Multi-class METANET model

The multi-class METANET model represents freeway networks as links and nodes, describing each segment with class-specific traffic variables and modeling flows, densities, speeds, splitting, and queues. In this work, the model is used both to simulate the real multi-class freeway network and as the MPC prediction model.

  • The multi-class METANET model is used both to simulate the real multi-class freeway network and as the MPC prediction model.
  • The multi-class METANET model represents freeway links and nodes, with each segment described by class-specific density, mean speed, and outflow.Links represent approximately uniform freeway stretches, while nodes represent junctions, ramps, or major geometric changes.
  • Class-specific density, speed, outflow, traffic-volume fraction, and segment dynamics are computed using vehicle lengths, lane counts, sampling intervals, and segment lengths.
  • The model includes density-dependent speed relationships, downstream-density effects, and speed reductions caused by merging phenomena.
  • At network nodes, entering class-specific flow is distributed among leaving links using per-class splitting rates that sum to one.
  • Origin queues are updated from class-specific demand and outflow, while on-ramp outflow depends on capacity, ramp metering rate, and downstream density conditions.

Appendix B. Training algorithms

Appendix B describes the DDPG and SAC actor-critic algorithms used to train the DRL agent in the DRL-MPC framework, including their training procedure, architectures, and hyperparameters.

  • The appendix details DDPG and SAC training algorithms, the overall training procedure, network architectures, and case-study hyperparameters.

Appendix B.1. Training the framework using DDPG

The DDPG training procedure uses actor-critic networks, replay-buffer learning, target networks, exploration noise, and n-step temporal-difference targets. Training alternates high-level MPC and low-level DRL actions during environment interaction before updating the networks from sampled transitions.

  • DDPG uses an actor to map observed states to continuous DRL actions and a critic to estimate discounted returns for state-action pairs.Target networks are used to improve learning stability.
  • The n-step temporal-difference target represents estimated discounted rewards over multiple future steps and improves learning efficiency and action quality in delayed transportation systems.
  • Critic parameters minimize a loss, actor parameters maximize estimated return, and target networks are updated using Polyak averaging.
  • Exploration adds temporally correlated Ornstein-Uhlenbeck noise to DRL actions, which is suited to control settings with inertia-like dynamics.
  • The training algorithm updates MPC actions at the high-level frequency and samples DRL actions at the low-level frequency during environment interaction.
  • Transitions are stored in a replay buffer, and critic and actor networks are updated from randomly sampled mini-batches after sufficient data have been collected.

Appendix B.2. Training the framework using SAC

The framework trains its lower-level controller with Soft Actor-Critic, combining entropy-regularized return optimization with twin-critic learning. The procedure includes target-value computation, adaptive entropy weighting, critic and actor updates, and Polyak-averaged target critics.

  • SAC training: SAC learns a stochastic policy for continuous actions by jointly optimizing expected discounted return and policy entropy.Higher entropy encourages broader exploration of the action space.
  • Target computation: The target value is computed with an n-step temporal-difference method using the target critic and an action sampled from the policy.The supplied training description identifies the target computation as analogous to DDPG and references equation (B.1).
  • Entropy adaptation: The entropy loss weight is adapted using a target entropy, with values closer to zero promoting a more stochastic policy.The target entropy is denoted H < 0.
  • Network updates: Each critic is updated by minimizing its loss, after which the actor maximizes the entropy-regularized return based on the minimum value from the two critics.The actor objective combines the minimum estimated Q-value with policy entropy weighted by α.
  • Target-network updates: The target critics are updated with Polyak averaging during the SAC training procedure.The training procedure is summarized in Algorithm 1, while the supplied passage introduces the Polyak update step.

Appendix B.3. Network architectures and hyperparameters

The appendix specifies separate DDPG and SAC actor-critic architectures and shared training settings. Both agents use bounded continuous actions, replay-based learning, n-step targets, and periodically updated target networks.

  • DDPG architecture: The DDPG actor uses two 256-unit ReLU layers, a tanh output, and affine scaling to bound ramp-metering actions to [0, 1].The DDPG critic has separate state and action paths with widths 256 and 128.
  • SAC architecture: The SAC agent uses twin critics matching the DDPG critic architecture and an actor with shared 256-unit features plus separate mean and standard-deviation heads.The standard-deviation output uses softplus, and sampled actions are mapped to the action bounds through tanh and affine scaling.
  • Shared hyperparameters: Each agent is trained for 3500 episodes with a replay buffer of length 2 · 10^5, batch size N = 512, γ = 0.99, τ = 10^-2, and n = 10.These settings are stated as shared DDPG and SAC training hyperparameters.
  • Exploration and entropy: DDPG uses Ornstein-Uhlenbeck exploration, while SAC initializes α = 1 and tunes entropy using target H = −2.The target entropy is set as the negative of the action dimension.
  • Training schedule: Learning begins after 512 replay samples, uses at most 100 mini-batches per iteration, and applies Polyak updates every 10 critic updates.The update schedule applies to the target networks specified for DDPG and SAC.

Appendix C. Tuned PI-ALINEA parameters

The appendix reports PI-ALINEA parameters tuned separately for four scenarios. Each scenario has five independently tuned parameter sets, with a per-scenario mean shown for comparison.

  • Parameter tuning: PI-ALINEA parameters were obtained from five independent Bayesian optimization runs for each scenario.The tuning runs are described in Section 5.2.3.
  • Figure interpretation: Figure C.12 compares the five tuned parameter sets across four scenarios and marks each scenario’s mean with a solid horizontal tick.Each marker represents one independently tuned parameter set.
Loading 2608.20858v1…