PaperlayerView exact revision on arXiv

Source-linked AI summary

Comparison of Deep Reinforcement Learning and Model Predictive Control for Adaptive Cruise Control

Yuan Lin, John McPhee, Nasser L. Azad

arXiv:1910.12047v3eess.SY

TL;DR

The paper asks how DRL and MPC compare for ACC car-following control under matched conditions, including modeling errors and out-of-range testing inputs. It trains a DDPG policy and solves IPO-based MPC using a first-order COM, then tests both controllers in simulation. Without modeling errors, DRL approaches sufficiently long-horizon MPC, while with significant modeling errors DRL generally has lower episode costs.

  • Problem

    The study addresses limited fair evidence comparing DRL and MPC for state-space optimal control, particularly under modeling errors.

  • Method

    The paper compares an offline DDPG policy with IPO-based MPC for ACC car-following control using a shared first-order COM and HFM testing.

  • Results

    Without modeling errors and within the training range, DRL is equivalent to sufficiently long-horizon MPC and its average episode cost is 5.8% higher than the IPO benchmark; with significant modeling errors, DRL has lower episode costs than MPC.

  • Takeaways & Limitations

    DRL can achieve near-optimal ACC control under familiar conditions and is more tolerant than MPC when the same modeling errors are significant.

Abstract

from arXiv · show

This study compares Deep Reinforcement Learning (DRL) and Model Predictive Control (MPC) for Adaptive Cruise Control (ACC) design in car-following scenarios. A first-order system is used as the Control-Oriented Model (COM) to approximate the acceleration command dynamics of a vehicle. Based on the equations of the control system and the multi-objective cost function, we train a DRL policy using Deep Deterministic Policy Gradient (DDPG) and solve the MPC problem via Interior-Point Optimization (IPO). Simulation results for the episode costs show that, when there are no modeling errors and the testing inputs are within the training data range, the DRL solution is equivalent to MPC with a sufficiently long prediction horizon. Particularly, the DRL episode cost is only 5.8% higher than the benchmark solution provided by optimizing the entire episode via IPO. The DRL control performance degrades when the testing inputs are outside the training data range, indicating inadequate generalization. When there are modeling errors due to control delays, disturbances, and/or testing with a High-Fidelity Model (HFM) of the vehicle, the DRL-trained policy performs better with large modeling errors while having similar performance as MPC when the modeling errors are small.

I. INTRODUCTION

The paper establishes a fair DRL–MPC comparison for ACC optimal control, examining optimality, generalization, and modeling errors under matched conditions. It develops controllers for a Prius PHEV using a simplified vehicle model and an HFM for testing.

  • I. INTRODUCTION: MPC repeatedly solves a constrained finite-horizon optimization online, applying only the first control value before updating the state.This receding-horizon procedure is contrasted with the learned DRL policy.
  • I. INTRODUCTION: The central questions are whether DRL or MPC achieves lower cost without modeling errors and which better handles delays, disturbances, and COM–HFM differences.The study also examines DRL performance when testing inputs leave the training-data range.
  • I. INTRODUCTION: The ACC controllers target a 2015 Toyota Prius power-split PHEV, using a first-order COM and an HFM that includes control delay and detailed powertrain elements.The HFM is based on Autonomie, while the COM does not include control delay.
  • I. INTRODUCTION: The scope excludes robust or stochastic MPC and transfer or meta-learning, and centers on a low-dimensional ACC task with three state variables.The authors note that these advanced methods could improve robustness or uncertainty handling.
  • I. INTRODUCTION: The study compares DDPG-based DRL with IPO-based MPC for ACC car-following control under matched control-oriented-model conditions.The comparison focuses on minimum episode cost without and with modeling errors.
  • I. INTRODUCTION: The paper's contribution is a quantitative comparison spanning prediction horizons, generalization, no-error testing, and modeling errors.The authors position the comparison as a reference for optimal-control researchers.

II. LITERATURE REVIEW

Prior work compares reinforcement learning and MPC in several control settings, but often uses unequal costs, different policy types, or no modeling errors. This paper addresses that gap with matched conditions and a Prius HFM for practical error testing.

  • II. LITERATURE REVIEW: Existing comparisons cover electrical oscillation damping, wildfire-coverage control with unmanned aerial vehicles, high-dimensional robotics, and dense-traffic merging.These studies report different control settings and comparison designs.
  • II. LITERATURE REVIEW: Prior studies variously use a random-tree policy, stochastic disturbances, integrated MPC–DRL, or unequal cost functions, limiting direct episode-cost comparisons.The dense-traffic study did not compare episode costs because DRL and MPC used different cost functions.
  • II. LITERATURE REVIEW: The literature lacks fair DRL–MPC comparisons for traditional state-space optimal control, especially when modeling errors are present.This paper holds most conditions equal and tests practical errors using a Prius HFM.
  • II. LITERATURE REVIEW: ACC research using DRL includes neural policies trained from car-following models or naturalistic driving data, while this work builds on earlier state-space DRL control.The present model uses constant time headway rather than the previous constant distance headway.
  • II. LITERATURE REVIEW: ACC MPC studies commonly use multi-objective costs for tracking error, energy consumption, and jerk, with first-order vehicle models often considered sufficient.The first-order approximation represents acceleration-command dynamics.

III. ACC PROBLEM FORMULATION

The ACC formulation models two-car following with state-space equations and a multi-objective cost balancing gap keeping, control effort, and jerk.

  • The following vehicle is controlled to maintain a constant 1s time gap from the preceding vehicle.
  • Gap-keeping error and velocity difference define the car-following tracking states.
  • A first-order system approximates the following vehicle’s acceleration dynamics, with commanded acceleration as the control input.
  • The cost function minimizes gap-keeping error, control effort, and jerk, the rate of acceleration.
  • The formulation uses equal weights for the three objectives and smooths the cost with ε so IPO can converge.
  • DRL uses the negative MPC cost as reward with γ=0.99, whereas MPC does not use discounting; hard constraints are excluded from both methods.

IV. METHODOLOGIES

Both controllers use the same discretized ACC dynamics and update frequency for training, testing, and optimization.

  • RK4 discretizes the ACC state-space equations for both DRL and MPC.
  • 10Hz is the shared control update frequency because both methods use Δt=0.1s.

A. DRL

The DRL controller learns a continuous ACC policy with DDPG from simulated state transitions, using actor-critic training and stabilizing techniques.

  • DDPG learns a deterministic state-action policy with separate actor and critic networks.
  • The critic estimates cumulative discounted reward, while the actor is updated to increase the critic’s action value.
  • Training stability uses target networks, mini-batches, experience replay, batch normalization, and Gaussian exploration noise.
  • The policy receives three ACC states and outputs the control input, with randomized normal car-following initial conditions.
  • Training runs for 1 million time steps over 20s episodes.

B. MPC

MPC solves the ACC optimal-control problem online over a prediction horizon and applies the resulting control sequence receding-horizon style.

  • IPO solves the single-shooting optimization problem formulated from the ACC dynamics and cost function.
  • MPC applies only the first control in each optimized sequence, then repeats optimization with updated states.
  • A one-time IPO optimization over the entire episode provides the benchmark solution.

V. RESULTS

Without modeling errors, the study compares DRL and MPC on the exact ACC state-space equations, examining prediction-horizon effects and broader testing conditions. Longer-horizon MPC approaches the IPO benchmark, while the cost response can change sharply near a specific horizon.

  • Testing setup: Testing uses the exact ACC state-space equations to isolate inherent DRL–MPC differences without modeling errors.The study also considers testing inputs both within and outside the training range, plus separate modeling-error experiments.
  • Single-condition comparison: For the single tested initial condition, DRL, MPC, and IPO produce similar response curves, although MPC with h=2.8s decelerates earlier and more strongly than IPO.The initial condition is [e0,ev,0,ai,0] = [5m,5m/s,0m/s2], within the training range.
  • Prediction-horizon effects: MPC episode cost decreases as the prediction horizon increases, with an acute change between 2.7s and 2.8s.The authors attribute the sharp transition to a cost function resembling an absolute-value cost; quadratic-cost results transition more smoothly.
  • Single-condition comparison: 2.6% higher DRL episode cost versus the IPO benchmark contrasts with MPC at h=5s, which is 0.02% lower than IPO.The study therefore treats h=5s as sufficiently long and uses it for subsequent testing.
  • Computational cost: 0.7 seconds for DRL execution contrasts with at least 50 seconds for MPC and IPO optimization in Python on the same computer.These timings are reported for the single-condition comparison.

B. Testing without modeling errors - different initial conditions

Testing across 75 initial conditions compares normal car-following and cut-in scenarios against the IPO benchmark. DRL remains near-optimal within its training range but loses optimality for out-of-range cut-in conditions, whereas h=5s MPC stays close to IPO.

  • Normal car following: 5.8% larger DRL average episode cost than the IPO benchmark occurs for normal car following within the training initial-condition range.The episode costs are averaged over 75 initial conditions.
  • Cut-in scenarios: 17.2% larger DRL average episode cost than the IPO benchmark occurs in cut-in scenarios outside the training range.The authors identify this increase as evidence of reduced optimality and a generalization issue in the trained neural network.
  • MPC comparison: Less than 0.1% difference from IPO is achieved by MPC with h=5s across the tested initial conditions.For larger-magnitude cut-in conditions, the reported MPC difference is 0.0%.
  • Generalization: The DRL cost increase grows as the testing initial gap-keeping error moves farther from the training range.The analysis averages over different initial relative velocities and following-vehicle accelerations for each e0.

C. Testing with control delays - different initial conditions

The control-delay experiment adds delay to the COM for testing and compares DRL and MPC across three delay values. Their costs are similar for small delays, but MPC becomes substantially worse at the largest delay because of oscillatory steady-state behavior.

  • Experimental setup: 0.1s, 0.2s, and 0.4s are the tested control delays, representing pure electric vehicles, hybrid or combustion-engine vehicles, and combustion-engine trucks.For each delay, average episode cost is computed over 75 initial conditions.
  • Small delays: For control delays τd ≤0.2s, DRL and MPC have similar average episode costs.The delay is added to the COM only during testing, delaying execution of the control action.
  • Large delay: At τd=0.4s, MPC has a significantly higher average episode cost than DRL.MPC exhibits oscillatory steady state, whereas DRL produces zero steady-state errors and low costs.

D. Testing on the HFM - constant-speed following

HFM testing shows that increasing speed and the resulting modeling errors raise both controllers’ episode costs, but DRL generally tolerates these errors better than MPC. At 25 m/s, MPC exhibits a large control-input jump and substantially higher cost.

  • HFM modeling errors: The HFM replaces the first-order COM and introduces modeling errors from control delay, mode switching, low-level PI effects, and vehicle power limits.The experiment considers low- and high-speed testing because power-limit effects contribute strongly to modeling error.
  • Constant-speed following: At vi,0=25m/s, MPC produces a large control-input jump at 10s, whereas DRL does not.The following vehicle starts at 25m/s while the preceding vehicle travels at 30m/s; the jump occurs when the vehicle experiences the power-limit constraint.
  • Constant-speed following: The MPC jump at vi,0=25m/s causes a significantly larger MPC episode cost.Setting the control delay to zero eliminates the jump, but that condition is impractical.
  • Speed dependence: For vi,0≤15m/s, DRL and MPC produce similar results.At the smallest speed, DRL’s episode cost exceeds its COM-testing cost because the HFM includes an unmodeled 0.2s control delay.
  • Speed dependence: As modeling errors increase with speed, both episode costs increase, but MPC’s cost rises more than DRL’s, especially at high speeds.The results suggest that DRL is better than MPC at coping with modeling errors in this experiment.

E. Testing on the HFM - drive cycles

Drive-cycle HFM testing introduces additional modeling errors because preceding-vehicle acceleration is nonzero while the ACC equations assume it is zero. Across HWFET, FTP-75, and US06, MPC has smaller gap-keeping errors but larger jerk and episode costs than DRL.

  • Drive-cycle setup: Drive-cycle testing uses HWFET, FTP-75, and US06, with nonzero preceding-vehicle acceleration creating additional modeling errors.The ACC state-space equations assume ai−1=0m/s2, unlike the drive cycles.
  • Drive-cycle results: Across all three drive cycles, MPC has smaller gap-keeping errors but larger jerk magnitudes than DRL.The comparison uses the same initial condition for every drive cycle.
  • Drive-cycle results: MPC episode costs are consistently larger than DRL’s across HWFET, FTP-75, and US06.The results support DRL’s stronger tolerance of larger modeling errors in drive-cycle testing.
  • Drive-cycle results: 52.2% is the MPC cost increase over DRL for HWFET, compared with 7.5% for FTP-75.The smaller FTP-75 increase is attributed to less frequent power-limit and control-input saturation during city driving.

VI. CONCLUSION

The study compares DRL and MPC under matched ACC modeling, discretization, update-frequency, and testing conditions, using both the COM and HFM. DRL approaches MPC under accurate, in-range conditions and is more tolerant of significant modeling errors, but its theoretical basis and broader applicability remain open questions.

  • Comparison design: DRL training and MPC optimization use the same three-state COM equations, cost function, discretization, update frequency, and testing conditions.DRL alone uses a reward discount factor γ=0.99 for convergence.
  • Conclusion: Without modeling errors and within the training-data range, DRL is equivalent to MPC with a sufficiently long prediction horizon.Testing covers the exact ACC state-space equations and both the COM and HFM, with the COM approximating HFM acceleration-command dynamics.
  • Conclusion: DRL performance degrades significantly when testing inputs fall outside the training-data range.This reflects the generalization issue identified for the DRL policy.
  • Conclusion: With significant modeling errors, DRL has lower episode costs than MPC; with small errors, their performances are similar.Overall, DRL shows smaller cost increases under the same modeling errors.
  • Discussion: The proposed explanation for DRL’s modeling-error tolerance is that DDPG’s probabilistic environment transitions represent stochasticity associated with modeling errors.The authors identify the underlying reason as requiring further theoretical analysis.
  • Future work: Future work includes higher-degree-of-freedom systems, robust and stochastic MPC, transfer and meta-learning, environmental noise, and combined DRL–MPC approaches.The paper contrasts DRL’s lack of theoretical assurance with MPC’s established modeling and optimization theory and online-computation burden.
Loading 1910.12047v3…