PaperlayerView exact revision on arXiv

Source-linked AI summary

Learned Terminal Costs for Short-Horizon Model Predictive Control of Autonomous Trucks in Interactive Highway Traffic

Hooman Hamze, Benjamin Houshmand, Deepthi Pathare, Erik Börve, Morteza Haghir Chehreghani, Nikolce Murgovski

arXiv:2610.40257v1cs.LGcs.AIcs.ROcs.SYeess.SY

TL;DR

Heavy-truck control must balance safety and long-horizon decisions, while MPC’s terminal-cost design limits short-horizon effectiveness. The paper combines finite-horizon MPC with a learned terminal cost and reports that it resolves short-horizon pure-MPC failure while remaining within real-time computational budget.

  • Problem

    The ideal terminal cost cannot be computed exactly for the constrained system because surrounding-vehicle behavior is not modeled in closed form and the Bellman recursion has no analytical solution.

    Source: §2.3 · p. 4

  • Method

    A chance-constrained MPC solved by dynamic programming over a discretized state-action space uses a DQN-learned approximation of the cost-to-go as its terminal cost.

    Source: §3.4 · p. 5, §5 · p. 8

  • Results

    The learned terminal cost resolves pure MPC’s short-horizon failure, outperforms zero-terminal-cost MPC at longer horizons, and remains within real-time computational budget.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Short-horizon MPC can achieve better driving quality than zero-terminal-cost MPC at longer horizons, while extending rollout horizon can degrade performance when base-policy accuracy limits rollout quality.

    Source: §5 · p. 8

    The evaluation assumes a straight homogeneous road, a kinematic ego model without continuous lateral control, and surrounding-vehicle predictions with constant acceleration and fixed lanes.

    Source: §5 · p. 8

Abstract

from arXiv · show

Autonomous highway driving for heavy-duty trucks is a challenging control problem. The greater mass, dimensions and intricate dynamics increase the requirements for cooperation from surrounding traffic in critical maneuvers, requiring a careful balance between safety constraints and efficient long-horizon decision making. Model Predictive Control (MPC) provides rigorous constraint satisfaction but relies on terminal cost which can limit its effectiveness at short, computationally affordable horizons. Reinforcement Learning (RL) captures long-term driving behavior but offers no constraint guarantees and tends toward overly conservative policies when trained under sparse collision penalties. In this paper, we propose a hybrid MPC and RL framework for highway driving in autonomous trucks, where a finite-horizon MPC, solved via dynamic programming, uses a pre-trained RL model for terminal cost estimation. We evaluate the framework in a simulation environment with reactive surrounding traffic and compare it against pure MPC with zero terminal cost, rollout with terminal cost estimation using a base policy, and pure RL. Results show that the hybrid approach achieves zero collisions with the lowest cost of any configuration.

1 Introduction

Heavy-truck highway control must balance safety and efficient long-horizon decisions amid difficult dynamics and reactive traffic. This paper combines MPC with a learned terminal cost and reports a strong safety, driving-performance, and computational-efficiency balance at short horizons.

1 Introduction

  • Heavy-truck lane changes and merges require more cooperation from surrounding traffic, whose responses also make highway control interactive.Truck-trailer dynamics and limited maneuverability further complicate motion planning and control.Source: §1 · p. 1
  • MPC enforces constraints but faces a speed-versus-horizon trade-off, whereas RL learns from interaction without guaranteeing physical or safety constraints.A short MPC horizon with a crude terminal cost can perform poorly; longer horizons cost more online, while RL behavior outside training situations is unpredictable.Source: §1 · pp. 1–2
  • Prior learned-terminal-cost methods cover continuous low-level control in single-agent, non-reactive settings, while traffic hybrids do not learn the terminal cost.The paper identifies the unaddressed combination as discrete tactical highway driving with genuinely reactive surrounding traffic.Source: §1 · p. 2
  • A pre-trained DQN supplies terminal costs to an MPC that handles speed control and lane changes over a discrete state-action grid.The framework is evaluated in highway simulation against conventional MPC and rollout-based terminal-cost methods.Source: §1 · p. 2
  • The learned terminal cost lets short-horizon MPC achieve the best balance of safety, driving performance, and computational efficiency.The three-step controller avoids computationally expensive long-horizon optimization while remaining suitable for real-time implementation.Source: §1 · p. 2

2 PROBLEM FORMULATION

The formulation uses finite-horizon, chance-constrained MPC to manage uncertain, reactive traffic, with a terminal cost standing in for costs beyond the planning horizon. Because the ideal cost-to-go cannot be computed exactly, the paper approximates it with a learned value function.

2 PROBLEM FORMULATION

  • Surrounding vehicles respond to the ego truck, so the traffic environment is interactive and optimal decisions depend on predicting traffic over multiple steps.Source: §2 · p. 3

2.1 System Model

  • The ego truck uses a discrete-time kinematic model, with lane and lane-change-counter updates jointly enforcing transition and cooldown durations.A new lane-change command is accepted only after the counter clears its transition time and the truck reaches its target lane.Source: §2.1 · p. 3
  • Traffic-state uncertainty comes from noisy initial observations and simplified predictions of surrounding vehicles whose behavior is not fully predictable.Both sources of uncertainty accumulate over the prediction horizon.Source: §2.1 · p. 3

2.2 The MPC Problem

  • Finite-horizon MPC truncates an intractable infinite-action problem and uses DP on the ego state while treating predicted surrounding states as time-varying parameters.DP over the full joint state space would be intractable because its grid grows exponentially with the number of surrounding vehicles.Source: §2.2 · p. 3
  • Chance-constrained MPC requires the ego to remain safe with probability at least 1 − ε, accounting for uncertain traffic states.Under Gaussian uncertainty, the safety constraint becomes a deterministic equivalent with a margin proportional to predicted position standard deviation; suppressing the margin yields a nominal-prediction baseline.Source: §2.2 · p. 3

2.3 The Terminal Cost Approximation Problem

  • Setting the terminal cost to the optimal value function makes the truncated problem equivalent to the infinite-horizon optimum for any horizon of at least one step.Exactness requires matching the discount factors in the MPC objective, terminal cost, and value function.Source: §2.3 · p. 4
  • The optimal value function cannot be obtained exactly because surrounding-vehicle dynamics are not modeled in closed form and the Bellman recursion has no analytical solution.A reasonable approximation can be corrected by MPC look-ahead; residual error decreases sharply as the horizon increases.Source: §2.3 · p. 4
  • The paper uses an RL-based model to approximate the optimal value function and inserts it as the terminal cost in the DP backward pass.The terminal cost, rather than horizon length, is the effective design variable for short-horizon performance.Source: §2.3 · p. 4

3 Proposed Method

The controller combines finite-horizon MPC with learned or rollout-based terminal-cost estimates. It predicts surrounding traffic, penalizes unsafe and inefficient behavior under uncertainty, and repeats optimization after applying only the first action.

3 Proposed Method

  • At each step, the controller builds a discretized state-action grid, evaluates terminal costs, runs backward dynamic programming, and sends only the first action to the simulator.It repeats this loop at the next step; alternatives include zero-terminal-cost MPC, rollout terminal cost, and pure RL.Source: §3 · p. 4, Figure 1 · p. 4

3.1 Trajectory Prediction

  • Constant-acceleration extrapolation with lanes held fixed gives the lowest mean position error among tested prediction strategies at all tested horizon lengths.Vehicles changing lanes are represented in both their origin and target lanes, so the controller accounts for their presence in either lane during transition.Source: §3.1 · pp. 4–5

3.2 Cost Function under Uncertainty

  • The stage cost combines seven weighted terms, while safety and collision terms alone depend on predicted surrounding-vehicle states.Speed deviation and control effort are penalized; lane-change and cooldown penalties discourage repeated maneuvers, while a progressive left-lane penalty encourages returning to the slow lane.Source: §3.2 · p. 5

3.3 Discounted Objective

  • The discounted objective down-weights later costs because prediction evidence weakens with horizon, and applies the same discount to terminal cost.Using the same discount in RL training keeps the learned terminal cost and stage costs on a common scale.Source: §3.3 · p. 5

3.4 RL-Based Terminal Cost Approximation

  • A DQN Q-network estimates optimal cost-to-go, and its value is rescaled into MPC cost units for use as the terminal cost.The MPC terminal state is mapped into the RL observation representation before the network is queried.Source: §3.4 · pp. 5–6

3.5 Rollout-Based Terminal Cost Approximation

  • The rollout baseline extends planning online by simulating a base policy forward from each terminal state and accumulating subsequent stage costs.The DP backward pass and rollout use the same discount factor, keeping the combined horizon estimate consistent across their boundary.Source: §3.5 · p. 6

4 Results and Discussion

The evaluation uses SUMO with TraCI to connect the controller to a simulated three-lane highway. The ego vehicle is a heavy-duty truck amid surrounding traffic modeled with SUMO car-following and lane-change models.

4.1 Experimental Setup

  • The controller interacts with SUMO through TraCI, which supplies surrounding-vehicle states and accepts longitudinal and lateral ego commands at each step.SUMO is an open-source microscopic traffic simulator.Source: §4.1 · p. 6
  • The scenario is a three-lane, one-way highway with a heavy-duty truck as the ego vehicle and surrounding traffic using SUMO’s Krauss car-following and LC2013 lane-change models.Surrounding-vehicle desired speeds are sampled uniformly within a fixed range, maintaining approximately constant traffic density around the ego.Source: §4.1 · p. 6
  • Episodes end at a fixed maximum duration or on collision, and the dynamics and cost parameters are listed in Appendix D.Source: §4.1 · p. 6

4.2 RL Training

Across five training runs, the selected DQN produced collision-free policies at normal driving speed. Runs improved rapidly early in training and then fluctuated around a common holdout reward level.

4.2 RL Training

  • −17.00 ± 8.63 mean reward and 20.1–20.8 m/s average speeds were achieved by collision-free policies across all five runs.The first-run model was selected as the terminal cost used throughout.Source: Figure 2 · p. 6, §4.2 · p. 6
  • Training used perfect state observations in a deterministic simulation so the agent could learn long-term cost structure without noisy-measurement variance.The resulting value function was used as an MPC terminal cost, with measurement uncertainty handled by the chance constraint.Source: §4.2 · p. 6
  • Checkpoint selection re-evaluated the ten highest-scoring checkpoints on 500 independent seeds and chose the model using mean reward and mean speed.Mean speed excluded policies that avoided collisions by driving well below reference speed; the selected configuration was trained five times.Source: §4.2 · p. 6
  • All five runs improved rapidly within the first few million steps and then fluctuated around a common holdout reward level.Figure 2 reports holdout reward as mean ± one standard deviation across the five training runs.Source: §4.2 · p. 6, Figure 2 · p. 6

4.3 Controller Comparison

The controller comparison evaluates the hybrid method against zero-terminal-cost MPC, rollout terminal costs, and pure RL. The configurations are tested over independent stochastic episodes, with deterministic variants used to isolate uncertainty handling.

4.3 Controller Comparison

  • The hybrid MPC-RL controller and three baselines are evaluated on 100 independent stochastic episodes.Deterministic results are reported in Appendix A as a baseline for isolating uncertainty handling.Source: §4.3 · p. 6
  • Pure MPC uses the same dynamic-programming formulation with zero terminal cost, isolating the effect of lookahead length.The hybrid controller is evaluated across horizons N∈{1, . . . , 5}.Source: §4.3 · p. 6
  • Rollout substitutes a simulated base-policy terminal estimate at rollout horizons 10 and 20, while pure RL applies the trained DQN without MPC optimization.The comparison reports collision rate, per-step cost, cost variability, mean speed, computation time, and cost decomposition.Source: §4.3 · pp. 6–7

4.4 Discussion

The learned terminal cost lets the hybrid controller achieve better driving quality at shorter horizons than zero-terminal-cost MPC, while prediction-uncertainty-aware safety distances affect performance differently across horizons. Rollout estimates depend on base-policy accuracy, pure RL is collision-free but costly, and the best-performing hybrid configuration stays within the simulation-time budget.

4.4.1 Effect of the terminal cost.

  • At N=1, pure MPC collides in 86% of episodes, while the hybrid eliminates this failure mode and one-step lookahead reduces collisions to zero.The learned terminal cost flags slow-leader proximity as costly; uncertainty-aware safety distances at N=3 complement the terminal value without significantly penalizing it.Source: §4.4.1 · p. 7

4.4.2 Rollout horizon length and base policy quality.

  • With rollout horizon Nr=10, N=1 collisions fall from 86% to 39%, but the estimate never eliminates collisions at every horizon and costs up to 44 times more than the hybrid.At Nr=20, performance degrades beyond N=1 because accumulated prediction mismatch outweighs discounting; rollout quality is bounded by base-policy accuracy.Source: §4.4.2 · p. 7

4.4.3 Pure RL as a direct policy.

  • Pure RL achieves zero collisions at total cost 551, but its speed cost (18.0) and acceleration cost (2.4) are higher than those of safe MPC or hybrid controllers.The paper attributes its conservative driving to sparse, large collision penalties; that risk-averse value signal is useful as the hybrid's terminal cost.Source: §4.4.3 · p. 7

4.4.1 Effect of the terminal cost.

  • At N=3, the hybrid achieves 329 cost with 0% collisions, beating pure MPC at N=5 (754 cost) with a shorter horizon.Its safety cost is 309, 59% below pure MPC at N=5 (747).Source: §4.4.1 · p. 7, Table 1 · p. 8

4.4.4 Comparison with deterministic baseline.

  • Three configurations become collision-free while reducing cost under uncertainty accounting; the hybrid at N=3 has the largest improvement, from 1,015 to 329.Uncertainty accounting can instead increase cost for configurations already collision-free deterministically; Table 2 presents representative deterministic-versus-stochastic comparisons.Source: §4.4.4 · pp. 7–8, Table 2 · p. 8

4.4.5 Computational cost.

  • All controllers stay within the 200 ms simulation-time budget; hybrid N=3 takes 91.6 ms per step, while hybrid N=5 is most expensive at 176.0 ms.Reported timing is influenced by shared-cluster workload, and each reachable terminal state requires a Q-network forward pass in addition to DP expansion.Source: §4.4.5 · p. 8

5 Conclusion

The conclusion presents a chance-constrained hybrid MPC-RL controller that uses a DQN-learned cost-to-go as its terminal cost and evaluates it against three baselines in reactive traffic. The reported results favor the learned terminal cost at short horizons, while the stated limitations bound the evaluation to simplified road, vehicle, and traffic models.

5 Conclusion

  • The framework combines chance-constrained finite-horizon MPC over a discretized state-action space with a DQN-learned approximation of the true cost-to-go.It is evaluated against pure MPC, rollout-based, and pure-RL baselines in a simulator with reactive surrounding vehicles.Source: §5 · p. 8
  • The learned terminal cost resolves pure MPC’s short-horizon failure, improves driving quality over longer-horizon zero-terminal-cost MPC, and stays within real-time computational limits.Extending the rollout horizon degrades performance rather than improving it, with rollout quality bounded by base-policy accuracy.Source: §5 · p. 8
  • The evaluation is limited to a straight homogeneous road, a kinematic ego model without continuous lateral control, and surrounding-vehicle predictions that assume constant acceleration and fixed lanes.The prediction model becomes less accurate at longer horizons and misses reactive lane-change behavior.Source: §5 · p. 8

A Deterministic Controller Results

The deterministic evaluation suppresses measurement noise and uses nominal predicted gaps, with other settings held identical to the stochastic evaluation. It reports aggregate controller results and a cost decomposition.

A Deterministic Controller Results

B Trajectory Prediction Methods

The appendix compares reactive multi-agent prediction with constant-acceleration extrapolation for surrounding-vehicle trajectories. Constant-acceleration extrapolation has lower reported prediction errors and is adopted throughout the work, while reactive prediction has a sensor-boundary error that compounds over longer horizons.

Reactive Multi-Agent Prediction

  • Reactive multi-agent prediction propagates neighbors with SUMO’s car-following logic and the Krauss safe-speed model, using the ego’s constant-speed, fixed-lane projection as a reference.At each step, each neighbor adjusts speed based on its gap to the vehicle immediately ahead.Source: B Trajectory Prediction Methods · p. 10
  • The reactive model treats the sensor-front vehicle as having clear road ahead, causing artificial acceleration that propagates backward and compounds position errors over longer horizons.Freezing that boundary leader at constant speed mitigates the error only to some extent.Source: B Trajectory Prediction Methods · p. 10

Constant-Acceleration Extrapolation

  • Constant-acceleration extrapolation uses each neighbor’s observed position, speed, and acceleration while holding its lane fixed, without requiring car-following parameters or inter-vehicle dependencies.This avoids boundary effects associated with the reactive model.Source: B Trajectory Prediction Methods · p. 10

Comparison

  • The methods are compared using predicted versus ground-truth SUMO positions at each of five horizon steps across 100 seeds, with Table 5 reporting i=5.The longest horizon is identified as having the most error.Source: B Trajectory Prediction Methods · p. 10, Table 5 · p. 10
  • Constant-acceleration extrapolation has lower mean absolute error and standard deviation than the reactive model at every tested horizon step.The comparison evaluates predicted positions against ground-truth SUMO positions across 100 seeds; Table 5 reports the longest horizon, i=5.Source: B Trajectory Prediction Methods · p. 10, Table 5 · p. 10
  • Below 0.15 m, the position-error standard deviation at i=5 (1 s ahead) supports current acceleration as the dominant predictor at the DP solver’s short timescales.Constant-acceleration extrapolation is adopted as the prediction model throughout the work.Source: B Trajectory Prediction Methods · pp. 10–11

C Rollout Base Policy

The rollout base policy extends the ego and surrounding vehicles forward using lightweight analytical models. It estimates a terminal cost from the rollout, but omits surrounding-vehicle lane changes, making it optimistic when a neighbor cuts in.

C Rollout Base Policy

  • The base policy propagates the ego and surrounding vehicles for N_r steps with lightweight analytical models, avoiding runtime SUMO simulation.Source: C Rollout Base Policy · p. 11
  • Surrounding vehicles follow the Krauss safe-speed model and hold their lanes throughout the rollout.Source: C Rollout Base Policy · p. 11
  • The ego considers a lane change only when leader and follower gaps exceed d_safe and the target lane offers a speed closer to v_ref.If all three conditions hold, it changes lanes; otherwise, it remains in its current lane.Source: C Rollout Base Policy · p. 11
  • The rollout accumulates discounted stage cost under its rollout policy.Source: C Rollout Base Policy · p. 11
  • The rollout omits surrounding-vehicle lane changes, making its estimate optimistic when a neighboring vehicle cuts in during the extension.The rollout policy is used to estimate terminal cost.Source: C Rollout Base Policy · p. 11

D RL Model and Simulation Hyperparameter Tuning

Hyperparameter tuning screened settings on 10 holdout seeds, and only the final configuration received a full 500-seed evaluation, so these outcomes are qualitative and not directly comparable to Table 6. Tuning addressed discounting, learning rate, sensing range, exploration, and alignment of the speed weight with the MPC cost.

D RL Model and Simulation Hyperparameter Tuning

  • Only the final configuration was evaluated on 500 seeds; earlier settings were screened on 10 holdout seeds, making these outcomes qualitative and not directly comparable to Table 6.Source: D RL Model and Simulation… · p. 11
  • A discount factor of 0.995 yielded near-standstill collision avoidance, while 0.95 and 0.90 removed that behavior but produced unstable training; 0.98 was retained.No checkpoint at 0.95 or 0.90 survived the 500-seed evaluation.Source: D RL Model and Simulation… · pp. 11–12
  • Reducing the learning rate from 10^-4 to 10^-5 reduced holdout-curve oscillation without eliminating it; sensing range was raised to 150 m and final exploration set to 0.01.An exploration rate of 0.001 gave comparable holdout results but more degenerate checkpoints.Source: D RL Model and Simulation… · p. 12
  • The speed weight must match the MPC cost at w_v = 1 so learned Q-values remain in the same units as the horizon objective.Other tested weights degraded the hybrid controller, not the RL policy itself.Source: D RL Model and Simulation… · p. 12
Loading 2610.40257v1…