Source-linked AI summary
Automated vehicle's behavior decision making using deep reinforcement learning and high-fidelity simulation environment
Yingjun Ye, Xiaohui Zhang, Jian Sun
TL;DR
Automated-vehicle decision making remains underdeveloped because behavioral training data are limited and existing DRL setups use unrealistic scenes, discretized actions, and inconsistent reward methods. The paper combines high-fidelity simulation with DRL, trains car-following and integrated lane-changing behavior, and reports effective learning, while noting that lane changing remains conservative.
Problem
Automated-vehicle decision-making research is limited by scarce behavioral data, difficult action labeling, unrealistic training scenes, discretized states and actions, and nonuniform reward methods.
Method
The paper combines a VISSIM high-fidelity simulation environment with modified DDPG-based deep reinforcement learning to train car-following and integrated lane-changing behavior.
Results
The framework effectively learns automated-vehicle driving tasks, with normalized reward training converging after about 800 episodes and producing safe, comfortable car-following behavior.
Takeaways & Limitations
High-fidelity simulation and DRL provide an effective platform for automated-vehicle decision-making learning within the studied driving scenarios.
Takeaways & Limitations
The integrated lane-changing behavior remains conservative and requires reward modification and thousands of additional training episodes for refinement.
Abstract
from arXiv · showhide
Automated vehicles are deemed to be the key element for the intelligent transportation system in the future. Many studies have been made to improve the Automated vehicles' ability of environment recognition and vehicle control, while the attention paid to decision making is not enough though the decision algorithms so far are very preliminary. Therefore, a framework of the decision-making training and learning is put forward in this paper. It consists of two parts: the deep reinforcement learning training program and the high-fidelity virtual simulation environment. Then the basic microscopic behavior, car-following, is trained within this framework. In addition, theoretical analysis and experiments were conducted on setting reward function for accelerating training using deep reinforcement learning. The results show that on the premise of driving comfort, the efficiency of the trained Automated vehicle increases 7.9% compared to the classical traffic model, intelligent driver model. Later on, on a more complex three-lane section, we trained the integrated model combines both car-following and lane-changing behavior, the average speed further grows 2.4%. It indicates that our framework is effective for Automated vehicle's decision-making learning.
1. Introduction
Automated-vehicle research has advanced environment recognition and control more than decision making, which remains constrained by limited behavioral data and shortcomings in current training setups. The paper proposes an integrated high-fidelity simulation and deep reinforcement learning framework for decision-making training.
- 1. Introduction: Decision-making research for automated vehicles remains limited because behavioral data are scarce and difficult to label.Unlike perception tasks, decision-making lacks public training data, and actions depend on both surrounding conditions and other factors.
- 1. Introduction: Existing autonomous-driving DRL studies use unrealistic traffic scenes, discretized states and actions, and inconsistent reward-function methods.These limitations can reduce prediction accuracy and leave reward design without a uniform methodology.
- 1. Introduction: The paper proposes an integrated decision-making training framework combining VISSIM high-fidelity simulation with deep reinforcement learning.The framework is designed to address automated-vehicle decision-making rather than environment recognition or vehicle control alone.
- 1. Introduction: The framework adopts DDPG for continuous-state autonomous-driving decisions and investigates reward functions, convergence acceleration, and effectiveness verification.The study trains car-following first, then extends the model to integrate car-following and lane-changing behavior.
2. Work related to AV’s decision-making
Prior automated-vehicle decision-making research progressed from field challenges to virtual driving environments and machine-learning approaches. DRL expanded this work, but existing methods still face data demands and limitations from imitation-based training.
- 2. Work related to AV’s decision-making: Early AV competitions emphasized environment recognition and vehicle control before the Urban Challenge introduced more complicated decisions involving other vehicles.The Urban Challenge used a simplified urban environment requiring interaction with surrounding vehicles and traffic-rule compliance.
- 2. Work related to AV’s decision-making: Simulation offered less expensive, repeatable alternatives for comparing agents under identical conditions and became a focus of driving research.This shift moved attention from field testing toward virtual simulator training.
- 2. Work related to AV’s decision-making: Supervised machine-learning approaches require large amounts of hand-labeled data and remain limited by imitation of human driving.Such systems can learn to drive without necessarily achieving excellent performance.
- 2. Work related to AV’s decision-making: DRL introduced a different approach to driving behavior prediction and was subsequently applied across TORCS and more realistic game-based platforms.These platforms included environments with richer graphics and other cars or pedestrians.
3. Methodology
The paper replaces simplified racing-game training with a VISSIM-based traffic simulator linked to a DRL program, using DDPG to learn continuous driving decisions. The framework combines actor–critic learning and experience replay for autonomous-driving behavior training.
- 3.2 Training environment--VISSIM: VISSIM provides realistic traffic-flow scenes and road-user behavior, addressing simplifications associated with racing-game training environments.The simulator includes road infrastructure and traffic behavior rather than focusing primarily on 3D visual presentation.
- 3.1 Framework of the training platform: The training platform couples a VISSIM traffic simulator with a DRL program that exchanges recognition information and vehicle actions recursively.Data is passed between VISSIM and the DRL program through TCP/IP communication.
- 3.3 Training methods—DRL: Reinforcement learning models autonomous decision making as trial-and-error interaction with an environment that returns numerical rewards.The policy seeks actions that maximize expected total discounted rewards.
- 3.3.2 Methodology of DDPG: DDPG extends deep Q learning with deterministic policy gradients and actor–critic networks to support continuous-state and continuous-action driving decisions.The critic evaluates actor-produced actions, while the actor updates its policy through gradient ascent.
- 3.3.2 Methodology of DDPG: Experience replay stores recent experiences and randomly samples batches, reducing data relevance and exposing the network to more varied past interactions.The actor and critic perform separate but correlated roles in the DDPG architecture.
4. Case Study
The case study first trains car-following behavior from scratch, then extends the platform toward broader driving tasks that incorporate lane changing and overtaking.
- 4. Case Study: The study begins with car-following, requiring the vehicle to maintain proper distance and steady acceleration without prior driving experience.After car-following succeeds, the platform is extended to more general tasks incorporating lane changing and overtaking.
4.1 Effectiveness and stability evaluation
The study evaluates reinforcement-learning training through stability measures and visual or model-based effectiveness checks. Because traffic simulation lacks a universal score, effectiveness is assessed first by observation and then against IDM.
- 4.1 Effectiveness and stability evaluation: Training stability is assessed using episode-level total reward, average Q value, critic loss, and an approximated upper bound of average action value.The derived upper bound is intended to make convergence assessment more quantitative than visual inspection alone.
- 4.1 Effectiveness and stability evaluation: Effectiveness is first judged by visual observation, then compared with the intelligent driver model because the traffic simulator has no scoring system or universal autonomous-driving standard.The passage frames IDM as a comparison model for strategies that appear effective.
4.2 Variable definition
The case-study variables describe vehicle spacing, speed, relative speed, desired headway, jerk, and discrete driving-state conditions used to characterize car-following behavior.
- 4.2 Variable definition: The variables include net distance, vehicle speed, speed difference, normalized speed difference, jerk, and desired time headway.Desired time headway is defined as 2 s, while speed difference is expressed relative to the preceding vehicle.
- 4.2 Variable definition: The driving-state labels identify collision, low-speed following, uncomfortable jerk, reverse motion, and normal movement using threshold conditions.Examples include collision when d<0m and uncomfortable jerk when Δa>5.6m/s2.
4.3 Training tips about model training
The training experiments examine how reward design and optimization choices affect DRL convergence and driving behavior, then evaluate two ways to accelerate training. Normalized rewards with suitable optimization settings support effective learning, while pre-training reduces training episodes.
- Discussion on reward function: Addition-form rewards caused either collision-prone acceleration or uncomfortable speed fluctuations, while the multiplication form showed no convergence.The addition form produced persistently negative total rewards, whereas the multiplication form reached relatively high positive rewards without converging.
- Discussion on reward function: About 800 episodes were required for the normalized reward r_a3 to converge while producing safe, comfortable following behavior.The trained vehicles maintained proper headway and avoided frequent or sharp acceleration and braking; the maximum average Q value was 42.0, close to the evaluated upper bound.
- Discussion on reward function: 100-time reward magnification destabilized learning, but reducing the learning rate to 1% and regularizing the action network restored convergence in about 700 episodes.The analysis attributes the failure to oversized updates and local solutions associated with critic-network ReLU and action-network tanh behavior.
- Discussion on reward function: The reward-setting guidance is to bound reward and punishment, adjust the learning rate, and regularize the action network.These choices address magnitude mismatch and excessive parameter-update steps during training.
- Ways to accelerate training: Pre-training the action network on IDM calibration data reduced convergence from almost 800 episodes to about 400 episodes.This approach lets the vehicle start from an experienced driving policy rather than learning entirely from zero.
- Ways to accelerate training: Increasing the simulation time step from 0.1 s to 1 s was feasible, with convergence at about 800 episodes similar to the 0.1 s model.The paper theoretically associates this change with a tenfold efficiency increase while retaining effectiveness.
4.4 Evaluation of the DRL trained CF model
The DRL car-following model is evaluated against IDM for efficiency and comfort, with training acceleration also tested using a larger simulation time step.
- Efficiency: 12.44m/s average speed is 7.9% higher for DRL than IDM’s 11.53m/s in the same VISSIM environment.The comparison uses 800 episodes for each model, with DRL evaluated over its last 800 converged episodes.
- Comfort: Only 2.6% of sampled DRL jerks exceed the 5.6 m/s2 comfort threshold, despite higher jerk values than IDM.The study extracts 2000 jerk steps randomly from 800 episodes.
- Overall evaluation: Under the stated comfort premise, the DRL model is reported as more effective than IDM, and the overall training framework is considered feasible.The conclusion combines the efficiency comparison with the reported comfort evaluation.
4.5 Extended model integrating CF and LC behavior
The model is extended from car-following to integrated car-following and lane-changing on a three-lane section, using nearby-vehicle information and lane-change rewards.
- 4.5.1 Model design: The extended model receives current-lane and adjacent-lane vehicle information and outputs acceleration plus LK, LLC, or RLC decisions.The input includes nearby vehicles’ spacing and longitudinal speed information.
- 4.5.1 Reward design: The lane-change reward accounts for AV speed and acceleration effects on followers, penalizing changes that force sharp braking despite AV acceleration.The reward design refers to the MOBIL model.
- 4.5.1 Model assumption: Lane-changing is simplified by assuming the AV follows the nearer vehicle on both current and target lanes throughout the 2s lane-change process.This is an explicit modeling assumption for reducing model complexity.
- 4.5.2 LC decision analysis: Lane-change training converges over 5000 episodes, while training statistics show fewer instructions and optimization of lane-change decisions over time.The frequency of lane-change instructions falls from 267 to 20, and the success rate stabilizes around 4%.
- 4.5.2 LC decision analysis: The converged lane-change frequency is 0.81 veh/km/ln, similar to NGSIM values but obtained under lighter simulated traffic flow.The reported NGSIM values are 0.63 veh/km/ln for US-101 and 0.80 veh/km/ln for I-80, versus about 700 veh/ln in simulation.
- 4.5.2 LC decision analysis: Both lead and lag time headways are significantly higher than NGSIM field data, consistent with a reward that considers lag vehicles and overall efficiency.The paper contrasts this objective with human drivers’ preference for their own driving condition.
- 4.5.3 Model effectiveness: 12.74m/s average speed for the integrated model is 2.4% higher than for the DRL car-following model.The result is reported for the extended model integrating car-following and lane-changing behavior.
- 4.5.3 Model effectiveness: The integrated model is more efficient, but its lane-changing remains conservative and requires reward modification and extensive further training.The paper identifies more specific lane-change analysis as future work.
5. Conclusion
The paper proposes DRL with high-fidelity simulation for automated-vehicle decision learning and evaluates it on car-following and integrated lane-changing tasks.
- Framework: The framework combines DDPG-based DRL with VISSIM and investigates reward functions, convergence verification, and training acceleration.The paper considers bounded reward, learning rate, regularized action networks, and enlarged time steps.
- Car-following evaluation: The DRL car-following model is reported to maintain acceptable comfort, with only 2.6% of jerks exceeding the comfort threshold.The paper also reports an efficiency comparison with IDM in the conclusion passage.
Appendix A. Estimation of the approximated upper bound of average action value
The appendix derives an approximated upper bound for average action value to evaluate convergence and avoid instability in commonly used convergence indexes.
- Upper-bound estimation: The proposed upper bound is introduced to evaluate current-prediction convergence and avoid instability in three commonly used indexes.The appendix then expands the action value and derives the approximated average-action-value bound.
- Derivation: The appendix uses an action-value expansion as the intermediate step toward the approximated upper bound.The proof explicitly proceeds from Eq.(1) by expanding the action value.
- Derivation assumption: The derivation assumes that, after convergence, every step has maximized reward before obtaining the approximated bound.This assumption is stated for any step t in the proof.