Source-linked AI summary
Safe, Efficient, and Comfortable Velocity Control based on Reinforcement Learning for Autonomous Driving
Meixin Zhu, Yinhai Wang, Ziyuan Pu, Jingyun Hu, Xuesong Wang, Ruimin Ke
TL;DR
The paper addresses autonomous car-following velocity control that should optimize safety, efficiency, and comfort rather than only imitate human driving. It proposes a deep-RL model trained through trial-and-error interaction with a simulation environment and NGSIM data, then compares simulated behavior with empirical driving. The model demonstrates safe, efficient, and comfortable velocity control, including fewer dangerous minimum TTC values than the NGSIM human-driver data.
Problem
Human imitation may not provide optimal autonomous driving, while car-following control should also optimize safety, efficiency, and comfort.
Method
A DDPG-based deep-RL model learns car-following velocity control through simulation interaction using a reward function combining safety, efficiency, and comfort.
Results
The model demonstrated safe, efficient, and comfortable driving, with 8% of DDPG minimum TTCs below 5s versus 35% for NGSIM data.
Takeaways & Limitations
The results indicate that reinforcement learning methods could contribute to developing autonomous driving systems.
Abstract
from arXiv · showhide
A model used for velocity control during car following was proposed based on deep reinforcement learning (RL). To fulfil the multi-objectives of car following, a reward function reflecting driving safety, efficiency, and comfort was constructed. With the reward function, the RL agent learns to control vehicle speed in a fashion that maximizes cumulative rewards, through trials and errors in the simulation environment. A total of 1,341 car-following events extracted from the Next Generation Simulation (NGSIM) dataset were used to train the model. Car-following behavior produced by the model were compared with that observed in the empirical NGSIM data, to demonstrate the model's ability to follow a lead vehicle safely, efficiently, and comfortably. Results show that the model demonstrates the capability of safe, efficient, and comfortable velocity control in that it 1) has small percentages (8\%) of dangerous minimum time to collision values (\textless\ 5s) than human drivers in the NGSIM data (35\%); 2) can maintain efficient and safe headways in the range of 1s to 2s; and 3) can follow the lead vehicle comfortably with smooth acceleration. The results indicate that reinforcement learning methods could contribute to the development of autonomous driving systems.
I. INTRODUCTION
Car-following velocity control must balance safety, efficiency, and comfort rather than merely imitate human driving. The paper proposes a deep reinforcement-learning model that learns this multi-objective control through interaction with a simulation environment and evaluates it against NGSIM behavior.
- Car following is a frequent driving scenario centered on maintaining safe and comfortable following gaps.
- Rule-based and supervised-learning driver models traditionally approximate human car-following behavior from rules or demonstrations.
- Human imitation alone may be inadequate because human driving may not be optimal, users may prefer different behavior, and control should also optimize safety, efficiency, and comfort.
- The proposed deep reinforcement-learning model directly optimizes safety, efficiency, and comfort while learning from trial-and-error interaction with a simulation environment.
- DDPG learns actor and critic networks, with the actor outputting accelerations from speed, relative speed, and spacing and the critic improving the policy.
- The model is trained with NGSIM driving data and compared with empirical NGSIM car-following behavior for safety, efficiency, and comfort.
B. Reinforcement Learning
Reinforcement learning formulates velocity control as sequential interaction between an agent and environment, with actions selected from observed states and evaluated through accumulated rewards. The section distinguishes value-based and policy-based learning and introduces actor-critic and deep-RL foundations relevant to continuous control.
- At each time step, an RL agent observes a state, selects an action under a policy, receives a reward, and transitions to the next state.
- The agent seeks maximum discounted accumulated reward, with discount factor γ ∈(0, 1].
- Value-based RL estimates action-value functions, where Qπ(s, a) is the expected return after selecting action a in state s and following policy π.
- The optimal value-based policy selects the action with the highest Q(s, a) to maximize expected future rewards.
- Policy-based methods directly update policy parameters using gradient ascent on expected return.
- Actor-critic methods combine an actor that chooses actions with a critic that evaluates action quality and guides policy adjustment.
- Deep reinforcement learning uses neural networks to approximate value functions, policies, or system models.
- DDPG extends actor-critic methods for continuous control, addressing the limitation of DQN in continuous action spaces.
1) Deep Q-Network:
The study applies DDPG to continuous car-following velocity control using actor and critic networks, replay and target networks, simulated vehicle dynamics, and filtered NGSIM events. Training proceeds through exploratory interaction, transition storage, minibatch updates, and policy improvement.
- 1) Deep Q-Network:: DDPG uses separate actor and critic networks to represent the policy and estimate action values.
- 1) Deep Q-Network:: Experience replay stores environment transitions and trains both networks on random minibatches to reduce correlation among sequential samples.
- 1) Deep Q-Network:: Target networks slowly track the main networks, constraining target-value updates and improving learning stability.
- 1) Deep Q-Network:: NGSIM trajectory data from eastbound I-80 in Emeryville, California, provide the real-world driving data used in the study.
- 1) Deep Q-Network:: Algorithm 1 identifies DDPG as the deep deterministic policy-gradient method used for car-following velocity control.
- 1) Deep Q-Network:: Training initializes actor, critic, and target networks before episodes begin with exploratory actions and observed car-following states.
- 1) Deep Q-Network:: Each interaction calculates a reward, applies follower acceleration through a kinematic point-mass model, stores the transition, and updates targets from sampled transitions.
- 1) Deep Q-Network:: The actor policy is updated using a sampled policy gradient.
IV. FEATURES FOR REWARD FUNCTION
The section introduces features that represent the relevant objectives of car-following velocity control. These features are intended to support construction of a reward function.
- Relevant car-following velocity-control features were proposed to construct a proper reward function.
A. Safety
Safety is represented with time to collision (TTC), while efficiency is represented with time headway. The study derives these measures and features from empirical NGSIM car-following data.
- Safety: TTC represents the time remaining before two vehicles collide and is used as the safety indicator.Smaller TTC values correspond to higher crash risks.
- Safety: A 7-second safety limit was selected from the NGSIM TTC distribution at the 10th percentile.TTC below 7s produces a negative safety feature, approaching negative infinity as TTC approaches zero.
- Efficiency: Time headway measures the elapsed time between the lead and following vehicles reaching a designated point.Short headways within safety bounds correspond to greater roadway capacity.
- Efficiency: A lognormal distribution was fit to time headways from 1,341 extracted NGSIM car-following events.The empirical estimates were µ = 0.4226 and σ = 0.4365.
- Efficiency: Headways around 1.3 seconds produced large feature values of about 0.65, while excessively short or long headways produced low values.The feature encourages efficient headways and discourages unsafe or overly long headways.
C. Comfort
Comfort is measured through jerk, the rate of change of acceleration, and incorporated into the velocity-control learning approach.
- Comfort: Jerk is used to measure passenger comfort because it strongly influences comfort.The jerk feature is constructed from changes in vehicle acceleration.
- Comfort: The squared jerk is divided by 3600 to scale the comfort feature into the range [0, 1].The base value follows the assumed 0.1s sample interval and observed acceleration bounds of -3 to 3 m/s2.
- Comfort: DDPG is used to learn the velocity-control strategy because vehicle acceleration is a continuous variable.The algorithm is introduced as the method for learning the control strategy.
A. State and Action
The car-following process uses vehicle speed, spacing, and relative speed as the state, with following-vehicle acceleration as the action in a simulated environment.
- A. State and Action: The state contains following-vehicle speed, spacing, and relative speed, while the action is longitudinal following-vehicle acceleration.These variables describe the car-following process at time step t.
- A. State and Action: The next state is generated with a kinematic point-mass model using the current state and acceleration.The simulation interval is 0.1s, and the lead-vehicle velocity is externally inputted.
- A. State and Action: Each simulation event is initialized with empirical following-vehicle speed, spacing, and velocity differences.The agent then computes acceleration and iteratively generates future velocity, relative speed, and spacing.
- A. State and Action: The reward function combines safety, efficiency, and comfort features linearly to encourage or discourage car-following behaviors.The three feature coefficients are all set to 1 in this study.
D. Network Architecture
The model uses separate actor and critic neural networks for continuous car-following control, with shallow three-layer architectures and standard optimization components.
- D. Network Architecture: The actor maps the state to following-vehicle acceleration, while the critic maps a state-action pair to a scalar Q-value.The state contains following-vehicle speed, relative speed, and spacing.
- D. Network Architecture: Both networks contain input, output, and one hidden layer with 30 neurons.Deeper networks with more than one hidden layer did not perform significantly better in testing.
- D. Network Architecture: ReLU is used in hidden layers, while tanh bounds actor outputs and therefore accelerations between -3 and 3 m/s2.The paper states that ReLU can accelerate convergence of network-parameter optimization.
- D. Network Architecture: Adam updates the network parameters, with critic learning based on a loss function and actor learning based on a policy gradient.The hyperparameters were selected from prior work and tests on a randomly sampled training dataset.
F. Exploration Noise of Action
The DDPG agent was trained with temporally correlated exploration noise across empirical car-following events, and its learned trajectories were evaluated against NGSIM behavior for safety.
- F. Exploration Noise of Action: Training performance began converging around episode 20, reaching an average reward of about 0.64.The reward reflects action choices producing near-zero TTC and jerk features while maximizing the headway feature.
- F. Exploration Noise of Action: The model achieved similar performance on training and testing data, indicating generalization to new data.
- F. Exploration Noise of Action: 8% of DDPG minimum TTC values were below 5s, compared with 35% for NGSIM empirical data.Minimum TTC was used to evaluate driving safety.
- F. Exploration Noise of Action: In an illustrative event, DDPG maintained a safe following gap around 10m, whereas the NGSIM driver produced very small inter-vehicle spacing.
B. Efficient Driving
The DDPG model was assessed for efficient and comfortable car following using time headway and jerk distributions, alongside comparisons with NGSIM data.
- B. Efficient Driving: DDPG trajectories always maintained time headway from 1s to 2s, while NGSIM headways ranged from 0s to 6s.The NGSIM range included dangerous headways below 1s and inefficient headways above 3s.
- C. Comfortable Driving: DDPG jerk values occupied a narrower range, −5 to 5 m/s3, than NGSIM values, −10 to 10 m/s3.The DDPG jerk distribution was also centered more closely to zero.
- C. Comfortable Driving: Smaller absolute jerk values correspond to more comfortable driving, supporting DDPG’s smoother velocity control than human NGSIM behavior.In an illustrative event, DDPG maintained nearly constant acceleration and produced low jerk values, unlike the driver’s frequent acceleration changes.
- C. Comfortable Driving: The model demonstrated safe, efficient, and comfortable driving through low dangerous-TTC percentages, 1s-to-2s headways, and smooth acceleration.
VII. DISCUSSION AND CONCLUSION
The study concludes that deep RL can control velocity during car following while directly optimizing safety, efficiency, and comfort, with performance that may exceed human drivers. It also identifies extensions involving objectives, reward design, network depth, and experience replay.
- The DDPG model combines human driving data with a reward function covering safety, efficiency, and comfort.
- Future extensions could add energy-saving objectives, adjust objective weights for user preferences, and replace the linear reward combination with nonlinear forms.
- The current DDPG implementation uniformly samples experience transitions, while prioritized replay is proposed to replay important transitions more often and improve learning efficiency.
- The model demonstrated safe, efficient, and comfortable driving and may perform better than human drivers.
- Reinforcement learning methods could contribute to autonomous driving system development.