Source-linked AI summary
Deep Reinforcement Learning for Autonomous Driving
Sen Wang, Daoyuan Jia, Xinshuo Weng
TL;DR
Autonomous driving is difficult for deep reinforcement learning because it combines complex real-world states, continuous fine-grained actions, and functional-safety requirements. The paper addresses these challenges by adapting DDPG to TORCS with selected sensors, a custom rewarder, and tailored actor–critic networks, and reports fast simulator driving while maintaining functional safety.
Problem
Autonomous driving presents complex state spaces, continuous fine-grained actions, and functional-safety requirements that differ from many successful game environments.
Method
The paper applies DDPG in TORCS using selected sensor inputs, a custom rewarder, and tailored actor–critic network architectures.
Results
The agent runs fast in the simulator while ensuring functional safety, with evaluation across TORCS modes containing different visual information.
Takeaways & Limitations
Synthetic simulation enables the paper to investigate autonomous-driving navigation without physical damage while testing continuous-control learning.
Takeaways & Limitations
During experiments, episodes were manually terminated when the car rushed off the track and became stuck rather than waiting for recovery.
Abstract
from arXiv · showhide
Reinforcement learning has steadily improved and outperform human in lots of traditional games since the resurgence of deep neural network. However, these success is not easy to be copied to autonomous driving because the state spaces in real world are extreme complex and action spaces are continuous and fine control is required. Moreover, the autonomous driving vehicles must also keep functional safety under the complex environments. To deal with these challenges, we first adopt the deep deterministic policy gradient (DDPG) algorithm, which has the capacity to handle complex state and action spaces in continuous domain. We then choose The Open Racing Car Simulator (TORCS) as our environment to avoid physical damage. Meanwhile, we select a set of appropriate sensor information from TORCS and design our own rewarder. In order to fit DDPG algorithm to TORCS, we design our network architecture for both actor and critic inside DDPG paradigm. To demonstrate the effectiveness of our model, We evaluate on different modes in TORCS and show both quantitative and qualitative results.
1 Introduction
Autonomous driving combines highly complex perception with continuous, fine-grained control, making successes from simpler deep reinforcement learning environments difficult to transfer. The paper explores this problem in TORCS using DDPG, selected sensors, a custom rewarder, and tailored actor–critic networks.
- Challenges: Deep reinforcement learning successes often involve limited discrete actions and simpler visual state spaces than autonomous driving.Examples include games with only four actions and visually easy-to-understand board states.
- Challenges: Lidar and IMU can reconstruct 3D surfaces but are expensive, heavy to deploy, and do not themselves provide environmental understanding.These limitations constrain the broader popularity of autonomous driving technology.
- Challenges: Autonomous driving requires perception under substantial environmental variation and simultaneous continuous actions such as steering and braking.The paper highlights variation in object appearance, backgrounds, viewpoints, and changing environments.
- Motivation: Synthetic driving simulators provide a way to study navigation policies without physical damage while reinforcement learning methods address real-world complexity.The project specifically investigates autonomous driving within synthetic simulators.
- Approach: The paper applies DDPG in TORCS with selected sensor inputs, a rewarder encouraging speed, safety, and road-center adherence, and custom actor–critic architectures.Effectiveness is evaluated across TORCS modes containing different visual information.
2 Related Work
Prior work applies reinforcement learning and deep learning to autonomous driving and develops value-based and policy-based algorithms for control. The paper situates DDPG among methods designed for continuous action spaces and end-to-end policy learning.
- Autonomous Driving: Earlier autonomous-driving studies used Q-learning in TORCS or CNNs to decompose driving into car-detection and lane-detection tasks.The cited TORCS work evaluates heuristic exploration, while the CNN-based work uses a real-world highway dataset.
- Reinforcement Learning: Value-based reinforcement learning includes vanilla Q-learning and deep variants such as DQN, Double DQN, and Dueling DQN.These methods are described as successful across a variety of games.
- Reinforcement Learning: Policy-based methods learn policies directly by outputting actions from current states, unlike value-based methods.The distinction is presented as a difference in how actions are produced.
- Reinforcement Learning: Deterministic policy gradients efficiently address continuous action spaces, and DDPG combines them with deep Q-learning and actor–critic ideas.The cited DDPG work is characterized as achieving end-to-end policy learning.
3 Methods
The method applies DDPG to autonomous driving in TORCS, combining deterministic policy gradients, actor-critic learning, deep networks, selected sensors, and a custom reward function. The design targets continuous control while encouraging speed, road-center tracking, and stable learning.
- 3.1 Deterministic Policy Gradient (DPG): DDPG is selected because autonomous driving uses continuous action spaces, making policy-based control preferable to value-based methods.The paper notes that steering and acceleration vary continuously and that random exploration can have undesirable consequences.
- 3.1 Deterministic Policy Gradient (DPG): Deterministic policy gradients reduce sampling requirements by mapping each state to a fixed action instead of integrating over the entire action space.This is presented as making gradient estimation more efficient than the stochastic-policy version.
- 3.1 Deterministic Policy Gradient (DPG): The actor produces actions from observations, while the critic estimates Q values and updates through TD learning; the actor is updated by policy gradients.The DPG setup uses off-policy learning with separate actor and critic roles.
- 3.2 Deep Deterministic Policy Gradient (DDPG): The model uses separate actor and critic networks, with the critic taking observations and actions as input and estimating rewards for each action.The network architecture uses ReLU hidden layers and merges action information at the second hidden layer.
- 3.2 Deep Deterministic Policy Gradient (DDPG): The custom TORCS reward encourages speed along the track while penalizing transverse speed and deviation from the road center.The reward assigns weights α, β, and γ to these terms, including penalties based on |trackPos| and Vx|trackPos|.
4 Experiments
Experiments trained the DDPG agent in TORCS and evaluated it across training and compete modes. Performance generally improved, but environmental failures and driving instability remained visible during training.
- Experimental setup: The experiments used TORCS training and compete modes, with training on Aalborg for about 200 episodes and evaluation against nine competitors.Episodes terminated when the car left the track or faced the wrong direction, with a maximum length of 60000 iterations.
- Experimental setup: The model gradually stabilized in training, although it initially drove shakily and frequently collided with walls.The compete-mode setup introduced other cars whose sensor effects differed from training conditions.
- Compete-mode behavior: The agent learned to release the accelerator before corners, reducing drifting-related wrong-direction failures, and often overtook competitors near or after turns.It still did not learn to avoid collisions with competitors because competitors were absent during training.
- Training curves: Figure 5 tracks speed, step gain, travel distance, total reward, center-distance variance, and episode length across aligned training episodes.These measures cover both driving performance and stability over training.
- Training curves: Average speed and step gain stabilized after about 100 episodes, while center-distance variance decreased and stabilized after about 150 episodes.The later stabilization indicates that driving remained unstable after speed and episode rewards had stabilized.
- Training limitations: Sudden performance drops were associated with cars becoming stuck, including repeated failures at the same map location, and the authors manually terminated such episodes.The paper attributes these failures to imprecise out-of-track detection in TORCS rather than the learning algorithm.
5 Conclusion
The paper bridges autonomous driving and reinforcement learning by training an agent with DDPG in TORCS using continuous actions, tailored rewards, and actor-critic networks. The resulting agent runs fast while maintaining functional safety in the simulator.
- DDPG trains the autonomous-driving agent in the TORCS simulator.
- The system uses selected TORCS sensor information as inputs and defines actions in a continuous domain.
- The method includes dedicated actor and critic network architectures within the DDPG paradigm.
- The agent runs fast in the simulator while ensuring functional safety.