Source-linked AI summary
End-to-End Deep Reinforcement Learning for Lane Keeping Assist
Ahmad El Sallab, Mohammed Abdou, Etienne Perot, Senthil Yogamani
TL;DR
The paper addresses the challenge of applying reinforcement learning to autonomous driving, where interaction with the environment makes supervised learning difficult. It develops discrete-action DQN and continuous-action DDAC methods, evaluates them in TORCS, and reports successful lane keeping while examining termination restrictions and convergence.
Problem
Autonomous driving remains difficult for reinforcement learning because it involves strong interaction with vehicles, pedestrians, roadworks, and other environmental elements.
Method
The paper develops end-to-end deep reinforcement-learning methods using discrete-action DQN and continuous-action DDAC, with environmental inputs and autonomous driving actions evaluated in TORCS.
Results
Both DQN and DDAC successfully learn lane keeping in TORCS, while DDAC produces smoother actions and better performance on curved sections.
Takeaways & Limitations
The study compares action-space choices for autonomous maneuvering and finds that adding termination conditions slows learning convergence.
Takeaways & Limitations
Restricted termination conditions limit exploration by ending episodes when the car becomes stuck or leaves the track, increasing the number of episodes needed for convergence.
Abstract
from arXiv · showhide
Reinforcement learning is considered to be a strong AI paradigm which can be used to teach machines through interaction with the environment and learning from their mistakes, but it has not yet been successfully used for automotive applications. There has recently been a revival of interest in the topic, however, driven by the ability of deep learning algorithms to learn good representations of the environment. Motivated by Google DeepMind's successful demonstrations of learning for games from Breakout to Go, we will propose different methods for autonomous driving using deep reinforcement learning. This is of particular interest as it is difficult to pose autonomous driving as a supervised learning problem as it has a strong interaction with the environment including other vehicles, pedestrians and roadworks. As this is a relatively new area of research for autonomous driving, we will formulate two main categories of algorithms: 1) Discrete actions category, and 2) Continuous actions category. For the discrete actions category, we will deal with Deep Q-Network Algorithm (DQN) while for the continuous actions category, we will deal with Deep Deterministic Actor Critic Algorithm (DDAC). In addition to that, We will also discover the performance of these two categories on an open source car simulator for Racing called (TORCS) which stands for The Open Racing car Simulator. Our simulation results demonstrate learning of autonomous maneuvering in a scenario of complex road curvatures and simple interaction with other vehicles. Finally, we explain the effect of some restricted conditions, put on the car during the learning phase, on the convergence time for finishing its learning phase.
1 INTRODUCTION
The paper frames autonomous driving as a difficult control problem and proposes evaluating discrete- and continuous-action deep reinforcement-learning approaches through exploration, model solution, and testing.
- Autonomous driving is challenging because it requires substantial driver-like skill, attention, and experience from machine intelligence.
- Deep learning has enabled strong representation learning, while reinforcement learning supplies planning for control tasks.
- The proposed workflow separates environment exploration, model solution, and testing, including comparison with other algorithms.
- The paper compares discrete-action and continuous-action algorithms and studies how restricted conditions affect learning convergence time.
2 DEEP REINFORCEMENT LEARNING
The paper reviews deep reinforcement learning for discrete and continuous action spaces, emphasizing DQN and actor–critic methods while noting partial observability in driving environments.
- The choice between discrete and continuous action spaces strongly affects which reinforcement-learning algorithms are applicable.
- Deep Q Networks (DQN): DQN approximates the Q-function with a deep neural network and minimizes mean squared error of estimated Q-values using gradient-based optimization.
- Deep Deterministic Actor Critic (DDAC): For continuous actions, DDAC learns an actor that maps states to actions and a critic that evaluates the selected action through policy-gradient methods.
- Autonomous-driving environments can be partially observable, requiring information integration across multiple observations rather than relying on a single revealed state.
- The review also discusses attention models and inverse reinforcement learning as related approaches for selecting relevant sensory features and learning from demonstrated behavior.
3 DRL SYSTEM FOR LANE KEEPING ASSIST
The proposed lane-keeping system maps environmental inputs to autonomous driving actions and compares discrete- and continuous-action deep reinforcement-learning methods for control behavior.
- The lane-keeping system constructs input states from the environment and outputs driving actions autonomously.
- Data Collection and Environment Exploration: Data collection and environment exploration use sensors such as cameras, LIDAR, and RADAR, with sensor fusion combining information about the car and surroundings.
- The system evaluates discrete-action and continuous-action algorithm categories, including Q-learning, DQN, and DDAC.
- DQN uses discretized actions, whereas DDAC uses an actor–critic structure for continuous actions and backpropagates error through the network.
4 RESULTS AND DISCUSSION
Lane-keeping was tested end-to-end in TORCS using sensor inputs and driving-action outputs. Both discrete and continuous approaches learned lane keeping, but continuous DDAC produced smoother and better performance, especially on curves.
- TORCS setup: The TORCS setup used track position and speed as inputs and produced steering, gear, acceleration, and braking outputs end-to-end.TORCS supplied vehicle, track, and opponent information for the simulation.
- Algorithm comparison: Both tiled-discrete actions and continuous DDAC actions achieved successful lane keeping.The discrete formulation tiled and discretized steering, gear, braking, and acceleration, while DDAC used continuous policy-gradient actions.
- Track performance: On straight track sections, Q-learning and DDAC had approximately similar survival performance.The comparison used the same track containing both straight and curved sections.
- Track performance: On curved sections, Q-learning produced delayed discrete actions and an abrupt path, whereas DDAC produced a smooth curve.The difference was attributed to discrete versus continuous action behavior.
- Overall result: DDAC performed well on both straight and curved sections, while Q-learning was judged unsuitable for the curved section.The reported conclusion favors DDAC for the combined track conditions.
4.1 TORCS Termination Conditions
The TORCS experiments varied how training episodes terminated after undesirable vehicle states. Conditions included no restricted termination, out-of-track termination, stuck termination, and a hybrid condition combining both.
- Conditions: The termination-condition study considered no termination, out-of-track, stuck, and combined out-of-track-with-stuck conditions.Termination criteria determine when training converges and when TORCS is relaunched for a new episode.
- No termination: Without restricted termination, the car received negative rewards for leaving the track or becoming stuck while continuing to explore.The only stated termination was when the car became horizontally positioned on the track after attempting a sharp curve.
- Out of track: Under out-of-track termination, leaving the track produced a high negative reward and immediately started a new episode.This condition prevented the car from leaving the track during learning.
- Stuck: The stuck condition terminated and relaunched TORCS when the car stalled or became stuck.The broader setup gave the car about 100 time steps before applying the stuck-related termination criterion.
- Stuck: The stuck threshold was tied to vertical speed, with speedX reaching 5 km/h treated as a tuning criterion for low-speed behavior.The stated purpose was to prevent the car from approaching low speeds during learning.
- Combined condition: The hybrid condition started a new episode whenever either leaving the track or becoming stuck occurred.It combined both restricted termination criteria in one experiment.
4.2 Convergence Time vs. Termination Conditions
Fewer termination restrictions were expected to shorten convergence because episodes lasted longer and enabled broader track exploration. Adding out-of-track and stuck restrictions progressively increased the expected number of episodes.
- No Termination Condition: The unrestricted experiment was expected to converge in a low number of episodes.Longer episodes allowed the car to complete laps and explore more of the track before termination.
- Out of Track: The out-of-track condition was expected to converge in a moderate number of episodes, exceeding the basic experiment.Restarting after leaving the track required more episodes to explore and complete a lap.
- Stuck: The stuck condition was also expected to require a moderate number of episodes beyond the basic experiment.Restarting after becoming stuck increased the episodes needed to explore the full track.
- Out of Track with Stuck: The combined out-of-track and stuck condition was expected to require the highest number of episodes.Both restrictions could trigger a restart and constrained autonomous exploration of the track.
4.3 Experimental Results for the Termination Conditions
The experiments measure learning by whether the car completes 10 laps per episode and compare convergence under different termination conditions. Learning is fastest without termination conditions and progressively slower as restrictions are added.
- 10 laps per episode defines successful learning of the track across the four termination-condition experiments.
- No termination conditions converge faster than stuck termination, out-of-track termination, and their combination.
- Restricted termination conditions trade faster avoidance of repeated failures for the loss of rapid convergence.
5 CONCLUSION
The paper presents a deep reinforcement learning lane-keeping system using discrete and continuous action categories. It compares algorithm performance and studies how termination restrictions affect learning convergence.
- The proposed lane-keeping system uses algorithms with either discrete or continuous driving actions.
- The study compares Q-learning and DDAC on performance and action smoothness over the same track.
- Adding more termination conditions slows convergence time for learning the same track.
APPENDIX
The appendix provides supplementary training and demonstration sequences for the paper’s deep reinforcement learning experiments. It includes videos for DQN, DDAC, and regression-neural-network lane keeping.
- Supplementary material provides sample deep reinforcement learning training and demonstration sequences.
- The listed videos cover DQN training, DDAC training, and lane keeping with a regression neural network.