PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning framework for Autonomous Driving

Ahmad El Sallab, Mohammed Abdou, Etienne Perot, Senthil Yogamani

arXiv:1704.02532v1stat.MLcs.LGcs.RO

TL;DR

Autonomous driving remains difficult to formulate as supervised learning because vehicle behavior depends on strong interactions and variable scenarios. The paper proposes an end-to-end deep reinforcement-learning framework combining recurrent networks and attention, and tests it in TORCS. The simulations demonstrate successful lane keeping and learned autonomous maneuvering, including supervised-to-autonomous apprenticeship learning with a speed limit.

  • Problem

    Autonomous driving involves strong interactions and variable behaviors, making exhaustive supervised-learning coverage difficult and planning hard to integrate with recognition and prediction.

  • Method

    The paper proposes an end-to-end deep reinforcement-learning framework that maps sensor and temporally aggregated environment states to driving actions using sensor fusion, recurrent networks, and attention.

  • Results

    The framework demonstrated successful lane keeping in TORCS with both discretized DQN and continuous-action setups, and apprenticeship learning achieved lane keeping with a speed limit.

  • Takeaways & Limitations

    The simulations support learning autonomous maneuvering with recurrent and attention-based deep reinforcement learning in the tested driving scenarios.

Abstract

from arXiv · show

Reinforcement learning is considered to be a strong AI paradigm which can be used to teach machines through interaction with the environment and learning from their mistakes. Despite its perceived utility, it has not yet been successfully applied in automotive applications. Motivated by the successful demonstrations of learning of Atari games and Go by Google DeepMind, we propose a framework for autonomous driving using deep reinforcement learning. This is of particular relevance as it is difficult to pose autonomous driving as a supervised learning problem due to strong interactions with the environment including other vehicles, pedestrians and roadworks. As it is a relatively new area of research for autonomous driving, we provide a short overview of deep reinforcement learning and then describe our proposed framework. It incorporates Recurrent Neural Networks for information integration, enabling the car to handle partially observable scenarios. It also integrates the recent work on attention models to focus on relevant information, thereby reducing the computational complexity for deployment on embedded hardware. The framework was tested in an open source 3D car racing simulator called TORCS. Our simulation results demonstrate learning of autonomous maneuvering in a scenario of complex road curvatures and simple interaction of other vehicles.

INTRODUCTION

Autonomous driving combines recognition, prediction, and planning, with planning requiring coherent integration of environment understanding and dynamics. The proposed framework uses deep reinforcement learning, recurrent networks, and attention to map raw sensor inputs to driving actions while handling partial observability and embedded-computation constraints.

  • INTRODUCTION: Recognition identifies surrounding components, whereas prediction integrates past information to model future environmental states.The paper associates recognition with deep learning and prediction with recurrent neural networks.
  • INTRODUCTION: Autonomous driving tasks comprise recognition, prediction, and planning, with planning integrating environmental understanding and dynamics to select safe future actions.Planning must avoid penalties and reach destinations through appropriate rewards.
  • INTRODUCTION: Planning is especially difficult because it must coordinate recognition and prediction while accounting for other vehicles, pedestrians, and roadworks.
  • INTRODUCTION: Attention models can filter raw sensory inputs to retain driving-relevant information, improving computational and memory efficiency.Examples include moving vehicles, free road space, and kerb positions.
  • INTRODUCTION: The proposed end-to-end model takes raw sensor inputs, integrates information over time with RNNs, and outputs driving actions using deep reinforcement learning.The framework is intended to handle partially observable scenarios and support real-time embedded systems.

REVIEW OF REINFORCEMENT LEARNING

The reinforcement-learning review frames driving as a sequential decision problem in which an agent maximizes accumulated rewards through a policy. It motivates RL for multi-agent driving because interaction variability makes exhaustive supervised-learning coverage difficult and because reward signals can coordinate system behavior.

  • REVIEW OF REINFORCEMENT LEARNING: Reinforcement learning seeks a policy that maximizes total accumulated rewards from the current state to a terminal state.
  • REVIEW OF REINFORCEMENT LEARNING: Driving is a multi-agent interaction problem because traffic involves variable vehicle configurations and uncertain driver behavior.These variations make exhaustive supervised-learning datasets challenging to design.
  • REVIEW OF REINFORCEMENT LEARNING: Standard autonomous-driving pipelines decouple perception and related subproblems before combining them in post-processing.The paper argues that isolated subproblems may be unnecessarily difficult and may not combine coherently.
  • REVIEW OF REINFORCEMENT LEARNING: An MDP is represented by states, actions, transition probabilities, a discount factor, and rewards.
  • REVIEW OF REINFORCEMENT LEARNING: Q-learning selects actions using an action-value function and updates its estimates recursively through temporal-difference learning.The agent repeatedly observes states and rewards until convergence or an episode limit.

DEEP REINFORCEMENT LEARNING

Deep reinforcement learning approximates action values with neural networks for large or continuous state spaces, while actor–critic methods address continuous actions. DQN and continuous-policy methods therefore differ primarily in how actions are represented and selected.

  • DEEP REINFORCEMENT LEARNING: The choice between DQN and continuous-action methods depends on whether the action space is discrete or continuous.
  • DEEP REINFORCEMENT LEARNING: DQN approximates the Q-function with a deep neural network when tabular state representations become impractical.Its training objective minimizes mean squared error of Q-values using differentiable optimization.
  • DEEP REINFORCEMENT LEARNING: Continuous-action algorithms learn an actor that maps states to actions and a critic that evaluates those actions.Both functions can be represented by neural networks under policy-gradient learning.

Deep Recurrent Reinforcement Learning

Deep recurrent reinforcement learning addresses partial observability by integrating information across observations over time. Attention mechanisms further focus computation on relevant learned features, motivating recurrent attention-based models for autonomous driving.

  • Deep Recurrent Reinforcement Learning: Partial observability invalidates the full-state Markov assumption used by Q-learning in scenarios such as object occlusion, tracking, mapping, and localization.
  • Deep Recurrent Reinforcement Learning: Recurrent neural networks integrate observations across time so the environment state can be inferred from multiple measurements.
  • Deep Recurrent Reinforcement Learning: Attention models select only part of CNN-extracted features for classification, learning the filtering process alongside the original model.The paper relates this approach to glimpse networks and DARQN.
  • Deep Recurrent Reinforcement Learning: The proposed framework combines recurrent information integration with attention-based extraction of relevant sensory information.The stated aim is to reduce computational complexity for autonomous-driving deployment.

PROPOSED DRL FRAMEWORK FOR AUTONOMOUS DRIVING

The proposed framework is an end-to-end, pipelined deep reinforcement learning system that maps aggregated environmental states to driving actions. It combines sensor fusion, spatial feature extraction, attention, and reinforcement-learning planning.

  • The pipeline takes environmental states and their temporal aggregations as inputs and produces driving actions.
  • Spatial aggregation: Spatial aggregation fuses sensor information and extracts representations of the car and surrounding environment.The car state includes position, orientation, velocity, and acceleration, while environmental state includes nearby objects and their motion.
  • Spatial aggregation: The DNN can learn sensor fusion by receiving raw sensor information and adjusting weights through gradient-based training.
  • Attention: Attention directs convolution toward relevant data regions, reducing input dimensionality and computation for embedded-system integration.

Recurrent temporal aggregation

Because the environment state is only indirectly observed through sensors, the framework integrates information over time. Recurrence, particularly LSTMs, is used to handle partially observable driving situations and retain relevant historical information.

  • Sensor readings provide indirect evidence of the environment state, so single snapshots are insufficient for decoding it.
  • Temporal information integration is mandatory because the environment state is revealed as the vehicle moves.
  • Recurrence enables the framework to handle partially observable driving situations.
  • LSTMs address RNN limitations in retaining long-term dependencies by controlling what information is kept, forgotten, and added to hidden states.
  • The temporal-planning pipeline follows the reinforcement-learning network after recurrent information aggregation.

RESULTS AND DISCUSSION

The framework was tested on lane keeping in TORCS using both discrete and continuous action formulations, as well as apprenticeship learning. Both reinforcement-learning setups achieved successful lane keeping, with continuous policy outputs producing smoother and better-performing actions.

  • The framework was tested for lane keeping in TORCS using simulated sensor inputs and end-to-end DQN-objective training.Inputs included track position and speed; outputs included steering, gear, acceleration, and braking.
  • Action formulations: Both tiled discrete actions and continuous DDAC actions produced successful lane keeping.
  • Action formulations: Removing replay memory in DQN accelerated convergence and improved performance, while tiled steering actions became more abrupt.
  • Action formulations: Continuous DDAC policy outputs smoothed the actions and provided better performance.
  • Apprenticeship learning: Apprenticeship learning produced successful lane keeping while enforcing a speed limit as supervision was gradually reduced.

CONCLUSION

The paper surveys deep reinforcement learning and proposes an end-to-end autonomous-driving pipeline that combines recurrent and attention models. Testing in TORCS showed successful lane-keeping learning, while future work targets broader simulated and real-driving scenarios.

  • The paper surveys recent advances in deep reinforcement learning.
  • The proposed pipeline integrates RNNs to account for partially observable driving scenarios.
  • Attention models use glimpse and action networks to direct CNN kernels toward input regions relevant to driving, improving computational complexity.
  • Testing on a lane-keeping assist algorithm produced successful learning in the simulated environment.
  • Future work includes deployment in a simulated environment with artificially controlled sensors and actuators as a potential extension toward real driving scenarios.

APPENDIX

The appendix provides supplementary DRL training and driving demonstration materials through YouTube links, including DQN, DDAC, and regression-neural-network examples.

  • APPENDIX: Supplementary material includes sample DRL training and demonstration sequences for review.The materials are offered through linked online videos.
  • APPENDIX: The linked training examples use DQN and DDAC approaches.Separate URLs are provided for each training method.
  • APPENDIX: A separate video demonstrates DRL driving using a regression neural network.The driving demonstration is provided alongside the DQN and DDAC training links.

Author Biography

The authors work across deep learning, machine learning, computer vision, reinforcement learning, embedded development, and automotive systems at Valeo and related organizations.

  • Author Biography: Ahmad El Sallab is Valeo’s Chief Engineer of Deep Learning in the Driving Active Safety unit.He has 11 years of machine-learning and deep-learning experience and 15 publications in the field.
  • Author Biography: Mohammed Abdou is a software engineer and researcher at Valeo Cairo pursuing a master’s degree focused on reinforcement learning.His work is in the Driving Active Safety unit’s systems and functions team.
  • Author Biography: Etienne is a research engineer applying deep learning to perception and control, with an emphasis on low-power embedded devices.He has worked in computer vision and machine learning at Valeo Driving Assistance Research for four years.
  • Author Biography: Senthil Yogamani leads computer-vision architecture research for next-generation automated parking systems at Valeo.He has over 10 years of computer-vision and machine-learning experience, including eight years in industrial automotive systems.
Loading 1704.02532v1…