Source-linked AI summary
Learning to Drive in a Day
Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, Amar Shah
TL;DR
Autonomous-driving research has often relied on maps, explicit rules, or direct supervision, motivating a more general learning framework. The paper applies continuous, model-free deep reinforcement learning on-vehicle and demonstrates rapid real-world lane-following learning, while identifying important reward limitations.
Problem
Existing autonomous-driving approaches rely heavily on maps, explicit rules, or direct supervision, leaving a need to examine reinforcement learning as a more general framework.
Method
The paper poses driving as a Markov decision process and learns a policy from monocular camera images using a continuous, model-free deep reinforcement-learning algorithm with a distance-before-intervention reward.
Results
The agent learned real-world lane following in a handful of trials, with under thirty minutes of training on on-board computers.
Takeaways & Limitations
Deep reinforcement learning is presented as a viable approach to autonomous driving using an image-based input and a reward that requires no maps or additional environmental information.
Takeaways & Limitations
The distance-without-intervention reward does not condition on a navigation goal and is extremely sparse, weakening the training signal as interventions become less frequent.
Abstract
from arXiv · showhide
We demonstrate the first application of deep reinforcement learning to autonomous driving. From randomly initialised parameters, our model is able to learn a policy for lane following in a handful of training episodes using a single monocular image as input. We provide a general and easy to obtain reward: the distance travelled by the vehicle without the safety driver taking control. We use a continuous, model-free deep reinforcement learning algorithm, with all exploration and optimisation performed on-vehicle. This demonstrates a new framework for autonomous driving which moves away from reliance on defined logical rules, mapping, and direct supervision. We discuss the challenges and opportunities to scale this approach to a broader range of autonomous driving tasks.
I. INTRODUCTION
The paper frames autonomous driving as a reinforcement-learning problem that can operate without maps and explicit rules. It demonstrates that deep reinforcement learning can learn driving policies for a real vehicle using on-board computation.
- Existing autonomous-driving approaches often rely on formal logic and annotated 3D maps, creating dependence on external mapping infrastructure.
- The paper advocates driving systems that understand the immediate environment without maps or explicit rules, while following high-level route commands.
- The authors pose autonomous driving as a Markov decision process and use reinforcement learning to learn driving policies.
- The proposed actor-critic agent maximises distance travelled before a safety driver intervenes.
- The system uses a continuous deep reinforcement-learning algorithm with exploration and optimisation performed on-board.
- The work reports the first demonstration of a deep reinforcement-learning agent driving a real car.
II. RELATED WORK
Related work includes mapping-based, modular, imitation-learning, and reinforcement-learning approaches to autonomous driving. The paper positions its contribution as deep reinforcement learning for image-based lane following with a sparse reward.
- Mapping approaches: Mapping-based systems traditionally combine perception, state estimation, mapping, planning, and control as separately engineered components.
- Mapping approaches: These modular mapping approaches remain a complex systems-engineering challenge for commercial autonomous-driving efforts.
- Imitation learning: Imitation learning jointly optimises end-to-end models from expert demonstrations, but expert examples are unavailable for every possible scenario.
- Reinforcement learning: Reinforcement learning formulates decision-making as a Markov decision problem with states, actions, transitions, rewards, and a discount factor.
- Reinforcement learning: Model-free reinforcement learning estimates action values directly, trading potential generality for lower data efficiency than model-based methods.
- Reinforcement learning: Unlike prior GPS-trajectory work using dense tracking-error rewards, this paper learns image-based lane following with a sparse reward.
A. Driving as a Markov Decision Process
The paper formulates autonomous driving as an MDP whose state, actions, and rewards are designed for simple, general driving tasks. It uses compact observations and a reward based on forward progress until traffic-rule violations.
- The paper formulates driving as an MDP by defining the state space, action space, and reward function while leaving vehicle transitions to the simulator or vehicle mechanics.
- State space: A monocular camera image, vehicle speed, and steering angle are sufficient observations for the simple driving tasks considered.The observation itself is treated as an adequate approximation of the Markov state.
- State space: The image can be processed either directly with convolutions or through a compressed VAE representation, whose reinforcement-learning performance is compared experimentally.The VAE is trained online using five purely random exploration episodes.
- Action space: The action space can represent driving controls directly or reparameterize throttle as a speed set-point handled by a classical controller.
- Reward function: The reward is forward speed, and episodes terminate after traffic-rule infractions, making state value correspond to average distance travelled before an infraction.The authors note that command-conditional rewards may be needed to discourage avoiding difficult manoeuvres.
B. Reinforcement Learning Algorithm – Deep Deterministic Policy Gradients
The paper uses DDPG, a continuous-action, model-free reinforcement-learning algorithm, and adapts its online training for vehicle deployment. Exploration noise balances state-action coverage against safety-driver comfort.
- DDPG is selected as a simple continuous-action, model-free algorithm to solve the driving MDP without task-specific adaptation.
- DDPG uses a critic to estimate action values and an actor to approximate a Q-optimal policy from experience tuples stored in a replay buffer.
- Training is online, with prioritised experience replay sampling tuples in proportion to the critic’s temporal-difference error.
- Exploration: Exploration adds Ornstein-Uhlenbeck process noise to the actor’s actions to collect diverse state-action data outside the optimal policy’s narrow distribution.
- Exploration: Lower-variance, strongly mean-reverting noise is easier for the safety driver to anticipate, whereas higher variance improves state-action space coverage.
- Figure 2 outlines the workflow and architecture for training the algorithm from safety-driver feedback.
C. Task-based Training Architecture
The real-vehicle training system is organized as an interactive state machine that accommodates driver intervention, manual resets, invalid episodes, and on-demand stopping.
- Real-world deployment requires modified training procedures to account for safety-driver intervention and external variables affecting learning.
- The state machine defines train, test, undo, and done tasks, enabling interactive, stateful execution instead of a fixed episode schedule.
- Train and test: Training adds exploration noise and updates the model, whereas testing executes the model output directly; early episodes skip optimisation to encourage exploration.The experiment continues until test reward stops increasing.
- Episodes end when the safety driver intervenes, after which the driver manually resets the vehicle while the model is optimised between episodes.
- Undo and done: The undo task discards episodes terminated for valid non-failure reasons, while the done task permits graceful termination of the interactive experiment.Other drivers entering the road are given as an example of a valid non-failure termination.
IV. EXPERIMENTS
The experiments use lane-following as the showcase task in both simulation and a real vehicle, using image input without explicit lane-position knowledge. Both settings use a small convolutional network.
- Lane-following is evaluated in simulation and on a real vehicle from image input without knowledge of lane position.The task is presented as central to driving and associated with the seminal ALVINN system.
- The model is a small convolutional neural network with four convolutional layers shared between the actor and critic models.The layers use 3 × 3 kernels, stride 2, and 16 feature dimensions.
A. Simulation
The simulator generates varied country-road environments for image-based lane-following experiments. It enabled tuning of reinforcement-learning settings and reliable learning from raw images within 10 training episodes.
- The 3D simulator uses a generative model for country roads with varied weather conditions and road textures.Procedural generation also varies road texture, lane markings, and road topology across episodes.
- Each episode randomly varies road texture, lane markings, and road topology while training uses a forward-facing driver-view image.
- The simulator was essential for tuning learning rates, optimization steps, and termination procedures, with conservative termination producing a better policy.It also supported continuous actions and indicated that DDPG was suitable.
- 10 training episodes were sufficient to reliably learn lane following in simulation from raw images.A compressed VAE state representation offered little advantage in simulation.
B. Real-world driving
The real-world experiments used a modified electric vehicle on a 250-meter road section with onboard sensing and computation. Reinforcement learning solved the task in a handful of trials, while an online-trained VAE improved reliability compared with raw pixels.
- The real-world experiment uses a 250 meter road section where episodes begin at the start and reset after unrecoverable lane deviation.Environmental factors are uncontrolled, so real-time safety and control systems are required.
- Using a VAE with DDPG greatly improves data efficiency over DDPG from raw pixels in the real-world experiments.The figure identifies the experimental route as 250m.
- The modified Renault Twizy uses a single monocular forward-facing camera and onboard computation to control braking, steering, and throttle.
- A handful of trials were sufficient for reinforcement learning to solve the real-world lane-following problem.Using 250 optimization steps with batch size 64 took approximately 25 seconds, while resetting the car took approximately 10 seconds.
V. DISCUSSION
The discussion presents on-vehicle deep reinforcement learning as a viable approach for lane following, while identifying state representation, reward design, and broader scaling as open challenges.
- The vehicle learned to lane follow with under thirty minutes of training using only on-board computers.
- A reward based on speed and termination when the safety driver intervenes requires no maps or further environmental information.
- The framework is presented as general enough to scale to more complex driving tasks with more data, vehicles, and larger models.
- Scaling reinforcement learning to autonomous driving will require translating advances from the wider field and improving core reinforcement learning algorithms.
- A better state representation is identified as a development priority, with the experiments showing that a Variational Autoencoder improves DDPG performance on a real vehicle.
- Unsupervised state encoding alone is unlikely to suffice because reward-linked information is needed to identify which image elements matter for learning.