Source-linked AI summary
End to End Learning for Self-Driving Cars
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, Karol Zieba
TL;DR
Reliable autonomous driving traditionally relied on manually decomposed perception and control pipelines. This paper trains an end-to-end CNN from camera pixels to steering and reports approximately 98% autonomous steering on a typical road test, with zero intercepts over 10 highway miles.
Problem
Prior end-to-end driving systems were not reliable enough to replace modular off-road approaches, motivating a robust alternative that avoids hand-designated features and rule collections.
Method
The paper trains a CNN to map camera video directly to steering commands, learning the complete driving pipeline from human steering supervision without manual decomposition.
Results
Approximately 98% autonomous steering was achieved on a typical Monmouth County drive, while 10 miles on the Garden State Parkway had zero intercepts.
Takeaways & Limitations
End-to-end CNNs can learn meaningful road features, including road outlines, from steering alone and follow roads across diverse conditions with less than 100 hours of driving data.
Takeaways & Limitations
Further work is needed to improve network robustness, verify robustness, and visualize internal processing.
Abstract
from arXiv · showhide
We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands. This end-to-end approach proved surprisingly powerful. With minimum training data from humans the system learns to drive in traffic on local roads with or without lane markings and on highways. It also operates in areas with unclear visual guidance such as in parking lots and on unpaved roads. The system automatically learns internal representations of the necessary processing steps such as detecting useful road features with only the human steering angle as the training signal. We never explicitly trained it to detect, for example, the outline of roads. Compared to explicit decomposition of the problem, such as lane marking detection, path planning, and control, our end-to-end system optimizes all processing steps simultaneously. We argue that this will eventually lead to better performance and smaller systems. Better performance will result because the internal components self-optimize to maximize overall system performance, instead of optimizing human-selected intermediate criteria, e.g., lane detection. Such criteria understandably are selected for ease of human interpretation which doesn't automatically guarantee maximum system performance. Smaller networks are possible because the system learns to solve the problem with the minimal number of processing steps. We used an NVIDIA DevBox and Torch 7 for training and an NVIDIA DRIVE(TM) PX self-driving car computer also running Torch 7 for determining where to drive. The system operates at 30 frames per second (FPS).
1 Introduction
The paper presents a CNN that learns the entire processing pipeline for steering an automobile, extending earlier end-to-end driving systems with the goal of robust public-road driving. Its motivation is to avoid hand-designed visual features and rule collections, addressing the limited reliability of prior systems.
- CNN foundations: CNNs learn features automatically from training examples rather than relying on hand-crafted feature extraction followed by classification.The convolution operation is especially effective for image recognition because it captures images’ 2D structure.
- Contribution: The paper’s CNN learns the entire processing pipeline needed to steer an automobile, building on the earlier DARPA Autonomous Vehicle project.DAVE used hours of human driving data in similar but not identical environments to train a sub-scale radio-control car.
- Related work: 1989 ALVINN demonstrated that an end-to-end neural network could steer a car on public roads, while this work applies more data, computation, and CNN technology.The authors describe their results as building on 25 years of advances since ALVINN.
- Prior limitations: DAVE’s mean distance between crashes was about 20 meters in complex environments, leaving it insufficiently reliable as a full alternative to modular off-road-driving approaches.This limitation motivated subsequent efforts to create a more robust public-road driving system.
- Motivation: The new effort seeks to avoid recognizing human-designated features such as lane markings, guard rails, and other cars, along with manually authored “if, then, else” rules.The paper reports preliminary results from this effort to build on DAVE for driving on public roads.
2 Overview of the DAVE-2 System
DAVE-2 collects time-synchronized camera images and human steering commands, augments them with off-center and rotated views, and trains a CNN to predict steering. After training, the network generates steering from a single center-camera video stream.
- Data collection: Three windshield-mounted cameras capture time-stamped video simultaneously with human steering commands obtained from the vehicle CAN bus and represented as 1/r.The 1/r representation is intended to make the system independent of vehicle geometry.
- Data augmentation: Training images pair single video frames with corresponding 1/r commands and include shifts from the lane center and rotations to teach recovery from mistakes.Human-driver data alone is insufficient because the car can otherwise slowly drift off the road.
- Data augmentation: Off-center views come from the side cameras, while intermediate shifts and rotations are simulated using an approximate viewpoint transformation based on horizon depth assumptions.The method assumes points below the horizon lie on flat ground and points above it are infinitely far away.
- Training and deployment: The CNN predicts a steering command, compares it with the desired command, and updates its weights through back propagation implemented in Torch 7.After training, the network generates steering from video images captured by a single center camera.
3 Data Collection
The training data covered diverse road types, geographic locations, lighting, and weather conditions. It was collected using two camera-equipped vehicles, with about 72 hours recorded by March 28, 2016.
- Road and environmental diversity: Data covered highways and roads with or without lane markings, residential streets, tunnels, and unpaved roads across several states.Most road data came from central New Jersey; highway data also came from Illinois, Michigan, Pennsylvania, and New York.
- Road and environmental diversity: Collection included clear, cloudy, foggy, snowy, and rainy conditions during both daytime and nighttime.
- Vehicles and driving procedure: About 72 hours of driving data had been collected by March 28, 2016 using a 2016 Lincoln MKZ or a 2013 Ford Focus with similarly positioned cameras.The system was designed without dependence on a particular vehicle make or model, and drivers were encouraged to remain fully attentive while driving normally.
4 Network Architecture
The network is a nine-layer end-to-end model that normalizes YUV input, extracts features through five convolutional layers, and produces an inverse-turning-radius control value through three fully connected layers. Its weights are trained by minimizing mean squared steering-command error, without a clean separation between feature extraction and steering control.
- Architecture: 9 layers comprise a normalization layer, 5 convolutional layers, and 3 fully connected layers receiving input split into YUV planes.The weights minimize mean squared error between network and human or adjusted steering commands.
- Architecture: The first layer performs hard-coded image normalization that is excluded from learning.Embedding normalization in the network allows the scheme to change with the architecture and use GPU acceleration.
- Architecture: The first three convolutional layers use strided 5×5 convolutions with 2×2 stride, while the final two use non-strided 3×3 convolutions.The convolutional configuration was selected empirically to perform feature extraction.
- Architecture: Three fully connected layers lead to an output control value representing the inverse turning radius.Although these layers are designed as a steering controller, end-to-end training prevents a clean division between feature extraction and control.
5 Training Details
Training data is filtered to lane-following frames, sampled to reduce redundancy, and balanced toward curves. The resulting data is augmented with random shifts and rotations to teach recovery from poor position or orientation.
- Frame selection: Lane-following training retains only frames where the driver stayed in a lane and discards other activities.Collected data is labeled by road type, weather, and driver activity before filtering.
- Frame selection: 10 FPS sampling limits highly similar images that provide little additional training value.The paper contrasts this rate with higher sampling rates that would include redundant frames.
- Data balancing: A higher proportion of curve frames reduces bias toward driving straight.
- Data augmentation: Artificial shifts and rotations train recovery from poor position or orientation, with magnitudes sampled from a zero-mean normal distribution.The augmentation standard deviation is twice the standard deviation measured from human drivers, but larger perturbations introduce undesirable artifacts.
6 Simulation
Section 6 evaluates trained CNNs in simulation by transforming synchronized human-driving video frames according to the CNN’s predicted steering, then tracking virtual vehicle behavior against calibrated lane-center ground truth.
- Simulation setup: The simulator uses forward-facing human-driving videos synchronized with the steering commands recorded during data collection.It generates images approximating what would appear if the CNN were steering instead.
- Simulation setup: Each video frame is manually calibrated to define a lane-center position called the “ground truth”.This accounts for human drivers not always driving in the lane center.
- Closed-loop simulation: The simulator adjusts each frame for departures from ground truth, feeds it to the CNN, and applies the returned steering command through a dynamic vehicle model.The CNN command and recorded human-driver command are both used in the simulation update.
- Closed-loop simulation: The simulator repeatedly modifies subsequent frames to reflect the virtual position produced by following the CNN’s steering commands.Each modified frame is fed back into the CNN, creating an iterative closed-loop process.
- Evaluation measures: When off-center distance exceeds one meter, the simulator triggers a virtual human intervention and resets the vehicle to the corresponding ground-truth position and orientation.It also records off-center distance, yaw, and distance traveled.
7 Evaluation
The networks were evaluated first in simulation and then in on-road tests, using autonomy as the primary performance measure. The CNN also learned useful road features from human steering angles without explicit road-outline training.
- Evaluation procedure: Evaluation proceeded in two stages: simulation followed by on-road testing.Simulation supplied steering commands on prerecorded routes before road deployment.
- Simulation: The simulator covered about three hours and 100 miles across highways, local roads, and residential streets in diverse lighting and weather.The routes were recorded in Monmouth County, NJ.
- Simulation: Autonomy was estimated from simulated human interventions triggered when the vehicle departed the center line by more than one meter.Each intervention was assumed to require six seconds for human takeover, re-centering, and self-steering restart.
- On-road tests: Approximately 98% of a typical Monmouth County drive was autonomous, excluding lane changes and turns between roads.The road test ran from the office in Holmdel to Atlantic Highlands.
- Learned representations: The CNN learned useful road features using only human steering angles as the training signal, without explicit training to detect road outlines.On an unpaved road, early feature-map activations clearly outlined the road; on a forest image, they were mostly noise.
8 Conclusions
The study demonstrates that CNNs can learn lane and road following end to end from less than a hundred hours of driving, without manually decomposing the task. The authors also identify remaining needs to improve and verify robustness and visualize internal processing.
- CNNs learned the entire lane- and road-following task without manual decomposition into detection, abstraction, planning, and control.The system was trained using less than a hundred hours of driving and operated on highways, local, and residential roads in varied weather.
- The system learned to detect road outlines without explicit training labels.
- Further work is needed to improve network robustness, verify it, and visualize internal processing steps.