Source-linked AI summary
Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car
Mariusz Bojarski, Philip Yeres, Anna Choromanska, Krzysztof Choromanski, Bernhard Firner, Lawrence Jackel, Urs Muller
TL;DR
PilotNet addresses how an end-to-end driving network makes steering decisions from visual input. The paper develops a saliency visualization method and finds that the network identifies both obvious and subtle road-relevant features without hand-crafted rules.
Problem
The paper asks which image elements most influence PilotNet’s steering decisions and what the network learns from human driving data.
Method
The authors visualize salient image regions by propagating feature-map activations through the network to produce an input-sized mask.
Results
PilotNet highlights lane markings, road edges, cars, bushes, and atypical vehicle classes as relevant visual features.
Takeaways & Limitations
The results show that PilotNet learns human-interpretable driving cues, including subtle features difficult for engineers to anticipate and program.
Abstract
from arXiv · showhide
As part of a complete software stack for autonomous driving, NVIDIA has created a neural-network-based system, known as PilotNet, which outputs steering angles given images of the road ahead. PilotNet is trained using road images paired with the steering angles generated by a human driving a data-collection car. It derives the necessary domain knowledge by observing human drivers. This eliminates the need for human engineers to anticipate what is important in an image and foresee all the necessary rules for safe driving. Road tests demonstrated that PilotNet can successfully perform lane keeping in a wide variety of driving conditions, regardless of whether lane markings are present or not. The goal of the work described here is to explain what PilotNet learns and how it makes its decisions. To this end we developed a method for determining which elements in the road image most influence PilotNet's steering decision. Results show that PilotNet indeed learns to recognize relevant objects on the road. In addition to learning the obvious features such as lane markings, edges of roads, and other cars, PilotNet learns more subtle features that would be hard to anticipate and program by engineers, for example, bushes lining the edge of the road and atypical vehicle classes.
1 Introduction
PilotNet learns steering directly from road images paired with human steering commands, avoiding hand-coded driving rules. The paper introduces saliency analysis to understand which image regions influence its decisions.
- PilotNet outputs steering angles from front-facing road images paired with time-synchronized human-driver steering commands.
- The system was motivated by replacing hand-coded rules with knowledge learned by observing human drivers.
- The paper studies which image regions most influence steering and calls those salient image sections salient objects.
- The saliency method is presented as simple, fast on the test car’s NVIDIA DRIVE PX 2 computer, and nearly pixel-level in resolution.
- Training data include augmented images simulating off-center and off-orientation vehicle positions, with commands adjusted to return the vehicle toward lane center.
2 PilotNet Network Architecture
PilotNet combines convolutional feature extraction with fully connected steering control in an end-to-end network. Its learned design does not impose a hard boundary between perception and control.
- PilotNet has nine layers: one normalization layer, five convolutional layers, and three fully connected layers.
- The input image is split into YUV planes, and the first layer performs hard-coded normalization that is not learned.
- The convolutional layers were empirically configured for feature extraction, using strided early convolutions and non-strided later convolutions.
- The fully connected layers lead to an output control value equal to the inverse turning radius.
- End-to-end training means no hard boundary separates layers functioning primarily as feature extractors from those serving as the steering controller.
3 Finding the Salient Objects
The visualization method propagates feature-map activations toward the input to identify salient objects, then overlays the resulting mask on the camera image. The identified regions correspond to driving-relevant visual cues.
- Finding the Salient Objects: Salient objects are image regions corresponding to locations where feature maps have the greatest activations.
- Finding the Salient Objects: The method averages activations in each layer and upsamples higher-level maps using deconvolution with matching convolutional parameters.
- Finding the Salient Objects: At each stage, the upscaled map is multiplied by the averaged map from the layer below to form an intermediate mask, repeated until the input is reached.
- Finding the Salient Objects: The final normalized visualization mask highlights input pixels that contribute most to the network output and is overlaid on the original image.
- Finding the Salient Objects: PilotNet highlights car bases, lane lines, parked cars, and road-edge grass while ignoring a crosswalk line, mirroring human use of visual cues.
- Finding the Salient Objects: The visualization identifies a partially occluded construction vehicle that was reportedly absent from the training data.
4 Analysis
The analysis tests whether PilotNet’s identified salient regions actually control steering. Shifting salient objects changes steering nearly as much as shifting the full image, whereas shifting background pixels has a much smaller effect.
- Validation experiment: The experiment divides each input into salient regions and background pixels, then shifts these regions to measure their effects on steering.Salient regions are selected from visualization-mask locations above a threshold and dilated by 30 pixels.
- Visualization method: Figure 3 visualizes averaged feature maps and intermediate masks at successive network levels.At each layer, feature-map activations are averaged, and an upper-level map is combined with the map below after up-scaling.
- Salient objects: Figure 4 shows salient cues including car bases, lane lines, parked cars, and roadside grass across different road scenes.The examples include scenes without painted lane markings, where parked cars or grass indicate the road edge.
- Validation experiment: The shifted salient regions produce a linear steering change nearly as large as shifting the entire image.The plots compare shifts of salient pixels, nonsalient pixels, and all input pixels.
- Validation experiment: Shifting only background pixels has a much smaller effect on the steering angle.This contrast supports the claim that the method identifies the image regions most important for steering.
5 Conclusions
The paper presents a method for identifying the image regions PilotNet uses for steering and evidence that these regions correspond to meaningful driving cues. PilotNet also learns subtle cues, including roadside bushes and atypical vehicle classes, without hand-crafted rules.
- Contribution: The paper describes a method for finding the input-image regions by which PilotNet makes steering decisions.These regions are called salient objects.
- What PilotNet learns: The identified features correspond to human-interpretable driving cues while irrelevant image structures are ignored.The paper states that this capability is derived from data without hand-crafted rules.
- Examples: Figure 6 presents the PilotNet monitor, including the camera input and the salient regions identified in that image.The monitor example highlights a partially occluded construction vehicle that was reportedly absent from the training data in that pose.
- What PilotNet learns: PilotNet learns subtle features that would be difficult for engineers to anticipate and program, including roadside bushes and atypical vehicle classes.These examples extend beyond obvious cues such as lane markings, road edges, and other cars described in the paper context.