End-to-end Driving via Conditional Imitation Learning

Felipe Codevilla; Matthias Müller; Antonio López; Vladlen Koltun; Alexey Dosovitskiy

Source-linked AI summary

End-to-end Driving via Conditional Imitation Learning

Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, Alexey Dosovitskiy

arXiv:1710.02410v2Oct 6, 2017cs.ROcs.CVcs.LG

TL;DR

End-to-end imitation-learning policies cannot be guided to take a specific turn because observations alone may not reveal the driver’s intention. The paper conditions imitation learning on high-level commands and evaluates the resulting vision-based controllers in simulation and on a physical robotic vehicle. The command-conditional formulation significantly improves performance in both scenarios, with the branched controller completing 88% of Town 1 and 64% of Town 2 episodes.

Problem
Observations alone do not determine the intended action at urban intersections, limiting test-time control of imitation-learned driving policies.
Method
The paper trains a deep controller on visual observations, measurements, expert actions, and high-level categorical commands that expose useful information about expert intent.
Results
The branched command-conditional controller completed 88% of Town 1 and 64% of Town 2 episodes, while the formulation improved performance in simulation and on a physical vehicle.
Takeaways & Limitations
Command-conditional imitation learning yields a human-controllable vision-based driving policy responsive to navigational commands.

Abstract

from arXiv · show

Deep networks trained on demonstrations of human driving have learned to follow roads and avoid obstacles. However, driving policies trained via imitation learning cannot be controlled at test time. A vehicle trained end-to-end to imitate an expert cannot be guided to take a specific turn at an upcoming intersection. This limits the utility of such systems. We propose to condition imitation learning on high-level command input. At test time, the learned driving policy functions as a chauffeur that handles sensorimotor coordination but continues to respond to navigational commands. We evaluate different architectures for conditional imitation learning in vision-based driving. We conduct experiments in realistic three-dimensional simulations of urban driving and on a 1/5 scale robotic truck that is trained to drive in a residential area. Both systems drive based on visual input yet remain responsive to high-level navigational commands. The supplementary video can be viewed at https://youtu.be/cFtnflNe5fM

I. INTRODUCTION

Standard imitation learning can map visual inputs to driving controls, but observations alone do not determine the intended action at intersections. Conditional commands address this ambiguity and enable controllable vision-based driving, evaluated in simulation and on a physical truck.

Motivation: Earlier imitation-learning systems handled lane and road following but required human takeover for lane changes or turns between roads.
Motivation: At intersections, identical camera observations can correspond to turning left, turning right, or going straight, so image-to-control mapping is not a function.The missing information is the driver’s intended destination or internal state.
Approach: Conditional imitation learning supplies expert-intention commands during training and corresponding commands at test time to resolve perceptuomotor ambiguity.
Approach: Commands let a passenger or topological planner guide the trained network while it focuses on sensorimotor driving rather than planning.
Evaluation: The approach was evaluated in realistic urban-driving simulations and on a 1/5-scale robotic truck, with simulation supporting baselines and ablations.
Scope: The method remains end-to-end while adding commands that specify driver intent, using a predefined vocabulary rather than natural-language communication.

III. CONDITIONAL IMITATION LEARNING

The paper reformulates imitation learning by exposing useful information about the expert’s latent intent through a command input. The controller then maps observations and commands to actions, resolving ambiguities that observations alone cannot explain.

Standard imitation learning: Standard imitation learning assumes expert actions are fully explained by observations through a function E mapping observations to actions.This assumption works for simpler tasks such as lane following but breaks down when latent intent changes the correct action.
Standard imitation learning: In urban driving, the same observation can lead to different actions because the driver’s intended destination is not visible in the observation.
Conditional formulation: The method models expert internal state h and introduces a command c=c(h) that exposes useful information about intentions, goals, or prior knowledge.
Conditional formulation: Training data becomes observation-command-action triples, allowing the learner to use information about the expert’s latent state.
Controller loop: Figure 2 depicts a controller receiving observation o_t and command c_t, producing action a_t that affects the environment and advances time.

IV. METHODOLOGY

The practical implementation represents each observation as an image plus measurements and uses a deep network conditioned on a categorical command to predict driving actions.

Implementation: In the driving experiments, the network predicts a continuous two-dimensional action consisting of steering angle and acceleration.Negative acceleration represents braking or driving backwards.
Implementation: The implementation code is available in the CARLA simulator imitation-learning repository.

A. Network Architecture

The controller receives images, measurements, and a high-level command, then uses either a shared command-input architecture or command-selected specialist branches. The branched design forces different branches to learn command-specific sub-policies while sharing perception.

Command-conditioned controller: The network maps an image, measurements, and one-hot command to a continuous action containing steering angle and acceleration.The action space in the driving experiments is two-dimensional.
Command input: The command-input architecture independently processes image, measurement, and command features before concatenating them into a joint representation.A fully connected control module maps the joint representation to the action.
Branched architecture: The branched architecture removes the command module and uses the command as a switch selecting one specialist branch.Each branch receives the shared image and measurement processing.
Command input: The command-input architecture can also support goal-conditional learning by replacing the command with a vector pointing to the goal.This configuration is described as a baseline architecture.
Branched architecture: The branched design forces branches to learn sub-policies for different commands, such as lane following, right turns, and left turns.All branches share the perception stream.

B. Network Details

All controllers use the same core module architectures while differing in how modules and branches are configured. Training uses a weighted per-sample action loss and balanced command minibatches with Adam optimization.

Inputs: The observation is a 200×88 image, and the measurement is the car’s current speed when available.No speed measurement was used on the physical system because its estimates were very noisy.
Training objective: The per-sample loss compares predicted and ground-truth two-dimensional actions comprising steering angle and acceleration.The action vector is written as a pair of steering and acceleration values.
Optimization: Models are trained with Adam using minibatches of 120 samples and an initial learning rate of 0.0002.Command-conditional minibatches contain equal numbers of samples for each command.

C. Training Data Distribution

The training data is augmented to expose policies to disturbances and appearance variation, because expert-only trajectories may not teach recovery. Noise injection creates gradual drift, while online image transformations improve generalization without geometric changes.

Motivation: Expert-only trajectories can produce unstable policies because the model may not learn to recover from disturbance or drift.Training data should therefore include observations of recoveries from perturbations.
Perturbation recovery: The authors inject temporally correlated control noise and let the expert recover, simulating gradual drift away from the desired trajectory.Only the driver-provided control signal is used for training in the illustrated sequence.
Image augmentation: Online augmentation applies random subsets of appearance and corruption transformations during network training.Transformations include contrast, brightness, tone, blur, Gaussian noise, salt-and-pepper noise, and region dropout.
Image augmentation: Geometric transformations such as translation and rotation are excluded because control commands are not invariant to them.The restriction preserves the correspondence between image geometry and control commands.

V. SYSTEM SETUP

The system is evaluated in CARLA and on a 1/5-scale truck using central and lateral camera views, two-dimensional control, and four navigational commands. CARLA training and testing use separate towns, while human operators provide commands near intersections.

Shared sensing and control: Both platforms use one central camera and two lateral cameras rotated 30 degrees from the center, with steering and acceleration as recorded controls.Both control signals are scaled between -1 and 1.
Command interface: The command set contains continue, left, straight, and right, represented as one-hot vectors.The turn commands specify the intended action at the next intersection.
Command interface: Drivers indicate the intended command when the course of action becomes clear near an intersection, using a steering-wheel or remote-control button.This timing is intended to reflect turn indicators or mapping-app navigation instructions.
Simulated environment: CARLA provides a dynamic urban environment with traffic, buildings, vegetation, traffic signs, vehicles, and pedestrians.The simulator is implemented using Unreal Engine 4.
Simulated environment: Town 1 is used for training and Town 2 exclusively for testing, with different visual styles shown through maps and onboard views.The separate towns support evaluation on a distinct simulated environment.
Physical system: The physical platform is an off-the-shelf 1/5-scale truck equipped with an Nvidia TX2, three webcams, and a Pixhawk flight controller.The embedded computer communicates bidirectionally with the Pixhawk, which converts controls to low-level PWM signals.

1) Data collection:

The physical truck records synchronized multi-camera observations and control signals during human driving, while test-time operation uses only the central camera and supplied commands.

Data collection: During data collection, all three camera streams are synchronized with controls, GPS, and IMU measurements and recorded.Control signals pass through the TX2 to support noise injection and reproduce test-time delay.
Data collection: The physical system uses three command inputs: left, straight, and right.These commands correspond to the three-way switch available on the remote control.
Test-time operation: At test time, the model receives the central webcam image and a command, predicts controls end to end, and forwards them to the Pixhawk.The Pixhawk converts the predicted control into PWM signals for the speed controller and steering servo.
Camera inputs: All three cameras are used during training with adjusted steering commands, but only the central camera is used at test time.This training setup provides multiple viewpoints while preserving a single-camera test configuration.

1) Experimental setup:

The simulation evaluates command-conditional driving across training and unseen towns, comparing branched control against baselines and ablations. The branched controller reaches 88% of episodes in Town 1 and 64% in Town 2, while noise injection and augmentation materially affect performance.

Experimental setup: CARLA evaluates episodic driving to commanded destinations, measuring success rate and distance traveled without collisions or lane departures.Each episode starts at a new location and uses high-level turn commands from a topological planner.
Experimental setup: Town 1 is used for training, whereas Town 2 is reserved exclusively for testing.Evaluation uses 50 start-goal pairs per town, with locations at least 1 km apart.
Training data: The dataset contains 2 hours of human driving, including roughly 12 minutes with injected steering noise.The authors report that this relatively small noisy subset substantially stabilizes the learned policy.
Baseline comparison: 20% and 24%: standard imitation learning succeeds in Town 1 and Town 2, while goal-conditioned imitation learning reaches 24% and 30%, respectively.The goal-conditioned controller can shortcut toward the goal and veer off the road rather than learning a sequence of turns.
Results: 88% in Town 1 and 64% in Town 2: the branched command-conditional controller successfully completes more episodes than the baseline methods.The comparison includes standard imitation learning and goal-conditioned imitation learning.
Ablations: The ablation study finds that branching, steering-noise data, data augmentation, and sufficient network depth all contribute to performance.Without augmentation, the model completes no episode in the unseen Town 2; only 12 minutes of noisy data are used in training.

B. Physical System

The physical-system evaluation tests command-conditional driving on a 14-intersection residential route. The branched architecture performs best, while removing noise injection or augmentation causes substantial instability and route failures.

Physical-system setup: The physical truck is trained with 2 hours of remote-controlled driving and evaluated on a residential route containing 14 intersections.The route contains roughly equal numbers of left, straight, and right intersections.
Evaluation measures: Performance is measured by missed intersections, interventions, and course-completion time.A first missed turn allows rerouting for a second attempt; a second miss counts as missed and triggers manual intervention.
Main results: The branched architecture achieves the best performance among the physical-system variants.Table 2 compares branched and command-input architectures with models trained without noise injection or data augmentation.
Ablations: 8.67 versus 0.67 interventions: removing noise-injected data makes the physical model highly unstable.The no-noise model also misses almost 25% of intersections and takes twice as long to complete the course.
Ablations: Almost 40 interventions: removing data augmentation causes the truck to miss most intersections and frequently leave the lane.The no-augmentation model takes more than four times longer to complete the course.
Generalization: The authors attribute the augmentation benefit to generalization under changing real-world conditions such as weather and lighting.They emphasize that proper augmentation improves performance when training data are limited.

3) Generalization to new environments:

The approach generalizes qualitatively to previously unseen environments with very different appearance while remaining responsive to commands. The paper frames command-conditional imitation learning as a controllable driving approach, but notes that substantial progress is still needed for large-scale autonomous urban driving.

Generalization to new environments: The truck consistently follows the lane and responds to commands across three previously unseen environments with very different appearance.These tests assess qualitative generalization beyond naturally varying weather conditions.
Command-conditional imitation learning: Command-conditional imitation learning uses high-level commands during training to resolve perceptuomotor ambiguities and during testing to direct the controller.Commands provide a communication channel between the user or planner and the learned driving policy.
Evaluation: Experiments on a physical robotic vehicle and in realistic simulations show that the command-conditional formulation significantly improves performance in both scenarios.The evaluation spans vision-based driving in dynamic urban simulations and on a physical robotic vehicle.
Remaining limitations: Larger datasets and more sophisticated, higher-capacity architectures are still needed to support autonomous urban driving on a large scale.The authors describe the current results as encouraging but identify substantial room for progress.
Remaining limitations: The work does not address guidance of autonomous vehicles through unstructured natural-language communication.Natural-language communication is left as an important direction for future work.