Learning by Cheating

Dian Chen; Brady Zhou; Vladlen Koltun; Philipp Krähenbühl

Source-linked AI summary

Learning by Cheating

Dian Chen, Brady Zhou, Vladlen Koltun, Philipp Krähenbühl

arXiv:1912.12294v1Dec 27, 2019cs.ROcs.AIcs.CVcs.LG

TL;DR

Vision-based urban driving is difficult because imitation learning must connect complex visual input to actions. The paper trains a privileged state-based agent and uses it to supervise a purely vision-based sensorimotor agent, achieving complete success on the original CARLA benchmark and strong NoCrash results. The approach separates learning to act from learning to see, though physical-world transfer remains future work.

Problem
Direct imitation learning for vision-based urban driving remains difficult even with tens of millions of examples.
Method
The paper trains a privileged agent with ground-truth environment information, then trains a sensorimotor agent to imitate it using only visual input.
Results
The system achieves 100% success on all original CARLA tasks, sets a NoCrash record, and reduces infractions by at least an order of magnitude in most conditions versus CILRS.
Takeaways & Limitations
The decomposition partially decouples learning to act from learning to see and substantially outperforms prior state of the art on standard urban-driving benchmarks.
Takeaways & Limitations
The training procedure relies on simulation for privileged-state access, and transfer to the physical world is left for future work.

Abstract

from arXiv · show

Vision-based urban driving is hard. The autonomous system needs to learn to perceive the world and act in it. We show that this challenging learning problem can be simplified by decomposing it into two stages. We first train an agent that has access to privileged information. This privileged agent cheats by observing the ground-truth layout of the environment and the positions of all traffic participants. In the second stage, the privileged agent acts as a teacher that trains a purely vision-based sensorimotor agent. The resulting sensorimotor agent does not have access to any privileged information and does not cheat. This two-stage training procedure is counter-intuitive at first, but has a number of important advantages that we analyze and empirically demonstrate. We use the presented approach to train a vision-based autonomous driving system that substantially outperforms the state of the art on the CARLA benchmark and the recent NoCrash benchmark. Our approach achieves, for the first time, 100% success rate on all tasks in the original CARLA benchmark, sets a new record on the NoCrash benchmark, and reduces the frequency of infractions by an order of magnitude compared to the prior state of the art. For the video that summarizes this work, see https://youtu.be/u9ZCxxD-UUw

1 Introduction

Vision-based urban driving remains difficult because direct imitation learning must connect complex visual observations to actions. The paper decomposes learning into privileged-agent training followed by vision-based imitation, yielding stronger supervision and improved benchmark performance.

Motivation: Direct imitation learning remains difficult because agents must organize complex visual input while correlating it with demonstrated actions.A recent study reported that even tens of millions of examples did not produce satisfactory driving policies.
Approach: The two-stage method first trains a privileged agent on environment state and then trains a sensorimotor agent to imitate it using only legitimate visual input.The privileged agent observes layout and traffic participants, while the final agent uses a single forward-facing camera and no privileged information.
Approach: The decomposition partially separates learning to act from learning to see: the privileged agent focuses on acting, while the student learns visual control from its supervision.Direct state access removes the privileged agent’s need to learn perception during the first stage.
Advantages: The privileged agent’s compact bird’s-eye-view representation supports faster learning, better generalization, and effective data augmentation.The paper identifies the bird’s-eye view as an intermediate representation that facilitates generalization.
Advantages: The teacher can be queried from arbitrary states and provide adaptive on-policy supervision, turning passive demonstrations into DAgger-like training.This supervision is gathered through online rollouts of the sensorimotor agent.
Results: 100% success was achieved on all tasks in the original CARLA benchmark, while NoCrash performance improved by 18 percentage points in dense traffic.The approach also reduced infractions by at least an order of magnitude in most conditions versus CILRS.

2 Method

The method separates privileged-map learning from vision-based control: a privileged agent predicts waypoints, then trains a sensorimotor agent that uses images, speed, and commands. Both agents feed waypoint predictions to low-level PID controllers, while adaptive on-policy supervision and command-branch training improve student training.

Sensorimotor agent: The sensorimotor agent receives an RGB image, vehicle speed, and a high-level command, then predicts steering, throttle, and braking through waypoint-based control.The commands are follow-lane, turn left, turn right, and go straight.
Privileged agent: The privileged agent observes a ground-truth map containing lanes, traffic lights, vehicles, and pedestrians, plus speed and the high-level command, and predicts K waypoints.Its map is anchored at the agent’s position and uses binary indicators for environmental features.
Privileged agent: The privileged agent is trained by behavior cloning with an L1 loss between predicted and future trajectory waypoints.Training examples include maps, commands, velocities, positions, orientations, and generated future waypoints.
Privileged agent: The privileged agent’s map is augmented by random rotations and shifts, with the same transformations applied to waypoints to create perturbed configurations offline.This mimics camera-angle and trajectory perturbations without modifying expert data collection.
Sensorimotor agent: The sensorimotor agent imitates the privileged agent using an L1 loss over corresponding maps, images, and velocities, with camera-coordinate waypoints aligned through a perspective transformation.The student uses a similar heatmap-and-waypoint representation but receives only visual input rather than the privileged map.
Sensorimotor agent: On-policy training lets the student sample states from its own rollouts and receive supervision across all waypoints and command-conditioned branches.The privileged agent provides adaptive on-demand supervision in states reached by the student.
Low-level controller: Two independent PID controllers translate predicted or projected waypoints into steering, throttle, and braking commands.The longitudinal controller targets the average waypoint-passing velocity, while the lateral controller fits an arc and steers toward a projected waypoint.

3 Implementation details

The implementation uses lightweight convolutional waypoint predictors with different backbones for privileged and sensorimotor inputs, followed by PID-based control. Training combines augmentation, trajectory pretraining, and online DAgger supervision with loss-based resampling.

Low-level controller: The low-level lateral controller aims at a waypoint projected onto a fitted arc rather than steering directly toward an individual predicted point.Figure 4 illustrates projection of the second waypoint onto the arc.
Network architecture: The privileged agent uses a randomly initialized ResNet-18, while the sensorimotor agent uses an ImageNet-pretrained ResNet-34.Both networks use three up-convolutional layers and receive vehicle velocity at each up-convolutional layer.
Privileged agent: The privileged agent uses 192 × 192 inputs and 48×48 waypoint heatmaps, with random rotations of [−5, 5] degrees and shifts of [−5, 5] pixels.The shift corresponds to a 1m offset in the simulated world.
Sensorimotor agent: The sensorimotor agent processes a 384 × 160 RGB image and produces 96 × 40 heatmaps in camera coordinates.Its augmentations include pixel dropout, blurring, Gaussian noise, and color perturbations before projection into vehicle coordinates.
Training: Training first uses the trajectories for sensorimotor pretraining and then applies online DAgger with the privileged agent as an oracle.Pretraining accelerates overall training because online DAgger is slower, while the second stage alone reaches equal final accuracy.
Training: Critical states with higher loss are resampled more frequently during training.

4 Results

The evaluation compares the approach with baselines and prior systems on CARLA and NoCrash, including ablations, generalization, and infractions. The full method combines white-box supervision with on-policy training and achieves strong benchmark performance.

Ablation study: Direct one-stage imitation of the autopilot expert performs poorly in the experiments.The authors attribute part of this result to missing trajectory augmentations in the training data.
Ablation study: White-box supervision substantially improves performance by training all conditional branches, including actions for commands not taken on the demonstrated trajectory.The authors describe this as effective data augmentation and report that it helps decorrelate branch outputs.
Comparison to the state of the art: 100% success rate is achieved on all routes in the full-generalization setting of the original CARLA benchmark.The setting uses a test town and test weather; the comparison includes prior work evaluated on earlier CARLA versions.
Comparison to the state of the art: 85% success rate or higher is reached in all NoCrash traffic conditions, including the condition with dense traffic.The approach achieves 100% success rate without traffic and is compared with prior approaches in the test town.
Ablation study: 100% success rate is achieved in all conditions when white-box multi-branch supervision is combined with student on-policy rollouts.The ablation uses the CoRL2017 navigation condition in CARLA 0.9.5, test town, and test weather.
Infraction analysis: At least an order-of-magnitude reduction in infraction frequency is reported in most NoCrash conditions.Infractions are measured as traffic-light violations and collisions per 10 km against CILRS.

5 Conclusion

The paper concludes that training a privileged agent before training a purely vision-based system makes imitation learning for urban driving more effective. The authors report benchmark gains while noting that physical-world demonstration remains future work.

Conclusion: Two-stage training first teaches a privileged agent and then uses it as a teacher for a purely vision-based system.The decomposition partially decouples learning to act from learning to see.
Conclusion: The resulting vision-based driving system substantially outperforms the state of the art on standard benchmarks.The conclusion states that the advantages were validated experimentally.
Limitations and future work: Physical-world demonstration is left to future work because accessing ground-truth state for cheating is difficult outside simulation.The final policy is described as transferable using sim-to-real methods, but that transfer is not demonstrated here.
Future work: Combining the approach with reinforcement learning is proposed as a way to train systems beyond the capabilities of the initial expert.This is identified as an opportunity for future work.

Appendix A Additional details

The appendix details the map-view projection, simulation data collection, optimization settings, and image-model warm-up used in the training pipeline.

Map-view perspective transformation: Predicted camera-coordinate waypoints are projected onto a constant ground plane using camera geometry, avoiding depth estimation.The differentiable transformation supports end-to-end training.
Map-view perspective transformation: The projection shifts points backward by 4 meters to prevent near-vehicle points from being clipped at the image bottom.The camera is positioned at the vehicle hood and uses a 384 × 160 resolution.
Data collection: Privileged-agent training uses 157K training frames and 39K validation frames collected at 10 fps by a handcrafted autopilot.The data use four training weather conditions and include 100 other vehicles, plus 250 pedestrians in the 0.9.6 implementation.
Hyperparameters: All models are trained with Adam using an initial learning rate of 10^-4 and no weight decay.Batch sizes differ between the privileged and sensorimotor models in the 0.9.6 implementation.
Image model warm-up: The image model is warmed up for 1K iterations in projected image coordinates before two-stage training.This addresses exploding backward-pass gradients caused by randomly initialized spatial-argmax outputs.

Appendix B Additional experiments

Additional experiments test a simpler perception-control decomposition in which RGB images predict a privileged map representation before waypoint prediction. This baseline is vulnerable to out-of-distribution observations and underperforms the two-stage approach.

Map prediction baseline: Predicting a privileged map from RGB before acting resembles explicitly decoupling perception and control.The appendix presents this as an alternative to the paper’s approach.
Map prediction baseline: The perception network predicts a 192 × 192 privileged representation from resized RGB input using a ResNet34 backbone and L1 loss.The map is produced through five bilinear-upsampling, convolution, and ReLU layers.
Map prediction baseline: The action network predicts waypoints from the perception network’s output, with the perception weights frozen during action training.The action network uses the same architecture and training procedure as the privileged agent in the main experiments.
Results: Out-of-distribution observations during evaluation produce erroneous map predictions that cause the waypoint network to fail.Adding trajectory noise to 20% of frames broadens the states seen by the perception network, but both map-prediction agents perform significantly worse than the two-stage agent.
Results: The map-prediction comparison uses 150K-frame expert datasets and evaluates against two-stage LBC on the CoRL2017 Navigation task.The table evaluates CARLA 0.9.5 results with and without trajectory noise.

Appendix C Benchmark results

The appendix reports training-town performance for the CARLA CoRL 2017 and NoCrash benchmarks and examines performance across CARLA versions. It also compares CILRS on the Navigation Dynamic task with and without a working pedestrian autopilot.

CARLA version comparison: CILRS performance on Navigation Dynamic is compared between CARLA versions 0.8 and 0.9.5, respectively with and without a working pedestrian autopilot.The comparison uses test and training weathers across the two CARLA versions.
CARLA version comparison: CILRS performance in CARLA 0.9.5 matches the older version in test weathers and is slightly lower in training weathers.The authors interpret this as indicating that CARLA 0.9.5 does not make the task easier and report the higher numbers from the CILRS paper.