PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Urban Driving by Imitating a Reinforcement Learning Coach

Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, Luc Van Gool

arXiv:2108.08265v3cs.CVcs.RO

TL;DR

End-to-end autonomous-driving imitation learning needs dense, informative supervision, but human experts are poor coaches and existing automated urban-driving experts can be rule-heavy and suboptimal. The paper trains Roach, a reinforcement-learning expert mapping bird’s-eye-view images to continuous actions, then uses its richer supervision to train camera-based imitation-learning agents. Roach sets a new CARLA performance upper bound, while its supervised single-camera agent reaches expert-level NoCrash performance and state-of-the-art CARLA LeaderBoard performance.

  • Problem

    End-to-end driving imitation learning relies on expert demonstrations, but human experts provide limited dense on-policy supervision and existing automated urban-driving experts can be suboptimal.

  • Method

    The paper trains Roach, a reinforcement-learning expert that maps bird’s-eye-view images to continuous low-level actions and provides action distributions, values, and latent features for imitation learning.

  • Results

    Roach sets a new performance upper bound on CARLA, while a single-camera imitation-learning agent supervised by it achieves expert-level NoCrash performance and state-of-the-art CARLA LeaderBoard performance.

  • Takeaways & Limitations

    Informative supervision from a learned reinforcement-learning coach can bring a monocular end-to-end imitation-learning agent to expert-level or state-of-the-art performance within the reported CARLA evaluations.

  • Takeaways & Limitations

    The reported evaluation is conducted on CARLA, and applying Roach to real-world on-policy data requires addressing several sim-to-real gaps.

Abstract

from arXiv · show

End-to-end approaches to autonomous driving commonly rely on expert demonstrations. Although humans are good drivers, they are not good coaches for end-to-end algorithms that demand dense on-policy supervision. On the contrary, automated experts that leverage privileged information can efficiently generate large scale on-policy and off-policy demonstrations. However, existing automated experts for urban driving make heavy use of hand-crafted rules and perform suboptimally even on driving simulators, where ground-truth information is available. To address these issues, we train a reinforcement learning expert that maps bird's-eye view images to continuous low-level actions. While setting a new performance upper-bound on CARLA, our expert is also a better coach that provides informative supervision signals for imitation learning agents to learn from. Supervised by our reinforcement learning coach, a baseline end-to-end agent with monocular camera-input achieves expert-level performance. Our end-to-end agent achieves a 78% success rate while generalizing to a new town and new weather on the NoCrash-dense benchmark and state-of-the-art performance on the challenging public routes of the CARLA LeaderBoard.

1. Introduction

The paper frames expert demonstrations as a central challenge for end-to-end autonomous driving, especially because supervised imitation learning suffers from covariate shift and human experts provide limited dense on-policy supervision. It proposes Roach, a reinforcement-learning coach that uses privileged simulation information to generate richer supervision for camera-based imitation-learning agents.

  • Supervised imitation learning for end-to-end driving is vulnerable to covariate shift, motivating a focus on improving expert demonstrations.
  • Automated experts can generate large-scale dense labels for both on-policy and off-policy data, but may depend on privileged information unsuitable for deployment.
  • CARLA’s Autopilot provides ground-truth state but is limited as an urban-driving expert, motivating a learned alternative.
  • Roach maps bird’s-eye-view images to continuous actions and is trained as a reinforcement-learning expert for CARLA.
  • Roach supervises imitation-learning agents with action distributions, value estimations, and latent features beyond deterministic action labels.
  • A single-camera end-to-end imitation-learning agent supervised by Roach achieves state-of-the-art performance on NoCrash and CARLA LeaderBoard evaluations.

2. Related Work

Related work spans end-to-end imitation learning, structured intermediate representations, reinforcement learning, and imitation from automated experts in CARLA and related driving settings. Roach differs by targeting a learned reinforcement-learning expert and using its supervision for end-to-end imitation.

  • End-to-End IL: End-to-end imitation-learning work in CARLA includes behavior cloning, branched action heads, and DAGGER-based critical-state sampling.
  • Mid-to-X IL: Mid-to-X methods reduce direct camera-to-action complexity by using structured intermediate representations such as segmentation masks and driving affordances.
  • RL: CARLA reinforcement-learning studies include A3C, DDPG, and value-based or actor-critic methods using BEV inputs or pretrained visual encoders.
  • RL: Roach is distinguished from the cited reinforcement-learning methods by achieving high sample efficiency without expert demonstrations.
  • IL with Automated Experts: Automated-expert approaches train agents by imitating on-policy actions from experts with privileged or expensive sensing, including CARLA proxy-expert methods.

3. Method

Roach is an end-to-end RL expert that uses privileged BEV and measurement inputs to predict continuous actions, then supplies IL agents with diverse supervision signals. Its design combines PPO training, structured action distributions, event-specific exploration, and losses for transferring values, action distributions, and latent features.

  • 3.1. RL Coach: Roach maps BEV images and measurement vectors to an action distribution, while a value network estimates scalar value from the same inputs.The policy and value networks are parameterized separately and share the BEV-plus-measurement input design.
  • 3.1. RL Coach: Ground-truth BEV semantic segmentation encodes drivable areas, routes, road markings, dynamic agents, pedestrians, and traffic controls to reduce problem complexity.The representation also includes temporal image channels and renders stop-sign state after the ego-vehicle stops.
  • 3.1. RL Coach: Roach directly predicts bounded steering and acceleration distributions with Beta distributions, avoiding controller tuning, action clipping, and squashing layers.Positive acceleration represents throttle and negative acceleration represents brake.
  • 3.1. RL Coach: PPO training combines clipped policy-gradient and maximum-entropy objectives with an exploration loss that shifts recent actions toward event-specific priors after terminal failures.The priors encourage slowing for collisions or traffic violations, stronger acceleration when blocked, and steering exploration after route deviation; Nz = 100.
  • 3.2. IL Agents Supervised by Roach: The Roach training scheme supervises IL agents through value regression, action-distribution KL divergence, and latent-feature matching alongside existing imitation learning.The value target represents expected future return, while the latent feature is intended to transfer driving-relevant information across rendering and weather conditions.
  • 3.2. IL Agents Supervised by Roach: CILRS combines camera and measurement encoders with command-specific branches that predict continuous actions and ego-vehicle speed.Roach can provide deterministic actions through its distribution mode, or richer distributional, value, and feature supervision.

4. Experiments

The experiments evaluate Roach and camera-based imitation-learning agents across CARLA generalization benchmarks, then analyze training efficiency, expert performance, and supervision ablations. Roach provides a strong expert and enables improved IL agents, including expert-level driving scores under supported settings.

  • Benchmarks: NoCrash evaluates generalization from Town 1 to Town 2 and from four training weather types to two new weather types, focusing on dense traffic.The offline LeaderBoard evaluates 50 training and 26 testing public routes, including varied urban and freeway scenarios.
  • Metrics: Success rate measures collision- and blockage-free route completion, while driving score multiplies route completion by infraction penalty.The driving score also incorporates the percentage of route distance completed.
  • RL expert training: 10M training steps produce Roach, combining Beta action distributions with exploration loss for stable and sample-efficient PPO training.Training uses approximately 1.7M steps in each of six CARLA servers and takes roughly a week on an AWS EC2 g4dn.4xlarge.
  • Performance of Experts: Roach and other RL experts achieve comparable success rates and higher driving scores than Autopilots, partly because they handle traffic lights better.The Autopilots often run red lights after waiting too long at junctions, whereas PPO+beta and Roach achieve the best RL-expert performance.
  • Performance of IL Agents: Feature matching helps when the IL measurement vector contains information needed to reproduce Roach’s policy, while missing navigation information hurts NoCrash feature-matching variants.Augmenting the measurement vector with a command enables LK+LF(c) and LK+LV+LF(c) to achieve the best NoCrash generalization scores among IL agents.
  • Performance of IL Agents: Roach-supervised LK+LF(c) loses less than 10% while generalizing to the NoCrash new town and weather setting, but remains below Roach on NoCrash.The performance gap on the offline LeaderBoard indicates remaining room for improvement.
  • Performance of IL Agents: Replacing the Autopilot with Roach improves LA driving score by 23%, while soft targets and latent features add another 22% absolute improvement.The resulting LK+LF(c) agent reaches an 88% expert-level driving score using a single camera image.

5. Conclusion

The paper presents Roach as an RL expert and an effective imitation target, combining privileged BEV supervision with informative targets for camera-based imitation learning. The resulting agent reaches state-of-the-art performance and expert-level performance on NoCrash-dense, while future work remains for simulation improvement and real-world deployment.

  • Roach uses BEV representations, Beta distributions, and exploration loss to set a new performance upper-bound on CARLA with high sample efficiency.
  • Soft targets, values, and latent features from Roach provide more effective supervision for imitation learning.
  • A single-camera end-to-end imitation-learning agent achieves state-of-the-art performance and expert-level performance on the NoCrash-dense benchmark.
  • Future work includes improving simulation benchmarks, deploying in the real world, increasing model capacity, and addressing sim-to-real gaps and realistic road-user behavior.

B.1. Videos

The supplementary videos illustrate Roach’s natural driving in challenging urban situations, Autopilot’s conservative failures, and failure modes of the best Roach-supervised camera-based imitation agent. The appendix also documents reproduction scripts and repository modules.

  • Videos: Roach handles traffic lights, unprotected left turns, roundabouts, and stop signs in a Town03 dynamic-weather evaluation run.
  • Videos: Autopilot’s over-conservative behavior in dense traffic can lead to red-light infractions and blockage in a new-town, new-weather run.
  • Videos: The best Roach-supervised camera-based imitation agent exhibits incomplete stops, overcrowded-intersection collisions, and steering oscillations under sun glare.
  • Reproduction: Reproduction scripts cover Roach training, imitation-learning training, benchmarking, and on/off-policy data collection.
  • Repository: The repository contains a configurable CARLA Gym environment and implementations of Autopilot, Roach, and DA-RB.

B.3. Rendering issues

CARLA 0.9.11 can render after-rain puddle reflections as black pixels under testing weather. Camera-based end-to-end imitation agents may interpret accumulated black pixels as obstacles, making weather generalization exceptionally difficult.

  • CARLA 0.9.11 sometimes renders after-rain puddle reflections as black pixels that camera-based agents recognize as obstacles.
  • Because the artifact appears under testing weather but not training weather, it makes camera-based imitation-learning generalization exceptionally hard.

C.1. Roach

Roach combines a configurable CARLA training setup with BEV inputs, shaped rewards, terminal penalties, and deterministic action-distribution outputs. Its design uses privileged simulation information and behavior-oriented reward components to train the RL expert.

  • BEV: Cyclists and pedestrians are rendered larger than their actual sizes in BEV inputs to reduce encoder size and encourage caution around vulnerable road users.
  • Optimization: The policy and value networks are updated together with Adam, while excessive empirical KL divergence interrupts the update phase and reduces the learning rate after patience thresholds.
  • Rollout: Rollouts fill a fixed-size trajectory buffer from six CARLA servers corresponding to six LeaderBoard maps.
  • Terminal conditions: Episodes terminate on red lights, stop signs, collisions, route deviation, or prolonged blocking, with terminal penalties assigned to the final observation/action pair.
  • Reward shaping: The reward combines speed, route position, heading, steering-change, and terminal components.
  • Reward shaping: The desired speed decreases linearly with obstacle distance, using Autopilot hazard detection to provide a dense speed reward.

C.2. IL Agent Supervised by Roach

The IL agent uses a branched architecture and combines off-policy expert demonstrations with on-policy DAGGER data. Training incorporates perturbations, replay-buffer mixing, image augmentation, and optional Roach-initialized heads.

  • Network Architecture: The IL agent uses six branches for junction turning, lane following, and left or right lane changes.
  • Off-policy Data Collection: Triangular action perturbations are applied during off-policy collection to alleviate covariate shift.
  • Off-policy Data Collection: Off-policy episodes are retained as correct demonstrations by removing the final 30 seconds after traffic-rule violations terminate an episode.
  • On-policy Data Collection: DAGGER uses critical-state sampling and a fixed replay buffer that mixes newly collected data with off-policy data.
  • On-policy Data Collection: At each DAGGER iteration, new data fills around 15-25% of the replay buffer while at least 20% remains off-policy.
  • Training Details: Roach weights initialize distributional action and value heads at the first behavior-cloning iteration, while training continues from the previous DAGGER epoch.

D. Benchmarks

The benchmarks evaluate urban-driving performance across varying traffic, towns, and weather conditions. The offline LeaderBoard provides stricter control over training and testing settings and supports generalization studies, but setup changes require re-evaluation.

  • Scope: The offline LeaderBoard includes more traffic scenarios and longer routes across six maps than the NoCrash benchmark.
  • Weather: Training uses four weather types, while SoftRainSunset and WetSunset serve as new weather types for evaluation.
  • Background Traffic: Background vehicles and pedestrians are randomly spawned from CARLA 0.9.11's complete blueprint library.
  • Pros and cons of the online and the offline Leaderboard: The online LeaderBoard standardizes evaluation conditions but does not restrict training procedures or training-data collection.
  • Pros and cons of the online and the offline Leaderboard: The offline LeaderBoard prescribes both training and testing environments and gives full control and observation over the benchmark.
  • Pros and cons of the online and the offline Leaderboard: Offline LeaderBoard setup changes, such as a CARLA-version change, require re-evaluating other methods.
  • Pros and cons of the online and the offline Leaderboard: The offline LeaderBoard can be used when a thorough study of method generalization ability is desired.

E. Additional Experimental Results

Additional experiments examine feature alignment, iteration-wise driving performance, benchmark conditions, and implementation details. Feature loss aligns the IL agent's latent representation with Roach's, while results are reported across NoCrash and offline LeaderBoard settings.

  • Feature Loss: Without feature loss, LK learns a latent space independent of Roach, making test feature loss effectively noise across test conditions.
  • Experimental Configuration: The reported experiments include expert and IL-agent configurations alongside benchmark scope, traffic, network, and hyper-parameter tables.
  • Feature Loss: With feature loss, LK + LF(c) has test feature loss below 1 and increases during generalization tests as expected.
  • Driving Performance: Figures 8 and 9 report driving scores for experts and IL agents at each DAGGER iteration on NoCrash-busy and LeaderBoard-busy.
  • Driving Performance: The supplementary tables provide detailed performance and infraction analyses for fifth-iteration agents across NoCrash-busy and offline LeaderBoard conditions.
  • Generalization Settings: NoCrash-busy and offline LeaderBoard results cover combinations of training or new towns with training or new weather.
Loading 2108.08265v3…