Source-linked AI summary
Learning from All Vehicles
Dian Chen, Philipp Krähenbühl
TL;DR
Autonomous-driving policies struggle with diverse and partially observed scenarios, especially when learning from other vehicles without their sensors or control signals. LAV addresses this with viewpoint-invariant perception, all-vehicle trajectory supervision, and privileged distillation. In closed-loop CARLA evaluation, it led the public leaderboard with a 61.85 driving score and 94.46 route completion rate, while the authors note simulation-only evaluation and remaining traffic infractions.
Problem
Autonomous-driving systems need to generalize beyond predominantly easy driving data, but other vehicles provide only partially observed trajectories and no accessible sensors or control signals.
Method
LAV learns viewpoint-invariant perception features from detection and segmentation, trains a privileged planner on all vehicles’ future waypoints, and transfers it through distillation.
Results
61.85 driving score and 94.46 route completion rate were achieved on the public CARLA leaderboard, exceeding the prior state of the art by 25 and 24 points respectively.
Takeaways & Limitations
Learning from all nearby vehicles provides richer trajectory supervision and helps predict their behavior for collision avoidance.
Takeaways & Limitations
The approach is trained and evaluated only in simulation, still incurs traffic infractions, and its behavior predictor does not model multi-modality beyond high-level commands.
Abstract
from arXiv · showhide
In this paper, we present a system to train driving policies from experiences collected not just from the ego-vehicle, but all vehicles that it observes. This system uses the behaviors of other agents to create more diverse driving scenarios without collecting additional data. The main difficulty in learning from other vehicles is that there is no sensor information. We use a set of supervisory tasks to learn an intermediate representation that is invariant to the viewpoint of the controlling vehicle. This not only provides a richer signal at training time but also allows more complex reasoning during inference. Learning how all vehicles drive helps predict their behavior at test time and can avoid collisions. We evaluate this system in closed-loop driving simulations. Our system outperforms all prior methods on the public CARLA Leaderboard by a wide margin, improving driving score by 25 and route completion rate by 24 points. Our method won the 2021 CARLA Autonomous Driving challenge. Code and data are available at https://github.com/dotchen/LAV.
1. Introduction
LAV addresses the long tail of driving scenarios by learning from nearby vehicles, despite their partial observability and lack of sensor information. It uses viewpoint-invariant representations and privileged distillation, achieving leading CARLA leaderboard performance.
- Motivation: Driving data is dominated by easy behaviors, while unseen and safety-critical scenarios create a generalization challenge for autonomous-driving systems.The paper contrasts this challenge with hand-designed planners, learning-based methods’ long tail, simulation-based scenario design, and large-scale real-world collection.
- Motivation: Other vehicles’ trajectories expose interesting or safety-critical states without requiring additional data and can help the ego-vehicle avoid collisions.The paper also links this training signal to improved sample efficiency and greater exposure to interesting scenarios.
- Challenge: Partial observability makes direct learning from other vehicles difficult because their trajectories, control commands, goals, and sensor inputs are unavailable.The model must infer surrounding state for other vehicles using only the ego-vehicle’s sensors.
- Approach: LAV learns viewpoint-invariant perception features with 3D detection and segmentation, then uses privileged distillation to transfer all-vehicle motion planning to ego-vehicle sensor inputs.Future waypoints represent motion plans, while the privileged planner predicts nearby vehicles’ trajectories and infers their high-level commands.
- Results: 61.85 driving score and 94.46 route completion rate were achieved on the public CARLA leaderboard, exceeding the prior state-of-the-art by 25 and 24 points respectively.The method ranked first at submission time and won the 2021 CARLA Autonomous Driving challenge.
2. Related Work
Prior autonomous-driving work spans perception, behavior prediction, and learning-based motion planning, including methods that learn from other agents. LAV differs by operating on raw sensor inputs while learning viewpoint-invariant representations rather than assuming privileged simulator state.
- Perception: Autonomous-driving perception commonly processes LiDAR, sometimes fused with RGB, to detect objects and represent surrounding road structures.The related work also distinguishes map-based from map-free perception systems.
- Learning from other vehicles: Using every encountered vehicle as a supervisory signal has been proposed to increase training-data diversity.This line of work directly relates to LAV’s use of other vehicles’ experiences.
- Behavior prediction: Behavior-prediction methods forecast dynamic-object trajectories using perception representations or raw sensors, with deterministic, conditional multi-modal, and Gaussian-mixture approaches represented.These approaches differ in how they represent uncertainty and future behavior.
- Motion planning: Learning-based motion planners use imitation or reinforcement learning to map sensor inputs, affordances, or commands to future trajectories or controls.The cited work includes conditional branching, cost-volume prediction, and on-policy distillation.
- Learning from other vehicles: Earlier privileged imitation learning avoids partial observability by using simulator ground truth or other agents’ sensors, whereas LAV uses raw sensor inputs and viewpoint-invariant intermediate features.LAV’s distinction is operational: it does not assume perfect perception or access to other agents’ sensors.
3. Learning from All Vehicles
LAV combines vehicle-independent perception, all-vehicle motion supervision, privileged distillation, trajectory refinement, and vehicle-aware control in an end-to-end pipeline. The design transfers supervision from surrounding vehicles while retaining ego-vehicle execution mechanisms.
- Pipeline: The pipeline maps sensor readings and navigation commands through perception, motion planning, and low-level control to steering and acceleration commands.The controller converts predicted motion plans into executable commands.
- Vehicle-independent perception: The perception module learns map-view, vehicle-invariant features using semantic segmentation and detection supervision, including explicit ego-vehicle labeling.Supervised pre-training is intended to improve generalization and align representations across ego and nearby vehicles.
- Motion planning: The motion planner predicts n = 10 future 2D waypoints conditioned on high-level commands and an intermediate GNSS goal.The paper uses n = 20 for its second leaderboard submission and refines trajectories with a recurrent network.
- All-vehicle supervision: All vehicles provide future-trajectory supervision, while the model infers their unavailable high-level commands instead of relying on ambiguous rule-based labels.The privileged planner is trained from ground-truth perception and then distilled into a planner using inferred perception outputs.
- Motion planning: The planner produces coarse trajectories for all visible vehicles, but ambiguous high-level commands can make the resulting motion plan noisy.The refinement model is trained only on ego-vehicle trajectories because GNSS goals are available only for the ego vehicle.
- Vehicle-aware control: A vehicle-aware controller uses other vehicles’ motion plans for emergency-stop decisions while PID controllers produce steering and acceleration outputs.A separate classifier overwrites braking for traffic-light and hazard stoppages.
4. Implementation details
The implementation combines multimodal perception, vehicle-level prediction, waypoint planning, collision-aware control, and image-based braking. Its components process camera and LiDAR information into representations used for detection, mapping, trajectory prediction, and vehicle control.
- Perception: PointPainting concatenates semantic camera classes with corresponding LiDAR points before PointPillars processes the multimodal representation.The semantic classes are background, vehicles, roads, lane markings, and pedestrians, with perspective transformation aligning image and LiDAR information.
- Perception: Sparse pillar processing and sparse PointNet representations allow all LiDAR pillars to be processed efficiently in space and time.Pillars cover x ∈ [−10m, 70m] and y ∈ [−40m, 40m], with each pillar representing a 0.25m×0.25m spatial region.
- Perception: Vehicle detection and mapping use separate heads to predict BEV object and semantic-map outputs for vehicles, pedestrians, roads, and lane markings.The detection head predicts centerness, orientation, and bounding-box maps, while mapping predicts roads and solid or broken lane markings.
- Prediction and planning: The prediction and planning stack crops rotated vehicle regions into shared embeddings, then uses command-specific GRUs and waypoint outputs to generate trajectories.The motion planner and refined planner provide trajectories used by the controller, while the system predicts future motion for detected vehicles.
- Control: The controller hard-stops when the refined ego-trajectory predicts a collision and otherwise follows it with separate lateral and longitudinal PID controllers.An image-only brake classifier can overwrite the longitudinal brake command when its predicted brake score is larger.
5. Experiments
The experiments evaluate LAV in closed-loop CARLA simulation against leaderboard methods and through local ablations of its design choices. LAV achieves leading leaderboard performance, while ablations support training from other vehicles, privileged distillation, staged perception training, and iterative refinement.
- Experimental setup: LAV is evaluated in closed-loop CARLA driving, using all eight towns and 400K training frames for the online leaderboard submission.Local ablations train on four towns and evaluate on two unseen towns across four weather conditions, with three runs per setup.
- Ablation study: Training on other vehicles’ trajectories and viewpoints improves route completion and infraction performance relative to ego-only training, while privileged distillation produces a larger ablation drop.The full model achieves the highest scores across all three reported metrics.
- Ablation study: Training with vehicles within 5 or 15 meters performs equally well, whereas extending the range to 25 meters is slightly worse.The authors attribute the degradation to distant vehicles’ different sensor appearance and sparse LiDAR measurements.
- Ablation study: Staged perception training outperforms both no perception pre-training and joint perception-motion training.Without perception training, viewpoint-induced distribution shifts hurt performance; joint training is harder than disentangled training.
- Ablation study: Five iterative refinement steps perform best, because refinement learns residuals and combines high-level command semantics with goal geometry.K = 0 directly uses motion-planner trajectories for driving.
- Qualitative analysis: Qualitative visualizations show predicted maps, detections, and motion plans for all vehicles, with non-ego vehicles producing distributions over plausible future plans.These plans align with the inferred map-view road representation and other vehicles.
6. Discussion
The discussion presents LAV as a mapless, end-to-end system trained from nearby vehicles’ experiences, while identifying simulation-only evaluation and limited behavior multimodality as boundaries.
- Discussion: LAV trains a mapless, end-to-end driving system from the experiences of all nearby vehicles and achieves state-of-the-art closed-loop simulation performance.The paper reports that it outperforms prior leading methods by a wide margin.
- Limitations and potential negative social impacts: The system is trained and evaluated only in simulation and still incurs traffic infractions.The authors state that direct real-world deployment would most likely result in traffic accidents.
- Technical limitations: The conditional motion planner’s behavior predictor models multimodality only through high-level commands.The authors propose probabilistic formulations and richer motion predictors as directions for handling diverse behaviors.
Appendix A. Detailed Infractions
The appendix defines infractions as normalized occurrence counts per kilometer traveled.
- Detailed Infractions: Infractions are measured as the number of occurrences normalized per 1 kilometer traveled.
A.1. Comparison with state-of-the-art
On the CARLA public leaderboard, LAV leads the reported red-light, offroad, and blocked infraction comparisons against prior methods.
- Comparison with state-of-the-art: LAV leads the red-light, offroad, and blocked infraction numbers among all compared leaderboard methods.
B.1. Ablation study
The ablation study examines key design choices, including sensor fusion, training on other vehicles’ experiences, perception training, and motion refinement. The system also does not rely on HD-Maps.
- Removing PointPainting lowers Driving Score and Route Completion compared with full LAV, supporting the benefit of multi-modal sensor fusion.The LiDAR-only variant keeps the rest of the backbone unchanged.
- The ablations separately evaluate how training on other vehicles’ experiences affects driving performance.
- The study also compares different perception training schemes and evaluates iterative motion refinement.
- The system does not rely on HD-Maps.
Appendix D. Dataset Statistics
The dataset statistics describe training and test-town layouts for the online submission and local ablation models. The ablations are tested on four routes in unseen towns.
- The online leaderboard submission trains on all towns, while local ablation models train on Town01, Town03, Town04, and Town06.The local ablation models test on Town02 and Town05.
- The ablation models are tested on four routes in unseen towns.Figure 7 visualizes these test routes.
Appendix E. License of Assets
The appendix documents asset licenses, sensor configurations, hyperparameters, dataset statistics, and experimental ablation tables. It also includes the CARLA leaderboard comparison and test-route visualization.
- CARLA is released under the MIT license, and its assets are under the CC-BY license.
- The Waymo open dataset uses a customized non-commercial license, while included ResNet and ERFNet code carries MIT and CC-BY-NC licenses.
- Table 6 compares methods on the public CARLA leaderboard, ranking them primarily by Driving Score and reporting route completion and infractions.
- The appendix lists ego-vehicle sensor configuration, hyperparameters, training-town statistics, and four driving-performance ablation tables.
- Figure 7 visualizes test routes in unseen towns for the ablation models.