Source-linked AI summary
DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Driving
Xiaosong Jia, Yulu Gao, Li Chen, Junchi Yan, Patrick Langechuan Liu, Hongyang Li
TL;DR
End-to-end autonomous driving must learn planning from noisy raw sensors, while teacher-student methods still require behavior-cloned students to learn planning heads and can suffer causal confusion. DriveAdapter uses adapters to align student perception features with a frozen teacher planner and masks alignment loss when hand-crafted rules override imperfect teacher actions. The method achieves state-of-the-art performance on two closed-loop benchmarks, although perfect student perception remains bounded by the imperfect teacher.
Problem
Teacher-student driving systems still train the student planning head from raw sensor data through behavior cloning, leaving causal confusion and a gap between predicted privileged inputs and teacher training inputs.
Method
DriveAdapter uses layer-wise adapters with masked feature alignment and action guidance to connect student perception to a frozen reinforcement-learning teacher planner.
Results
DriveAdapter achieves state-of-the-art performance on two closed-loop autonomous-driving evaluation benchmarks.
Takeaways & Limitations
The decoupled paradigm directly utilizes driving knowledge learned by the reinforcement-learning teacher while keeping perception and planning learning separate.
Takeaways & Limitations
Even perfect BEV segmentation from the student remains upper-bounded by the imperfect learning-based teacher’s performance.
Abstract
from arXiv · showhide
End-to-end autonomous driving aims to build a fully differentiable system that takes raw sensor data as inputs and directly outputs the planned trajectory or control signals of the ego vehicle. State-of-the-art methods usually follow the `Teacher-Student' paradigm. The Teacher model uses privileged information (ground-truth states of surrounding agents and map elements) to learn the driving strategy. The student model only has access to raw sensor data and conducts behavior cloning on the data collected by the teacher model. By eliminating the noise of the perception part during planning learning, state-of-the-art works could achieve better performance with significantly less data compared to those coupled ones. However, under the current Teacher-Student paradigm, the student model still needs to learn a planning head from scratch, which could be challenging due to the redundant and noisy nature of raw sensor inputs and the casual confusion issue of behavior cloning. In this work, we aim to explore the possibility of directly adopting the strong teacher model to conduct planning while letting the student model focus more on the perception part. We find that even equipped with a SOTA perception model, directly letting the student model learn the required inputs of the teacher model leads to poor driving performance, which comes from the large distribution gap between predicted privileged inputs and the ground-truth. To this end, we propose DriveAdapter, which employs adapters with the feature alignment objective function between the student (perception) and teacher (planning) modules. Additionally, since the pure learning-based teacher model itself is imperfect and occasionally breaks safety rules, we propose a method of action-guided feature learning with a mask for those imperfect teacher features to further inject the priors of hand-crafted rules into the learning process.
1. Introduction
End-to-end autonomous driving seeks to map raw sensor data directly to planned trajectories or controls, but existing teacher-student systems still train the student’s planning head through behavior cloning. DriveAdapter instead connects perception to a frozen teacher planner with adapters and masked action-guided feature learning, achieving state-of-the-art performance on two benchmarks.
- End-to-end autonomous driving directly maps raw sensor data to planned trajectories or control signals in a fully differentiable model.
- Existing teacher-student methods reduce perception noise during planning learning, but the student still learns a planning head from scratch through behavior cloning.This can produce causal confusion, such as copying surrounding vehicles instead of using traffic-light information.
- Directly generating the privileged inputs required by a frozen teacher creates a domain-transfer problem because the teacher has seen only ground-truth BEV segmentation.
- DriveAdapter inserts learnable adapters between student perception outputs and teacher planning inputs, aligning features layer by layer across the two domains.
- Action-guided feature learning masks alignment loss when rules override the teacher, encouraging adapters to learn features that generate good actions rather than merely mimic the teacher.
- DriveAdapter with masked feature distillation achieves state-of-the-art performance on two public benchmarks and is accompanied by ablation studies of the decoupled paradigm.
2. Related Works
Related work establishes end-to-end driving through behavior cloning, reinforcement learning, and teacher-student supervision. This work further decouples perception and planning by directly adopting a frozen teacher for planning while using adapters to bridge predicted privileged inputs and the teacher’s domain.
- 2.1. End-to-End Autonomous Driving: Early end-to-end driving methods used behavior cloning from rule-based experts to map camera images directly to control signals.
- 2.1. End-to-End Autonomous Driving: State-of-the-art student models generally use behavior cloning, motivating further decoupling of perception and planning.
- 2.1. End-to-End Autonomous Driving: The proposed paradigm directly adopts the frozen teacher model for planning while keeping the overall system end-to-end differentiable.
- 2.2. Adapter: Adapters preserve the teacher’s knowledge while bridging the gap between predicted privileged inputs and ground-truth inputs.
3. Method
DriveAdapter connects a raw-sensor student perception model to a frozen privileged-information teacher through layer-wise adapters. Feature alignment reduces the prediction-to-ground-truth feature gap, while masked alignment and action guidance incorporate rule-based interventions.
- Student Model for Perception Learning: The student converts camera and LiDAR observations into BEV features and predicted BEV segmentation for the teacher and adapter modules.The BEV representation encodes privileged scene elements such as surrounding agents, lanes, and traffic signs.
- Student Model for Perception Learning: Directly feeding predicted BEV segmentation to the frozen teacher performs poorly because predictions differ substantially from the ground-truth inputs used during teacher training.The perception model reports mIoU 0.35 on test unseen scenes, while predicted agents and lanes are incomplete or blurry and traffic-light states can be incorrect.
- Adapter Module: DriveAdapter inserts learnable adapters between teacher modules to transform imperfect student features toward the teacher’s ground-truth feature domain.Adapters receive teacher-layer features and downsampled raw BEV features, preserving the teacher feature tensor shape.
- Adapter Module: For Roach, adapters are implemented with CNNs for 2D feature maps and MLPs for 1D feature maps, except for the measurement encoder and output layer.The measurement encoder receives ego state directly, while the output linear layer generates actions rather than intermediate features.
- Adapter Module: Each adapter is trained with a feature-alignment regression loss that progressively reduces the distribution gap in a layer-by-layer supervised manner.The regression function uses smooth L1 loss, with targets obtained from the teacher processing ground-truth BEV segmentation.
- Mask & Action Guidance: When rules override the teacher, DriveAdapter masks feature-alignment losses and backpropagates action loss through all adapters to learn rule-consistent features.This avoids recovering teacher features associated with wrong decisions and injects hand-crafted rule priors into adapter training.
4. Experiments
Experiments evaluate DriveAdapter in CARLA closed-loop benchmarks and ablate its loss terms, adapter placement, teacher freezing, and learning targets. The results support combining feature alignment, action guidance, masking, multistage adapters, raw BEV features, and a frozen teacher.
- Dataset and Benchmark: Experiments use CARLA closed-loop evaluation on Town05Long and Longest6, with training datasets of 189K and 2 million frames.The standard collection uses 189K frames from four towns; extra-data models use 2 million frames on eight towns.
- Comparison with State-of-the-Art Works: DriveAdapter performs best under limited data, matches competitors trained on 10× data with dual outputs, and sets new records after using more data.The major gain from 10× data is attributed to improved red-light detection, highlighting BEV segmentation’s interpretability.
- Loss Design: Feature alignment prevents the route-completion drop associated with action-only supervision and the inertia issue.Without feature alignment, adapter supervision becomes similar to behavior cloning.
- Loss Design: Masking is necessary because action and feature-alignment supervision conflict when rules override teacher decisions, causing overall performance to drop without masking.The mask suppresses feature-alignment losses for teacher mistakes.
- Loss Design: Removing action loss sharply lowers Infraction Score because the relatively aggressive teacher produces more collisions.Both loss terms and masking are reported as significant; action guidance and masking inject hand-crafted-rule information for safer driving.
- Adapter Design: Adapters at both early and late stages outperform restricted placement, while omitting raw BEV features or unfreezing the teacher worsens performance.Early-only adapters become reckless, late-only adapters get stuck more often, and unfreezing approaches behavior-cloning performance.
- Learning Targets: Deeper student learning targets generally improve driving performance, except pure behavior cloning, which suffers severe inertia and low route completion.The paper retains BEV segmentation as the target because early features contain scene detail and segmentation is human-readable.
5. Conclusion
The paper proposes DriveAdapter to use reinforcement-learning teacher knowledge for planning while addressing imperfect perception and teacher decisions. It reports state-of-the-art performance on two closed-loop autonomous-driving benchmarks.
- DriveAdapter directly utilizes driving knowledge from a reinforcement-learning teacher in an end-to-end autonomous-driving pipeline.
- Masked feature alignment and action guidance objectives address imperfect perception and imperfect teacher models.
- DriveAdapter achieves state-of-the-art performance on two closed-loop autonomous-driving evaluation benchmarks.
A. Case Study of Causal Confusion
The case study illustrates causal confusion through an intersection inertia failure and contrasts it with frozen-teacher planning using a route mask. The latter avoids the described stationary behavior and yields higher route completion than listed variants.
- At an intersection, behavior cloning can make the ego vehicle remain stopped because it learns the improper correlation between surrounding vehicles’ inactivity and its own action.
- TeacherAdapter feeds a drawn route mask into the frozen teacher, so the model has a tendency to move in the described intersection scenario.
- TeacherAdapter has much higher Route Completion than the Unfrozen Teacher Model and Action variants in the cited comparisons.
- Using a frozen teacher for decision-making avoids causal confusion and lets the student focus on feature extraction and perception learning, although failure cases remain.
B. Failure Case Analysis
DriveAdapter’s remaining failures arise from perception difficulty, inherited teacher-model behavior, and benchmark simulation logic. The authors identify traffic-light detection, excessive safety distance, and over-conservative decision-making as key limitations.
- A red-light violation occurs when a small traffic light blends into a green background, making detection difficult.The issue is alleviated when scaling from limited data to 10x data.
- An intersection collision results from stopping too far from the stop line, causing the ego vehicle and neighbor to accelerate simultaneously.The authors attribute this behavior to a bad habit inherited from the Roach teacher model.
- A stuck case results from excessive safety distance around a jaywalking pedestrian, preventing both the pedestrian and ego vehicle from moving.Changing pedestrian logic or using object detection with cautious creeping are proposed, but creeping may increase rule violations or collisions.
- The decision-making process still needs improvement, particularly to avoid excessive conservatism, while a better learning-based teacher could further improve performance.The reported Driving Score is 71, indicating that performance remains far from perfect.
C. Discussion about Real World Application of DriveAdapter
BEV segmentation abstracts the driving scene for sim2real transfer. The authors suggest training a behavior policy in simulation or offline datasets, then adapting it with perception and an adapter in the real world.
- BEV segmentation provides a driving-scene abstraction that is useful for sim2real settings.
- A behavior policy could be trained with CARLA or offline nuPlan and Waymo Motion Prediction data, then finetuned with perception and an adapter for real-world use.
- The proposed real-world workflow separates behavior-policy training from later perception-module and adapter adaptation.
D. Implementation Details
The paper provides implementation details for data collection, model configuration, training hyper-parameters, and data augmentation, and states that code and models will be publicly available.
- Implementation details cover data collection, model configuration, training hyper-parameters, and data augmentation.
- The authors state that the code and model will be made publicly available.
- The implementation description is intended to document the configuration of the large end-to-end autonomous-driving system.
D.1. Data Collection
Data collection uses Roach as the expert, multiple calibrated cameras, LiDAR, vehicle-state sensors, route commands, and stored teacher features and actions. The dataset also includes visualized failure cases such as inertia and red-light violations.
- Roach serves as the expert, with a collision detector for emergency stops similar to prior work.
- The setup uses four 150°-FOV RGB cameras positioned around the ego vehicle, with image calibration for CARLA’s Brown-Conrady distortion.Depth and semantic-segmentation labels are also collected.
- Failure visualizations include inertia at a green light and an ego vehicle running a red light.
- The sensor suite includes a 64-channel LiDAR, IMU, GPS, and speedometer for environmental and ego-motion information.
- Target points and high-level commands such as keeping straight or turning are saved according to the benchmark protocol.
- Feature maps from multiple Roach layers, final actions, and rule-override indicators are stored for feature alignment and action guidance.
D.2. Model Configuration
The model configuration combines multi-scale image and temporal BEV processing with task-specific adapters, segmentation inputs, and an optional trajectory-prediction branch. It uses a 135M-parameter model trained with AdamW for 60 epochs.
- Backbone and feature processing: The image-processing stack uses ResNet50, PAFPN, LSS, a depth module, and a U-Net-like semantic-segmentation structure, with images downsampled to 450x800.The implementation uses official OpenMMLab and ImageNet-pretrained backbone components where applicable.
- Temporal processing: Temporal inputs contain two frames, with history features transformed into the current ego coordinate system before concatenation in LSS.SECOND stacks point clouds with an additional timestep channel, while LSS concatenates transformed history BEV features with current features.
- Privileged inputs: The privileged BEV representation contains 24 types spanning road, route, lane, vehicle, pedestrian, and traffic-light masks across historical timesteps.Lane markings and traffic-light states use distinct numeric encodings in the masks.
- Adapter design: Adapters use a ResNet bottleneck with squeeze-and-excitation for 2D feature maps and a two-layer MLP for 1D feature maps.
- Trajectory prediction: The +TCP setting adds an MLP that consumes the last-layer 1D feature map and predicts expert-generated trajectories.
- Model scale: The model has 135M parameters, 1719G MACs, and approximately 5G of inference GPU memory usage.
- Optimization: Training uses AdamW with a 1e-4 learning rate, cosine decay, effective batch size 96, weight decay 1e-7, and 60 epochs.Loss weights are tuned so each loss begins near 1, and gradients are clipped at an L2 threshold of 35.
D.4. Data Augmentation
Data augmentation is applied only to images, using random color transformations and random cropping before projection into the BEV grid.
- Image augmentation: Image augmentation uses random color transformation and random cropping before image features are projected to the BEV grid.