Source-linked AI summary
Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline
Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, Yu Qiao
TL;DR
End-to-end autonomous driving has separately studied trajectory planning and direct control, leaving their potential combination underexplored. TCP unifies both in a shared framework with multi-step, trajectory-guided control prediction and situation-based output fusion, achieving first place on the CARLA Leaderboard using only a monocular camera.
Problem
Trajectory planning and direct control have been studied separately, while current-step control prediction may miss sequential dependencies and trajectory-to-control conversion is nontrivial.
Method
TCP jointly learns trajectory and direct-control branches, using multi-step control prediction with temporal reasoning, trajectory-guided attention, shared features, and situation-based fusion.
Results
75.137 driving score and 1st place on the public CARLA Leaderboard were achieved using a monocular camera.
Takeaways & Limitations
The unified trajectory-and-control framework achieves state-of-the-art CARLA Leaderboard performance with only monocular-camera input.
Abstract
from arXiv · showhide
Current end-to-end autonomous driving methods either run a controller based on a planned trajectory or perform control prediction directly, which have spanned two separately studied lines of research. Seeing their potential mutual benefits to each other, this paper takes the initiative to explore the combination of these two well-developed worlds. Specifically, our integrated approach has two branches for trajectory planning and direct control, respectively. The trajectory branch predicts the future trajectory, while the control branch involves a novel multi-step prediction scheme such that the relationship between current actions and future states can be reasoned. The two branches are connected so that the control branch receives corresponding guidance from the trajectory branch at each time step. The outputs from two branches are then fused to achieve complementary advantages. Our results are evaluated in the closed-loop urban driving setting with challenging scenarios using the CARLA simulator. Even with a monocular camera input, the proposed approach ranks first on the official CARLA Leaderboard, outperforming other complex candidates with multiple sensors or fusion mechanisms by a large margin. The source code is publicly available at https://github.com/OpenPerceptionX/TCP
1 Introduction
End-to-end autonomous driving commonly predicts either trajectories or direct controls, but each paradigm has distinct weaknesses. TCP unifies both through shared learning, multi-step trajectory-guided control prediction, and situation-based output fusion.
- End-to-end methods typically predict either future trajectories or direct control actions, with no clear winner across all circumstances.
- Trajectory-based methods can struggle with big turns, while control-based methods may react late to abrupt obstacles and produce unstable actions.
- TCP packs trajectory planning and control prediction into one multi-task framework instead of separately training and directly ensembling two models.A shared backbone reduces computational complexity and can exploit correlations between the related tasks.
- The control branch predicts multiple future actions using a temporal module, while trajectory-guided attention supplies location information for different future time steps.The temporal module models interactions between the ego agent, other agents, and the changing environment.
- A situation-based fusion scheme adaptively combines trajectory and control outputs so the more suitable representation receives greater weight.The combination weight can vary across situations or individual control signals.
- 75.137 driving score: monocular-camera TCP ranked first on the public CARLA Leaderboard, surpassing multi-camera-and-LiDAR methods by 13.291 points.
2 Related Work
Related work in learning-based end-to-end driving spans reinforcement learning and imitation learning, with imitation methods predicting trajectories or direct controls. Multi-task learning and ensembles provide established strategies for combining related capabilities and models.
- Learning-based end-to-end autonomous driving research commonly includes reinforcement learning and imitation learning.Reinforcement learning is studied for robustness to dataset distribution shifts, while imitation learning uses recorded expert data.
- Imitation-learning systems use expert trajectories or direct control actions, with control baselines such as CIL and CILRS established in CARLA.Subsequent work adds multimodal inputs, multi-task learning, dataset aggregation, and knowledge distillation.
- Multi-task learning trains several related tasks simultaneously to help each other and improve generalization.Autonomous-driving examples combine tasks including object detection, lane detection, semantic segmentation, and depth estimation.
- Ensemble approaches combine model outputs, including test-time augmentation and fusion of experts trained on different input subsets.
3 Trajectory-guided Control Prediction
TCP unifies trajectory planning and direct control prediction in two connected branches. The trajectory branch guides multi-step control prediction, and situation-based fusion combines both outputs into final actions.
- Architecture Design: TCP combines trajectory and control models in one framework rather than using two separately trained models whose outputs are directly ensembled.Both branches are trained with supervision, sharing encoded features within a unified architecture.
- Trajectory Planning Branch: The trajectory branch predicts planned waypoints, which can be converted into throttle, brake, and steering actions through longitudinal and lateral PID controllers.Waypoint-vector magnitudes determine desired speed, while orientations determine steering.
- Multi-step Control Prediction: The control branch predicts current and future actions using a temporal module that reasons about interactions between predicted actions, the environment, and the ego agent.Its GRU-based temporal module updates hidden states using current features and predicted actions, acting as a coarse simulator.
- Trajectory-guided Attention: Trajectory guidance helps the control branch attend to spatially appropriate image regions at each future time step.Attention maps use trajectory- and control-branch hidden states to aggregate the encoded image feature map.
- Loss Design: Training averages losses across future steps and adds them to the current-step loss because the immediately executed action is the primary optimization target.The loss includes trajectory planning, control prediction, and auxiliary components.
- Situation-based Fusion: TCP averages trajectory-derived and directly predicted controls with a situation-dependent combination weight to exploit their complementary advantages.The combination weight α lies between 0 and 0.5, with the more suitable representation receiving greater influence according to the situation.
4 Experiments
Experiments evaluate TCP in closed-loop CARLA driving and compare its components, model combinations, and fusion settings. The method achieves strong leaderboard performance with a monocular camera, while ablations quantify gains from trajectory guidance, temporal prediction, and situation-based fusion.
- Experimental Setup: The closed-loop CARLA task requires agents to follow routes toward destinations under challenging scenarios including obstacle avoidance and unsignalized intersections.Evaluation reports Driving Score, Route Completion, and Infraction Score.
- State-of-the-art Comparison: TCP-Ens achieves a 75.137 driving score on the public CARLA Leaderboard using only a monocular camera.TCP alone achieves 69.714, while TCP-Ens ranks first overall.
- Ablative Study: Adding a trajectory branch improves performance by 2.5 points, multi-step temporal prediction adds 7.9 points, and trajectory-guided attention adds 3.2 points.These component ablations use the control branch output except for the final fusion experiment.
- Ablative Study: Situation-based fusion with α = 0.3 significantly improves infraction score and raises the overall driving score to 57.The fusion weight is evaluated through driving-score comparisons across multiple α values.
- Multi-task vs. Ensemble: Joint multi-task learning performs better than directly combining separate models while using a smaller model and fewer GFLOPs.A conventional ensemble of TCP and TCP-SB provides further gains at higher computational complexity.
- Visualization: Trajectory-guided attention supplies location-related information that directs the control branch toward regions useful for future control prediction.The visualization compares input images with predicted trajectories, top-down trajectories, and attention maps w1 and w3.
5 Conclusion
The paper studies trajectory-based and direct-control paradigms within one unified end-to-end driving framework. It combines a trajectory branch, a multi-step control branch, and situation-based fusion, achieving state-of-the-art CARLA Leaderboard performance with monocular input.
- TCP unifies trajectory planning and direct control through two interacting prediction branches.The control branch uses multi-step prediction, while the trajectory branch provides guidance.
- The method includes a situation-based fusion scheme to combine outputs from the trajectory and control branches.
- TCP achieves state-of-the-art performance on the CARLA Leaderboard using only a monocular camera.
Supplementary Materials: Trajectory-guided Control Prediction for End-to-end
The supplementary document provides implementation context and detailed experimental material for Trajectory-guided Control Prediction. It covers dataset details, training procedures, infraction statistics, qualitative results, limitations, future directions, and social impact.
- The supplementary document is titled “Autonomous Driving: A Simple yet Strong Baseline.”
- The document lists Xiaosong Jia, Hongyang Li, and Junchi Yan among the authors.
- The supplement covers dataset, implementation, infraction statistics, qualitative results, limitations, future directions, and potential social impact.
A.1 Dataset Collection
The dataset is collected and tested in CARLA 0.9.10.1 using Roach as the expert, with an added rule-based detector to improve obstacle avoidance. Routes are randomly generated and use the scenario configurations from Transfuser.
- Data collection and testing use CARLA 0.9.10.1, with Roach serving as the expert.
- A rule-based vehicle and pedestrian detector from Transfuser is added to the expert to avoid possible collisions.
- Routes are randomly generated with lengths ranging from 50 meters to 300 meters.
- The experiments use the scenario configurations provided in Transfuser.
A.2 Dataset Statistics
The study trains its leaderboard submission on all eight towns, while ablation experiments train on four towns and test on designed routes across four weathers in two held-out towns.
- Leaderboard submission training uses all eight towns.
- Ablation experiments train on Town01, Town03, Town04, and Town06.
- Ablations test four designed routes under four different weathers in Town02 and Town05.
B Implementation Details
TCP uses a monocular-camera setup with four future steps predicted at 2 Hz for both trajectory and control branches, evaluated across repeated routes, weathers, and scenarios.
- The image encoder is ImageNet-pretrained ResNet-34, with 900 × 256 inputs and a 100° camera field of view.
- Both trajectory and control branches predict K = 4 future steps at 2 Hz.
- Training uses Adam for 60 epochs with batch size 128, a 1 × 10^-4 learning rate, and learning-rate halving after 30 epochs.
- The fusion scheme classifies situations using recent steering magnitude to select trajectory-specialized or control-specialized behavior.
- The validation set contains 48 route evaluations from four routes, four weather conditions, and three repetitions, with random official leaderboard scenarios added.
C.3 Qualitative Results
Qualitative results show TCP handling diverse challenging traffic situations, while visualizations indicate that future-action prediction attends to more distant image regions and benefits from step-specific re-aggregation.
- Qualitative driving cases: TCP reacts to changing traffic lights, brakes for a suddenly crossing cyclist, yields to crossing vehicles, and negotiates an unprotected left turn.
- Evaluation context: The evaluation tables report leaderboard, trajectory-guided control ablation, and multi-task-versus-ensemble statistics.
- Attention visualizations: Future-action GradCam maps focus on regions farther from the ego vehicle than current-action maps.
- Attention visualizations: The Control-Only model’s single global-average aggregation produces one broad highlighted area rather than distinct regions for different future steps.
- Attention visualizations: The visualization supports re-aggregating image information with different highlighted regions for each future prediction step.
D.1.1 Failure Cases and Future Work Directions
The paper identifies rule-based fusion, limited single-camera visibility, and unmodeled other-vehicle trajectories as boundaries of TCP, motivating adaptive fusion and richer perception or prediction.
- Fusion limitations and future directions: The rule-based situation fusion scheme may require many experiments and specific prior knowledge, motivating more general learning-based adaptive fusion.
- Failure cases: A single camera can miss vehicles entering the ego path from outside the front view, leading to delayed collisions when emergency braking fails.
- Failure cases: Adding multi-view cameras or LiDAR is proposed as a direction for addressing limited visibility.
- Failure cases: TCP can fail when it does not predict other vehicles’ trajectories, resulting in blocking or collisions.
- Real-world scope: The method is trained and tested in simulation, so direct real-world deployment could cause traffic accidents and negative societal impacts.