Source-linked AI summary
TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving
Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, Andreas Geiger
TL;DR
End-to-end imitation learning with existing sensor-fusion methods underperforms in complex, dense-traffic scenarios, motivating better integration of complementary camera and LiDAR representations. TransFuser uses multi-resolution transformer attention to fuse image and LiDAR features, achieving state-of-the-art CARLA performance with fewer infractions. The study’s scope remains limited by simulation-only evaluation, restricted sensing, and single-timestep inputs.
Problem
Existing sensor-fusion imitation-learning policies underperform in complex driving scenarios with dense dynamic traffic, despite complementary information from LiDAR and cameras.
Method
TransFuser connects independent image and LiDAR encoder branches with transformers that fuse perspective-view and bird’s-eye-view representations at multiple resolutions.
Results
TransFuser achieves state-of-the-art CARLA performance with a significant reduction in infractions and outperforms prior work on the CARLA leaderboard.
Takeaways & Limitations
Attention-based multimodal fusion provides a simple, flexible, and generic approach for end-to-end driving and is highly competitive with more complex alternatives.
Takeaways & Limitations
Experiments are conducted only in simulation, with no rear sensing and only single-timestep inputs; real-world data may be more diverse and noisier.
Abstract
from arXiv · showhide
How should we integrate representations from complementary sensors for autonomous driving? Geometry-based fusion has shown promise for perception (e.g. object detection, motion forecasting). However, in the context of end-to-end driving, we find that imitation learning based on existing sensor fusion methods underperforms in complex driving scenarios with a high density of dynamic agents. Therefore, we propose TransFuser, a mechanism to integrate image and LiDAR representations using self-attention. Our approach uses transformer modules at multiple resolutions to fuse perspective view and bird's eye view feature maps. We experimentally validate its efficacy on a challenging new benchmark with long routes and dense traffic, as well as the official leaderboard of the CARLA urban driving simulator. At the time of submission, TransFuser outperforms all prior work on the CARLA leaderboard in terms of driving score by a large margin. Compared to geometry-based fusion, TransFuser reduces the average collisions per kilometer by 48%.
1 INTRODUCTION
End-to-end imitation learning must combine complementary camera and LiDAR information while reasoning over global scene context in dense, complex traffic. TransFuser addresses this with attention-based multimodal fusion and is evaluated on challenging CARLA settings.
- LiDAR provides accurate 3D information, but missing information such as traffic-light states must be recovered from RGB cameras.
- Interactions between nearby dynamic agents and distant traffic lights require global context beyond locality-based architectures.
- TransFuser connects independent image and LiDAR encoder branches with transformers to integrate multimodal representations.The model integrates representations between complementary image and LiDAR modalities within an autoregressive waypoint-prediction framework.
- The Longest6 benchmark uses approximately 1.5 km routes, increased traffic density, and challenging pre-crash scenarios.The study also evaluates TransFuser and baselines on the official CARLA leaderboard.
- Existing sensor-fusion imitation-learning policies struggle in challenging CARLA scenarios with dense traffic.
- The journal extension adds an improved demonstrator, wider multi-camera sensing, a stronger vision backbone, and multi-task training.It also introduces the image-only Latent TransFuser baseline and releases code, data, and trained models.
2 RELATED WORK
Prior autonomous-driving research applies multimodal fusion mainly to perception and forecasting, while end-to-end driving studies explore complementary image, depth, and semantic representations. TransFuser instead focuses on integrating image and LiDAR representations for end-to-end driving.
- Multi-Modal Autonomous Driving: Recent end-to-end methods improve driving by complementing RGB images with depth and semantic representations.
- Multi-Modal Autonomous Driving: Image and LiDAR are complementary scene representations and readily available in autonomous-driving systems.
- Forecasting: Most sensor-fusion research targets perception tasks such as object detection and motion forecasting.Common strategies project LiDAR features into image space or image features into BEV or range-view space.
- Attention for Autonomous Driving: Attention-based driving research covers lane changing, detection, forecasting, driver attention, and end-to-end driving.
3 TRANSFUSER
TransFuser integrates RGB and LiDAR BEV representations through multi-scale transformer fusion and predicts ego-vehicle waypoints for end-to-end navigation. It uses imitation learning with auxiliary tasks and also provides an image-only Latent TransFuser variant.
- Architecture: TransFuser combines image and LiDAR inputs through interconnected convolutional encoder branches and a multi-modal fusion transformer.The fused representation feeds an auto-regressive waypoint prediction framework.
- Problem Setting: The policy maps single-time-step camera and LiDAR observations to future BEV waypoints, which a separate PID controller converts into vehicle actions.Training uses behavior cloning with an L1 trajectory loss and auxiliary losses.
- Multi-Modal Fusion Transformer: Self-attention incorporates global context by treating intermediate feature-map elements as tokens and fusing image and LiDAR features at multiple scales.Learnable positional embeddings provide spatial information for the tokenized feature maps.
- Multi-Modal Fusion Transformer: The two branches produce 512-dimensional vectors that are combined by element-wise summation before MLP processing and waypoint prediction.The resulting vector is described as a compact representation encoding global 3D-scene context.
- Auxiliary Tasks and Latent TransFuser: Auxiliary BEV supervision detects vehicles, while the image-only Latent TransFuser replaces LiDAR histograms with positional encodings and retains the architecture and losses.Its BEV branch is supervised for HD-map and bounding-box prediction, enabling attention-based projection from perspective view to BEV.
4 EXPERIMENTS
The experiments evaluate imitation-learning driving agents in CARLA across routes and adversarial scenarios, using expert demonstrations generated with privileged simulator information. The setup includes route following, low-level control, and model-based infraction forecasting.
- Evaluation Setting: The evaluation covers predefined routes through freeways, urban areas, and residential districts with scenarios including obstacle avoidance and unprotected turns.Scenarios are initialized at predefined positions to test responses to adversarial situations.
- Training Dataset: Training and testing use CARLA 0.9.10 across all eight publicly available towns.The training dataset contains around 2500 junction routes and around 1000 curved-highway routes.
- Expert Demonstrations: The expert policy uses privileged simulator information, and its waypoints provide ground-truth labels for imitation learning.Simulator-provided labels also supervise the auxiliary tasks.
- Expert and Control: The expert follows planned paths with PID lateral control and model-predictive longitudinal control using target speeds of 4.0 m/s, 3.0 m/s in intersections, and 0.0 m/s when an infraction is predicted.The system predicts red-light and collision infractions using trigger-box tests and forecasts based on a kinematic bicycle model.
4.4 Longest6 Benchmark
Longest6 is a locally evaluable CARLA benchmark designed to approximate the official leaderboard while enabling repeated evaluations. It uses long, balanced routes, dense traffic, and metrics covering completion, infractions, driving score, and normalized infractions.
- Benchmark design: Longest6 avoids the official leaderboard’s evaluation-time constraints, which make ablations and repeated detailed evaluations impractical.A single official evaluation takes over 100 hours, while Longest6 can be evaluated locally without computational budget restrictions.
- Benchmark design: The benchmark selects the six longest routes from each of six towns, producing 36 routes averaging 1.5 km.The average route length is similar to the official leaderboard’s 1.7 km.
- Benchmark design: Evaluation maintains high dynamic-agent density by spawning vehicles at every permitted spawn point.This design choice is motivated by the official leaderboard.
- Metrics: Route Completion averages completed route distance across N routes and is reduced when the agent drives outside route lanes.The off-route multiplier is 1 - the percentage of off-route distance.
- Metrics: Infraction Score starts at 1.0 and applies predefined penalty coefficients for each infraction, while Driving Score weights route completion by the infraction multiplier.Collision and traffic-rule penalties are specified separately, and infractions per kilometer are normalized by total distance driven.
4.6 Baselines
The evaluation compares TransFuser with established and image-only baselines using driving performance and infraction metrics. The Longest6 results identify TransFuser as the strongest overall model by driving score.
- Baselines: TransFuser is compared with WOR and Latent TransFuser among the reported baselines.WOR uses multi-stage Q-function supervision, while Latent TransFuser tests the importance of LiDAR by removing that modality.
- Baseline comparison: TransFuser obtains the best Driving Score by a large margin on the Longest6 benchmark.The table reports mean and standard deviation for Driving Score, Route Completion, and Infraction Score across three evaluation runs.
4.7 Implementation Details
The implementation uses front-facing cameras and a LiDAR point cloud represented in bird’s-eye view as two sensor modalities. Separate RegNetY-3.2GF encoders process the image and LiDAR inputs.
- Sensor inputs: The system uses two modalities: concatenated front-facing camera images and a LiDAR point cloud converted to a BEV representation.The image encoder is pretrained on ImageNet, whereas the LiDAR encoder is trained from scratch.
4.8 Longest6 Benchmark Results
On Longest6, TransFuser achieves the strongest reported driving score, while fusion baselines can underperform the image-only Latent TransFuser. The benchmark also exposes collision failures in dense lane changes and a runtime cost for transformer fusion.
- Evaluation protocol: Three evaluation runs are used to account for CARLA nondeterminism and variation from training seeds.The evaluation therefore reports performance across repeated runs rather than a single trial.
- Image-only baselines: Latent TransFuser obtains the best Route Completion with zero route deviations among the image-based methods.The authors attribute this to BEV-aligned waypoint prediction and inverse-dynamics control, unlike WOR’s coarse commands and discrete actions.
- Sensor fusion methods: LAV performs worse than TransFuser in Driving Score because its Route Completion is 23% lower.The comparison also notes differences in steering, blocked infractions, and vehicle-collision avoidance.
- Sensor fusion methods: Late Fusion and Geometric Fusion perform worse than the image-only Latent TransFuser baseline.This comparison motivates attention-based fusion rather than simply adding LiDAR through these alternative fusion mechanisms.
- Limitations: TransFuser fails during dense-traffic lane changes, producing a high number of consecutive collisions on affected routes.The failure is illustrated with two examples whose timelines progress from left to right.
- Interpretation: Attention is reported to incorporate global 3D-scene context, which allows safer driving.This result is presented as the interpretation of the sensor-fusion comparison.
- Limitations: TransFuser has around 9× more vehicle collisions per kilometer than the expert in the maximum-density evaluation setting.Collisions primarily occur during unprotected turns and lane changes.
- Runtime: A single TransFuser model runs in real time on an RTX 3090, despite transformers increasing runtime by 17% over Late Fusion.An ensemble of three models incurs a 28% increase relative to the Late Fusion baseline.
4.9 Leaderboard Results
On the official CARLA leaderboard, TransFuser achieves state-of-the-art results, while Latent TransFuser leads among models without LiDAR inputs. The evaluation reports driving score, route completion, and infraction score over 100 secret routes.
- Latent TransFuser achieves a driving score of 45.20, nearly 10 points above the next-best image-based method, GRIAD.GRIAD uses 45M simulator samples, compared with 228k training frames for this work.
- Latent TransFuser and TransFuser improve infraction score by a large margin compared with existing methods.
- TransFuser achieves state-of-the-art results on the CARLA leaderboard by incorporating global attention between image and LiDAR features.The leaderboard reports driving score, route completion, and infraction score over 100 secret routes.
4.10 Attention Statistics and Visualizations
TransFuser’s cross-modal attention becomes extensive in later transformer blocks and focuses on vehicles and traffic lights near intersections. These patterns indicate aggregation of complementary image and LiDAR information.
- Attention setup: The transformers process 110 image tokens and 64 LiDAR tokens, each representing a 32×32 input patch.The analyzed tokens correspond to intersection regions containing traffic lights and vehicles.
- Attention statistics: Later transformers show substantially more LiDAR-to-image attention, with nearly all LiDAR tokens aggregating information from image features.
- Attention statistics: T4 exhibits extensive cross-attention for both image and LiDAR tokens, indicating strong information exchange between modalities.
- Attention visualizations: Attention visualizations show that TransFuser attends to areas near vehicles and traffic lights at intersections.The pattern appears in both image-to-LiDAR and LiDAR-to-image visualizations.
4.11 Global Safety Heuristic
The global safety heuristic improves some Longest6 results but can reduce TransFuser’s leaderboard performance. Large training and evaluation variance further complicates comparisons between methods.
- Global safety heuristic: For TransFuser, the global safety heuristic improves Longest6 driving score by 7 points but reduces performance on the CARLA leaderboard.For Latent TransFuser, it improves driving score by 8 points on Longest6 and 5 points on the leaderboard.
- Ensembling: Ensembling improves TransFuser’s leaderboard driving score by more than 10 points, while its Longest6 gain is 5 points.The authors associate the larger leaderboard benefit with greater training variance.
- Variance: Individual-model scores can differ by 10–15 points across training and evaluation seeds, with TransFuser ranging from 44.15 to 59.45.Evaluation randomness from CARLA traffic, physics, and sensor noise is identified as the primary source of variance.
- Variance: The reported variance makes method comparisons problematic when studies vary evaluation seeds but not training seeds.The authors recommend varying training seeds for both baselines and proposed methods.
4.12 Ablation Studies
Ablations show that auxiliary supervision, bidirectional multi-scale fusion, backbone capacity, and camera field of view materially affect performance. Removing auxiliary tasks sharply reduces route completion, while several fusion variations have smaller effects.
- Auxiliary tasks: Removing all auxiliary tasks reduces route completion from 92.28 to 78.17.Removing any single auxiliary task alone does not have a large impact.
- Architecture: Bidirectional fusion performs better than fusion in only one direction, while restricting fusion to deeper scales causes a small performance drop.Using 2–8 attention layers produces similar performance.
- Architecture: RegNetY-3.2GF backbones outperform the evaluated lower-capacity and alternative backbone configurations.The encoder backbone has the highest impact on final driving score among the architecture ablations.
- Model inputs: Increasing LiDAR range or reducing camera field of view reduces infraction score and driving score.The default configuration uses a 32m × 32m LiDAR range and 132° camera field of view.
- Controller and inputs: Creeping substantially improves route completion while maintaining a similar infraction score, whereas adding velocity input sharply reduces driving score.The velocity-input drop is not recovered by creeping.
5 DISCUSSION AND CONCLUSIONS
TransFuser achieves strong CARLA driving performance with a simple imitation-learning architecture, while the study identifies limitations involving inertia, sensing coverage, temporal inputs, latency, and simulation-only evaluation.
- TransFuser uses attention-based multimodal fusion and achieves state-of-the-art CARLA performance with significantly fewer infractions.The method integrates representations from different modalities and captures global 3D scene context, including dynamic agents and traffic lights.
- The proposed long-route, dense-traffic benchmark is intended to support ablation studies and detailed statistics beyond the CARLA leaderboard.The authors position it as a suitable community evaluation option for analyses that are difficult to conduct through the leaderboard alone.
- The study identifies unresolved issues with creeping, rearward sensing during lane changes, single-timestep inputs, and collision reduction at intersections.Temporal inputs may help estimate other traffic participants’ acceleration and velocity, while rear sensing matters for lane changes.
- Latency is not evaluated because CARLA waits for the agent’s computation before resuming world simulation, limiting conclusions about real-world performance.The paper notes that latency has been shown to matter for real-world applications.
- The experiments are simulation-only and rely on high-quality labels that real-world datasets may not provide.The authors caution that real-world data is more diverse, noisier, and may lack the full set of labels used in the study.
- Imitation learning with TransFuser remains highly competitive despite rapid CARLA leaderboard progress and the use of more complex alternatives.The paper identifies principled solutions to inertia and reduced training and evaluation variance as key future challenges.