Source-linked AI summary
Multi-Modal Fusion Transformer for End-to-End Autonomous Driving
Aditya Prakash, Kashyap Chitta, Andreas Geiger
TL;DR
Existing sensor-fusion policies underperform in complex urban driving because geometric or local fusion does not adequately capture global interactions among dynamic agents and traffic lights. TransFuser uses attention to fuse image and LiDAR representations within an end-to-end waypoint policy, achieving state-of-the-art CARLA performance and reducing collisions by 76% against geometry-based fusion.
Problem
Existing sensor-fusion policies struggle in dense, complex urban scenarios that require global context, while cameras and LiDAR each miss complementary scene information.
Method
TransFuser uses transformer attention to fuse image and LiDAR features across modality branches and feed the resulting representation to an auto-regressive waypoint policy.
Results
TransFuser achieves state-of-the-art CARLA driving performance, including a 76.11% collision reduction compared with geometry-based fusion.
Takeaways & Limitations
Attention-based fusion captures global 3D context and focuses the driving policy on dynamic agents and traffic lights.
Takeaways & Limitations
All fusion methods struggle with red-light violations because distant red lights are barely visible and receive weak learning supervision without semantic labels.
Abstract
from arXiv · showhide
How should representations from complementary sensors be integrated for autonomous driving? Geometry-based sensor fusion has shown great promise for perception tasks such as object detection and motion forecasting. However, for the actual driving task, the global context of the 3D scene is key, e.g. a change in traffic light state can affect the behavior of a vehicle geometrically distant from that traffic light. Geometry alone may therefore be insufficient for effectively fusing representations in end-to-end driving models. In this work, we demonstrate that imitation learning policies based on existing sensor fusion methods under-perform in the presence of a high density of dynamic agents and complex scenarios, which require global contextual reasoning, such as handling traffic oncoming from multiple directions at uncontrolled intersections. Therefore, we propose TransFuser, a novel Multi-Modal Fusion Transformer, to integrate image and LiDAR representations using attention. We experimentally validate the efficacy of our approach in urban settings involving complex scenarios using the CARLA urban driving simulator. Our approach achieves state-of-the-art driving performance while reducing collisions by 76% compared to geometry-based fusion.
1. Introduction
Existing end-to-end sensor-fusion policies struggle in complex urban scenarios because geometric or local fusion misses global interactions among traffic lights and dynamic agents. TransFuser addresses this with attention-based integration of image and LiDAR representations.
- Motivation: Image-only policies perform unsatisfactorily in adversarial CARLA scenarios with red-light violations, uncontrolled intersections, and occluded pedestrians.These settings contain many dynamic agents and require 3D scene information.
- Motivation: LiDAR provides 3D information but is sparse, especially at distance, and misses signals such as traffic-light states.Additional sensors are therefore needed to capture information absent from LiDAR scans.
- Motivation: Cameras and LiDAR offer complementary representations, motivating questions about how independently to process them and which fusion mechanism maximizes driving performance.Most prior sensor-fusion work targets perception tasks rather than end-to-end driving.
- Motivation: Locality assumptions in existing fusion architectures hamper performance when vehicles must reason about multiple agents and traffic lights at intersections.The paper uses transformer attention to model global contextual interactions between modalities.
- Contribution: TransFuser incorporates global 3D scene context into modality-specific feature extraction and is integrated with an auto-regressive waypoint prediction framework.The contribution is evaluated in complex urban CARLA settings and achieves state-of-the-art performance.
2. Related Work
Prior autonomous-driving fusion research combines modalities mainly for perception or uses relatively simple late fusion for driving. TransFuser instead targets global cross-modal context in challenging urban scenarios.
- Multi-Modal Autonomous Driving: Recent end-to-end driving methods improve performance by complementing RGB images with depth or semantic representations.These studies examine early, mid, and late fusion or use explicit intermediate representations.
- Sensor Fusion Methods: Sensor-fusion methods for object detection and motion forecasting commonly project features between camera, BEV, and range-view spaces.Their primary focus is perception rather than end-to-end driving control.
- Attention for Autonomous Driving: Attention has been applied to vehicle controls, actor dependencies, lane changing, and motion forecasting in autonomous-driving systems.These approaches use spatial, temporal, or recurrent attention within particular modalities or representations.
- Attention for Autonomous Driving: Prior attention approaches do not jointly model multiple modalities or encode global 3D scene context for adversarial urban driving.TransFuser applies attention to feature fusion between image and LiDAR modalities.
3. Method
The method learns waypoint-based urban driving from front-camera and LiDAR observations, using transformer attention to fuse modality features across scales before autoregressive prediction and control.
- Method Overview: The architecture combines a MultiModal Fusion Transformer with an auto-regressive waypoint prediction network.The transformer integrates image and LiDAR information, while the waypoint network produces the driving trajectory.
- Problem Setting: Behavior cloning trains a policy to map high-dimensional environment observations to expert BEV waypoints for point-to-point urban navigation.The policy is trained from expert rollouts and safely reacts to dynamic agents while following traffic rules.
- Problem Setting: The observation uses a single front-camera image and a single-time-step LiDAR point cloud.The setup uses one time step because prior IL studies found observation histories may not improve performance.
- Input and Output Parameterization: LiDAR is converted into a two-channel 256 × 256 BEV pseudo-image covering a 32m × 32m region around the vehicle.The two histogram bins represent points on or below, and above, the ground plane.
- Waypoint Prediction: The GRU predicts differential waypoints for T = 4 future time steps, which are converted to absolute waypoints and passed to PID controllers.The controllers produce steering, throttle, and brake values from the predicted trajectory.
- Multi-Modal Fusion Transformer: Transformer self-attention uses token feature sequences, positional encoding, learned queries, keys, and values to incorporate global context.The attention mechanism is repeated across layers and heads before producing output features with the input shape.
- Waypoint Prediction: The two streams produce 512-dimensional vectors that are summed element-wise into a compact environment representation before waypoint prediction.The representation is average-pooled, passed through an MLP, and reduced to 64 dimensions for the GRU waypoint network.
- Multi-Modal Fusion Transformer: Intermediate image and LiDAR feature maps are treated as tokens and fused at multiple encoder scales.Features from different modalities are stacked with positional and velocity embeddings, transformed jointly, reshaped, and added back to each branch.
4. Experiments
Experiments in CARLA 0.9.10 evaluate image-based, sensor-fusion, attention-visualization, and ablation settings in complex urban driving scenarios. TransFuser improves driving performance and reduces infractions relative to the compared fusion baselines, while ablations show benefits from multi-scale fusion, multiple attention layers, and positional embeddings.
- Experimental Setup: CARLA 0.9.10 evaluation uses predefined routes, seven training towns, held-out Town05 evaluation, and metrics for route completion, driving score, and infractions.The task includes freeways, urban areas, and residential districts, with routes specified by sparse GPS goals and navigational commands.
- Image-Based Baselines: AIM outperforms CILRS on all evaluation settings, including 58.21% higher performance on short routes and 275.89% higher performance on long routes.AIM uses an inverse-dynamics PID controller, BEV waypoint coordinates, and autoregressive waypoint prediction; LBC instead suffers red-light and vehicle-collision infractions on long routes.
- Sensor Fusion Methods: TransFuser outperforms geometric fusion by 31.02% in DS and reduces collisions by 76.11% on Town05 Long.It also achieves 51.58% fewer collisions than late fusion and reduces red-light violations by 23.5% versus late fusion and 21.93% versus geometric fusion.
- Limitations: All fusion methods struggle with red-light violations because distant red lights in Town05 are barely visible and receive weak learning supervision.The experiments do not use semantic supervision for red lights.
- Attention Map Visualizations: Attention maps show cross-modal aggregation: 88.87% of image tokens attend to at least one LiDAR token, while 98.95% of LiDAR tokens attend to at least one image token.The attended regions commonly include vehicles and traffic lights at intersections.
- Ablation Study: Reducing fusion from four scales to one degrades performance, while shared transformers, fewer attention layers, or removing positional embeddings also reduce driving score.Eight attention layers increase DS, and positional embeddings improve performance despite a 25% lower route completion in the compared configuration.
5. Conclusion
The paper presents TransFuser, a Multi-Modal Fusion Transformer that integrates representations from different modalities using attention to capture global 3D scene context. It reports state-of-the-art performance on CARLA and identifies extensions to additional sensors and embodied AI tasks as future directions.
- TransFuser integrates representations from different modalities using attention to capture global 3D scene context.
- The method focuses on dynamic agents and traffic lights in complex driving scenarios.
- The authors suggest exploring TransFuser with additional sensors such as radar and in other embodied AI tasks.