Source-linked AI summary
Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer
Hao Shao, Letian Wang, RuoBing Chen, Hongsheng Li, Yu Liu
TL;DR
Autonomous driving remains difficult to deploy because rare adverse events require comprehensive scene understanding, while multi-sensor fusion and model interpretability remain challenging. InterFuser combines multi-modal, multi-view sensor fusion with interpretable intermediate outputs used to constrain actions, and it ranked first on the public CARLA Leaderboard while outperforming prior methods. The authors note remaining traffic infractions and identify more advanced prediction, probabilistic modeling and theoretical safety analysis as future directions.
Problem
Rare adverse events and incomplete scene understanding remain safety challenges for autonomous driving, especially in dense traffic scenes.
Method
InterFuser fuses multi-modal, multi-view sensor information and generates interpretable intermediate features that serve as safety constraint heuristics.
Results
InterFuser outperformed prior methods on CARLA benchmarks and ranked first on the public CARLA Leaderboard.
Takeaways & Limitations
The framework’s interpretable outputs support global-context perception and a safety-enhanced controller that constrains actions within the safe set.
Takeaways & Limitations
The method still has traffic infractions, while advanced prediction, probabilistic modeling and rigorous theoretical safety analysis remain future work.
Abstract
from arXiv · showhide
Large-scale deployment of autonomous vehicles has been continually delayed due to safety concerns. On the one hand, comprehensive scene understanding is indispensable, a lack of which would result in vulnerability to rare but complex traffic situations, such as the sudden emergence of unknown objects. However, reasoning from a global context requires access to sensors of multiple types and adequate fusion of multi-modal sensor signals, which is difficult to achieve. On the other hand, the lack of interpretability in learning models also hampers the safety with unverifiable failure causes. In this paper, we propose a safety-enhanced autonomous driving framework, named Interpretable Sensor Fusion Transformer(InterFuser), to fully process and fuse information from multi-modal multi-view sensors for achieving comprehensive scene understanding and adversarial event detection. Besides, intermediate interpretable features are generated from our framework, which provide more semantics and are exploited to better constrain actions to be within the safe sets. We conducted extensive experiments on CARLA benchmarks, where our model outperforms prior methods, ranking the first on the public CARLA Leaderboard. Our code will be made available at https://github.com/opendilab/InterFuser
1 Introduction
InterFuser addresses autonomous-driving safety challenges by fusing multi-modal, multi-view sensors for global scene understanding and by exposing interpretable intermediate features that constrain actions. It was validated on CARLA benchmarks, where it outperformed prior methods and ranked first on the public leaderboard.
- Motivation: High-traffic-density scenes contain many obstacles and dynamic objects, while rare adverse events can lead to incorrect or unexpected driving behavior.Examples include pedestrians suddenly emerging from roadside areas and vehicles running red lights.
- Motivation: Single-image and single-LiDAR systems provide incomplete scene information, while existing fusion methods seldom model relationships across modalities, views, agents and traffic lights.The paper identifies locality-based geometric matching and simple feature concatenation as limitations of prior fusion approaches.
- Approach: InterFuser fuses LiDAR with left, front, right and focus camera views in a one-stage architecture for global contextual perception and reasoning.The framework is designed to overcome the sensor and view limitations described for TransFuser.
- Approach: The framework outputs intermediate interpretable features, called a safety mind map, describing surrounding objects and traffic signs while exposing failure conditions and causes.These features supplement the generated driving actions with interpretable information about perception and decision-making.
- Safety: InterFuser uses interpretable intermediate features as safety heuristics to constrain actions within safe action sets.This mechanism is intended to enhance the safety of end-to-end driving while preserving interpretability.
- Evaluation: InterFuser outperformed all prior methods on several CARLA benchmarks and ranked first on the public CARLA Leaderboard.The supplied result passage reports the overall ranking but does not provide a leaderboard score here.
2 Related work
Prior urban end-to-end driving research developed simulator-based, conditional, mimicking and reinforcement-learning approaches. Transformer-based vision and modality fusion, along with safety and interpretability methods, form related lines of work.
- End-to-end autonomous driving in urban scenarios: Urban end-to-end driving research grew around the CARLA simulator and conditional imitation learning, followed by conditional driving methods in urban scenarios.The passage also describes mimicking methods that train image-input networks using supervision from privileged or compressed models.
- End-to-end autonomous driving in urban scenarios: Mimicking and privileged-model approaches provide training supervision or demonstrations for image-input and imitation-learning agents.Roach uses a privileged reinforcement-learning model as an expert agent, while LBC trains image-input networks from privileged-model supervision.
- Transformer model in vision comprehension: Transformer research established attention-based relationships among image regions and extended the mechanism toward computer vision and modality fusion.The cited related work includes Vision Transformer and TransformerFusion for vision comprehension and 3D-scene reconstruction.
- Safe and interpretable driving: Safety research in autonomous driving considers uncertain behaviors, diverse driver preferences and numerous driving situations, while traditional methods use hand-crafted rules.The passage characterizes such rule-based designs as requiring substantial human engineering effort.
3 Method
InterFuser fuses multi-view RGB and LiDAR features with a transformer, producing driving actions alongside interpretable scene and traffic features. A safety controller uses these intermediate outputs to constrain control actions within a safe set.
- Model architecture: The framework integrates signals from multiple RGB cameras and LiDAR through a multi-view, multi-modal fusion transformer encoder.The inputs include three camera views plus a cropped focus view for distant traffic lights and one LiDAR sensor.
- Output representations: The transformer decoder generates L = 10 ego-vehicle waypoints, an object density map, and traffic-rule information.The density map represents potential objects and their properties, while traffic outputs cover traffic lights, stop signs, and intersections.
- Transformer decoder: Decoder queries retrieve spatial information from fused multi-modal, multi-view features for waypoints, density-map cells, and traffic rules.The decoder uses separate waypoint, density-map, and traffic-rule queries with multi-headed self-attention.
- Prediction headers: Waypoint prediction autoregressively estimates differential displacements and accumulates them to recover future positions.A GRU predicts the waypoint sequence using decoder embeddings and a goal-location-conditioned initial hidden state.
- Training objective: The method jointly trains waypoint, object-map, and traffic-information predictions using a weighted sum of three loss terms.The weights λ balance Lpt, Lmap, and Ltf.
- Safety controller: The safety controller combines PID control with object and traffic-rule predictions to constrain lateral steering and longitudinal acceleration.Object detections support safe-distance estimation and velocity optimization, while non-green lights or stop signs trigger emergency stopping.
4 Experiments
Experiments evaluate InterFuser in CARLA using multiple benchmarks, leaderboard comparisons, sensor-input ablations, module ablations, and visualizations. The results show that multi-view, multi-modal fusion and safety-related components improve reported driving performance and reduce selected infractions.
- Experiment Setup: InterFuser was evaluated in CARLA 0.9.10.1 across 8 towns and 21 weather conditions using benchmarks with adversarial urban driving scenarios.The training and evaluation dataset contained 3M frames collected from a rule-based expert agent.
- Comparison to the state of the art: InterFuser ranks first on the public CARLA leaderboard, with driving score 76.18, route completion 88.23, and infraction score 0.84.The reported route completion and infraction scores are the second highest among compared methods.
- Ablation Study: Adding left and right cameras, a focusing view, and LiDAR progressively improved driving and infraction scores while reducing collision and red-light infractions.The focusing view reduced the red-light infraction rate by 80% relative to front, left, and right cameras.
- Ablation Study: Removing sensor and position embeddings reduced driving score by 3% and infraction score by 6%.These embeddings help the transformer distinguish tokens across sensors and positions.
- Ablation Study: Concatenated inputs reduced driving and infraction scores by 58% and 63%, while restricted cross-view attention reduced them by 9% and 15%.The compared fusion variants were evaluated against the proposed sensor-fusion approach.
- Ablation Study: Removing the safety controller reduced driving score by 15% and infraction score by 22%.Higher safety factors produced more conservative behavior, higher driving and infraction scores, and lower route completion.
5 Limitation
The method still incurs traffic infractions and has been evaluated only in simulation. The authors identify probabilistic prediction, advanced detection and prediction models, theoretical safety analysis, and real-road generalizability as future needs.
- Limitations: Although InterFuser ranks first on the public CARLA Leaderboard, it still has traffic infractions.The paper provides failure-case visualizations and statistical analysis of failure causes.
- Limitations: The evaluation was conducted in simulation, where driving scenarios are limited compared with the countless situations on actual roads.The authors identify improved real-road generalizability as vital for scalable deployment.
- Limitations: The current system uses two-threshold detection criteria and dynamic propagation with moving-average trajectory prediction.The authors suggest more advanced models and probabilistic formulations to address uncertainty and multi-modality.
6 Conclusion
InterFuser combines multi-modal sensor fusion, interpretable intermediate outputs, and a safety controller that constrains driving actions within safe sets.
- 6 Conclusion: InterFuser fuses multi-view, multi-modal inputs and generates interpretable intermediate outputs for safety-enhanced autonomous driving.The framework uses an interpretable sensor fusion transformer and a safety-enhanced controller.
- 6 Conclusion: The training objective combines waypoint, object-density-map, and traffic-information losses.The total loss is L = λptLpt + λmapLmap + λtfLtf, with λ balancing the three terms.
- 6 Conclusion: The object density map predicts object probabilities and six meta features describing object geometry and motion.The map contains seven channels: one probability channel and six meta-feature channels.
- 6 Conclusion: Traffic-information prediction recognizes traffic-light status, stop-sign presence, and whether the vehicle is at a road junction.These signals are trained using three binary cross-entropy loss terms.
B Safety controller - desired speed optimization
The safety controller selects desired velocities through a constrained linear program that balances rapid progress, collision avoidance, and vehicle dynamics over a future horizon.
- B Safety controller - desired speed optimization: The controller maximizes desired velocity subject to collision-avoidance, dynamic, and actuation constraints.The optimization considers the ego vehicle’s goal of reaching the destination quickly while maintaining safety.
- B Safety controller - desired speed optimization: The optimization uses a 1-second horizon with desired velocities defined at 0.5-second and 1-second steps.The current ego velocity is v0, and the time step duration T is 0.5s.
- B Safety controller - desired speed optimization: Safety constraints use maximum safe distances at two steps, together with maximum velocity and acceleration limits.The controller denotes these quantities as s1, s2, vmax, and amax.
- B Safety controller - desired speed optimization: The first-step safe distance is obtained by augmenting other objects’ shapes with an extra safety distance.The augmented distance is denoted ¯s, while s′1 and s′2 represent collision-free route distances.
- B Safety controller - desired speed optimization: The second-step velocity remains a free variable so the optimization accounts for future actuation and dynamic constraints.This future-step constraint supports optimization of the first-step desired velocity.
C Implementation Details
The implementation uses multiple RGB views and a LiDAR bird’s-eye-view representation, encoded by convolutional backbones and fused with a transformer.
- C Implementation Details: The camera inputs use 800 × 600 resolution, 100° horizontal FOV, and distinct crops for front, left, and right views.The front crop is 224 × 224, while the side crops are 128 × 128.
- C Implementation Details: LiDAR point clouds are converted into a three-channel BEV histogram encoding points above ground, below ground, and total point counts.The BEV grid resolution is 0.125m2 and covers 28 meters forward and 14 meters on each side.
- C Implementation Details: Figure 4 evaluates driving-preference changes under 100% and 150% safety-factor settings on Town05 Long with adversarial events.The 100% setting uses ¯s = 2 and vmax = 6.5; the 150% setting scales ¯s by 150% and vmax by 1/150%.
- C Implementation Details: ResNet-50 encodes multi-view RGB images, ResNet-18 processes LiDAR BEV inputs, and six-layer transformer encoder and decoder modules use feature dimension 256.The RGB backbone is ImageNet-pretrained, whereas the LiDAR backbone is trained from scratch.
- C Implementation Details: Training runs for at most 35 epochs with five warm-up epochs, random scaling, and color jittering.The models use AdamW, cosine learning-rate scheduling, and weight decay 0.07.
- C Implementation Details: Inference takes about 0.04 second per frame on a GeForce GTX 1060 and 0.02 second per frame on a GeForce GTX 1080 Ti.The model contains 52,935,567 parameters and was trained for about 30 hours on eight Tesla V100 32G GPUs.
D Benchmark details
The evaluation spans CARLA leaderboard routes, Town05 short and long routes, and a 42-route multi-town benchmark covering varied urban traffic conditions.
- D Benchmark details: The CARLA Autonomous Driving Leaderboard evaluates autonomous agents across realistic traffic and weather conditions using 76 public routes and 100 secret evaluation routes.Online leaderboard evaluation usually takes about 150 hours.
- D Benchmark details: Town05 evaluation uses separate training towns and includes Town05 Short and Town05 Long settings.Each setting contains 10 routes, with short routes of 100–500m and long routes of 1000–2000m.
- D Benchmark details: The CARLA 42 routes benchmark covers six towns with intersections, freeways, roundabouts, stop signs, urban scenes, and residential districts.Traffic density is configured to be comparable to busy traffic conditions.
- D Benchmark details: Figure 5 presents four cases with predicted waypoints and reconstructed traffic scenes using ego-vehicle and detected-object positions.Blue points are predicted waypoints; yellow denotes the ego vehicle; white and grey rectangles denote current and future object positions.
- D Benchmark details: Attention visualizations show how object-density-map queries aggregate features from different camera views.Queries on the left side of the map mostly attend to left-view features.
- D Benchmark details: Failure-case visualizations identify impending collisions using three RGB views and predicted object-density maps at current, 1-second, and 2-second horizons.Orange boxes mark objects the ego vehicle is about to collide with.
E Success and Failing Cases Discussion
InterFuser provides interpretable intermediate outputs for understanding driving scenes, while failure analysis identifies object detection, motion estimation, and traffic-light recognition as major error sources. On CARLA benchmarks, it outperformed prior methods across reported metrics and scenarios.
- InterFuser produced four additional good cases demonstrating scene understanding through intermediate outputs.
- 45% of failing cases were caused by failures to detect objects such as vehicles and bicycles.
- 15% of failing cases were caused by inaccurate detection of speed, heading, and related results.
- InterFuser outperformed six state-of-the-art methods on Town05 in driving score and road completion across all metrics and scenarios.
- InterFuser outperformed other methods on CARLA 42 routes in road completion, infraction score, and driving score across all metrics and scenarios.
- 15% of failing cases were caused by traffic-light misrecognition.
F Additional Experimental Results
Additional experiments compare InterFuser with prior methods on Town05 and CARLA 42 routes, while implementation details include hyper-parameters and a safety-oriented object-size adjustment.
- Tables 4 and 5 compare InterFuser with prior state-of-the-art methods on CARLA Town05 and the CARLA 42 routes benchmark.
- InterFuser renders cyclists and pedestrians larger than their actual sizes during object-density-map scene reconstruction to add safety for these road users.
- The hyper-parameter values used in InterFuser are listed in Table 6.