Source-linked AI summary
Multi-modal Sensor Fusion-Based Deep Neural Network for End-to-end Autonomous Driving with Scene Understanding
Zhiyu Huang, Chen Lv, Yang Xing, Jingda Wu
TL;DR
The paper addresses limited performance and generalization in end-to-end autonomous driving by combining multimodal sensor fusion with scene understanding. It develops a network that fuses visual and depth inputs early, jointly predicts semantic segmentation and vehicle controls, and evaluates the system in simulated urban driving. The model achieves 100% success in static navigation tasks in both training and unobserved situations and outperforms prior models on other reported tasks.
Problem
End-to-end driving may suffer from spurious modality-specific correlations when trained only with control commands, limiting reliable use of multimodal information.
Method
The proposed network early-fuses visual images and depth maps, performs pixel-wise semantic segmentation, and concurrently predicts vehicle control commands.
Results
100% success rate was achieved in static navigation tasks in both training and unobserved situations, with better success rates than prior models in other tasks.
Takeaways & Limitations
The ablation results support the combined use of multimodal sensor fusion and scene understanding for better generalization in unobserved environments.
Abstract
from arXiv · showhide
This study aims to improve the performance and generalization capability of end-to-end autonomous driving with scene understanding leveraging deep learning and multimodal sensor fusion techniques. The designed end-to-end deep neural network takes as input the visual image and associated depth information in an early fusion level and outputs the pixel-wise semantic segmentation as scene understanding and vehicle control commands concurrently. The end-to-end deep learning-based autonomous driving model is tested in high-fidelity simulated urban driving conditions and compared with the benchmark of CoRL2017 and NoCrash. The testing results show that the proposed approach is of better performance and generalization ability, achieving a 100% success rate in static navigation tasks in both training and unobserved situations, as well as better success rates in other tasks than the prior models. A further ablation study shows that the model with the removal of multimodal sensor fusion or scene understanding pales in the new environment because of the false perception. The results verify that the performance of our model is improved by the synergy of multimodal sensor fusion with scene understanding subtask, demonstrating the feasibility and effectiveness of the developed deep neural network with multimodal sensor fusion.
I. INTRODUCTION
The paper motivates end-to-end driving with multimodal perception and scene understanding, then proposes a network that jointly performs semantic segmentation and vehicle control.
- End-to-end driving maps high-dimensional sensor inputs directly to vehicle control commands, unlike modular pipelines that separate perception, localization, planning, and control.
- Multimodal inputs such as RGB images and depth maps can provide richer environmental representations for learning driving policies.
- Control-only supervision can produce spurious modality-specific correlations, motivating scene understanding as an auxiliary task.
- The proposed network fuses visual and depth data early, performs pixel-wise semantic segmentation, and learns a conditional driving policy.
- The system is evaluated in simulated urban scenarios using CoRL2017 and NoCrash benchmarks, with ablations testing multimodal fusion and scene understanding.
II. METHODOLOGY
The methodology encodes multimodal sensor observations into a latent representation, decodes it for scene understanding, and passes it to downstream driving components.
- The model maps multimodal observations and navigational guidance to driving commands and scene understanding outputs.
- The multimodal fusion encoder converts high-dimensional sensor inputs into a low-dimensional latent representation.
- A decoder projects the latent representation into high-dimensional scene representations such as pixel-wise semantic segmentation.
- The latent representation is explicitly shaped by scene understanding before being used for the downstream driving task.
B. Conditional driving policy
The conditional driving policy uses the scene representation together with high-level navigational commands to generate low-level vehicle controls.
- High-level commands such as going straight or turning left resolve route-selection ambiguity at intersections.
- The policy takes the latent driving-scene representation and navigational command as inputs, then outputs low-level vehicle control commands.
C. End-to-end learning
The model jointly trains its fusion encoder, scene-understanding decoder, and conditional policy using demonstrations with driving and scene-understanding supervision.
- The encoder, decoder, and conditional policy are integrated and trained jointly end-to-end to obtain task-specific features efficiently.
- Training uses multimodal observations, scene representations, recorded control actions, and navigational guidance from demonstrations.
- The objective combines steering, speed-control, and scene-understanding losses with weights that balance their contributions.
- Pixel-wise cross-entropy is used for image semantic segmentation, while steering and speed actions use the specified control losses.
D. The neural network architecture
The network fuses RGB and depth inputs early, then jointly produces scene understanding and control outputs. Its conditional policy uses navigational commands to select guidance-specific speed and steering outputs.
- The observation combines an RGB image with its associated depth map, while scene understanding is represented as a pixel-wise semantic map.
- RGB and depth channels are concatenated into an RGBD input for the multimodal sensor fusion encoder.
- The architecture comprises a multimodal sensor fusion encoder, a scene understanding decoder, and a conditional driving policy.
- The driving policy uses a navigational command to activate a dedicated branch that outputs desired speed and steering control signals.
III. MODEL TRAINING AND TESTING
Training data were collected in two distinct urban CARLA scenarios with different road layouts, obstacles, and dynamically adjustable weather conditions.
- The dataset was collected in Town 01 and Town 02, two urban CARLA scenarios containing two-lane roads, curves, and intersections.
- Town 01 has 2.9 km of road and 11 intersections, whereas Town 02 has 1.4 km of road and eight intersections.
B. Model training
Training balances the lane-keeping data distribution and evaluates the model across benchmark tasks, traffic densities, and weather conditions in training and testing scenarios.
- Data balancing: Dataset balancing targets both steering and speed controls for the lane-keeping command because straight driving dominates the training data.
- Data balancing: Nearly 90% of lane-keeping samples have steering angles between -5 and 5 degrees, motivating downsampling of the majority range and upsampling of minority ranges.
- Evaluation: Success rate is the percentage of episodes successfully completed in a task.
- Testing protocol: The evaluation follows CoRL2017 and NoCrash, using navigation tasks that vary in route type, traffic participation, and traffic density.
- Testing protocol: Training uses four weather settings, while testing uses distinct weather conditions for both CoRL2017 and NoCrash scenarios.
IV. TESTING RESULTS AND DISCUSSIONS
MSFSU improves success rates and generalization relative to prior models across CoRL2017 and NoCrash evaluations, especially in static navigation and unobserved conditions. Failures mainly arise from out-of-distribution scenes and insufficiently trained traffic-rule behavior.
- Benchmark results: MSFSU achieves a 100% success rate in static navigation tasks and outperforms prior models in dynamic navigation tasks.
- Generalization: In testing scenarios with different road layouts and weather, static-task success does not decline and dynamic-traffic success drops only slightly.
- Benchmark results: MSFSU outperforms prior models on NoCrash, with no success-rate reduction for navigation in empty traffic.
- Failure analysis: Dense-traffic performance is expected to be poor because the model was not trained to obey traffic rules such as stopping for red lights.
- Failure analysis: Observed failures mostly involve collisions with vehicles or pedestrians in scenes outside the training distribution.
B. Ablation study
The ablation study separates the effects of multimodal sensor fusion and scene understanding, showing that both are important for generalization and control quality, especially in unobserved conditions.
- Ablation setup: The ablation compares the complete model with variants removing depth-based multimodal fusion or the scene-understanding decoder.The scene understanding (SU) model removes depth information, while the multimodal sensor fusion (MSF) model removes semantic segmentation.
- Success-rate results: Both ablated models perform approximately like the original model during training but show substantially lower success rates in unobserved situations.The impairment is especially evident in testing scenarios with changed road layouts and weather conditions.
- Failure analysis: False perception causes unexpected stops in testing: the MSF model brakes before a puddle, while the SU model falsely detects a vehicle.These examples occur in static navigation tasks under unobserved weather settings.
- Failure analysis: The combined model avoids these false-perception mistakes, supporting a joint contribution from multimodal sensor fusion and scene understanding.The paper attributes this complementarity to scene understanding learning relevant general features and depth information further enhancing scene understanding.
- Trajectory and control quality: The proposed MSFSU model stays closer to expert demonstrations, while the MSF-only model deviates more, particularly during turning.The trajectory comparison uses RMSE in meters between the agent and expert trajectories; the MSFSU model also shows smoother yaw-rate behavior.
C. Discussions
The discussion identifies simulation-to-real transfer as the most significant limitation, alongside challenges in real-world semantic labeling, onboard computation, and ego-speed integration.
- Limitations: Simulation-to-real transfer is identified as the most significant limitation of the approach.The discussion attributes this difficulty to differences between simulated and real-world camera imagery and depth sensing.
- Limitations: Real-world semantic labeling is laborious and time consuming for network training.
- Limitations: Computational burden remains a major concern for deployment onboard a real vehicle.
- Limitations: Adding ego-speed as an input caused an inertia problem in which the agent could not restart after obstacle avoidance.The authors report that the model relied heavily on feedback speed information and propose incorporating speed through scene understanding as future work.
V. CONCLUSIONS
The proposed network combines multimodal sensor fusion, scene understanding, and conditional driving policy for end-to-end autonomous driving. It achieves strong navigation performance and improved control quality, while ablation results attribute the gains to the combined design.
- Conclusions: The network fuses visual images and depth, performs pixel-wise semantic segmentation, and outputs vehicle control commands concurrently.
- Conclusions: 100% success in static navigation tasks was achieved in both training and unobserved circumstances.
- Conclusions: Removing multimodal sensor fusion or scene understanding reduced navigation success rates by 28% and 22%, respectively.