Source-linked AI summary
Multimodal End-to-End Autonomous Driving
Yi Xiao, Felipe Codevilla, Akhil Gurram, Onay Urfalioglu, Antonio M. López
TL;DR
The paper asks whether RGBD sensor data can improve end-to-end autonomous driving beyond single-modality inputs. It evaluates RGB, depth, and RGBD with CIL across early, mid, and late fusion in CARLA, finding that multimodal models outperform single-modal ones and that early fusion performs best. The study also examines both active depth and estimated monocular depth settings.
Problem
Most end-to-end driving models rely on RGB vision alone, despite autonomous vehicles also using active depth sensors and multimodality improving several perception tasks.
Method
The paper evaluates branched conditional imitation learning with RGB, depth, and RGBD inputs using early, mid, and late fusion in the CARLA simulator.
Results
Multimodal RGBD end-to-end models outperform single-modal counterparts, with early fusion showing better performance than mid and late fusion.
Takeaways & Limitations
RGB-depth multimodality, particularly early fusion, is worth pursuing for end-to-end autonomous driving.
Abstract
from arXiv · showhide
A crucial component of an autonomous vehicle (AV) is the artificial intelligence (AI) is able to drive towards a desired destination. Today, there are different paradigms addressing the development of AI drivers. On the one hand, we find modular pipelines, which divide the driving task into sub-tasks such as perception and maneuver planning and control. On the other hand, we find end-to-end driving approaches that try to learn a direct mapping from input raw sensor data to vehicle control signals. The later are relatively less studied, but are gaining popularity since they are less demanding in terms of sensor data annotation. This paper focuses on end-to-end autonomous driving. So far, most proposals relying on this paradigm assume RGB images as input sensor data. However, AVs will not be equipped only with cameras, but also with active sensors providing accurate depth information (e.g., LiDARs). Accordingly, this paper analyses whether combining RGB and depth modalities, i.e. using RGBD data, produces better end-to-end AI drivers than relying on a single modality. We consider multimodality based on early, mid and late fusion schemes, both in multisensory and single-sensor (monocular depth estimation) settings. Using the CARLA simulator and conditional imitation learning (CIL), we show how, indeed, early fusion multimodality outperforms single-modality.
I. INTRODUCTION
The paper motivates multimodal end-to-end driving by contrasting modular and end-to-end paradigms and asking whether RGBD improves driving over single-modality inputs. Using CIL and CARLA, it reports that multimodal models, especially early fusion, perform better.
- Modular pipelines divide driving into perception, planning, and control modules, whereas end-to-end models map raw sensor data directly to vehicle controls.
- End-to-end learning reduces the need for manually annotated sensor-content ground truth, using vehicle variables such as steering angle and speed as supervision.
- Most end-to-end driving models rely on vision alone despite multimodality improving several perception tasks, motivating multimodal end-to-end driving.
- The study asks whether combining RGB and depth improves end-to-end driving over single modalities, evaluating early, mid, and late fusion with CIL in CARLA.
- Multimodal RGBD models outperform single-modal counterparts, and early fusion performs better than the other fusion schemes.
II. RELATED WORK
Related work shows multimodality improving several individual perception tasks, while this paper extends the question to end-to-end driving rather than isolated perception subtasks.
- Prior multimodal research spans object detection, tracking, and semantic segmentation, with reported gains in detection accuracy and segmentation of small objects.
- The related literature therefore connects established multimodal perception gains with a less explored driving-level evaluation.
- This paper challenges multimodality in end-to-end driving by exploring early, mid, and late fusion instead of focusing on individual perception tasks.
B. End-to-end driving
End-to-end driving research progressed from simple sensorimotor road following to imitation-learning models with temporal reasoning, control specialization, and high-level navigation commands. This paper evaluates RGB-depth fusion without requiring difficult intermediate representations.
- ALVINN demonstrated early end-to-end road following by combining camera images and laser range-finder depth through early fusion.
- Later imitation-learning models used CNNs and LSTMs for steering or speed control, often relying on vision-based inputs and specialized control targets.
- CIL introduced branched CNNs that condition steering, throttle, and brake on high-level navigation commands and vehicle speed.
- The present work explores RGB-depth end-to-end driving using CIL, complementing prior vision-based models.
- Unlike approaches requiring intermediate representations, this study evaluates early, mid, and late fusion on CARLA scenarios with dynamic actors and unseen towns and weather.
III. MULTIMODAL FUSION
The multimodal fusion design adapts branched CIL to combine perception inputs with vehicle measurements and navigation commands. Its control output is a three-signal maneuver command, with high-level commands selecting specialized branches.
- The paper adapts CIL to leverage multimodal perception data using early, mid, and late fusion schemes.
- CIL processes perception data and vehicle-state measurements separately before forming joint features for maneuver control.
- The base configuration uses a 200 × 88 RGB image, vehicle speed, and three real-valued outputs for steering angle, throttle, and brake.
- A branch selector chooses among turn-left, turn-right, go-straight, and continue according to the high-level navigation command.
- High-level navigation commands are reported as more effective than alternative approaches.
B. Fusion schemes
The study compares three ways to fuse RGB and depth within conditional imitation learning: at the input, in intermediate features, or after separate control predictions. All fusion schemes assume aligned, similarly scaled RGB and depth channels.
- Early fusion feeds raw RGB and depth channels directly into CIL, changing only the first perception convolution to accept four channels.
- Mid fusion processes RGB and depth through separate perception branches before concatenating their features with vehicle measurements.
- Late fusion runs separate RGB and depth CIL streams, then combines their control outputs through additional fully connected layers.
- All channels are assumed pixel-aligned and normalized to a common magnitude range, with depth rescaled to [0 ... 255].
C. Loss function
The models train with a weighted combination of action prediction and auxiliary speed prediction losses. Conditional imitation learning activates the action branch corresponding to the current navigation command.
- The action loss is an L1 loss over steering angle, throttle, and brake, with only the command-selected action branch active for each input.
- An auxiliary three-layer branch predicts vehicle speed from perception features during training using an L1 loss.
- All single- and multimodal networks use the same total loss, weighting action and speed losses through β.
A. Environment
The experiments use CARLA, an open-source simulator with multiple towns and weather conditions, enabling on-board evaluation of end-to-end driving models. The study also compares RGBD fusion architectures within CIL.
- CARLA is selected because prior end-to-end driving studies use it, supporting comparison with previous literature.
- CARLA enables evaluation in an on-board driving regime because static offline metrics do not correlate well enough with actual driving.
- The study explores early, mid, and late RGB-depth fusion within the CIL architecture.
- The simulator contains two towns with roads, intersections, traffic infrastructure, vegetation, and dynamic vehicles and pedestrians.
B. Driving benchmark
The CARLA benchmark separates pre-recorded dataset training from episode-based validation and testing across four driving tasks and multiple town-weather blocks. Success is measured by completed routes within a time budget.
- The benchmark evaluates straight driving, one turn, unrestricted navigation, and navigation with dynamic obstacles.
- The environment includes two towns and six weather conditions used to construct the benchmark scenarios.
- An episode succeeds when the driver completes its route within the time budget; success rate is 100 × (ES/ET).
- Training uses pre-recorded data, whereas validation and testing use actual driving episodes rather than pre-recorded datasets.
- The benchmark includes training conditions, a new town, new weather, and new town-and-weather combinations.
C. Training dataset
The study trains CIL models on a 25-hour CARLA dataset containing balanced weather conditions, sensor imagery, depth, semantic labels, navigation commands, and vehicle information.
- Dataset composition: The dataset contains 25 hours of driving in CARLA Town 1 with balanced weather conditions.It was collected by a hard-coded autopilot operating at 35 km/h on straight segments and slowing for turns.
- Evaluation materials: The dataset and benchmark materials support comparisons among RGB, depth, RGBD fusion variants, and a perfect-semantic-segmentation upper bound.The supplied tables cover training validation performance, benchmark success rates, previous-method comparisons, and dynamic-navigation infractions.
- Dataset composition: Training data include dense depth, pixel-wise semantic classes, navigation commands, and vehicle control information.The metadata include speed, steering angle, throttle, and braking, while perfect semantic segmentation is used to develop an upper-bound driver.
- Depth data: Depth values in CARLA span 0 to 1,000 meters and are encoded with 24-bit precision.The depth ground truth comes directly from the simulator’s Z-buffer.
- Evaluation materials: Model selection uses five training runs and chooses intermediate models by validation performance VP.VP combines validation success rates across weather, town, and town-weather conditions.
D. Training protocol
All CIL models use a shared preprocessing protocol: sensor channels are cropped to remove sky and close regions, resized to 200 × 88 pixels, and trained without data augmentation.
- Preprocessing: Original R/G/B/D channels are trimmed to remove sky and very close areas before down-scaling.The resulting channels have 200 × 88 pixel resolution.
- Preprocessing: Traditional photometric and geometric data-augmentation procedures are not used.Initial experiments found that these recipes did not produce better driving models.
E. Experimental results
Across CARLA experiments, multimodal RGBD generally outperforms single-modality inputs, with early fusion performing best for active depth; monocular-depth fusion remains promising but is less robust outside training conditions.
- Active-depth fusion: 91% validation performance is achieved by early fusion, compared with 48% for RGB, 74% for depth, 74% for mid fusion, and 67% for late fusion.These are the best validation performances among five training runs.
- Active-depth fusion: 94.00±0.00 success rate is achieved by active-depth early fusion in new town and weather conditions with dynamic objects.The corresponding rates are 46.67 ± 6.43 for RGB alone and 69.33±2.31 for depth alone.
- Active-depth fusion: Active depth alone clearly outperforms RGB in non-training conditions, while RGBD usually outperforms either single modality.For new towns, early fusion clearly outperforms mid and late fusion.
- Comparison with prior work: Compared with prior methods, active-depth RGBD early fusion has the highest success rate in the presence of dynamic traffic participants.The comparison is based only on reported original-CARLA benchmark performance because training details and datasets are not consistently available.
- Monocular-depth fusion: Monocular-depth early fusion outperforms RGB and depth alone under training conditions and shows performance similar to active depth.Its performance worsens after changing from training conditions, as does monocular depth estimation itself.
- Monocular-depth fusion: Static depth-estimation metrics may be insufficient for judging usefulness in driving tasks.The paper suggests end-to-end driving as a task-based evaluation protocol, while noting that static evaluations can mislead for driving models too.
- Dynamic navigation: RGB performs worst across the reported dynamic-navigation infraction metrics, while active depth fails frequently at intersections.The active-depth model avoids sidewalks and uses the curbside as a lane-keeping cue except at intersections.
V. CONCLUSION
The paper compares single- and multimodal RGB-depth perception for end-to-end driving using CIL in CARLA. Across early, mid, and late fusion settings, the experiments conclude that RGBD multimodality benefits end-to-end driving.
- The study compares single-modal RGB or depth models with RGBD models using early, mid, and late fusion.It evaluates both multisensory RGB-plus-active-depth and single-sensor RGB-plus-estimated-depth settings.
- CARLA provides the simulation environment, while branched conditional imitation learning supplies the end-to-end driving model.CARLA depth is post-processed to produce more realistic distance ranges and depth accuracy.
- Multimodality is evaluated with both active depth and depth estimated from RGB, covering multisensory and single-sensor configurations.The estimated-depth model extends the analysis beyond vehicles equipped with an active depth sensor.
- The experiments conclude that RGBD multimodality is beneficial for end-to-end autonomous driving.