Source-linked AI summary
DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection
Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Bo Wu, Yifeng Lu, Denny Zhou, Quoc V. Le, Alan Yuille, Mingxing Tan
TL;DR
Multi-modal 3D detection must combine complementary lidar and camera signals despite difficult alignment after feature transformation and aggregation. DeepFusion addresses this with InverseAug and LearnableAlign for aligned deep-feature fusion, achieving state-of-the-art Waymo performance with improved robustness.
Problem
Effective fusion of complementary lidar and camera signals remains challenging because transformed deep features from the two modalities are difficult to align.
Method
DeepFusion fuses deep lidar and camera features using InverseAug to reverse geometric augmentations and LearnableAlign to learn cross-modal correlations with cross-attention.
Results
DeepFusion achieves state-of-the-art performance on Waymo Open Dataset, improving PointPillars, CenterPoint, and 3D-MAN by 6.7, 8.9, and 6.2 LEVEL 2 APH, respectively.
Takeaways & Limitations
DeepFusion is a simple, generic detector family compatible with voxel-based methods and more robust than single-modal baselines to corruptions and out-of-distribution data.
Takeaways & Limitations
Deep-feature fusion makes camera-lidar alignment less straightforward, particularly when heterogeneous geometric augmentations create misalignment.
Abstract
from arXiv · showhide
Lidars and cameras are critical sensors that provide complementary information for 3D detection in autonomous driving. While prevalent multi-modal methods simply decorate raw lidar point clouds with camera features and feed them directly to existing 3D detection models, our study shows that fusing camera features with deep lidar features instead of raw points, can lead to better performance. However, as those features are often augmented and aggregated, a key challenge in fusion is how to effectively align the transformed features from two modalities. In this paper, we propose two novel techniques: InverseAug that inverses geometric-related augmentations, e.g., rotation, to enable accurate geometric alignment between lidar points and image pixels, and LearnableAlign that leverages cross-attention to dynamically capture the correlations between image and lidar features during fusion. Based on InverseAug and LearnableAlign, we develop a family of generic multi-modal 3D detection models named DeepFusion, which is more accurate than previous methods. For example, DeepFusion improves PointPillars, CenterPoint, and 3D-MAN baselines on Pedestrian detection for 6.7, 8.9, and 6.2 LEVEL_2 APH, respectively. Notably, our models achieve state-of-the-art performance on Waymo Open Dataset, and show strong model robustness against input corruptions and out-of-distribution data. Code will be publicly available at https://github.com/tensorflow/lingvo/tree/master/lingvo/.
1. Introduction
DeepFusion argues that effective alignment is the central challenge in combining complementary lidar and camera signals for 3D detection. It addresses this challenge with deep-feature fusion, InverseAug, and LearnableAlign, achieving state-of-the-art Waymo results and improved robustness.
- Motivation: Existing multi-modal detectors commonly fuse camera features with raw lidar points or combine features after separate extraction, but correspondence remains a major challenge.The paper focuses on effective mid-level fusion rather than input-level point decoration.
- Method: InverseAug reverses geometry-related augmentations, while LearnableAlign uses cross-attention to dynamically learn correlations between lidar and camera features.Together, the techniques support accurate deep-feature-level alignment with marginal computational cost.
- Method: DeepFusion is an end-to-end, generic plug-in compatible with voxel-based detectors including PointPillars and CenterPoint.The models are designed as building blocks for existing 3D detection frameworks.
- Results: 6.7, 8.9, and 6.2 LEVEL 2 APH improvements are reported over PointPillars, CenterPoint, and 3D-MAN, respectively, on Waymo Open Dataset.DeepFusion also improves Pedestrian LEVEL 2 APH by 7.4 over PointAugmenting on the validation set.
- Results: DeepFusion achieves state-of-the-art Waymo performance and is more robust than its single-modal baseline against input corruptions and out-of-distribution data.The largest reported improvements come from recognition and localization of long-range objects.
2. Related Work
Related work combines lidar and camera information through point decoration, voxel- or pillar-based lidar processing, or mid-level feature fusion. The paper identifies effective alignment between camera and lidar features as a missing component of prior mid-level approaches.
- Lidar 3D Object Detection: Point clouds may be processed directly as unordered points or represented as dense range images containing depth information.These representations support distinct families of point-cloud detection methods.
- Lidar 3D Object Detection: Voxel-based methods discretize point clouds into 3D grids, whereas pillar-based methods collapse voxels sharing a z-axis into bird’s-eye-view 2D pillars.PointPillars is selected as the paper’s baseline because 2D pillars are relatively easy and fast to process.
- Lidar-Camera Fusion: Input-level fusion methods decorate raw lidar points with camera-derived segmentation or detection features before 3D point-cloud processing.PointPainting and PointAugmenting exemplify this approach and typically use pretrained camera feature extractors.
- Lidar-Camera Fusion: Mid-level fusion methods share information between 2D and 3D backbones, but prior work lacks an effective mechanism for aligning camera and lidar features.The paper presents alignment as the key missing piece for effective end-to-end multi-modal detection.
3. DeepFusion
DeepFusion fuses deep camera and lidar features mid-level, addressing the alignment challenges introduced by feature extraction, aggregation, and geometric augmentation. InverseAug restores geometric correspondence, while LearnableAlign uses cross-attention to select relevant camera features for each lidar voxel.
- 3.1. Deep Feature Fusion Pipeline: Deep feature fusion avoids sending camera features through point-cloud-specific voxelization while enabling end-to-end camera-feature learning.Camera and lidar features are fused after feature extraction rather than decorating raw lidar points.
- 3.1. Deep Feature Fusion Pipeline: Deep feature alignment is challenging because heterogeneous geometric augmentation can misalign transformed camera and lidar features.The pipeline’s advantages therefore come with a less straightforward alignment problem at the deep-feature level.
- 3.2. Impact of Alignment Quality: +2.6 AP is the multi-modal gain with no RandomRotation, but only +0.4 AP remains at a 45° maximum rotation.The experiment varies lidar rotation while keeping camera images unchanged; the reported metric is Waymo pedestrian LEVEL 1 AP improvement.
- 3.3. Boosting Alignment Quality: InverseAug reverses geometric augmentations at fusion time before projecting augmented 3D key points into camera coordinates.It can handle different key points, including voxel centers, and cases where both modalities are augmented.
- 3.3. Boosting Alignment Quality: LearnableAlign uses cross-attention to dynamically model correlations between each lidar voxel and its corresponding camera features.Because a voxel maps to many camera pixels, the method transforms voxel features into queries and camera features into keys and values.
4. Experiments
Experiments evaluate DeepFusion on Waymo using multiple baselines, implementation variants, alignment components, distance ranges, and robustness settings. DeepFusion achieves leading benchmark performance, consistently improves lidar-only models, gains most for long-range objects, and remains robust to corruptions and OOD data.
- State-of-the-art performance on Waymo Data: DeepFusion-Ens achieves the best results on the Waymo Challenge Leaderboard, improving 2.42 APH/L2 over AFDetV2.
- DeepFusion is a generic fusion method: DeepFusion consistently improves all six evaluated single-modal baselines and is presented as a generic plug-in for voxel-based detection frameworks.The comparisons cover PointPillars, CenterPoint, 3D-MAN, and their “++” versions.
- Where does the improvement come from?: 6.6% LEVEL 2 gain for objects beyond 50m exceeds the 1.5% gain for objects within 30m.DeepFusion improves accuracy in every distance range, with larger gains for long-range objects.
- Alignment analysis: LearnableAlign focuses attention on discriminative regions and object extremities, while both InverseAug and LearnableAlign are identified as critical in ablation results.LearnableAlign improves LEVEL 2 performance from 66.4 APH to 67.0 APH.
- DeepFusion is more robust: Multi-modal models are more robust to input corruptions, with only 0.2/0.5 L2 APH drops under Laser/Pixel Noise and an 8.0 LEVEL 2 APH OOD gain.The OOD gain is compared with a 4.0 LEVEL 2 APH improvement on in-distribution data.
5. Conclusion
The paper addresses multi-modal 3D detection by aligning deep lidar and camera features with InverseAug and LearnableAlign. The resulting generic DeepFusion detectors achieve state-of-the-art Waymo performance.
- DeepFusion targets late-stage fusion by aligning deep features from different modalities with InverseAug and LearnableAlign.
- DeepFusion detectors are described as simple, generic, and effective, achieving state-of-the-art performance on the Waymo Open Dataset.
A.1. Impact of Alignment Quality
Additional experiments show that larger data-augmentation perturbations reduce the benefit of multi-modal fusion, including under RandomFlip.
- As RandomFlip magnitude increases, the performance benefit from multi-modal fusion diminishes, indicating the importance of accurate alignment.RandomFlip is tested at probabilities of 0%, 50%, and 100%.
A.2. Implementation Details of 3D Detectors
The implementation details describe improved point-cloud detectors, training-time frame dropping, and model ensembling across resolutions and modalities.
- PointPillars is improved by replacing its single-layer voxel encoder with a NAS-designed multilayer perceptron and changing ReLU to SILU.
- During DropFrame training, randomly selected frames are removed before collapsing, while inference uses all frames.The illustrated 5-frame example drops Frames 2 and 4.
- The ensemble contains five model types spanning single-modality resolutions of 512, 704, and 1024 and multi-modality resolutions of 512 and 704.
- Each model type is trained five times with different random seeds, and the top-k models are selected using validation performance.
A.3. Comparison with Larger Single-Modal Models
Under matched computational budgets, DeepFusion achieves higher detection performance than larger single-modal models, while further single-modal scaling provides only marginal gains.
- At 0.32s latency, DeepFusion achieves 66.5 L2 APH versus 65.7 L2 APH for the single-modal model.
- Further scaling the single-modal model yields marginal gains and caps performance at 66.5 L2 APH, still below DeepFusion.
- DeepFusion significantly outperforms single-modal models across all evaluated latency budgets.