MSMDFusion: Fusing LiDAR and Camera at Multiple Scales with Multi-Depth Seeds for 3D Object Detection

• Depth affects virtual-point location and semantics, but existing methods mainly focus on point interactions while underusing seed depths.Depth provides spatial placement through perspective projection and color-insensitive cues that help pixels of similar depths capture instance-related semantics.Source: §1 · p. 2
• Fine-grained voxel-space interaction is needed to select useful virtual-point information under LiDAR guidance, but point-cloud scale and structure make it costly.Prior alternatives use simple concatenation [28] or addition [9] in voxel space, or cross-attention in compressed BEV space [26].Source: §1 · p. 2
• MDU explores multiple depths around reference neighborhoods and combines camera features with depth to generate more reliable, semantically stronger virtual points.GMA-Conv selects camera voxel information under LiDAR-voxel guidance, aggregates it with grouped sparse convolutions, and uses voxel subsampling to obtain reliable LiDAR references efficiently.Source: §1 · p. 2
• MSMDFusion aggregates multi-granularity voxel features through cascade connections across scales and achieves state-of-the-art performance with 100 times fewer virtual points than the two BEVFusion methods [13] [16].The framework aims to combine camera semantics with LiDAR points consistently across scales.Source: §1 · p. 2
• MSMDFusion achieves state-of-the-art nuScenes detection performance with 71.5% mAP and 74.0% NDS using a single model; with simple greedy tracking, it achieves 74.0% AMOTA.Source: §1 · p. 2

Related Work

• Early methods projected 3D queries into camera images; MV3D and AVOD [7] associated 3D proposals with 2D RoI features.PointPainting [22] and PointAugmenting instead decorated raw 3D points with 2D semantics.Source: §2 · pp. 2–3
• EP-Net and 3D-CVF [29] fused modalities at point and proposal levels, but sparse 3D points meant this hard association wasted dense 2D semantics.Source: §2 · p. 3
• BEVFusion methods [13] [16] densely lifted every image-feature pixel into 3D and encoded the lifted points as a BEV map, but modalities did not interact before BEV fusion.MVP [28] instead selected foreground-object seeds and unprojected them into the LiDAR frame to augment the point cloud.Source: §2 · p. 3
• UVTR [9] built LiDAR and camera voxel branches and interacted them by addition, but its single-scale voxel space lost multilevel abstraction and simple addition limited fine-grained interaction.Source: §2 · p. 3

3.1. Framework Overview

• MSMDFusion extracts multi-scale features from LiDAR and camera, interacts across voxel-space scales with MDU and GMA-Conv, then sends the resulting features toward prediction.The framework also provides cross-scale connections to progressively combine features of different granularities.Source: §3.1 · p. 3

3.2. LiDAR and Camera Feature Extraction

• LiDAR features are extracted from voxelized points with 3D sparse convolutions, while ResNet50 with FPN extracts multi-scale semantic features from multi-view images.LiDAR features at different scales represent different abstraction levels and interact with their 2D counterparts through GMA-Conv.Source: §3.2 · pp. 3–4

3.3. Multi-Modal Interaction in Voxel Space

• Across scales, MDU and GMA-Conv combine geometric LiDAR and semantic camera features through voxel-space interaction.The interaction is designed to produce a unified scene representation from modalities that focus on geometric and semantic information, respectively.Source: §3.3 · p. 4

3.3.1 Multi-Depth Unprojection

• MDU assigns each image seed K neighboring reference-point depths, producing K virtual points to improve the chance of recovering actual 3D points.It decorates the points with depth-aware semantic features formed by combining image features with sparse reference-point depth, with per-depth weights controlling the semantic feature.Source: §3.3.1 · p. 4

3.3.2 Gated Modality-Aware Convolution

• LiDAR voxels guide camera-feature selection, after which sparse convolutions transform modality-specific features into a joint space and aggregate them into multimodal voxel features.To avoid quadratic retrieval costs, the implementation subsamples 2,048 camera voxels and achieves overall GMA-Conv complexity of O(M + N).Source: §3.3.2 · pp. 5–6

3.3.3 Multi-Scale Progressive Interaction

• Cascade connections align and combine features from current and previous scales, enabling thorough multi-granularity LiDAR-camera interaction in voxel space.Voxel downsampling aligns spatial resolutions between scales.Source: §3.3.3 · p. 6

3.4. Prediction in BEV Space

• Voxel features are compressed in height into BEV, fused with multimodal BEV features by a lightweight 2D convolution, and passed to a BEV encoder and detection head.Source: §3.4 · p. 6

4.1.1 Dataset and Metrics

• The experiments use nuScenes, with mAP averaged across ten classes at distance thresholds of 0.5m, 1m, 2m, and 4m, and NDS combining mAP with five true-positive metrics.nuScenes contains 10,000 scenarios split into 700 training, 150 validation, and 150 test scenes. The model uses ResNet-50 with FPN and VoxelNet backbones, extracts FPN levels C2–C5, and trains a LiDAR-only detector for 20 epochs followed by six joint-training epochs; inference uses neither test-time augmentation nor model ensemble.Source: §4.1.1 · p. 6

4.2. Comparison with State-of-the-art

• 71.5 mAP and 74.0 NDS establish MSMDFusion’s new state of the art on the nuScenes test set, with a 3.5% absolute AP gain on Bike over the strongest competitor.Table 1 reports single-model results without ensemble or test-time augmentation.Source: §4.2 · p. 6, Table 1 · p. 7
• With 16k virtual points per frame—100 times fewer than BEVFusion’s 2M/5M—MSMDFusion still outperforms both methods in mAP and NDS.BEVFusion [16] has the fastest inference speed due to its CUDA camera-to-BEV transformation; without such a hardware-friendly implementation, BEVFusion [13] processes many virtual points and MSMDFusion runs much faster.Source: §4.2 · pp. 6–7, Table 3 · p. 7
• MSMDFusion consistently outperforms other single-model competitors across all nuScenes tracking evaluation metrics using the greedy tracker.The comparison excludes test-time augmentation and ensemble methods.Source: §4.2 · p. 7, Table 2 · p. 7

4.3.1 Ablation of proposed components

• With all proposed components, mAP rises from 60.26% to 66.93% and NDS from 65.62% to 68.93% over the baseline.The ablations use TransFusion-L as the baseline and a randomly sampled quarter of nuScenes training data for efficiency; virtual points, early depth-color interaction, and GMA-Conv each contribute improvements.Source: §4.3.1 · pp. 7–8, §4.3 · p. 7, Table 4 · p. 8

4.3.2 Effects of multi-scale interaction

• Multi-scale interaction consistently outperforms single-scale models, with the complete model gaining approximately 2% mAP and 1% NDS over the best single-scale model.Among single-scale settings, larger spatial-resolution scales generally yield larger gains over the baseline, attributed to finer-grained image and LiDAR features.Source: §4.3.2 · p. 8, Table 5 · p. 8

4.3.3 Discussion on MDU

• Increasing depths per seed from K=1 to 6 improves virtual-point recall without significantly increasing mean distance error, while K=10 introduces noise and reduces detection performance.Increasing seeds to N=200 with K=1 does not match the multi-depth gains and can hurt performance due to added noise.Source: §4.3.3 · pp. 8–9, Figure 6 · p. 8, Table 6 · p. 8

5. Conclusion

• MSMDFusion achieves state-of-the-art performance on the nuScenes dataset for 3D object detection.Source: §5 · p. 9
• Multi-Depth Unprojection produces reliable virtual points with depth-aware semantics from images.Source: §5 · p. 9
• Gated Modality-Aware Convolution enables fine-grained, controllable LiDAR-camera fusion at each scale, with multi-granularity information combined across scales for the prediction head.Source: §5 · p. 9