Source-linked AI summary
FusionPainting: Multimodal Fusion with Adaptive Attention for 3D Object Detection
Shaoqing Xu, Dingfu Zhou, Jin Fang, Junbo Yin, Zhou Bin, Liangjun Zhang
TL;DR
Accurate 3D obstacle detection requires combining LiDAR’s range information with camera-derived semantic detail despite their complementary segmentation errors. FusionPainting adaptively fuses 2D and 3D semantic information, paints the point cloud with the fused labels, and improves detection across nuScenes baselines while outperforming prior methods.
Problem
LiDAR and camera sensors provide complementary information, but 2D segmentation blurs object boundaries and 3D segmentation often gives worse category classification.
Method
FusionPainting obtains 2D and 3D semantic segmentation, adaptively fuses the results with voxel-level attention, and sends the painted point cloud to a 3D detector.
Results
FusionPainting consistently improves mAP and NDS over three baselines and outperforms previous multimodal methods on nuScenes.
Takeaways & Limitations
The framework is detector independent and can be used with other 3D object detectors while combining complementary 2D and 3D semantic information.
Abstract
from arXiv · showhide
Accurate detection of obstacles in 3D is an essential task for autonomous driving and intelligent transportation. In this work, we propose a general multimodal fusion framework FusionPainting to fuse the 2D RGB image and 3D point clouds at a semantic level for boosting the 3D object detection task. Especially, the FusionPainting framework consists of three main modules: a multi-modal semantic segmentation module, an adaptive attention-based semantic fusion module, and a 3D object detector. First, semantic information is obtained for 2D images and 3D Lidar point clouds based on 2D and 3D segmentation approaches. Then the segmentation results from different sensors are adaptively fused based on the proposed attention-based semantic fusion module. Finally, the point clouds painted with the fused semantic label are sent to the 3D detector for obtaining the 3D objection results. The effectiveness of the proposed framework has been verified on the large-scale nuScenes detection benchmark by comparing it with three different baselines. The experimental results show that the fusion strategy can significantly improve the detection performance compared to the methods using only point clouds, and the methods using point clouds only painted with 2D segmentation information. Furthermore, the proposed approach outperforms other state-of-the-art methods on the nuScenes testing benchmark.
I. INTRODUCTION
LiDAR supplies accurate obstacle ranges but loses detailed geometry, while cameras provide high-resolution texture and color without depth. FusionPainting addresses their complementary errors by adaptively combining 2D and 3D semantic information before 3D detection, improving nuScenes results.
- LiDAR provides accurate obstacle range measurements, whereas camera sensors provide high-resolution texture and color but lose depth during projection.
- 2D segmentation suffers boundary blurring that becomes more severe after reprojection, while 3D segmentation preserves boundaries but often classifies categories less accurately.The paper attributes these complementary weaknesses to limited 2D feature-map resolution and missing RGB texture information in 3D point clouds.
- FusionPainting adaptively fuses 2D and 3D semantic information at voxel level, then sends point clouds painted with fused labels to a 3D detector.The framework uses an attention module that learns context features to combine the semantic results.
- The framework is designed as a general semantic-level fusion approach and reports superiority over other methods on the nuScenes benchmark.The stated contributions include semantic fusion, context-based attention, and SOTA nuScenes results.
II. RELATED WORK
Point-cloud 3D object detectors are organized into projection-based, voxel-based, and point-based approaches, with representative methods using range images, sparse voxels, pillars, or raw points.
- Existing point-cloud 3D object detection methods are categorized as projection-based, voxel-based, and point-based approaches.
- RangeRCNN uses range images and BEV anchors, while VoxelNet learns unified voxel representations and SECOND uses sparse convolution to reduce 3D convolutional cost.
- CenterPoint is an anchor-free center-based detector, PointPillars encodes pillar features as a pseudo-image, and PointRCNN generates proposals directly from raw point clouds.
- PV-RCNN combines multi-scale 3D voxel CNNs with a PointNet-based network to learn more discriminative features.
B. Multi-modal Fusion
Multimodal detection methods combine LiDAR and camera data at different stages, while FusionPainting performs semantic-level fusion before detector processing. Its adaptive attention module combines complementary 2D and 3D segmentation information.
- Multimodal detection approaches include early fusion of raw data, deep fusion of learned features, and late fusion of detection boxes.
- FusionPainting uses 2D images and 3D point clouds to obtain accurate obstacle locations through adaptive semantic-level integration.
- The framework contains multimodal semantic segmentation, adaptive attention, and 3D detector modules arranged sequentially.
- Off-the-shelf 2D and 3D segmentors produce semantic results, the attention strategy combines their strengths and suppresses drawbacks, and painted point clouds enter modern 3D detectors.
A. Multi-modal Semantic Segmentation
FusionPainting obtains semantic labels independently from 2D images and 3D point clouds, then adaptively fuses them at voxel level before painting the point cloud for detection.
- 2D Painting Module: The 2D painting module segments images into pixel-wise labels and reprojects them onto corresponding 3D points as one-hot semantic features.It uses camera projection matrices to transfer image semantics into the point cloud.
- 3D Painting Module: The 3D painting module obtains point-wise semantic labels from Cylinder3D or other segmentors, using 3D bounding boxes to generate labels without extra point-wise annotations.Points inside obstacle boxes receive class labels, while points outside all boxes are treated as background.
- Semantic Fusion Motivation: 2D segmentation can blur object boundaries and misclassify background regions after reprojection, motivating fusion with clearer point-cloud semantics.The paper illustrates a background frustum behind a truck that is incorrectly classified as foreground.
- Adaptive Attention Module: The adaptive attention module combines point coordinates, attributes, and 2D/3D labels in fixed-size voxels to learn context-aware semantic fusion.Voxel-level processing reduces memory consumption compared with point-level fusion.
- Adaptive Attention Module: Local and global voxel features are formed with MLPs and max pooling, then an attention score weights the semantic vectors before producing adaptively fused painted voxels.The resulting enhanced voxels contain the raw point coordinates together with fused semantic information.
C. 3D Object Detection Module
The 3D detector consumes painted voxels generated by adaptive semantic fusion, and the evaluation table reports nuScenes validation results for FusionPainting-enhanced baselines.
- 3D Detector Module: Any off-the-shelf 3D object detector can predict detections from the painted point cloud produced by the adaptive attention module.The detector receives the adaptively painted voxels as input.
- Evaluation Results: The reported validation table compares baseline detectors with versions marked “*” after adding FusionPainting.The table defines NDS and mAP and abbreviates traffic cone, motorcycle, and construction vehicle categories.
IV. EXPERIMENTAL RESULTS
The experiments evaluate FusionPainting on a large-scale autonomous-driving 3D object-detection dataset and report results across three baseline detectors.
- Experimental Results: The evaluation first describes the experiments and then reports results for three baseline 3D object detectors.The stated purpose is to evaluate the effectiveness of FusionPainting.
A. Detectors, Dataset and Implementation Details
The experiments test FusionPainting across three established detectors on nuScenes, using standard detection metrics and specified segmentation implementations.
- Detectors: FusionPainting is implemented on SECOND, PointPillars, and CenterPoint to test its universality across detector architectures.The baselines span sparse voxel, pillar-based, and anchor-free center-based detection designs.
- Dataset: nuScenes contains 1,000 scenes divided into 700 training, 150 validation, and 150 testing scenes with annotated key frames from a 32-line LiDAR.The split includes 28,130 training, 6,019 validation, and 6,008 testing samples.
- Evaluation Metrics: Evaluation uses nuScenes mAP based on BEV center-distance thresholds and NDS, which combines mAP with error metrics including ATE and ASE.The center-distance thresholds are {0.5, 1, 2, 4} meters.
- Implementation Details: HTCNet and Cylinder3D serve as the 2D and 3D segmentors, pretrained on nuImages and nuScenes-related data respectively.The adaptive attention module uses m = 11, C1 = 64, and C2 = 128.
B. Evaluation Results
FusionPainting improves 3D detection across multiple baselines and exceeds prior published methods on nuScenes. Its gains are reported on both mAP and NDS, including stronger performance for difficult small-object categories.
- FusionPainting was evaluated on both validation and test splits of the nuScenes benchmark.
- More than 10 mAP points and 5 NDS points were gained over all three integrated baselines.For PointPillars, the reported improvements reached 17.31 mAP points and 8.45 NDS points.
- 36.19% AP improvement for Bicycle was reported with Second, compared with 48.83% for PointPillars and 26.84% for CenterPoint.The authors note that these categories are small objects with few LiDAR points.
- FusionPainting improved CenterPoint by 3.1 NDS points and 6.0 mAP points on the nuScenes test benchmark.
- Compared with the published multimodal method 3DCVF, FusionPainting improved NDS by 8.1 points and mAP by 13.6 points.
C. Qualitative Results
Qualitative comparisons show that FusionPainting combines complementary 2D and 3D semantic information to produce more accurate detections than raw, 2D-painted, or 3D-painted point clouds.
- 2D painting causes false positives from frustum blurring, while 3D painting produces worse foreground segmentation than 2D image segmentation.
- FusionPainting combines complementary 2D and 3D segmentation information and detects objects more accurately.
D. Ablation Studies
Ablation studies show that both semantic sources contribute to detection, with the largest average gains coming from their combination with adaptive attention.
- The ablation study evaluates the impact of different fusion modules on the validation split under consistent settings.
- 3D semantic segmentation alone contributes about 9.81% mAP and 3.92% NDS on average.
- 2D semantic segmentation contributes about 21.90% mAP and 8.46% NDS on average.The higher improvement is attributed to better recall for distant objects, where point clouds are sparse.
V. CONCLUSION AND FUTURE WORKS
The paper concludes that FusionPainting aggregates 2D and 3D semantic information through adaptive attention and can be used with different 3D detectors. Future work will integrate segmentation branches more deeply and pursue joint instance segmentation and 3D detection.
- FusionPainting aggregates semantic information from 2D and 3D segmentation networks for 3D object detection.
- The framework uses an adaptive attention module to learn separate attention masks for the 2D and 3D segmentation results.
- FusionPainting is detector independent and can be used with other 3D object detectors.
- The planned integrated framework will simultaneously address instance segmentation and 3D object detection.