PaperlayerView exact revision on arXiv

Source-linked AI summary

MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving

Jiale Li, Hang Dai, Hao Han, Yong Ding

arXiv:2303.08600v1cs.CV

TL;DR

LiDAR-only 3D segmentation struggles with small and distant objects, while multi-modal methods face heterogeneous features, limited FOV overlap, and augmentation challenges. MSeg3D jointly extracts and fuses LiDAR-camera features, completes missing camera features, and applies asymmetric augmentation, achieving state-of-the-art results across nuScenes, Waymo, and SemanticKITTI with reported robustness under degraded inputs.

  • Problem

    LiDAR-only segmentation suffers on small and distant objects, while robust multi-modal segmentation remains under-explored because of modality heterogeneity, limited FOV intersection, and augmentation challenges.

  • Method

    MSeg3D jointly optimizes intra-modal feature extraction and inter-modal fusion using GF-Phase, cross-modal feature completion, SF-Phase, and asymmetric multi-modal data augmentation.

  • Results

    MSeg3D achieves state-of-the-art 3D segmentation performance on nuScenes, Waymo, and SemanticKITTI.

  • Takeaways & Limitations

    The model improves multi-modal segmentation beyond the shared sensor FOV and remains robust under malfunctioning multi-camera input and multi-frame point-cloud input.

  • Takeaways & Limitations

    Multi-camera input makes the image backbone the efficiency bottleneck, motivating further study of real-time image segmentation networks.

Abstract

from arXiv · show

LiDAR and camera are two modalities available for 3D semantic segmentation in autonomous driving. The popular LiDAR-only methods severely suffer from inferior segmentation on small and distant objects due to insufficient laser points, while the robust multi-modal solution is under-explored, where we investigate three crucial inherent difficulties: modality heterogeneity, limited sensor field of view intersection, and multi-modal data augmentation. We propose a multi-modal 3D semantic segmentation model (MSeg3D) with joint intra-modal feature extraction and inter-modal feature fusion to mitigate the modality heterogeneity. The multi-modal fusion in MSeg3D consists of geometry-based feature fusion GF-Phase, cross-modal feature completion, and semantic-based feature fusion SF-Phase on all visible points. The multi-modal data augmentation is reinvigorated by applying asymmetric transformations on LiDAR point cloud and multi-camera images individually, which benefits the model training with diversified augmentation transformations. MSeg3D achieves state-of-the-art results on nuScenes, Waymo, and SemanticKITTI datasets. Under the malfunctioning multi-camera input and the multi-frame point clouds input, MSeg3D still shows robustness and improves the LiDAR-only baseline. Our code is publicly available at \url{https://github.com/jialeli1/lidarseg3d}.

1. Introduction

MSeg3D addresses multi-modal 3D segmentation difficulties by jointly learning modality-specific features and fusing information across and beyond the sensors’ shared field of view. Its components improve segmentation metrics, reduce the mIoU–mIoU1 gap, and achieve state-of-the-art results across three autonomous-driving datasets.

  • 1. Introduction: The method jointly optimizes intra-modal feature extraction and inter-modal feature fusion to exploit complementary information from heterogeneous LiDAR and camera modalities.This design targets the mismatch between sparse point clouds and dense images, whose feature extractors are typically developed separately.
  • 1. Introduction: Cross-modal feature completion supplies pseudo-camera features for points outside the shared sensor FOV, while SF-Phase fuses semantic information for all visible points.SF-Phase uses semantic relations between points and categories to complement geometry-based fusion.
  • 1. Introduction: MSeg3D achieves state-of-the-art 3D segmentation performance on nuScenes, Waymo, and SemanticKITTI.The method also placed second in the Waymo 3D semantic segmentation challenge at CVPR 2022.
  • 1. Introduction: As proposed components accumulate, mIoU and mIoU1 increase while the gap between them decreases.mIoU1 evaluates only points inside the sensor FOV intersection, whereas mIoU includes the broader point set.
  • 1. Introduction: Asymmetric transformations applied independently to point clouds and images diversify multi-modal training samples and improve robustness.The augmentation is designed to address the difficulty of training multi-modal models with reliable performance.

2. Related Work

Prior work develops efficient LiDAR-only segmentation through point, projected-image, and voxel representations, while multi-modal methods remain limited by feature quality and sensor-FOV overlap. MSeg3D is presented as an integrated architecture for broader multi-modal fusion.

  • 2. Related Work: LiDAR-only segmentation methods use point, 2D image, or 3D voxel representations, each trading off computation, projected 3D information, or sparse-convolution efficiency.Point methods can be computationally heavy, projection can damage 3D information, and voxel methods explicitly preserve 3D structure.
  • 2. Related Work: mIoU1 measures segmentation performance only on points inside the sensor FOV intersection, excluding outside points.This metric follows the evaluation practice used by PMF and related methods.
  • 2. Related Work: Existing multi-modal methods commonly add RGB or separately learned image features to point-cloud representations, but PMF discards points outside the camera FOV.This leaves multi-modal fusion and segmentation unavailable for those outside-FOV points.
  • 2. Related Work: MSeg3D’s overview combines geometry-based GF-Phase with semantic-based SF-Phase modules for multi-modal feature fusion.GF-Phase mainly uses GFFM, while SF-Phase contains LiDAR SFAM, camera SFAM, and SFFM.

3. Method

MSeg3D jointly extracts LiDAR and camera features, then fuses them through geometry-based, completion, and semantic-based modules. Its semantic fusion models category relations and point-specific modality importance, while asymmetric augmentation diversifies training samples.

  • 3.2. Intra-modal Feature Extraction: MSeg3D jointly optimizes parallel LiDAR and image backbones with inter-modal fusion to learn relevant representations from heterogeneous modalities.LiDAR features are extracted with a sparse 3D U-Net, while camera features come from a trainable image backbone.
  • 3.2. GF-Phase: Geometry-based Feature Fusion: Geometry-based fusion projects point-wise LiDAR and camera features into a shared space, concatenates them, and produces fused point features with an MLP.Point-wise features are obtained by devoxelizing voxel features and bilinearly interpolating image feature maps at projected coordinates; points outside camera FOV are temporarily zero-padded.
  • 3.3. SF-Phase: Semantic-based Feature Fusion: Cross-modal completion supplies pseudo-camera features for points outside the sensor FOV intersection under cross-modal supervision.This addresses the absence of realistic camera features outside the overlapping field of view.
  • 3.3. SF-Phase: Semantic-based Feature Fusion: Semantic fusion aggregates LiDAR and camera features into category-wise embeddings, models their intra- and inter-modal relations with self-attention, and uses cross-attention to select relevant semantics for each point.The point-wise query attends to multimodal semantic embeddings, allowing modality importance to vary across points and supporting points inside and outside the FOV intersection.
  • 3.6. Asymmetric Multi-modal Data Augmentation: Asymmetric augmentation independently transforms LiDAR, camera images, and local camera views, substantially diversifying multimodal training samples.LiDAR transformations can be preserved while camera transformations are applied after synchronously transforming image coordinates; local cameras can also be transformed independently.

4. Experiment

MSeg3D is evaluated across three autonomous-driving datasets, against state-of-the-art methods, through fusion ablations, distance analyses, and robustness tests. The experiments show strong multi-modal performance, improved handling of distant and out-of-view points, and robustness to camera failures and multi-frame inputs.

  • Results on nuScenes: MSeg3D achieves the best reported nuScenes mIoU and fwIoU among compared methods, especially improving segmentation of small objects such as pedestrians and traffic cones.The comparison includes public and private multi-modal submissions.
  • Results on Waymo: MSeg3D achieves state-of-the-art performance on Waymo despite denser LiDAR and incomplete camera coverage without a rear camera.The authors note that stronger point-cloud backbones could further improve performance.
  • Results on SemanticKITTI: On SemanticKITTI, MSeg3D improves over LiDAR-only methods and outperforms other multi-modal methods using the inside-FOV mIoU1 evaluation protocol.The protocol excludes points outside the camera-LiDAR FOV intersection.
  • Multi-modal feature fusion: The feature-fusion ablation shows that supervised cross-modal completion and SF-Phase improve both mIoU and mIoU1 while narrowing their gap.The analysis decomposes supervised completion into Lpixel2point and pseudo-camera feature completion.
  • Distance analysis: Distance-based evaluation shows that the multi-modal model alleviates long-range degradation, with a 0.56 mIoU gap on Waymo despite extensive out-of-view points.The benefit over LiDAR-only segmentation increases with distance on nuScenes and Waymo as point sparsity grows.
  • Robustness analysis: MSeg3D remains effective when cameras malfunction and gains further improvements from multi-frame point clouds after LiDAR-only gains saturate.With all cameras removed, MSeg3D still outperforms the LiDAR-only baseline; multi-frame input is presented as an optional extension.

5. Conclusion

MSeg3D combines LiDAR and multi-camera sensing for 3D semantic segmentation, addressing fusion outside the shared field of view and training reliability. It achieves state-of-the-art performance across nuScenes, Waymo, and SemanticKITTI, with experiments validating improvements and robustness.

  • MSeg3D achieves state-of-the-art 3D semantic segmentation performance on nuScenes, Waymo, and SemanticKITTI.
  • Cross-modal feature completion and semantic-based feature fusion address multi-modal fusion outside the sensor field-of-view intersection.
  • Asymmetric multi-modal data augmentation enables effective training with reliable performance.
  • The work was supported by the National Key Research and Development Program of China and YUNJI Technology Co. Ltd.
Loading 2303.08600v1…