Source-linked AI summary
2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds
Xu Yan, Jiantao Gao, Chaoda Zheng, Chao Zheng, Ruimao Zhang, Shenghui Cui, Zhen Li
TL;DR
Fusion-based segmentation depends on paired images, LiDAR, and strict point-to-pixel alignment, limiting practical point-cloud-only use. 2DPASS distills multiscale multimodal knowledge into a pure 3D network during training, achieving state-of-the-art results on SemanticKITTI and NuScenes, including first place in both SemanticKITTI scan settings.
Problem
Fusion-based methods require paired camera images and LiDAR with strict point-to-pixel mappings during training and inference, limiting practical deployment.
Method
2DPASS uses auxiliary multimodal fusion and multi-scale fusion-to-single knowledge distillation to transfer image priors into a pure 3D network.
Results
2DPASS achieves state-of-the-art performance on SemanticKITTI and NuScenes, ranking first in SemanticKITTI single- and multiple-scan tracks.
Takeaways & Limitations
The scheme enhances pure LiDAR semantic segmentation while requiring only point clouds as inputs at inference.
Abstract
from arXiv · showhide
As camera and LiDAR sensors capture complementary information used in autonomous driving, great efforts have been made to develop semantic segmentation algorithms through multi-modality data fusion. However, fusion-based approaches require paired data, i.e., LiDAR point clouds and camera images with strict point-to-pixel mappings, as the inputs in both training and inference, which seriously hinders their application in practical scenarios. Thus, in this work, we propose the 2D Priors Assisted Semantic Segmentation (2DPASS), a general training scheme, to boost the representation learning on point clouds, by fully taking advantage of 2D images with rich appearance. In practice, by leveraging an auxiliary modal fusion and multi-scale fusion-to-single knowledge distillation (MSFSKD), 2DPASS acquires richer semantic and structural information from the multi-modal data, which are then online distilled to the pure 3D network. As a result, equipped with 2DPASS, our baseline shows significant improvement with only point cloud inputs. Specifically, it achieves the state-of-the-arts on two large-scale benchmarks (i.e. SemanticKITTI and NuScenes), including top-1 results in both single and multiple scan(s) competitions of SemanticKITTI.
1 Introduction
2DPASS addresses the practical limits of camera–LiDAR fusion by using images during training while retaining point-cloud-only operation at inference. Its multi-scale fusion-to-single distillation improves 3D segmentation across major benchmarks.
- Motivation: Camera and LiDAR provide complementary appearance, texture, depth, and sensing-range information, motivating multimodal semantic segmentation.Cameras offer dense color and fine-grained texture, whereas LiDAR provides robust, wide-ranging depth but sparse, textureless data.
- Limitations: Fusion-based methods require point-to-pixel mappings, but differing camera and LiDAR fields of view leave some LiDAR points uncovered.This limitation is especially pronounced when front-facing cameras observe only a small portion of a 360-degree LiDAR scan.
- Approach: 2DPASS uses a model-independent training scheme that enhances 3D representation learning with image-derived semantic and structural information.The method is designed to improve any 3D semantic segmentation network with minor structural modification.
- Approach: For aligned regions, auxiliary multimodal fusion transfers image knowledge to the 3D model through multi-scale fusion-to-single knowledge distillation.Pure point clouds train the model for non-overlap regions, while MSFSKD preserves modal-specific 3D ability during knowledge transfer.
- Results: 2DPASS achieves significant gains on SemanticKITTI and NuScenes, ranking first in SemanticKITTI single- and multiple-scan tracks.The introduction reports state-of-the-art performance on both large-scale benchmarks.
2 Related Work
Prior work includes camera-only, LiDAR-only, multimodal fusion, and knowledge-distillation methods. These approaches differ in representation, efficiency, information loss, and whether multimodal knowledge remains available during inference.
- Camera-Based Methods: Camera-based segmentation predicts pixel-wise image labels but remains ambiguous in depth sensing and less robust in low-light conditions.Representative advances explore multiscale features, dilated convolution, and attention mechanisms.
- LiDAR-Based Methods: LiDAR segmentation uses point-based, projection-based, or voxel-based representations with different accuracy–efficiency trade-offs.Point-based methods can be time-consuming, projection loses information, and voxel frameworks balance efficiency and effectiveness.
- Multi-Sensor Methods: Multisensor methods fuse complementary camera and LiDAR information by mapping image features or logits into point-cloud representations.Examples include RGBAL and PointPainting, which use polar-grid, bird’s-eye, or spherical mappings.
- Multi-Sensor Methods: 2DPASS transfers multimodal knowledge to a single point-cloud modality through multiscale fusion-to-single distillation.Its workflow independently encodes image and point-cloud features before transferring complementary 2D knowledge to the 3D network.
- Knowledge Distillation: Knowledge distillation research transfers information between networks by matching logits, features, attention maps, or other representations.The paper places its cross-modal strategy within this broader knowledge-transfer literature.
3 Method
2DPASS trains a point-cloud segmentation model with auxiliary 2D image priors, using paired multi-scale features during training and retaining only the enhanced 3D network for inference. Its MSFSKD module fuses modality-specific features before unidirectionally distilling multi-modal knowledge into the point-cloud representation.
- Overall workflow: 2DPASS independently encodes multi-scale image and point-cloud features, then uses MSFSKD to enhance the 3D network with texture- and color-aware 2D priors.A cropped 480 × 320 image patch accelerates training without performance drop, while the 3D branch uses sparse convolution.
- Point-to-pixel correspondence: Point-to-pixel correspondence projects LiDAR points into the image plane so image features can be transferred into point-wise 2D features.The projection uses camera calibration and extracts features from the corresponding image locations after upsampling each feature map.
- Point-to-voxel correspondence: Point-wise 3D features are generated by mapping points to voxels and interpolating sparse-convolution features back onto the original point cloud.Points outside the image field of view are filtered, while projected 2D ground truths supervise the 2D branch.
- Multi-Scale Fusion-to-Single Knowledge Distillation: At each scale, modality fusion first combines transformed 2D and 3D features, after which unidirectional distillation transfers the fused knowledge to the point-cloud features.This fusion-then-distillation design is intended to preserve modality-specific information while improving 3D representations.
- Inference design: The fusion branch is used only during training, so the enhanced model can run with almost no extra computational cost during inference.The 3D decoder upsamples features from multiple scales, concatenates them, and feeds them to the classifier.
4 Experiments
Experiments evaluate 2DPASS on SemanticKITTI and NuScenes, compare knowledge-distillation designs, and test generality across 3D networks. The method improves benchmark performance, including top results under SemanticKITTI scan settings and NuScenes comparisons.
- Qualitative results: 2DPASS recognizes small objects better than the baseline, which has higher errors on small objects and region boundaries.The qualitative comparison is conducted on the SemanticKITTI validation set.
- SemanticKITTI: 2DPASS outperforms (AF)2-S3Net by 2.1% in SemanticKITTI single-scan mIoU while achieving the best reported running speed.The improvement does not add inference burden to the original baseline.
- SemanticKITTI: 5.5% improvement over (AF)2-S3Net is reported for SemanticKITTI multiple-scan mIoU, along with better overall accuracy.The multiple-scan evaluation covers all 25 classes.
- NuScenes: 2DPASS ranks first among published methods on the NuScenes benchmark and outperforms the listed fusion-based approaches using only point clouds at inference.The model ranks third when unpublished works are included.
- Knowledge distillation: Pure knowledge distillation and cross-modal feature alignment provide limited or marginal gains, whereas MSFSKD performs significantly better in the comparison.The comparison evaluates the transfer methods at each scale.
- Generality: 2DPASS improves MinkowskiNet from 63.1 to 66.2 and SPVCNN from 63.8 to 66.9, supporting its model-independent training-scheme claim.The comparison keeps setups unchanged except for the 2D-related components.
5 Conclusion
The conclusion presents 2DPASS as a general training scheme that uses 2D-related knowledge distillation to improve pure 3D LiDAR segmentation. It reports state-of-the-art performance on SemanticKITTI and NuScenes and identifies possible future use in other 3D tasks.
- Conclusion: 2DPASS boosts LiDAR point-cloud semantic segmentation through auxiliary modal fusion and multi-scale knowledge distillation from multimodal data.The resulting information is distilled to a pure 3D network.
- Conclusion: 2DPASS achieves state-of-the-art results on the SemanticKITTI and NuScenes large-scale benchmarks.
- Conclusion: The authors identify 3D detection and tracking as possible future application scenarios for 2DPASS.
A Training and Inference Details
Training uses standard geometric and image augmentations, cropped 2D inputs, and end-to-end SGD optimization. The segmentation and distillation objectives are combined during training.
- Training: 3D inputs use random global scaling from [0.95, 1.05] and random rotation around the Z axis, while 2D inputs use horizontal flipping and color jitter.
- Training: Each 2D image is cropped to 480 × 320 for faster training, and 2DPASS is trained end-to-end with SGD.
B.1 Comparing with Multi-Sensor Architecture
The multi-sensor comparison examines projection, prediction concatenation, feature interaction, and a lightweight 2DPASS variant. 2DPASS (light) provides a favorable accuracy–speed trade-off compared with methods that process camera inputs during inference.
- Compared architectures: PointPainting projects image segmentation logits into LiDAR space, while the multi-branch baseline concatenates output logits from the two modalities.
- 2DPASS (light): 2DPASS (light) reduces hidden dimensions to 64 because full-image multi-sensor methods are time-consuming and memory-intensive.
- Results: Naive prediction combinations do not obviously improve segmentation and introduce substantial computational burden from processing six 1600×900 camera images per point cloud.
- Results: 2DPASS (light) achieves the second-best NuScenes validation mIoU while running 60× faster than multi-sensor methods.
B.2 Concrete Results
This section presents detailed semantic segmentation results on the NuScenes validation set, reported in Table 2 as a benchmark for future work.
- The NuScenes results are presented as a benchmark for future work.
- The section provides detailed results for the NuScenes dataset.
- Table 2 reports semantic segmentation results on the NuScenes valid set.