Source-linked AI summary
Deep Projective 3D Semantic Segmentation
Felix Järemo Lawin, Martin Danelljan, Patrik Tosteberg, Goutam Bhat, Fahad Shahbaz Khan, Michael Felsberg
TL;DR
Semantic segmentation of 3D point clouds is constrained by voxelization costs, reduced resolution, and scarce annotated 3D data. The paper projects point clouds into synthetic 2D images for CNN segmentation, re-projects scores to points, and fuses multiple modalities. On Semantic3D, it reports a new state of the art, with a relative gain of 7.9% over the previous best approach.
Problem
3D-CNN point-cloud segmentation is limited by voxelization-induced resolution and memory costs and by scarce annotated 3D data.
Method
The framework projects point clouds into synthetic 2D images, segments them with a 2D-CNN, re-projects prediction scores, and fuses color, depth, and surface-normal modalities.
Results
7.9% relative gain over the previous best approach establishes a new state of the art on Semantic3D.
Takeaways & Limitations
Projected 2D representations let the approach exploit image-segmentation techniques and annotated image datasets while avoiding voxelization.
Takeaways & Limitations
The rendering uses a simple camera-view heuristic and the segmentation network is trained on limited data.
Abstract
from arXiv · showhide
Semantic segmentation of 3D point clouds is a challenging problem with numerous real-world applications. While deep learning has revolutionized the field of image semantic segmentation, its impact on point cloud data has been limited so far. Recent attempts, based on 3D deep learning approaches (3D-CNNs), have achieved below-expected results. Such methods require voxelizations of the underlying point cloud data, leading to decreased spatial resolution and increased memory consumption. Additionally, 3D-CNNs greatly suffer from the limited availability of annotated datasets. In this paper, we propose an alternative framework that avoids the limitations of 3D-CNNs. Instead of directly solving the problem in 3D, we first project the point cloud onto a set of synthetic 2D-images. These images are then used as input to a 2D-CNN, designed for semantic segmentation. Finally, the obtained prediction scores are re-projected to the point cloud to obtain the segmentation results. We further investigate the impact of multiple modalities, such as color, depth and surface normals, in a multi-stream network architecture. Experiments are performed on the recent Semantic3D dataset. Our approach sets a new state-of-the-art by achieving a relative gain of 7.9 %, compared to the previous best approach.
1 Introduction
The paper addresses the challenges of semantic segmentation in sparse, unstructured 3D point clouds by transferring the strengths of 2D image segmentation to projected point-cloud views. It evaluates multi-modal fusion and reports state-of-the-art performance on Semantic3D.
- 3D point-cloud segmentation remains less developed than image segmentation despite its importance for applications including scene understanding and robotics.
- Voxelization for 3D-CNNs increases memory consumption and decreases spatial resolution, while annotated 3D data remains scarce.
- The proposed framework projects point clouds into synthetic images, applies a 2D-CNN, and re-projects pixel-wise scores to obtain point labels.
- Color, depth, and surface-normal modalities are fused in a multi-stream architecture to produce final prediction scores.
- The approach leverages existing image datasets, avoids voxelization, and thereby supports higher spatial resolution with reduced dependence on annotated 3D data.
- On Semantic3D, modality fusion significantly improves segmentation performance and the method establishes a new state of the art over classical and 3D-CNN approaches.
2 Related Work
Prior 3D segmentation methods commonly combine hand-crafted features, classifiers, and spatial models, while CNN-based approaches face structural, computational, contextual, and annotation constraints. The paper instead uses dense projected 2D representations to exploit mature image-segmentation methods and annotated image data.
- Most earlier 3D segmentation methods combine hand-crafted features, discriminative classifiers, and spatial smoothness models.
- Common handcrafted representations include structure-tensor, histogram-based, Spin Images, SHOT, and color features.
- Earlier classifiers often use maximum-margin methods or random forests, while spatial correlations are modeled with graphical models such as CRFs.
- Deep CNNs have advanced image segmentation by learning high-level features and benefiting from large datasets and GPU computing.
- Applying CNNs to point clouds is hindered by sparse, scattered structure, voxelization-related resolution and memory costs, limited context, and scarce annotations.
- The proposed approach projects point clouds into dense 2D images, eliminating voxelization and enabling efficient 2D convolutions with established image methods and annotated data.
3 Method
The method renders unstructured point clouds into multiple synthetic 2D views, segments those views with a multi-stream CNN, and fuses re-projected scores into point-wise 3D labels. Its rendering and fusion design preserves visibility information, combines complementary modalities, and avoids costly 3D preprocessing.
- 3.1 Render views: The point cloud is projected into synthetic views containing color, depth, and surface-normal images, which are processed for image-based semantic segmentation.The approach can also project other point-cloud features, while tracking point visibility for mapping predictions back to 3D.
- 3.1 Render views: Point splatting distributes each 3D point into virtual-camera image coordinates with a Gaussian spread function, enabling image-space rendering.The projected points retain depths and feature vectors such as RGB color and surface normals; splatting avoids demanding 3D surface-reconstruction preprocessing and supports parallel implementations.
- 3.1 Render views: Mean-shift clustering of projected depths determines visible surface clusters while suppressing foreground noise and occluded points during rendering.Clusters are ranked using kernel-density estimates and a camera-proximity reward; the selected cluster supplies each pixel’s depth, and features are proximity-weighted.
- 3.2 Deep Multi Stream Image Segmentation: A multi-stream architecture processes color, depth, and surface-normal inputs with separate Fully Convolutional Networks and sums their prediction scores.The streams are trained end-to-end, with VGG16-based FCNs initialized from ImageNet; the pipeline remains agnostic to the particular image-segmentation architecture.
- 3.3 Mapping Back to 3D: Scores from every rendered view are mapped to visible 3D points, summed across projections, and converted to labels using the largest aggregate score.Stored contributing-point indices make the re-projection from CNN outputs to the original point cloud direct.
4 Experiments
Experiments use Semantic3D and compare network configurations across color, depth, and surface-normal modalities. The multi-stream RGB+D+N model improves performance over prior methods, while the authors note scope for better view generation and segmentation components.
- Experimental setup: Each point cloud is rendered from 120 camera views generated across four pitch angles and camera translations.Each scan uses 30 equally spaced views over a 360° rotation.
- Experimental setup: The study evaluates single-stream, RGB+D+N, and multi-stream configurations without color to assess modality contributions and missing-color applicability.All network configurations are listed in Table 1 and trained using the same parameters.
- Evaluation: The benchmark reports per-class intersection over union and overall accuracy using confusion-matrix assignments against ground-truth labels.Table 2 covers eight semantic categories in the reduced Semantic3D test set.
- Results: The three streams provide complementary information, and their fusion significantly improves final segmentation performance over single-stream alternatives.The best multi-stream approach also significantly improves over the previous state-of-the-art method.
- Limitations: The authors use a simple camera-view heuristic and a basic segmentation network trained on limited data, leaving better alternatives outside the paper’s scope.They report promising results despite these choices and suggest replacements could improve performance further.
5 Conclusion
The paper proposes projecting 3D point clouds into synthetic 2D images for CNN-based segmentation, then reprojecting predictions to points. On Semantic3D, the approach outperforms existing methods and establishes a new state of the art.
- 5 Conclusion: The proposed framework projects point clouds into synthetic 2D images, applies a 2D-CNN, and reprojects prediction scores to obtain point labels.It also investigates multiple modalities in a multi-stream architecture.
- 5 Conclusion: Experiments on Semantic3D show that the approach outperforms existing methods and sets a new state of the art on the dataset.The conclusion reports the dataset-level outcome without specifying a numerical gain.