Source-linked AI summary
Point Cloud Oversegmentation with Graph-Structured Deep Metric Learning
Loic Landrieu, Mohamed Boussaha
TL;DR
The paper addresses the challenge of producing semantically pure superpoints from 3D point clouds for downstream semantic segmentation. It learns graph-structured embeddings from local geometry and radiometry, then partitions the point cloud using those embeddings. The approach improves state-of-the-art oversegmentation on indoor and outdoor datasets and also improves superpoint-based semantic segmentation.
Problem
Existing unsupervised oversegmentation methods assume geometrically or radiometrically homogeneous segments are semantically homogeneous, motivating supervised learning of semantically pure superpoints.
Method
The method combines a lightweight Local Point Embedder with graph-structured contrastive loss, cross-partition weighting, and graph partitioning of the learned embeddings.
Results
The approach significantly improves state-of-the-art point-cloud oversegmentation on two different datasets and improves superpoint-based semantic segmentation to a new state-of-the-art.
Takeaways & Limitations
Learned embeddings can provide semantically pure superpoints that improve downstream superpoint-based semantic segmentation.
Takeaways & Limitations
The embedding optimization constrains variables differently for efficiency, which can produce suboptimal approximate partitions; borders without recognizable geometric or radiometric cues cannot be recovered.
Abstract
from arXiv · showhide
We propose a new supervized learning framework for oversegmenting 3D point clouds into superpoints. We cast this problem as learning deep embeddings of the local geometry and radiometry of 3D points, such that the border of objects presents high contrasts. The embeddings are computed using a lightweight neural network operating on the points' local neighborhood. Finally, we formulate point cloud oversegmentation as a graph partition problem with respect to the learned embeddings. This new approach allows us to set a new state-of-the-art in point cloud oversegmentation by a significant margin, on a dense indoor dataset (S3DIS) and a sparse outdoor one (vKITTI). Our best solution requires over five times fewer superpoints to reach similar performance than previously published methods on S3DIS. Furthermore, we show that our framework can be used to improve superpoint-based semantic segmentation algorithms, setting a new state-of-the-art for this task as well.
1. Introduction
The paper introduces a supervised framework that learns point embeddings to produce semantically pure 3D superpoints, targeting improved semantic segmentation. Its graph-structured loss, local point embedder, and partitioning strategy yield strong oversegmentation and downstream segmentation results.
- 1. Introduction: The framework formulates 3D point-cloud oversegmentation as deep metric learning structured by an adjacency graph.It learns embeddings that are homogeneous within objects and contrastive at object interfaces.
- 1. Introduction: The method is presented as the first supervised framework for 3D point-cloud oversegmentation.The paper addresses the limitation of unsupervised methods that assume geometric or radiometric homogeneity implies semantic homogeneity.
- 1. Introduction: The graph-structured contrastive loss and cross-partition weighting strategy produce embeddings with high contrast at object borders.The loss adapts learned embeddings to the nondifferentiable oversegmentation task.
- 1. Introduction: The Local Point Embedder is a lightweight architecture that compactly encodes each point’s local geometry and radiometry.The embedder is inspired by PointNet and operates on local point neighborhoods.
- 1. Introduction: The approach significantly improves state-of-the-art point-cloud oversegmentation on two substantially different datasets.The evaluation covers dense indoor and sparse outdoor data, while the framework is also intended to provide semantically pure superpoints for semantic segmentation.
- 1. Introduction: The learned embeddings support clustering-based oversegmentation that closely follows ground-truth boundaries in a difficult white-board-on-white-wall scene.The figure shows the progression from colored input points to embeddings, oversegmentation, and ground truth.
2. Related work
Prior work covers graph- and cluster-based oversegmentation for images and several handcrafted approaches for 3D point clouds. The paper positions its contribution as the first supervised 3D point-cloud oversegmentation method using deep-learning embeddings.
- 2. Related work: Image and video oversegmentation methods include graph-based and cluster-based approaches, with recent supervised deep-learning variants.These methods exploit regular pixel structure that does not directly carry over to point clouds.
- 2. Related work: 3D point-cloud oversegmentation requires special treatment because point clouds are unordered point sets with irregular distributions.Prior 3D methods vary in graph construction, edge weighting, subgraph merging, and use of LiDAR structure.
- 2. Related work: Deep learning for 3D point clouds had been applied to semantic segmentation, detection, and reconstruction, but not supervised oversegmentation using learned embeddings.The paper identifies this absence as the relevant gap in prior work.
- 2. Related work: Metric learning learns task-oriented similarity functions through embedding vectors and losses such as contrastive or triplet loss.The paper adapts this general paradigm to graph-structured 3D point-cloud oversegmentation.
3. Method
The method learns compact local embeddings of 3D point geometry and radiometry, then converts them into superpoints through graph-based partitioning. Its training objective emphasizes object borders while its weighting strategy accounts for partition purity.
- The framework targets object-pure, border-aligned, and regular superpoints for downstream point-cloud analysis.These properties are defined as object-purity, border recall, and regularity.
- A lightweight local cloud embedder encodes each point’s local geometry and radiometry into a compact embedding.The LPE operates on a point and its local neighborhood rather than the whole cloud.
- The spatial transform normalizes neighboring coordinates while retaining neighborhood radius, height, and orientation information.It uses a learned rotation around the z axis and tracks the normalization operations.
- Superpoints are obtained by approximating the embeddings as piecewise constant on an adjacency graph and extracting constant connected components.The graph may derive from the point cloud, while smaller neighborhoods can suffice for adjacency than for local embedding description.
- The graph-structured contrastive loss promotes similar embeddings within objects and high contrasts across object interfaces.Its intra-edge and inter-edge terms respectively encourage homogeneity and separation.
- Cross-partition weighting incorporates the consequences of missed borders and distributes interface penalties across their constituent edges.The strategy uses a cross-segmentation graph between the superpoint and object partitions.
4. Numerical Experiments
The experiments evaluate oversegmentation and semantic segmentation on dense indoor S3DIS and sparse outdoor vKITTI data. SSP outperforms competing approaches across oversegmentation metrics, while improving superpoint-based semantic segmentation.
- Datasets: S3DIS provides dense indoor scans with object annotations, whereas vKITTI contains sparse outdoor urban scenes whose objects are defined by connected semantic-label components.
- Metrics: Boundary Recall and Boundary Precision evaluate whether predicted transitions align with object borders, using a one-edge tolerance.
- Metrics: Oracle Overall Accuracy measures the accuracy obtained by assigning each superpoint its majority ground-truth label, providing an upper bound for superpoint-based classification.
- Oversegmentation results: SSP significantly outperforms other approaches on all metrics and needs under 350 superpoints to match VCCS with over 1,800 superpoints on S3DIS.
- Oversegmentation results: The improvement is less significant on vKITTI, potentially because sparse acquisitions make adjacency-graph construction difficult; removing color further degrades performance.
- Efficiency: Embedding computation exceeds 3 million embeddings per second on a 1080Ti GPU, while graph partitioning processes around 100,000 points per second.
- Semantic segmentation: Replacing unsupervised superpoint computation with SSP significantly improves SPG semantic segmentation and beats concurrent methods on both datasets.
5. Conclusion
The paper presents a supervised 3D point-cloud oversegmentation framework using point embeddings and a graph-structured loss. It reports improvements over state-of-the-art oversegmentation and semantic segmentation methods.
- The paper presents the first supervised framework for 3D point-cloud oversegmentation.
- A lightweight point-embedding network and graph-structured loss produce significant improvements over the state of the art in point-cloud oversegmentation.
- Combining the method with superpoint-based semantic segmentation sets a new state of the art for semantic segmentation.
Supplementary Material
The supplementary material describes the full hyper-parameterization of networks used for oversegmentation and semantic segmentation on both datasets.
- The supplementary material specifies network hyper-parameters for both oversegmentation and semantic segmentation tasks across both datasets.
A.1. Models configuration for oversegmentation
The model configuration uses lightweight networks, local neighborhoods, graph-specific weighting, regularization paths, and data augmentation to configure oversegmentation experiments.
- Local neighborhoods and graphs: A local neighborhood size of 20 is sufficient for embeddings to detect object borders while keeping memory usage low.
- Network configuration: The LPE and spatial-transform networks use shallow, wide architectures, allowing over 250,000 points to be embedded simultaneously with gradients on 11GB of RAM.
- Loss weighting: The graph-structured loss scales inter-edge weight by average graph connectivity, with µ = ˜µc.
- Regularization: For regularization strength ˜λ ≤ 1/(2c), the partition should place borders at edges whose endpoint embeddings differ by at least 1.
- Regularization: Regularization paths vary ˜λ from 0.2 to 6 after training at ˜λ = 1, without fine-tuning.
- Superpoint size: The minimum superpoint size is adjusted with regularization strength; when n_min^(1) = 50, it is 33 at ˜λ = 0.2 and 70 at ˜λ = 6.
- Optimization: Training uses Adam with gradient clipping at 1 and takes around 2 hours per fold on an 11GB VRAM 1080Ti GPU.
- Mini-batches: Clustering-based segmentation uses batch size 1 and 100,000-point subgraphs, while graph-based clustering uses batches of 16 and 10,000-point subgraphs.
A.2. Models configuration for semantic segmentation
The semantic segmentation configuration adapts the superpoint-graph pipeline to the learned oversegmentation, using dataset-specific settings to improve boundary handling and semantic propagation.
- Configuration: The implementation changes only the oversegmentation step and selected hyper-parameters relative to the open-source superpoint-graph framework.The complete parameterization is reported in Table 5.
- Configuration: Lowering the regularization strength improves accuracy and border recall, while semantic context propagation compensates for reduced border precision.This adjustment addresses edges missed by the ℓ0-cut pursuit approximation and its treatment of spherical embeddings.
- Reported settings and results: Tables 4 and 5 specify the embedding and semantic segmentation network configurations, while Table 6 reports per-class S3DIS IoU results.Table 6 covers Area 5 and micro-averaged results across all six folds.
- Configuration: The S3DIS configuration uses a lower superpoint size based on the segmentation experiments.
- Configuration: SSP uses four-hop graph subsampling because it tends to produce thin components near interfaces, while vKITTI uses smaller networks to mitigate overfitting.The smaller networks reflect vKITTI’s smaller dataset size.
B. Residual Point Embedder
The Residual Point Embedder builds embeddings hierarchically: initial layers use local geometry and radiometry, while later layers incorporate neighboring embeddings to expand context and correct prior errors.
- Residual Point Embedder: The Residual Point Embedder modifies the local point embedder by adding a supplementary input and computing a residual added to the initial embedding before normalization.
- Layered architecture: First-layer RPEs compute embeddings from local geometric and radiometric information with zero initial embeddings, matching the behavior of local point embedders.
- Layered architecture: Subsequent RPE layers combine local geometry and radiometry with embeddings from neighboring points computed at the previous layer.A point is processed only after all its neighbors have been embedded by the preceding layer.
- Layered architecture: The layered design provides increasingly broader receptive fields and can correct errors introduced by earlier layers.
- Layered architecture: Geometric information is processed by the spatial transform once and then cascaded through the residual layers.
- Alternative configuration: Setting all initial embeddings to zero makes each layer compute a new embedding from local position and previous-layer embeddings, but its benefits shrink when a simple LPE has many parameters.
C. Detailed results and illustration
The S3DIS evaluation reports per-class IoU results and visualizes semantic segmentation outcomes, including both successful scenes and a failure caused by excessive oversegmentation.
- Qualitative results: Figure 8 illustrates semantic segmentation results for S3DIS scenes, including successful semantization and a whiteboard failure case.The failure produces many small superpoints, making classification harder for the semantic segmentation network.
- S3DIS classes: The S3DIS class breakdown includes ceiling, floor, wall, column, beam, window, door, table, chair, bookcase, sofa, board, clutter, and unlabelled.
vKITTI
The vKITTI evaluation covers outdoor semantic classes and includes a successful urban-scene visualization alongside a background case involving road signs.
- vKITTI classes: The vKITTI class breakdown includes terrain, tree, vegetation, building, road, guard rail, traffic sign, traffic light, pole, misc, truck, car, van, and unlabelled.
- Qualitative results: Figure 8 shows a successful semantic segmentation of an urban outdoor vKITTI scene and road signs visible in the background.