Source-linked AI summary
Know What Your Neighbors Do: 3D Semantic Segmentation of Point Clouds
Francis Engelmann, Theodora Kontogianni, Jonas Schult, Bastian Leibe
TL;DR
Unordered, sparse point clouds with variable density are difficult to process directly with traditional convolutions. This paper groups points using world-space and learned-feature-space neighborhoods and adds losses to structure features, reporting state-of-the-art results on indoor and outdoor datasets.
Problem
Unordered, sparse point clouds with variable density and very large point counts are difficult to process directly with traditional convolutional neural networks.
Source: §1 · p. 2Method
The approach combines world-space and learned-feature-space neighborhoods with pairwise-distance and centroid losses to structure point features.
Source: §1 · p. 2, §3 · p. 4, §3.3 · p. 6Results
The method reports state-of-the-art results on current indoor and outdoor datasets.
Source: §1 · p. 2, §6 · p. 11Takeaways & Limitations
The framework incorporates neighborhood information from both feature space and world space for 3D point-cloud semantic segmentation.
Source: §6 · p. 11The pairwise similarity loss assumes points of the same semantic class are likely to be nearby in feature space.
Source: Pairwise Similarity Loss Lpair · p. 6
Abstract
from arXiv · showhide
In this paper, we present a deep learning architecture which addresses the problem of 3D semantic segmentation of unstructured point clouds. Compared to previous work, we introduce grouping techniques which define point neighborhoods in the initial world space and the learned feature space. Neighborhoods are important as they allow to compute local or global point features depending on the spatial extend of the neighborhood. Additionally, we incorporate dedicated loss functions to further structure the learned point feature space: the pairwise distance loss and the centroid loss. We show how to apply these mechanisms to the task of 3D semantic segmentation of point clouds and report state-of-the-art performance on indoor and outdoor datasets.
1 Introduction
Unordered, sparse point clouds with variable density and very large point counts are difficult to process with traditional convolutions. The paper introduces adaptive neighborhoods and feature-shaping losses for point-cloud segmentation, evaluated on indoor and outdoor datasets.
1 Introduction
- Unordered, sparse point clouds with sensor-dependent density and very large point counts make direct processing with traditional CNNs difficult.Point clouds can contain multiple orders of magnitude more points than high-resolution images.Source: §1 · p. 2
- Voxel-based 3D convolutions incur cubic memory and computation growth, while coarse grids introduce discretization artifacts and lose geometric information such as point density.Direct point-cloud methods show promise, but prior neighborhoods are global or statically defined coarse-to-fine.Source: §1 · p. 2
- Adaptive neighborhoods use world-space K-means clustering and learned-feature-space k-nearest neighbors to group points.The world-space grouping is primarily sensitive to local geometry, while feature-space neighborhoods are dynamic.Source: §1 · p. 2
- Two dedicated losses further structure the learned point feature space.The losses are the pairwise distance loss and centroid loss.Source: §1 · p. 2
- The approach is evaluated on Stanford 3D Indoor Scene, ScanNet, and Virtual KITTI 3D for 3D semantic segmentation.The evaluation includes two indoor datasets and one outdoor dataset, with an ablation study of the introduced mechanisms.Source: §1 · p. 2
2 Related Work
Prior 3D semantic segmentation methods either convert point clouds to dense representations or operate directly on points. Direct methods introduce pointwise and hierarchical approaches, while the paper’s feature network draws on earlier learned-feature architectures.
2 Related Work
- Related work divides deep-learning approaches into methods that impose structure through voxelization or projection and methods that operate directly on point clouds.Source: §2 · p. 3
Voxelized Methods
- Voxelization and projection convert sparse point clouds to dense representations, increasing memory use and causing discretization artifacts and information loss.Voxelization enables 3D convolutions, while projection maps points into 2D images for 2D convolutions.Source: Voxelized Methods · p. 3
Point Cloud Methods
- PointNet processes points individually with MLPs, max-pools global features, then fuses point and global representations for label prediction.Later direct-point methods include spatial partitions, hierarchical grouping, generalized neighborhood convolutions, and superpoint graphs.Source: Point Cloud Methods · p. 3
- Compared with PointNet++ [22], which uses static metric-distance neighborhoods, the described approach iteratively updates points using learned feature-space kNN neighborhoods.Each iteration passes information one hop through the neighbors, allowing it to flow across more points when repeated.Source: Point Cloud Methods · pp. 3–4
Feature Networks
- The feature network learns high-dimensional features through repeated feature blocks; the caption details block 2 and says the other blocks are equal.The caption points to Sec. 3.1 for architecture details and Fig. 5 for the complete model.Source: Figure 2 · p. 4
- As a first step, a dedicated feature network learns strong features, reflecting the prominence of initial feature extraction; prior work uses image-space CNN features for 2.5 semantic segmentation and VoxelNet’s PointNet-like cascade for detection.Source: Feature Networks · p. 4
3 Our Approach
The approach learns point features using a feature network, defines neighborhoods in learned feature and input world spaces, and uses regional descriptors to inform point features. Pairwise and centroid losses further structure the learned feature space.
3 Our Approach
- The network combines a feature network, two neighborhood types, regional descriptors, and two dedicated losses to learn point features and structure their feature space.The neighborhoods are defined in learned feature space and input world space.Source: §3 · p. 4
3.1 Feature Network
- Each feature block applies MLPs and global max-pooling, then concatenates the repeated global feature with point features; blocks can be stacked to arbitrary depth.The feature network uses pathway connections across blocks, preferring addition over concatenation for scalability and using 17 blocks in the final network.Source: §3.1 · p. 5
3.2 Neighborhoods
- The method defines feature-space neighborhoods with kNN and world-space neighborhoods with K-means clustering on input point features such as position and color.These groupings are used to update point features and learn regional descriptors.Source: §3.2 · p. 5
Feature space neighborhood NF
- An NF-module finds each point’s k nearest feature-space neighbors, learns a neighborhood representation with an MLP, and max-pools it into an updated feature.Concatenating NF-modules expands the receptive field one hop at a time.Source: Feature space neighborhood NF · pp. 5–6, Figure 3 · p. 6
World space neighborhood NW
- The NW-module uses world-space K-means clusters with variable point counts, averages each cluster’s features, concatenates that average to its members, and max-pools a regional descriptor.World-space clustering also acts as pooling that reduces the input space and captures long-range dependencies.Source: World space neighborhood NW · p. 6
3.3 Loss Functions
- Training minimizes the sum of cross-entropy classification loss, pairwise distance loss, and centroid loss.The additional losses are intended to shape the feature space.Source: §3.3 · p. 6
Pairwise Similarity Loss Lpair
- The pairwise loss brings same-class point features within a near-distance threshold and pushes different-class features beyond a far-distance threshold.It uses pairwise distances already computed in the NF-module, with almost no additional memory cost.Source: Pairwise Similarity Loss Lpair · pp. 6–7
Centroid Loss Lcent
- The centroid loss makes features more compact by minimizing the distance between each point feature and its representative class centroid.The centroid can be computed during training as the mean feature of points from the same semantic class; cosine distance was more effective than L1 or L2 distance.Source: Centroid Loss Lcent · pp. 7–8, Figure 4 · p. 7
4 Implementation Details
The implementation builds on a Feature Network, then applies three NF-modules and a world-space clustering module before producing class predictions. Pairwise loss is applied early, while centroid loss is applied after feature concatenation and an MLP.
4 Implementation Details
- Three NF-modules precede a world-space clustering module, and regional descriptors are concatenated with features from the second and third NF-modules before an MLP.The final features are reduced to 13 dimensions, one for each semantic class in the datasets.Source: §4 · p. 8
- The pairwise distance loss is applied at the second NF-module, where the authors found it most effective among tested locations.The authors say this provides early layers a stronger learning signal and simplifies gradient propagation because it passes through fewer layers.Source: §4 · p. 8
- The centroid loss is computed after the regional descriptors are concatenated with point features and passed through another MLP.Source: §4 · p. 8
5 Evaluation
The evaluation covers indoor S3DIS and ScanNet data and outdoor vKITTI3D, using overall accuracy, mean class accuracy, and mean IoU. The authors report state-of-the-art results on the more difficult S3DIS six-fold cross-validation, with slightly inferior results on Area 5.
5.1 Indoor Datasets
- Evaluation spans S3DIS and ScanNet indoors and vKITTI3D outdoors, reporting overall accuracy, mean class accuracy, and mean intersection-over-union.S3DIS contains six areas and 272 rooms with 13 classes; ScanNet has 1,523 scans with 20 classes, while vKITTI3D has six sequences with 13 classes.Source: §5 · p. 8, §5.1 · p. 9, §5.2 · p. 9
5 Evaluation
- S3DIS results are state of the art on the more difficult six-fold cross-validation, but slightly inferior on Area 5.The caption identifies Table 2 as the six-fold cross-validation results.Source: Table 2 · p. 9
5.3 Training Details
- On S3DIS, the method outperforms the current state of the art on average, especially for bookcase and board, but is slightly worse on beam and sofa.Table 4 reports per-class IoU comparisons.Source: Table 4 · p. 10
- For S3DIS, rooms are divided into 1 m2 ground-plane blocks, with 4,096 sampled points per block, kNN k = 30 using L1 distance, and K = ⌊N/52⌋ for K-means.The evaluation uses six-fold cross-validation across areas and also reports Area 5 scores.Source: §5.3 · pp. 9–10
- ScanNet training and evaluation use the PointNet++ reference implementation with unchanged hyperparameters; deeper feature networks generally perform better, and addition is slightly stronger and more memory-efficient than concatenation.Table 5 reports overall point accuracy, mean semantic class accuracy, and mean IoU.Source: §5.3 · p. 10, Table 6 · p. 11, Table 5 · p. 11
5.2 Outdoor Dataset
- For vKITTI3D, scenes are divided into 9 m2 blocks with 256 sampled points, and the evaluation also tests geometric-only inputs without color.The study reports quantitative results in Table 7 and qualitative results in Figure 7.Source: §5.3 · pp. 10–11
6 Conclusion
The framework combines NF- and NW-modules to use neighborhoods in feature and world spaces, alongside pairwise distance and centroid losses. The authors report state-of-the-art results on current indoor and outdoor datasets.
6 Conclusion
- The framework uses NF- and NW-modules for feature- and world-space neighborhoods, plus pairwise distance and centroid losses, and produces state-of-the-art results on current indoor and outdoor datasets.The losses are introduced in the context of 3D point-cloud semantic segmentation.Source: §6 · p. 11