Source-linked AI summary
Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds
Francis Engelmann, Theodora Kontogianni, Alexander Hermans, Bastian Leibe
TL;DR
Direct semantic segmentation of unstructured 3D point clouds remains difficult, while PointNet processes each block individually and therefore has limited spatial context. This paper extends PointNet with input-level and output-level context mechanisms, improving segmentation results on challenging indoor and outdoor datasets.
Problem
Semantic segmentation of unstructured 3D point clouds remains difficult, and PointNet aggregates context only among points within each block.
Method
The paper extends PointNet with input-level context from multiple scales or adjacent regions and output-level context that exchanges information across larger spatial neighborhoods.
Results
The proposed spatial-context mechanisms improve semantic segmentation quality on challenging indoor and outdoor datasets.
Takeaways & Limitations
Spatial context can be incorporated into the PointNet pipeline through two extension mechanisms, although other design combinations remain possible.
Takeaways & Limitations
PointNet’s aggregated block-feature passes information only among points inside the same block.
Abstract
from arXiv · showhide
Deep learning approaches have made tremendous progress in the field of semantic segmentation over the past few years. However, most current approaches operate in the 2D image space. Direct semantic segmentation of unstructured 3D point clouds is still an open research problem. The recently proposed PointNet architecture presents an interesting step ahead in that it can operate on unstructured point clouds, achieving encouraging segmentation results. However, it subdivides the input points into a grid of blocks and processes each such block individually. In this paper, we investigate the question how such an architecture can be extended to incorporate larger-scale spatial context. We build upon PointNet and propose two extensions that enlarge the receptive field over the 3D scene. We evaluate the proposed strategies on challenging indoor and outdoor datasets and show improved results in both scenarios.
1. Introduction
The paper addresses the difficulty of directly segmenting unstructured 3D point clouds and the limited spatial context of block-wise PointNet processing. It introduces input-level and output-level context mechanisms, which improve semantic segmentation quality on indoor and outdoor datasets.
- Motivation: Direct semantic segmentation of unstructured 3D point clouds remains harder and less established than image-based semantic segmentation.Point clouds may come from LiDAR or stereo sensors, while most state-of-the-art approaches operate on 2D images.
- Motivation: PointNet processes small 3D blocks independently, restricting neighborhood context to points within each block.It computes local point features, aggregates them into a block-level global feature, and predicts per-point class scores.
- Proposed mechanisms: The paper proposes input-level context by jointly processing multiple scales or adjacent regions.This enlarges the spatial neighborhood considered at the network input.
- Proposed mechanisms: The paper proposes output-level context by exchanging information among point descriptors over larger spatial neighborhoods.This consolidates estimated descriptors after the initial point-processing stage.
- Results: Both context mechanisms improve semantic segmentation quality on challenging indoor and outdoor datasets.The contributions are evaluated experimentally in both scenarios.
2. Related Work
Related work spans traditional feature-based methods and deep-learning approaches that regularize, project, track, or graph-structure 3D point-cloud data. PointNet provides the architectural basis for the paper's context extensions.
- Traditional Methods: Traditional approaches use 3D geometric features with classifiers such as random forests, associative Markov networks, and sequential parsing procedures.Examples include covariance-tensor features, multiscale nearest-neighbor computation, sensor fusion, and spatial relationship learning.
- Deep-learning Methods: Deep-learning approaches represent point clouds as volumetric grids, 2D projections, snapshots, tracked sequences, or shape graphs.These representations enable 3D or 2D convolutions, label projection, tracking-based segmentation, or spectral graph processing.
- PointNet: PointNet takes N points from a 3D block, transforms them with per-point MLPs, and uses max pooling to produce a permutation-invariant global feature.The global feature is concatenated with point features before MLP-based prediction of M class scores.
3. Method
The method extends PointNet by incorporating spatial context across multiple blocks, either at the input level, output level, or both, to produce per-point semantic scores.
- PointNet: PointNet splits a point cloud into blocks, maps points into higher-dimensional features, aggregates them with max-pooling, and predicts per-point class scores.The global feature is concatenated with point features before the final MLP stages.
- Context Extension: Because PointNet aggregates information only within each block, the proposed extensions share context across spatially related blocks.The paper targets context outside an individual block as an additional source for class-label prediction.
- Input-Level Context: Input-level context processes multiple blocks jointly, selecting either multiple scales at one position or neighboring cells in a regular grid.Multi-scale blocks use scale-specific descriptors, while grid blocks use one shared single-scale descriptor.
- Consolidation: Consolidation Units transform point features with MLPs, max-pool them into a common block-feature, concatenate that feature back, and can be chained sequentially.This mechanism further consolidates block-features produced by the input-level stage.
- Recurrent Consolidation: Recurrent Consolidation Units use GRUs over spatially nearby block-features to return updated features that incorporate neighboring-block information.The unrolled GRU processes the whole input sequence before returning the updated block-features.
- Multi-Scale Architecture: The exemplary MS architecture concatenates multi-scale block-features with transformed point-features and passes them through chained CUs before per-point scoring.Multi-scale blocks share a center while using multiple radii to select points.
- Grid Architecture: The grid architecture uses four blocks from a 2x2 neighborhood, shared MLP weights, an RCU, and concatenation of updated and original block-features for classification.The resulting features are combined with local features before fully connected layers compute per-point class scores.
4. Experiments
The experiments compare the proposed context mechanisms with PointNet on indoor and outdoor point-cloud datasets, using geometry with or without color and qualitative transfer to real laser scans. Both input-level and consolidation-based context schemes improve segmentation, while cross-domain results remain strongest for common classes.
- Datasets and protocol: The evaluation compares the proposed architectures with PointNet on S3DIS and vKITTI, with additional qualitative tests on real KITTI laser scans.S3DIS uses dense Matterport scans, while vKITTI provides synthetic urban point clouds projected from annotated depth.
- Input features: The experiments distinguish XYZ geometry from XYZ+RGB features when analyzing input-block schemes and consolidation units.The geometry-only setting tests performance without color information.
- Geometry with Color: The G+RCU model improves over PointNet by conveying context among overlapping neighboring blocks.Training selects four 1×1 m blocks from a 2×2 neighborhood, with each block containing 4096 points.
- Geometry with Color: Multi-scale input blocks alone provide limited gains over PointNet, whereas consolidation units outperform the multi-scale input-block feature.The full MS+CU(2) model combines multi-scale blocks with two consolidation units.
- Cross-domain qualitative results: On real Velodyne and 3DRMS laser data, models trained on vKITTI transfer successfully mainly for common classes such as road, buildings, cars, and trees.The training and test datasets can differ in semantic labels, limiting direct class correspondence.
5. Conclusion
The paper extends PointNet with input-level and output-level spatial context mechanisms for 3D semantic segmentation and validates them indoors and outdoors. The reported results improve over PointNet, while the full design space remains unexplored.
- Conclusion: The proposed input-level and output-level context extensions are successfully applied to indoor and outdoor datasets.The extensions are built on the PointNet pipeline.
- Future work: The paper leaves full exploration of the possible context-design combinations for future work.The conclusion notes that numerous other combinations remain possible.
- Experimental findings: With XYZ-RGB features on S3DIS, adding context components improves mean IoU and achieves state-of-the-art results in mean IoU and individual class IoU.The comparison includes different input- and output-level context techniques.
- Experimental findings: The methods outperform PointNet consistently on S3DIS and vKITTI using XYZ input features without color.The reported mean-IoU improvements are more considerable when color is unavailable.
- Qualitative findings: Indoor qualitative results are more consistent and less noisy, while outdoor outputs are less fragmented and better recognize fine structures such as street lights and poles.These qualitative comparisons are made against PointNet and ground-truth labels.