PaperlayerView exact revision on arXiv

Source-linked AI summary

SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation

Weiyue Wang, Ronald Yu, Qiangui Huang, Ulrich Neumann

arXiv:1711.08588v2cs.CV

TL;DR

3D instance segmentation on point clouds remains less developed than 2D approaches, motivating a framework that can represent and group variable numbers of instances. SGPN uses one network with a learned point-pair similarity matrix, confidence map, and semantic map to generate classified grouping proposals. Experiments report effective 3D instance segmentation across scenes while also evaluating detection, semantic segmentation, and RGBD-feature integration.

  • Problem

    3D instance segmentation lags behind 2D methods, while volumetric approaches face high memory and computation costs on 3D scenes.

  • Method

    SGPN learns point embeddings and pairwise similarities, then uses similarity, confidence, and semantic branches to generate and classify instance group proposals.

  • Results

    Experiments show effective 3D instance segmentation on varied scenes and evaluate benefits for 3D object detection, semantic segmentation, and RGBD CNN-feature integration.

  • Takeaways & Limitations

    A similarity-matrix representation provides an intuitive way to define point-cloud instances through relationships between points and supports multiple 3D understanding tasks.

  • Takeaways & Limitations

    The similarity matrix scales quadratically with the number of points, preventing SGPN from processing extremely large scenes on the order of 10^5 or more points.

Abstract

from arXiv · show

We introduce Similarity Group Proposal Network (SGPN), a simple and intuitive deep learning framework for 3D object instance segmentation on point clouds. SGPN uses a single network to predict point grouping proposals and a corresponding semantic class for each proposal, from which we can directly extract instance segmentation results. Important to the effectiveness of SGPN is its novel representation of 3D instance segmentation results in the form of a similarity matrix that indicates the similarity between each pair of points in embedded feature space, thus producing an accurate grouping proposal for each point. To the best of our knowledge, SGPN is the first framework to learn 3D instance-aware semantic segmentation on point clouds. Experimental results on various 3D scenes show the effectiveness of our method on 3D instance segmentation, and we also evaluate the capability of SGPN to improve 3D object detection and semantic segmentation results. We also demonstrate its flexibility by seamlessly incorporating 2D CNN features into the framework to boost performance.

1. Introduction

SGPN addresses the gap in 3D instance-aware semantic segmentation by representing point relationships with a learned similarity matrix and generating grouped, classified instance proposals from one network. Experiments cover varied 3D scenes and related detection and semantic-segmentation tasks, including integration of RGBD CNN features.

  • 3D instance segmentation has lagged behind 2D methods, while volumetric CNNs face high memory and computation costs for scene understanding.
  • SGPN learns a similarity matrix that represents relationships between point pairs and forms object-instance group proposals.Point features are extracted with PointNet/PointNet++, and pairwise feature distances indicate whether points belong to the same instance.
  • Rows of the similarity matrix serve as instance candidates, while confidence scores produce plausible proposals and semantic segmentation assigns each instance a class.The same outputs also support deriving tight 3D bounding boxes for object detection.
  • SGPN uses three output branches: a similarity matrix for point-wise group proposals, a confidence map for pruning, and a semantic map for class labels.
  • Experiments on ShapeNet, Stanford Indoor Semantic Dataset, and NYUV2 report state-of-the-art 3D instance segmentation and evaluate 3D detection and semantic segmentation.The framework also incorporates RGBD-image CNN features to boost performance.

2. Related Works

Related work spans successful 2D proposal-based detection and instance segmentation, emerging 3D detection systems, volumetric and octree CNNs, point-cloud networks, and similarity metric learning. The paper positions SGPN against limited prior work on learned 3D instance segmentation.

  • 2D detection and instance segmentation commonly generate region or segment proposals, including RPN-based and grid-based systems.
  • Prior 3D detection methods use volumetric CNN proposals, RGBD appearance features, or sliding-shape CRF models.
  • The related-work discussion states that no previous work had learned 3D instance segmentation.
  • Volumetric CNNs support several 3D tasks but incur high memory and computational costs and have limitations on 3D scenes.Octree-based CNNs address volumetric processing but are described as less flexible.
  • PointNet and PointNet++ provide memory-efficient, flexible point-cloud representations for classification, part segmentation, and semantic scene segmentation.SGPN bases its network architecture on these point-cloud methods.
  • Similarity metric learning has been applied to re-identification, matching, retrieval, face recognition, tracking, one-shot learning, and associative pixel grouping.

3. Method

SGPN learns point relationships in an embedded feature space to produce instance grouping proposals, semantic labels, and directly derived 3D detections. Its similarity-matrix representation is designed for shape-centric point clouds and is processed with confidence scoring and proposal merging.

  • Framework: SGPN maps each input point cloud to point-wise instance labels and class labels for each instance.The network outputs instance proposals containing grouped points and their class labels.
  • Framework: The network extracts point features, then branches into similarity, confidence, and semantic segmentation outputs with a summed training loss.The three branches produce similarity, confidence, and semantic maps from shared extracted features.
  • Similarity representation: Its Np × Np similarity matrix classifies whether point pairs belong to the same instance, with each row representing a candidate grouping.Pairwise feature distances define the matrix, making instance segmentation an embedding-space learning problem.
  • Similarity representation: The representation matches point clouds’ shape-centric structure by defining objects through relationships between points rather than grid positions or image patches.This contrasts with 2D methods that localize patches and predict binary masks within them.
  • Similarity representation: A double-hinge loss separates same-instance pairs, same-class different-instance pairs, and different-class pairs by progressively increasing feature-space distances.The loss avoids precisely regressing feature values and instead enforces relative separation among similarity classes.
  • Proposal processing: Confidence scores filter grouping candidates, semantic point predictions assign group classes, and non-maximum suppression merges overlapping proposals before point assignment.Approximately 2% of points belong to multiple final proposals after merging and are randomly assigned among those proposals.

4. Experiments

SGPN is evaluated on synthetic shapes and real 3D scenes for instance segmentation, semantic segmentation, and 3D detection. Results show strong qualitative and quantitative performance, efficient inference, benefits from similarity-based grouping, and further gains from integrating 2D CNN features.

  • Datasets: SGPN is evaluated on S3DIS, NYUV2, and ShapeNet Part Segmentation, covering real scans, partial RGBD point clouds, and annotated object parts.S3DIS contains 271 rooms across six areas; NYUV2 uses 795 training and 654 testing images; ShapeNet contains 16,881 shapes.
  • S3DIS instance segmentation: On S3DIS, SGPN separates nearby objects sharing the same semantic class more effectively than Seg-Cluster.Seg-Cluster struggles with small same-label objects that are close together, whereas SGPN separates difficult same-class instances in feature space.
  • S3DIS 3D object detection: SGPN derives 3D object detections from its instance segmentation results and outperforms PointNet by a large margin without requiring an additional category-specific network.Detection is evaluated over four categories using AP at IoU threshold 0.5.
  • S3DIS semantic segmentation: SGPN outperforms its PointNet baseline on S3DIS semantic segmentation, with the similarity matrix contributing to the improvement.The comparison uses mean IoU over 13 classes, including clutter.
  • NYUV2: On NYUV2, SGPN's improvement over Seg-Cluster is smaller because same-label objects are usually far apart, while partial scans also make naive separation easier.The method can still use CNN features, combining them with 3D features through geometric relationships.
  • Efficiency and CNN integration: SGPN improves efficiency relative to cited systems, while SGPN-CNN incorporates 2D CNN features and jointly trains them with the point-cloud network.SGPN takes 170 ms and about 400M GPU memory per sample; SGPN-CNN takes 300 ms and 1.4G GPU memory per sample.
  • ShapeNet Part Segmentation: On ShapeNet Part Segmentation, SGPN produces qualitatively strong instance grouping, including separating chair legs into four instances under challenging conditions.Only qualitative results are provided because the generated ground truths are not real ground truths.

5. Conclusion

SGPN is presented as an intuitive, simple, and flexible point-cloud framework whose similarity-matrix representation generates group proposals and supports related tasks. Its main scalability limitation is quadratic similarity-matrix growth, preventing processing of extremely large scenes.

  • SGPN uses a similarity matrix to generate group proposals with class predictions from a single network.The framework is designed for 3D instance segmentation on point clouds.
  • Experiments report good instance-segmentation performance across various 3D scenes and support for 3D detection and semantic segmentation.
  • The similarity matrix scales quadratically with the number of points, limiting SGPN on extremely large scenes of roughly 10^5 or more points.The paper notes that SGPN remains more memory-efficient than volumetric methods despite this limitation.

A. Network Architecture

The architecture uses PointNet or PointNet++ as baseline feature extractors, with task-specific layers and dataset-dependent configurations. The NYUV2 configuration follows PointNet++ set-abstraction and feature-propagation notation.

  • S3DIS uses PointNet as its baseline, while ShapeNet part uses PointNet++ with the architecture reported in the PointNet++ paper.
  • The S3DIS network retains the PointNet semantic-segmentation architecture except for its last two layers.Its final feature layer has 256 output channels, and the task-specific 1×1 convolution layers have 128 channels.
  • NYUV2 uses PointNet++ as its baseline and describes the network through set-abstraction and feature-propagation levels.Set-abstraction levels specify local-region counts, radii, and convolutional output channels; feature-propagation levels specify convolutional channels.
  • The NYUV2 architecture uses set-abstraction levels with 1024, 256, 128, 64, and 16 local regions and radii from 0.1 to 1.2.
  • The reported task-specific layers use 1×1 convolutions with 128 output channels for S3DIS and 64 output channels for ShapeNet part.

B.1. S3DIS Dataset

S3DIS scenes are processed as overlapping blocks and merged through a voxel-based procedure that reconciles point-instance labels across blocks. Wall-associated objects remain difficult because they contain little depth information.

  • Block Merging: Scenes are divided into overlapping 1m × 1m blocks using sliding windows with 0.5m stride in a snake pattern.The procedure is illustrated in Figure 9.
  • Block Merging: BlockMerging combines block point-instance labels with a 400 × 400 × 400 voxel grid to produce whole-scene point instance labels.The grid stores instance labels for cells, which are used to merge object instances across blocks.
  • Results: ScanNet results report 0 percent AP for wall-associated items such as doors, windows, and pictures.The text attributes this to limited depth information and semantic mislabeling as wall.

C.1. ScanNet

ScanNet experiments use overlapping-block processing with uniformly sampled training points and evaluate instance segmentation using AP at IoU threshold 0.25. Qualitative results are shown alongside comparison results with Seg-Cluster.

  • Dataset and Evaluation: ScanNet contains 1513 reconstructed indoor scenes, split into 1201 training scenes and 312 test scenes.
  • Dataset and Evaluation: Each ScanNet scene is divided into 1.5m × 1.5m blocks, with 4096 points uniformly sampled per block for training.All points in each block are used at test time.
  • Dataset and Evaluation: Each point is represented by a 9D vector containing XYZ, RGB, and normalized room-relative location.
  • Results: ScanNet instance segmentation is evaluated with average precision at an IoU threshold of 0.25 and compared quantitatively with Seg-Cluster.Figure 11 presents qualitative instance-segmentation results.
Loading 1711.08588v2…