Source-linked AI summary
GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloud
Li Yi, Wang Zhao, He Wang, Minhyuk Sung, Leonidas Guibas
TL;DR
3D instance segmentation needs object proposals that remain reliable under clutter, noise, and incomplete observations. The paper introduces GSPN, which reconstructs shapes from noisy point clouds and integrates them into R-PointNet for refinement and segmentation, achieving state-of-the-art performance on several tasks.
Problem
3D object proposal is difficult because direct bounding-box approximations can be unfaithful and produce proposals containing multiple objects or object parts.
Method
GSPN uses analysis-by-synthesis to sample candidate objects from a learned natural object distribution guided by noisy scene observations, then R-PointNet refines and segments them.
Results
The GSPN-based R-PointNet achieves state-of-the-art performance on complete indoor, partial indoor, and object part instance-segmentation tasks.
Takeaways & Limitations
Geometric understanding produces higher-objectness proposals and instance-aware features that support proposal refinement and segmentation-mask generation.
Abstract
from arXiv · showhide
We introduce a novel 3D object proposal approach named Generative Shape Proposal Network (GSPN) for instance segmentation in point cloud data. Instead of treating object proposal as a direct bounding box regression problem, we take an analysis-by-synthesis strategy and generate proposals by reconstructing shapes from noisy observations in a scene. We incorporate GSPN into a novel 3D instance segmentation framework named Region-based PointNet (R-PointNet) which allows flexible proposal refinement and instance segmentation generation. We achieve state-of-the-art performance on several 3D instance segmentation tasks. The success of GSPN largely comes from its emphasis on geometric understandings during object proposal, which greatly reducing proposals with low objectness.
1. Introduction
The paper frames 3D instance segmentation as a challenging point-cloud problem and proposes geometric, generative object proposals within the flexible R-PointNet framework. The approach reconstructs object shapes from noisy observations and achieves state-of-the-art performance across several tasks.
- Motivation: 3D instance segmentation requires delineating and classifying objects, but volumetric CNN approaches incur expensive computation and memory costs.Point clouds offer an efficient representation for capturing details in large scenes.
- Motivation: 3D object proposals must handle varied categories, scales, clutter, sensor noise, and incomplete data, making faithful proposal generation difficult.Direct bounding-box approximations can create proposals containing multiple objects or only parts of objects.
- GSPN: GSPN uses analysis-by-synthesis: a generative model represents the natural object distribution, while noisy scene observations guide sampling candidate objects.The method emphasizes geometric understanding and reconstructs objects before proposing them.
- GSPN: GSPN produces higher-objectness proposals and instance-aware features that delineate boundaries for proposal refinement and segmentation-mask generation.Its explicit geometric understanding is intended to reduce blind-box proposals that do not correspond to single objects.
- R-PointNet: R-PointNet accepts and refines GSPN proposals through proposal classification, refinement, and segmentation components for unstructured point-cloud data.The framework is designed to use both object proposals and instance-sensitive features generated by GSPN.
- Results: R-PointNet with GSPN achieves state-of-the-art performance on complete indoor reconstructions, partial indoor scenes, and object part instance segmentation.The paper also reports extensive evaluation and ablation studies to validate design choices and generalizability.
2. Related Work
Prior work developed region-based 2D detection and segmentation, while 3D methods explored volumetric, projected, and point-cloud representations. The paper builds on point-cloud architectures and presents GSPN within this broader context.
- Object Detection and Instance Segmentation: R-CNN introduced region proposals for 2D object detection, followed by methods that accelerated proposals, improved features, and extended the framework to instance segmentation.The related progression includes Fast R-CNN, Faster R-CNN, and Mask R-CNN.
- Object Detection and Instance Segmentation: Early learning-based 3D detection and segmentation methods included volumetric region proposal networks and approaches borrowing mature 2D proposal techniques.Volumetric processing was limited by high memory and computation costs.
- 3D Generative Models: VAEs model data distributions and support sampling, whereas GANs can generate faithful shapes but are known to suffer from mode collapse.The paper uses CVAE rather than CGAN because it better complies with the conditioning information on average.
- Deep Learning on Point Cloud: Point-cloud representations are increasingly used for 3D learning because of their memory efficiency and intuitiveness.The paper uses PointNet, PointNet++, and Point Set Generation networks as bases for its instance-segmentation network.
- GSPN: GSPN architecture contains prior and recognition subnetworks that parameterize pθ(z|c) and qφ(z|x, c) as Gaussian distributions.Figure 2 presents the data flow and key building blocks alongside detailed block architectures.
3. Method
GSPN generates 3D shape proposals from centered, multi-scale scene contexts using a conditional generative model, then R-PointNet classifies, refines, and segments those proposals.
- Generative Shape Proposal Network: GSPN treats proposal generation as conditional object generation, sampling latent representations to reconstruct a shape proposal for a seed point.The context comprises the scene point cloud and seed point; generated points include per-point confidence scores.
- Generative Shape Proposal Network: The CVAE uses prior and recognition networks to model latent distributions, while the generation network decodes context features and sampled latents into point clouds.The recognition network additionally consumes the centered object during training to parameterize the proposal distribution.
- Generative Shape Proposal Network: A center prediction network estimates each object’s axis-aligned bounding-box center and recenters multi-scale context before generation.Three spherical context scales are processed with separate PointNets, whose features are concatenated for center regression and subsequent encoding.
- Generative Shape Proposal Network: GSPN is trained with shape generation, per-point confidence, KL, center prediction, and objectness losses.Chamfer distance supervises generated shapes, while binary cross-entropy supervises confidence and objectness and Smooth L1 supervises centers.
- Region-based PointNet: R-PointNet converts high-confidence proposals into axis-aligned boxes, extracts region features with Point RoIAlign, and predicts classification, box refinement, and segmentation.Its feature backbone combines GSPN instance-sensitive features with semantic features from a pretrained PointNet++ network.
- Region-based PointNet: Training separates stages: GSPN and the semantic segmentation network are trained first, their weights are fixed, and R-PointNet is then trained on filtered proposals.R-PointNet applies non-maximum suppression and retains up to 128 proposals, using positive and negative RoIs at an IoU threshold of 0.5.
4. Experiment
Experiments evaluate R-PointNet with GSPN on ScanNet, PartNet, and NYUv2, including scene and part instance segmentation. The framework achieves strong results, while ablations and visualizations link performance to faithful, high-objectness proposals and geometric shape priors.
- Datasets and tasks: Experiments cover ScanNet, PartNet, and NYUv2, spanning complete indoor reconstructions, partial indoor scenes, and object part instance segmentation.ScanNet contains 1,613 reconstructed indoor scenes; PartNet provides fine-grained part-instance annotations; NYUv2 supplies RGB-D images for partial point clouds.
- ScanNet: R-PointNet achieves the leading position on the ScanNet 3D semantic instance segmentation benchmark and outperforms previous state-of-the-art methods on most categories.The approach performs particularly well on categories with small geometric variation and improves results on texture-relevant categories compared with SGPN.
- ScanNet: Qualitative ScanNet comparisons show GSPN proposals have higher objectness, whereas SGPN often predicts partial or multiple objects.SGPN may break one object into pieces or miss objects; GSPN generates proposals that support better segmentation quality.
- PartNet: R-PointNet outperforms SGPN on all evaluated PartNet categories and segments parts across varied scales, including enclosed parts, with smoother and less noisy instances.Both methods struggle when chair bars are treated as separate instances, but the proposed approach still produces smoother segments with less noise.
- NYUv2: On NYUv2, R-PointNet outperforms SGPN-CNN, Mask RCNN, and Mask R-CNN* in mAP and provides the best results on 12/19 classes.It remains effective on partial point clouds with severe data missing, while Mask R-CNN performs best for categories with strong appearance but weak geometric features.
- Ablation study: Replacing GSPN with binary segmentation or direct box regression produces proposals covering partial or multiple objects, while GSPN achieves the best performance on both evaluation metrics.Removing either the instance-sensitive context feature or semantic feature also causes performance degeneration.
5. Conclusion
The paper presents GSPN as an object proposal network for 3D point-cloud instance segmentation and incorporates it into R-PointNet, achieving state-of-the-art performance on several benchmarks.
- GSPN generates high-objectness object proposals for instance segmentation in 3D point-cloud data.Its proposals are designed to improve the performance of an instance segmentation framework.
A. Architecture Details
The architecture details describe GSPN and R-PointNet, including their network components, proposal filtering settings, and refinement-to-segmentation flow.
- GSPN: GSPN uses center prediction, prior, recognition, and generation networks built on PointNet or PointNet++ backbones.The architecture notation includes set abstraction, feature propagation, deconvolution, MLP, and fully connected layers.
- Reference tables: Table 7 reports the detailed architecture of GSPN, while Table 8 reports the main configuration parameters used during training and inference.The tables provide the detailed network design and configuration reference for the framework.
- R-PointNet: R-PointNet has classification, segmentation, and bounding box refinement heads for proposal processing and instance segmentation.Its classification and refinement heads transform pooled features into category scores and category-specific box updates.
- R-PointNet: During inference, predicted RoIs are refined by the bounding box head and passed through Point RoIAlign before segmentation.The segmentation head predicts one segment per category, while training updates use the ground-truth category prediction.
- Configuration: Configuration parameters control seed-point sampling, post-GSPN objectness filtering, proposal-count limits after NMS, and the 3D IoU threshold.The passage defines these settings as main training and inference parameters, with their overall organization reported in Table 8.