Source-linked AI summary
JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds with Multi-Task Pointwise Networks and Multi-Value Conditional Random Fields
Quang-Hieu Pham, Duc Thanh Nguyen, Binh-Son Hua, Gemma Roig, Sai-Kit Yeung
TL;DR
The paper addresses the challenge of jointly understanding semantic categories and object instances in 3D point clouds. It combines a multi-task pointwise network with a multi-value conditional random field, and reports robust joint performance and state-of-the-art semantic segmentation.
Problem
Semantic and instance segmentation are mutually dependent 3D scene-understanding tasks that have often been addressed separately.
Method
MT-PNet predicts semantic labels and instance embeddings, which are fused in MV-CRF to jointly optimise semantic and instance segmentation.
Results
Joint semantic-instance segmentation outperformed each individual task, and the method achieved state-of-the-art performance on semantic segmentation.
Takeaways & Limitations
The proposed framework provides a unified approach for semantic and instance segmentation of 3D point clouds.
Takeaways & Limitations
The variational inference formulation assumes semantic and instance labels are independent in the joint variational distribution.
Abstract
from arXiv · showhide
Deep learning techniques have become the to-go models for most vision-related tasks on 2D images. However, their power has not been fully realised on several tasks in 3D space, e.g., 3D scene understanding. In this work, we jointly address the problems of semantic and instance segmentation of 3D point clouds. Specifically, we develop a multi-task pointwise network that simultaneously performs two tasks: predicting the semantic classes of 3D points and embedding the points into high-dimensional vectors so that points of the same object instance are represented by similar embeddings. We then propose a multi-value conditional random field model to incorporate the semantic and instance labels and formulate the problem of semantic and instance segmentation as jointly optimising labels in the field model. The proposed method is thoroughly evaluated and compared with existing methods on different indoor scene datasets including S3DIS and SceneNN. Experimental results showed the robustness of the proposed joint semantic-instance segmentation scheme over its single components. Our method also achieved state-of-the-art performance on semantic segmentation.
1. Introduction
The paper targets joint semantic and instance segmentation of 3D point clouds, motivated by their mutual dependence. It combines a multi-task pointwise network with a unified conditional random field framework and reports strong benchmark results.
- 3D scene understanding remains challenging because 3D data can be large-scale and noisy.
- Semantic segmentation assigns object categories to points, whereas instance segmentation clusters points into object instances.
- MT-PNet predicts pointwise object categories while embedding points so same-instance points can be clustered together.
- MV-CRF jointly optimises semantic class labels and object instances in a unified framework using variational mean field inference.
- The pipeline scans overlapping point-cloud windows, extracts vertices, predicts labels and embeddings, then optimises MV-CRF for final results.
- Experiments on benchmark datasets found joint semantic-instance segmentation outperformed individual tasks, while semantic segmentation achieved state-of-the-art performance.
2. Related Work
Prior 3D scene-understanding work uses multi-view, volumetric, or point-cloud representations and commonly treats semantic and instance segmentation separately. Point-cloud methods offer compact geometry-aware inputs, while CRFs provide label-consistency mechanisms.
- 3D scene-understanding methods are commonly grouped by multi-view images, volumetric representations, or point clouds.
- Multi-view approach: Multi-view methods back-project image predictions into 3D but can suffer from inconsistent views and dependence on view placement.
- Volumetric approach: Volumetric methods apply convolutions on regular 3D grids, with hierarchical structures used to support high-resolution segmentation and reduce memory demands.
- Point cloud approach: Point clouds compactly store scene geometry through vertex coordinates and normals, although convolutions are not directly straightforward on them.
- Conditional Random Fields: CRFs combine unary and binary potentials, with higher-order potentials adding prior cues for semantic class inference.
- A common instance-segmentation strategy extends semantic segmentation with procedures that propose and merge object instances.
3. Proposed Method
The proposed method jointly segments 3D point clouds semantically and by object instance using MT-PNet and MV-CRF. MT-PNet predicts classes and instance embeddings, while MV-CRF jointly optimizes both label types using semantic, geometric, and embedding-based constraints.
- 3.1. Multi-Task Pointwise Network (MT-PNet): MT-PNet predicts each point’s semantic class while embedding points so same-instance points cluster together and different instances separate.Its two branches produce semantic labels and pointwise instance embeddings from overlapping 3D windows.
- 3.1. Multi-Task Pointwise Network (MT-PNet): The MT-PNet objective combines semantic prediction and embedding losses: L = Lprediction + Lembedding.The prediction loss uses cross-entropy, while the embedding loss combines pull, push, and regularization terms.
- 3.1. Multi-Task Pointwise Network (MT-PNet): The embedding loss attracts points toward their instance centroid, separates different centroids, and regularizes centroids toward the origin.When δd > 2δv, each embedding is closer to its own centroid than to other centroids.
- 3.2. Multi-Value Conditional Random Fields (MV-CRF): Unlike higher-order CRFs using detections as prior knowledge, MV-CRF treats semantic and instance labels as equally unknown and optimizes them simultaneously.Mean field variational inference approximates the posterior, with Gaussian filtering accelerating message passing for the fully connected model.
- 3.2. Multi-Value Conditional Random Fields (MV-CRF): MV-CRF assigns semantic and instance labels to graph nodes and jointly minimizes an energy incorporating physical and semantic constraints.The energy uses classification scores, geometric properties, instance embeddings, and semantic-instance consistency.
- 3.2. Multi-Value Conditional Random Fields (MV-CRF): The semantic-instance potential favors labelings with low semantic-label entropy within each object instance.This encourages vertices in one instance to share a consistent semantic class and can favor larger instances over tiny noise-induced ones.
4. Experiments
Experiments evaluate the method on S3DIS and SceneNN for semantic and instance segmentation, comparing MT-PNet with the full MV-CRF pipeline and existing methods. The joint pipeline improves segmentation performance, achieves state-of-the-art semantic segmentation, and shows category-dependent instance results.
- Experimental Setup: The full pipeline initializes semantic labels from MT-PNet, clusters its embeddings with mean shift, and jointly optimizes labels using MV-CRF.For large scenes, overlapping-window embeddings are merged before joint optimization and non-maximal suppression produces final predictions.
- Experimental Setup: Experiments use S3DIS building-level indoor scans with 13 categories and SceneNN room-scale indoor meshes with 40 semantic classes.S3DIS spaces typically contain about 10–80 object instances; both datasets use overlapping windows for segmentation.
- Semantic Segmentation: The multi-task MT-PNet significantly outperforms PointNet on semantic segmentation, while disabling its instance-embedding branch matches vanilla PointNet.This ablation attributes the improvement over PointNet to the multi-task architecture.
- Semantic Segmentation: MV-CRF improves MT-PNet’s semantic segmentation results and achieves state-of-the-art performance on S3DIS and SceneNN.Semantic evaluation reports per-class accuracy and micro-mean accuracy, with micro-mean equivalent to overall accuracy in this context.
- Instance Segmentation: MV-CRF significantly boosts instance segmentation performance compared with applying mean shift directly to MT-PNet embeddings.Instance segmentation is evaluated as object detection using average precision at an IoU threshold of 0.5.
- Instance Segmentation: Compared with existing methods, the method improves some instance categories but has low precision for doors, mainly because of low semantic segmentation accuracy.Reported improvements include floor, sofa, board, and clutter.
5. Conclusion
The paper jointly performs semantic and instance segmentation of point clouds using a multi-task pointwise network and a multi-value conditional random field. Experiments show advantages over individual components and state-of-the-art semantic segmentation performance.
- The method jointly performs semantic and instance segmentation through a multi-task pointwise network and a multi-value conditional random field.The network learns class labels and embeddings for clustering points into object instances, while the field model integrates 3D and embedded features.
- The multi-task network learns both 3D point class labels and embedded representations that enable clustering points into object instances.
- The multi-value conditional random field integrates 3D and high-dimensional embedded features for joint segmentation.
- Joint semantic-instance segmentation outperformed the method's individual components in experiments on challenging indoor datasets.
- The proposed method compared favourably with existing methods and achieved state-of-the-art performance on semantic segmentation.