Source-linked AI summary
3D Graph Embedding Learning with a Structure-aware Loss Function for Point Cloud Semantic Instance Segmentation
Zhidong Liang, Ming Yang, Chunxiang Wang
TL;DR
3D semantic instance segmentation must handle sparse, geometrically structured point clouds while producing both semantic and instance outputs. The paper combines sparse convolution, structure-aware embedding supervision, and attention-based graph aggregation, then clusters refined embeddings. It reports outperforming state-of-the-art methods on ScanNet and NYUv2, while leaving multi-sensor fusion for future work.
Problem
3D semantic instance segmentation requires discriminative instance embeddings that account for both embedding similarity and geometric relations in point clouds.
Method
The method uses a submanifold sparse convolutional network, a structure-aware loss, attention-based KNN graph convolution, and mean-shift clustering to produce semantic and instance predictions.
Results
The proposed approach outperforms all state-of-the-art methods on the ScanNet benchmark [Dai et al., 2017] and NYUv2 dataset.
Takeaways & Limitations
The framework provides an end-to-end approach that jointly outputs semantic predictions and instance predictions from point clouds.
Takeaways & Limitations
The paper leaves multi-sensor fusion as future work to further increase precision.
Abstract
from arXiv · showhide
This paper introduces a novel approach for 3D semantic instance segmentation on point clouds. A 3D convolutional neural network called submanifold sparse convolutional network is used to generate semantic predictions and instance embeddings simultaneously. To obtain discriminative embeddings for each 3D instance, a structure-aware loss function is proposed which considers both the structure information and the embedding information. To get more consistent embeddings for each 3D instance, attention-based k nearest neighbour (KNN) is proposed to assign different weights for different neighbours. Based on the attention-based KNN, we add a graph convolutional network after the sparse convolutional network to get refined embeddings. Our network can be trained end-to-end. A simple mean-shift algorithm is utilized to cluster refined embeddings to get final instance predictions. As a result, our framework can output both the semantic prediction and the instance prediction. Experiments show that our approach outperforms all state-of-art methods on ScanNet benchmark and NYUv2 dataset.
1 Introduction
The paper targets 3D semantic instance segmentation, using geometric structure and embedding information to produce semantic predictions and instance embeddings. It proposes attention-based neighbour aggregation and reports state-of-the-art performance on ScanNet and NYUv2.
- 1 Introduction: The paper focuses on 3D semantic instance segmentation, motivated by 3D data's geometric information and the sparsity and computational cost of 3D understanding.The target applications include augmented reality and autonomous driving.
- 1 Introduction: The framework outputs semantic predictions and instance embeddings, then uses mean-shift clustering to obtain final instance predictions.The network input contains point-cloud xyz coordinates and RGB attributes.
- 1 Introduction: Proposal-free embedding methods simplify implementation and jointly produce semantic and instance segmentation, while avoiding proposal-based methods' overlapping-label and complex-training drawbacks.They can also retain objects without instance labels, such as sky or road.
- 1 Introduction: The method uses a submanifold sparse convolutional network to obtain semantic labels and initial embeddings, with a structure-aware loss combining geometric and embedding information.The loss is designed because embeddings can differ within an instance, especially near object edges.
- 1 Introduction: An attention-based graph convolutional network aggregates useful information from KNN neighbours while reducing incorrect message passing across instance boundaries.Nearby 3D points are more likely than adjacent image pixels to belong to the same instance, but edge points can still have neighbours from different instances.
- 1 Introduction: Experiments report that the proposed method outperforms all state-of-the-art methods on the ScanNet benchmark [Dai et al., 2017] and NYUv2 dataset.This is the paper's reported overall benchmark comparison.
2 RELATED WORK
Prior work includes proposal-based and embedding-based instance segmentation in images, point-cloud instance methods, sparse voxel convolutions, PointNet, and graph CNNs. These approaches motivate embedding learning, efficient sparse processing, and neighbour-based aggregation for point clouds.
- Instance Segmentation in 2D Images: Embedding-based 2D methods learn pixel embeddings and cluster them, offering simpler implementation while jointly supporting semantic- and instance-level segmentation.The cited embedding approaches include [Fathi et al., 2017; Kong and Fowlkes, 2018; De Brabandere et al., 2017].
- Instance Segmentation in Point Clouds: Point-cloud instance segmentation has used per-point embeddings with a double-hinge loss, shape-reconstruction proposals with PointNet++, and detection-based methods.The cited examples include [Wang et al., 2018a], [Yi et al., 2018], and [Hou et al., 2018].
- Deep Learning on Point Clouds: Sparse convolutional methods exploit 3D sparsity to improve voxel-based resolution and efficiency, while PointNet directly processes raw point clouds without quantization errors.The relevant methods are cited as [Tchapmi et al., 2017], [Riegler et al., 2017; Wang et al., 2017; Graham et al., 2018], and [Qi et al., 2017].
- Graph CNN: Graph CNNs generalize convolution to non-Euclidean data and pass messages between neighbours, making point clouds suitable for local graph-based feature extraction.The cited graph-CNN work includes [Bruna et al., 2013; Kipf and Welling, 2016; Simonovsky and Komodakis, 2017; Wang et al., 2018b].
3 METHOD DESCRIPTION
The method jointly predicts semantics and instance embeddings, then refines embeddings with attention-based graph convolutions and supervises both initial and refined representations using a structure-aware loss.
- 3.1 Network Architecture: The network uses a submanifold convolutional backbone, separate MLPs for semantic predictions and instance embeddings, GCN refinement, and three loss terms in end-to-end multi-task training.The initial and refined embeddings both contribute to the structure-aware loss.
- 3.2 Structure-aware Loss Function: Structure-aware supervision combines within-instance compactness with between-instance separation using Euclidean embedding distances and thresholds α and β.The within-instance term penalizes large embedding distances, while the between-instance term stops contributing once mean embeddings are sufficiently separated.
- 3.3 Attention-based K Nearest Neighbour: Attention-based KNN searches neighbours spatially, learns normalized weights from neighbour and center embeddings, and aggregates a weighted average for refined representations.Unlike standard averaging, the learned weights allow different neighbours to receive different importance.
- 3.4 Graph Convolutional Neural Network using Attention-based KNN: The graph convolutional network concatenates the aggregated neighbour embedding with the input embedding and maps them through a trainable fully connected updater.The proposed aggregator uses attention-based KNN rather than uniform KNN averaging.
- 3.4 Graph Convolutional Neural Network using Attention-based KNN: The spatial KNN formulation has complexity O(n × k) instead of the O(n^2) complexity of a Laplacian-matrix formulation, avoiding eigendecomposition.The method is presented as a spatial graph convolution that does not require calculating the graph Laplacian or its eigendecomposition.
- 3.4 Graph Convolutional Neural Network using Attention-based KNN: The central design choice is learning different importances for neighbouring points through attention-based KNN aggregation.This is intended to provide a more meaningful aggregation operation for point clouds than simple averaging.
4 EXPERIMENTS
Experiments evaluate the point-cloud-only method on ScanNet and NYUv2, with comparisons spanning benchmark performance, qualitative predictions, and network ablations. The method outperforms prior approaches across both datasets, while structure-aware loss, GCN depth, and backbone capacity materially affect results.
- Experimental Setup: The experiments use ScanNet and NYUv2 3D instance-segmentation datasets, with AP0.5 on ScanNet and AP0.25 on NYUv2.ScanNet follows the official 1201/300/100 train-validation-test split, while NYUv2 uses 795 training and 654 testing samples.
- 4.1 Instance Segmentation on ScanNet: The ResNet-backbone model outperforms all methods by a large margin on ScanNet, while the UNet model outperforms almost all state-of-the-art methods except image-using 3D-SIS.Both proposed models use only point-cloud data, whereas 3D-SIS additionally uses multiple images.
- 4.2 Instance Segmentation on NYUv2: The point-cloud-only method outperforms all state-of-the-art methods on NYUv2 and achieves the highest precision for many categories.Image-based methods perform better for some categories that are difficult to recognize from NYUv2’s partial point clouds; multi-sensor fusion is left for future work.
- 4.3 Ablation Study: Structure-aware loss improves AP0.5 and AP0.25 by more than 1% without improving semantic-segmentation IoU, indicating gains from more discriminative embeddings.The comparison is against a vanilla loss that weights points within an instance equally.
- 4.3 Ablation Study: Two GCN layers achieve the best result, whereas additional layers may cause overfitting, oversmoothing, or greater training difficulty.The ablation varies the number of GCN layers to assess their contribution.
- 4.3 Ablation Study: The ResNet backbone substantially outperforms the UNet backbone, showing that backbone choice strongly affects the model’s performance.The ResNet-like backbone has larger capacity and slower speed than the UNet-like backbone.
5 CONCLUSIONS
The paper presents a 3D semantic instance segmentation approach that combines geometric structure with embedding learning and refines embeddings through attention-based graph convolution. It outperforms state-of-the-art methods on ScanNet and NYUv2, while identifying multi-sensor fusion as future work.
- The approach outperforms all state-of-the-art methods on the ScanNet benchmark and NYUv2 dataset.
- Its structure-aware loss uses geometric information to generate discriminative embeddings for 3D instance segmentation.
- An attention-based KNN graph convolutional network refines initial embeddings through automatic feature selection and aggregation.