PaperlayerView exact revision on arXiv

Source-linked AI summary

JSNet: Joint Instance and Semantic Segmentation of 3D Point Clouds

Lin Zhao, Wenbing Tao

arXiv:1912.09654v1cs.CV

TL;DR

3D point-cloud semantic and instance segmentation remains challenging, and prior work often handles the tasks separately. JSNet jointly processes the tasks with feature fusion and reciprocal feature exchange; the paper reports improvements on S3DIS and benefits for ShapeNet part segmentation.

  • Problem

    3D semantic and instance segmentation face challenges with large-scale noisy data and high computation and memory demands, while prior approaches often handle the tasks separately.

    Source: Introduction · p. 1, Semantic&Instance Segmentation on… · p. 3

  • Method

    JSNet uses a shared encoder and parallel decoders with PCFF feature fusion, while JISS exchanges semantic and instance features to support both tasks.

    Source: Introduction · p. 2

  • Results

    JSNet achieves significant improvement in both instance and semantic segmentation on S3DIS, and experiments indicate it is beneficial for ShapeNet part segmentation.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    The authors conclude that JSNet learns instance-aware semantic and semantic-aware instance features that can make point predictions more accurate.

    Source: Introduction · p. 2

Abstract

from arXiv · show

In this paper, we propose a novel joint instance and semantic segmentation approach, which is called JSNet, in order to address the instance and semantic segmentation of 3D point clouds simultaneously. Firstly, we build an effective backbone network to extract robust features from the raw point clouds. Secondly, to obtain more discriminative features, a point cloud feature fusion module is proposed to fuse the different layer features of the backbone network. Furthermore, a joint instance semantic segmentation module is developed to transform semantic features into instance embedding space, and then the transformed features are further fused with instance features to facilitate instance segmentation. Meanwhile, this module also aggregates instance features into semantic feature space to promote semantic segmentation. Finally, the instance predictions are generated by applying a simple mean-shift clustering on instance embeddings. As a result, we evaluate the proposed JSNet on a large-scale 3D indoor point cloud dataset S3DIS and a part dataset ShapeNet, and compare it with existing approaches. Experimental results demonstrate our approach outperforms the state-of-the-art method in 3D instance segmentation with a significant improvement in 3D semantic prediction and our method is also beneficial for part segmentation. The source code for this work is available at https://github.com/dlinzhao/JSNet.

Introduction

Joint 3D semantic and instance segmentation remains challenging, while prior approaches often treat the tasks separately. JSNet combines feature fusion with joint task-specific feature exchange and reports results across S3DIS and ShapeNet.

Introduction

  • 3D segmentation faces challenges with large-scale noisy data and high computation and memory consumption, while prior work often handled the two tasks separately.The paper also notes that ASIS requires choosing a kNN K value and distance metric and has high training memory cost.Source: Introduction · pp. 1–2
  • JSNet processes raw point clouds to produce instance and semantic predictions, using a shared encoder, parallel decoders, feature fusion, and a joint segmentation module.The joint module transforms semantic features into instance embedding space and aggregates instance features into semantic feature space.Source: Figure 1 · p. 1, Introduction · p. 2
  • PCFF fuses high- and low-level features to generate more discriminative features and improve point-prediction accuracy.The paper describes PCFF as an efficient feature-fusion module.Source: Introduction · p. 2
  • JISS is designed to make instance and semantic segmentation mutually promote each other while maintaining acceptable GPU memory consumption during training.It fuses transformed semantic features with instance features and aggregates instance features into semantic feature space.Source: Introduction · p. 2
  • JSNet reports a significant improvement in 3D instance segmentation on S3DIS and favorable performance for ShapeNet part segmentation.The ShapeNet instance annotations are described as fake ground-truth labels, so the paper presents qualitative results for part instance segmentation.Source: Introduction · p. 2

Related Work

The related work reviews deep neural approaches to point-cloud processing and methods for 3D semantic and instance segmentation. It highlights feature-learning challenges and the limited integration of the two segmentation tasks.

Related Work

  • The review focuses on deep neural methods applied to 3D point clouds, citing their robustness and efficiency in the field.Source: Related Work · p. 2

Deep learning for 3D Point Clouds

  • PointNet uses shared MLPs and max pooling but struggles with local features; PointNet++ adds hierarchy, while max pooling may discard useful detail.Continuous-filter methods include PointConv, which reweights learned filters for non-uniform sampling but requires high GPU memory during training.Source: Deep learning for 3D Point Clouds · p. 2

Semantic&Instance Segmentation on Point Clouds

  • Prior 3D semantic and instance methods use varied architectures, but few exploit instance embeddings for semantics or semantic fusion for instances.MT-PNet uses a CRF after its CNN, while ASIS exchanges semantic and instance features using kNN, with K-selection, distance-metric, and memory costs noted as limitations.Source: Semantic&Instance Segmentation on… · pp. 2–3

Proposed Method

JSNet combines a shared point-cloud backbone and parallel semantic and instance branches with PCFF and JISS. PCFF fuses decoder-layer features, while JISS exchanges semantic and instance features to support both predictions.

Network Architecture

  • The backbone combines a PointNet++ set-abstraction encoder with three PointConv feature-encoding layers and decoders that pair PointConv layers with PointNet++ feature propagation.This combination addresses PointNet++ detail loss and PointConv’s high training memory consumption.Source: Network Architecture · p. 3
  • A shared encoder feeds parallel semantic and instance decoders, producing per-point class scores and instance embeddings whose distances reflect instance relationships.The outputs have shapes Na × C and Na × K, respectively.Source: Network Architecture · p. 3
  • Training adds semantic-segmentation and instance-embedding losses; testing assigns instance labels with mean-shift clustering and semantic categories with argmax.The embedding loss pulls points toward their instance mean and pushes different instance means apart.Source: Network Architecture · p. 3

Point Cloud Feature Fusion Module

  • PCFF fuses the decoder’s last three layers through upsampling, addition, and convolution to produce refined features with acceptable computation and memory consumption.The module uses inverse-square-distance interpolation based on three nearest neighbors.Source: Point Cloud Feature Fusion Module · p. 4

Joint Instance and Semantic Segmentation

  • JISS transforms semantic features into instance space and fuses them with instance features, while also aggregating instance features into semantic space.The instance branch produces embeddings for mean-shift clustering; the semantic branch produces per-point class predictions.Source: Joint Instance and Semantic Segmentation · pp. 4–5

Experiments

JSNet is evaluated on S3DIS for instance and semantic segmentation and on ShapeNet for part segmentation. The experiments report gains over comparison methods, examine components and training strategies, and describe the evaluation setups.

Datasets and Evaluation Metrics

  • S3DIS evaluation uses six areas, 272 rooms, and 13 categories; ShapeNet contains 16,881 CAD models across 16 categories with 50 part types.S3DIS uses k-fold cross-validation and separately reports Area 5; ShapeNet follows the official 795/654 train/test split.Source: Datasets and Evaluation Metrics · p. 5

Implementation Details

  • For S3DIS, points use 9-dimensional XYZ, RGB, and normalized-room-location inputs in 1 m × 1 m blocks of 4,096 points; ShapeNet models use 2,048 points with XYZ and normals.At S3DIS test time, mean-shift clustering uses bandwidth 0.6, followed by BlockMerging across blocks.Source: Implementation Details · p. 5

Instance Segmentation on the S3DIS dataset

Semantic Segmentation on the S3DIS dataset

  • On six-fold S3DIS evaluation, JSNet improves over PointNet by 11.4 mAcc, 8.4 oAcc, and 12.8 mIoU; Area 5 gains are 9.3, 4.2, and 11.1, respectively.The reported metrics are mAcc, oAcc, and mIoU; semantic results are shown in Table 2.Source: Semantic Segmentation on the S3DIS… · p. 6, Table 2 · p. 6

ShapeNet Results

  • On ShapeNet semantic segmentation, JSNet improves mIoU by 0.9 points over PointNet++ and by 0.8 over ASIS.Because the instance annotations are generated as ground truth, part instance segmentation is presented qualitatively.Source: ShapeNet Results · p. 6, Table 3 · p. 7

Ablation Study

  • Ablations report gains from density-weighted real convolution and cross-layer feature fusion; combining instance fusion with semantic awareness improves performance more than either alone.Random sampling is reported to avoid overfitting and improve generalization; with random sampling and early stopping, the model reaches 62.9 mPrec and 55.0 mIoU.Source: Ablation Study · p. 7, Table 4 · p. 7

Conclusion

JSNet combines a shared feature encoder, parallel decoders with PCFF modules, and a JISS module for 3D point-cloud instance and semantic segmentation. The authors report significant improvement in both tasks on S3DIS and identify adding spatial geometric topology as future work.

  • JSNet achieves significant improvement in both instance and semantic segmentation on S3DIS.Source: Conclusion · p. 7
  • JSNet uses a shared feature encoder and two parallel feature decoders, each followed by a PCFF module.Source: Conclusion · p. 7
  • The JISS module lets instance and semantic segmentation take advantage of each other.Source: Conclusion · p. 7
  • The authors identify adding spatial geometric topology to the framework as future work for better segmentation results.Source: Conclusion · p. 7
Loading 1912.09654v1…