PaperlayerView exact revision on arXiv

Source-linked AI summary

Associatively Segmenting Instances and Semantics in Point Clouds

Xinlong Wang, Shu Liu, Xiaoyong Shen, Chunhua Shen, Jiaya Jia

arXiv:1902.09852v2cs.CV

TL;DR

The paper addresses the limited treatment of joint instance and semantic segmentation in informative 3D point clouds. It introduces an end-to-end framework that lets the tasks assist each other through semantic-aware instance embeddings and instance-level semantic feature fusion. ASIS outperforms state-of-the-art 3D instance-segmentation methods while significantly improving 3D semantic segmentation, and also benefits ShapeNet part segmentation.

  • Problem

    Prior work made progress on individual tasks, but did not associatively segment instances and semantics in 3D point clouds.

  • Method

    ASIS jointly trains parallel instance and semantic branches, using semantic-aware point-level instance embeddings and fused same-instance semantic features.

  • Results

    ASIS outperforms state-of-the-art 3D instance-segmentation methods on S3DIS and significantly improves 3D semantic segmentation, while also benefiting ShapeNet part segmentation.

  • Takeaways & Limitations

    The mutually assisted framework provides a flexible approach for jointly segmenting instances and semantics in point clouds and supports part segmentation.

  • Takeaways & Limitations

    The method can mistakenly merge two nearby chairs into one instance and does not improve this type of distinction between same-class instances.

Abstract

from arXiv · show

A 3D point cloud describes the real scene precisely and intuitively.To date how to segment diversified elements in such an informative 3D scene is rarely discussed. In this paper, we first introduce a simple and flexible framework to segment instances and semantics in point clouds simultaneously. Then, we propose two approaches which make the two tasks take advantage of each other, leading to a win-win situation. Specifically, we make instance segmentation benefit from semantic segmentation through learning semantic-aware point-level instance embedding. Meanwhile, semantic features of the points belonging to the same instance are fused together to make more accurate per-point semantic predictions. Our method largely outperforms the state-of-the-art method in 3D instance segmentation along with a significant improvement in 3D semantic segmentation. Code has been made available at: https://github.com/WXinlong/ASIS.

1. Introduction

ASIS addresses the lack of methods that jointly segment instances and semantics in 3D point clouds. It uses an end-to-end framework in which the two tasks cooperate through semantic-aware instance embeddings and instance-based semantic feature fusion.

  • Motivation: Instance and semantic segmentation are related point-grouping tasks, but prior methods did not tackle them associatively despite their broad applications.The tasks can share information because different classes must form different instances, while points in one instance share a category.
  • Framework: ASIS integrates both tasks into an end-to-end parallel training framework instead of using a step-wise pipeline.The framework shares benefits between the tasks in a soft and learnable fashion.
  • Framework: The baseline uses parallel branches for per-point semantic predictions and point-level instance embeddings, with same-instance points close and different-instance points separated.The baseline is tailored to 3D point clouds and reportedly trains and infers faster than SGPN while achieving better performance.
  • Mutual aid: ASIS learns semantic-aware instance embeddings by fusing semantic information, further separating points from different semantic classes.Figure 2 visualizes the embeddings with t-SNE on S3DIS test data, comparing baseline patches with ASIS patches through color differences representing embedding distances.
  • Mutual aid: ASIS fuses semantic features from points belonging to the same instance to improve per-point semantic predictions.The method is trained end-to-end and is reported to outperform state-of-the-art 3D instance-segmentation methods on S3DIS with significant semantic-segmentation improvement.
  • Results: Experiments on ShapeNet show that ASIS also benefits part segmentation.The paper reports this as an additional experimental finding beyond its primary instance- and semantic-segmentation tasks.

2. Related Work

Prior work advanced 2D and 3D instance or semantic segmentation separately, using proposals, convolutional architectures, point-level networks, and superpoint graphs. Few approaches used instance embeddings to improve semantic segmentation in either 2D or 3D point clouds.

  • Instance Segmentation: 2D instance-segmentation methods commonly propose segment candidates or proposals, then classify them to obtain final instance masks.Some approaches generate segment proposals from bounding-box proposals, while others use region-based strategies.
  • Semantic Segmentation: Semantic segmentation on 2D images has been dominated by fully convolutional network approaches.The passage frames these methods as part of the recent progress in semantic segmentation.
  • Semantic Segmentation: 3D semantic-segmentation methods include coarse voxel-level prediction, fine-grained point-level prediction, and superpoint graphs for large-scale point clouds.PointNet and subsequent methods use multilayer perceptrons to produce point-level segmentation.
  • Semantic Segmentation: Few previous methods used instance embeddings to benefit semantic segmentation in either 2D images or 3D point clouds.This observation distinguishes the paper’s cross-task motivation from the reviewed single-task approaches.
  • Deep Learning on Point Clouds: Point-cloud learning methods use multiview projections or voxelized volumes, but projection can discard part of the point cloud’s contextual information.These representations enable use of established 2D CNNs or regular volumetric processing.

3. Our Method

The method uses a shared encoder with parallel semantic and instance branches, then links both tasks through semantic-aware instance features and instance-fused semantic features. These interactions provide a soft, learnable way for each segmentation task to support the other.

  • 3.1. A Simple Baseline: A shared encoder feeds parallel decoders for per-point semantic predictions and instance embeddings.The semantic branch outputs class predictions, while the instance branch learns embeddings that cluster points from the same instance and separate different instances.
  • 3.1. A Simple Baseline: The baseline supervises semantic predictions with cross entropy and instance embeddings with a discriminative loss.The loss pulls embeddings toward instance centers, repels different instances, and regularizes embedding magnitudes; α is set to 0.001.
  • 3.1. A Simple Baseline: At test time, mean-shift clustering converts instance embeddings into instance labels, while each instance receives the mode of its points’ semantic labels.This combines clustered instance assignments with category prediction from the semantic branch.
  • 3.2. Mutual Aid: The ASIS module adds adapted semantic features to instance features, producing semantic-aware instance features before generating instance embeddings.The semantic features are transformed through a fully connected layer with batch normalization and ReLU, then added element-wise to the instance features.
  • 3.2. Mutual Aid: Semantic awareness further repels different-category instances while leaving same-category instances comparatively unaffected in instance feature space.The resulting feature matrix is used to generate the final instance embeddings.
  • 3.2. Mutual Aid: Instance-fused semantic segmentation selects nearby points in instance-embedding space, filters outliers, and channel-wise max-aggregates their semantic features.The resulting fused features are passed to the final semantic classifier, refining the semantic representation for each centroid point.

4.1. Experiment Settings

Experiments evaluate the method on S3DIS and ShapeNet using established segmentation metrics, cross-validation procedures, and specified training settings.

  • Datasets: Experiments use S3DIS, containing 272 rooms across six scanned areas, with instance labels and 13 semantic categories per point.
  • Datasets: ShapeNet part provides an additional benchmark comprising 16,881 3D shapes from 16 categories.
  • Evaluation Metrics: Semantic segmentation is evaluated with oAcc, mAcc, mIoU, and per-class IoU, while instance segmentation uses Cov, WCov, mPrec, and mRec.
  • Evaluation Metrics: S3DIS experiments follow six-fold cross-validation with micro-averaging and additionally report results on Area 5, the fifth fold.
  • Training and Inference Details: S3DIS training uses 1m×1m overlapped blocks containing 4096 points, with 9-dimensional point features and specified embedding, fusion, and optimization settings.

4.2. S3DIS Results

On S3DIS, the framework jointly segments instances and semantics, while ASIS improves both tasks through semantic awareness and instance fusion. It also maintains efficient computation and benefits from stronger backbones.

  • Baseline Method: 46.3 mWCov gives the baseline a 5.5-point advantage over SGPN, while training takes 4–5 hours instead of SGPN’s 16–17 hours.
  • ASIS: ASIS reaches 48.2 mWCov, improving over the baseline by 1.9 points; mean precision gains 2.6 points, with further Area 5 gains of 2.7 mWCov and 2.2 mPrec.
  • Qualitative Results: Qualitative results show clearer separation of nearby different-class instances, while nearby chairs of the same class can still be merged incorrectly.
  • ASIS: ASIS improves semantic segmentation by 1.6 mIoU overall, including 2.8 mAcc and 1.7 mIoU gains on Area 5.
  • Stronger Backbone: With PointNet++, ASIS gains 2.1 mWCov and 2.6 mIoU on Area 5, supporting its use with different backbone networks.
  • Analysis: Instance fusion alone adds 0.5 mIoU, semantic awareness alone adds 1.1 mWCov, and combining both produces larger gains through mutual improvement.

4.3. ShapeNet Results

On ShapeNet, ASIS is evaluated for part segmentation using generated instance annotations and improves semantic segmentation with both PointNet and PointNet++.

  • ShapeNet Evaluation: ShapeNet instance annotations are generated rather than real ground truths, so only qualitative part-instance results are reported.
  • Instance Segmentation: ASIS groups car tires and chair legs into individual instances in the qualitative ShapeNet results.
  • Semantic Segmentation: ASIS improves semantic segmentation by 0.6 points with PointNet and by 0.7 mIoU with PointNet++.

5. Conclusion

ASIS associates instance and semantic segmentation in point clouds through a novel framework, with experiments demonstrating its effectiveness and efficiency on S3DIS and ShapeNet part datasets.

  • ASIS explicitly associates instance and semantic segmentation in point clouds through a novel segmentation framework.
  • Experiments on S3DIS and ShapeNet part datasets demonstrate ASIS's effectiveness and efficiency.
  • The proposed method is intended for applications in 3D instance and semantic segmentation and may provide insights for future segmentation work.
Loading 1902.09852v2…