PaperlayerView exact revision on arXiv

Source-linked AI summary

DyCo3D: Robust Instance Segmentation of 3D Point Clouds through Dynamic Convolution

Tong He, Chunhua Shen, Anton van den Hengel

arXiv:2011.13328v2cs.CV

TL;DR

3D point-cloud instance segmentation is challenging, while bottom-up approaches can require complex processing and be sensitive to hyperparameters. DyCo3D generates instance-conditioned convolution kernels and uses a lightweight transformer to capture long-range dependencies. The paper reports state-of-the-art results with improved robustness and inference speed.

  • Problem

    Bottom-up point-cloud instance segmentation methods can rely on complex processing and tuned hyperparameters, while 3D data sparsity and irregularity make instance segmentation challenging.

    Source: §1 · pp. 1–2

  • Method

    DyCo3D is a proposal-free approach that dynamically generates convolution filters conditioned on instance category and position, with a lightweight transformer for long-range dependencies.

    Source: §1 · p. 2

  • Results

    DyCo3D is reported to achieve state-of-the-art results with improved robustness and inference speed over comparators.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper presents dynamic convolution as a proposal-free alternative to bottom-up point-cloud instance segmentation.

    Source: §1 · p. 2

    Optimization difficulties and heavy hardware-resource requirements remain a limitation.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Previous top-performing approaches for point cloud instance segmentation involve a bottom-up strategy, which often includes inefficient operations or complex pipelines, such as grouping over-segmented components, introducing additional steps for refining, or designing complicated loss functions. The inevitable variation in the instance scales can lead bottom-up methods to become particularly sensitive to hyper-parameter values. To this end, we propose instead a dynamic, proposal-free, data-driven approach that generates the appropriate convolution kernels to apply in response to the nature of the instances. To make the kernels discriminative, we explore a large context by gathering homogeneous points that share identical semantic categories and have close votes for the geometric centroids. Instances are then decoded by several simple convolutional layers. Due to the limited receptive field introduced by the sparse convolution, a small light-weight transformer is also devised to capture the long-range dependencies and high-level interactions among point samples. The proposed method achieves promising results on both ScanetNetV2 and S3DIS, and this performance is robust to the particular hyper-parameter values chosen. It also improves inference speed by more than 25% over the current state-of-the-art. Code is available at: https://git.io/DyCo3D

1. Introduction

3D point-cloud instance segmentation is difficult because sparse, irregular scenes complicate instance identification, while bottom-up methods can depend on tuned heuristics and complex processing. DyCo3D instead generates instance-conditioned convolution filters and uses a lightweight transformer; the paper reports state-of-the-art results with improved robustness and inference speed.

1. Introduction

  • Instance segmentation must identify each individual object instance and its visible extent, a more challenging task than semantic segmentation.The passage also notes that 3D point-cloud instance segmentation is more challenging than image-domain instance segmentation because of data irregularity, sparsity, and scene diversity.Source: §1 · p. 1
  • Bottom-up approaches can be sensitive to manually tuned hyperparameters, require complex processing, and depend heavily on proposal quality.PointGroup's mAP drops by more than 6% when its clustering radius changes from 3 cm to 2 cm; 3D-MPA uses an extra 10-layer graph network and clustering.Source: §1 · pp. 1–2
  • DyCo3D generates convolution filters on the fly, conditioned on instance category and position, and decodes masks with a few convolution layers.Its weight generator gathers homogeneous points with shared semantic labels and close votes for instance centroids to create discriminative filters.Source: §1 · p. 2
  • A lightweight transformer captures long-range dependencies and high-level interactions among point-cloud regions.The transformer addresses the limited receptive field of sparse convolution.Source: §1 · p. 2
  • DyCo3D is reported to achieve state-of-the-art results with improved robustness and inference speed over comparators.The introduction also characterizes it as a proposal-free approach more robust than bottom-up strategies.Source: §1 · p. 2

2. Related Work

Prior 3D point-cloud work spans point-, voxel-, and multiview-based representations, while instance segmentation methods are commonly organized as top-down or bottom-up. DyCo3D is positioned among dynamic-convolution approaches tailored to point clouds.

Deep Learning for 3D Point Cloud

  • 3D point-cloud representations include point-based, voxel-based, and multiview-based methods, each with distinct processing strategies.Voxel methods rasterize points and apply convolutions; sparse volumetric convolution focuses computation on occupied data rather than void space.Source: §2 · p. 2

Instance Segmentation of 3D Point Cloud

  • 3D instance segmentation methods are broadly divided into top-down detection-and-segmentation approaches and bottom-up grouping approaches.Bottom-up methods range from clustering to graph-based algorithms, with examples including ASIS, PointGroup, and OccuSeg.Source: §2 · pp. 2–3

Dynamic Convolution

  • Dynamic convolution has been used for 2D image instance segmentation, but direct application to 3D point clouds faces computation and receptive-field constraints.The passage identifies optimization difficulties from added computation and limitations from sparse convolution's receptive field and representation capacity.Source: §2 · p. 3

3. Methods

DyCo3D predicts instance masks without proposals, using a sparse-convolution backbone with a lightweight transformer and dynamically generated, instance-aware filters. Semantic and centroid information guide context grouping before a three-layer decoder produces instance masks.

3.1. Overall Architecture

  • DyCo3D is proposal-free: a sparse-convolution backbone with a lightweight transformer feeds a weight generator and a three-layer instance decoder.The transformer is intended to enlarge the receptive field and capture long-range dependencies.Source: §3.1 · p. 3, §3.2 · p. 3, §3.4 · p. 4

3.2. Backbone Network

  • Three parallel backbone branches predict semantic labels, centroid offsets, and instance-mask features; a lightweight transformer enhances long-range interactions beyond sparse convolution's limited receptive field.Centroid offsets shift points toward instance centroids, with floor and wall excluded from offset prediction on ScanNetV2.Source: §3.2 · pp. 3–4

3.3. Dynamic Weight Generator

  • Homogeneous points with matching semantic predictions and nearby centroid votes are grouped, then their broader context generates instance-aware convolution filters.Each cluster is voxelized and pooled by a lightweight subnetwork to produce its filter parameters; the method reports fewer than 60 instance candidates.Source: §3.3 · p. 4, Figure 3 · p. 5

3.4. Instance Decoder

  • For each cluster, the decoder appends position embeddings to point features and applies three 1 × 1 convolution layers to predict that instance's binary mask.The decoder operates on points sharing the cluster's semantic prediction.Source: §3.4 · pp. 4–5

C · LBCE(mj

The mask loss combines semantic-label-matched binary cross-entropy with Dice loss for foreground–background imbalance. Training uses four loss terms and a staged schedule before mask-based inference with non-maximum suppression.

C · LBCE(mj

  • LBCE computes binary cross-entropy only for points whose semantic predictions match their cluster label, with a normalization term.The indicator restricts the loss to matching points.Source: C · LBCE(mj · p. 5
  • Dice loss is also used to address imbalance between foreground and background points.Source: C · LBCE(mj · p. 5

3.5. Training details

  • The training objective combines semantic segmentation, centroid supervision, and two instance-segmentation losses, with all loss weights set to 1.0.Source: §3.5 · p. 5
  • Training first runs 6k iterations on semantic segmentation and centroid prediction, then 24k iterations with all loss items.Dynamic filters depend on the semantic and centroid predictions.Source: §3.5 · p. 5
  • Inference applies NMS to instance masks scored by mean foreground semantic scores, using IoU 0.3 and ignoring clusters with fewer than 50 points.Voxel sizes are 0.02m for ScanNetV2 and 0.05m for S3DIS; the search radius is 0.03m.Source: §3.5 · p. 5

4. Experiments

Experiments evaluate DyCo3D on ScanNetV2 and S3DIS, testing its components and comparing it with prior methods. The reported results cover instance segmentation, object detection, robustness to clustering radius, and inference time.

4.1. Datasets

  • The experiments use S3DIS Area-5 and ScanNetV2, with ScanNetV2 evaluated on 18 common categories.S3DIS contains six areas; Area-5 is used for evaluation and the other sets for training. ScanNetV2 has 1,613 scans and 40 categories.Source: §4.1 · pp. 5–6

4.2. Evaluation Metrics

  • ScanNetV2 is evaluated with mAP, AP@50, and AP@25; S3DIS uses mConv, mWConv, mPrec, and mRec.For S3DIS, mConv is mean instance-wise IoU; mWConv weights it by instance size, while mPrec and mRec are mean precision and recall.Source: §4.2 · p. 6

4.3. Ablation Studies

  • The grouping operation on homogeneous points surpasses the baseline by a large margin across all metrics and reduces the number of instance candidates to fewer than 60.The baseline generates filters for each foreground point without clustering and randomly selects 150 points for instance decoding.Source: §4.3 · p. 6
  • Category masks improve mAP from 31.8% to 34.1%, and the transformer brings a further 1.3% mAP improvement.Category-aware decoding applies convolution only to category-specific points; the transformer is added at the bottleneck to capture long-range dependencies.Source: §4.3 · p. 6
  • Changing PointGroup’s clustering radius from 3cm to 2cm drops mAP by 6.3%, while DyCo3D is reported as robust to the radius.Changing the radius from 3cm to 1cm drops PointGroup mAP by 23.9%; small radii can over-segment and large radii can merge adjacent objects.Source: §4.3 · pp. 6–7, Figure 4 · p. 6
  • DyCo3D reaches 0.28s per ScanNetV2 validation scan on a 1080TI GPU, compared with PointGroup’s 0.39s on the same facility.DyCo3D takes the whole scene as input and uses NMS as its only post-processing step.Source: §4.3 · p. 7

4.4. Comparison with State-of-the-art Methods

  • On ScanNetV2 validation, DyCo3D surpasses PointGroup by 3.1% AP@25 and 3.0% AP@50 for 3D object detection.The detection boxes are obtained by fitting axis-aligned bounding boxes around the instance masks.Source: §4.4 · p. 7, Table 2 · p. 7
  • 60.9% AP@50 on S3DIS is 3.1% higher than PointGroup [20]; on ScanNetV2, Ours-S surpasses PointGroup by 0.6% mAP and Ours-L surpasses 3D-MPA [11] by 4.9% mAP.DyCo3D achieves the highest performance on S3DIS across all reported metrics; the ScanNetV2 comparisons are on the validation set.Source: §4.4 · pp. 7–8, Table 3 · p. 7, Table 4 · p. 8

5. Conclusion

DyCo3D uses dynamic convolution to respond to data characteristics at test time, improving robustness while achieving state-of-the-art results and greater efficiency than existing methods.

  • DyCo3D achieves state-of-the-art results and improved efficiency over existing methods.Source: §5 · p. 8
  • Dynamic convolution lets the segmentation method respond to data characteristics at test time, improving robustness.Source: §5 · p. 8
  • The proposed method avoids many pitfalls associated with bottom-up methods.Source: §5 · p. 8

Related papers

Loading 2011.13328v2…