PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Boundary Learning for Point Cloud Segmentation

Liyao Tang, Yibing Zhan, Zhe Chen, Baosheng Yu, Dacheng Tao

arXiv:2203.05272v2cs.CV

TL;DR

3D point cloud segmentation methods often perform unsatisfactorily at scene boundaries, motivating explicit boundary analysis. The paper introduces Contrastive Boundary Learning (CBL), which contrasts features across boundaries at multiple scales and consistently improves boundary and overall segmentation performance across baselines.

  • Problem

    Existing 3D point cloud segmentation methods often overlook scene boundaries and perform unsatisfactorily in boundary areas.

  • Method

    CBL quantifies boundary performance and enhances feature discrimination across scene boundaries using contrastive learning with multi-scale sub-scene boundary mining.

  • Results

    CBL consistently improves boundary-area and overall performance across different baseline methods and datasets.

  • Takeaways & Limitations

    The results demonstrate the effectiveness of CBL and the importance of scene boundaries in 3D point cloud segmentation.

  • Takeaways & Limitations

    The boundary contrastive learning uses a local neighborhood radius of 0.1 for sampling positive and negative points.

Abstract

from arXiv · show

Point cloud segmentation is fundamental in understanding 3D environments. However, current 3D point cloud segmentation methods usually perform poorly on scene boundaries, which degenerates the overall segmentation performance. In this paper, we focus on the segmentation of scene boundaries. Accordingly, we first explore metrics to evaluate the segmentation performance on scene boundaries. To address the unsatisfactory performance on boundaries, we then propose a novel contrastive boundary learning (CBL) framework for point cloud segmentation. Specifically, the proposed CBL enhances feature discrimination between points across boundaries by contrasting their representations with the assistance of scene contexts at multiple scales. By applying CBL on three different baseline methods, we experimentally show that CBL consistently improves different baselines and assists them to achieve compelling performance on boundaries, as well as the overall performance, eg in mIoU. The experimental results demonstrate the effectiveness of our method and the importance of boundaries for 3D point cloud segmentation. Code and model will be made publicly available at https://github.com/LiyaoTang/contrastBoundary.

1. Introduction

3D point cloud segmentation methods often overlook scene boundaries, where accuracy is unsatisfactory. The paper quantifies boundary quality and proposes Contrastive Boundary Learning (CBL), which improves boundary and overall segmentation across baselines and datasets.

  • Motivation: Scene boundaries receive little attention in existing 3D point cloud segmentation methods despite their importance for accurate scene understanding.Boundary errors can affect recognition of smaller object categories and applications such as autonomous driving.
  • Contribution: The paper introduces boundary-focused metrics, including boundary-area mIoU, to quantify segmentation quality beyond overall performance.The study separately examines boundary and inner areas to expose boundary-specific weaknesses.
  • Method: CBL contrasts point features across scene boundaries and uses sub-scene boundary mining to incorporate boundary information at multiple semantic scales.The method operates on sub-sampled point clouds and optimizes feature representations around boundary areas.
  • Results: Experiments across three baselines and four datasets show consistent improvements in boundary and overall segmentation performance.CBL helps RandLANet surpass state-of-the-art methods on Semantic3D and enables a basic ConvNet to achieve leading performance on S3DIS.

2. Related work

Prior point cloud segmentation work includes projection-based and point-based methods, but scene boundaries remain comparatively underexplored. Existing boundary methods use explicit prediction or local aggregation, whereas CBL applies contrastive learning with multi-scale sub-sampled context.

  • Point cloud segmentation: Recent point cloud segmentation methods are broadly divided into projection-based and point-based approaches.Projection-based methods map points to 2D images or 3D voxels, while point-based methods operate directly on points.
  • Boundary segmentation: Scene boundaries have rarely been explored in point cloud segmentation compared with their longer history in 2D image processing.Only a few 3D works explicitly recognize the significance of boundaries.
  • Boundary segmentation: Existing 3D boundary methods rely on complex explicit boundary prediction or local aggregation modules.CBL instead demonstrates that contrastive learning can improve segmentation quality around boundaries.
  • Contrastive Boundary Learning: CBL uses sub-sampled point clouds to examine scene context at multiple scales rather than relying only on input-point features.This distinguishes the proposed approach from the cited boundary-related methods.

3. Segmentation on Boundaries

The paper defines ground-truth and predicted boundary sets from neighboring points with different labels, then evaluates segmentation separately on boundary and inner regions. B-IoU additionally measures alignment between predicted and ground-truth boundaries.

  • Boundary definition: A ground-truth boundary point has a differently labeled neighbor, while a predicted boundary point has a neighbor with a different predicted label.Both definitions use local neighborhoods around each point.
  • Boundary definition: The boundary neighborhood radius is set to 0.1 following common practice.This radius defines the local neighborhood used for boundary identification.
  • Boundary metrics: mIoU@boundary measures mean intersection-over-union within boundary points, while mIoU@inner measures it within non-boundary points.Comparing these values reveals differences between boundary and inner-area segmentation quality.
  • Boundary metrics: B-IoU measures the intersection-over-union between predicted and ground-truth boundary sets.It captures boundary alignment and accounts for false predicted boundaries.

4. Method

CBL improves feature discrimination across scene boundaries by contrasting boundary-point representations with nearby same-category and different-category points. It extends this boundary-focused learning to sub-sampled point clouds at multiple scales.

  • Contrastive Boundary Learning: CBL contrasts boundary-point features with nearby same-category positives and different-category negatives to improve discrimination across scene boundaries.Positive and negative samples are restricted to each boundary point’s local neighborhood.
  • Contrastive Boundary Learning: Boundary points are identified from ground-truth labels, and contrastive optimization is applied only to these points.The method uses boundary points Bl and local neighborhoods Ni for contrastive pairing.
  • Sub-scene Boundary Mining: CBL examines boundaries in sub-sampled point clouds to apply contrastive learning across multiple backbone stages and scales.Sub-scene labels are iteratively derived from the groups aggregated during sub-sampling.
  • Sub-scene Boundary Mining: Each sub-sampled point receives a class-distribution annotation from its aggregated input points, with average pooling used during propagation.The distribution’s kth entry represents the proportion of class k in the corresponding point group.
  • Sub-scene Boundary Mining: The arg max of each sub-scene annotation determines boundary-point labels for evaluation and feature optimization at later stages.The paper chooses arg max for simplicity and a non-parametric procedure.

5. Implementation Details and Baselines

The implementation uses a continuous-space 3D ConvNet encoder-decoder with an optional multi-scale prediction head, and applies CBL to multiple backbone baselines. The reported table evaluates methods on S3DIS Area 5.

  • 3D ConvNet Baseline: The 3D ConvNet baseline uses convolution in continuous 3D space, with a continuous kernel approximated by a one-layer MLP.The convolution is based on spatial relations between a center point and its neighbors.
  • Multi-scale Prediction: The multi-scale prediction head concatenates point features from each sub-sampled point cloud into the final output layer.The paper reports that this concatenation fails without CBL in its ablation study.
  • Evaluation: Table 1 reports results on the S3DIS test set Area 5 using each method’s officially released code.Methods marked with an asterisk also consider boundaries.
  • Baselines: CBL is applied to RandLA-Net and CloserLook3D in addition to the ConvNet baseline to assess its generalizability across backbones.RandLA-Net uses random sampling and attentive local aggregation, while CloserLook3D uses a parameter-free PosPool module.

6. Experiments

Experiments show that boundary regions are substantially harder to segment than interior regions, and that CBL improves boundary and overall performance across baselines, datasets, and scene types.

  • The Boundary Problem in Experiment: Boundary-region mIoU is much lower than inner-region mIoU, with overall performance between them, indicating boundaries degrade segmentation quality.The comparison uses mIoU, mIoU@boundary, mIoU@inner, and B-IoU.
  • The Boundary Problem in Experiment: CBL produces larger gains on mIoU@boundary and B-IoU than on overall mIoU and mIoU@inner across all three baselines.This directly supports the framework’s intended focus on boundary areas.
  • Performance Comparison: On S3DIS Area 5, CBL consistently improves three baselines, with especially large gains for boundary-sensitive classes such as column.ConvNet reaches 69.4 mIoU, while the column class improves by +13 relative to the ConvNet baseline.
  • Performance Comparison: Six-fold S3DIS cross-validation reports +9.5 for column and improvements across all classes except one (-0.2), with ConvNet achieving 73.1 mIoU.The results are presented as evidence that CBL is effective beyond a single Area 5 split.
  • Performance Comparison: On Semantic3D, CBL consistently improves ConvNet and RandLA-Net, including vegetation classes, while RandLA-Net reaches 78.4 mIoU.The authors connect the vegetation gains to improved separation of challenging scene boundaries.
  • Performance Comparison: CBL also shows consistent improvement on ScanNet and NPM3D, indicating robustness across baselines, datasets, and indoor or outdoor scenes.The reported experiments cover S3DIS, ScanNet, Semantic3D, and NPM3D.
  • Ablation Studies: Ablation results show that input-level CBL helps, sub-scene boundary mining yields a larger improvement, and the multi-scale head further boosts ConvNet with CBL.A direct multi-scale head without CBL can hurt performance by -0.09 in OA, whereas CBL improves both mIoU and OA.

7. Conclusion

The paper identifies unsatisfactory boundary segmentation, introduces CBL to optimize boundary features, and reports consistent improvements across baselines and datasets. It also acknowledges that the analysis mainly focuses on scene boundaries rather than broad inner areas.

  • CBL explicitly optimizes boundary features and improves segmentation performance on scene boundaries.
  • CBL achieves leading performance and consistent improvements across various baselines and datasets.
  • The supplementary material details baseline architecture, boundary analysis, visualizations, training, temperature, annotation design, and further experiments.
  • CBL achieves a new state-of-the-art result on S3DIS with a newly released transformer model.

C. Further Analysis on Boundary Problem

The analysis separates boundary and inner-area performance and shows that CBL’s gains are concentrated more strongly around scene boundaries. Qualitative visualizations further indicate enhanced feature discrimination and cleaner boundary predictions across scenes.

  • Per-class IoU is separately calculated on boundary area B_l and inner area X − B_l.
  • CBL improves performance on inner areas as well as boundaries, including larger gains for small objects such as columns.
  • CBL produces much larger improvements on boundary areas than on inner areas across all three baselines.
  • The separated metrics show that CBL’s overall improvement mainly comes from boundary areas.
  • Visualizations cover rooms, cluttered spaces, hallways, and offices, with feature discrimination shown between center points and neighboring points.
  • CBL enhances feature distances around scene boundaries and produces more detailed and cleaner boundary predictions across different scenes.

E. Training Setup in Details

The supplementary analysis describes training settings, studies the contrastive temperature, and examines alternatives for representing sub-scene annotations. It finds that maintaining distributions avoids information loss and that KL-based treatment slightly improves boundary performance but adds complexity.

  • RandLA-Net and CloserLook3D follow their released training and evaluation instructions, with the sin/cos spatial-embedding module used for CloserLook3D.
  • The ConvNet baseline trains for 600 epochs with SGD, weight decay 0.001, learning rate 0.01, momentum 0.98, and decay rate 0.11/200.Training takes roughly 24 hours on four Nvidia V100 GPUs, with no obvious increase after applying CBL.
  • The proper CBL temperature lies within (0.5, 2), with τ = 1 used by default.
  • Sub-scene annotations are distributions, but boundary evaluation uses their arg max, motivating analysis of distribution preservation and alternative utilization.
  • Using arg max discards minority classes through subsampling, producing imprecise boundaries; CBL (argmax) improves B-IoU but compromises overall performance.
  • KL divergence measures semantic distance between sub-scene annotations, with threshold 0.5 yielding small overall and slightly larger boundary improvements.The authors choose arg max in the main paper because KL thresholding introduces extra hyperparameters and complexity.

H. Further Experiments

Further experiments evaluate CBL with transformer results and qualitative scenes, reporting improved overall performance and consistent gains for classes such as columns. The experiments also document the evaluation datasets and visualization settings.

  • The experiments report quantitative results on ScanNet and NPM3D and reproduce point Transformer performance on S3DIS Area 5.
  • CBL with point Transformer improves column performance and raises overall S3DIS performance to 71.0 mIoU, achieving a new state-of-the-art result.
  • Qualitative comparisons on large rooms show improved segmentation and enhanced feature discrimination across boundary points with CBL.
  • Additional qualitative results cover cluttered spaces, hallways, offices, and large rooms.
  • The visualizations compare ground truth, baseline, CBL improvement, and input.
Loading 2203.05272v2…