Source-linked AI summary
Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds
Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, Niki Trigoni
TL;DR
3D point-cloud instance segmentation is difficult because point clouds are unordered, unstructured, and non-uniform, while voxelization is costly. 3D-BoNet directly predicts bounding boxes and point masks through parallel branches in a single-stage, anchor-free, end-to-end framework, improving objectness and efficiency without post-processing. Its design is supported by improvements over baselines and ablation studies, while remaining limited by fixed loss weighting, separate semantic prediction, and training on divided small blocks.
Problem
3D point-cloud instance segmentation must handle unordered, unstructured, and non-uniform data, while voxelization incurs high computational and memory costs.
Method
3D-BoNet uses a backbone with parallel bounding-box prediction and point-mask branches in a single-stage, anchor-free, end-to-end architecture without post-processing.
Results
3D-BoNet significantly improves over baselines, with extensive ablations demonstrating the effectiveness of its design choices.
Takeaways & Limitations
Explicitly learning 3D object boundaries provides high objectness without dense proposals, while single-forward mask learning improves computational efficiency.
Takeaways & Limitations
The framework uses unweighted loss criteria, a separate semantic-prediction branch, and divided small-block inputs rather than directly training and testing on large-scale point clouds.
Abstract
from arXiv · showhide
We propose a novel, conceptually simple and general framework for instance segmentation on 3D point clouds. Our method, called 3D-BoNet, follows the simple design philosophy of per-point multilayer perceptrons (MLPs). The framework directly regresses 3D bounding boxes for all instances in a point cloud, while simultaneously predicting a point-level mask for each instance. It consists of a backbone network followed by two parallel network branches for 1) bounding box regression and 2) point mask prediction. 3D-BoNet is single-stage, anchor-free and end-to-end trainable. Moreover, it is remarkably computationally efficient as, unlike existing approaches, it does not require any post-processing steps such as non-maximum suppression, feature sampling, clustering or voting. Extensive experiments show that our approach surpasses existing work on both ScanNet and S3DIS datasets while being approximately 10x more computationally efficient. Comprehensive ablation studies demonstrate the effectiveness of our design.
1 Introduction
3D-BoNet addresses the computational and structural challenges of 3D point-cloud instance segmentation with a single-stage framework that predicts object boxes and point masks without expensive proposal or post-processing pipelines.
- 3D point clouds are unordered, unstructured, and non-uniform, while voxelization incurs high computational and memory costs.
- Existing proposal-free methods group per-point features but require computationally heavy clustering, whereas proposal-based methods use two-stage training and non-maximum suppression.
- 3D-BoNet uses efficient MLPs to loosely detect objects and then precisely segment each instance with a point-level binary classifier.
- The framework extracts local and global features before parallel branches predict instance-level bounding boxes and point-level masks.
- The bounding-box branch directly predicts unique, unoriented rectangular boxes without predefined spatial anchors or a region proposal network.
- The predicted boxes and point features feed a mask branch that classifies each point inside each box as instance or background.
- The method explicitly learns object boundaries to improve objectness, avoids dense proposals, and learns instance masks in a single forward pass without post-processing.
2 3D-BoNet
3D-BoNet predicts a fixed set of bounding boxes from global point-cloud features and associates them optimally with ground-truth boxes for multi-criteria supervision. It then fuses box information with local and global features to predict instance masks, using differentiable point coverage criteria to handle sparse, non-uniform point clouds.
- The backbone extracts local point features and one global feature vector, which feed the bounding-box and point-mask branches.
- Bounding Box Prediction: The box branch directly regresses H fixed boxes and scores from the global feature vector, without predefined anchors or a region proposal network.
- Point Mask Prediction: Each predicted box, score, and shared local-global features produces a point-level binary mask through shared layers, making the branch class-agnostic.
- Bounding Box Prediction: Optimal assignment pairs each ground-truth box with a unique predicted box despite variable box counts and unordered predictions.
- Bounding Box Prediction: The reordered paired boxes are trained with a multi-criteria loss, while score loss discourages multiple similar boxes for one instance.
- Bounding Box Prediction: The association cost combines vertex distance, soft point IoU, and cross-entropy coverage to balance geometric boundaries with inclusion of valid points.
3 Experiments
Experiments on ScanNet and S3DIS evaluate 3D-BoNet against existing methods, while ablations examine its box prediction, association, supervision, and mask-loss components.
- ScanNet Evaluation: On ScanNet, 3D-BoNet surpasses leading approaches across 18 object categories using point clouds only.Compared methods include clustering-based, proposal-based, and RGB-supported approaches.
- S3DIS Evaluation: On S3DIS, 3D-BoNet surpasses the PartNet baseline by large margins and outperforms ASIS, though not significantly.The authors attribute the limited margin over ASIS mainly to weaker semantic prediction from vanilla PointNet++.
- Ablation Study: The box score sub-branch improves overall instance segmentation by penalizing duplicated box predictions.The analysis reports results for all ablation experiments in Table 3.
- Ablation Study: The differentiable sIoU cost tends to outperform Euclidean distance and cross-entropy for box association and supervision.The three criteria prefer different point structures, so their simple combination may not be optimal on every dataset.
- Ablation Study: Removing box prediction supervision significantly reduces performance because predicted boundaries and point masks deteriorate.The ablation study evaluates six component and loss variants on Area 5 of S3DIS.
- Ablation Study: Focal loss is more effective than standard cross-entropy for point mask prediction under imbalanced instance and background point counts.The comparison replaces focal loss with standard cross-entropy.
4 Related Work
Related 3D point-cloud segmentation methods use voxelization, semantic backbones, projection or proposal pipelines, while 3D-BoNet directly predicts masks within detected object boundaries.
- Feature Extraction: Learning-based 3D feature extraction mainly uses voxel-based or point-based schemes, alongside traditional manually crafted features.The paper notes that point clouds are unordered, unstructured, and non-uniform, making voxelization costly.
- Semantic Segmentation: Semantic segmentation networks can serve as 3D-BoNet backbones and be trained in parallel to learn per-point semantics.PointNet is cited as a leading classification and semantic-segmentation approach that lacks context features.
- Object Detection: Many 3D object detectors rely on predefined anchors and two-stage region proposal networks, which the paper describes as inefficient for 3D point clouds.Other detection pipelines project points to images, fuse RGB, or divide point clouds into voxels.
- Instance Segmentation: Proposal-free instance methods group or classify per-point features but do not explicitly detect object boundaries, whereas proposal-based methods use dense proposals and post-processing.3D-BoNet predicts point-level masks within explicitly detected boundaries without post-processing.
5 Conclusion
The conclusion presents 3D-BoNet as simple, effective, and efficient, while identifying adaptive loss weighting, feature fusion, and large-scale input processing as future directions.
- Conclusion: 3D-BoNet is described as simple, effective, and efficient for 3D point-cloud instance segmentation.The conclusion frames these properties as the framework’s overall assessment.
- Future Work: Future work includes automatically learning criterion weights to adapt to different types of input point clouds.This would replace the current unweighted combination of three criteria.
- Future Work: The authors propose advanced feature fusion to mutually improve semantic and instance segmentation instead of using a separate semantic branch.The current semantic and instance predictions are trained through separate branches.
- Future Work: The MLP design is agnostic to point number and order, motivating direct training and testing on large-scale point clouds rather than divided small blocks.The proposed direction draws on recent work for large-scale inputs.
A Experiments on ScanNet Benchmark
The ScanNet evaluation divides scenes into overlapping blocks for training, then merges block predictions for complete-scene testing. Qualitative results show generally complete instances but failures on very similar objects.
- Dataset and Protocol: ScanNet(v2) contains 1,613 complete indoor scenes, split into 1,201 training, 312 validation, and 100 hidden testing scenes.Large point clouds are divided into 1m × 1m blocks with 0.5m overlap.
- Dataset and Protocol: Training samples 4,096 points per block, while testing uses all points from each block.The same block-based preprocessing is also used for S3DIS in the cited setup.
- Qualitative Results: Qualitative ScanNet results show that 3D-BoNet tends to predict complete object instances rather than tiny invalid fragments.The authors interpret this as evidence of high objectness for segmented instances.
- Qualitative Results: Very similar instances remain a failure case in the ScanNet qualitative results.These cases are marked by red circles in Figure 8.
B Experiments on S3DIS Dataset
On S3DIS, 3D-BoNet’s losses converge consistently, while its predicted boxes provide inclusive high-objectness regions that support point-level instance masks.
- Training: The proposed loss functions converge consistently across S3DIS Areas 1, 2, 3, 4, and 6.This jointly optimizes semantic segmentation, bounding box prediction, and point mask prediction end-to-end.
- Bounding boxes: Predicted boxes are generally inclusive rather than tight, but maintain high objectness for detected instances.The bounded points can then be segmented by the subsequent point-level mask branch.
- Instance masks: The mask visualization represents near-zero instance probabilities with black points and near-one probabilities with brighter points.
- Qualitative results: Figure 10 compares predicted red boxes with tight blue ground-truth boxes on S3DIS Area 2.The point clouds inside the blue boxes are provided to the framework for prediction.
- Qualitative results: Figure 11 presents qualitative results of the predicted instance masks.
C Experiments for Computation Efficiency
The computation-efficiency experiment compares four approaches on ScanNet(v2)’s 312-scene validation split, measuring time consumption under specified hardware and implementation settings.
- Experimental setup: The comparison evaluates SGPN, ASIS, GSPN, 3D-BoNet, and 3D-SIS using their released code on the ScanNet(v2) validation split.The split contains 312 scenes.
- Experimental setup: All methods run on a single Titan X GPU, while pre- and post-processing run on one i7 CPU core with a single thread.The implementations use TensorFlow 1.4 for SGPN, ASIS, GSPN, and 3D-BoNet, and PyTorch 0.4 for 3D-SIS.
- Results: The table reports time consumption in seconds for different approaches on the ScanNet(v2) validation split.
D Gradient Estimation for Hungarian Algorithm
For Hungarian matching, the method differentiates the Euclidean-distance loss through the assignment process; in implementation, the permutation-gradient term can be omitted for convergence.
- Gradient formulation: The predicted and ground-truth boxes produce an assignment cost matrix that the Hungarian algorithm converts into a permutation matrix.The Euclidean-distance component is the focus of the displayed derivative.
- Gradient formulation: The derivative contains a term for the permutation gradient with respect to the assignment cost matrix, which is zero nearly everywhere.The implementation sets this term to zero and still obtains convergence.
- Alternative estimation: A straight-through estimator can set the permutation derivative to 1 to speed convergence.This permits backpropagation of bounding-box alignment error and reinforcement of the assignment.
E Generalization to Unseen Scenes and Categories
A model trained on S3DIS is directly tested on ScanNet(v2), where it predicts high-quality instance labels despite unseen scenes and some unseen categories.
- Cross-dataset evaluation: The framework learns bounding boxes and point masks from raw point clouds without coupling them to semantic information.The authors test a model trained on S3DIS Areas 1, 2, 3, 4, and 6 directly on the ScanNet(v2) validation split.
- Results: The S3DIS-trained model predicts high-quality instance labels on ScanNet despite scenes and some object categories being unseen during training.
- Qualitative results: Figure 12 shows input point clouds, predicted instance labels, and ground truth for qualitative ScanNet results.