Source-linked AI summary
3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nießner
TL;DR
3D-MPA addresses proposal processing for 3D instance segmentation by combining object-centric voting with proposal-level feature learning and aggregation. It replaces NMS with clustering of learned aggregation features, retaining multiple proposals and reporting improved performance across 3D detection and semantic instance segmentation benchmarks.
Problem
3D proposal processing must balance top-down methods’ sensitivity to incorrect anchors against bottom-up methods’ costly, parameter-dependent clustering and O(N^2) pairwise relationships.
Method
3D-MPA has points vote for object centers, groups voted points into proposals, models proposal interactions with a graph convolutional network, and clusters learned aggregation features instead of applying NMS.
Results
3D-MPA improves over NMS and outperforms previous state-of-the-art methods on 3D object detection and semantic instance segmentation on ScanNetV2 and S3DIS.
Takeaways & Limitations
Multi-proposal aggregation is a promising direction for object detection, particularly in the 3D domain.
Takeaways & Limitations
The paper identifies combining detection with tracking in semi-dynamic sequences as a future avenue rather than an addressed capability.
Abstract
from arXiv · showhide
We present 3D-MPA, a method for instance segmentation on 3D point clouds. Given an input point cloud, we propose an object-centric approach where each point votes for its object center. We sample object proposals from the predicted object centers. Then, we learn proposal features from grouped point features that voted for the same object center. A graph convolutional network introduces inter-proposal relations, providing higher-level feature learning in addition to the lower-level point features. Each proposal comprises a semantic label, a set of associated points over which we define a foreground-background mask, an objectness score and aggregation features. Previous works usually perform non-maximum-suppression (NMS) over proposals to obtain the final object detections or semantic instances. However, NMS can discard potentially correct predictions. Instead, our approach keeps all proposals and groups them together based on the learned aggregation features. We show that grouping proposals improves over NMS and outperforms previous state-of-the-art methods on the tasks of 3D object detection and semantic instance segmentation on the ScanNetV2 benchmark and the S3DIS dataset.
1. Introduction
3D-MPA combines object-centric voting with proposal-level feature learning and clustering to avoid discarding potentially correct proposals through NMS. Its hybrid design targets limitations of top-down and bottom-up 3D detection approaches.
- Top-down methods refine many rough box proposals but have little tolerance for wrongly detected anchors.
- Bottom-up methods handle outliers effectively but require manually tuned clustering parameters and O(N^2) pairwise inference relationships.
- 3D-MPA generates object-centric proposals by having points vote for object centers, then groups those proposals using learned higher-level features instead of NMS.
- Multi-proposal aggregation keeps multiple proposals per object, tolerates outlier proposals, and remains efficient because proposal counts are far smaller than raw point counts.
- The method uses a graph convolutional network to model higher-order interactions between neighboring proposal features alongside lower-level point features.
- The method reports significantly improved scores over NMS when producing final object detections and semantic instances.
2. Related Work
Prior work includes proposal-based and proposal-free approaches for 3D detection and instance segmentation, alongside diverse point- and voxel-based 3D learning architectures. Figure 2 situates 3D-MPA as a three-stage proposal-generation, consolidation, and object-generation pipeline.
- Proposal-based 2D detection and instance-segmentation methods such as Faster R-CNN and Mask R-CNN have been extended to 3D settings.
- 3D research also includes proposal-free methods and advances in semantic segmentation, instance segmentation, detection, relocalization, flow estimation, scenegraph reconstruction, and scene over-segmentation.
- Point-based architectures operate directly on unstructured point sets, whereas voxel-based approaches use volumetric representations.
- 3D-MPA’s architecture comprises proposal generation, proposal consolidation, and object generation stages.
3. Method
3D-MPA generates object-centric proposals from per-point center votes, consolidates them with graph-based proposal features, and aggregates proposals instead of applying NMS. Its proposal pipeline predicts semantic labels, objectness, aggregation features, and point-level masks for final instances.
- Proposal Generation: A fixed number K of proposals is sampled from predicted centers, each containing a position, feature vector, and associated points.Associated points voted for centers within radius r of the sampled proposal position; random sampling performed better than FPS in the described comparison.
- Proposal Generation: The sparse volumetric backbone jointly predicts per-point semantic labels and object-center offsets, using foreground supervision to ignore background votes.The combined point loss uses semantic cross-entropy and Huber center regression, with λ set to 0.1.
- Proposal Consolidation: A proposal graph connects positions within 2 m, and a GCN models higher-order interactions to produce refined proposal features.The method reports its best results with l = 10 graph-convolutional layers, while also operating without refinement.
- Object Generation: The object generator predicts each proposal’s semantic class, aggregation features, objectness score, and foreground-background mask over its associated points.Positive proposals receive supervision for these outputs, whereas negative proposals are ignored for semantic, aggregation, and mask prediction.
- Object Generation: Instead of NMS, proposals with similar aggregation features are clustered and merged, with final object points formed by the union of their foreground masks.With K ≈500 proposals versus N ≈10^6 points, aggregation takes approximately 8 ms and is faster than clustering full point clouds.
- Object Generation: Aggregation features can be geometric or embedding-based, with geometric features encoding refined center offsets and object-radius estimates.The geometric feature vector has dimension E = 4; the embedding loss pulls same-instance features together and separates different instances.
4. Experiments
Experiments evaluate 3D-MPA on ScanNetV2 and S3DIS for detection and instance segmentation, with ablations isolating proposal grouping, aggregation features, and graph convolution. The method consistently outperforms prior approaches, while qualitative results show strong boundary and size handling alongside plausible instance-separation failures.
- Qualitative results: Qualitative results show segmentation across substantially different object sizes and clear object boundaries, while two failures incorrectly split single instances but remain plausible against the input.Figure 3 also visualizes center votes, sampled proposals, grouped point features, and proposal groups.
- Comparison with state-of-the-art methods: +5.8 mAP@25% and +15.7 mAP@50% over previous methods on ScanNetV2 3D object detection.Scores use tight axis-aligned bounding boxes around predicted object point-masks and the evaluation methodology from VoteNet.
- Comparison with state-of-the-art methods: +17.8 mAR@50% on S3DIS Area 5 and +16.5 mAR@50% under six-fold cross-validation, while simultaneously achieving higher precision.The reported metric is computed on per-point instance masks at a 50% IoU threshold.
- Comparison with state-of-the-art methods: +18.1 mAP@50% and +17.0 mAP@25% over previous methods on ScanNetV2 instance segmentation.The method outperforms all other methods at mAP@50% across object classes and trails only on chair and sofa at mAP@25%.
- Ablation study: +4.9 mAP results from replacing NMS with proposal aggregation in the ScanNetV2 validation ablation.The comparison uses a baseline with NMS at a 25% IoU suppression threshold versus naive grouping by proposal positions.
- Ablation study: +5.4 mAP from learned aggregation features over position grouping, +1.3 mAP from the GCN, and +11.6 mAP overall over the unchanged baseline.Geometric features outperform embedding features by +1.1 mAP in the reported ablation.
5. Conclusion
3D-MPA combines object-centric voting, proposal graphs, and multi-proposal aggregation to achieve strong 3D instance segmentation and detection results. The authors identify extending aggregation to detection and tracking in semi-dynamic sequences as future work.
- 3D-MPA combines object-centric voting, graph-based proposal interactions, and proposal aggregation rather than NMS.Each object may receive multiple proposals, which are aggregated using graph relationships and proposal-feature similarities.
- The method achieves state-of-the-art instance segmentation and object detection results on ScanNetV2 and S3DIS.
- The largest improvement originates from the multi-proposal aggregation strategy, while graph convolutions also improve evaluation scores.
- Future work includes combining detection with tracking in semi-dynamic sequences and accumulating proposals across a 4D time-space axis.