Source-linked AI summary
OneFormer3D: One Transformer for Unified Point Cloud Segmentation
Maxim Kolodiazhnyi, Anna Vorontsova, Anton Konushin, Danila Rukhovich
TL;DR
Existing 3D segmentation tasks are typically handled by separate models, limiting use of their shared structure and increasing training costs. OneFormer3D unifies semantic, instance, and panoptic segmentation with a single transformer-based model trained once, achieving state-of-the-art results across all three tasks and multiple indoor benchmarks.
Problem
Existing 3D semantic, instance, and panoptic segmentation methods typically use distinct architectures and separately trained models.
Method
OneFormer3D jointly trains semantic and instance queries in a transformer decoder, using unified kernels plus query selection and disentangled matching.
Results
OneFormer3D achieves state-of-the-art results in 3D semantic, instance, and panoptic segmentation across ScanNet, ScanNet200, and S3DIS.
Takeaways & Limitations
A single model trained once can solve all three 3D segmentation tasks while outperforming approaches trained separately for each task.
Abstract
from arXiv · showhide
Semantic, instance, and panoptic segmentation of 3D point clouds have been addressed using task-specific models of distinct design. Thereby, the similarity of all segmentation tasks and the implicit relationship between them have not been utilized effectively. This paper presents a unified, simple, and effective model addressing all these tasks jointly. The model, named OneFormer3D, performs instance and semantic segmentation consistently, using a group of learnable kernels, where each kernel is responsible for generating a mask for either an instance or a semantic category. These kernels are trained with a transformer-based decoder with unified instance and semantic queries passed as an input. Such a design enables training a model end-to-end in a single run, so that it achieves top performance on all three segmentation tasks simultaneously. Specifically, our OneFormer3D ranks 1st and sets a new state-of-the-art (+2.1 mAP50) in the ScanNet test leaderboard. We also demonstrate the state-of-the-art results in semantic, instance, and panoptic segmentation of ScanNet (+21 PQ), ScanNet200 (+3.8 mAP50), and S3DIS (+0.8 mIoU) datasets.
1. Introduction
OneFormer3D addresses the fragmentation of 3D segmentation by jointly solving semantic, instance, and panoptic segmentation with one model trained once. It combines unified queries and new matching strategies to achieve state-of-the-art performance across the three tasks.
- 3D point cloud segmentation comprises semantic, instance, and panoptic formulations that group points into categories, individual objects, or objects plus background regions.
- Existing 3D methods typically use different architectures for semantic, instance, and panoptic segmentation.
- Joint training can avoid the three-times-larger time and memory footprint caused by separately training task-specific models.
- OneFormer3D adds semantic queries alongside instance queries in a transformer decoder and introduces query selection and efficient matching strategies.
- OneFormer3D is trained once and outperforms task-specific 3D semantic, instance, and panoptic segmentation methods.
2. Related Work
Related work develops separate approaches for 3D semantic, instance, and panoptic segmentation, while unified 2D methods motivate a single architecture trained jointly across tasks. OneFormer3D extends this unified strategy to 3D point clouds.
- 3D Semantic Segmentation: 3D semantic segmentation commonly uses point- or voxel-based U-Net-like networks, including sparse convolutional models.
- 3D Instance Segmentation: 3D instance segmentation typically combines semantic segmentation with feature aggregation, proposals, grouping, or transformer decoders.
- 3D Panoptic Segmentation: 3D panoptic methods are few and generally lift 2D image masks into 3D before aggregating them point-wise.
- Unified 2D Segmentation: Unified 2D methods still often require task-specific training for best performance, whereas OneFormer achieves joint state-of-the-art segmentation with one model.
- Unified 2D Segmentation: OneFormer3D follows this unified 2D direction for 3D point cloud segmentation.
3. Proposed Method
OneFormer3D uses a sparse 3D backbone, flexible pooling, and a transformer decoder with semantic and instance queries that produce corresponding masks. Its query selection and disentangled matching improve training efficiency and instance segmentation quality.
- Framework: The framework inherits SPFormer’s pipeline while adding unified semantic and instance queries and other improvements.
- Backbone and Pooling: A sparse 3D U-Net extracts point-wise features before flexible pooling produces superpoint or voxel features for transformer processing.
- Query Decoder: The query decoder transforms semantic and instance queries into kernels that generate semantic and instance masks from superpoint features.
- Backbone and Pooling: Flexible pooling reduces millions of input points to hundreds of superpoints or thousands of voxels, lowering subsequent computational cost.
- Query Selection: Query selection initializes queries from pooled backbone features and randomly retains half for extra training augmentation, while inference keeps all queries.
- Disentangled Matching: Disentangled matching replaces Hungarian matching with direct proposal selection, reducing complexity from O(K_ins^3) to O(K_ins).
- Inference: During inference, predicted kernels produce masks whose classification and mask scores are combined for instance ranking and matrix-NMS re-ranking.
4. Experiments
Experiments evaluate OneFormer3D across indoor benchmarks, segmentation tasks, object detection, and targeted ablations. The unified model achieves state-of-the-art results while joint training, pretraining, query selection, and matching each affect performance or efficiency.
- Evaluation setup: Experiments use ScanNet, ScanNet200, and S3DIS, with mIoU, mAP, mAP50, mAP25, PQ, mean precision, and mean recall as evaluation metrics.ScanNet results include validation and hidden test splits; instance segmentation uses IoU thresholds from 50% to 95% for mAP.
- Comparison to prior work: 80.1 mAP50 on the ScanNet hidden test leaderboard ranks OneFormer3D first, improving 2.1 mAP50 over Mask3D.It also reaches 89.6 mAP25, 1.2 mIoU above PointTransformerV2, and 71.2 PQ, 21.0 PQ above TUPPer-Map.
- Comparison to prior work: OneFormer3D sets state-of-the-art results on S3DIS across segmentation tasks and improves 6-fold instance segmentation by 1.5 mAP50 and 1.2 mAP over Mask3D.It also outperforms TD3D and Mask3D in mPrec50 and mRec50 in both evaluated S3DIS benchmarks.
- Comparison to prior work: On ScanNet200, OneFormer3D achieves at least 3-point gains in mAP25, mAP50, and mAP, while providing panoptic scores where none had previously been reported.These scores establish a basis for future ScanNet200 and S3DIS panoptic research.
- Ablation studies: The query-selection and disentangled-matching combination improves mAP25, mAP50, and mAP by at least 1.3, whereas query selection alone does not improve baseline matching.The ablation evaluates instance segmentation because the two components do not affect semantic segmentation.
- Ablation studies: ScanNet and Structured3D pretraining together add at least 11.5 to both mAP50 and mIoU, while removing superpoint clustering yields at least 2x speed-up with gains of at least 1.2 in both metrics.Joint training reduces training time threefold; semantic accuracy increases by as much as 3.4 mIoU while instance accuracy remains unchanged.
5. Conclusion
OneFormer3D unifies instance, semantic, and panoptic 3D point-cloud segmentation in a transformer-based framework. Across ScanNet, ScanNet200, and S3DIS, it establishes state-of-the-art performance in all three tasks.
- OneFormer3D unifies instance, semantic, and panoptic 3D point-cloud segmentation.
- The framework uses novel query selection and disentangled matching strategies to address weaknesses in transformer-based 3D instance segmentation.
- OneFormer3D establishes state-of-the-art results across all three 3D segmentation tasks on ScanNet, ScanNet200, and S3DIS.
A. Per-category Scores
The analysis reports per-category segmentation performance to complement averaged scores and expose category-specific behavior. OneFormer3D is reported to segment every category more precisely than competing methods on the ScanNet test split.
- Per-category scores provide a more complete view because averaged scores can obscure performance issues in imbalanced category distributions.
- The evaluation reports per-class panoptic PQ on ScanNet and S3DIS, plus per-category instance mAP on ScanNet.
- OneFormer3D segments every reported category more precisely than competitors on the ScanNet test split.
B. Performance
OneFormer3D adds semantic queries and a different instance-query initialization to SPFormer while preserving nearly the same inference speed. Its measured inference-time increase is below 3%.
- Less than 3% additional inference time results from semantic queries and a different instance-query initialization compared with SPFormer.
- The profiler decomposes runtime into superpoint creation, sparse 3D CNN feature extraction, pooling, and query decoding.
- OneFormer3D is reported to be on par with SPFormer, the fastest among the profiled approaches.
C. Qualitative Results
Qualitative visualizations compare original point clouds with ground-truth and predicted instance, semantic, and panoptic masks. They are provided for ScanNet and S3DIS, alongside per-category and runtime tables.
- The paper provides these visualizations to relate segmentation scores to actual segmentation quality.
- Additional reported analyses include per-class panoptic PQ, per-class instance mAP50, and inference-time comparisons.
- ScanNet visualizations show an input point cloud, ground-truth panoptic mask, and predicted instance, semantic, and panoptic masks.
- S3DIS Area-5 visualizations use the same input, ground-truth, and three predicted-mask views.