Source-linked AI summary
SAM3D: Segment Anything in 3D Scenes
Yunhan Yang, Xiaoyang Wu, Tong He, Hengshuang Zhao, Xihui Liu
TL;DR
3D segmentation needs fine-grained point-level masks, while SAM provides strong segmentation for RGB images. SAM3D projects SAM masks into 3D point clouds and merges adjacent-frame results into whole-scene masks, with qualitative ScanNet results showing accurate and fine-grained segmentation without training or finetuning SAM.
Problem
3D segmentation seeks point-level labels for scene point clouds, while visual perception models have shown strong fine-grained segmentation from large-scale RGB image data.
Method
SAM3D applies SAM to posed RGB frames, projects the 2D masks into 3D, and iteratively merges adjacent-frame masks using bidirectional bottom-up merging.
Results
SAM3D obtains accurate and fine-grained 3D scene masks, including detailed wall paintings not annotated in the ground truth, without training or finetuning SAM.
Takeaways & Limitations
SAM segmentation results provide a strong baseline and initial state for 3D segmentation.
Abstract
from arXiv · showhide
In this work, we propose SAM3D, a novel framework that is able to predict masks in 3D point clouds by leveraging the Segment-Anything Model (SAM) in RGB images without further training or finetuning. For a point cloud of a 3D scene with posed RGB images, we first predict segmentation masks of RGB images with SAM, and then project the 2D masks into the 3D points. Later, we merge the 3D masks iteratively with a bottom-up merging approach. At each step, we merge the point cloud masks of two adjacent frames with the bidirectional merging approach. In this way, the 3D masks predicted from different frames are gradually merged into the 3D masks of the whole 3D scene. Finally, we can optionally ensemble the result from our SAM3D with the over-segmentation results based on the geometric information of the 3D scenes. Our approach is experimented with ScanNet dataset and qualitative results demonstrate that our SAM3D achieves reasonable and fine-grained 3D segmentation results without any training or finetuning of SAM.
1. Introduction and Background
3D segmentation combines point-cloud geometry with visual information from RGB images, where SAM offers fine-grained image masks. SAM3D projects these masks into 3D and iteratively merges adjacent-frame results to form whole-scene masks without training or finetuning SAM.
- SAM3D addresses fine-grained 3D scene segmentation by leveraging SAM's segmentation results from RGB images.
- The framework applies SAM to RGB frames, projects the resulting 2D masks into 3D, and merges partial-scene masks iteratively.
- SAM3D can optionally ensemble its 3D masks with over-segmentation masks to refine the final scene segmentation.
- Adjacent point-cloud frames are merged with a bidirectional approach within a bottom-up process that expands masks toward the whole scene.
- The approach is demonstrated qualitatively on ScanNet scenes and does not require training or finetuning SAM.
2. Method
SAM3D lifts SAM’s RGB-image masks into 3D point clouds, then combines partial-frame predictions through bidirectional and bottom-up merging. It can optionally fuse these semantic masks with geometry-based over-segmentation results.
- 2.1. Single-frame 3D masks from SAM: SAM masks from posed RGB images are projected into corresponding 3D point-cloud frames to produce single-frame 3D masks.The projection uses RGB-D depth and camera calibration, followed by grid pooling to downsample the masks.
- 2.2. Bidirectional Merging between Two Point Clouds: Bidirectional merging matches points across adjacent frames and merges masks when their corresponding regions sufficiently overlap.The process is applied symmetrically in both directions between the two point clouds.
- 2.3. Bottom-up Merging of the Point Clouds in the Whole Scene: Bottom-up merging repeatedly combines adjacent local point clouds until all frame-level masks form a single whole-scene point cloud.Adjacent pairs are merged at successive levels, requiring ⌈log2 n⌉ rounds, with grid pooling after each step.
- 2.3. Bottom-up Merging of the Point Clouds in the Whole Scene: The bidirectional merging operation ⊕ combines adjacent point clouds at each bottom-up level.At step t, X_t^{2i} and X_t^{2i+1} are merged into X_{t+1}^i.
- 2.4. Ensembling with Over-segmentation: SAM3D masks can be ensembled with normal-based geometric over-segmentation to combine RGB semantics and edges with 3D geometric information.The paper states that the ensembled masks demonstrate higher segmentation quality.
3. Results
On ScanNet 3D scenes, SAM3D produces accurate, fine-grained qualitative segmentation masks. Its masks can reveal details absent from both ground-truth annotations and geometric over-segmentation.
- 3. Results: SAM3D achieves accurate and fine-grained segmentation results on ScanNet 3D scenes.The results are presented qualitatively in Figure 5.
- 3. Results: SAM3D segments paintings or pictures on walls that are absent from the ground-truth masks and over-segmentation results.The paper gives wall paintings in the first row of Figure 5 as an example.
4. Conclusion and Discussion
SAM3D projects SAM segmentation results from posed RGB images into 3D point clouds and merges masks across frames to segment whole scenes. The resulting masks are accurate and fine-grained, supporting SAM as a strong baseline and initialization for 3D segmentation.
- SAM3D projects SAM-predicted 2D masks into 3D point clouds and merges masks from different frames to obtain whole-scene segmentation.
- SAM3D produces accurate and fine-grained masks for 3D scenes.
- SAM segmentation results can provide a strong baseline and initial state for 3D segmentation.