PaperlayerView exact revision on arXiv

Source-linked AI summary

SAM2Point: Segment Any 3D as Videos in Zero-shot and Promptable Manners

Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Chengzhuo Tong, Peng Gao, Chunyuan Li, Pheng-Ann Heng

arXiv:2408.16768v1cs.CVcs.AIcs.CL

TL;DR

Adapting SAM to 3D remains challenging, with projection-based methods adding complexity and losing spatial information, while existing approaches can limit prompting flexibility and domain transfer. SAM2POINT voxelizes point clouds into six orthogonal-direction videos for zero-shot SAM 2 processing and supports 3D points, bounding boxes, and masks. Demonstrations indicate robust generalization across objects, indoor and outdoor scenes, and raw sparse LiDAR.

  • Problem

    Adapting SAM to 3D remains challenging, with projection-based methods adding complexity and losing spatial information, while existing approaches can limit prompting flexibility and domain transfer.

    Source: §1 · p. 2

  • Method

    SAM2POINT voxelizes point clouds into six orthogonal-direction videos for zero-shot SAM 2 processing and supports 3D points, bounding boxes, and masks.

    Source: §2.1 · p. 3, §1 · p. 3

  • Results

    Demonstrations indicate robust generalization across objects, indoor and outdoor scenes, and raw sparse LiDAR.

    Source: §1 · p. 3, §4 · p. 5

  • Takeaways & Limitations

    SAM2POINT offers a preliminary starting point for research on promptable 3D segmentation and adapting SAM 2 to 3D understanding.

    Source: §5 · p. 10

    As a preliminary exploration, SAM2POINT does not target the performance limit.

    Source: §1 · p. 2

Abstract

from arXiv · show

We introduce SAM2Point, a preliminary exploration adapting Segment Anything Model 2 (SAM 2) for zero-shot and promptable 3D segmentation. SAM2Point interprets any 3D data as a series of multi-directional videos, and leverages SAM 2 for 3D-space segmentation, without further training or 2D-3D projection. Our framework supports various prompt types, including 3D points, boxes, and masks, and can generalize across diverse scenarios, such as 3D objects, indoor scenes, outdoor environments, and raw sparse LiDAR. Demonstrations on multiple 3D datasets, e.g., Objaverse, S3DIS, ScanNet, Semantic3D, and KITTI, highlight the robust generalization capabilities of SAM2Point. To our best knowledge, we present the most faithful implementation of SAM in 3D, which may serve as a starting point for future research in promptable 3D segmentation. Online Demo: https://huggingface.co/spaces/ZiyuG/SAM2Point . Code: https://github.com/ZiyuGuo99/SAM2Point .

1 INTRODUCTION

Adapting SAM to 3D remains challenging: projection-based methods can add processing complexity, lose spatial information, restrict interactive prompting, and transfer poorly across domains. SAM2POINT instead uses voxelized 3D data as multi-directional videos for zero-shot, promptable segmentation across varied scenarios.

1 INTRODUCTION

  • Existing 2D–3D projection approaches add processing complexity, degrade fine-grained spatial information, limit interactive prompting, and can be tied to specific domains.Training a 3D promptable network from scratch avoids projection but demands substantial training and data resources.Source: §1 · p. 2
  • SAM2POINT adapts SAM 2 for efficient, projection-free, promptable, zero-shot 3D segmentation as an initial exploration rather than an effort to maximize performance.Source: §1 · p. 2
  • Voxelization represents 3D data in a video-like format that retains spatial information and lets SAM 2 perform zero-shot segmentation without additional training or 2D–3D projection.Source: §1 · pp. 2–3
  • Users can prompt segmentation with 3D points, bounding boxes, or masks, whose multi-directional results are integrated into a 3D prediction.Source: §1 · p. 3
  • SAM2POINT demonstrates generalization across single objects, indoor and outdoor scenes, and raw LiDAR with varying point-cloud distributions.Source: §1 · p. 3

2 SAM2POINT

SAM2POINT voxelizes point clouds and treats the resulting grid as multi-directional videos for SAM 2. It supports point, box, and mask prompts, with demonstrations spanning objects, indoor and outdoor scenes, and raw LiDAR.

2 SAM2POINT

  • SAM2POINT converts object- or scene-level point clouds into voxels, then uses multi-directional video representations to make 3D data processable by SAM 2.Source: §2.1 · p. 3

2.1 3D DATA AS VIDEOS

  • Voxelization assigns each voxel the RGB value of the nearest point to its center, producing a w × h × l × 3 grid that resembles video data while remaining isotropic across spatial dimensions.Source: §2.1 · p. 3

2.2 PROMPTABLE SEGMENTATION

  • Point prompts anchor three orthogonal sections, box prompts use the box center and projected 2D boxes, and mask prompts use the mask’s center of gravity and section intersections.A 3D mask can also refine a previously predicted mask; Figure 2 illustrates dividing prompted voxel space into six directional videos and integrating their results.Source: §2.2 · pp. 3–4, Figure 2 · p. 4

2.3 ANY 3D SCENARIOS

  • The framework is presented as generalizing zero-shot from objects to scenes and from indoor to outdoor environments.Source: §2.3 · p. 4
  • The demonstrations cover object part segmentation, indoor scenes, and large-scale outdoor scenes, while raw KITTI LiDAR uses voxel RGB values set from LiDAR intensity.Raw LiDAR is sparse and lacks RGB information, so segmentation relies on geometric cues and inferred missing semantics.Source: §2.3 · p. 4

3 DISCUSSION AND INSIGHT

The discussion frames transferring 2D foundation models to 3D as a response to 3D data scarcity, while noting trade-offs among projection, training, and more complex approaches. SAM2POINT’s voxel-video representation is proposed as a possible balance of performance and efficiency, with further experiments needed; the authors also outline possible roles across 3D applications.

3 DISCUSSION AND INSIGHT

  • The 3D field’s scarcity of large-scale, high-quality data has motivated transferring pretrained 2D models, but bridging the modalities remains challenging.Source: §3.1 · p. 5

3.1 HOW TO ADAPT 2D FOUNDATION MODELS TO 3D?

  • Voxelizing 3D data as videos may balance performance and efficiency by preserving spatial geometry through a simple transformation into a grid that 2D models can process directly.The authors state that further experiments are needed to validate this observation.Source: §3.1 · p. 5

3.2 WHAT IS THE POTENTIAL OF SAM2POINT IN 3D DOMAINS?

  • SAM2POINT is proposed as a potential unified backbone, annotation tool, joint 2D–3D–video embedding space, and 3D encoder for language models across objects, scenes, and raw LiDAR.The discussion links automatic diverse-scenario labels to mitigating data scarcity and promptable features to instruction-following capabilities in 3D LLMs.Source: §3.2 · p. 5

4 DEMOS

The demos illustrate SAM2POINT segmentation across 3D objects, indoor and outdoor scenes, and raw LiDAR. They span Objaverse, S3DIS, ScanNet, Semantic3D, and KITTI.

4 DEMOS

  • The figures demonstrate SAM2POINT segmentation on Objaverse objects, S3DIS and ScanNet indoor scenes, Semantic3D outdoor scenes, and KITTI raw LiDAR.The prompts and segmentation results are highlighted in red and green, respectively.Source: §4 · p. 5, Figure 3 · p. 6, Figure 4 · p. 7
  • The Objaverse figure shows 3D object segmentation, with prompts in red and segmentation results in green.Objaverse contains a wide array of object categories.Source: Figure 3 · p. 6
  • The S3DIS and ScanNet figures show indoor scene segmentation, with prompts in red and segmentation results in green.Source: Figure 4 · p. 7, Figure 5 · p. 8
  • The Semantic3D figure shows outdoor scene segmentation, with prompts in red and segmentation results in green.Source: Figure 6 · p. 9
  • The KITTI figure shows raw LiDAR segmentation, with prompts in red and segmentation results in green.Source: Figure 7 · p. 10

5 CONCLUSION

SAM2POINT uses SAM 2 in a zero-shot, promptable 3D segmentation framework by representing 3D data as multi-directional videos. It supports point, box, and mask prompts across varied 3D scenarios and is proposed as a foundational baseline for further research.

5 CONCLUSION

  • SAM2POINT leverages SAM 2 for zero-shot, promptable 3D segmentation by representing 3D data as multi-directional videos.Source: §5 · p. 10
  • The framework supports 3D point, box, and mask prompts across objects, indoor and outdoor scenes, and raw sparse LiDAR.Source: §5 · p. 10
  • The authors present SAM2POINT as a preliminary investigation and a potential foundational baseline for promptable 3D segmentation research.Source: §5 · p. 10

Related papers

Loading 2408.16768v1…