PaperlayerView exact revision on arXiv

Source-linked AI summary

Point-SAM: Promptable 3D Segmentation Model for Point Clouds

Yuchen Zhou, Jiayuan Gu, Tung Yen Chiang, Fanbo Xiang, Hao Su

arXiv:2406.17741v2cs.CVcs.AI

TL;DR

3D segmentation lacks unified representations and architectures, scalable models, and diverse labeled data. Point-SAM combines a promptable point-cloud model with a SAM-assisted pseudo-labeling engine. The authors report improved out-of-distribution performance from pseudo-label training and better performance than AGILE3D on KITTI360 and PartNet-Mobility, though worse performance on S3DIS.

  • Problem

    3D segmentation faces heterogeneous representations and architectures, higher computational costs, and scarce labels with diverse masks.

    Source: §1 · pp. 1–2

  • Method

    Point-SAM uses a transformer-based point-cloud model with point and mask prompts, trained with heterogeneous data and SAM-assisted pseudo-labels.

    Source: §1 · p. 2, §3 · pp. 3–4

  • Results

    Point-SAM outperforms AGILE3D on the OOD datasets KITTI360 and PartNet-Mobility, but performs worse on the in-domain dataset S3DIS.

    Source: §5.2 · p. 9

  • Takeaways & Limitations

    The authors report that SAM-assisted pseudo-labels improve zero-shot transferability and substantially boost overall performance.

    Source: §1 · p. 2, §5.2 · p. 10

    The approach depends on SAM pseudo-label quality and variability, which may not capture complex 3D structure; 3D data scale and diversity also lag behind 2D.

    Source: §6 · p. 10

Abstract

from arXiv · show

The development of 2D foundation models for image segmentation has been significantly advanced by the Segment Anything Model (SAM). However, achieving similar success in 3D models remains a challenge due to issues such as non-unified data formats, poor model scalability, and the scarcity of labeled data with diverse masks. To this end, we propose a 3D promptable segmentation model Point-SAM, focusing on point clouds. We employ an efficient transformer-based architecture tailored for point clouds, extending SAM to the 3D domain. We then distill the rich knowledge from 2D SAM for Point-SAM training by introducing a data engine to generate part-level and object-level pseudo-labels at scale from 2D SAM. Our model outperforms state-of-the-art 3D segmentation models on several indoor and outdoor benchmarks and demonstrates a variety of applications, such as interactive 3D annotation and zero-shot 3D instance proposal. Codes and demo can be found at https://github.com/zyc00/Point-SAM.

1 INTRODUCTION

The introduction argues that 3D segmentation lacks the unified representations, scalable architectures, and diverse labels available to 2D segmentation. Point-SAM addresses these gaps with a promptable point-cloud model and a SAM-assisted pseudo-labeling engine.

1 INTRODUCTION

  • Existing 2D-to-3D lifting is constrained by image quality and view selection, can produce inconsistent masks, and cannot label internal structures.These limits motivate native 3D segmentation.Source: §1 · p. 1
  • 3D foundation models face heterogeneous representations and architectures, higher computational costs, and scarce high-quality labels with diverse masks.The largest part-labeled dataset cited contains about 26,671 shapes and 573,585 part instances.Source: §1 · pp. 1–2
  • Point-SAM is a promptable segmentation model for point clouds, which the authors select because other representations can be converted to point clouds and real-world data is often captured that way.It encodes point clouds and point or mask prompts to predict segmentation masks.Source: §1 · p. 2
  • The model is trained on heterogeneous datasets with part- and object-level annotations, while a SAM-assisted data engine generates pseudo-labels to expand mask diversity.The authors report that these pseudo-labels significantly improve zero-shot transferability.Source: §1 · p. 2
  • The authors report unified processing of point clouds from varied sources and zero-shot transfer to unseen point-cloud distributions and new tasks.They position Point-SAM as a step toward a 3D foundation model.Source: §1 · p. 2

2 RELATED WORK

Related work adapts 2D foundation models to 3D by merging multiview results or distilling their knowledge into native 3D models. Point-SAM is situated among work on 3D foundation models and interactive segmentation, with a focus on both object- and part-level point-cloud tasks.

2 RELATED WORK

  • The review identifies 2D foundation models including CLIP, GLIP, and SAM as resources used to address scarce 3D segmentation labels.Existing 3D approaches either merge predictions across views or distill 2D-model knowledge into 3D models.Source: §2 · pp. 2–3
  • Multiview lifting methods use 2D models to segment projected views, but depend on rendering quality and require difficult post-processing to integrate results consistently.Methods that distill SAM into scene-level point clouds still have difficulty labeling internal structures.Source: §2 · p. 3
  • Other 3D foundation-model work covers point-cloud pretraining, multimodal representations, and 3D-language or embodied tasks, while this paper concentrates on segmentation.The related work characterizes open-world segmentation methods as primarily scene-level and trained on relatively small datasets.Source: §2 · p. 3
  • Prior interactive point-cloud segmentation methods are described as ScanNet-only, whereas Point-SAM is designed for object- and part-level segmentation across CAD models and real scans.The authors also contrast its feed-forward, seconds-scale responses with implicit-representation methods that require additional optimization.Source: §2 · p. 3

3 POINT-SAM

Point-SAM combines a point-cloud encoder, prompt encoder, and mask decoder to produce masks from point-cloud and prompt inputs. Its Voronoi tokenizer groups points into patch tokens, while transformer components process cloud and prompt features.

3 POINT-SAM

  • Point-SAM takes a point cloud and prompts, embeds the cloud, and uses a prompt encoder and mask decoder in a three-component architecture.The architecture is designed around point-cloud computation efficiency, scalability, and irregularity.Source: §3 · p. 3, Figure 2 · p. 4
  • The Voronoi tokenizer assigns points to their nearest center, extracts pointwise features from relative positions, and scatter-max pools them into patch tokens.This avoids the K-nearest-neighbor grouping described for the conventional tokenizer.Source: §3 · p. 4, Figure 2 · p. 4
  • The prompt encoder represents point prompts with positional and learned label embeddings, and incorporates mask-prompt logits into the point-cloud embedding.Mask prompts are typically derived from previous predictions.Source: §3 · p. 4
  • A two-way transformer updates prompt and point-cloud features, which are upsampled and passed to a dynamic classifier to predict a mask.The decoder uses three-neighbor inverse-distance interpolation to restore point resolution.Source: §3 · p. 4
  • Multiple masks can be produced for a single point prompt only when no mask prompt is supplied, with an additional token predicting each mask’s IoU score.The training description uses a hindsight loss for multiple outputs and supervises predicted IoU scores.Source: §3 · pp. 4–5
  • At inference, the model adjusts patch count and size to handle point sets larger than those used in training.For inputs above 32,768 points, the default settings are 2,048 patches and patch size 512.Source: §3 · p. 5

4 TRAINING DATASETS

Point-SAM training combines synthetic object and part datasets with scene-level ScanNet data. To expand mask diversity beyond existing labels, the authors generate ShapeNet pseudo-labels through iterative interaction between SAM and Point-SAM across rendered views.

4 TRAINING DATASETS

  • Training combines PartNet, PartNet-Mobility, and Fusion360 synthetic data with ScanNet200 scenes split into ScanNet-Block blocks.PartNet and Fusion360 use 32,768 points sampled from mesh faces; PartNet-Mobility uses fused rendered RGB-D clouds, and ScanNet blocks are 3 m × 3 m with 1.5 m stride.Source: §4 · p. 5
  • The data engine addresses limited mask diversity by generating pseudo-labels for large 3D datasets such as ShapeNet that lack part-level segmentation labels.Existing segmentation datasets often assign each point to a single instance.Source: §4 · p. 5
  • For each of six rendered views, SAM generates 2D proposals that are matched to 3D masks by lifting prompts to the fused point cloud and iteratively updating Point-SAM predictions.The process continues until the 2D proposal and projected 3D mask exceed an IoU threshold, then uses additional views to refine the proposal with SAM priors.Source: §4 · pp. 5–6, Figure 3 · p. 6
  • The pipeline generates pseudo-labels for 20,000 ShapeNet shapes, with an average of 17 masks per shape.The paper describes this mask diversity as comparable to PartNet.Source: §4 · p. 6

5 EXPERIMENTS

The experiments evaluate zero-shot point-prompted segmentation across indoor and outdoor scenes and part-level datasets, then test dataset scaling, model design, and data-engine choices. Point-SAM shows stronger transfer and prompt efficiency, while the ablations identify training-data diversity and 2D SAM pseudo-labels as important factors.

5.1 ZERO-SHOT POINT-PROMPTED SEGMENTATION

  • The evaluation covers part-level PartNet-Mobility and ScanObjectNN, plus scene-level S3DIS and KITTI-360, using simulated point clicks and instance-averaged IoU@k.PartNet-Mobility evaluation holds out three categories; scene-level scans are cropped around each instance.Source: §5.1 · pp. 6–7, Table 2 · p. 7
  • Point-SAM outperforms MV-SAM, especially with few prompts, and surpasses AGILE3D across datasets, particularly on OOD PartNet-Mobility and KITTI360.Table 3 reports quantitative zero-shot results; Figure 4 provides qualitative comparisons, including better quality with a single prompt and faster convergence. MV-SAM samples k prompts per view for IoU@k.Source: §5.1 · p. 7, Table 3 · p. 8, Figure 4 · p. 9
  • The Voronoi tokenizer raises FPS by 163.4% (5.2 →13.7 shape/sec) and lowers GPU memory by 18.4% (3890 →3172 MB) versus KNN, with comparable performance.Efficiency was measured on one Nvidia RTX-4090 GPU using KITTI360 point clouds and 10 prompt points per cloud.Source: §5.1 · p. 7, Table 3 · p. 8

5.2 ABLATIONS

  • Adding larger and more diverse training datasets improves zero-shot OOD performance; ShapeNet especially helps PartNet-Mobility, while ScanNet has a stronger effect on S3DIS.On held-out PartNet-Mobility, PartNet+ShapeNet outperforms PartNet+ScanNet, particularly for IoU@1, which probes mask diversity under an ambiguous single prompt.Source: §5.2 · p. 8, Table 4 · p. 8
  • With ScanNet-only training, Point-SAM* beats AGILE3D on OOD KITTI360 and PartNet-Mobility but performs worse on in-domain S3DIS.The comparison uses Point-SAM trained under AGILE3D's ScanNet-only setting alongside the original Point-SAM.Source: §5.2 · pp. 8–10
  • Pseudo-labels generated by Point-SAM alone help, but incorporating 2D SAM substantially boosts overall performance on PartNet-Mobility.The baseline generates instance proposals with pretrained Point-SAM; the pipeline uses 1,024 point prompts and NMS to filter duplicate instances.Source: §5.2 · pp. 9–10
  • For S3DIS point clouds averaging about 500K points, increasing patch count and patch size is important for zero-shot prompt segmentation.The study varies both hyperparameters because S3DIS has about 50 times as many points as the training datasets and different neighborhood densities.Source: §5.2 · p. 10, Table 6 · p. 10

6 CONCLUSION

Point-SAM is a transformer-based point-cloud model that responds to 3D point and mask prompts, trained across mixed datasets with SAM-distilled pseudo-labels. The authors also identify limits in 3D data scale and diversity, computational demands, and dependence on 2D pseudo-label quality.

6 CONCLUSION

  • Point-SAM implements promptable 3D segmentation for point clouds and responds to point and mask prompts.Its training strategy uses mixed datasets such as PartNet and ScanNet, enhanced with pseudo-labels generated by distilling knowledge from SAM.Source: §6 · p. 10
  • Training across mixed datasets is beneficial, especially when enhanced with SAM-distilled pseudo-labels.The conclusion describes this as a robust training strategy.Source: §6 · p. 10
  • The diversity and scale of available 3D datasets lag behind those available in 2D, constraining generalization across 3D environments and tasks.The authors also identify computational demands and the complexity of efficient 3D-specific operations as hurdles; pseudo-labels depend on 2D SAM labels that may miss complex 3D structure.Source: §6 · p. 10

A TRAINING DETAILS

Point-SAM training uses AdamW with a scheduled learning rate and different hardware and iteration settings for ViT-g and ViT-l. Training also uses point resampling, random scaling, and rotations.

A TRAINING DETAILS

  • The ViT-g model trains for 100k iterations with AdamW and a step-wise learning-rate decay at 60k and 90k iterations.It uses a total batch size of 128 across 8 NVIDIA H100 GPUs, with 4 samples per GPU and gradient accumulation of 4.Source: A TRAINING DETAILS · p. 14
  • The ViT-l model trains for 50k iterations across 2 NVIDIA A100 GPUs, with a batch size of 16 per GPU and gradient accumulation of 4.Its learning-rate decay milestones are 30k and 40k iterations.Source: A TRAINING DETAILS · p. 14
  • Training samples 10,000 points online from 32,768 pre-sampled points and applies random scaling and rotation.The scale factor is [0.8, 1.0]; object clouds also receive x- and z-axis rotation perturbations sampled with standard deviation 0.06 and clipped to [0, 0.18].Source: A TRAINING DETAILS · p. 14

B.1 ZERO-SHOT OBJECT PROPOSALS

The appendix evaluates Point-SAM on zero-shot object proposals and few-shot part segmentation, and reports results for AGILE3D normalization, prompt sampling, and tokenizer efficiency. The evaluations include transfer to Replica and ShapeNet-Part as well as tokenizer comparisons across datasets.

B.1 ZERO-SHOT OBJECT PROPOSALS

  • Point-SAM performs strongly against OpenMask3D on zero-shot object proposals on Replica, despite zero-shot evaluation on unseen data and no training on such large point counts.The evaluation uses average recall, excludes undefined and floor categories, and truncates both methods to their top 250 proposals.Source: §B.1 · pp. 14–15

B.2 FEW-SHOT PART SEGMENTATION

  • On ShapeNetPart, Point-SAM surpasses PointBERT and Uni3D (close) in 1-shot and 2-shot part segmentation.The evaluation reports category-averaged mIoU; the Point-SAM encoder is frozen while the feature propagation layer and MLP are optimized.Source: §B.2 · p. 15

B.3 NORMALIZATION SCALE FOR AGILE3D

  • AGILE3D achieves its best zero-shot prompt-segmentation performance on held-out PartNet-Mobility categories with normalization scale 5.Table 8 reports the normalization-scale comparison.Source: §B.3 · p. 15, Table 8 · p. 15

C PROMPT SAMPLING

  • Prompt sampling selects foreground points with the largest minimum distance to background points, using false positives for negative prompts and false negatives for positive prompts.The final prompt is selected from the two candidates by the furthest-distance criterion.Source: C PROMPT SAMPLING · pp. 15–16

D VORONOI TOKENIZER

  • The Voronoi-based tokenizer outperforms the KNN-based tokenizer in time and memory efficiency across all evaluated datasets.The authors note that Point-SAM's mask encoder also requires a point-cloud tokenizer.Source: D VORONOI TOKENIZER · p. 16, Table 9 · p. 16

E TRAINING DATASETS ABLATION

The training-dataset ablation finds that adding data improves performance across evaluation datasets, with the effect varying by training-data and evaluation-dataset combination. Additional visualizations report transfer to Waymo and few-shot ShapeNet-Part results.

E TRAINING DATASETS ABLATION

  • Adding training data consistently improves performance, but the gains vary by evaluation dataset: ShapeNet helps PartNet-Mobility more, while ScanNet has a stronger effect on S3DIS.The ablation also reports improved KITTI360 performance as dataset size increases, suggesting greater transferability with more training data.Source: E TRAINING DATASETS ABLATION · p. 16, Table 10 · p. 17

F MORE VISUALIZATION

  • Waymo visualizations show Point-SAM segmenting outdoor objects such as cars and trees in a fully out-of-distribution setting.Figure 5 presents qualitative Waymo Open dataset results.Source: Figure 5 · p. 16, F MORE VISUALIZATION · p. 17
  • Figure 4 compares ground-truth segmentation with results from varying numbers of prompt points on Waymo, where Point-SAM demonstrates superior transferability.The dataset is described as out of distribution.Source: F MORE VISUALIZATION · p. 17
  • Figure 6 presents one-shot linear-probing results on ShapeNet-Part, which the authors describe as evidence of Point-SAM's pre-training quality for segmentation.One sample per category is used for training before inference on the evaluation set.Source: F MORE VISUALIZATION · p. 17, Figure 6 · p. 17

G EVALUATION FOR INTERIOR SEGMENTATION

The evaluation compares Point-SAM and MV-SAM on PartNet-Mobility StorageFurniture, using Point-SAM trained only on PartNet and ScanNet. Point-SAM performs better with interior segmentation masks.

G EVALUATION FOR INTERIOR SEGMENTATION

H EVALUATION ON REPLICA

On Replica, Point-SAM outperforms AGILE3D using the whole training dataset and a Voronoi-based tokenizer. The evaluation follows the block-based scene-splitting procedure used for S3DIS and KITTI360.

H EVALUATION ON REPLICA

I MORE INTERACTIVE SEGMENTATION RESULTS

Additional interactive segmentation visualizations show Point-SAM results on complicated scenes and objects, with both mesh projections and raw point-cloud outputs. The examples are described as showing superior transferability.

I MORE INTERACTIVE SEGMENTATION RESULTS

Related papers

Loading 2406.17741v2…