PaperlayerView exact revision on arXiv

Source-linked AI summary

SAMPart3D: Segment Any Part in 3D Objects

Yunhan Yang, Yukun Huang, Yuan-Chen Guo, Liangjun Lu, Xiaoyang Wu, Edmund Y. Lam, Yan-Pei Cao, Xihui Liu

arXiv:2411.07184v2cs.CV

TL;DR

3D part segmentation is constrained by limited part annotations and zero-shot methods' reliance on text prompts, which complicates scaling and flexible handling of part ambiguity. SAMPart3D distills text-independent 2D features into a 3D backbone and uses scale-conditioned grouping, then reports stronger results than prior zero-shot methods and supports part-level applications.

  • Problem

    3D part segmentation is limited by scarce, narrow annotations and by zero-shot methods' reliance on predefined text labels, with ambiguity in part semantics and granularity.

    Source: §1 · p. 2

  • Method

    SAMPart3D distills text-independent DINOv2 features into a 3D backbone, uses scale-conditioned grouping for segmentation, and queries MLLMs for part labels from multi-view renderings.

    Source: §1 · p. 2

  • Results

    SAMPart3D achieves outstanding part-segmentation results on complex and diverse 3D objects compared with existing zero-shot methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework supports part-level editing and interactive segmentation, and the paper introduces PartObjaverse-Tiny with detailed annotations for 200 complex objects.

    Source: §1 · p. 2

    Inaccurate SAM masks can affect grouping-field results, and training a grouping field for each object remains slow.

    Source: §6.3 · p. 11

Abstract

from arXiv · show

3D part segmentation is a crucial and challenging task in 3D perception, playing a vital role in applications such as robotics, 3D generation, and 3D editing. Recent methods harness the powerful Vision Language Models (VLMs) for 2D-to-3D knowledge distillation, achieving zero-shot 3D part segmentation. However, these methods are limited by their reliance on text prompts, which restricts the scalability to large-scale unlabeled datasets and the flexibility in handling part ambiguities. In this work, we introduce SAMPart3D, a scalable zero-shot 3D part segmentation framework that segments any 3D object into semantic parts at multiple granularities, without requiring predefined part label sets as text prompts. For scalability, we use text-agnostic vision foundation models to distill a 3D feature extraction backbone, allowing scaling to large unlabeled 3D datasets to learn rich 3D priors. For flexibility, we distill scale-conditioned part-aware 3D features for 3D part segmentation at multiple granularities. Once the segmented parts are obtained from the scale-conditioned part-aware 3D features, we use VLMs to assign semantic labels to each part based on the multi-view renderings. Compared to previous methods, our SAMPart3D can scale to the recent large-scale 3D object dataset Objaverse and handle complex, non-ordinary objects. Additionally, we contribute a new 3D part segmentation benchmark to address the lack of diversity and complexity of objects and parts in existing benchmarks. Experiments show that our SAMPart3D significantly outperforms existing zero-shot 3D part segmentation methods, and can facilitate various applications such as part-level editing and interactive segmentation.

1. Introduction

Supervised 3D part segmentation is constrained by scarce, narrow annotations, while zero-shot approaches face challenges in learning 3D priors and handling semantic and granularity ambiguity. SAMPart3D uses text-independent feature distillation and scale-conditioned grouping, and introduces a benchmark for complex objects.

1. Introduction

  • Supervised methods achieve strong closed-set results but struggle with complex objects because annotated 3D part data are limited in quantity and diversity.Annotating large 3D asset collections is time-consuming and labor-intensive.Source: §1 · p. 2
  • Zero-shot methods must generalize without 3D part labels, learn geometric priors from unlabeled shapes, and address semantic and granularity ambiguity.Text-prompted methods depend on predefined label sets, while objects may be segmented at multiple levels of detail.Source: §1 · p. 2
  • SAMPart3D replaces text-dependent feature distillation with DINOv2 features and uses a scale-conditioned MLP distilled from SAM for controllable grouping.After segmentation, MLLMs assign semantic descriptions using multi-view renderings of each part.Source: §1 · p. 2
  • SAMPart3D introduces PartObjaverse-Tiny, with detailed semantic and instance annotations for 200 complex objects, and reports stronger results than existing zero-shot methods.The authors also report applications including interactive segmentation and part-level editing.Source: §1 · p. 2

2. Related Work

Prior work spans image-based foundation models and supervised 3D segmentation, while zero-shot methods transfer 2D priors or distill them into 3D features. SAMPart3D extends cross-modal distillation to large-scale unlabeled data without part text prompts and supports controllable granularity.

2D Foundation Models

  • 2D foundation models include self-supervised traditional models, textually prompted image-text models, and visually prompted models such as SAM.The passage notes that text prompts are less effective for fine-grained tasks such as segmentation.Source: §2 · p. 2

3D Part Segmentation

  • Supervised 3D segmentation research has developed 3D representation architectures, but requires labor-intensive annotations and struggles to generalize to complex open-world objects.PointNet++ uses neighborhoods at multiple scales, and Point Transformer layers support high-performing point-cloud segmentation backbones.Source: §2 · p. 3

Zero-shot 3D Part Segmentation

  • Zero-shot methods transfer 2D foundation-model priors through bounding-box, pixel-mask, or multi-view correspondence pipelines.PartSLIP uses GLIP, PartSLIP++ incorporates SAM, and ZeroPS bridges multi-view correspondences with prompting.Source: §2 · p. 3
  • PartDistill learns point-specific geometric features from unlabeled shapes, whereas SAMPart3D extends cross-modal distillation to Objaverse without part text prompts and adds granularity control.The overview depicts large-scale backbone pretraining, scale-conditioned grouping, and semantic querying from multi-view renderings.Source: §2 · p. 3, Figure 2 · p. 3

3. Method

SAMPart3D combines large-scale 3D feature pre-training, object-specific scale-conditioned grouping, and training-free semantic querying. It uses unlabeled objects and multi-view 2D supervision to produce parts at multiple granularities and assign them semantic labels.

3. Method

  • The framework has three stages: large-scale pre-training on unlabeled 3D objects, sample-specific scale-conditioned grouping, and training-free semantic querying with a multimodal language model.The stages learn a 3D feature backbone, produce grouped parts at selected scales, and assign labels to those parts.Source: §3 · p. 3

3.1. Large-scale Pre-training: Distilling 2D Visual Features to 3D Backbone

  • The 3D backbone is trained on Objaverse’s over 800K diverse 3D assets, using text-independent DINOv2 visual features upsampled with FeatUp as supervision for feature distillation.Across K rendered views, 2D features are averaged for the point cloud; depth identifies occlusion, and occluded points retain their original 3D features.Source: §3.1 · pp. 3–4

3.2. Sample-specific Fine-tuning: Distilling 2D Masks for Multi-granularity Segmentation

  • A scale-conditioned lightweight MLP groups backbone features using SAM masks, enabling segmentation at different granularities.Mask scale is computed from coordinate standard deviations; paired points from the same or different masks train the grouping features with a contrastive loss, then clustering produces parts.Source: §3.2 · p. 5

3.3. Semantic Querying with MLLMs

  • After part segmentation, the method highlights each part in multi-view renderings and queries an MLLM for its semantic label.It selects canonical views and uses the view with the largest rendered area for the part of interest.Source: §3.3 · p. 5

4. Experiments

The experiments introduce a diverse, fine-grained benchmark, evaluate segmentation across datasets and against prior methods, examine design choices, and illustrate several editing and interaction applications.

4.1. PartObjaverse-Tiny

  • PartObjaverse-Tiny contains 200 finely annotated shapes across eight categories, addressing the limited categories and incomplete annotations of existing part datasets.Its categories include human shapes, animals, daily-used objects, buildings and outdoor objects, transportation, plants, food, and electronics.Source: §4.1 · p. 5

4.2. Experiments Results

  • The Objaverse-pretrained model is visualized across GSO, OmniObject3D, Vroid, and generated meshes, and evaluated against prior methods on PartObjaverse-Tiny and PartNetE.PartObjaverse-Tiny comparisons cover semantic, class-agnostic, and instance segmentation; PartNetE comparisons cover semantic segmentation.Source: §4.2 · pp. 5–6, Figure 4 · p. 6

4.3. Ablation Analysis

  • Ablations report that removing pretraining weakens semantic encoding and training stability, while SAM features fail to teach useful part knowledge and the backbone exceeds direct DINOv2 feature fusion.The authors also report convergence difficulty without the long skip connection, which removes direct 3D input to the grouping field.Source: §4.3 · pp. 6–7

4.4. Applications

  • The segmentation supports 2D-mask-guided parts, material and shape editing, animation, and click-based hierarchical segmentation.For hierarchical segmentation, a 3D click is combined with scale control to adjust the segmented area.Source: Figure 6 · p. 8, §4.4 · p. 8

5. Conclusion and Discussions

SAMPart3D segments 3D objects into semantic parts at multiple granularities. The paper also introduces PartObjaverse-Tiny, and reports that experiments demonstrate the framework's effectiveness.

  • SAMPart3D is a zero-shot framework for segmenting 3D objects into semantic parts at multiple granularities.Source: §5 · p. 9
  • PartObjaverse-Tiny is introduced to address diversity and complexity shortcomings in existing annotated datasets.Source: §5 · p. 9
  • The authors report that experimental results demonstrate SAMPart3D's effectiveness.Source: §5 · p. 9

6. Supplemental Material

The supplemental material specifies training and scale-evaluation procedures, identifies limitations, and illustrates segmentation outputs and PartObjaverse-Tiny annotations. These examples span datasets, objects, and levels of granularity.

6.1. Implementation Details

  • Training uses 200K Objaverse objects for pretraining, then object-specific SAM masks and MLP training before HDBSCAN grouping and MLLM semantic querying.The reported stages take 7 days on eight A800 GPUs for pretraining, and 1 minute for mask generation plus 5 minutes for MLP training per object.Source: §6.1 · p. 11

6.2. Ablation Analysis of Segmentation Scale

  • Scale evaluation tests five values, selects the result closest to ground truth, and compares individual scales with mixed-scale results; manually annotated objects vary in suitable scales.Table 6 reports zero-shot class-agnostic part segmentation on PartObjaverse-Tiny in mIoU (%).Source: §6.2 · p. 11, Table 6 · p. 12

6.3. Limitations

  • SAM mask inaccuracies can affect final results, and training a grouping field for each object remains slow.The authors suggest using the method to annotate many parts and then training a large 3D part segmentation model.Source: §6.3 · p. 11

6.4. Visualization on PartNetE Dataset

  • SAMPart3D segments fine-grained PartNetE parts, including parts not annotated in the dataset.The supplemental visualization is shown in Figure 8.Source: §6.4 · p. 11, Figure 8 · p. 12

6.5. More Qualitative Results of Our SAMPart3D

  • Additional visualizations show multi-granularity segmentation results for point clouds and meshes.Figure 9 presents these results.Source: §6.5 · p. 11, Figure 9 · p. 13

6.6. More PartObjaverse-Tiny Visualization

Loading 2411.07184v2…