Source-linked AI summary
PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Models
Minghua Liu, Yinhao Zhu, Hong Cai, Shizhong Han, Zhan Ling, Fatih Porikli, Hao Su
TL;DR
Generalizable 3D part segmentation is limited by scarce fine-grained annotations and weak low-shot generalization. PartSLIP transfers pretrained GLIP knowledge from rendered multi-view images to 3D point clouds, using prompt tuning and 3D-aware aggregation. It reports strong zero-shot results and few-shot performance that exceeds existing few-shot methods while remaining competitive with full supervision.
Problem
Fine-grained 3D part annotations are costly to collect, while supervised models can struggle with unseen shapes and low-shot generalization.
Method
PartSLIP renders point clouds from multiple views, applies GLIP with part-name prompts, lifts predictions to 3D, and adapts prompts using few-shot data.
Results
The method achieves excellent zero-shot segmentation, outperforms existing few-shot approaches, and reaches 59.4% mIoU after 8-shot prompt tuning.
Takeaways & Limitations
Pretrained image-language knowledge can support low-shot 3D part segmentation across benchmark datasets and iPhone-scanned point clouds.
Takeaways & Limitations
The pipeline cannot handle interior object points and has long running times from rendering and repeated GLIP inference.
Abstract
from arXiv · showhide
Generalizable 3D part segmentation is important but challenging in vision and robotics. Training deep models via conventional supervised methods requires large-scale 3D datasets with fine-grained part annotations, which are costly to collect. This paper explores an alternative way for low-shot part segmentation of 3D point clouds by leveraging a pretrained image-language model, GLIP, which achieves superior performance on open-vocabulary 2D detection. We transfer the rich knowledge from 2D to 3D through GLIP-based part detection on point cloud rendering and a novel 2D-to-3D label lifting algorithm. We also utilize multi-view 3D priors and few-shot prompt tuning to boost performance significantly. Extensive evaluation on PartNet and PartNet-Mobility datasets shows that our method enables excellent zero-shot 3D part segmentation. Our few-shot version not only outperforms existing few-shot approaches by a large margin but also achieves highly competitive results compared to the fully supervised counterpart. Furthermore, we demonstrate that our method can be directly applied to iPhone-scanned point clouds without significant domain gaps.
1. Introduction
3D part segmentation supports important machine capabilities, but limited annotated 3D data and poor generalization make low-shot learning difficult. PartSLIP addresses this by transferring pretrained image-language knowledge to 3D point clouds through GLIP and multi-view processing.
- 3D part segmentation supports robotic manipulation, AR/VR, and shape analysis by helping machines understand object structure, semantics, mobility, and functionality.
- 3D part datasets are orders of magnitude smaller than image datasets because building models and annotating fine-grained parts are laborious and time-consuming.
- Standard supervised methods achieve strong results but often struggle with out-of-distribution shapes, including unseen classes.
- Few-shot methods improve vanilla supervised learning, yet a substantial gap remains between their capabilities and downstream application needs.
- PartSLIP uses pretrained image-language models to address zero-shot and few-shot 3D part segmentation, leveraging their visual concepts and natural-language-referenced knowledge.
- The method renders point clouds into multi-view images, applies GLIP with part-name prompts, and transfers its predictions back to 3D segmentation.
- On PartNetE, the method achieves excellent zero-shot performance, large few-shot gains over existing approaches, competitive results against full supervision, and direct application to iPhone scans.
2. Related Work
Related work spans conventional semantic and instance segmentation, low-shot representation learning, label-efficient annotation settings, and pretrained image-language models applied to 3D tasks.
- 3D part segmentation combines semantic labeling of geometric primitives with instance grouping or region-proposal strategies.
- Low-shot 3D segmentation methods use pretext tasks, auxiliary losses, compositional generalization, deformation, 2D contrastive learning, autoencoders, prototypes, reinforcement learning, and augmentation.
- Some label-efficient methods assume only a small fraction of point labels are annotated, a setting distinguished from setups with few training shapes.
- Pretrained image-language models learn visual concepts from large-scale image-text pairs and have been applied to diverse 3D tasks, including classification, grounding, generation, and neural radiance fields.
3. Proposed Method: PartSLIP
PartSLIP transfers GLIP’s open-vocabulary 2D part-recognition ability to 3D point clouds through multi-view rendering, 2D-to-3D lifting, and optional few-shot adaptation. Its pipeline combines super-point voting and grouping with prompt tuning and multi-view feature fusion.
- 3.1. Overview: 3D Part Segmentation with GLIP: GLIP receives rendered multi-view point-cloud images and text prompts, producing 2D part bounding boxes for 3D segmentation.The point cloud is rendered from uniformly spaced camera poses intended to cover the shape, then processed by GLIP.
- 3.2. Detected 2D BBoxes to 3D Point Segmentation: A learning-free lifting module converts 2D detections into 3D semantic and instance segmentation through super-point generation, 3D voting, and instance grouping.Super points receive semantic labels from bounding-box voting and are merged when their labels, 3D adjacency, and box-coverage patterns agree.
- 3.3. Prompt Tuning w/ Few-Shot 3D Data: Prompt tuning adapts each part name’s language embedding using few-shot 3D shapes while freezing the pretrained GLIP parameters.Trainable token-wise offset features adjust the part definition in language-embedding space and remain fixed after tuning.
- 3.4. Multi-View Visual Feature Aggregation: Multi-view feature aggregation uses 3D point correspondences to combine corresponding feature cells across views, addressing GLIP’s sensitivity to unfamiliar camera angles.The method replaces isolated view features with aggregated features, allowing information from other views to support predictions.
- 3.4. Multi-View Visual Feature Aggregation: Early fusion before vision-language fusion is preferred because late fusion can fail when final features already encode predicted 2D bounding-box shape information.The paper reports that late fusion does not improve or can degrade the original performance, whereas early fusion performs better.
4. Experiments
Experiments evaluate PartSLIP on PartNetE across semantic and instance segmentation, low-shot settings, ablations, input variations, and real iPhone-scanned point clouds. The method shows strong zero- and few-shot performance, while sparse inputs and misunderstood prompts remain important limitations.
- Dataset and evaluation: PartNetE combines PartNet and PartNet-Mobility to evaluate low-shot generalization across complementary category and instance coverage.PartNet-Mobility provides more categories but fewer shapes, whereas PartNet provides more shapes but fewer categories.
- Semantic segmentation: 59.4% mIoU is achieved after 8-shot prompt tuning, outperforming few-shot baselines and remaining competitive with the fully supervised comparison at 56.3% versus 58.5%.The fully supervised comparison uses 28k additional shapes for 17 overlapping categories.
- Instance segmentation: 18.0% mAP50 is obtained zero-shot and 44.8% mAP50 with 8-shot tuning, outperforming baseline approaches in the evaluated training settings.These results concern instance segmentation on PartNetE.
- Ablation studies: 3D voting and grouping converts 2D detections into 3D semantic and instance segmentation, while prompt tuning improves recognition of difficult part names such as kettle spouts.Without the proposed grouping module, 3D instance segmentation is unavailable; prompt tuning particularly improves the Kettle category.
- Ablation studies: Multi-view feature aggregation improves performance without extra finetuning, and integrating the proposed modules yields the final reported performance.The ablation attributes the improvement to multi-view 3D priors helping GLIP interpret 3D shapes.
- Robustness and real-world deployment: Performance drops for sparse point clouds, whereas partial or texture-free inputs remain effective; the method also produces similar results on iPhone-scanned point clouds.The sparse-input degradation is attributed to larger rendering domain gaps and weaker super-point generation.
5. Discussion and Limitations
The pipeline relies on GLIP-predicted bounding boxes and has several practical limitations. It cannot handle interior object points and is computationally slow because rendering and repeated GLIP inference are required.
- Bounding-box pipeline: The current pipeline uses GLIP-predicted bounding boxes rather than pretrained 2D segmentation outputs.The authors note that 2D segmentation would be more natural and accurate, but the relevant pretrained model was unavailable at submission time.
- Bounding-box pipeline: Even with multi-view 2D segmentation, obtaining 3D instance segmentation would remain non-trivial, so the method’s components would still require adaptation.
- Practical limitations: The method cannot handle interior points of objects.
- Practical limitations: Point-cloud rendering and multiple GLIP inferences make the method suffer from long running time.
S. Supplementary Material
The supplementary material provides dataset details, real-world demonstrations, ablations, model comparisons, qualitative results, baseline implementation details, and full quantitative tables.
- Supplement contents: The supplement details PartNet-Ensembled, presents real-world demos, and visualizes ablation studies.
- Supplement contents: It compares CLIP and GLIP for object-part recognition and provides qualitative comparisons with baseline approaches.
- Supplement contents: The supplement also documents baseline implementations and provides full quantitative comparison tables.
S.1. PartNet-Ensembled Dataset
PartNet-Ensembled combines PartNet and PartNet-Mobility with consistent part definitions and selected annotations spanning coarse and fine-grained object parts.
- Dataset construction: PartNetE uses few-shot and test shapes from PartNet-Mobility and additional training shapes from PartNet, with consistent part definitions across sets.
- Dataset construction: The dataset selects 100 existing object parts and manually annotates three additional parts.
- Dataset construction: The construction filters extremely fine-grained, ambiguous, inconsistently annotated, and rarely seen parts.
- Part coverage: Each object category contains 1-6 parts, covering common coarse-grained parts and fine-grained parts relevant to downstream tasks such as robotic manipulation.
S.2. Real-World Demo
The real-world demo evaluates the method on colored iPhone LiDAR point clouds and finds that it generalizes without significant domain gaps, unlike the baselines shown.
- Data preparation: The demo uses Polycam to scan daily objects with an iPhone 12 Pro Max LiDAR sensor, then removes ground points and computes point normals in MeshLab.
- Results: Our method directly generalizes to iPhone-scanned point clouds without significant domain gaps, whereas baseline methods perform poorly.
- Results: PointNext still fails to recognize many parts despite using an additional 28k training shapes.
- Figure setup: The figure compares iPhone-scanned point clouds, text prompts, and outputs from the proposed method and baseline approaches across few-shot settings.“45x8” denotes training with 8 shapes per object category, while “45x8+28k” adds 28k training shapes.
- Results: The few-shot baseline performs even worse and identifies only a few parts.
S.3. Visualization of Ablation Studies
Figure S9 shows that few-shot prompt tuning helps pretrained GLIP adapt from misunderstanding many part names to recognizing part definitions and generalizing to unseen instances.
- Few-Shot Prompt Tuning: Before tuning, pretrained GLIP fails to understand the meaning of many part names.
- Few-Shot Prompt Tuning: After tuning with one or a few segmented 3D shapes, GLIP quickly adapts to part definitions.
- Few-Shot Prompt Tuning: The tuned model can generalize to unseen instances.
Multi-View Visual Feature Aggregation
Multi-view visual feature aggregation improves GLIP’s ability to interpret unfamiliar camera views of 3D shapes, producing more accurate predictions.
- Without multi-view feature aggregation, GLIP fails to detect parts from some unfamiliar camera views.
- After aggregating visual features from multiple views, GLIP more comprehensively understands 3D shapes and predicts unfamiliar views more accurately.
- The ablation compares prompt-tuned detection without and with multi-view visual feature aggregation.
S.4. CLIP vs. GLIP
CLIP is unsuitable for this part-segmentation pipeline because it focuses on image classification, lacks direct region-level outputs, and struggles with fine-grained parts.
- CLIP focuses mainly on image classification and cannot directly generate region-level outputs such as segmentation masks or bounding boxes.
- The pretrained CLIP model fails to determine whether an object contains a fine-grained part.
S.5. Qualitative Comparison on PartNetE
Qualitative comparisons show that PartSLIP’s few-shot version outperforms existing few-shot methods, including a baseline trained with 28k additional 3D shapes, while its zero-shot version also performs impressively.
- Qualitative comparison: PartSLIP’s 45x8 few-shot version outperforms all existing few-shot methods and the 45x8+28k PointNext version.
- Qualitative comparison: The method is particularly effective at detecting small parts such as wheels, bulbs, screws, handles, knobs, and buttons.
- Qualitative comparison: The zero-shot version achieves impressive results without any 3D training.
- Evaluation settings: The comparison uses semantic segmentation results under 45x8 and 45x8+28k training settings, with baselines trained on PartNetE.