PaperlayerView exact revision on arXiv

Source-linked AI summary

UniPart: Towards Zero-shot Language-Grounded 3D Part Segmentation for Embodied Interaction

Xinqiang Yu, Zekun qi, Jiawei He, Wenyao Zhang, Xuchuan Chen, Guaocai Yao, Li Yi, Zhaoxiang Zhang, He Wang

arXiv:2609.12898v1cs.CVcs.AIcs.RO

TL;DR

Open-world manipulation needs fine-grained part understanding from language, but existing 3D approaches are often coarse or tied to fixed taxonomies. UniPart combines large-scale language-aligned part supervision with a feed-forward text-conditioned 3D Transformer, and the paper reports open-vocabulary generalization and real-sensor grasping use.

  • Problem

    Open-world manipulation needs robots to identify action-relevant parts on unseen objects from instructions, while existing 3D approaches are either object-aware but coarse or part-aware but limited to fixed taxonomies.

    Source: §1 · p. 1

  • Method

    UniPart is a feed-forward cross-modal 3D Transformer conditioned on CLIP text embeddings, trained with large-scale multi-view language-to-part supervision and manually labeled data.

    Source: §1 · pp. 1–2

  • Results

    UniPart achieves strong generalization across open-vocabulary benchmarks and produces usable real-sensor part masks for language-conditioned grasping.

    Source: §5 · pp. 10–11

  • Takeaways & Limitations

    Language-grounded part masks are presented as a semantic interface and useful primitive for embodied interaction.

    Source: §5 · p. 10

    The work focuses on isolated objects; future work will extend part understanding beyond them and strengthen closed-loop integration with robotic control.

    Source: §5 · pp. 10–11

Abstract

from arXiv · show

Fine-grained robotic manipulation depends on understanding parts, not only whole objects. Existing 3D foundation models tend to be either generalized but object-aware, or part-aware but limited to closed-set taxonomies, which weakens zero-shot transfer. We study text-conditioned 3D part segmentation, where a free-form phrase selects a functional part on point cloud. We introduce UniPart, a feed-forward cross-modal 3D Transformer that conditions CLIP text embedding. To scale supervision, we build LangPart-1M with 160K+ Objaverse assets and 8M text to part pairs using multi-view consistent part generation. We further manually label a high-quality subset, LangPart-4K, for fine-tuning and evaluation. UniPart achieves strong zero-shot results on open-vocabulary part benchmarks and transfers to language-conditioned part grasping in real world.

1 Introduction

Open-world manipulation requires identifying action-relevant parts on unseen objects from language, but existing 3D approaches are either coarse or limited to fixed part vocabularies. UniPart pairs large-scale language-aligned supervision with a text-conditioned 3D model and reports real-sensor grasping use.

1 Introduction

  • Robots must identify semantically relevant parts on unseen objects from human instructions, rather than recognize only whole objects.The introduction frames this ability as important for open-world manipulation.Source: §1 · p. 1
  • Promptable 3D methods such as Point-SAM [87] separate object instances but miss internal parts, while fixed-taxonomy models such as ShapeNetPart struggle with open-vocabulary generalization.Source: §1 · p. 1
  • UniPart is a feed-forward cross-modal 3D Transformer that adds frozen CLIP text embeddings at every layer, with a controlled study separating data and architecture effects.Source: §1 · p. 2
  • LangPart-1M comprises 160K objects and 8M text-to-part pairs, while LangPart-1K is a 1K-object benchmark split with more than 4K annotations for open-vocabulary evaluation.The multi-view pipeline combines SAM proposals, Set-of-Mark prompting, and text-guided merging; LangPart-4K is manually verified and receives text labels for part masks.Source: §1 · p. 2
  • UniPart produces part masks on real sensor point clouds that are useful for language-conditioned grasping.Source: §1 · pp. 2–3

2 Related Works

Related work spans closed-set part segmentation, open-vocabulary 3D approaches, language-guided manipulation, and 3D representation learning. UniPart is positioned as a language-queryable part-mask model with multi-view-consistent supervision and feed-forward inference.

2.1 3D Part Segmentation

  • Classic part segmentation offers fine granularity within fixed taxonomies, while open-vocabulary 3D segmentation largely targets instances or scenes; concurrent FIND3D [38] also queries parts with language.UniPart differs from FIND3D through text-guided mask merging for 3D consistency and feed-forward point-based inference without multi-view querying.Source: §2.1 · p. 3

2.2 Language-Grounded Robot Manipulation

  • Language-guided manipulation includes end-to-end action models and systems that separate high-level reasoning from low-level actions; UniPart belongs to the latter group.The paper presents fine-grained part understanding as empowering UniPart within this group.Source: §2.2 · p. 3

2.3 3D Representation Learning

  • 3D representation learning includes point-, voxel-, and multiview-based methods, with self-supervised and cross-modal work extending representations toward open-world settings.Source: §2.3 · p. 3

3.1 Datasets

The datasets balance scalable but potentially noisy automatic annotations with a manually verified subset for high-fidelity supervision. They provide broad object and part coverage and a benchmark designed to test transfer across categories and part types.

3.1 Datasets

  • LangPart-1M provides broad, automatically generated coverage, while manually verified LangPart-4K supplies high-fidelity part supervision for fine-tuning and evaluation.The automated subset may contain minor inconsistencies from model-induced noise.Source: §3.1 · pp. 3–4, §3.1.5 · p. 5

3.1.1 Dataset Statistics

  • Over 160K objects span more than 250 categories, with over 1,000 part types and approximately 8 million text–part pairs.Source: §3.1.1 · p. 4

3.1.2 Data Source

  • LangPart-1M is built from Objaverse assets, which are re-rendered in Blender under controlled lighting to produce over 8 million images.Source: §3.1.2 · p. 4

3.1.3 Data Filtering

  • Filtering retains clean, meaningful, high-quality, distinguishable object-centric assets without benchmark leakage, using GPT-5.2-Pro and a fixed prompt for pass decisions.The authors say they will release the prompt, retained asset IDs, and per-asset decisions for reproducibility.Source: §3.1.3 · p. 4

3.1.4 LangPart-1M Dataset Generation

  • The pipeline queries part names, segments 12 rendered views with SAM and Set-of-Mark identifiers, text-guidedly merges regions, then back-projects masks into a labeled point cloud.The merging step targets cross-view coherence, over-segmentation, and isolated outliers.Source: §3.1.4 · p. 5

3.1.5 Manual Annotation for Quality Improvement

  • LangPart-4K selects objects with functional or structurally meaningful parts, then annotators verify part-name-to-region matches and correct errors in the generated masks.The verified masks support fine-tuning and benchmark construction.Source: §3.1.5 · pp. 5–6

3.1.6 Text-Alignment Part Segmentation Benchmark

  • LangPart-1K holds out 1,000 objects to evaluate seen and unseen categories and part types, addressing benchmark limits from narrow categories and fixed vocabularies.LangPart-4K is split into LangPart-3K for training and LangPart-1K for evaluation.Source: §3.1.6 · p. 6

3.2 A Cross-Modal 3D Transformer for Language-Grounded Part Segmentation

UniPart predicts a part mask from a point cloud and arbitrary language description using a cross-modal 3D Transformer. It uses local geometric tokens and injects the CLIP text embedding additively at every Transformer layer.

3.2 A Cross-Modal 3D Transformer for Language-Grounded Part Segmentation

  • UniPart is a cross-modal point-cloud Transformer that takes an object point cloud and language description and predicts the corresponding text-conditioned part mask.Source: §3.2 · p. 6

3.2.1 Language and Point Cloud Embeddings

  • Farthest-point sampled neighborhoods become local geometric tokens, while a global OpenAI CLIP text embedding conditions the fused representations for per-point binary mask prediction.A decoder upsamples [CLS] token features to point-level predictions.Source: §3.2.1 · p. 6

3.2.2 Cross-Modal Fusion

  • The model adds the global text feature to every point token at each Transformer layer, a fusion strategy reported to offer the best trade-off between simplicity and effectiveness.Source: §3.2.2 · pp. 6–7

3.3 Optimization

Optimization first aligns point features with rendered-image CLIP features, then trains segmentation masks against ground-truth labels. The CLIP parameters remain frozen.

3.3 Optimization

  • The second and third fine-tuning stages minimize binary cross-entropy between predicted segmentation masks and one-hot ground-truth labels, with CLIP kept frozen.Source: §3.3 · p. 7
  • Pretraining aligns each point feature with its corresponding rendered-image CLIP patch feature to transfer CLIP’s semantic structure into the 3D representation.The paper says this initialization facilitates subsequent text-guided part understanding.Source: §3.3 · p. 7
  • The point-to-patch mapping π(i) identifies the corresponding image patch, and the alignment uses normalized point and CLIP patch features with their inner product.Source: §3.3 · p. 7

3.4 Comparison with Related Works

The comparison with related work centers on text-prompted part segmentation and evaluates UniPart’s supervision and architecture under matched supervision. The study also tests predicted masks in a real robotic grasping pipeline.

3.4 Comparison with Related Works

  • Compared with text-prompted part segmentation such as FIND3D [38], the study evaluates two differentiators under matched supervision.Source: §3.4 · p. 7
  • LangPart-1M provides large-scale point-level supervision built with a multi-view consistency mechanism based on text-guided mask merging.Source: §3.4 · p. 7
  • UniPart uses a feed-forward point-based model with layer-wise additive text injection to produce dense part masks without multi-view querying at inference.The study also tests the resulting masks in a real robotic grasping pipeline.Source: §3.4 · p. 7

4 Experiments

The experiments evaluate UniPart on Objaverse-General, ShapeNetPart, PartNet-E, and LangPart-1K, using established open-world baselines and two query formats.

4.1 Experimental Settings

  • Evaluation covers Objaverse-General, ShapeNetPart, PartNet-E, and LangPart-1K, with mean IoU averaged across annotated parts and point clouds.Source: §4.1 · p. 7

4.2.1 Open-world Baselines

  • The open-world comparisons follow FIND3D’s baseline selection and protocol, using both “{part} of a {object}” and “{part}” queries.Source: §4.2.1 · p. 7
  • PointCLIPV2 [88] repeatedly queries CLIP [56] across rendered views and selects from top-k prompts, while PartSLIP++ [35] transfers GLIP [30] detections through superpoints and uses its released zero-shot checkpoint.Source: §4.2.1 · p. 7
  • OpenMask3D [61] is an open-vocabulary 2D-to-3D approach trained on 3D scenes; dense predictions assign every point a label, with “other” used for unlabeled regions.Source: §4.2.1 · p. 7
  • Tables 2 and 3 compare open-world methods on Objaverse-General and ShapeNetPart using both query formats, while Table 4 compares methods on LangPart-1K.Source: Table 2 · p. 8, Table 3 · p. 8, Table 4 · p. 8

4.3 Experimental Results

UniPart performs best in the reported comparisons across open-world benchmarks and LangPart-1K. On real sensor point clouds, it predicts target-part masks that support language-conditioned grasping, with reported segmentation and grasp success rates.

4.3 Experimental Results

  • UniPart performs best on Objaverse-General, ShapeNetPart, and PartNet-E in the reported open-world comparisons.Tables 2, 3, and 5 present the benchmark comparisons.Source: §4.3 · p. 8, Table 5 · p. 9
  • UniPart has the best reported performance on LangPart-1K under both partial-observation and complete-point-cloud test protocols.Separate models are trained for partial observations and full geometry.Source: §4.3 · p. 8

4.4.1 Real World Robot Setups

  • The real-world setup uses one end-mounted RealSense D415 camera with Franka Panda and UR arms, reducing environmental setup and multi-camera calibration overhead.Source: §4.4.1 · p. 8

4.4.2 Language-Conditioned Grasping in Real-World Scenarios

  • The grasping pipeline feeds UniPart’s language-selected target-part point cloud to off-the-shelf GraspNet [64] or DexGrasp-Net2.0 [77].Source: §4.4.2 · p. 9
  • On 20 real objects, UniPart achieves 90.5% part segmentation accuracy and the downstream pipeline achieves 85.0% grasp success on the intended part.Source: §4.4.2 · p. 9
  • Under sensor noise and occlusion, UniPart delineates parts including handles and grips in qualitative real-world examples.Source: §4.4.2 · p. 9

4.5 Ablation Study

The controlled study separates architecture and dataset effects, while further ablations examine cross-modal pretraining and fusion. The reported results favor UniPart trained on LangPart-1M, show pretraining gains across partial and complete inputs, and identify addition as the strongest tested fusion choice.

4.5.1 Controlled Study

  • On Objaverse-General, UniPart exceeds FIND3D by 14.69% mIoU with FIND3D data, and FIND3D gains 24.94% mIoU with LangPart-1M; UniPart trained on LangPart-1M performs best.The controlled study cross-evaluates both models with both training datasets.Source: §4.5.1 · pp. 9–10

4.5.2 Ablation study on Pretraining

  • Pretraining on single-view partial point clouds improves fine-tuning on complete point clouds, with gains typically larger for partial inputs but consistent on complete geometry.The pretraining stage uses point-to-image patch correspondences.Source: §4.5.2 · p. 10, Table 7 · p. 10

4.5.3 Cross-Modal Fusion Choices

  • Among cross-attention, multiplication, addition, and concatenation, simple addition achieves the best performance in the fusion ablation.The authors suggest short instructions and the text CLS token’s high-level semantics may explain this result.Source: §4.5.3 · p. 10, Table 8 · p. 10

5 Conclusion and Future Work

The paper presents UniPart as a language-to-part-mask interface for embodied interaction, supported by large-scale text-part supervision and manually verified evaluation data. It reports open-vocabulary generalization and usable masks for grasping, with broader object understanding and closed-loop robotic control left for future work.

5 Conclusion and Future Work

  • UniPart maps natural-language phrases to fine-grained 3D part masks, backed by LangPart-1M’s 8 million text-to-part alignments on 160K+ objects and manually verified LangPart-4K and LangPart-1K.Source: §5 · p. 10
  • The authors report strong generalization across open-vocabulary benchmarks and usable part masks on real sensor point clouds for language-conditioned grasping.Source: §5 · p. 11
  • Future work will extend part understanding beyond isolated objects and strengthen closed-loop integration with robotic control.Source: §5 · p. 11

Related papers

Loading 2609.12898v1…