PaperlayerView exact revision on arXiv

Source-linked AI summary

Name That Part: 3D Part Segmentation and Naming

Soumava Paul, Prakhar Kaushik, Ankit Vaidya, Anand Bhattad, Alan Yuille

arXiv:2512.18003v2cs.CV

TL;DR

Semantic 3D part segmentation must assign meaningful names as well as geometric boundaries, but annotations are sparse and inconsistent and prior methods can leave parts unnamed or return only query-specific masks. ALIGN-Parts aligns shape-conditioned partlets with text descriptions using set matching and combined geometric, appearance, and affordance cues. It reports superior fine-part localization to PartField despite using 10× fewer points.

  • Problem

    3D part understanding requires both segmentation and naming, yet annotations are sparse and inconsistent, and prior methods may produce unnamed decompositions or retrieve only individual queried parts.

    Source: §1 · p. 2

  • Method

    ALIGN-Parts matches shape-conditioned partlets to text descriptions through set-level matching, combining geometric, appearance, and affordance-based semantic cues.

    Source: §1 · p. 2, §3.1 · p. 6

  • Results

    Despite using 10× fewer points than PartField, ALIGN-Parts achieves superior fine-part localization.

    Source: §6 · p. 14

  • Takeaways & Limitations

    The annotation pipeline converts raw meshes into training data with 5–8× less human effort than manual annotation.

    Source: §1 · p. 3

    The model’s robustness and coverage are constrained by the limited annotated data available for training, reflecting scarcity of dense part annotations and a unified operational definition of a part.

    Source: §5 · p. 14

Abstract

from arXiv · show

We address semantic 3D part segmentation: decomposing objects into parts with meaningful names. While datasets exist with part annotations, their definitions are inconsistent across datasets, limiting robust training. Previous methods produce unlabeled decompositions or retrieve single parts without complete shape annotations. We propose ALIGN-Parts, which formulates part naming as a direct set alignment task. Our method decomposes shapes into partlets - implicit 3D part representations - matched to part descriptions via bipartite assignment. We combine geometric cues from 3D part fields, appearance cues from multi-view vision features, and semantic knowledge from language-model-generated affordance descriptions. Text-alignment loss ensures partlets share embedding space with text, enabling a theoretically open-vocabulary matching setup, given sufficient data. Our efficient and novel, one-shot, 3D part segmentation and naming method finds applications in several downstream tasks, including serving as a scalable annotation engine. As our model supports zero-shot matching to arbitrary descriptions and confidence-calibrated predictions for known categories, with human verification, we create a unified ontology that aligns PartNet, 3DCoMPaT++, and Find3D, consisting of 1,794 unique 3D parts. We introduce two novel metrics appropriate for the named 3D part segmentation task. We also show examples from our newly created TexParts dataset.

1. Introduction

Semantic 3D part understanding requires both geometric segmentation and meaningful names, but sparse, inconsistent annotations and incomplete prior methods leave a bottleneck. ALIGN-Parts matches shape-conditioned partlets to descriptions, combining geometric, appearance, and affordance cues; the paper also presents a unified ontology and human-in-the-loop annotation pipeline.

1. Introduction

  • Existing datasets have sparse part annotations and inconsistent definitions, while prior methods either produce unnamed geometric regions or retrieve a single part without a complete set of names.The paper frames consistently named annotations as useful for scalable training data and human-in-the-loop annotation.Source: §1 · p. 2
  • ALIGN-Parts formulates part naming as set alignment: shape-conditioned partlets are matched to candidate descriptions by bipartite assignment, with a null class allowing the number of named parts to adapt per shape.Each predicted part receives at most one name, and each name is used at most once.Source: §1 · p. 2
  • Set-level matching compares a handful of partlets with descriptions rather than matching every point to every label, and the authors report faster generation of named segments.The matching also provides permutation consistency: each predicted part receives at most one name, and each name is used at most once.Source: §1 · p. 2
  • The method combines 3D part-field geometry, multi-view appearance features, and LLM-generated affordance descriptions to ground partlets in both shape and function.Descriptions are embedded in a continuous space, supporting matching to user-provided description sets without retraining; the paper also introduces two evaluation metrics.Source: §1 · p. 2
  • The paper constructs a unified ontology across PartNet, 3DCoMPaT++, and Find3D, then bootstraps TexParts from unlabeled TexVerse assets with human verification.TexParts contains 8,450 objects spanning 14,000 part categories; annotation uses 5–8× less human effort than manual annotation.Source: §1 · pp. 2–3

2. Related Work

Prior 3D part methods often separate geometric segmentation from semantic naming, or restrict naming to predefined vocabularies, queries, or object classes. ALIGN-Parts instead aligns part-level representations with text to produce complete, non-overlapping named decompositions with dynamic part cardinality.

2. Related Work

  • Earlier geometric methods partition shapes without semantic names, while multi-stage 2D-foundation-model pipelines do not produce complete, non-overlapping decompositions with coherent names.PartField learns dense per-point feature fields but produces unlabeled regions.Source: §2 · p. 3
  • Joint segmentation-and-labeling methods [15] use fixed category-specific label sets, limiting open-vocabulary naming and instance-consistent part identities.They frame labeling as classification over predefined semantics rather than language-grounded naming.Source: §2 · p. 3
  • Language-grounded systems use per-category predefined vocabularies, query-by-query retrieval, or a chair-only reference game rather than complete decompositions across object classes.Find3D retrieves masks for text queries, while PartGlot is limited to chairs and a predefined small number of parts.Source: §2 · p. 3
  • ALIGN-Parts aggregates point features into part-level representations and matches them to text via optimal transport, enabling complete, non-overlapping decompositions and dynamic part cardinality without predefined part counts.The method jointly learns segmentation and semantic alignment, with the number of parts emerging from activated partlets.Source: §2 · pp. 3–4
  • Part-based generative models discover latent structures without open-vocabulary naming, while existing annotated datasets use inconsistent taxonomies and category-specific semantics.ALIGN-Parts constructs a unified ontology across datasets for cross-dataset evaluation under a shared semantic framework.Source: §2 · p. 4

3. Method

ALIGN-Parts treats semantic 3D part segmentation as set alignment: shape-conditioned Partlets aggregate points and are matched to candidate part descriptions. Its pipeline combines geometry, appearance, and affordance-aware language, with training alignment supporting open-vocabulary matching at inference.

3.1. Architecture

  • ALIGN-Parts maps fused point features to 32 Partlets, each predicting a soft mask, partness score, and semantic embedding; 32 accommodates most shapes, which have at most 28 parts.Partlets refine shape-conditioned representations by interacting with one another and aggregating shape-specific point evidence.Source: §3 · p. 4, §3.1 · p. 5
  • Geometry features from PartField and multi-view appearance features are fused through BiCo Fusion, using bidirectional cross-attention over local 16-neighbor graphs with O(Nk) complexity.The fused features feed the Partlet decoder; geometry encodes shape characteristics while appearance provides texture and visual cues.Source: §3.1 · p. 4, Figure 2 · p. 5

3.2. Training: Partlet ↔Part Names

  • Partlets are matched to ground-truth parts through differentiable optimal transport, with text alignment, mask and partness supervision, and auxiliary coverage, overlap, and global-alignment losses.The text-alignment loss is essential for open-vocabulary grounding; without it, Partlets remain geometrically meaningful but semantically ambiguous.Source: §3.2 · p. 6

3.3. Inference Modes and Use Cases

  • At inference, known categories use a predicted class and its candidate part list, while novel categories use candidate descriptions supplied by users or generated with an LLM for Partlet matching.The method also supports single-part text retrieval for comparison with Find3D; novel-category confidence is less calibrated because training statistics are unavailable.Source: §3.3 · p. 7

3.4. Implementation Details

  • MPNet distinguishes functionally equivalent parts across classes from parts with different affordances, whereas SigLIP assigns uniformly high similarities in the reported comparison.The method uses MPNet for long-form affordance descriptions; its input feature modules are not trained end-to-end because of compute requirements and insufficient fine-tuning details.Source: §3.1 · p. 4, §3.4 · p. 8, Figure 3 · p. 8
  • The approach uses 40,982 training shapes from 3DCoMPaT++, PartNet, and Find3D, and evaluates on 206 held-out shapes spanning 50 categories.The held-out set includes eight novel objects from eight Find3D categories.Source: §3.4 · p. 8

4. Experiments

The experiments align inconsistent part vocabularies, evaluate named 3D part segmentation, and test ALIGN-Parts’ components and inference modes. They also use the model to support a human-verified TexParts annotation workflow.

4.1. 3D Part Annotation Alignment & Vocabulary Compression

  • MPNet similarity proposes vocabulary matches, Gemini verifies them, and confirmed duplicates are merged while location- or orientation-distinct parts remain separate.Examples include merging laptop_computer with laptop and retaining distinct front and rear bumpers.Source: §4.1 · p. 9

4.2. Comparison for Named 3D Part Segmentation

  • Relaxed LA-mIoU weights overlap by text-embedding cosine and correlates with class-agnostic mIoU at Pearson r = 0.978, versus 0.739 for strict LA-mIoU.Across label mismatches, the relaxed-versus-strict mIoU increase shows no large outliers.Source: Figure 5 · p. 10, §4.2 · p. 10, Figure 6 · p. 10

4.3. Results

  • ALIGN-Parts outperforms all baselines on class-agnostic and named segmentation, improving average mIoU over PartField by 15.8%.It also segments fine-grained parts that baselines miss and produces labeled parts in approximately 0.05 seconds, excluding feature preprocessing.Source: §4.3 · pp. 10–11
  • With 10k sampled points, ALIGN-Parts localizes a scissors screw that PartField misses despite training with 100k points.Find3D and PartField+MPNet also miss fine parts or produce noisy, overlapping, or under-segmented results.Source: §4.3 · p. 11

4.4. Ablations

  • Raw geometry and naive feature concatenation do not improve the baseline; text alignment improves named-part metrics, and auxiliary regularizers bring the full model to peak performance.Adding DINOv2 appearance features alone yields modest gains.Source: §4.4 · p. 11

4.5. Inference-time Ablations

  • Ground-truth part counts only slightly improve rLA-mIoU, while the default dynamic activation robustly estimates part cardinality without explicit count guidance.Alternative inference tests use k-means on fused features or retain the top M partlets when the true count is supplied.Source: §4.5 · pp. 11–12, Table 4 · p. 12

4.6. TexParts Dataset

  • TexParts uses ALIGN-Parts predictions for human validation, with phase one covering approximately 8,000 objects and 14,000 unique part categories.The dataset is intended for public release.Source: §4.6 · p. 13
  • Model-assisted annotation reduces average per-shape annotation time from 15–25 minutes to 3–5 minutes, a reported 5–8× saving.Source: §4.6 · p. 13

5. Limitations and Future Work

ALIGN-Parts’ coverage is constrained by the limited annotated training data available relative to real-world part variety. The authors propose future work on supervision, 3D priors, and geometric feature training to address these constraints.

5. Limitations and Future Work

  • The trained object and part set is restricted relative to the variety of parts found in the real world.The authors link this boundary to scarce large-scale dense annotations and the absence of a unified operational definition of a part.Source: §5 · p. 14
  • Limited annotated training data constrains ALIGN-Parts’ robustness and coverage, despite its aim of enabling annotation at scale.The authors describe this as a chicken-and-egg problem.Source: §5 · p. 14
  • Future work will explore self- or weak supervision, stronger 3D priors, and end-to-end geometric feature training.End-to-end training was not pursued primarily because of computational constraints; the authors also note that parties with abundant compute and proprietary 3D assets can scale the framework to richer datasets.Source: §5 · p. 14

6. Discussion

ALIGN-Parts frames semantic 3D part segmentation as set alignment and reports fast, coherent named parts without part-count supervision. Its discussion highlights both fine-part localization and limits on generalization and real-world scans.

6. Discussion

  • Partlets trained through optimal transport jointly learn geometry and semantics without part-count supervision.The approach frames semantic 3D part segmentation as a set-alignment problem.Source: §6 · p. 14
  • ALIGN-Parts produces fast, coherent named parts directly, unlike Find3D’s per-point alignments or PartField’s brittle clustering.The comparison is reported in Fig. 4.Source: §6 · p. 14
  • LLM-generated affordance descriptions help disambiguate fine-grained parts, and MPNet handles long-form text better than CLIP/SigLIP.The discussion gives a door-handle affordance description as an example and refers to Fig. 3.Source: §6 · p. 14
  • Despite using 10× fewer points than PartField, ALIGN-Parts achieves superior fine-part localization, including on a scissors screw.The authors identify noisy real-world scans, distribution-shift degradation in Mahalanobis confidence, and training-similar-category limits on open-vocabulary generalization.Source: §6 · p. 14
  • The authors propose extending the framework to articulated objects and integrating part-level alignments into foundation 3D models for manipulation and generation.They describe the broader contribution as bridging dense geometry and structured language for semantically rich 3D asset creation.Source: §6 · p. 14

A. Architecture

The dense feature fusion module combines geometric and appearance cues through localized, bidirectional cross-attention. It uses learned gates and a projected fused representation to incorporate cross-modal information while limiting computation through a 3D KNN graph.

A.1. Dense Feature Fusion Module

  • Geometry encodes shape characteristics while appearance contributes texture and visual cues; a KNN graph supports localized bidirectional cross-attention between them.The graph operates on 3D coordinates to maintain computational tractability.Source: §A.1 · p. 17
  • Fourier-encoded relative positional biases are computed for neighboring point pairs to incorporate 3D spatial structure.The frequency vector uses F=6 logarithmically spaced frequencies, and its MLP has two layers with ReLU activation.Source: §A.1 · p. 17
  • Geometric features attend to neighboring appearance features, while appearance features symmetrically attend to neighboring geometric features.The two directions produce cross-modal features that capture information from the other modality.Source: §A.1 · p. 17
  • Learned sigmoid gates control how much attended cross-modal information is added to each original feature.The gates use both the original feature and the attended information.Source: §A.1 · p. 17
  • After gated addition and layer normalization, the modalities are concatenated and projected through a two-layer MLP; local k=16 neighborhoods yield O(Nk) complexity.The stated feature dimensions are dg=448, da=768, and df=256.Source: §A.1 · p. 17

B. Experiments and Analysis

The experiments compare ALIGN-Parts with existing methods while noting that no published work is directly comparable and prior training data are often undisclosed. The authors report competitive or superior performance and show qualitative retrievals that closely align with ground truth.

B. Experiments and Analysis

  • The authors caution that no published work is directly comparable, and that prior methods often use proprietary Objaverse-scale data with rarely disclosed annotations.They characterize comparisons with class-agnostic segmentation methods as not entirely fair.Source: B. Experiments and Analysis · p. 17
  • ALIGN-Parts achieves competitive or superior performance relative to existing baselines despite the stated data and compute constraints.The experiments use open, publicly available datasets and 10,000 points per shape, compared with 100,000 points typically used by class-agnostic segmentation baselines.Source: B. Experiments and Analysis · p. 17
  • On airplane and motorbike examples, ALIGN-Parts’ text-driven part retrievals are described as more semantically and spatially consistent than Find3D’s and close to ground truth.The figure attributes this to the point → partlet → part label hierarchy and highlights spatially coherent point groups for natural-language queries.Source: Figure 11 · p. 18
  • The authors present their open-dataset methodology and efficiency emphasis as supporting reproducibility, accessibility, and research on open 3D datasets.Their design processes 10,000 input points per shape, versus the 100,000 points typically used by class-agnostic baselines.Source: B. Experiments and Analysis · pp. 17–18

B.1. PartField+MPNet baseline

PartField + MPNet is a clustering-based semantic baseline that fuses geometric and appearance features before matching clusters to candidate part names.

B.1. PartField+MPNet baseline

  • The stronger baseline variant uses fused per-part PartField and DINOv2 features, combining geometric and appearance information through cross-attention.Its feature fuser has 2.8M parameters, with separate projectors for local part features and semantic text.Source: §B.1 · p. 18
  • At inference, the baseline predicts an object category and part count, clusters fused point features, then assigns labels by cosine similarity and Hungarian matching.Training uses symmetric local part-text InfoNCE, global shape-class contrastive loss, and part-count cross-entropy.Source: §B.1 · p. 18
  • Dependent clustering transfers source cluster means to a target, while independent clustering segments the target separately and matches cluster centers afterward.The dependent strategy can fail when target part counts or geometry differ; independent clustering is more reliable on difficult co-segmentation cases.Source: Figure 12 · p. 19

B.2. Part-Retrieval Comparison with Find3D

The retrieval comparison evaluates ALIGN-Parts under object-specific vocabulary and part-count constraints, with qualitative examples showing coherent and meaningful retrieved groups.

B.2. Part-Retrieval Comparison with Find3D

  • Retrieval restricts candidate labels to parts known for the object class and sets active partlets to the ground-truth part count.These oracle constraints reduce search space and assignment ambiguity, supporting fairer, more interpretable comparisons.Source: §B.2 · p. 19
  • The point cloud → partlet → part label hierarchy is described as encouraging connected groups with consistent semantic meaning.The passage contrasts this with alternatives that may fragment or over-segment parts.Source: §B.2 · p. 19
  • ALIGN-Parts retrieves more spatially coherent and semantically meaningful part groups than Find3D in examples from airplane and motorbike objects.The examples are from the Objaverse-General benchmark’s closed-vocabulary evaluation set [26].Source: §B.2 · pp. 19–20

B.3. 3D Shape Co-Segmentation and Part Label Transfer

ALIGN-Parts predicts named segmentations without source-shape initialization, avoiding clustering brittleness across varying part structures; named decompositions also connect to dataset construction and downstream applications.

B.3. 3D Shape Co-Segmentation and Part Label Transfer

  • Independent clustering segments each target autonomously and matches clusters post hoc, improving robustness to topological differences but losing direct geometric correspondence to the source.Dependent clustering can blur boundaries when target part structure differs substantially, such as merging a chair backrest into its seat.Source: §B.3 · p. 20
  • ALIGN-Parts jointly predicts segmentation masks and semantic labels without source initialization or explicit co-segmentation, producing robust segmentations across diverse part structures and semantics.The reported comparison contrasts this feedforward approach with clustering strategies that can be brittle to part-count variation and geometric mismatch.Source: §B.3 · p. 20

C. Applications

  • Named part decompositions can support dataset taxonomy normalization, robotics and embodied AI, scene understanding, reconstruction, and 3D content editing.The paper identifies affordance-aligned names as interfaces for manipulation and task planning, and named components as handles for targeted edits.Source: C. Applications · p. 20
Loading 2512.18003v2…