PARTFIELD: Learning 3D Feature Fields for Part Segmentation and Beyond

Minghua Liu; Mikaela Angelina Uy; Donglai Xiang; Hao Su; Sanja Fidler; Nicholas Sharp; Jun Gao

Source-linked AI summary

PARTFIELD: Learning 3D Feature Fields for Part Segmentation and Beyond

Minghua Liu, Mikaela Angelina Uy, Donglai Xiang, Hao Su, Sanja Fidler, Nicholas Sharp, Jun Gao

arXiv:2504.11451v1Apr 15, 2025cs.CV

TL;DR

Limited scale and diversity in 3D part-annotated datasets, along with slow per-shape optimization in prior open-world methods, motivate a more general and efficient approach. PARTFIELD learns continuous 3D feature fields feedforward from diverse 2D and 3D data, improving class-agnostic segmentation quality and speed while supporting hierarchical and cross-shape applications.

Problem
Limited scale and diversity in 3D part-annotated datasets restrict generalization, while predefined templates or text prompts do not cover inconsistent or language-ambiguous parts and prior open-world methods require lengthy per-shape optimization.
Method
PARTFIELD predicts a continuous 3D feature field feedforward, training with triplet-based contrastive learning that distills diverse 2D and 3D part proposals without predefined part templates or text names.
Results
PARTFIELD improves class-agnostic part segmentation by more than 20% while being an order of magnitude faster, with feature-field prediction taking less than one second and clustering taking a few seconds.
Takeaways & Limitations
The learned features support hierarchical part decomposition and exhibit consistency across shapes, enabling downstream applications such as correspondence and co-segmentation.
Takeaways & Limitations
The method is evaluated at object scale, cross-shape applications are small-scale, and those applications require consistently oriented shapes because the feature space is weakly correlated with 3D position.

Abstract

from arXiv · show

We propose PartField, a feedforward approach for learning part-based 3D features, which captures the general concept of parts and their hierarchy without relying on predefined templates or text-based names, and can be applied to open-world 3D shapes across various modalities. PartField requires only a 3D feedforward pass at inference time, significantly improving runtime and robustness compared to prior approaches. Our model is trained by distilling 2D and 3D part proposals from a mix of labeled datasets and image segmentations on large unsupervised datasets, via a contrastive learning formulation. It produces a continuous feature field which can be clustered to yield a hierarchical part decomposition. Comparisons show that PartField is up to 20% more accurate and often orders of magnitude faster than other recent class-agnostic part-segmentation methods. Beyond single-shape part decomposition, consistency in the learned field emerges across shapes, enabling tasks such as co-segmentation and correspondence, which we demonstrate in several applications of these general-purpose, hierarchical, and consistent 3D feature fields. Check our Webpage! https://research.nvidia.com/labs/toronto-ai/partfield-release/

1. Introduction

PARTFIELD learns a template-free, hierarchical notion of 3D parts through a continuous feature field predicted in one feedforward pass. It improves class-agnostic segmentation quality and speed while supporting multiple modalities and cross-shape applications.

Motivation: 3D part segmentation is limited by scarce, diverse annotations, while prior open-world methods often require lengthy per-shape multi-view optimization.These pipelines render and segment multiple views before fusing them into 3D, taking minutes to hours at inference.
Motivation: PARTFIELD avoids predefined templates and text names, learning parts across semantic, geometric, mobility, and granularity criteria.This supports multiple valid part decompositions rather than enforcing one fixed notion of a part.
Approach: Given a 3D shape, PARTFIELD predicts a continuous feature field whose distances encode part membership and whose clustering yields hierarchical decompositions.Features can be queried continuously at any location and reused for downstream applications.
Approach: PARTFIELD trains with contrastive learning on unconstrained 2D and 3D part proposals, bringing same-part samples closer than distinct-part samples.The proposals may use different semantics, geometries, or granularities, enabling training from diverse large-scale data.
Results and Applications: More than 20% performance improvement and an order-of-magnitude speed advantage are reported for class-agnostic part segmentation.The learned field also supports hierarchical decomposition, cross-modal inputs, co-segmentation, selection, and correspondence.
Results and Applications: The model learns a versatile part concept across modalities, and its features are consistent across shapes despite no explicit cross-shape supervision.The authors identify correspondence and co-segmentation as downstream uses of this emergent consistency.

2. Related Work

Related work includes supervised and open-world 3D segmentation methods, but existing feedforward fine-grained models remain limited by closed-domain training and weak open-world generalization.

Prior Approaches: Traditional methods predefine part templates and rely on supervised datasets whose limited scale and diversity restrict generalization.Recent open-world methods increasingly leverage 2D image foundation models to reduce reliance on 3D annotations.
Positioning: PARTFIELD is among the first feedforward models to learn a feature field for open-world part decomposition.The cited comparison contrasts it with fine-grained models trained on closed-domain datasets.

3. Method

PARTFIELD learns a continuous, hierarchical 3D part feature field from diverse 2D and 3D part proposals using feedforward contrastive learning. The field supports multi-scale clustering, fast inference across 3D modalities, and consistent downstream representations.

Feature-field representation: PARTFIELD predicts a continuous feature field from a 3D shape, with feature distances indicating whether points belong to the same part.The field can be queried at arbitrary spatial locations and clustered into part-aware decompositions.
Training data: Part proposals from diverse 2D and 3D sources need not follow predefined templates, semantics, or granularity.Image proposals are obtained by rendering multi-view RGB and normal images, applying 2D foundation models, and projecting masks back onto the shape.
Contrastive objective: Triplet-based contrastive learning encourages two points within a proposal to be more similar than an outside point, supporting multi-scale parts without explicit scale conditioning.Hard negatives are sampled near part boundaries in Euclidean or feature space to improve training efficiency.
Evaluation: Qualitative comparisons on PartObjaverse-Tiny include Find3D, PartSLIP, Ultrametric Feature Field, SAMesh, and SAMpart3D.Each color denotes a different predicted part.
Feedforward architecture: The feedforward 3D model replaces per-shape optimization with a single network evaluation, improving inference speed and robustness to noisy or inconsistent proposals.It produces a complete feature field that extends into the shape interior.
Architecture and inference: The model accepts point samples from meshes, point clouds, or Gaussian particles and represents the output feature field with a triplane.Clustering methods such as k-means or agglomerative clustering can convert sampled features into part decompositions.

4. Experiments

PARTFIELD is evaluated on open-world class-agnostic part segmentation and related cross-shape applications. It combines large-scale 2D and 3D proposals with feedforward feature-field inference, producing accurate, fast, hierarchical, and cross-shape-consistent decompositions.

Evaluation Setup: Training uses approximately 340k Objaverse shapes with 2D SAM2 proposals and around 30k PartNet shapes with hierarchical 3D labels.Six RGB and six normal images per Objaverse shape generate multi-granularity proposals; PartNet contributes labels from all hierarchy levels.
Evaluation Setup: PARTFIELD is compared on PartObjaverse-Tiny and PartNetE against five recent open-world part-segmentation baselines using instance-level mean IoU.The evaluation aligns predictions to input mesh faces when required and selects the best result across scales for multi-scale methods.
Segmentation Results: 22.3% over the second-best method is the PartObjaverse-Tiny improvement, while feature-field prediction takes less than one second per shape.Clustering adds only a few seconds, whereas most baselines require minutes and Ultrametric can exceed one hour.
Hierarchical Decomposition: PARTFIELD learns meaningful multi-scale hierarchies, initially grouping related components before splitting them into individual parts.Agglomerative clustering extracts the discrete hierarchy from the continuous feature field.
Cross-Shape Applications: Cross-shape feature consistency supports co-segmentation and point-to-point correspondence applications.The visualizations compare corresponding parts across shapes and transfer source-shape colormaps to target shapes using Functional Maps.
Ablations and Analysis: Hard negative mining improves performance and produces crisper part boundaries, with the full combination of triplets and hard negatives achieving the best overall performance.The ablation uses PartObjaverse-Tiny; additional 3D data provides a modest open-world gain despite smaller scale and limited category diversity.

5. Limitations and Future Work

The current architecture is fast but weakly tied to 3D position, constraining cross-shape use to consistently oriented shapes. Cross-shape applications are also small-scale and motivate further study.

Limitations: PARTFIELD’s extrinsic PVCNN and triplane architecture weakly correlates features with 3D position, so cross-shape applications require consistently oriented shapes.Part segmentation remains agnostic to this positional correlation.
Future Work: The demonstrated cross-shape applications are small-scale, and future work may extend the object-scale method to large scenes.The authors hope these applications motivate further research into part contrastive learning for foundational 3D shape features.

1. Details of Hard Negative Mining

Hard negative mining augments contrastive training with negatives selected near the query in 3D or feature space. The loss then contrasts positive-pair similarity against multiple negative samples.

Sampling Strategy: Each sampled part mask supplies N = 64 positive pairs, M1 = 256 random negatives, and M2 = 256 hard negatives.Masks come from SAM for 2D proposals or part-label masks for 3D proposals.
Sampling Strategy: 3D-hard negatives are sampled with probability weighted by the negative Euclidean distance from the query point.Feature-hard sampling uses the same strategy but computes distance in feature space.
Contrastive Objective: The triplet loss uses exponential cosine similarity with a learnable temperature to make same-part features more similar than negative features.The supplied equation fragments show symmetric similarity terms for the two positive-pair directions.

2. Details of Baseline Comparison

The baseline comparison covers text-based and text-agnostic methods with differing input modalities and inference procedures. Predictions are generated using released implementations and aligned to mesh faces where necessary.

Baseline Procedures: SAMPart3D predicts mesh-face labels across scales from 0.0 to 2.0 and selects the scale with the best metric for each shape.The evaluation uses its official codebase and released checkpoint.
Baseline Procedures: Find3D samples 5,000 points per shape, uses ground-truth text prompts, and transfers predicted point labels to mesh faces by nearest neighbor.The sampled points receive dummy white color before inference.
Baseline Procedures: SAMesh uses official code and dataset-specific settings, with PartNetE requiring refined hyperparameters because of low-poly meshes and poor topology.The authors report that default settings work reasonably on PartObjaverseTiny but fail on PartNetE.
Baseline Procedures: PartSLIP renders multi-view RGB and depth images, fuses them into a dense colored point cloud, and predicts labels conditioned on ground-truth part names.The evaluation uses the official zero-shot implementation and released checkpoint.
Baseline Procedures: Ultrametric renders multi-view images, distills multi-view SAM predictions through NeRF optimization, and is omitted from PartNetE because of lengthy runtime.The method follows the official code and default configuration.

3. Multiclass Regression Interactive Cosegmentation

PartField features support interactive cosegmentation across shapes through either clustering or lightweight user-supervised classification. User annotations can be propagated in real time using precomputed feature fields.

A small number of user clicks on any guitar shape can extend to real-time segmentations across all guitar shapes.This interactive cosegmentation uses the precomputed feature field for each shape.
User-annotated points can train a one-vs-rest multiclass logistic regression model over PartField features.The model maps features to segmentation labels instead of using clustering.
The logistic regression models are fit and evaluated on all shapes in real time on the GPU.This provides interactive feedback during annotation.

4. PartNetE Class Grouping

For reporting PartNetE results, the original 45 classes are grouped into five broader clusters. The grouping covers electronics, appliances, kitchen items, furniture, and tools or miscellaneous objects.

45 original PartNetE classes are grouped into five clusters to save space in reporting.
The five clusters include Electronics & Computing Devices and Large Home Appliances.Examples include keyboards, laptops, washing machines, and refrigerators.
The remaining clusters cover Kitchen & Food-Related Items, Furniture & Household Infrastructure, and Tools, Office Supplies, & Miscellaneous.Examples include kettles, tables, lamps, staplers, scissors, and suitcases.