PaperlayerView exact revision on arXiv

Source-linked AI summary

PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding

Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, Hao Su

arXiv:1812.02713v1cs.CV

TL;DR

Part-level understanding matters for vision, graphics, and robotics, but fine-grained, hierarchical, instance-level 3D part recognition remains challenging. PartNet addresses this gap with a large-scale annotated dataset, three segmentation benchmarks, and a novel instance-segmentation method that outperforms existing baselines.

  • Problem

    Fine-grained, hierarchical, and instance-level 3D part recognition is challenging, despite its importance for vision, graphics, robotics, manufacturing, assembly, and interaction applications.

  • Method

    PartNet provides 573,585 fine-grained part annotations for 26,671 shapes across 24 categories and establishes fine-grained semantic, hierarchical semantic, and instance segmentation benchmarks.

  • Results

    The authors benchmark four state-of-the-art algorithms for fine-grained semantic segmentation, propose three hierarchical-segmentation baselines, and report that their novel instance-segmentation method outperforms existing baselines.

  • Takeaways & Limitations

    PartNet offers a platform for part-level object understanding and supports future work on richer annotations, shape grammars, articulation, and robotic learning.

  • Takeaways & Limitations

    Hierarchical segmentation methods face trade-offs: top-down inference accumulates prediction errors, while ensemble inference improves performance but adds hyper-parameters and training time.

Abstract

from arXiv · show

We present PartNet: a consistent, large-scale dataset of 3D objects annotated with fine-grained, instance-level, and hierarchical 3D part information. Our dataset consists of 573,585 part instances over 26,671 3D models covering 24 object categories. This dataset enables and serves as a catalyst for many tasks such as shape analysis, dynamic 3D scene modeling and simulation, affordance analysis, and others. Using our dataset, we establish three benchmarking tasks for evaluating 3D part recognition: fine-grained semantic segmentation, hierarchical semantic segmentation, and instance segmentation. We benchmark four state-of-the-art 3D deep learning algorithms for fine-grained semantic segmentation and three baseline methods for hierarchical semantic segmentation. We also propose a novel method for part instance segmentation and demonstrate its superior performance over existing methods.

1. Introduction

PartNet addresses the need for large-scale, richly structured 3D part annotations by providing fine-grained, hierarchical, instance-level data and benchmarks for three segmentation tasks. Its structure exposes challenges for existing methods, especially recognizing small, heterogeneous, and semantically similar parts.

  • Existing datasets offer either relatively few annotated instances or coarse, non-hierarchical annotations, limiting fine-grained part understanding.
  • PartNet provides fine-grained, hierarchical, instance-level 3D part information on top of ShapeNet.
  • Expert-defined hierarchical templates organize part concepts selected for well-defined, consistent, compact, atomic, and complete annotations.
  • The paper benchmarks four semantic-segmentation algorithms, proposes three hierarchical-segmentation baselines, and introduces a novel instance-segmentation method that outperforms an existing baseline by a clear margin.
  • PartNet reveals that coarse-part algorithms struggle with small fine-grained parts, geometrically similar but semantically different parts, and heterogeneous shape variation.
  • 573,585 fine-grained part annotations cover 26,671 shapes across 24 object categories in PartNet.
  • The paper establishes fine-grained semantic, hierarchical semantic, and instance segmentation as three part-level benchmarking tasks.

2. Related Work

Prior 3D part datasets and methods advanced shape segmentation but remained limited in scale, granularity, hierarchy, or annotation coverage. PartNet addresses these gaps while connecting part understanding to functionality and articulation.

  • Earlier Benchmarks: Early shape-part benchmarks relied on qualitative evaluation or small collections of manually annotated meshes.One benchmark compared five mesh segmentation algorithms using 11 surface meshes, while another used 380 meshes across 19 categories.
  • Related Directions: Prior work also studied fine-grained grouping, labeling, language-related part properties, functionality, and articulation using smaller datasets.PartNet assigns consistent semantic labels that entail functionality and articulation information.
  • PartNet: PartNet contributes 573,585 fine-grained and hierarchical part annotations across 26,671 models and 24 object categories.This scale exceeds the related datasets summarized in the passage.

3. Data Annotation

PartNet constructs hierarchical part annotations with expert-defined templates, guided annotation, and 3D mesh operations. The design balances consistent reusable concepts with coverage of heterogeneous object structures and exceptions.

  • Annotation Interface: The annotation process uses expert templates, a single-thread GUI, 66 trained professional annotators, and at least one verification pass per annotation.Average annotation time is 8 minutes per shape.
  • Expert-Defined Part Hierarchy: Experts define category-specific hierarchical templates using well-defined, consistent, compact, hierarchical, atomic, and complete part concepts.Templates proceed from coarse semantic parts to fine-grained primitive-level components and use And-nodes for decomposition and Or-nodes for subcategorization.
  • Expert-Defined Part Hierarchy: Template design addresses ambiguity by examining broad object variation, refining structures after annotation, and collecting multiple annotations on reserved shapes.The authors state that even deep templates cannot cover all cases, so annotators may add out-of-scope parts.
  • Annotation Interface: A depth-first question-answering workflow traverses the template, records subpart counts or subtypes, and captures leaf-part geometry in 3D.Textual definitions and visual examples support annotator understanding of part specifications.
  • Annotation Interface: The interface accelerates annotation by exposing original mesh groupings and providing mesh cutting tools for finer segmentation.Annotators can select predefined pieces or split coarser pieces into smaller parts.
  • Annotation Interface: Compared with prior interfaces, PartNet directly supports fine-grained hierarchical instance collection on 3D models rather than only 2D annotation or existing over-segmentations.Direct 3D operation helps expose subtle details hidden in 2D renderings.

4. PartNet Dataset

PartNet provides large-scale, fine-grained, hierarchical, instance-level annotations across diverse 3D shapes. Its statistics also characterize part counts, hierarchy depth, and a multiply annotated consistency subset.

  • Dataset Statistics: 26,671 shapes across 24 object categories contain 573,585 fine-grained hierarchical instance-level part annotations.Most models and categories come from ShapeNetCore, with three added categories and seven augmented categories from 3D Warehouse.
  • Dataset Statistics: PartNet templates have a median hierarchy depth of 3 and a maximum depth of 7.The reported depth treats the root node as depth 0.
  • Dataset Statistics: Each shape contains a median of 14 part instances and a maximum of 230 part instances.These statistics quantify the fine-grained structure represented in the dataset.
  • Annotation Consistency: A subset of 771 shapes receives multiple annotations to study annotation consistency.This subset provides repeated labels for consistency analysis.

5. Tasks and Benchmarks

PartNet benchmarks fine-grained, hierarchical, and instance-level part understanding while exposing challenges from small, similar, and heterogeneous parts. Experiments compare established segmentation methods and a proposed instance method under geometry-based evaluation settings.

  • Tasks and benchmarks: PartNet benchmarks fine-grained semantic segmentation, hierarchical semantic segmentation, and instance segmentation.The study evaluates four state-of-the-art semantic segmentation algorithms, proposes three hierarchical baselines, and introduces a novel instance segmentation method.
  • Data preparation: Parts requiring texture or material information are excluded because evaluation considers parts fully determined by shape geometry.Examples include cabinet-door glass and microwave buttons; rarely appearing parts are also removed because insufficient samples hinder training and evaluation.
  • Experimental setup: 10,000 points per model and a 70%:10%:20% train-validation-test split define the common experimental input and partition.Only 3D coordinates are used as neural-network inputs, and shapes with multiple human annotations are excluded from experiments.
  • Fine-grained semantic segmentation: Fine-grained segmentation requires distinguishing small and visually similar parts, motivating features that capture local geometry and global context.Door handles average 77 of 10,000 points, while beds contain several geometrically similar part types.
  • Fine-grained semantic segmentation: All four semantic segmentation algorithms lose substantial performance from coarse to fine-grained levels, with small-part predictions remaining difficult.PointCNN qualitative results particularly show failures on door handles, stair steps, and bed-frame horizontal bars.
  • Hierarchical semantic segmentation: The hierarchical ensemble performs slightly better, especially for structurally varied categories, but requires more hyperparameters and longer training.It jointly considers predictions from multiple levels, whereas top-down inference accumulates errors.
  • Instance segmentation: The proposed instance method produces more robust and cleaner complete-instance predictions than SGPN by learning category-level structural priors.SGPN can produce partial or noisy instances because its post-processing depends on thresholding hyperparameters.

6. Conclusion

PartNet is a large-scale benchmark for fine-grained, hierarchical, and instance-level 3D shape segmentation. Its dataset and tasks support current evaluations while opening directions for richer part annotations, shape synthesis, articulation, and robotic learning.

  • PartNet contains 573,585 part annotations across 26,671 ShapeNet models from 24 object categories.
  • The dataset supports fine-grained semantic, hierarchical semantic, and instance segmentation benchmarks, including a novel instance method that outperforms an existing baseline.
  • The supplementary material provides dataset visualizations, statistics, hierarchical template details, and task architectures and training details.

B. More Dataset Visualization and Statistics

Supplementary visualizations and statistics characterize PartNet’s annotation scale, hierarchical structure, semantic diversity, and heterogeneous shape variation across categories.

  • PartNet visualizations show fine-grained instance-level annotations and heterogeneous shape variation within object categories.
  • Hierarchical annotations are visualized as tree structures with associated 2D part renderings at tree nodes.
  • Statistics report annotation counts, unique shapes, multiply annotated shapes, part-semantic counts, part-instance counts, and hierarchy depths by category.Figures 9–12 summarize these dataset properties using counts and maximum or median values.

C. More Template Design Details and Visualization

PartNet’s hierarchical templates are designed to cover shared, fine-grained part semantics while refinement procedures address annotation ambiguity and coverage limits. The appendix reports template statistics, annotation consistency, and supporting visualizations.

  • C.1. Template Design Details: Templates are designed using part concepts that are well-defined, consistent, compact, hierarchical, atomic, and complete.The design also uses online references, language-related shape studies, and shape-space visualization to cover variation and semantics.
  • C.1. Template Design Details: Template statistics report maximum and median part-instance counts and tree depths for each object category.These statistics are presented separately for part instances and hierarchy depth.
  • C.1. Template Design Details: Annotators may define parts outside the templates because uncommon object parts cannot all be covered in advance.User-defined parts account for 1.3% of annotated part instances, and 13.1% of shapes contain user-defined labels.
  • C.1. Template Design Details: Templates labeled 26,671 of 27,260 collected shapes, achieving at least 97.8% coverage; the remaining 2.2% were mainly excluded for data-quality or processing issues.The reported causes include poor mesh quality, classification errors, and mesh-splitting errors.
  • C.1. Template Design Details: PartNet templates encode coarse and fine-grained semantics down to primitive-level parts, while shared labels support correspondences across shapes and categories.Examples include chair back vertical bars, bed base surface panels, legs, and wheels.
  • C.2. Template Refinement Details: Fine-grained annotation suffers from boundary, granularity, and part-labeling ambiguity, including unclear bottle boundaries and competing chair-part labels.Similar concepts may apply to the same physical part, such as chair seat support versus chair base connector.
  • C.2. Template Refinement Details: The refinement process uses multiple annotations and confusion matrices to merge highly confused concepts, remove unclear segmentation, and reduce ambiguity.The confusion analysis is conducted at leaf-node level, using complete path labels as histories.
  • C.2. Template Refinement Details: After refinement, average data consistency reaches 83.3%, a 13.5% improvement over raw annotation.Table 6 compares average diagonal confusion scores and standard deviations before and after refinement.

D. Tasks and Benchmarks

The appendix supplements the benchmark description with additional evaluation metrics, specifically shape mIoU and shape mAP, and reports quantitative results using them.

  • D. Tasks and Benchmarks: Additional evaluations use shape mean Intersection-over-Union and shape mean Average-Precision.These metrics complement the benchmark results described elsewhere in the paper.

D.1. Fine-grained Semantic Segmentation

Fine-grained semantic segmentation is evaluated with multiple point-cloud networks, category-specific training, and shape-level metrics across segmentation levels. The appendix also specifies inputs and metric definitions.

  • D.1. Fine-grained Semantic Segmentation: The benchmark follows original PointNet, PointNet++, SpiderCNN, and PointCNN architectures with selected training modifications.The modifications address fine-grained part counts, point sampling, memory limits, and fair geometric-only inputs.
  • D.1. Fine-grained Semantic Segmentation: Category-specific networks are trained at each segmentation level because PartNet contains up to 480 part semantics across categories.The earlier coarse ShapeNet Part setting contained 50 parts across 16 categories.
  • D.1. Fine-grained Semantic Segmentation: Point clouds contain 10,000 points to preserve sampling of small fine-grained parts such as door handles and buttons.This exceeds the 1,000-, 2,000-, or 4,000-point inputs commonly used in the original papers.
  • D.1. Fine-grained Semantic Segmentation: Shape mIoU averages per-shape IoUs over part semantics present in ground truth or predictions, excluding ground-truth unlabeled points.Part-category mIoU instead averages performance over part semantics, so the metrics weight shapes and part categories differently.
  • D.1. Fine-grained Semantic Segmentation: The four algorithms are evaluated at coarse, middle, and fine-grained levels, with unavailable levels omitted.Shape mIoU results are reported by category and level, then averaged across levels and categories.
  • D.1. Fine-grained Semantic Segmentation: PointNet++ achieves the best Shape mIoU, whereas PointCNN achieves the best Part-category mIoU.The distinction reflects different weighting of shapes versus part semantics, especially under imbalanced part counts.

D.2. Hierarchical Semantic Segmentation

Hierarchical segmentation is benchmarked with bottom-up, top-down, and ensemble strategies that incorporate tree structure during training, inference, or both. The methods achieve similar performance, with the ensemble slightly ahead.

  • D.2. Hierarchical Semantic Segmentation: The bottom-up method trains on leaf parts and aggregates child predictions to obtain parent-node predictions.Parent semantics are therefore implicitly encoded through leaf definitions and the known hierarchy.
  • D.2. Hierarchical Semantic Segmentation: The top-down method trains a multi-label classifier over all tree nodes and predicts coarser nodes before finer nodes.Its loss increases the scores of all ground-truth labels along a point’s hierarchical path.
  • D.2. Hierarchical Semantic Segmentation: The ensemble method trains separate networks at coarse, middle, and fine-grained levels to learn level-specific features.Coarse-level signals can distinguish parts with similar local geometry, such as chair back and chair arm vertical bars.
  • D.2. Hierarchical Semantic Segmentation: At inference, path voting averages log-likelihoods along each root-to-leaf path and selects the highest-scoring path.This combines prediction scores from networks trained at multiple segmentation levels.
  • D.2. Hierarchical Semantic Segmentation: The three methods show similar shape mIoU performance, with the ensemble method performing slightly better.The comparison uses shape mIoU over relevant hierarchical part semantics.

D.3. Instance Segmentation

This section details training considerations and evaluation of PartNet part instance segmentation, including the Shape mAP metric and qualitative visualizations. The proposed method is more memory-efficient and faster to train than SGPN, while improving Shape mAP performance.

  • Architecture and Training Details: SGPN requires batch size 1 because its 10,000 × 10,000 similarity score matrix consumes excessive GPU memory.SGPN uses two-stage training before jointly optimizing semantic segmentation, similarity scores, and confidence scores.
  • Architecture and Training Details: The proposed method is more memory-efficient and trains faster than SGPN.Both methods are trained until convergence.
  • Shape mAP Metric and Results: Shape mAP evaluates part instance segmentation by averaging mean AP across test shapes within each object category.For each shape, AP is averaged over part semantics present in the ground truth or prediction, then averaged across test shapes.
  • Shape mAP Metric and Results: The proposed method shows a clear Shape mAP improvement over SGPN.Table 9 reports the comparison under Shape mAP scores.
  • Visualization: The appendix visualizes fine-grained and hierarchical instance-level segmentation annotations across the 24 object categories.Hierarchical visualizations distinguish And-nodes with solid lines and Or-nodes with dashed lines.
  • Visualization: Template visualizations cover the object categories used for structured part representations.The templates use solid lines for And-nodes and dashed lines for Or-nodes.
Loading 1812.02713v1…