PaperlayerView exact revision on arXiv

Source-linked AI summary

HoloPart: Generative 3D Part Amodal Segmentation

Yunhan Yang, Yuan-Chen Guo, Yukun Huang, Zi-Xin Zou, Zhipeng Yu, Yangguang Li, Yan-Pei Cao, Xihui Liu

arXiv:2504.07943v1cs.CV

TL;DR

3D part amodal segmentation asks how to recover complete semantic parts when standard segmentation sees only visible surface patches. The paper uses existing segmentation followed by HoloPart, a diffusion-based part-completion model, and reports superior completion and full-task results while identifying input-mask quality as a limitation.

  • Problem

    3D part segmentation identifies visible surface patches rather than complete semantic parts, limiting its usefulness for 3D content creation tasks.

  • Method

    The paper uses a two-stage pipeline that obtains incomplete segments with existing 3D segmentation and completes them with HoloPart, a diffusion-based model.

  • Results

    HoloPart significantly outperforms existing shape completion approaches and achieves superior results when integrated with existing segmentation techniques for full 3D part amodal segmentation.

  • Takeaways & Limitations

    The work establishes 3D part amodal segmentation benchmarks on ABO and PartObjaverse-Tiny and demonstrates a practical route to complete-part prediction.

  • Takeaways & Limitations

    HoloPart's outcome is influenced by input surface-mask quality, and unreasonable or low-quality masks may produce incomplete results.

Abstract

from arXiv · show

3D part amodal segmentation--decomposing a 3D shape into complete, semantically meaningful parts, even when occluded--is a challenging but crucial task for 3D content creation and understanding. Existing 3D part segmentation methods only identify visible surface patches, limiting their utility. Inspired by 2D amodal segmentation, we introduce this novel task to the 3D domain and propose a practical, two-stage approach, addressing the key challenges of inferring occluded 3D geometry, maintaining global shape consistency, and handling diverse shapes with limited training data. First, we leverage existing 3D part segmentation to obtain initial, incomplete part segments. Second, we introduce HoloPart, a novel diffusion-based model, to complete these segments into full 3D parts. HoloPart utilizes a specialized architecture with local attention to capture fine-grained part geometry and global shape context attention to ensure overall shape consistency. We introduce new benchmarks based on the ABO and PartObjaverse-Tiny datasets and demonstrate that HoloPart significantly outperforms state-of-the-art shape completion methods. By incorporating HoloPart with existing segmentation techniques, we achieve promising results on 3D part amodal segmentation, opening new avenues for applications in geometry editing, animation, and material assignment.

1. Introduction

The paper introduces 3D part amodal segmentation to recover complete semantic parts from incomplete surface patches, and proposes HoloPart within a practical two-stage pipeline. Benchmarks on ABO and PartObjaverse-Tiny support strong performance for part completion and the full task.

  • Existing 3D part segmentation produces surface patches, which limits direct use in geometry editing, animation, and material assignment.
  • 3D part amodal segmentation separates a 3D shape into complete semantic parts, including geometry hidden by occlusion.
  • The proposed pipeline first obtains incomplete part segments with existing segmentation and then completes each part's 3D shape.
  • HoloPart is a diffusion-based completion model that uses learned 3D shape priors to generate plausible complete geometry from incomplete part segments.
  • The authors establish evaluation benchmarks on ABO and PartObjaverse-Tiny for 3D part completion and amodal segmentation.
  • HoloPart significantly outperforms existing shape completion approaches and achieves superior results when integrated with existing segmentation techniques.

2. Related Work

Prior 3D part segmentation methods rely on supervised learning or 2D foundation models projected onto 3D surfaces, but occlusion leaves their outputs incomplete. Shape completion research provides related reconstruction strategies, while this work targets complete individual parts within larger shapes.

  • Earlier 3D part segmentation methods learn 3D representations with fully supervised training that requires extensive manual annotations.
  • Limited dataset scale and diversity make conventional 3D part segmentation challenging in open-world scenarios.
  • Recent methods use 2D foundation models to segment rendered views and project the resulting masks onto 3D surfaces.
  • Because of occlusions, projected methods segment only visible surface areas, producing incomplete segmentations for downstream tasks.
  • Learning-based, encoder-decoder, patch-prior, and generative approaches have been explored for completing shapes from partial inputs.

3. 3D Part Amodal Segmentation

3D part amodal segmentation decomposes a shape into complete semantic parts, including occluded geometry, while enforcing geometric and semantic consistency. The proposed pipeline obtains incomplete surface segments, then uses HoloPart to complete them with pretrained shape priors and dual attention.

  • Task definition: 3D part amodal segmentation decomposes a shape into complete semantic parts, including portions occluded in the input.Each part should represent the entire geometry while remaining geometrically and semantically meaningful.
  • Two-stage approach: The pipeline first obtains incomplete surface segments and then generates corresponding complete parts with HoloPart.The first stage uses an existing segmentation method; the second stage performs part completion.
  • Model design: HoloPart adapts a pretrained 3D shape representation and diffusion model to complete parts from incomplete segments.Object-level pretraining learns generalizable shape representations and semantic correspondences before part-completion finetuning.
  • Model design: Context-aware and local attention preserve global shape consistency while capturing fine-grained part details and positional information.The resulting conditioning information is injected into the part diffusion model through cross-attention.
  • Data curation: The training pipeline creates whole-part pairs by processing component meshes, visibility, non-watertight geometry, and dataset-specific part distributions.The data sources include ABO and filtered Objaverse shapes, with additional processing for mesh validity and semantic distribution.
  • Evaluation: Table 1 evaluates ABO 3D part amodal completion using Chamfer Distance, IoU, F-Score, and Success Rate across completion baselines and HoloPart variants.The comparison includes PatchComplete, DiffComplete, Finetune-VAE, and HoloPart with or without context-aware attention.

4. Experiments

Experiments evaluate HoloPart on ABO and PartObjaverse-Tiny, including comparisons with completion baselines, zero-shot integration, ablations, and downstream applications. HoloPart consistently produces high-quality complete parts across diverse shapes and supports editing, material assignment, animation, geometry processing, and super-resolution.

  • 4.1. Experimental Setup: The benchmarks use ABO and PartObjaverse-Tiny, with 20,000 and 160,000 training parts respectively, while evaluation uses 60 ABO shapes and 200 PartObjaverse-Tiny objects.ABO covers four household-object categories; PartObjaverse-Tiny spans eight categories and contains 3,000 annotated parts.
  • 4.2. Main Results: HoloPart significantly outperforms PatchComplete, DiffComplete, and finetuned VAE on ABO, especially for parts with large missing regions or thin structures.The baselines struggle with bed sheets, lamp connections, bedsteads, and chair interiors, whereas HoloPart generates coherent parts.
  • 4.2. Main Results: HoloPart consistently outperforms competing methods on PartObjaverse-Tiny across Chamfer Distance, IoU, F-Score, and Success Rate.The method also completes intricate details such as an eyeball, strawberry, and house features that other methods fail to reconstruct.
  • 4.3. Zero-shot Generalization: By combining HoloPart with SAMPart3D, the system performs zero-shot 3D part amodal segmentation on generated meshes.The pipeline first segments visible surfaces and then generates complete, consistent parts.
  • 4.3. Ablation Study: Ablations show that context-aware attention supports invisible-area completion and global consistency, while local attention preserves detail and positional mappings.Guidance scale also materially affects generated-shape quality: small values provide insufficient control, whereas excessively large values can cause reconstruction failure.
  • 4.4. Application: Completed parts support geometry editing, material assignment, animation, reasonable remeshing, part-aware data creation, and geometric super-resolution.The application examples include modifying car-part dimensions, assigning unique textures, enabling part motion, and preserving part details with equal token counts.

5. Conclusion

The paper introduces 3D part amodal segmentation and addresses it through diffusion-based part shape completion with local and context-aware attention. Benchmarks and application demonstrations support the effectiveness of the approach and establish a foundation for further research.

  • 5. Conclusion: The paper introduces 3D part amodal segmentation and proposes benchmarks based on ABO and PartObjaverse-Tiny.The task separates shapes into semantic parts with complete geometry.
  • 5. Conclusion: HoloPart uses local and context-aware attention mechanisms to complete parts coherently.The approach focuses on 3D part shape completion within the broader amodal-segmentation task.
  • 5. Conclusion: HoloPart significantly outperforms prior shape completion approaches, while evaluations and application demonstrations support the approach’s effectiveness.The paper positions these results as a foundation for future research in 3D part amodal segmentation.

6. Supplementary Material

The supplementary material details implementation settings, data filtering, ablations, qualitative results, and a limitation concerning input surface-mask quality.

  • Implementation Details: HoloPart uses a 24-block VAE, a 10-layer DiT part diffusion model, and an 8-block context-aware attention module.The part diffusion model uses hidden size 2048, while completion uses 512 tokens and context latents of dimension (512, 512).
  • Implementation Details: Training samples 20,480 points from the overall shape and 512 points from each segmented part, using a 1e-4 AdamW learning rate.The overall-shape points provide keys and values, while segmented-part points provide queries.
  • Data Preparation: Filtering rules constrain mesh counts to 2–15 and remove objects with severe fragmentation, excessive floaters, or highly imbalanced part volumes.The rules target objects with reasonable part-wise semantic distributions for dataset construction.
  • Ablation Analysis: HoloPart handles both semantic and instance part completion, while removing context-aware attention produces inconsistent and lower-quality completions.Semantic completion groups repeated parts such as chair legs, whereas instance completion treats them separately.
  • Qualitative Results: On generated meshes, completing segmented surface masks enables consistent mesh reconstruction and supports geometry editing, geometry processing, and material editing.Qualitative examples include intricate components such as glasses, hats, and headsets.
  • Qualitative Results and Limitations: HoloPart effectively completes PartObjaverse-Tiny part details while maintaining overall consistency, but poor input surface masks can yield incomplete results.The limitation motivates future use of generated 3D part-aware shapes for training part-aware generation models.
Loading 2504.07943v1…