Source-linked AI summary
PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers
Yuchen Lin, Chenguo Lin, Panwang Pan, Honglei Yan, Yiqiang Feng, Yadong Mu, Katerina Fragkiadaki
TL;DR
Existing 3D generators generally produce whole objects or rely on segmentation followed by reconstruction, limiting unified part-aware generation. PartCrafter jointly denoises multiple part-specific latent token sets from one RGB image using compositional representations and local-global attention. It reports stronger structured generation across objects and scenes, including invisible parts, while using a relatively small 50K-part training set.
Problem
Whole-object 3D generators lack part-level decomposition, while two-stage segmentation-and-reconstruction methods face segmentation errors, extra computation, and scaling difficulties.
Method
PartCrafter jointly denoises multiple part-specific latent token sets using a compositional architecture with identity-aware local-global attention, shared decoding, and part-annotated training data.
Results
PartCrafter achieves higher generation quality and efficiency than existing two-stage methods across part-level object generation and scene reconstruction, including invisible 3D structures.
Takeaways & Limitations
The results support structured 3D generation from a single image without segmentation input for both individual objects and complex multi-object scenes.
Takeaways & Limitations
PartCrafter is trained on 50K part-level examples, relatively small compared with the millions used for typical 3D object generation models.
Abstract
from arXiv · showhide
We introduce PartCrafter, the first structured 3D generative model that jointly synthesizes multiple semantically meaningful and geometrically distinct 3D meshes from a single RGB image. Unlike existing methods that either produce monolithic 3D shapes or follow two-stage pipelines, i.e., first segmenting an image and then reconstructing each segment, PartCrafter adopts a unified, compositional generation architecture that does not rely on pre-segmented inputs. Conditioned on a single image, it simultaneously denoises multiple 3D parts, enabling end-to-end part-aware generation of both individual objects and complex multi-object scenes. PartCrafter builds upon a pretrained 3D mesh diffusion transformer (DiT) trained on whole objects, inheriting the pretrained weights, encoder, and decoder, and introduces two key innovations: (1) A compositional latent space, where each 3D part is represented by a set of disentangled latent tokens; (2) A hierarchical attention mechanism that enables structured information flow both within individual parts and across all parts, ensuring global coherence while preserving part-level detail during generation. To support part-level supervision, we curate a new dataset by mining part-level annotations from large-scale 3D object datasets. Experiments show that PartCrafter outperforms existing approaches in generating decomposable 3D meshes, including parts that are not directly visible in input images, demonstrating the strength of part-aware generative priors for 3D understanding and synthesis. Code and training data will be released.
1 Introduction
PartCrafter addresses the lack of part-level decomposition in 3D generation by jointly generating structured 3D parts from a single image. It combines a compositional architecture with part-annotated data and reports stronger quality, efficiency, and reconstruction capabilities than existing approaches.
- Whole-object 3D generators do not support part-level decomposition, while segmentation-then-reconstruction pipelines introduce segmentation errors, computational costs, and scaling difficulties.
- The model uses structured latent spaces, identity-aware part-global attention, shared decoders, and a dataset curated from existing 3D assets with part annotations.
- PartCrafter achieves higher generation quality and efficiency across part-level object generation and scene reconstruction than existing two-stage methods.It also infers invisible 3D structures and surpasses its underlying object generative model in object reconstruction fidelity.
- PartCrafter jointly generates semantically meaningful 3D parts from image prompts without requiring segmentation input.Its compositional latent space binds each part to dedicated latent variables, supporting independent editing, removal, and addition.
- The contributions include structured generation for individual objects and complex multi-object scenes, together with ablations validating the model components.
2 Related Work
Prior 3D part and scene generation methods use varied representations and often depend on segmentation or other intermediate structures. PartCrafter targets these limitations with unified structured generation that can handle invisible parts without an additional segmentation model.
- 3D part-level object generation includes methods that assemble existing parts or generate part geometry and structure using diverse representations.These representations include point clouds, voxels, implicit neural fields, and parametric forms.
- Recent part-generation approaches leverage multi-view diffusion and 2D segmentation, while HoloPart segments meshes before refining part geometry with a pretrained 3D DiT.
- Existing part-level methods are limited by dataset scale or representation expressiveness and can depend heavily on segmentation quality.
- Prior object-composed scene generation extracts layouts, graphs, or segmentations before generating or retrieving objects under input conditions.
- MIDI requires image-visible components, whereas PartCrafter generates parts that are not visible in the conditioning image without an additional segmentation model.
3 PARTCRAFTER: A Compositional 3D Diffusion Model
PARTCRAFTER is a compositional 3D generative model that simultaneously produces multiple part-specific meshes from an RGB image. It combines part-wise latent representations, local-global attention, pretrained mesh-generation components, and part-annotated data to generate coherent structured assets.
- Model overview: Given an RGB image and a user-specified part count, PARTCRAFTER simultaneously generates multiple semantically meaningful 3D meshes in a shared canonical space.Each part is decoded as a separate mesh, allowing direct assembly without additional transformations.
- Pretrained foundation: PARTCRAFTER reassembles pretrained 3D mesh-generation encoders, decoders, and DiT blocks into a compositional multi-entity architecture.The model builds on TripoSG and leverages its SDF-based decoder for mesh generation.
- Compositional latent space: Each 3D part receives a dedicated set of latent tokens and a learnable identity embedding, while concatenated tokens represent the complete asset.Training shuffles part order to support permutation invariance.
- Local-global denoising transformer: Hierarchical local-global attention preserves internal part structure while modeling interactions across all parts.The architecture replaces the original attention modules in pretrained DiT blocks with the proposed part-global mechanism.
- Local-global denoising transformer: Image features are injected into both local and global attention to align part-level semantics with the overall image-conditioned composition.This dual-conditioning design uses DINOv2 features and cross-attention at both levels.
- Dataset: A curated dataset retains part annotations mined from Objaverse, ShapeNet, and ABO instead of flattening modular assets into single meshes.The source datasets provide part-level metadata suitable for learning compositional generation.
4 Experiments
Experiments evaluate PARTCRAFTER for part-level object generation, object-composed scene reconstruction, and architectural design choices. Results show strong reconstruction quality, invisible-structure recovery, and sensitivity to local-global attention, part identity control, and attention ordering.
- Evaluation Setup: PARTCRAFTER is evaluated against segmentation-then-reconstruction baselines for both part-level objects and object-composed scenes.The evaluation covers object and scene settings, including questions about invisible parts and architectural contributions.
- 3D Part-level Object Generation: PARTCRAFTER outperforms HoloPart on object-level and part-level metrics while generating decomposable meshes in seconds.It also surpasses the number-of-token-aligned TripoSG backbone on object-level metrics.
- Qualitative Results: Qualitative results show that PARTCRAFTER infers invisible parts and supports different granularities of part decomposition from the same image.The model is presented for both part-level object generation and scene generation.
- 3D Object-Composed Scene Generation: PARTCRAFTER outperforms MIDI in reconstruction fidelity without segmentation and maintains high generation quality under severe occlusion.MIDI slightly surpasses PARTCRAFTER in IoU, partly because MIDI uses ground-truth 2D segmentation masks.
- Ablations: Removing local attention collapses generation, whereas removing global attention produces overlapping geometry and loses part decomposition.The ablation identifies local-global attention as essential for meaningful 3D representations and global structural relationships.
- Ablations: Removing part identity embeddings causes collapse, while cross-attention in both local and global modules balances mesh fidelity and geometry independence.The selected design achieves the best overall performance among the tested control configurations.
- Ablations: Alternating local- and global-level attention yields the best trade-off between local and global metrics.The arrangement is intended to support information exchange and integration across both levels.
5 Conclusion
PARTCRAFTER reconstructs part-level objects and scenes without 2D or 3D segmentation, supporting 3D structural understanding within generation. Its main limitation is the relatively small 50K-part-level training set, while large-scale training may have environmental implications.
- PARTCRAFTER reconstructs part-level objects and scenes without relying on 2D or 3D segmentation information.
- The model validates the feasibility of integrating 3D structural understanding into the generative process.
- 50K part-level training examples are relatively small compared with the millions typically used for 3D object generation models.The authors suggest scaling DiT training with more, higher-quality data.
- Large-scale training may have environmental implications, although the authors support further exploration of 3D vision.
A Dataset Details
The curated dataset combines part annotations from Objaverse, ShapeNet-Core, and Amazon Berkeley Objects into a collection of approximately 50,000 part-labeled objects and 300,000 individual parts.
- The dataset comprises approximately 50,000 part-labeled objects and 300,000 individual parts collected from three existing 3D repositories.The sources are Objaverse, ShapeNet-Core, and Amazon Berkeley Objects; filtering excludes objects without textures and selects assets with fewer than 16 parts and maximum IoU below 0.1.
B Implementation Details
Implementation comparisons use TripoSG as the underlying model, adapted HoloPart for part-level object generation, and MIDI as the baseline for 3D scene generation.
- For part-level object generation, the implementation first uses TripoSG to generate a mesh and then applies HoloPart to produce a part-level object.SAMPart3D provides the 3D segmentation model used for this pipeline.
- MIDI is adopted as the baseline for 3D scene generation.
C Texture Generation
The method generates textures for part-level 3D objects using Hunyuan3D-2 and assigns them to corresponding parts through manipulated UV maps.
- Hunyuan3D-2 generates textures for the generated 3D part-level objects, with qualitative results presented in Figure 7.Known vertex-to-part assignments allow the generated UV map to be manipulated so textures correspond to their parts.
D More Results
The supplementary results show additional image-conditioned 3D part-level object generations and 3D object-composed scene generations by PARTCRAFTER.
- The supplementary visualizations cover both part-level objects and object-composed scenes.The paper directs readers to the supplementary files and project website for improved visualization.
- Figures 8–11 present more image-conditioned 3D part-level object generation results.
- Figure 12 presents more image-conditioned 3D scene generation results.