Source-linked AI summary
From One to More: Contextual Part Latents for 3D Generation
Shaocong Dong, Lihe Ding, Xiao Chen, Yaokun Li, Yuxin Wang, Yucheng Wang, Qi Wang, Jaehyeok Kim, Chenjian Gao, Zhanpeng Huang, Zibin Wang, Tianfan Xue, Dan Xu
TL;DR
Holistic 3D latents, limited explicit part modeling, and global-only controls constrain detailed multi-part generation. CoPart jointly denoises contextual part latents with mutual guidance and bounding-box conditions, trained alongside a large Objaverse-derived part dataset. The authors report improved generation quality and part accuracy, with applications in editing, articulated objects, and scene composition.
Problem
Method
CoPart represents objects with contextual part latents and uses mutual guidance to synchronize parts and geometry with appearance, with bounding-box conditions for generation.
Source: §1 · p. 2, §3.2 · p. 4Results
Takeaways & Limitations
Part-based representation supports editing, articulated-object generation, mini-scene generation, and long part-sequence sampling without further training.
Source: §4 · p. 6Incorrect user-provided bounding boxes can cause assembly errors such as mesh clipping.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Recent advances in 3D generation have transitioned from multi-view 2D rendering approaches to 3D-native latent diffusion frameworks that exploit geometric priors in ground truth data. Despite progress, three key limitations persist: (1) Single-latent representations fail to capture complex multi-part geometries, causing detail degradation; (2) Holistic latent coding neglects part independence and interrelationships critical for compositional design; (3) Global conditioning mechanisms lack fine-grained controllability. Inspired by human 3D design workflows, we propose CoPart - a part-aware diffusion framework that decomposes 3D objects into contextual part latents for coherent multi-part generation. This paradigm offers three advantages: i) Reduces encoding complexity through part decomposition; ii) Enables explicit part relationship modeling; iii) Supports part-level conditioning. We further develop a mutual guidance strategy to fine-tune pre-trained diffusion models for joint part latent denoising, ensuring both geometric coherence and foundation model priors. To enable large-scale training, we construct Partverse - a novel 3D part dataset derived from Objaverse through automated mesh segmentation and human-verified annotations. Extensive experiments demonstrate CoPart's superior capabilities in part-level editing, articulated object generation, and scene composition with unprecedented controllability.
1. Introduction
3D-native latent diffusion retains geometric supervision, but holistic latents, weak part modeling, and global-only controls limit detailed multi-part generation. CoPart instead jointly generates contextual part latents, using part-specific representations and bounding-box conditioning, and is trained on PartVerse, a large Objaverse-derived part dataset.
1. Introduction
- Earlier multi-view methods use 2D image supervision but omit accurate 3D shape supervision, while CLAY uses 3D occupancy auto-encoding to preserve geometric supervision.The passage presents 3D-native latent diffusion as a shift from multi-view rendering approaches.Source: §1 · p. 2
- Earlier 3D latent diffusion methods use equal token counts for simple and intricate meshes, risking information loss in complex shapes and allowing simpler geometries to dominate training.The introduction also notes that methods generally lack explicit part representations and relationships, and that global conditions provide little local control.Source: §1 · p. 2
- Part-based modeling distributes complexity across parts, supports learning part-level information, and provides detailed part-level control.The paper motivates this design by contrasting it with holistic generation, which can fuse a person’s head and hat into one low-resolution region.Source: §1 · p. 2
- CoPart’s bottom-up approach learns from diverse Objaverse data and directly generates coherent parts, unlike category-restricted or image-segmentation-driven approaches.The paper says top-down methods depend heavily on segmentation quality and may not leverage part information during training.Source: §1 · p. 2
- CoPart generates coherent objects from contextual part latents, with geometry and image tokens per part and bounding-box conditions for generation.Mutual guidance exchanges information across parts and between each part’s geometry and image tokens; text descriptions can also condition generation.Source: §1 · p. 2
- PartVerse contains 91k parts from 12k objects, built from Objaverse through automated segmentation, manual filtering and grouping, and generated part prompts.The process addresses inconsistent Objaverse part labels, including over- and under-segmentation.Source: §1 · p. 3
- CoPart is reported to outperform prior work with higher-quality objects and more accurate parts, while supporting part editing, articulated-object generation, and scene composition.The paper describes resampling selected part latents for editing, using structure diffusion for articulated objects, and treating scene objects as parts for mini-scene generation.Source: §1 · p. 3
2. Related Work
Related work moves from image-based and reconstruction approaches toward 3D-native diffusion, while part-generation research spans structural models, part-aware latents, and reconstruction. CoPart is positioned as a bottom-up approach that jointly learns and generates coherent parts from diverse data.
2.1. 3D Generation
- Recent 3D-native diffusion models, including CLAY and follow-ups, achieve significantly improved performance, following SDS-based 3D generation, multi-view diffusion, and single-image reconstruction approaches.Earlier category-specific methods [2] [6] [10] [14] [18] [35] [47] [52] are contrasted with diverse text- or image-conditioned contemporary models.Source: §2.1 · p. 3
2.2. Part Generation
- Part-generation work includes structural and part-aware latent models, but many rely on category-specific data; image-segmentation-based reconstruction also depends on segmentation quality and limited patch information.CoPart is contrasted as a bottom-up strategy that learns part distributions from diverse data and jointly generates coherent parts.Source: §2.2 · p. 3
3. Synchronized Part Latent Diffusion
CoPart addresses the difficulty of generating consistent, high-quality parts and ambiguity in their ordering through synchronized diffusion over geometry and image latents. It combines part-level representations, mutual guidance, global and box conditioning, and refinement with pretrained models.
3. Synchronized Part Latent Diffusion
- CoPart targets part consistency, efficient use of limited part data, and part-order ambiguity with synchronized part latent diffusion.The framework also uses pretrained foundation models and conditions to address ordering and controllability.Source: §3 · p. 3
3.1. Part Representation Encoding
- Each part is represented by geometric tokens from a 3D part VAE and appearance tokens from multi-view images encoded by an image VAE.The geometric VAE is fine-tuned on part data, while the image tokens use a pretrained image VAE.Source: §3.1 · pp. 3–4
3.2. Synchronized Diffusion
- CoPart fine-tunes pretrained 3D and image diffusion models to generate geometry and image latents, synchronizing parts and modalities.Inter-part synchronization links distinct parts; intra-part synchronization links geometry and appearance within a part.Source: §3.2 · p. 4
3.2.1. Mutual Guidance
- Mutual guidance uses attention to exchange information across parts and between 3D and 2D features during denoising.The 2D branch also attends across views of the same part to support multi-view consistency.Source: §3.2.1 · pp. 4–5
3.2.2. Global Guidance
- A shared-parameter global branch jointly denoises holistic latents and provides global supervision while distinguishing global and part branches with learnable embeddings.The passage states that this supervision regularizes fine-tuning and prevents significant deviation from pretrained weights.Source: §3.2.2 · p. 5
3.2.3. Part Guidance Encoding
- Part-level text prompts distinguish parts, while bounding-box conditions constrain their locations in both diffusion branches.Boxes are encoded as meshes using the pretrained 3D mesh VAE and injected into geometry through cross-attention; rendered box wireframes enter the 2D global branch through ControlNet.Source: Figure 2 · p. 4, §3.2.3 · p. 5
3.3. Refinement
- A stage-II enhancer uses generated part images and geometry, replacing its original voxel input with detailed part voxels extracted from generated meshes.The described pipeline enhances parts and assembles them using an inverse transformation.Source: §3.3 · pp. 5–6
3.4. Optimization Loss
- Both the 3D and image branches are supervised with diffusion denoising loss.The passage identifies the denoiser as operating for 3D and 2D and the noise as Gaussian.Source: §3.4 · p. 6
4. Applications
CoPart's part representation supports editing, articulated-object generation, mini-scene generation, and longer part sequences. These applications use part-level resampling and box-guided generation, with some parts fixed while others are replaced.
4. Applications
- CoPart supports part-based editing, articulated-object generation, mini-scene generation, and long-sequence sampling without further training.The applications use the same part-based representation for different generation tasks.Source: §4 · p. 6
4.1. Part-based Editing
- For editing, selected parts are resampled while the remaining parts are encoded as contextual latents and kept unchanged.The preserved parts can still inform generation of the edited parts.Source: §4.1 · pp. 6–7
4.2. Articulated Object Generation
- Articulated-object generation combines externally generated bounding boxes and articulation relationships with CoPart-generated parts guided by text prompts.The approach also allows articulation information to be manually defined for each part.Source: §4.2 · p. 7
4.3. Mini-scene and Long Sequence Generation
- Mini-scenes are generated from specified boxes and text, while longer sequences extend beyond the training limit of eight parts by fixing some latents and replacing others.The longer-sequence strategy reuses the editing approach to avoid the GPU memory constraint during training.Source: §4.3 · p. 7
5. PartVerse Dataset
PartVerse is a diverse, annotated 3D part dataset built from Objaverse. Its processing pipeline combines mesh segmentation, human review, and part-level text descriptions.
5. PartVerse Dataset
- PartVerse is a diverse 3D part dataset comprising 91k high-quality parts from 12k objects with detailed text descriptions; PartNet contains only 24 categories of daily objects.Source: §5 · p. 7
- The dataset pipeline segments meshes, uses human post-correction, and generates text captions for part-level data.Reviewers refine segmentation by merging over-segmented parts or splitting under-segmented ones; captions describe appearance, shape, and part-whole relationships.Source: Figure 6 · p. 8, §5 · p. 8
6. Experimental Evaluation
The evaluation reports stronger qualitative generation than leading holistic methods, especially for small parts, and describes quantitative comparisons and ablations. The reported ablations identify contributions from global guidance and joint image-geometry refinement.
6. Experimental Evaluation
- CoPart outperforms state-of-the-art holistic methods qualitatively, particularly in the quality of small parts.The authors attribute this result to its part-based representation.Source: §6 · p. 9
- Quantitative comparisons and a 51-participant user study are reported as highlighting the approach's ability to produce decomposable, high-quality 3D assets.The passage does not provide the specific table values.Source: §6 · p. 9
- Ablations report that global guidance improves part coherence, especially appearance, and that refinement benefits from both part images and geometries.The two modalities together help the enhancer comprehend part shapes and optimize the parts.Source: §6 · p. 9, Figure 7 · p. 9
7. Discussion of Part Assembly
Refined part meshes are relocated to bounding boxes for assembly, but incorrect user-provided boxes can cause mesh clipping.
7. Discussion of Part Assembly
- After refinement, inverse-normalized part meshes are relocated to 3D bounding boxes for assembly.The paper says its bounding-box injection strategy generally ensures effective combinations.Source: §7 · p. 9
- Incorrect user-provided bounding boxes can cause assembly errors such as mesh clipping.The authors identify optimizing user-provided boxes as future work.Source: §7 · p. 9
8. Conclusion
CoPart combines mutual guidance and 3D box conditions for high-quality, diverse 3D part generation. The paper reports that it outperforms state-of-the-art results and introduces a larger-scale 3D part-aware dataset collected from Objaverse.
8. Conclusion
- CoPart uses mutual guidance for coherent part-latent denoising and 3D box conditions to eliminate part ambiguity.Source: §8 · p. 9
- The paper reports that CoPart outperforms state-of-the-art results.Source: §8 · p. 9
- The authors introduce a larger-scale 3D part-aware dataset collected from Objaverse for use in various tasks.Source: §8 · p. 9
- The authors discuss limitations of CoPart in the supplementary material.Source: §8 · p. 9