Source-linked AI summary
DreamPartGen: Semantically Grounded Part-Level 3D Generation via Collaborative Latent Denoising
Tianjiao Yu, Xinzhuo Li, Muntasir Wahed, Jerry Xiong, Yifan Shen, Ying Shen, Ismini Lourentzou
TL;DR
DreamPartGen targets the limited semantic and relational grounding of existing part-aware text-to-3D methods. It introduces complementary geometry-appearance and language-relation latents that are co-denoised, achieving state-of-the-art geometric fidelity and text–shape alignment across benchmarks.
Problem
Existing text-to-3D methods overlook semantic and functional part structure, while part-aware methods remain geometry-focused and weakly model textual alignment and inter-part relations.
Method
DreamPartGen jointly co-denoises Duplex Part Latents for part geometry and appearance with Relational Semantic Latents encoding language-derived attributes and inter-part relations.
Results
DreamPartGen achieves state-of-the-art performance across four benchmarks, improving geometric fidelity by ↓60% CD / ↓41% EMD and text–shape alignment by ≥↑20% CLIP/ULIP.
Takeaways & Limitations
Explicitly modeling semantically grounded part latents supports coherent, interpretable, controllable 3D synthesis and part-centric applications.
Abstract
from arXiv · showhide
Understanding and generating 3D objects as compositions of meaningful parts is fundamental to human perception and reasoning. However, most text-to-3D methods overlook the semantic and functional structure of parts. While recent part-aware approaches introduce decomposition, they remain largely geometry-focused, lacking semantic grounding and failing to model how parts align with textual descriptions or their inter-part relations. We propose DreamPartGen, a framework for semantically grounded, part-aware text-to-3D generation. DreamPartGen introduces Duplex Part Latents (DPLs) that jointly model each part's geometry and appearance, and Relational Semantic Latents (RSLs) that capture inter-part dependencies derived from language. A synchronized co-denoising process enforces mutual geometric and semantic consistency, enabling coherent, interpretable, and text-aligned 3D synthesis. Across multiple benchmarks, DreamPartGen delivers state-of-the-art performance in geometric fidelity and text-shape alignment.
1 Introduction
DreamPartGen addresses the lack of semantic and relational grounding in part-aware text-to-3D generation by representing parts and their language-derived relations explicitly. Its collaborative latent diffusion framework improves geometric fidelity and text–shape alignment.
- Research gap: Text-to-3D prompts specify both object parts and relations such as attachment, symmetry, and relative placement.These relations matter for controllable generation, part editing, and articulated synthesis.
- Research gap: Existing monolithic latents entangle geometry, appearance, and semantics without explicitly representing part identities or inter-part relations.
- Research gap: Recent part-aware methods improve geometric granularity but remain vulnerable to segmentation noise and treat parts as geometrically isolated units.Language is therefore largely non-operational in these frameworks.
- Approach: DreamPartGen uses Duplex Part Latents for part geometry and appearance, Relational Semantic Latents for text-derived attributes and relations, and synchronized co-denoising.Learnable identifiers preserve part slot identity across diffusion timesteps and instances.
- Reported outcomes: ↓60% CD / ↓41% EMD and ≥↑20% CLIP/ULIP summarize DreamPartGen’s reported improvements in geometric fidelity and text–shape alignment.The framework also supports relational part editing, articulated generation, and mini-scene synthesis.
2 Related Work
Prior text-to-3D work progressed from score-distillation methods toward explicit 3D representations, while part-level methods added decomposition and part supervision. These approaches established part-aware generation but left room for stronger relational and semantic modeling.
- Text-to-3D generation: Early methods such as DreamFusion, ProlificDreamer, and LucidDreamer use score distillation sampling with 2D diffusion priors to generate 3D assets.They often suffer from low fidelity and poor multi-view consistency.
- Text-to-3D generation: Later approaches incorporate differentiable rendering and explicit representations such as Gaussian splatting, voxels, or meshes to improve stability and geometric realism.
- Part-level 3D generation: Part-level methods introduce segmentation, assembly, decomposition, dual 2D/3D priors, or Gaussian splatting for articulated generation.Part supervision supports more controllable part-aware modeling across several formulations.
3 DreamPartGen Method
DreamPartGen formulates part-based 3D generation as collaborative diffusion between modular part latents and persistent language-derived semantic latents. Two-level synchronization aligns geometry, appearance, local semantics, and global inter-part constraints during denoising.
- Motivation: Prior part-level formulations improve local shape and texture but do not explicitly preserve text-derived semantics throughout denoising.DreamPartGen keeps relational semantic latents active rather than using text only as a one-shot condition.
- Duplex Part Latents: DPLs encode each semantic part with 3D geometry tokens, 2D appearance tokens, and a learnable identifier embedding.The identifiers prevent slot swapping and preserve part tracking across denoising.
- Duplex Part Latents: DPLs are permutation-robust, preserve slot persistence, support cross-object latent transfer, and remain modular for multi-part diffusion.
- Relational Semantic Latents: RSLs provide compact language-derived tokens for part attributes and inter-part relations, addressing global incoherence that can arise from independently synthesized parts.Global relational tokens encode relations such as support, attachment, symmetry, or articulation.
- Collaborative denoising: The model encodes part geometry and appearance with DPLs and text phrases with RSLs, then co-denoises them through intra-part and inter-part synchronization.Intra-part synchronization aligns geometry and appearance under local semantics; inter-part synchronization propagates context and enforces global relational constraints.
4 PartRel3D Dataset
PartRel3D extends geometric 3D data with canonicalized functional and spatial triplets that connect part geometry, appearance, and language. Its dense relational annotations provide supervision for assembly-level semantics.
- Dataset motivation: Existing datasets provide geometric diversity but limited semantic grounding, relational coverage, or consistent part correspondence.This limits training for models that must understand object assembly and part roles.
- Functional relations: PartRel3D augments objects with canonicalized triplets encoding functional dependencies such as support, attachment, and articulation.These triplets link parts through explicit semantic predicates.
- Spatial relations: Spatial triplets encode geometric and positional relations between indexed parts using interpretable assembly predicates.The vocabulary includes vertical, horizontal, containment, symmetry, alignment, proximity, and contact relations.
- Dataset scale: PartRel3D contains approximately 11K part-labeled objects across 175 categories, over 90K parts, and 300K canonicalized relational triplets.Each object averages 8.2 parts and 27 inter-part relations.
5 Experiments
DreamPartGen achieves strong geometric fidelity and text–shape alignment across benchmarks while producing more coherent part assemblies than prior methods. Ablations show that local and global relational semantics, together with part identity modeling, are important for stable, semantically consistent generation.
- Quantitative Results: ↓60% CD and ↓41% EMD on average across Objaverse, ShapeNet, ABO, and PartRel3D, with ↓27.2% average IoU relative to the strongest baseline.The results indicate improved geometric precision and stronger geometry independence for non-intersecting, composable parts.
- Quantitative Results: ≥20% better text–shape alignment at the object level and ≥35% at the part level than the strongest baseline on PartVerse.The evaluation covers both complete-object and individual-part descriptions.
- Qualitative Results: DreamPartGen preserves fine-grained geometry, inter-part relationships, and global structural constraints, whereas baselines show detached parts, spatial drift, tearing, holes, and broken attachments.The qualitative comparison spans diverse object categories and reports watertight meshes with intact intra-part connections for DreamPartGen.
- Ablations: Removing local semantic tokens raises CD to 0.781 and EMD to 5.764, while disabling the part identifier raises IoU to 0.438 and lowers ULIP-T to 0.091.These ablations show degradation in geometry, part separation, semantic alignment, and identity-consistent structure.
- Applications: DreamPartGen extends part-aware generation to coherent multi-object scenes by treating each object as a macro-part and preserving semantic slots for fine-grained components and part counts.The scene-generation examples respect part structure, spatial constraints, and global coherence.
6 Conclusion
DreamPartGen bridges geometric structure and semantic reasoning through collaborative part-latent denoising. By coupling DPLs with RSLs, it supports coherent, interpretable, controllable, and part-centric 3D synthesis.
- DreamPartGen is a part-aware text-to-3D framework that bridges geometric structure and semantic reasoning through collaborative part-latent denoising.
- Coupled DPLs and RSLs jointly model geometry, appearance, and inter-part relations.
- The framework enables coherent, interpretable, and controllable 3D synthesis.
- DreamPartGen supports relational part editing and compositional scene generation beyond single-object generation.
A PartRel3D Dataset
PartRel3D canonicalizes functional and spatial relations into a predicate vocabulary and validates generated relations geometrically. These relations become relational signals that attention can aggregate during training.
- Canonicalization: Functional metadata is converted into triplets, while unavailable relations are generated with a pretrained VLM from rendered views and part captions.
- Canonicalization: Free-form relational phrases are normalized through parsing relational clauses and mapping them to a spatial predicate vocabulary.
- Validation: Spatial triplets are validated by checking predicate-specific inequalities against axis-aligned bounding boxes computed from ground-truth part meshes.
- Validation: During training, triplets are embedded as relational semantic latents and aggregated through attention, which can down-weight inconsistent or noisy triplets.
B Implementation Details
DreamPartGen uses two-stage training on PartRel3D, combining semantic synchronization with joint part- and object-level denoising. Experiments use published evaluation protocols and report perceptual metrics defined for rendered and point-based representations.
- Training: DreamPartGen is trained in two stages using the relational PartRel3D dataset.
- Training: Stage one optimizes part latents with semantic synchronization, while stage two jointly fine-tunes the full model toward high-level semantic alignment.
- Architecture: The diffusion backbone uses a Transformer-based DiT with cross-attention, and the VAE is fine-tuned using PartVerse and PartNet.
- Evaluation: Perceptual evaluation reports r-FID/r-KID from multi-view renderings and P-FID/P-KID in PointNet++ feature space.
- Optimization: Training uses AdamW with learning rate 1 × 10^-4, cosine decay, and gradient clipping at 1.0 on four NVIDIA L40 GPUs.
C Additional Experiments
Additional experiments show strong perceptual, geometric, part-level, and text-shape results, with relational triplets providing complementary supervision. DreamPartGen also degrades less under rare-part and held-out-relation shifts, while prompt-only inference remains competitive.
- Perceptual Evaluation: DreamPartGen achieves the best perceptual performance across all four datasets, with HoloPart and PartCrafter alternating as strongest baselines by metric.
- Condition-wise Analysis: Spatial triplets provide the largest improvement over the text-only baseline among combined single-condition variants.
- Condition-wise Analysis: Text+FT+ST is competitive with and across several metrics nearly matches Text+Image+FT+ST without image input.
- Part-Level Generation: DreamPartGen consistently achieves the best per-part geometry quality across part-annotated datasets.
- Robustness to Relation Parsing: Prompt-only inference remains competitive, and the small Qwen2.5-VL versus GPT-5 gap suggests gains come from RSLs rather than parser-specific artifacts.
- Number of Local RSL Tokens: Performance stabilizes once the number of local RSL tokens Km reaches 16.
- Generalization Beyond Clean Part Decompositions: All methods degrade under OOD shifts, but DreamPartGen exhibits smaller performance drops than prior part-based baselines.
D.1 Mini-scene Generation
DreamPartGen generates small multi-object scenes from relational text prompts by treating objects as macro-parts while preserving their internal part structure.
- Each object is represented by aggregated DPLs and a scene-level relational graph derived from captions.The graph contains inter-object triplets describing spatial and functional relations.
- Scene semantic tokens guide object placement through cross-object attention, promoting spatial consistency while preserving internal structure.
- Objects are first sampled independently and then joined to synthesize the complete scene.
- Evaluation on 3D-Front and an Occluded subset reports CD↓, F-score↑, IoU↓, and inference runtime↓ per scene.
D.2 Articulated Object Generation
DreamPartGen models articulation through persistent part identities and transformations estimated between paired canonical poses, then reconstructs motion without retraining.
- Paired configurations of the same object provide opposite canonical poses for estimating per-part rigid transformations.Each transformation is represented as Ti = (Ri, ti).
- Part-identifier embeddings remain fixed during articulation estimation, linking corresponding parts across motion states.
- The resulting articulation field is applied to canonical part meshes and reassembled to reconstruct articulated objects.
- Reconstructed objects maintain structural integrity and semantic consistency across states without re-optimization or diffusion retraining.
- Part editing isolates target DPLs, freezes non-target slots and global relational context, then briefly re-synchronizes the edited result.
E Broader Impacts
DreamPartGen is positioned as a controllable source of structurally coherent, editable 3D assets for several application domains. The paper also identifies privacy, intellectual-property, and synthetic-media risks requiring responsible deployment.
- Part-level generation, composition, and editing are presented as relevant to robotics, simulation, virtual content creation, and digital twins.
- The framework is described as producing structurally coherent, fine-grained 3D assets directly from language.
- Additional qualitative results cover mini-scene generation, articulated objects, and part editing.
- Potential practical benefits include lowering the cost and expertise barrier for accurate, customizable 3D model creation.
- Deployment risks include privacy concerns, intellectual-property considerations, and misuse in synthetic media pipelines.
- The paper recommends oversight involving provenance, data consent, attribution, and domain-specific usage guidelines.