Source-linked AI summary
OmniPart: Part-Aware 3D Generation with Semantic Decoupling and Structural Cohesion
Yunhan Yang, Yufan Zhou, Yuan-Chen Guo, Zi-Xin Zou, Yukun Huang, Ying-Tian Liu, Hao Xu, Ding Liang, Yan-Pei Cao, Xihui Liu
TL;DR
Existing 3D generators commonly produce monolithic assets without editable part structure, while structured methods must balance independent components with coherent geometry. OmniPart addresses this with autoregressive mask-guided layout planning followed by spatially conditioned synthesis, achieving state-of-the-art part-aware generation and supporting editable downstream content creation.
Problem
Most 3D generative methods produce monolithic shapes without intrinsic part structure, limiting direct use for editing, animation, material assignment, and semantic understanding.
Method
OmniPart autoregressively plans variable-length 3D part bounding boxes from flexible 2D masks, then spatially conditions a pretrained holistic generator to synthesize all parts coherently.
Results
OmniPart achieves state-of-the-art performance in part-aware 3D generation while producing explicit, controllable, semantically meaningful part structures.
Takeaways & Limitations
The framework supports user-controlled part granularity and downstream applications including compositional editing, material assignment, animation, and geometry processing.
Abstract
from arXiv · showhide
The creation of 3D assets with explicit, editable part structures is crucial for advancing interactive applications, yet most generative methods produce only monolithic shapes, limiting their utility. We introduce OmniPart, a novel framework for part-aware 3D object generation designed to achieve high semantic decoupling among components while maintaining robust structural cohesion. OmniPart uniquely decouples this complex task into two synergistic stages: (1) an autoregressive structure planning module generates a controllable, variable-length sequence of 3D part bounding boxes, critically guided by flexible 2D part masks that allow for intuitive control over part decomposition without requiring direct correspondences or semantic labels; and (2) a spatially-conditioned rectified flow model, efficiently adapted from a pre-trained holistic 3D generator, synthesizes all 3D parts simultaneously and consistently within the planned layout. Our approach supports user-defined part granularity, precise localization, and enables diverse downstream applications. Extensive experiments demonstrate that OmniPart achieves state-of-the-art performance, paving the way for more interpretable, editable, and versatile 3D content.
1 INTRODUCTION
OmniPart addresses the limitations of monolithic 3D generation by separating controllable part-structure planning from detailed part synthesis. Its two-stage design uses flexible 2D masks to guide variable-granularity 3D layouts and spatial conditioning to generate coherent parts.
- Monolithic 3D generators often lack editable part structure, limiting compositional editing, animation, material assignment, and semantic understanding.
- OmniPart decouples high-level structural planning from detailed part synthesis in a unified two-stage framework.The design targets controllability, part coherence, and overall quality.
- Flexible 2D part masks guide an autoregressive planner to generate variable-length sequences of 3D bounding boxes without strict correspondences or explicit semantic labels.A Part Coverage Loss further encourages each predicted box to enclose its corresponding object part.
- A spatially-conditioned synthesis module generates all parts simultaneously within the planned layout by adapting a pretrained holistic 3D generator.Part representations use spatial regions, full-object context, and part-aware embeddings to promote semantic awareness and structural coherence.
- OmniPart generates explicit, controllable, semantically meaningful part structures and achieves state-of-the-art performance in part-aware 3D generation.The framework also supports downstream applications including fine-grained editing, material assignment, and animation.
2 RELATED WORK
Related work spans holistic 3D generation, compositional synthesis, and 3D part segmentation. Existing approaches use 2D diffusion or segmentation, autoregressive primitives, or multi-view reconstruction, but face fidelity, consistency, or annotation-scaling challenges.
- 2D diffusion-based methods generate multi-view images or optimize 3D scenes, then reconstruct geometry using multi-view consistency algorithms.
- PartGen segments and inpaints occluded regions across views before reconstructing parts, but multi-view inconsistency can reduce final geometric fidelity.
- 3D Part Segmentation: Traditional 3D part-segmentation methods rely on large-scale annotations, limiting scalability to diverse or open-world domains.Recent work instead uses 2D foundation models to segment projected views and lift masks onto 3D surfaces.
3 PART-AWARE 3D OBJECT GENERATION
OmniPart generates part-aware 3D objects by first planning variable-length part bounding boxes and then synthesizing spatially aligned part latents within that layout. It combines flexible 2D mask conditioning with a pretrained holistic 3D generator to support controllable decomposition and coherent part generation.
- OmniPart separates part structure planning from structured part latent generation in a two-stage pipeline.The framework builds on TRELLIS’s spatially structured sparse voxel latent representation.
- Controllable Structure Planning: The autoregressive planner generates an arbitrary number of 3D bounding boxes, with each box representing a meaningful object part.Boxes are tokenized into sequences with explicit beginning and end markers and sorted by minimum coordinates in z-y-x order.
- Controllable Structure Planning: Flexible 2D part masks guide variable-granularity decomposition without requiring one-to-one mask-box correspondences or explicit semantic labels.Visual features extracted from the input image are combined with learnable part embeddings before autoregressive prediction.
- Controllable Structure Planning: Part Coverage Loss encourages predicted boxes to be large enough to comprehensively enclose their corresponding parts.The design relies on the second-stage voxel-discarding mechanism to remove redundant voxels after ensuring broad initial coverage.
- Spatially-Conditioned Part Synthesis: The synthesis module aligns planned boxes with sparse voxel coordinates and generates all part latents while maintaining coherence across parts.It adapts TRELLIS’s pretrained rectified-flow generator and filters invalid or overlapping voxels using an additional validity dimension.
4 EXPERIMENTS
OmniPart is evaluated through bounding-box planning, part-aware and whole-object generation, qualitative comparisons, efficiency, and downstream applications. The experiments show controllable part decomposition, coherent high-quality outputs, and practical editing and processing uses.
- Evaluation Protocol: The evaluation uses 300 objects grouped by part count, covering planning, full-pipeline generation, and practical applications.The groups contain 0–5, 6–10, 11–15, and 16–50 parts, with proportional sampling for part-count diversity and category coverage.
- Planning Evaluation: Bounding-box planning is evaluated with BBox IoU, Voxel Recall, and Voxel IoU against ground-truth part layouts.The metrics measure box overlap, coverage of valid part voxels, and overall voxel-level consistency.
- Ablation: Removing coverage loss lowers voxel-level recall and IoU, while removing 2D masks weakens control over generated part size and placement.The ablation links coverage loss to downstream voxel coverage and 2D masks to structural controllability.
- Part-Aware 3D Content Generation: Compared with segmentation, completion, and direct part-generation baselines, OmniPart produces textured parts with low semantic coupling and high structural cohesion.Segmentation methods recover only surface-level masks, completion methods depend on segmentation quality, and PartGen exhibits lower geometric and semantic quality.
- Pipeline Results: The complete pipeline generates high-quality geometry and texture while enabling precise control over part granularity through 2D masks.Qualitative results show the input image and masks, planned bounding boxes, individual meshes, and the combined full-object mesh.
- Applications: Part-aware outputs support animation, mask-controlled and multi-granularity generation, independent material editing, and geometry processing without boundary artifacts.Applications include modifying part-specific textures and remeshing while preserving structural coherence.
5 CONCLUSION AND DISCUSSION
OmniPart concludes that explicit part modeling can produce controllable, semantically decoupled, and structurally coherent 3D assets. Its main scope boundary is the use of axis-aligned bounding boxes, which can excessively include surrounding regions in some cases.
- Conclusion: OmniPart plans part layouts with autoregressive bounding-box prediction and synthesizes all parts simultaneously using a spatially conditioned pretrained holistic generator.Flexible 2D masks guide controllable part granularity, while the synthesis stage generates detailed and coherent parts.
- Conclusion: The framework produces low semantic coupling, high structural cohesion, controllable part granularity, and support for animation, material editing, and geometric processing.The conclusion characterizes OmniPart as advancing interpretable and editable 3D assets.
- Limitation: The first stage currently uses axis-aligned bounding boxes, which can excessively include surrounding regions in some cases.This design simplifies training but constrains the precision of spatial part representation.