Source-linked AI summary
MultiCube: Compositional 3D Generation With Part-Level Semantic and Spatial Control
Ava Pun, Kangle Deng, Yiheng Zhu, Jun-Yan Zhu, Maneesh Agrawala, Tinghui Zhou
TL;DR
Existing compositional 3D generation methods lack precise control over each part's semantics and spatial arrangement, limiting their fit for professional workflows. MultiCube addresses this with per-part text schemas and bounding boxes in a two-stage diffusion pipeline, generating a monolithic shape before simultaneous part decomposition. The method produces high-quality compositional meshes with precise part-level control, while severely mis-specified boxes can yield strange shapes and parts can occasionally collide.
Problem
Existing compositional 3D methods provide limited control over the number, semantics, and spatial arrangement of individual parts needed in professional workflows.
Method
MultiCube conditions two-stage diffusion on a global prompt, part labels, and per-part bounding boxes, using a Part Layout Adapter before simultaneous part decomposition.
Results
MultiCube generates high-quality compositional 3D objects with precise semantic and spatial part-level control, including distinct layouts difficult to achieve with text prompting alone.
Takeaways & Limitations
Independent per-part semantic and spatial conditioning supports controllable multi-part mesh generation for varied compositional 3D objects.
Takeaways & Limitations
Severely mis-specified bounding boxes may produce strange shapes, and the model occasionally generates colliding parts; it also lacks iterative local editing.
Abstract
from arXiv · showhide
Digital 3D objects used in games and animation are often required to be compositional; that is, decomposed into semantically meaningful parts. Recent 3D generation methods can produce high-quality compositional objects conditioned on image or text prompts. Yet, such global conditioning lacks the precise part-level controllability required for professional creative workflows. To address this, we introduce MultiCube, a novel compositional 3D generation method that provides explicit, independent control over both the semantics and spatial arrangement of each part. MultiCube takes as input a global text prompt, a text schema specifying the desired parts, and a spatial layout indicating the bounding boxes of the parts in the given schema. It outputs a 3D object composed of distinct meshes, one per specified part, that adhere to the given semantic and spatial conditions. Our approach employs a two-stage diffusion process, first generating a schema- and layout-aligned monolithic mesh, then decomposing the mesh into individual parts simultaneously. A novel Part Layout Adapter is used to encode per-part conditions independently of the other parts. Experiments demonstrate that our method can generate high-quality compositional 3D objects with precise part-level control, including those with unique layouts difficult to achieve with text or image prompting alone. Project page: https://multi-cube.github.io
1. Introduction
MultiCube addresses the limited part-level controllability of compositional 3D generation by independently specifying part semantics and spatial layouts. It uses a two-stage process to produce coherent, distinct meshes aligned with these conditions.
- Motivation: Compositional 3D methods often cannot let users specify the number, semantics, or detailed arrangement of individual parts.Some methods generate arbitrary parts, while others control only the total number of parts; text, images, and 2D layouts also struggle with complex spatial arrangements.
- Motivation: MultiCube enables flexible, open-vocabulary part semantics and independent location and dimension control for each part.These capabilities are intended to support fine-grained control over the generated shape.
- MultiCube: MultiCube takes a global text prompt, a text schema of desired parts, and per-part bounding boxes, producing one distinct mesh for each specified part.The meshes form a coherent object aligned with the global prompt and multi-box layout, and layouts can be specified manually or through an LLM.
- MultiCube: A two-stage diffusion process first generates a schema- and layout-aligned monolithic mesh, then decomposes it into specified parts simultaneously.Cross-part attention supports global consistency during decomposition.
- MultiCube: The Part Layout Adapter processes each part separately to encode semantic and spatial conditions without entangling geometric constraints.This provides a strong per-part conditioning signal for monolithic shape generation.
2. Related Work
Prior 3D generation work has explored 2D lifting, 3D-native representations, part-aware generation, and spatial conditioning. MultiCube differs by targeting precise semantic and spatial control at the individual-part level.
- 3D Shape Generation: 2D-to-3D methods use image diffusion supervision, but view-dependent supervision provides only partial structural constraints.3D-native methods instead use latent spaces or sparse voxel grids for shape generation.
- Part-level 3D Generation: Early part-level methods learned compositional latent spaces, while later approaches used segmentation-then-reconstruction pipelines to improve part-aware generation.The earlier latent approaches often struggled with detailed and diverse geometries.
- Part-level 3D Generation: Recent direct part-aware methods support joint generation or arbitrary part counts but provide limited control beyond specifying the total number of parts.MultiCube is presented as providing precise control over both spatial and semantic part attributes.
- Spatially-controlled 3D Generation: Global bounding-box conditioning controls an object's overall dimensions and scale but does not control its internal structure or individual components.This motivates finer-grained spatial guidance for compositional 3D generation.
3. Controllable Compositional 3D Generation
MultiCube adds explicit per-part semantic and spatial control to compositional 3D generation through a two-stage diffusion pipeline. It independently encodes part conditions, generates a layout-aligned monolithic shape, and then decomposes it into coherent part meshes.
- Problem formulation: MultiCube targets compositional 3D generation conditioned on a global prompt, part labels, and per-part bounding boxes.The method extends CubePart, a VecSet diffusion model that generates multi-part objects from text prompts and part labels but lacks spatial controllability.
- Stage 1: Monolithic Generation: The Part Layout Adapter independently fuses each part label with its corresponding bounding box into localized part latents.Labels are encoded with Qwen-VL, boxes receive Fourier embeddings, and a Q-Former produces the part-latent sequence.
- Stage 1: Monolithic Generation: Stage 1 generates a monolithic mesh containing the specified semantic parts in their prescribed positions.The monolithic shape model receives the global text condition together with concatenated per-part conditions.
- Stage 2: Multi-part Generation: Stage 2 decomposes the monolithic shape into specified parts using semantic labels, spatial embeddings, and cross-part attention for global coherence.Spatial priors help distinguish parts with identical or similar labels, while a lightweight embedding supplies each part’s bounding-box condition.
- Training: Fine-tuning starts with the adapter alone and then jointly trains both stages, using the same flow-matching objective for each stage.The staged optimization preserves pre-trained shape priors while adapting the model to spatially guided generation.
- Ablation results: Ablations show that naive or entangled encodings can cause overlapping parts and artifacts, while removing Stage 2 spatial conditions can produce missing parts.The full method improves part alignment and slightly improves geometric quality in most evaluated cases.
4. Experiments
Experiments evaluate MultiCube on compositional 3D generation using part-level semantic and spatial metrics, ablations, external baselines, and application-oriented tests. The method strongly improves part-level accuracy and alignment while remaining competitive holistically, supporting customization, articulation, and scene generation.
- Evaluation Protocol: Evaluation uses Chamfer Distance, F-score, and bounding-box IoU to measure geometric quality and part-layout alignment.Metrics are computed at part and holistic levels where applicable, with shapes normalized to a [−1, 1]3 box and unordered parts greedily matched to references.
- Multi-part Generation: MultiCube strongly outperforms all baselines on part-level metrics while remaining competitive on holistic-level metrics.Qualitative comparisons show cleaner part decompositions and more detailed geometry than the evaluated baselines.
- Ablation Study: MultiCube’s ablation compares naive part-condition encoding, entangled adapter encoding, and removal of Stage 2 spatial conditions against the full design.The study evaluates whether semantic and spatial conditioning choices affect part-level alignment and geometric quality.
- Ablation Study: Naive or entangled encodings produce overlapping parts and artifacts, while removing Stage 2 spatial conditions can cause missing parts.The full method greatly improves part alignment and slightly improves geometric quality in most cases.
- Full Pipeline Evaluation: Against FullPart and OmniPart, MultiCube matches FullPart on layout compliance while winning on semantic alignment and geometric quality.Unlike the baselines, MultiCube uses text and layout alone, whereas the comparison baselines additionally receive ground-truth renderings and OmniPart receives segmentation information.
- Applications: MultiCube remains robust to small bounding-box inaccuracies and can preserve unmodified components during targeted part-level customization.Users can modify schemas or layouts to add, remove, retarget, or resize parts, although each operation triggers full regeneration.
- Applications: Explicitly disentangled, semantically labeled meshes are immediately usable for rigid-body articulation through assigned kinematic properties and pivot points.The decomposition is presented as reducing manual post-processing for downstream animation workflows.
5. Discussion and Limitations
MultiCube provides controllable multi-part mesh generation, but remains limited by mis-specified layouts, occasional part collisions, and the lack of iterative local editing.
- MultiCube generates compositional 3D meshes with part-level semantic and spatial control, surpassing baselines.The authors describe this as a strong step toward fully controllable 3D asset creation.
- Severely mis-specified bounding boxes can produce strange generated shapes, while occasional collisions can occur between parts.The method is robust to small bounding-box perturbations, but severe errors remain problematic; collision-avoidance losses are proposed as future work.
- MultiCube generates the full output at once and does not support modifying individual parts without regenerating the entire assembly.The authors identify local, iterative editing as an important extension for production use.
A. Additional Evaluation Details
The additional evaluation details describe automated and human comparisons of multi-part 3D models, including their inputs, spatial constraints, semantic alignment, and geometric quality criteria.
- VLM Evaluation: The evaluation uses VLM head-to-head comparisons of randomized multi-view renders to assess part semantic alignment and geometric quality.Parts are distinguished by solid colors, and each model is evaluated using four multi-view renders.
- Human Evaluation: Human evaluators compare interactive 3D models using part labels and text prompts to assess semantic alignment, spatial alignment, and geometric quality.The interface presents models side-by-side in random order.
- Human Evaluation: 27 participants rated 136 randomly ordered comparison pairs, with each pair receiving between one and three ratings.Participants came from graphics, vision, and machine learning research, and method identity was concealed.
- Baseline Inputs: Baseline methods receive varying inputs, including part labels, shapes, voxelized shapes, bounding boxes, 2D images, and 2D segmentations.OmniPart receives ground-truth part segmentation, which enhances its decomposition input.
- Layout Generation: The layout-generation prompt specifies normalized 3D coordinates, bounding-box dimensions and centers, adjacency, symmetry, and proportionality rules.The output represents each part with dimensions [size_x, size_y, size_z] and center [x, y, z].
- Evaluation Criteria: The VLM semantic criterion asks whether colored parts clearly represent their assigned labels, while the geometric criterion checks boundaries, detail, and artifacts.Final judgments are output as Model A, Model B, or Tie.