PaperlayerView exact revision on arXiv

Source-linked AI summary

CubePart: An Open-Vocabulary Part-Controllable 3D Generator

Yiheng Zhu, Kangle Deng, Jean-Philippe Fauconnier, Inaki Navarro, Daiqing Li, Ava Pun, Yinan Zhang, Peiye Zhuang, Xiaoxia Sun, Maneesh Agrawala, Kiran Bhat, Tinghui Zhou

arXiv:2605.28763v1cs.AI

TL;DR

Generative 3D models often lack application-specific semantic part structure. CubePart generates coherent multi-part meshes from global text and open-ended part schemas, enabling direct game-engine integration and consistent schema-controlled decomposition.

  • Problem

    Existing generative 3D models produce monolithic or arbitrarily decomposed meshes, limiting alignment with application-specific semantic part schemas for interactive assets.

  • Method

    CubePart combines a large open-vocabulary part-labeled dataset with a schema-driven two-stage generator that synthesizes global geometry before semantic part meshes.

  • Results

    CubePart outperforms all baselines across holistic and part-level metrics while modulating part identity and granularity according to the input schema.

  • Takeaways & Limitations

    The generated multi-part meshes can be integrated into game engines and driven by animation or behavior scripts without manual post-processing.

  • Takeaways & Limitations

    CubePart focuses on rigid-body decomposition and does not yet support skinned vertex weights for organic character mesh deformation.

Abstract

from arXiv · show

Interactive 3D assets used in games and simulation are typically decomposed into specific semantic parts to support animation, physics, and scripted behaviors, yet most generative 3D models produce either monolithic meshes or arbitrary part decompositions that cannot be aligned with application-specific requirements. We present CubePart, a generative framework for open-vocabulary, part-controllable 3D mesh generation that exposes part structure as an explicit inference-time control signal. Given a global text prompt and a user-defined parts schema expressed as an open-ended list of part names, our method generates a set of meshes - one per schema element - that assemble into a coherent object while respecting the specified semantic structure. To enable this capability, we introduce a scalable data pipeline to construct a large open-vocabulary, part-labeled 3D dataset, along with a two-stage generative architecture that separates global shape synthesis from part-level decoding. We demonstrate that the resulting assets can be directly integrated into game engines and driven by animation and behavior scripts without manual post-processing. Project Page: https://cubepart.github.io/

1 Introduction

CubePart addresses the need for 3D-native, schema-driven control in generative 3D modeling by generating coherent multi-part meshes from a global text prompt and an open-ended parts schema. It supports application-specific semantic decompositions through a scalable part-labeled dataset and a two-stage generative architecture.

  • Motivation: Interactive assets require semantic parts that match predefined schemas for animation, physics, simulation, and scripted behaviors.Game engines and interaction systems operate on specific components such as wheels, articulated character parts, or opening containers.
  • Problem: Existing workflows manually decompose geometry and assign labels, while generative models typically produce monolithic meshes or unsuitable arbitrary part decompositions.This manual process scales poorly with asset diversity and limits direct use in downstream applications.
  • Limitations: 2D grounding cannot specify hidden parts and cannot reliably provide the complete 3D semantic structure required for part control.A rear tail, for example, cannot be controlled from a single front-facing view, motivating a 3D-native interface.
  • CubePart: CubePart generates one coherent mesh per user-defined schema element from a global text prompt, enabling open-vocabulary, part-controllable 3D generation.The schema can vary by application, such as separating car doors for opening animations or a hood for engine access.
  • Method and Contributions: 462K assets and about 2M parts support CubePart’s data engine, which uses vision-language models and 3D-aware Set-of-Mark annotation for semantic grounding.The dataset is over 11 times larger than PartVerse-XL, and the framework uses a schema-driven two-stage architecture for globally coherent parts.

2 Related Work

Prior 3D generators improve visual quality or resolution but generally lack explicit semantic part decomposition. Part-aware methods introduce structured generation, yet existing approaches typically rely on fixed vocabularies, implicit structure, or costly multi-stage pipelines rather than user-defined open-vocabulary control.

  • 3D Generative Modeling: 2D-to-3D methods use pretrained 2D diffusion priors through Score Distillation Sampling, but view-dependent supervision weakly constrains 3D structure and semantic part decomposition.DreamFusion introduced SDS for optimizing implicit 3D representations, and follow-up methods adopted the paradigm to compensate for limited 3D data.
  • 3D Generative Modeling: 3D-native and sparse-voxel generators enable diffusion in 3D-aligned spaces, localized synthesis, and high-resolution detail, but typically produce monolithic meshes without semantic part control.This line of work became practical with large-scale datasets such as Objaverse and Objaverse-XL, while sparse voxel grids reduce dense voxelization costs.
  • Part-Aware 3D Generation: Early part-aware generators demonstrate decomposed shape synthesis using category-specific supervision with autoencoder-based or diffusion-based frameworks, but are restricted in scope.Representative methods include SPAGHETTI, Neural Template, SALAD, and DiffFacto.
  • Part-Aware 3D Generation: Recent part-generation pipelines combine multi-view image synthesis, 2D segmentation, and 3D reconstruction, while 3D-native methods address completion, unified synthesis, contact artifacts, or variable part counts.Examples include Part123, PartGen, HoloPart, PartCrafter, PartPacker, and AutoPartGen.
  • Part Controllability: Existing methods assume fixed or learned vocabularies or infer structure implicitly, whereas the proposed approach accepts an open-vocabulary part list at inference and aligns generated meshes to it.This alignment is intended to support direct integration with downstream animation and interaction pipelines.
  • Part Datasets: Part-aware generation depends on meaningful part-decomposed datasets, but existing open-vocabulary resources remain expensive to scale and limited in coverage.PartVerse and PartVerse-XL contain approximately 12k and 40k assets, respectively, while other datasets primarily contain monolithic meshes or closed-vocabulary labels.

3 Open-Vocabulary Part-Controllable 3D Generator

CubePart is a two-stage framework for open-vocabulary, part-controllable 3D generation: it first synthesizes a text-conditioned full mesh, then decodes that mesh into distinct parts specified by a user-defined schema. Schema-aware prompting and cross-part attention improve semantic alignment and inter-part communication while leveraging pretrained geometric priors.

  • Framework overview: CubePart separates generation into full-mesh synthesis and multi-part mesh generation conditioned on a global shape prompt and an explicit list of part names.The framework targets objects whose composing parts are defined by an open-ended text schema.
  • Stage 1: Full mesh generation: Schema-aware finetuning explicitly enumerates constituent parts because single-mesh generation may omit intended parts or disproportionately emphasize particular components.The finetuning dataset structures text prompts to enumerate the object’s constituent parts.
  • Stage 2: Multi-part generation: Stage 2 initializes from Stage 1 and adapts the model to output multiple part latents while preserving geometric priors learned during pretraining.A multi-part object is represented as a set of N parts, with each part encoded by latent tokens.
  • Stage 2: Multi-part generation: Part-aware prompts provide the full part list and identify a target part, helping the model interpret labels and determine segmentation boundaries.The prompt format is: “This object has the following parts: {list of all parts}. Target to segment: {target part name}.”
  • Stage 2: Multi-part generation: A zero-initialized cross-part attention block enables information exchange among part latents and full-shape latents, addressing overlapping or incomplete geometry from text-only global context.Four such blocks are inserted at the 1st, 5th, 9th, and 17th layers while retaining the pretrained model structure.

4 Dataset

CubePart introduces an approximately 462K-asset, 2.02M-part dataset for open-vocabulary part-based 3D generation, constructed through an automated pipeline combining artist segmentations with VLM priors. The pipeline filters and standardizes assets, clusters existing parts into semantically meaningful groups, and produces concise names suited to user queries.

  • Dataset scale and construction: The dataset contains approximately 462K assets and 2.02M parts, built by combining artist-provided segmentations with VLM priors and semantic naming.The data engine uses multi-view renders with Set-of-Mark overlays to cluster parts and assign names.
  • Dataset pipeline: The automated pipeline filters assets with 2–32 parts, removes defective meshes and scan artifacts, clusters and names parts, then converts meshes to watertight surfaces and samples points with normals.Postprocessing uses Dual Marching Cubes on a 512^3 unsigned distance field.
  • Part clustering and naming: VLM clustering merges over-segmented parts with related functions or logical relationships while preserving existing part boundaries, such as grouping rims, tires, and hubs into left and right wheels.Parts that already form coherent semantic units may remain singleton clusters.
  • Set-of-Mark rendering: Each asset is rendered from 14 orbital viewpoints as paired textured and part-colored images, combining semantic context with unambiguous part identification for VLM annotation.Markers share colors with their corresponding contours and solid-colored parts across both views.

5 Evaluations

CubePart is evaluated for schema-aligned mesh generation using part-level and holistic geometric metrics, qualitative comparisons, and ablations. The results show improved structural completeness, controllable part decomposition, and benefits from schema-aware fine-tuning, cross-part attention, and Stage 1 pre-training.

  • Ablation study: Schema-aware fine-tuning ensures that all requested parts are present and correctly generated, avoiding missing or incorrectly emphasized parts.Without fine-tuning, examples omit parts such as “Steering Wheel” or overemphasize parts such as “Exhaust Pipe.”
  • Ablation study: Stage 1 single-part generation provides global geometric pre-training that improves Stage 2 output quality at both part and holistic levels.This lets Stage 2 focus on inter-part boundaries and part-specific geometry rather than basic 3D structure.
  • Quantitative comparison: CubePart outperforms all compared baselines consistently on holistic and part-level Chamfer Distance and F-score evaluations.Evaluation uses PartObjaverse-Tiny, with CD and F-score computed for individual parts and concatenated holistic shapes.
  • Controllability: Varying the input schema accurately controls both the semantic identity and granularity of generated parts for a single mesh.Adding an explicit “fenders” part resolves ambiguity that otherwise causes fenders to merge with wheels.
  • Ablation study: Removing cross-part attention severely reduces part-level accuracy, while PartCrafter-style local-attention modifications perform worse than dedicated zero-initialized blocks.These findings highlight the importance of inter-part communication and preserving pre-trained geometric priors.

6 Application: Generating 3D Objects with Behaviors

CubePart’s schema conditioning enables generated 3D objects whose labeled parts can be directly controlled by Lua behavior scripts in an interactive gaming platform. Examples show driving, character, and flight behaviors supported by increasingly detailed, application-specific part schemas.

  • Behavior-Driven Generation: Schema conditioning designs part structures compatible with scripted behaviors, with Lua scripts directly controlling individual object parts in a gaming platform.The workflow applies behaviors to generated objects without requiring manual post-processing, using schema-specified part meshes as control targets.
  • Driving: A jellyfish car progresses from five driving parts to separately modeled body, headlights, exhaust pipe, and gun components for shooting.The initial schema includes the body and four wheels; later refinements add headlights, exhaust, and a gun with corresponding scripts.
  • Characters: Independently labeled robot, humanoid frog, and wizard parts support behaviors including arm extension, powered takeoff, laser emission, spinning, and prop interactions.The examples span characters with articulated limbs, sensory parts, weapons, and external props, demonstrating compatibility with varied structural and behavioral complexity.
  • Behavior-Driven Generation: The application pipeline decomposes input meshes into schema-defined parts before applying dynamic motions and visual effects.Stage 2 produces single-part meshes following the input schema, after which object behaviors are applied to the generated asset.
  • Flying: Separately segmented drone propellers enable asymmetric actuation for takeoff, hovering, and directional motion, while functional body components support blinking-light effects.The drone schema distinguishes propellers, landing gear, body shell, and lights so motion and visual effects can target individual parts.

7 Limitations and Future Work · Appendix

CubePart’s remaining challenges include rigid-only decomposition, boundary interpenetration, and unreliable spatial or positional referencing caused partly by ambiguous VLM-based labels. The appendix adds dataset-pipeline details, visual results, and animated results.

  • 7 Limitations and Future Work: The paper identifies several technical challenges that remain despite CubePart’s progress toward part-based control for game-ready assets.This frames the listed limitations as ongoing work rather than resolved capabilities.
  • 7 Limitations and Future Work: CubePart currently supports rigid-body decomposition but not skinned vertex weights for organic character deformation.Future work could predict skeletal rig weights alongside part geometry.
  • 7 Limitations and Future Work: Parts can still interpenetrate at boundaries, even when the input schema specifies disjoint parts.Cross-part attention reduces overlaps but does not eliminate them; Figure 11 shows failure cases.
  • 7 Limitations and Future Work: Open-vocabulary generation struggles to consistently interpret relative spatial identifiers such as “front-left” and “rear-right.”The dataset uses spatially aware naming, but inherits ambiguities from VLM-based labeling.
  • 7 Limitations and Future Work: Figure 11 also shows failures where complicated input geometry causes the model to drop input components.The same examples include overlap at contact points and misunderstandings of left-right relationships.
  • 7 Limitations and Future Work: VLM labeling can produce mirroring errors, confuse object-local coordinates with camera view-space, and miss occluded parts.These errors can cause symmetrical-part swaps or component misplacement along an axis.
  • Appendix: The appendix provides additional details about the dataset pipeline, visual results in Figure 14, and animated results in an attached video.These materials extend the paper’s documentation and demonstrations beyond the main discussion.

A Dataset Pipeline Details

This section details each stage of CubePart’s multi-part dataset pipeline and supplements the overview provided in the main paper.

  • Pipeline scope: The section describes the stages of the multi-part dataset pipeline.It focuses on detailed descriptions rather than introducing a separate pipeline overview.
  • Relationship to main paper: The pipeline details supplement the overview presented in the main paper.The section is intended as additional detail alongside the main-paper overview.
  • Stage-by-stage coverage: The section covers each stage of the dataset pipeline individually.Its organization follows the pipeline’s stages.

A.1 Data Sources … A.5 Postprocessing Stage

CubePart’s data engine aggregates and cleans permissively licensed 3D assets, filters them for quality and complexity, then uses multi-view VLM annotation to cluster and name existing parts before watertight-mesh postprocessing. The pipeline supports semantic part labels while preserving the limitation that fused source geometry cannot be subdivided.

  • A.1 Data Sources: The dataset combines Sketchfab sources, prioritizes human-corrected PartVerse segmentations during deduplication, restricts licensing, and excludes PartObjaverse-Tiny to prevent test contamination.Commercial and internal sources also contribute additional assets.
  • A.2 Preprocessing Stage: Preprocessing removes empty or degenerate parts and retains assets containing 2–32 parts, excluding single-part and overly complex assets.The lower bound supports clustering, while the upper bound aims to maximize VLM success.
  • A.3 VLM-Based Filtering Stage: VLM quality filtering evaluates assets from 8 viewpoints for mesh defects, scan artifacts, scene-level content, problematic geometry, and geometry or texture complexity.The model also produces brief descriptions alongside its quality assessments.
  • A.4 VLM-Based Clustering and Naming Stage: The annotation stage uses a VLM to cluster related parts by function or logical relationship and assign each cluster a concise semantic name.Identity clustering allows coherent parts to remain singleton clusters with names supplied by the VLM.
  • A.4.1 Set-of-Mark Rendering.: Set-of-Mark rendering adapts 2D visual grounding to 3D by using mesh-defined parts, 14 orbital viewpoints, and textured and part-colored image pairs.Markers are placed independently at maximum distance from part boundaries because the geometry-derived masks do not overlap.
  • A.4.1 Set-of-Mark Rendering.: Textured renders provide semantic context, while part-colored renders provide unambiguous segmentation; matching marker colors connect corresponding parts across both views.The paired representation was beneficial because textured-only renders produced poor clustering accuracy.
  • A.4.2 VLM Annotation.: VLM annotation returns structured JSON cluster assignments, resolves duplicate assignments by first occurrence, and filters assets whose parts collapse into one cluster.Existing source names and hierarchy are omitted to avoid noisy guidance and preserve generality across datasets.
  • A.4.2 VLM Annotation.: GPT-5 produced more accurate, finer-grained clusters and more consistent names than GPT-4o, especially for complex assets with many parts.The VLM cannot subdivide fused source geometry, so under-segmented assets receive names describing combined parts.

A.6 Dataset Limitations · B VLM Prompts

The dataset pipeline has acknowledged labeling limitations, especially for invisible, under-segmented, positional, and inconsistently named parts. Its VLM prompts specify structured quality filtering and semantic part clustering from multi-view renderings, with constrained tags, scores, descriptions, and JSON outputs.

  • A.6 Dataset Limitations: Invisible parts caused by small size or complete occlusion cannot be identified by the VLM and receive a special unlabeled designation.Under-segmented parts may receive names describing only their most visually prominent component.
  • A.6 Dataset Limitations: Positional adjectives such as “left” and “right” are occasionally confused, and some assets exhibit naming inconsistencies.
  • B.1 VLM Prompt for Quality Filtering: The quality-filtering VLM evaluates each asset from 8 rendered views using fixed flaw tags, geometric and texture complexity scores, and an overall quality score.The prompt also requests reasoning and a short searchable description in JSON format.
  • B.1 VLM Prompt for Quality Filtering: Quality filtering uses a controlled vocabulary covering defects and unsuitable assets, including mesh tearing, fragmented objects, cutaways, thin structures, occlusion, zero-volume meshes, baseplates, and empty images.The prompt requires returning only applicable tags as a JSON array and defines zero-volume meshes as flat, one-sided polygons with no measurable thickness.
  • B.1 VLM Prompt for Quality Filtering: The filtering prompt distinguishes poor, moderate, and excellent assets, retaining recognizable but simplistic assets as moderate and reserving excellent for high-quality standalone objects.Poor assets include broken, unrecognizable, fragmented, or scene-like content, while excellent assets can enter a small golden training set.
  • B.2 VLM Prompt for Part Clustering and Naming: Part clustering receives textured and part-colored image pairs with numbered parts and contours, then groups visible IDs into semantic clusters based on function, assembly, or logical relationship.The prompt structure includes system context, task instructions, clustering rules, output specifications, and few-shot examples.
  • B.2 VLM Prompt for Part Clustering and Naming: Clustering rules prohibit grouping solely by visual similarity or proximity, require every visible part ID to appear in exactly one cluster, and permit identity clustering when parts are already logically grouped.Cluster names should be concise and descriptive, with position adjectives grounded in the object perspective rather than the viewer’s perspective.
  • B.2 VLM Prompt for Part Clustering and Naming: The clustering stage requires a single JSON object containing only the “semantic_clusters” key and appends paired inputs across 14 orbital views.

C Data Distribution · D Behavior Script pipeline

The 462K-asset training set is dominated by objects with 2–10 parts, while the behavior-script pipeline assembles, rigs, controls, and drives generated parts through physical interaction. Its controllable part structure enables category-specific behaviors and transfers across compatible game-engine APIs.

  • C Data Distribution: 30% of the 462K training assets have exactly 2 parts, 45% have 3–5, 20% have 6–10, and 4% exceed 10, with a maximum of 32.The distribution is right-skewed, and the majority of assets fall within the 2–10 parts range.
  • D Behavior Script pipeline: The drone behavior-script pipeline follows four core stages for converting generated modular parts into an interactive asset.Figure 13 presents the pipeline skeleton using a commercial gaming platform’s Lua scripting API.
  • D Behavior Script pipeline: Welding assembles modular parts into a cohesive rigid body.This is the first stage of the four-stage behavior-script pipeline.
  • D Behavior Script pipeline: Rigging configures hinge constraints to define pivot points.This is the second stage of the pipeline.
  • D Behavior Script pipeline: Dynamic Force Control computes real-time physical thrust from angular velocity and lift formulas.This is the third stage of the pipeline.
  • D Behavior Script pipeline: Interaction synchronizes user commands by translating player input into physical motion.This is the fourth stage of the pipeline.
  • D Behavior Script pipeline: The controllable part structure lets developers inject category-specific behavior functions into generated assets.The four-stage structure transfers to other engines with comparable rigid-body and event APIs.

E Additional results

Additional results show that CubePart generates diverse assets across object categories and complex part schemas. The examples and accompanying video illustrate both open-vocabulary conditioning and the practical utility of the resulting multi-part meshes.

  • Additional results: Figure 14 presents a broader gallery of generated assets spanning diverse object categories and complex part schemas.The figure is described as containing more results generated by the full pipeline.
  • Additional results: An accompanying video highlights the practical utility of the generated multi-part meshes.The paper states that the video demonstrates practical utility rather than providing a quantitative evaluation.
  • Additional results: The gallery includes a sleek anti-gravity racing speeder specified with fuselage, engine pods, and control handlebars.Its prompt describes a single-seat racing speeder with glowing engine nacelles, while the schema names the component parts.
  • Additional results: It also includes a medieval plate-armor-inspired space suit with chainmail, thermal layer, gauntlet controls, oxygen tank, and pauldrons.The example demonstrates a complex semantic schema for a space suit.
  • Additional results: Another example is a floating surveillance unit housed inside a human skull, decomposed into anti-gravity, jaw bone, cranium, camera, and eye-socket parts.The prompt and schema jointly specify the object and its semantic components.
Loading 2605.28763v1…