Source-linked AI summary
KaiNinja: Extending Native 3D Generators to the Part Level
Ruihan Yu, Lian Fu, Muyao Niu, Zheng-hui Huang, Yu-Ju Tsai, Sho Kuno, Fengbo Lan, Yonghao Yu, Erwin Wu, Ming-Hsuan Yang, Kaipeng Zhang, Zhixiang Wang
TL;DR
Native 3D generators produce fused meshes, although downstream editing, rigging, and simulation require separate part meshes and segmentation-based solutions are slow and error-sensitive. KaiNinja extends TRELLIS.2 with dual-volume O-Voxel packing so touching interfaces remain representable and parts emerge during generation without masks or segmenters. It improves whole-object fidelity over the same fine-tuned backbone and outperforms compared part-generation pipelines on reported whole-object and part metrics.
Problem
Native image-to-3D generators return fused meshes, while downstream editing, rigging, simulation, and reuse require separate self-contained part meshes.
Method
KaiNinja extends TRELLIS.2 with dual-volume O-Voxel packing, adapting its two-stage cascade so parts are generated without masks or segmenters.
Results
KaiNinja improves whole-object fidelity over the same backbone fine-tuned on the same corpus, including 38% lower Chamfer distance, while retaining part-level generation speed and quality.
Takeaways & Limitations
Packing is presented as a representation that supports part interfaces and can improve whole-object fidelity, not merely as a container for separating parts.
Takeaways & Limitations
Dual-volume two-coloring is heuristic: densely interlocking parts may be merged beyond the intended annotation because the contact graph need not be bipartite.
Abstract
from arXiv · showhide
Native 3D generators turn one image into a single mesh. TRELLIS.2 and its peers deliver high-fidelity non-watertight geometry with materials, but the output is one fused object, while downstream work such as editing, rigging and simulation operates on part-level assets. A naive idea is to run a 3D segmentation network on the fused mesh that TRELLIS.2 generates, but such pipelines are slow and bounded by the accuracy of the segmentation. We want a simple way to extend an existing native 3D generator to the part level. But we face a critical problem: the O-Voxel grid stores one sheet of surface per voxel, so a single volume cannot represent the interface where two parts touch, at any resolution. We introduce a dual-volume representation to solve this problem and put forward KaiNinja, a part-level extension of TRELLIS.2 built on a dual-volume form of its O-Voxel representation. KaiNinja keeps the generation speed and quality of TRELLIS.2 while extending it to the part level, with no mask or segmenter in the pipeline. Its training data come from sources of many kinds, including CAD models and assets authored by an LLM-driven agent; to our knowledge it is the first 3D generative model trained on agent-authored part data. Surprisingly, we also find that whole-object fidelity improves over the same backbone fine-tuned on the same dataset. Against part generation pipelines of different paradigms, it lowers whole-object Chamfer distance by 40% and raises strict part F-score by 16%.
1 Introduction
KaiNinja extends TRELLIS.2 from fused meshes to part-level assets by using dual-volume O-Voxel packing and adapting the generator without masks or segmenters. It preserves native generation advantages while improving whole-object fidelity and supporting diverse training data, including agent-authored assets.
- Representation: Dual-volume packing avoids predicting the number of parts or running a segmenter, supports arbitrarily many parts in two streams, and prevents contact faces from collapsing.Parts are recovered as connected components of each volume, while the backbone runs once rather than generating parts autoregressively.
- Results: 38% lower Chamfer distance is achieved than the same backbone fine-tuned on the same corpus, while generating a part-separated asset in about 24 seconds per object on one H100.The reported result indicates improved whole-object fidelity alongside part-level output and retained generation speed and quality.
- Data and results: Training combines CAD, artist-made, and LLM-agent-authored assets, with exact part labels from Articraft-10K's programmatic construction.On a held-out test set spanning all four sources, KaiNinja achieves state-of-the-art performance.
- Motivation and contribution: KaiNinja extends TRELLIS.2 into a native part-level generator, producing separate meshes without masks, segmenters, or per-object optimization.Its recipe adapts the pretrained model's layout flow while leaving the refinement backbone untouched and using volume-restricted attention.
- Representation: A single O-Voxel volume cannot preserve touching-part interfaces because one sheet of surface per voxel discards one coincident contact surface.Dual-volume packing assigns touching parts to different volumes, preserving open surfaces, UVs, materials, and PBR attributes.
2 Related Work
Related work separates into whole-object native generation, explicit part-aware generation, post-generation segmentation, and program-driven asset creation. KaiNinja is positioned as a native generator extension that decides part boundaries during generation while retaining TRELLIS.2's structured latent and material capabilities.
- Part-aware generation: Part-aware methods explicitly model structure, but earlier systems often target small single-category collections rather than open-domain images.Examples include hierarchical, deformable-part, implicit-editing, and cascaded diffusion approaches.
- Segmentation and compositional pipelines: Existing pipelines either segment generated meshes or generate parts jointly, with segmentation-dependent systems vulnerable to mask errors and slower cascades.The generate-then-segment family is reported as an order of magnitude slower end to end than the strongest direct part-generation methods.
- Program-driven generation: Program-driven and agentic systems create correctly structured assets by construction, but their geometry is bounded by primitives and retrieval libraries.KaiNinja uses Articraft as a training-data source rather than replacing generative geometry with procedural assembly.
- Positioning: KaiNinja extends the native TRELLIS.2 line into part-level generation, unlike whole-mesh generators that produce no part structure.It retains O-Voxel's structured latent, open-surface support, and PBR materials while avoiding masks, boxes, and segmenters on the critical path.
- Positioning: Compared with prior dual-volume methods, KaiNinja generates every part in one pass and is the first such extension built on TRELLIS.2 and O-Voxel.Its structured latent supports open surfaces and PBR materials that vecset/SDF representations do not offer.
3 Method
KaiNinja extends TRELLIS.2 to part-level generation by packing an object into two O-Voxel volumes, preserving touching interfaces that a single volume cannot represent. Its two-stage design adapts layout generation for coordinated streams while retaining the pretrained refinement flow, producing separable parts without masks or segmenters.
- 3.1 The Dual-Volume O-Voxel Representation: Dual-volume packing preserves part interfaces by assigning touching parts to different O-Voxel volumes, where each volume can represent one surface sheet per voxel.The representation keeps open surfaces, UVs, materials, and PBR attributes while preventing contact faces from collapsing.
- 3.1 The Dual-Volume O-Voxel Representation: Parts are assigned by two-coloring an O-Voxel contact graph, with greedy odd-cycle contraction merging parts when the graph is not bipartite.Edges indicate shared O-Voxel contact voxels, and parts within each resulting stream do not touch by construction.
- 3.2 Two Streams, Two Stages: The layout flow denoises two occupancy grids jointly, using separate stream weights and periodically inserted zero-initialized global attention to coordinate them.A frozen DINOv3 image encoder conditions both stages, while the layout stage receives the main stream-specific adaptation.
- 3.2 Two Streams, Two Stages: The refinement flow keeps TRELLIS.2 weights and processes active voxels with scoped attention, treating fine geometry and appearance as a task close to the pretrained objective.The two streams coexist through attention scope rather than separate refinement weights.
- 3.3 Training: Training first adapts each stream independently, then merges and jointly fine-tunes them with a disjointness penalty that discourages overlapping voxel claims.This staged recipe addresses exposed-contact distribution shifts before requiring coordinated decomposition.
- 3.4 Data Curation: The training corpus combines CAD, artist-made, and agent-authored assets whose programmatically assembled parts provide exact supervision from authoring structure.Articraft-10K supplies assets created by an LLM-driven agent, with part labels derived from scene graphs or connected components.
4 Experiments
KaiNinja is evaluated against representative part-generation pipelines and whole-object references on a shared held-out benchmark. It achieves strong part quality while retaining competitive cost and improving whole-object fidelity.
- Cost: KaiNinja costs about 24 seconds per object at 512 resolution and sits near PartPacker in cost while ranking above every method in quality.The full measurement includes generation and mesh export; X-Part is charged 469 GPU seconds because its segmenter occupies a four-GPU node.
- Comparison with State of the Art: KaiNinja is best on all seven metrics in Table 1 and every source breakdown entry, including a 0.10 strict part F-score gain over Hunyuan3D-2.1 + X-Part.Its whole-object Chamfer distance is 40% lower than the strongest baseline.
- Cost: PartPacker is the cheapest method and weakest on parts, while KaiNinja costs 40% more and gains 0.20 strict part F-score.AutoPartGen spends three times KaiNinja’s budget, and segment-then-regenerate costs an order of magnitude more than the other methods.
- Whole-object quality: At the strict threshold, KaiNinja reaches 0.919 whole-object F-score, exceeding every reference while missing the shape on 0.7% of objects.The same-corpus fine-tuned TRELLIS.2@512 reaches 0.830, making KaiNinja another 0.089 higher at about 2.5× the generation cost.
- Comparison with State of the Art: X-Part largely inherits the whole-object and part quality of its upstream generator, so post-generation segmentation does not exceed the mesh it receives.Its Hunyuan3D-2.1 pairing reaches 0.816 whole-object strict F-score, within 0.009 of the upstream generator alone.
- Representation: The comparison with PartPacker shows a moderate whole-object gap but the widest part-quality gap, linking performance differences to the representation used for packed volumes.PartPacker uses a vecset latent decoded as an SDF field, whereas KaiNinja packs volumes on the backbone’s sparse voxel grid.
4.3 Generalization Beyond the Fine-Tuning Corpus
KaiNinja is tested on sources outside its fine-tuning corpus and on the external HY3D-Bench benchmark. Its geometry remains competitive or strongest, while benchmark part scores depend on annotation granularity.
- External sources: On the 95 Sketchfab and GitHub objects completed by every method, KaiNinja leads the next method on whole-object and strict part metrics.The sources do not enter the fine-tuning corpus or test split, although they overlap with data used by pretrained TRELLIS.2.
- HY3D-Bench: HY3D-Bench reverses the held-out ordering because its coarser annotations penalize KaiNinja for producing finer part decompositions.The benchmark therefore separates faithful geometry from agreement with its part-label granularity.
4.4 Qualitative Results
Qualitative comparisons evaluate separated parts across held-out, training, and out-of-training-source objects under aligned rendering conditions. KaiNinja produces clean and coherent parts across these groups.
- Qualitative Results: KaiNinja produces clean and coherent separated parts on rigid CAD shapes across the qualitative comparison groups.Figures compare held-out, training-corpus, and source-separated objects after best-axis alignment, with colors marking parts rather than materials.
4.5 Ablation
The ablation studies test volume count and packing behavior while holding the backbone, data, and training recipe fixed where specified. Two volumes are selected because they offer better robustness despite a small fidelity advantage for three volumes.
- Number of volumes: The ablation isolates volume count by keeping the backbone, data, and recipe identical while changing only the number of packed streams.The three-volume design uses a greedily three-colored contact graph and three flow streams.
- Number of volumes: Three volumes are slightly better on parts by 1.6 strict F-score points and 0.0067 CD, while tying two volumes on whole-object geometry.The difference is significant under a paired bootstrap.
- Number of volumes: Two volumes are chosen for robustness: three-volume models fail on 13 of 1000 objects and leave a stream empty on 7.7%, versus zero failures and 1.5% for two volumes.Three volumes also produce 4.73 connected components per object versus 3.84 for two and 5.05 in ground truth.
TRELLIS.2
The ablations examine volume count, disjointness regularization, and post-processing in KaiNinja's TRELLIS.2 extension. Two volumes avoid incomplete-arm failures, while the disjointness term improves intermediate occupancy and relabeling substantially reduces fragmented components.
- Volume-count ablation: Two volumes complete all 986 evaluated objects, whereas three volumes fail on 13.Table 6 scores Stage-1 occupancy only and includes objects on which both arms complete.
- Disjointness penalty: The disjointness penalty lowers median cross-volume overlap from 0.229 to 0.184 over 250 held-out objects.Its effect appears at Stage 1, where the layout is decided; final decoded metrics remain nearly unchanged.
- Disjointness penalty: The shipped model retains the disjointness penalty because it acts on layout formation despite requiring one decoder pass per step.The term is kept for its effect on the stage that determines part structure.
- Post-processing: Relabeling cuts predicted component count from 14.06 to 5.43, versus 5.85 ground-truth parts, and improves mIoUP and F10.05.The merge step changes metrics little and mainly safeguards against duplicated volumes, while relabeling reconnects floating facets and pieces of one part.
5 Conclusion
KaiNinja extends TRELLIS.2 in place with two interleaved volumes while preserving the backbone's VAE and material pipeline. It produces separate part meshes without masks or segmenters, improves whole-object fidelity, and remains subject to graph-coloring and stochastic-generation failures.
- Conclusion: KaiNinja extends TRELLIS.2 in place, producing separate meshes with a small constant generation-cost increase and no mask or segmenter.The extension packs O-Voxel into two interleaved volumes while leaving the VAE and material pipeline untouched.
- Conclusion: Extending the generator in place outperforms compared pipelines that attach a segmenter, on both whole-object and part-level measures.The conclusion reports this comparison across the evaluated pipelines.
- Conclusion: Dual-volume packing improves whole-object fidelity over the same backbone fine-tuned on the same corpus.The authors interpret this as evidence that packing represents whole objects better than serving only as a part container.
- Limitations: Dense non-bipartite contact graphs require merging parts, which can undersegment objects with heavily interlocking parts.The two-volume assumption therefore limits decomposition granularity.
- Limitations: Stochastic generation can undersegment, collapse onto one empty volume, or duplicate the whole object across both volumes.The duplicated-volume failure can produce nearly coincident copies that part metrics interpret as many small pieces.
- Limitations: The disjointness penalty alleviates duplicated-volume failures but does not resolve them, and Stage 2 can still drop facets and leave small holes.The released pipeline therefore applies light geometric cleanup before delivery.
A Hyperparameters
KaiNinja trains two-stage, two-stream flows with separate latent structures and cross-volume attention schedules. Stage 1 uses a disjointness penalty during joint fine-tuning, while Stage 2 reuses the released structured-latent autoencoder without training it.
- Shared settings: Both flows use width 1536, 30 blocks, 12 heads, rotary position encoding, bfloat16 AdamW training, EMA, gradient clipping, and classifier-free guidance.The global batch size is 32 throughout, with puncond = 0.1 and EMA rate 0.9999.
- Stage 1: Stage 1 operates on a 163 sparse structure latent with 8 channels per stream and inserts cross-volume attention at depths {6, 12, 18, 24, 29}.Frozen warmup lasts 10K steps at 10−4, followed by 100K joint fine-tuning steps at 5 × 10−5 with λov = 5.
- Stage 2: Stage 2 operates on a 32-channel structured latent at resolution 32 and merges attention pools every third block.Frozen warmup lasts 30K steps and full fine-tuning 100K steps, both at 3 × 10−5.
- Stage 2: The structured latent autoencoder is the released TRELLIS.2 component and is never trained.This keeps the autoencoder fixed while the flows are adapted.
B Two Volumes and Graph Coloring
Two-volume packing relies on two-coloring the part-contact graph, but this is only guaranteed for bipartite graphs. Non-bipartite 3D contact graphs may therefore force coarser decompositions or alternative future strategies.
- Graph coloring: Two-coloring is a heuristic because it works only when the part-contact graph is bipartite.When the graph is not bipartite, the method merges parts until a two-coloring is possible, reducing decomposition granularity.
- Graph coloring: A 3D contact graph need not be planar, so planar-map arguments cannot guarantee a bounded coloring strategy.The passage notes that arbitrarily many spatial regions can be pairwise adjacent.
- Future directions: Deleting contact edges by merging parts preserves assembly but costs a coarser decomposition; separating parts would preserve them but create exploded objects requiring reassembly.The latter route is left for future work.