Seeing Both the Forest and the Trees: Reusing Holistic 3D Priors for Part-Decomposed Generation

1 Introduction

• Part-decomposed generation must combine geometric detail within each part with structural coherence across the whole object, while holistic priors produce only fused meshes.Directly extending those priors to part-decomposed generation has proven non-trivial.Source: §1 · p. 2
• The model treats each part as a complete shape locally and as part of the whole object globally, reflecting the hierarchical duality of 3D shapes.Global context constrains parts toward coherence, while high-quality parts elevate the geometric fidelity of the whole.Source: §1 · p. 2
• A coarse mesh plus a face-level segmentation mask forms a segmented mesh scaffold that exposes part boundaries in the holistic prior’s native 3D representation.The prior can read this input without an objective shift.Source: §1 · pp. 2–3
• Dual-frame coordinate interleaving processes each latent in both a part-local frame and an object-global frame, reusing geometric detail and structural coherence from the same pretrained weights.A global-local interleaved transformer alternates intra-part and inter-part attention, with only lightweight finetuning.Source: §1 · pp. 2–3
• On PartObjaverse-Tiny, STRUCTPARTS matches or surpasses prior part-decomposed models with substantially lower training costs.It also supports part-mesh refinement and part-level editing.Source: §1 · p. 3

2 Related Work

• Holistic models capture global structure and local detail but emit a single fused mesh, motivating part-decomposed models that reuse their native 3D priors.Localizable VecSet models [3] [12] anchor tokens to explicit 3D positions.Source: §2 · p. 3
• Point-cloud conditioning [5] [6] [7] differs from holistic models’ native input and can require substantial retraining [5] [7].Two-stage methods [8] [9] shift the objective toward structural abstractions and away from the native shape latent space.Source: §2 · p. 3
• Predicting parts in an unnormalized object frame [10] [11] [29] creates a spatial-resolution mismatch that hurts tiny parts.Multi-view 2D-to-3D part lifting [30] sidesteps the native 3D latent but brings view-consistency and post-processing issues.Source: §2 · p. 3
• STRUCTPARTS uses existing 3D part masks as a scaffold and applies dual-frame interleaving in the native geometric domain with lightweight finetuning.Part-reasoning methods produce masks on existing meshes rather than synthesizing new geometry.Source: §2 · pp. 3–4

3.1 Preliminaries: structured shape latents for 3D diffusion

• Structured shape latents anchor tokens to occupied voxel positions, which provide positional embeddings for 3D shape diffusion.The representation is built from curvature-aware mesh samples quantized onto a regular grid.Source: §3.1 · p. 4

3.2 Scaffold-guided global-local interleaving in transformer

• A segmented scaffold pairs a coarse mesh from a holistic 3D generator with a face-level mask that partitions it into semantic parts.This keeps structural conditioning in the prior’s native 3D representation.Source: §3.2 · p. 4
• Each part’s latents receive positional embeddings in both its normalized local frame and the shared global frame.The frames are derived by sampling scaffold-surface points, normalizing them within each part, and mapping them back to object coordinates.Source: §3.2 · p. 4
• STRUCTPARTS alternates intra-part and inter-part attention, reusing the same pretrained weights for per-part geometric detail and whole-object structural coherence.Intra-part attention processes each part as a standalone shape; inter-part attention coordinates the parts as one anchored assembly.Source: §3.2 · p. 5

3.3 Downstream tasks

• The scaffold supports inference-time refinement from a coarse mesh and training-free merge or split editing through segmentation regrouping.During editing, latents in unedited regions are kept while the modified segmentation determines which latents are resampled.Source: §3.3 · pp. 5–6

4.1 Implementation Details

• The model is initialized from Hunyuan3D-2.1, trained on Partverse-XL for 40K steps, and evaluated on 200 held-out PartObjaverse-Tiny objects.Training used 768 H200-hours; evaluation reports CD and F-Score at 0.05 and 0.10 for both part-level and assembled-shape-level geometry.Source: §4.1 · p. 6, §4.2 · p. 6

4.2 Experimental Setup

• Evaluation uses separate protocols for layout-conditioned part-level assessment and image-to-decomposed-parts shape-level assessment, with a matched-budget X-Part retraining comparison.The matched comparison uses the same initialization, data, token count, and GPU-hour allocation as STRUCTPARTS.Source: §4.2 · p. 6

4.3 Main Results

• On a fixed 200-object ABO subset, STRUCTPARTS obtains the lowest mean and median CD and matches released X-Part at F@0.05.The evaluation uses identical inputs and metrics.Source: §4.3 · p. 7
• STRUCTPARTS achieves the best decomposition quality across layout-conditioned part-level metrics and is comparable to released X-Part while outperforming other baselines in image-to-decomposed-parts evaluation.The comparisons also report qualitative failure modes: missing detail, incomplete shapes, floaters, semantically weak decompositions, and boundary collisions.Source: §4.3 · p. 7, Table 1 · p. 8, Table 2 · p. 8

4.4 Ablation Studies

• Ablations show that local-only RoPE loses global coherence, global-only RoPE degrades small or off-center parts, and adding part embeddings still misses fine detail.The comparisons support the dual frame and local RoPE as important to preserving both global structure and per-part geometric quality.Source: §4.4 · pp. 7–8, Table 3 · p. 8, Figure 5 · p. 9
• At 5% scaffold geometry noise, F@0.05 falls from 0.63 to 0.51; with 20% boundary-label corruption, corresponding-part F@0.05 falls to 0.58.Whole-shape performance remains close to clean at 1% geometry noise; corresponding-part F@0.05 is 0.67 at 5% label corruption.Source: §4.4 · pp. 8–9

4.5 Applications

• With a coarse scaffold, STRUCTPARTS recovers fine geometric detail that faithfully matches the input image, unlike X-Part’s rigid following of the input geometry.The authors attribute this capability to alignment with the holistic shape latent space.Source: §4.5 · p. 9
• Part-level editing supports training-free changes in decomposition granularity by merging or splitting segmentation groups on the scaffold.Edited regions are resampled while tokens in unedited regions are kept consistent.Source: §4.5 · p. 9

• STRUCTPARTS is presented as inheriting both geometric detail and structural coherence from a pretrained holistic 3D prior.Source: §5 · p. 9
• Given a coarse scaffold, STRUCTPARTS recovers fine geometric detail from the image, whereas X-Part rigidly follows the input geometry.Source: Figure 6 · p. 10
• Merging or splitting scaffold segmentation groups yields different part granularities without training at inference time.Source: Figure 7 · p. 10
• Inaccurate input scaffolds can persist in generated parts even though the method generates geometric detail.Source: §5 · p. 10, Figure 8 · p. 10

• Training uses Partverse-XL, with 36K objects and about 240K parts, after cuBVH-based watertight remeshing of each part.Source: §A.1 · p. 14
• Scaffold construction remeshes the whole shape, transfers rendered segmentation labels onto its watertight surface, and filters labels outside ground-truth part bounding boxes.Source: §A.1 · p. 14
• The 21-layer DiT alternates intra-part and inter-part attention, using part-local 128^3 and shape-global 512^3 rotary position grids.Source: §A.1 · p. 14
• The DiT is initialized from Hunyuan3D-2.1 ShapeDiT weights and fine-tuned for 40K steps on eight H200 GPUs over four days.Source: §A.1 · p. 14
• The VAE is fine-tuned with token length 512 to encode and decode individual parts at the DiT’s per-part training scale.Source: §A.1 · p. 14
• At inference, a coarse mesh and per-face labels come from image-conditioned 3D generation and segmentation, while DINOv2 image features condition sampled part latents.Source: §A.1 · p. 14

• The retrained X-Part comparison uses the same initialization, training data, token budget, and eight-H200 four-day allocation as STRUCTPARTS.Source: §A.2 · p. 14, Table 4 · p. 15
• STRUCTPARTS uses 32.5% fewer TFLOPs per object and less memory than X-Part under the shared profiling setup.Source: §A.2 · p. 14, Table 4 · p. 15
• Both methods use online point sampling because fixing points offline can introduce a train–inference mismatch.Source: §A.2 · p. 15
• The 20K STRUCTPARTS checkpoint performs better on all four metrics than the 28K matched-budget X-Part model, despite fewer steps and processed parts.Source: §A.2 · p. 15, Table 5 · p. 15

• Geometry noise causes little change at 1%, moderate degradation at 2%, and substantial degradation at 5% of the scaffold bounding-box diagonal.The clean output preserves overall geometry, though aggregate metrics do not isolate high-frequency detail.Source: §A.3 · p. 15
• Boundary-label corruption changes whole-shape metrics only slightly, but part consistency worsens progressively at 10–20% corruption.Performance is nearly unchanged at 5%; the part-level comparison uses ground-truth labels with known correspondence.Source: §A.3 · p. 15

Cross-dataset evaluation

• On ABO, STRUCTPARTS achieves the lowest mean and median CD, matches released X-Part at F@0.05, and is within 0.01 at F@0.10.Evaluation uses a fixed 200-object subset with identical inputs and the same whole-shape CD/F-score protocol for all methods.Source: §A.4 · pp. 15–16, Table 8 · p. 16

Category-wise evaluation

• Across the six largest PartObjaverse-Tiny categories, STRUCTPARTS has the lowest median CD in all six, the lowest mean CD in five, and leads both F-scores in four.The higher Animals mean indicates a heavy-tailed failure; all evaluation UIDs are excluded from training, yielding zero object overlap, while semantic-category overlap with PartVerse-XL cannot be objectively measured.Source: §A.4 · p. 16, Table 9 · p. 17

Paired uncertainty estimates

• Only the paired-bootstrap 95% interval for F@0.05 excludes zero when comparing STRUCTPARTS with released X-Part.The bootstrap resamples objects with complete matched predictions 10,000 times, retaining paired parts within each sampled object; negative CD and positive F-score differences favor STRUCTPARTS.Source: §A.4 · p. 16, Table 11 · p. 17

Alternative initialization

• At 20K, UltraShape initialization achieves mean/median CD of 0.123/0.015 and F@0.05/F@0.10 of 0.64/0.77; its F-scores are nearly identical to the Hunyuan3D-2.1 baseline.The comparison uses the same halfway training point because the compatible UltraShape DiT requires retraining.Source: §A.5 · p. 17

Comparison with UniPart

• STRUCTPARTS processes one latent under alternating part-local and object-global frames, unlike UniPart’s two-stage generation of whole-object and separate global and canonical part latents.STRUCTPARTS accepts an editable segmented scaffold for training-free merge/split editing; UniPart code was unavailable, so the comparison is conceptual rather than matched quantitatively.Source: §A.5 · p. 17

Positive impact

• Part-decomposed assets support localized editing, animation rigging, retopology, and semantic asset reuse.The scaffold-conditioned interface and training-free merge/split operations can lower the barrier for nonexperts working on education, indie games, and rapid prototyping.Source: §A.6 · pp. 17–18
• Reusing a pretrained holistic prior with lightweight finetuning avoids the energy and data costs of training a part-aware generative model from scratch.The authors state this can lower the carbon footprint of part-decomposed 3D research and make the capability accessible to teams without large-scale compute infrastructure.Source: §A.6 · p. 17

Negative impact and mitigation

• Potential harms include unauthorized reproduction of copyrighted assets, misleading 3D media, and inherited training-data biases such as category over-representation or culturally skewed styles.The authors release the model only with publicly available, license-compliant datasets, encourage attribution and output review, and do not recommend identity-related or surveillance use.Source: §A.6 · p. 18

• The authors say the abstract and introduction accurately state the dual-frame mechanism, segmented mesh scaffold, and PartObjaverse-Tiny evaluation, supported by Sec. 3 and Sec. 4.Source: NeurIPS Paper Checklist · p. 19

Limitations

• The method depends on segmented-scaffold quality and inherits failure modes from the upstream holistic 3D generator and 3D segmentation model.The authors say these limitations are discussed in Sec. 5 and the supplementary appendix.Source: NeurIPS Paper Checklist · p. 20

3. Theory assumptions and proofs; 14. Crowdsourcing and research with human subjects; 15. Institutional review board approvals

• The paper reports no theoretical results and no crowdsourcing or human-subject research.Source: NeurIPS Paper Checklist · pp. 20, 25

4. Experimental result reproducibility; 5. Open access to data and code

• Reproduction information covers the architecture, training procedure, hyperparameters, evaluation protocol, and additional preprocessing and scaffold-construction details; code and checkpoints are promised upon acceptance.The training and evaluation datasets are reported as publicly available.Source: NeurIPS Paper Checklist · p. 21

7. Experiment statistical significance

• No error bars are reported: results use one training run because multiple-seed reruns exceed the compute budget, with complementary metrics evaluated at part and object levels.The metrics are Chamfer Distance and F-Score at thresholds 0.05 and 0.1.Source: NeurIPS Paper Checklist · p. 22

8. Experiments compute resources

• The paper reports training on 8 GPUs for about 4 days, totaling about 32 GPU-days, and cites original publications for baseline training-compute figures.The reported batch size is 128 per rank.Source: NeurIPS Paper Checklist · p. 23

• The authors report using publicly released datasets and pretrained models, discussing positive and negative societal impacts, and identifying no high-risk dual-use concerns beyond standard generative-model considerations.Source: NeurIPS Paper Checklist · pp. 23–24

12. Licenses for existing assets; new assets

• The authors say existing datasets, pretrained models, and baselines are credited and used under their public terms; trained checkpoints and code are to be released with documentation upon acceptance.Source: NeurIPS Paper Checklist · pp. 24–25

NeurIPS Paper Checklist

• The checklist marks the LLM-use question N/A because core method development did not involve LLMs as important, original, or non-standard components.The checklist notes that LLM use limited to writing, editing, or formatting does not require declaration if it does not affect the method, scientific rigor, or originality.Source: NeurIPS Paper Checklist · p. 26