PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Models

Motivation

• PartGen addresses the lack of meaningful, independently editable parts in otherwise high-quality but monolithic 3D assets.Such parts support reuse, editing, animation, replacement, and applications including robotics and embodied AI.Source: §1 · p. 2

Part segmentation

• The method models segmentation ambiguity by generating multiple plausible, view-consistent part decompositions from artist-created examples.A stochastic multi-view colouring model avoids assuming a fixed taxonomy and better captures artists’ intended decompositions than fine-tuned SAM or SAM2.Source: §1 · p. 2

Framework

• PartGen builds on the common pipeline of generating consistent 2D views and reconstructing 3D geometry from them.The framework applies this two-stage scheme to both segmentation and reconstruction ambiguities.Source: §1 · p. 2

Part completion

• A second multi-view generator completes occluded or invisible parts using whole-object context before 3D reconstruction.This addresses the ambiguity that deterministic reconstruction networks handle poorly and helps the reconstructed parts fit together coherently.Source: §1 · p. 2

Evaluation and applications

• PartGen starts from text, images, or real-world 3D scans, generates 3D assets with meaningful parts, is evaluated on artist-produced and scanned assets, and extends to 3D part editing.Source: §1 · p. 2

3D generation from text and images

• Existing 3D generation methods include latent, point-cloud, SDF, and score-distillation approaches, while two-stage systems generate multi-view images before 3D reconstruction.Related methods also explore compositional generation, but examples such as ComboVerse primarily compose distinct objects rather than semantic parts.Source: §2 · pp. 2–3

3D segmentation

• Prior 3D segmentation methods transfer semantic or language features from 2D into neural fields, Gaussian mixtures, point clouds, or other unstructured representations.These include semantic, instance, language-aware, scale-aware, and foundation-model approaches, alongside learned part-prior completion.Source: §2 · p. 3

Primitive-based representations

• Primitive-based methods decompose shapes into mixtures of geometric or neural primitives, which are generally related to parts but usually non-semantic.Other representations support part-based control or scene decomposition without necessarily producing human-interpretable semantic parts.Source: §2 · p. 3

Semantic part-based representations

• PartGen instead generates compositional 3D objects across modalities using multi-view diffusion for meaningful segmentation, complete reconstruction, and coherent assembly.Its generative formulation explicitly handles ambiguity in these tasks.Source: §2 · p. 3

3. Method

• PartGen takes text prompts, images, or 3D assets and produces human-interpretable, self-contained 3D parts through multi-view diffusion and reconstruction.Its pipeline first obtains multi-view images, then segments, completes, and reconstructs the parts.Source: §3 · p. 3, Figure 2 · p. 4

3.1. Background on 3D generation

• Multi-view 3D generation produces a four-view grid that a reconstruction model converts into a 3D object in shape and appearance.The grid contains views from four cardinal directions, arranged as a 2 × 2 image.Source: §3.1 · pp. 3–4

3.2. Multi-view part segmentation

• Part segmentation maps a multi-view image to stochastic, view-consistent masks for visible parts rather than imposing a deterministic part taxonomy.The model learns decompositions from artist-created multipart assets and can generate alternative segmentations by rerunning the stochastic diffusion process.Source: §3.2 · p. 4

3.3. Contextual part completion

• Contextual completion generates complete multi-view images for masked parts, using the unmasked object and mask so heavily occluded parts can fit the whole object.The contextual image becomes more important as occlusion increases, addressing the ambiguity that makes deterministic reconstruction unsuitable.Source: §3.3 · p. 5

3.4. Part reconstruction

• The reconstruction stage feeds completed, consistent part views into a pretrained reconstruction model to obtain each part in 3D.The authors report that no special finetuning is required to apply the reconstruction model to parts.Source: §3.4 · p. 5

3.5. Training data

• Training uses 140k artist-created assets, while filtered segmentation and completion data contains 45k objects and 210k parts.Assets with parts below 5% of object volume, more than 10 parts, or a single monolithic part are filtered out.Source: §3.5 · p. 5

4.1. Part segmentation

• PartGen’s mAP results are much higher than competing segmentation methods, including SAM2 fine-tuned on the dataset.The authors attribute this advantage primarily to the generator-based approach capturing segmentation ambiguity.Source: §4.1 · p. 7

4.2. Part completion and reconstruction

• Part completion and reconstruction largely surpass baselines, with joint multi-view reasoning and contextual completion both important for performance.Compared settings include no completion, omitted context, independently completed views, and an oracle.Source: §4.2 · p. 8

4.3. Reassembling parts

• Part-based reassembly achieves performance comparable to direct whole-object reconstruction while additionally producing structured parts for downstream editing.The reassembled object is formed by reconstructing and merging the predicted parts.Source: §4.3 · p. 8, Table 3 · p. 8

4.4. Applications

• PartGen supports part-aware text-to-3D generation, image-to-3D generation, and decomposition of real-world scanned objects.The text-to-3D examples include heavily occluded cases such as a gummy bear.Source: Figure 6 · p. 7, §4.4 · p. 8
• Decomposed parts can be modified through text prompts, including changes to their shape and appearance.The application is illustrated in Figure 7.Source: §4.4 · p. 8, Figure 7 · p. 8

• PartGen generates or reconstructs compositional 3D objects from text, images, or unstructured 3D objects.Source: §5 · p. 8
• The method reconstructs parts that are minimally visible or entirely invisible using a specially designed multiview diffusion prior.Source: §5 · p. 8
• PartGen supports downstream applications including text-guided part editing.Source: §5 · p. 8

• The supplementary material covers implementation details, additional experiment details, additional application examples, failure cases, and ethics and limitations.Source: Supplementary Material · p. 13

• The paper provides training details for PartGen and implementation details for part composition and part editing.Source: A. Implementation Details · p. 13

A.1. Text-to-multi-view generator

• The model converts a pretrained text-to-image diffusion model into a text-to-multi-view generator producing four 512 × 512 views arranged in a 1024 × 1024 grid.Training uses v-prediction, a DDPM scheduler, rescaled noise scheduling, 64 H100 GPUs, batch size 512, learning rate 10−5, and 10k steps.Source: §A.1 · p. 13

A.2. Image-to-multi-view generator

• The generator replaces text conditioning with an input image and adds image features through spatial concatenation and an additional cross-attention layer.The image is encoded with CLIP and reprojected into 157 tokens of dimension 1024 using a Perceiver-like architecture.Source: §A.2 · p. 13
• The image-to-multi-view generator supports downstream part-aware processing by providing multi-view inputs for editing and annotation workflows.Figure 8 shows masked-image and text conditioning for part editing, plus highlighted inputs for part captioning.Source: Figure 8 · p. 13
• Training uses all 140k 3D models with randomly sampled elevation and azimuth, while retaining fixed camera distance and field of view.The model is trained with a DDPM scheduler, rescaled SNR, v-prediction, 64 H100 GPUs, batch size 512, learning rate 10−5, and 15k steps.Source: §A.2 · p. 13

A.3. Multi-view segmentation network

• The multi-view segmentation network is fine-tuned with segmentation-image pairs to generate part masks across views.Its training uses expanded input channels, DDPM scheduling, v-prediction, rescaled SNR, 64 H100 GPUs, batch size 512, learning rate 10−5, and 10k steps.Source: §A.3 · pp. 13–14
• The segmentation model outperforms SAM2 and its variants on the reported recall curve.The comparison is shown in Figure 9.Source: Figure 9 · p. 14

A.4. Multi-view completion network

• The completion network conditions on the context image, masked image, and binary mask to generate complete multi-view part images before reconstruction.These inputs are combined into a 25-channel latent-space representation.Source: §A.4 · p. 14

A.5. Parts assembly

• PartGen assembles reconstructed parts by combining their implicit neural fields with their spatial locations during rendering.The generalized emission-absorption formulation uses part-specific feature and opacity functions to compute visibility and rendered features.Source: §A.5 · p. 14

A.6. 3D part editing

• The part-editing network is trained with part multi-view images, masks, and text descriptions, using the remaining-parts mask so the edited part can be regenerated without a fixed projection region.Part captions are generated by highlighting the target part with a red annulet and alpha blending before vision-language annotation.Source: §A.6 · p. 14
• Additional examples show PartGen processing varied modalities and generating or reconstructing objects with distinct parts.Figure 10 provides further examples of the part-aware pipeline.Source: Figure 10 · p. 15

B. Additional Experiment Details

• The evaluation ranks proposals using repeated segmentation samples, overlap-based reliability scores, and non-maxima suppression of highly overlapping regions.SAM2 methods use their own region-reliability scores, while the evaluation matches ranked predictions to ground truth by IoU thresholds.Source: B. Additional Experiment Details · pp. 15–16
• Mean average precision matches ranked predicted segments to ground-truth regions by IoU threshold, allowing each ground-truth segment to be matched only once.Average precision is averaged over test samples, and the number of matched predictions cannot exceed the number of ground-truth regions.Source: B. Additional Experiment Details · p. 16
• Recall at K measures the fraction of ground-truth segments recovered within the first K ranked predictions, and PartGen outperforms SAM2 variants by a large margin.The method also improves consistently as the number of segmentation samples increases; the comparison appears in Figure 9.Source: B. Additional Experiment Details · p. 17

C. Additional Examples

• PartGen supports part-aware text-to-3D generation, image-to-3D reconstruction, and decomposition of real-world 3D objects.These applications are presented as additional examples in Figure 10.Source: C. Additional Examples · p. 17
• Users can iteratively add parts from different inputs while preserving consistency and coherence in the composed 3D model.Figure 11 demonstrates this step-by-step composition capability.Source: Figure 11 · p. 16, C. Additional Examples · p. 17

D. Failure Cases

• Failures can arise from inconsistent generated views, incorrect grouping of distinct parts, or inaccurate depth reconstruction for complex geometry.The implementation is limited to objects with no more than 10 parts, and dense grass or leaves can produce poor depth quality.Source: D. Failure Cases · pp. 17–18, Figure 12 · p. 17

Ethics

• Dataset-derived biases may propagate into outputs, potentially producing culturally insensitive or inappropriate content.Source: E. Ethics and Limitation · p. 18

Limitation

• The method focuses on object-level generation using artist-created 3D assets, so extending it to scene-level generation requires further research.Its performance is heavily dependent on the training dataset’s quality and diversity.Source: E. Ethics and Limitation · p. 18