PaperlayerView exact revision on arXiv

Source-linked AI summary

Seg3DParts: Segmentation-Grounded Controllable Part-Level 3D Generation

Jiantao Lin, Meixi Chen, Yingjie Xu, Chenbo Fu, Leyi Wu, Hao Chen, Yinchuan Li, Ying-Cong Chen

arXiv:2609.36918v1cs.CV

TL;DR

Recovering parts from one image is difficult under occlusion and ambiguous boundaries, and existing methods may not combine explicit part control with coherent multi-part generation. Seg3DParts grounds part identity in segmentation and coordinates components during generation, reporting stronger geometric fidelity and part-level controllability than existing baselines. It directly decodes aligned meshes in shared canonical space, but its performance depends on segmentation quality.

  • Problem

    Single-image part recovery is ambiguous under occlusion and unclear boundaries, while existing methods leave part identity and spatial allocation implicit.

    Source: §1 · p. 2

  • Method

    Seg3DParts grounds part identity in segmentation and uses structured interaction to exchange context across components during generation.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Seg3DParts achieves stronger geometric fidelity and part-level controllability than existing baselines.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The method directly decodes well-aligned part meshes in shared canonical space without post-hoc alignment, supporting flexible segmentation-driven generation.

    Source: §1 · p. 3

    Performance depends on input part-segmentation quality; low-quality segmentation or semantically ambiguous boundaries can degrade generated geometry.

    Source: G Limitations · p. 16

Abstract

from arXiv · show

Part-level 3D assets are essential for editing, reassembly, and interaction, yet recovering such structure from a single image remains challenging due to occlusion, ambiguous boundaries, and the need for coherent multi-part reasoning. Existing approaches struggle to achieve both controllable part-level generation and coherent multi-part structure, as part identity and spatial allocation are typically inferred implicitly. We present Seg3DParts, a segmentation-grounded framework for controllable part-level 3D generation from a single image. By treating segmentation as an explicit grounding signal, our method defines part identity during generation, enabling each component to be anchored to a corresponding image region. To ensure coherent assemblies, we introduce structured cross-part interaction that allows components to exchange global context throughout the generative process. As a result, Seg3DParts directly generates well-aligned part meshes in a shared canonical space without post-hoc alignment, supporting flexible and controllable decomposition. We further introduce PartObjectNet, a large-scale dataset with over 200K objects and 1M annotated parts. Experiments demonstrate that Seg3DParts achieves superior geometry quality, cross-part coherence, and part-level controllability over existing methods.

1 Introduction

Single-image part generation must resolve ambiguous boundaries, occlusion, and cross-part relationships, while existing approaches leave part identity and spatial allocation implicit. Seg3DParts grounds identity in segmentation and combines part-level control with cross-part reasoning, direct canonical-space decoding, and PartObjectNet.

1 Introduction

  • Occlusion, ambiguous boundaries, and weak local visual cues make single-image part geometry and coherent 3D placement difficult to recover.The system must also infer missing geometry and reason about relative scale and attachment across interacting parts.Source: §1 · p. 2
  • Existing decomposition and joint-generation methods leave a gap between controllable part decomposition and coherent assemblies, with identity and spatial allocation often implicit.Decomposition pipelines rely on local cues and can misalign parts; joint models encourage coherence but use implicit latent slots.Source: §1 · p. 2
  • Seg3DParts uses segmentation to specify part identity and anchor each generated component to its corresponding image region.Localized appearance cues ground part-specific geometry and provide direct control under occlusion and ambiguous boundaries.Source: §1 · pp. 2–3
  • Structured cross-part interaction lets components reason about one another’s shape, scale, and placement while preserving explicit part identities.This mechanism supplies global structural context throughout generation.Source: §1 · pp. 2–3
  • Seg3DParts directly decodes independent part meshes in a shared canonical space, avoiding post-hoc alignment and supporting flexible segmentation-driven generation.The paper also introduces PartObjectNet, with over 200K objects and more than 1M annotated parts.Source: §1 · p. 3

2 Related Work

Prior 3D-generation work increasingly uses geometry-aware generative models, but most produces whole objects without explicit part structure. Part-level methods either reconstruct parts separately or generate them jointly, with trade-offs in cross-part coherence and explicit control.

2.1 3D Generation

  • Most 3D-generation methods focus on whole-object synthesis, leaving limited support for fine-grained editing, reassembly, and part-level control.More recent approaches use 3D-native geometry-aware latent models, while some generate meshes without post-processing.Source: §2.1 · p. 3

2.2 Part-Level 3D Generation

  • Part-level generation broadly divides into pipelines that segment and process components separately and models that represent multiple components jointly.These paradigms provide the organizing distinction for the related work.Source: §2.2 · p. 3

2.2.1 Decomposition-Based Part Generation Pipelines

  • Decomposition pipelines complete each segmented part using local surface cues and enforce cross-part consistency only after completion, risking misalignment or inconsistent scale.This is especially challenging when parts are heavily occluded or have large missing regions.Source: §2.2.1 · p. 3

2.2.2 Part-Structured Generative Models

  • Joint part-structured models encourage global coherence, but implicit latent slots leave part identity, number, and granularity difficult to control.Using segmentation only as a global partition, as in OmniPart [27], still leaves part allocation implicit and can produce ambiguous boundaries or merged components.Source: §2.2.2 · p. 4

3 Method

Seg3DParts adapts TRELLIS’s two-stage, part-aware latent-generation backbone to use segmentation-grounded conditioning and structured interaction between parts. The framework preserves aligned part representations through decoding and trains its stages with reconstruction and generative objectives.

3.1 Preliminaries

  • Seg3DParts builds on TRELLIS’s sparse-voxel, two-stage generation framework, maintaining latent representations separately for each part.The sparse-structure stage models occupancy, and the structured-latent stage generates fine-grained geometry for mesh decoding.Source: §3.1 · pp. 4–5

3.2 Segmentation-Grounded Part Conditioning

  • Segmentation-conditioned features ground each component in localized image evidence, explicitly specifying part identity and enabling control over decomposition.Part crops provide local appearance features, while global image features provide overall context; conditioning is used in both part-aware DiTs.Source: §3.2 · p. 5

3.3 Multi-Part Latent Interaction

  • Multi-part cross-attention exchanges context between part-separated latents during generation, supporting coordinated assembly while preserving part identities.Interaction follows intra-part self-attention, and the residual formulation retains each part’s representation while adding structural context.Source: §3.3 · pp. 5–6

3.4 Canonical-Space Part Decoding

  • Parts are decoded directly in a shared canonical space, so independently generated meshes are already positioned relative to one another without post-hoc alignment.The representations retain original scale and placement through data preparation, encoding, generation, and decoding.Source: §3.4 · p. 6

3.5 Optimization Loss

  • The two-stage framework combines VAE reconstruction objectives with KL regularization and trains both rectified-flow DiTs using flow matching.The sparse-structure VAE targets coarse part occupancy, while the structured-latent VAE emphasizes surface geometry reconstruction conditioned on sparse structure.Source: §3.5 · p. 6

4 Experiments

Seg3DParts is evaluated on a curated dataset and an external benchmark, with comparisons and ablations assessing geometry quality, part separation, and segmentation-driven control. Its reported results favor the method overall, while ablations identify distinct contributions from cross-part interaction and localized conditioning.

4.1 Dataset

  • PartObjectNet contains approximately 200K curated objects, preserving parts’ relative scale and placement; evaluation also uses 500 held-out objects and the independent PartObjaverse-Tiny benchmark.Automated filtering retains objects with 2–15 parts, followed by manual curation.Source: §4.1 · pp. 6–7

4.4 Comparison with State-of-the-Art Methods

  • Figure 3 compares Seg3DParts with decomposition-based pipelines and part-structured models; ground-truth segmentations condition Seg3DParts and OmniPart.Different colors identify semantic parts.Source: Figure 3 · p. 7

4.2 Implementation Details

  • The two-stage TRELLIS training pipeline adds multi-part interaction layers to both stages and uses a mesh-based latent encoder for structured latents.The encoder replaces TRELLIS’s voxel-feature projection and is described as improving geometric expressiveness and reconstruction of occluded regions.Source: §4.2 · p. 7

4.3 Evaluation Protocol

  • The evaluation measures global geometry with Chamfer Distance and F-Score, part geometry with per-part Chamfer Distance, F-Score, and IoU, and overlap with pairwise IoU.The protocol uses 16K sampled surface points for global and part-level metrics and 64 × 64 × 64 voxelization for volumetric consistency and overlap.Source: §4.3 · p. 7

4.4 Comparison with State-of-the-Art Methods

  • Seg3DParts achieves the highest F-Score and lowest Chamfer Distance among evaluated methods, with lower part-overlap IoU and gains over OmniPart across part-level metrics.The global comparison covers PartObjectNet and PartObjaverse-Tiny; the part-level comparison is limited to methods with explicit correspondence.Source: Table 1 · p. 8, §4.4 · p. 8, Table 2 · p. 8

4.5 Ablation Study

  • Removing multi-part latent interaction degrades all metrics and increases part intersections or overlap, while replacing localized conditioning with a whole-object map degrades part-level fidelity and overlap.Global shape quality remains comparable under whole-object conditioning, but boundaries become ambiguous and components can be incorrectly shaped or misplaced.Source: §4.5 · p. 9, Table 3 · p. 9
  • Changing only the input segmentation while fixing the image produces distinct, consistent part layouts, demonstrating control over decomposition and spatial configuration.The dataset’s spatially aligned parts provide supervision for inter-part relationships and controllable generation.Source: §4.1 · p. 6, §4.5 · p. 9

5 Conclusion

Seg3DParts uses semantic segmentation to guide structured 3D part synthesis, producing coherent, well-aligned meshes and supporting segmentation-driven part control.

5 Conclusion

  • With the image fixed, different segmentation inputs produce distinct, accurate part layouts.The generated layouts follow the provided segmentation.Source: Figure 5 · p. 9
  • Seg3DParts combines explicit 2D semantic grounding with joint multi-part reasoning to generate coherent, well-aligned meshes without post-hoc alignment.Experiments show stronger geometric fidelity and part-level controllability than existing baselines.Source: §5 · p. 9
  • Removing multi-part attention yields interpenetrating, incoherent parts, while replacing part-wise segmentation with an object mask causes unstable decomposition.The full model produces well-separated, structurally coherent parts that closely match the ground truth.Source: Figure 4 · p. 10

A Implementation and Architectural Details

The appendix details Seg3DParts’ architecture, training, and inference; its part-aware VAE stages preserve component separation while modeling structure and geometry.

A Implementation and Architectural Details

B.1 Stage-1 Part-Aware Sparse Structure VAE

  • Stage 1 models coarse part structure with sparse voxel occupancy, processing semantic parts in parallel with shared encoder–decoder weights while keeping components separate.Its hierarchical 3D convolutional encoder infers a Gaussian latent posterior, and a symmetric decoder reconstructs binary occupancy at the original resolution.Source: §B.1 · p. 13
  • At the latent bottleneck, cross-part attention lets components reason about other parts’ presence, placement, and extent while higher-resolution stages preserve local detail.The training objective uses Dice loss for sparse occupancy reconstruction and KL regularization; decoded voxel grids are normalized for numerical stability without changing canonical alignment.Source: §B.1 · p. 13

B.2 Stage-2 Part-Aware Structured Latent VAE

The stage-2 part-aware VAE encodes each part’s surface geometry in structured latent tokens, coordinates components through cross-part interaction, and decodes meshes in a shared canonical space.

B.2 Stage-2 Part-Aware Structured Latent VAE

  • Stage 2 encodes each part’s fine-grained surface geometry into sparse structured latent tokens, offering greater geometric expressiveness and robustness under partial or occluded observations.A sparse transformer encoder–decoder processes part-separated representations and predicts a Gaussian posterior for each part’s latent code.Source: §B.2 · p. 13
  • Cross-part latent attention in both encoder and decoder coordinates components without collapsing their separate representations.The interaction is integrated into a subset of transformer blocks, while remaining blocks process parts independently.Source: §B.2 · p. 14
  • The decoder uses progressive sparse-feature upsampling and FlexiCubes mesh extraction to decode each part directly into a shared canonical space, avoiding post-hoc alignment.FlexiCubes provides differentiable, topology-adaptive surface representation for mesh reconstruction.Source: §B.2 · p. 14
  • Stage-2 training focuses on geometry reconstruction using geometry-aware supervision and KL regularization, including rendered silhouette, depth, and truncated signed distance field losses.The perceptual normal loss combines ℓ1, SSIM, and LPIPS losses to enforce low-level and perceptual consistency on rendered surface normals.Source: §B.2 · p. 14

C Transformer Architecture Details

Seg3DParts uses a shared rectified-flow DiT backbone with part-wise and cross-part blocks, per-part AdaLN conditioning, and a two-stage inference pipeline.

C Transformer Architecture Details

  • Both rectified-flow DiTs use 24 transformer blocks, 1024 hidden dimensions, and 16 attention heads, with stage-specific representations for coarse structure and fine geometry.Stage 1 operates on voxel grids at resolution 16^3; Stage 2 processes sparse voxel features in a 64^3 coordinate space.Source: C Transformer Architecture Details · pp. 14–15

C.1 Single-Part and Multi-Part Transformer Blocks

  • Single-part blocks refine each component independently, while interleaved multi-part blocks exchange information to coordinate relative placement, scale, and structural compatibility.With one part, multi-part attention reduces to standard self-attention.Source: §C.1 · p. 15

C.2 Conditioning with AdaLN

  • AdaLN combines timestep, parts-count, and per-part image-region embeddings; each part’s conditioning affects only its own tokens.A shared MLP produces per-part AdaLN parameters applied by indexed broadcasting.Source: §C.2 · p. 15

D Inference and Sampling Procedure

  • Inference samples the two stages with rectified-flow integration using 50 and 30 steps, respectively, and applies image-conditioning CFG at scale 3.0.No test-time optimization or post-processing is used; decoded parts align naturally in a shared canonical space.Source: D Inference and Sampling Procedure · p. 15

E Dataset Construction Details

PartObjectNet combines curated, spatially aligned part decompositions at scale; Seg3DParts also handles fully occluded parts using blank image conditions and cross-part attention.

E Dataset Construction Details

  • PartObjectNet aggregates Objaverse, Texverse, and PartNet objects with 2–15 semantic parts, filtered through automatic checks and manual curation.Filtering removes low-quality meshes, scanned artifacts, scene-level assets, and semantically inconsistent decompositions.Source: E Dataset Construction Details · p. 15
  • Approximately 200K objects retain original part scales and relative placements, providing aligned part-level supervision.The dataset spans diverse categories and decomposition styles.Source: E Dataset Construction Details · p. 15

F Fully Occluded Parts

  • Figure 2 illustrates recovered occluded parts, including a juice tin’s pour-out opening and a toolbox’s inner lid.Source: Figure 2 · p. 16
  • Fully occluded parts receive all-black image crops, matching the training condition for parts hidden from the training view.Cross-part attention then infers their shape and placement from visible parts.Source: F Fully Occluded Parts · p. 16

G Limitations

Generated geometry may degrade when input part segmentation is low quality or part boundaries are semantically ambiguous.

G Limitations

  • Seg3DParts’ performance depends on input part-segmentation quality.Source: G Limitations · p. 16
  • Low-quality segmentation or semantically ambiguous boundaries may degrade generated geometry.Source: G Limitations · p. 16
  • Figure 3 presents failure cases associated with low-quality segmentation and ambiguous part boundaries.Source: Figure 3 · p. 16

H More Results

Additional qualitative results span diverse object categories and part decompositions, and are presented as evidence of generalization and coherence.

H More Results

  • Figures 4 and 5 show qualitative results across diverse categories and part decompositions, demonstrating generalization and coherence.Source: H More Results · p. 17
  • The additional results label condition images alongside generated results.Source: H More Results · p. 17
  • Figure 4 presents an additional set of qualitative results (Part I).Source: H More Results · p. 17
  • Figure 5 presents an additional set of qualitative results (Part II).Source: Figure 5 · p. 18
Loading 2609.36918v1…