PaperlayerView exact revision on arXiv

Source-linked AI summary

A Unified Framework for Image-to-3D Part Generation via Variable Granularity

Qiwen Gu, Xin Zhou, Xiwu Chen, Zhongyang Zhu, Zhuo Zhang, Junqiao Zhao

arXiv:2610.31671v1cs.CV

TL;DR

Existing image-to-3D part-generation methods lack flexibility across users’ desired levels of geometric control. FlexPart unifies point, box, and mask prompts with asymmetric geometric guidance; under mask conditioning on PartVerseXL, it reduces Part CD against two baselines while maintaining competitive whole-mesh quality.

  • Problem

    Existing part-generation methods typically specialize in one prompt granularity, limiting flexible transitions between sparse interaction and dense spatial control.

    Source: §1 · p. 2

  • Method

    FlexPart integrates point, box, and mask prompts in one model and uses asymmetric geometric guidance to align sparse-point representations with stronger spatial priors.

    Source: §1 · p. 2

  • Results

    Under mask conditioning on PartVerseXL, FlexPart reduces Part CD by 16.6% relative to TripoSG + SAM3D and by 10.5% relative to OmniPart, while maintaining competitive whole-mesh quality.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments support joint multi-granularity learning as a practical approach to flexible 3D part generation with improved part-level quality and competitive whole-mesh fidelity.

    Source: §7 · p. 10

    Single-view 2D conditions provide limited depth cues, and occluded parts may be placed incorrectly in visible regions.

    Source: §7 · p. 10

Abstract

from arXiv · show

Controllable 3D part generation is pivotal for digital asset creation, yet existing methods lack the flexibility to accommodate diverse user workflows requiring varying levels of control. In this paper, we introduce FlexPart, a unified image-to-3D part generation framework that supports variable-granularity geometric conditions within a single model. By seamlessly integrating heterogeneous inputs, ranging from sparse points to dense bounding boxes and masks, via gated adaptive modulation, FlexPart unlocks a powerful cross-modal synergy. We also propose an asymmetric geometric guidance mechanism that leverages strong spatial priors to align representations across granularities, improving geometric consistency and facilitating inter-task synergy. Under mask conditioning on PartVerseXL, FlexPart reduces Part CD by 16.6% relative to TripoSG + SAM3D and by 10.5% relative to OmniPart, while maintaining competitive whole-mesh quality. The code can be found at https://github.com/Gucci233/FlexPart

1 Introduction

Part generation must accommodate different desired segmentation granularities, but existing methods typically specialize in one prompt type. FlexPart unifies point, box, and mask control and reports improved part-level quality while preserving whole-mesh fidelity.

1 Introduction

  • Existing approaches specialize in a single prompt granularity, limiting transitions between dense-mask control and sparse-point interaction despite complementary information across modalities.Dense masks and boxes provide strong spatial constraints, while lighter forms of supervision reduce annotation effort but often sacrifice precise spatial control.Source: §1 · p. 2
  • FlexPart integrates points, boxes, and masks in one DiT through gated adaptive modulation, using strong priors to supervise sparse-point representations during training.The resulting model supports different prompt types at inference without retraining.Source: §1 · p. 2
  • Under mask conditioning on PartVerseXL, FlexPart reduces Part CD by 16.6% versus TripoSG + SAM3D and by 10.5% versus OmniPart, while preserving strong whole-mesh reconstruction fidelity.The paper frames these results as superior part-level generation quality with strong whole-mesh reconstruction fidelity.Source: §1 · p. 2

2 Related Work

Prior work spans holistic 3D generation, shape analysis and segmentation, and part-level generation, with differing approaches to representing and controlling parts.

2 Related Work

  • Holistic 3D generators produce whole meshes without explicit part decomposition, while ROAD, HERMES, and HERMES++ connect 3D analysis and generation [24] [51] [52].The related-work discussion positions FlexPart as focused on joint learning across geometric prompt granularities for controllable object-part generation.Source: §2 · pp. 2–3
  • Shape-analysis and segmentation methods analyze existing shapes or lift 2D foundation-model priors to 3D [15] [26] [29], whereas FlexPart generates complete parts from images and geometric prompts.HoloPart [43] completes initially segmented regions into full 3D parts, but these pipelines rely on input geometry.Source: §2 · p. 3
  • Part-generation methods trade explicit spatial control for easier interaction: masks and layouts [8] [9] [46] offer control, while PartCrafter [20] and PartPacker [35] internalize decomposition.FlexPart unifies point, box, and mask prompts in one model, allowing users to choose spatial-control granularity while benefiting from joint training.Source: §2 · p. 3

3 Preliminaries

Optimal transport models soft correspondences between feature sets, while partial optimal transport relaxes complete matching to accommodate ambiguous or unmatched elements.

3 Preliminaries

  • Optimal transport estimates a plan P between feature sets of sizes N and M, with entries representing transported mass and marginal constraints defining valid plans.The source and target mass vectors are r and q, respectively.Source: §3 · p. 3
  • Entropic regularization supports efficient, differentiable optimization; the solution has a factorized form and can be computed with Sinkhorn–Knopp.The cost matrix C and regularization parameter ε define the formulation.Source: §3 · p. 3
  • Partial optimal transport adds a dustbin row and column to absorb unmatched mass, relaxing strict mass conservation for partial or ambiguous assignments.The framework overview uses optimal transport during training to align weak and strong geometric priors.Source: §3 · p. 3, Figure 2 · p. 4

4 Methodology

FlexPart combines geometric prompt inputs in a DiT and uses asymmetric geometric guidance to transfer stronger spatial priors to sparse point representations.

4 Methodology

  • FlexPart uses a DiT with Gated AdaLN to inject point, box, and mask conditions of varying precision, alongside geometric guidance for weak supervision.The architecture supports heterogeneous user inputs within one model.Source: §4 · p. 4

4.1 Gated Adaptive Modulation

  • Modality-specific encoders represent point, box, and mask prompts; missing inputs are replaced by learnable null embeddings, and fused part conditions modulate DiT hidden states.DINOv2 image features enter through cross-attention for global context, while Gated AdaLN controls part-level geometry.Source: §4.1 · pp. 4–5

4.2 Asymmetric Geometric Guidance

  • Strong box and mask priors supervise weak point representations through asymmetric alignment, with student and teacher features projected into a shared visual space.Stop-gradient prevents the alignment loss from updating teacher prompt embeddings.Source: §4.2 · p. 5
  • Differentiable partial optimal transport aligns student and teacher features through global competition, transferring box and mask priors to point representations without dense inference prompts.The unified training objective combines conditional Rectified Flow with feature-alignment supervision; inference supports points alone or hierarchical combinations of points, boxes, and masks.Source: §4.2 · pp. 5–6

5 Implementation Details and Metrics

FlexPart is trained in two stages, then evaluated on held-out PartVerseXL and PartObjaverse-Tiny data using separate whole-mesh and matched-part metrics. The evaluation tables distinguish part-level structure from whole-mesh quality and flag entries excluded from direct comparison.

5.1 Implementation Details

  • Two-stage training scales from 1–8 parts with 512 tokens each to 2–16 parts with 1024 tokens per part, using modality-specific prompt dropout.Training uses a composite Objaverse and PartVerseXL dataset.Source: §5.1 · p. 6

5.2 Datasets and Metrics

  • Evaluation uses disjoint held-out PartVerseXL and PartObjaverse-Tiny test sets, selecting the view with the most visible pixels multiplied by visible parts.Source: §5.2 · p. 6
  • Whole-mesh quality is measured with Chamfer Distance and F-Score, while ULIP measures semantic consistency with the input image.The F-Score threshold is 0.1.Source: §5.2 · p. 6
  • Part metrics use CD-based Hungarian matching after filtering fully occluded ground-truth parts; unmatched parts are excluded, and Part CD, IoUB, and IoUV measure deviation and alignment.Benchmark boxes come from ground-truth masks, and points are sampled from their corresponding mask regions.Source: §5.2 · p. 6
  • Table 1 evaluates part-level structure, while Table 2 evaluates whole-mesh quality and excludes overlapping FullPart and full-mesh baseline entries from direct comparison.Source: Table 1 · p. 7, Table 2 · p. 7

6 Experiments

FlexPart directly generates parts across prompt granularities, with stronger part-level results and competitive whole-mesh quality. Ablations show benefits from joint training and asymmetric guidance, alongside robustness to varied prompts and stronger 3D conditions.

6.1 Main Results

  • 10.5% lower Part CD on PartVerseXL than OmniPart demonstrates FlexPart’s part-level mesh-quality advantage under mask conditioning.The comparison contrasts direct part geometry with OmniPart’s intermediate axis-aligned boxes, which can enclose empty space around non-axis-aligned shapes.Source: §6.1 · p. 7
  • FlexPart achieves Mean CD of 0.1418 on PartVerseXL, surpassing OmniPart by 8%, while remaining competitive on PartObjaverse-Tiny.The paper reports strong and consistent mesh-level quality across both benchmarks.Source: §6.1 · p. 7
  • Qualitatively, FlexPart produces distinct, semantically meaningful parts, while alternatives show shape, structure, or alignment shortcomings.Figure 3 places exploded part views above assembled meshes; cascaded SAM pipelines can yield fragments rather than complete functional parts.Source: §6.1 · p. 8, Figure 3 · p. 8, §6.2 · p. 10

6.2 Analysis and Ablation Study

  • Joint training improves Point-only inference by 7.9% in IoUV and 6.1% in CD, and reduces mask-conditioned Part CD from 0.3224 to 0.2938.Table 3 compares evaluation conditions and training settings, including single-modality baselines.Source: §6.2 · p. 8, Table 3 · p. 9
  • Asymmetric Geometric Guidance improves sparse-point results by 7.9% in Part IoUB and 5.0% in Part IoUV, while mask performance remains comparable.The OT-based guidance transfers structural priors from dense annotations to sparse inputs and adds no inference overhead.Source: §6.2 · p. 9, Table 4 · p. 9
  • Under the same 3D-box supervision on PartObjaverse-Tiny, FlexPart outperforms FullPart on most metrics.This comparison uses a FlexPart variant retrained solely with 3D boxes.Source: §6.2 · p. 10, Table 5 · p. 10
  • With box perturbations up to 20%, Part CD rises from 0.3042 to 0.3171; additional box or mask prompts keep point-strategy Part CD within 0.3036–0.3178 or 0.2995–0.3087.The paper also reports Part CD of 0.3287 for half-box, half-point inputs and 0.3190 for an unseen half-mask, half-box combination.Source: §6.2 · p. 10

7 Conclusion

FlexPart unifies geometric prompts of different granularities, using strong spatial priors to improve sparse-point representations during training. Experiments show improved part-level geometry and competitive whole-mesh fidelity, while occlusion remains challenging under single-view conditions.

7 Conclusion

  • FlexPart combines heterogeneous prompts in one model and uses strong spatial priors to improve sparse-point representations during training.Gated adaptive modulation integrates prompts; Asymmetric Geometric Guidance transfers spatial information from boxes and masks to point representations.Source: §7 · p. 10
  • Experiments show improved part-level geometric quality and competitive whole-mesh fidelity across prompt conditions.The conclusion presents joint multi-granularity learning as a practical approach to flexible 3D part generation.Source: §7 · p. 10
  • Occlusion remains challenging: single-view 2D conditions provide limited depth cues, and occluded parts may be placed incorrectly in visible regions.Improving spatial reasoning under occlusion is identified as future work.Source: §7 · p. 10

A Dataset Preparation and Benchmark Details

  • Training uses about 180K effectively utilized models from approximately 250K Objaverse and PartVerseXL models, with watertight representations prepared for training.The evaluation includes a held-out PartVerseXL test set of 100 shapes and PartObjaverse-Tiny shapes with up to 16 parts; meshes are rigidly aligned by ICP before geometric metrics.Source: A Dataset Preparation and Benchmark… · p. 14

A.1 Evaluation Metric Definitions

The appendix defines whole-mesh and matched-part metrics, specifies baseline and alignment protocols, and examines conditioning and alignment choices. The reported comparisons favor Gated AdaLN and partial optimal transport, while gradient flow trades a slight point-prompt improvement for weaker box- and mask-prompt CD.

A.1 Evaluation Metric Definitions

  • Whole CD is symmetric unsquared L2 Chamfer distance over 10K surface samples; part metrics use CD-based Hungarian matches among visible parts, excluding unmatched parts without penalties.Part evaluation also reports bounding-box IoU for coarse extent and surface-based voxel IoU for fine overlap; F-Score uses a distance threshold of 0.1.Source: §A.1 · p. 14

A.2 Baseline Protocol and Data Lineage

  • All baselines use official checkpoints without retraining; the main comparison uses differing native inputs and training sets, with controlled comparisons using specified condition types.Methods are re-evaluated using the same view selection, normalization, ICP alignment, and metrics; PartObjaverse-Tiny shares Objaverse asset lineage with several pretrained methods.Source: §A.2 · p. 15

A.3 Alignment Implementation and Overhead

  • The alignment branch adds 8.9 ms versus a 525 ms DiT forward pass, or 1.70%, in the profiled configuration; this is forward-pass overhead, not total training throughput.The branch has 1.18M parameters, or 0.077% of the model.Source: §A.3 · p. 15

B Conditioning Route Analysis

  • Under mask conditioning, token injection performs worse than Gated AdaLN and the count-conditioned PartCrafter reference; the authors suggest attention-softmax competition as a possible explanation.The comparison supports the conditioning route for this backbone, not a general disadvantage of cross-attention conditioning.Source: B Conditioning Route Analysis · p. 15, Table 6 · p. 15

C.1 Effect of Stop-Gradient Design

  • With gradient flow, whole-mesh CD improves from 0.1525 to 0.1518 for points but worsens from 0.1447 to 0.1531 for boxes and 0.1418 to 0.1512 for masks.The default stops gradients at box/mask embeddings before the Visual Bridge to prevent the alignment loss from updating teacher prompt embeddings.Source: §C.1 · p. 15, Table 7 · p. 16

C.2 Partial OT versus Contrastive Alignment

  • Under point prompts, partial OT reduces Part CD from 0.3499 to 0.3334 versus InfoNCE and achieves lower Part CD and higher voxel and box IoU across all three prompt conditions.The comparison changes only the alignment loss while holding initialization and training budget fixed.Source: §C.2 · p. 16, Table 8 · p. 16

C.3 Global Assignment on Frozen Features

Sinkhorn couples point-to-prior assignments through row and column marginals, addressing ambiguity that row-wise matching leaves unconstrained. A separate dustbin allows unmatched mass when distractor priors have no valid correspondence.

C.3 Global Assignment on Frozen Features

  • Sinkhorn couples assignments through row and column marginals, adding competition among points for available priors beyond independent paired L2 or row-wise InfoNCE matching.The matching comparison is conducted on the same frozen features.Source: §C.3 · p. 16
  • On frozen features, Sinkhorn raises Top-1 matching from 0.803 to 0.895 and reduces many-to-one collapse by approximately 5.5×; symmetric-pair accuracy is 0.516 versus 0.931 for other pairs.The diagnostic uses 199 PartObjaverse-Tiny objects and 1,330 parts without diffusion sampling or mesh decoding.Source: §C.3 · pp. 16–17

C.4 Dustbin for Unmatched Priors

  • With four nonmatching cross-object box-prior distractors, partial OT raises Top-1 matching from 0.677 to 0.747 versus strict OT.The dustbin provides a channel for unmatched mass rather than requiring assignment to irrelevant priors.Source: §C.4 · p. 17, Table 10 · p. 17

D Adaptability to Input Variations

FlexPart is evaluated under noisy, partial, mixed, and human-provided prompts, with efficiency and point-placement sensitivity also examined. Stronger geometric conditions reduce point-placement sensitivity, while single-view human-prompt scores measure agreement with dataset parts rather than satisfaction of user intent.

D Adaptability to Input Variations

  • With part-count-only input, spectral clustering derives point prompts from image patch features and coordinates, and this strategy outperforms PartCrafter in preliminary experiments.The method requires at least point-level prompts, so points are automatically derived from the image.Source: D Adaptability to Input Variations · pp. 17–18
  • FlexPart maintains reasonable performance across noisy, partial, and mixed input settings, with stronger geometric conditions providing more precise localization and weaker inputs offering greater flexibility.Tests include boxes perturbed by up to 20%, boxes for half the parts combined with points for the rest, and an unseen-at-training mix of masks and boxes.Source: D Adaptability to Input Variations · pp. 17–18

E Evaluation with Human-Provided Prompts

  • The human-prompt study uses points and boxes from three annotators on 50 PartObjaverse-Tiny objects, but its scores measure dataset agreement rather than satisfaction of intended decompositions.The small-scale study does not measure interaction time or downstream usability.Source: E Evaluation with Human-Provided Prompts · p. 18, Table 12 · p. 18

F Part Generation Efficiency

  • FlexPart has competitive inference speed, with slight latency overhead versus PartCrafter and more computation than latent-downsampling OmniPart at higher resolution.The authors describe a balance between generation quality and speed and note potential for further acceleration.Source: Figure 5 · p. 18, F Part Generation Efficiency · p. 19
  • Peak VRAM is approximately 6–8 GB, comparable to PartCrafter; OmniPart and FullPart use nearly twice as much, while doubling parts from 8 to 16 raises memory only marginally.The comparison is for object generation under the reported evaluation settings.Source: Figure 5 · p. 18, F Part Generation Efficiency · p. 19

G Sensitivity of Point Prompts

  • With Point Only inputs, Edge prompts raise CD from 0.3302 to 0.4190 and lower IoUV from 0.1905 to 0.1342, showing sensitivity to point placement.The point-placement strategies include Center, Edge, Offset, and Cluster.Source: G Sensitivity of Point Prompts · p. 19, Table 13 · p. 20
  • With Point + Box inputs, CD varies only from 0.3036 to 0.3178; with Point + Mask, IoUV ranges from 0.1962 to 0.2088.The reported trend is that point-placement influence is less pronounced when stronger geometric conditions are present.Source: G Sensitivity of Point Prompts · p. 19, Table 13 · p. 20
  • Qualitative comparisons show reduced geometric fidelity for ambiguous edge points in Point Only conditions, and overall sensitivity is substantially lower with stronger prompts.Figure 6 depicts outcomes under the four point-sampling strategies.Source: Figure 6 · p. 19, G Sensitivity of Point Prompts · p. 20

H Analysis of SAM-based Cascaded Pipelines

SAM-derived masks do not match ground-truth-mask performance, and direct point conditioning outperforms both tested SAM-mask settings. The authors suggest incomplete masks can carry segmentation errors into 3D generation, while FlexPart models heterogeneous conditions directly.

H Analysis of SAM-based Cascaded Pipelines

  • CD rises from 0.2938 to 0.3387 and IoUV falls from 0.2049 to 0.1731 with center-sampled SAM masks versus ground-truth masks; direct points outperform both SAM-mask settings.Direct point conditioning achieves Part CD 0.3334, IoUV 0.1902, and IoUB 0.2506.Source: H Analysis of SAM-based Cascaded… · p. 20
  • SAM typically segments a localized fragment rather than the intended complete functional part, and incomplete masks can propagate segmentation errors into 3D generation.Figure 7 compares SAM masks generated from a single point near the visual center with ground-truth segmentation masks.Source: H Analysis of SAM-based Cascaded… · p. 20, Figure 7 · p. 21
  • FlexPart directly models heterogeneous conditions, learning semantics from sparse prompts and accommodating structured 3D bounding boxes that do not reduce neatly to 2D masks.Source: H Analysis of SAM-based Cascaded… · p. 20

I Additional Qualitative Results

Extended qualitative results show FlexPart generating part-level 3D objects across geometric prompts, including objects with complex topologies. The results describe high-fidelity meshes with clean topology and alignment across interaction modes.

I Additional Qualitative Results

Related papers

Loading 2610.31671v1…