PaperlayerView exact revision on arXiv

Source-linked AI summary

FlexPart: Text-Controlled Semantics-Aware 3D Part Segmentation at Flexible Granularity

Yiqun Zhao, Jiachen Xu, Yihao Zhi, Yunfei Zhao, Zibo Zhao, Xintong Han, Chunchao Guo, Shenghua Gao

arXiv:2610.23659v1cs.CV

TL;DR

3D part segmentation must accommodate different valid decompositions, but current data and control mechanisms offer limited multi-granularity, holistic user-intent guidance. FLEXPART combines structured text prompts, VLM-generated annotations, and native 3D part-ID fields. It leads evaluated baselines at every FLEXPARTBENCH granularity and outperforms all baselines on PartObjaverse-Tiny.

  • Problem

    Existing 3D datasets typically lack multi-granularity semantic annotations, while localized or view-dependent control signals do not provide coherent holistic partitions.

    Source: §1 · p. 2

  • Method

    FLEXPART uses prompts pairing part names with location and shape clauses, VLM-generated multi-granularity training data, and a native 3D part-identity field with randomized background dropping.

    Source: §1 · p. 2, §3.2 · pp. 4–5

  • Results

    FLEXPART outperforms all baselines at every FLEXPARTBENCH granularity, surpassing PartField by 0.255 mIoU on coarse functional grouping.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    Users can specify target parts and granularity in text, and a VLM planner can complete location and shape clauses from renderings.

    Source: §1 · p. 2

    An agentic workflow for iteratively refining prompts after users inspect returned partitions remains a future direction.

    Source: §5 · p. 9

Abstract

from arXiv · show

A 3D asset admits multiple valid part decompositions, while semantics-aware segmentation should reflect both the desired granularity and the parts a user wants to extract. Ideally, a practical segmentation model should strive for a simple principle: what you ask for is what you get. However, realizing this goal is hindered by two core obstacles: the lack of multi-granularity training data and the absence of a holistic control mechanism. We propose FLEXPART, a unified framework for text-controlled 3D mesh segmentation. To establish flexible control, we introduce a holistic control prompt comprising semantic part names with location and shape clauses. To support this mechanism, we utilize an automated Vision-Language Model (VLM) pipeline to construct a large-scale dataset of multi-granularity partitions and their corresponding prompts. Driven by this data, FLEXPART generates a view-independent part-identity field in native 3D space, directly assigning discrete part IDs to mesh faces. To compel the model to respect semantic conditions rather than rely on geometric priors, we train the model using randomized background dropping. Furthermore, an inference-time VLM planner can automatically complete descriptive clauses from 2D renderings, allowing users to simply name the desired parts in ordinary words. FLEXPART with planner-completed prompts outperforms the strongest evaluated baseline by 15% in mIoU, even when that baseline receives ground-truth guidance. On our held-out FLEXPARTBENCH, FLEXPART leads at each granularity and improves average mIoU by 22% over the strongest baseline. With only the requested part names supplied by the user, it also improves IoU by 35% over the strongest click-driven baseline on selective segmentation.

1. Introduction

A single 3D asset can support different valid part decompositions, but existing data and control signals do not adequately capture user intent across granularities. FLEXPART addresses these gaps with text prompts, multi-granularity annotations, and a native 3D segmentation model.

1. Introduction

  • A single asset can be partitioned into coarse functional groups, accessories, or named parts depending on the user's task.Part segmentation provides mesh-surface boundaries useful for editing, rigging, animation, and 3D printing.Source: §1 · p. 2
  • Existing datasets generally offer one fixed partition per geometry, leaving methods without multi-granularity supervision biased toward geometric cues rather than varied user intent.For example, a fixed split may divide an arm even when a user wants the whole arm as one part.Source: §1 · p. 2
  • Global scale, threshold, or part-count controls do not identify semantically important boundaries, while single-view masks leave occluded regions unspecified.Text schemas also lack spatial and geometric grounding.Source: §1 · p. 2
  • A holistic prompt pairs target part names with location and shape clauses, while a VLM pipeline creates multi-granularity partitions and corresponding prompts.The pipeline annotates 57K objects, yielding 150K prompt-target pairs.Source: §1 · p. 2
  • FLEXPART generates a native 3D part-identity field and uses randomized background dropping to support selective segmentation and prioritize semantic prompts over geometric priors.A VLM planner can infer descriptive clauses from 2D renderings when users provide part names.Source: §1 · p. 2
  • FLEXPART outperforms evaluated baselines on PartObjaverse-Tiny, leads at every granularity on FLEXPARTBENCH, and significantly improves selective part segmentation.The introduction reports these results without giving numerical values.Source: §1 · p. 3

2. Related Work

Prior work uses 2D priors, learned 3D features, or control prompts, but these approaches leave gaps in semantic intent, holistic control, or user-directed structure. FLEXPART provides text-controlled segmentation of an existing shape at a user-specified granularity.

2. Related Work

  • 2D-prior and open-vocabulary methods can be costly or view-sensitive, while fixed-split 3D methods remain governed by geometric cues rather than user-intended semantic groupings.The related work also notes multi-view inconsistencies and fragmented mesh boundaries for approaches that aggregate 2D predictions.Source: §2 · p. 3
  • Global heuristics specify scale or region count but not semantic boundaries; clicks leave grouping implicit, and single-view guidance leaves occluded regions unspecified.Text-schema approaches provide text control but lack spatial and geometric grounding and multi-granularity training.Source: §2 · p. 3
  • Part-generation methods either discover structure without explicit user control or depend on an input decomposition, whereas FLEXPART segments the existing shape from text at a chosen granularity.Two-stage generation performance is bounded by its input segmentation.Source: §2 · p. 3

3. Method

FLEXPART generates view-independent 3D part-ID fields from text prompts, with a part-aware VAE and text-conditioned DiT. Its prompt format and multi-granularity training data support flexible partitions and selective segmentation.

3. Method

  • FLEXPART formulates segmentation as a continuous 3D part-identity field, generated from a holistic text prompt and decoded into face labels.A part-aware VAE encodes geometry and part IDs, while a text-conditioned dual-stream DiT generates the latent representation.Source: §3 · p. 3

3.1. Preliminary

  • Voxel-anchored latents provide spatial guidance for generating fine-grained surface attributes, which the decoder predicts at queried 3D locations.The field represents local geometry and, in FLEXPART, the SDF and binary-encoded part IDs.Source: §3.1 · pp. 3–4

3.2. Formulating segmentation as text-controlled part-ID generation

  • Each prompt item pairs a semantic part name with location and shape clauses; discrete binary IDs align output labels with prompt order.An all-ones ID is reserved for background, and the six-bit representation supports up to 63 distinct parts.Source: §3.2 · p. 4

3.3. Data construction

  • A VLM pipeline creates up to three partitions from artist-authored components, while randomized drop-to-background augmentation supports selective segmentation.The inference-time planner can use multi-view renderings to complete semantic prompts, including remaining background categories.Source: §3.3 · p. 5

4. Experiments

FLEXPART controls semantic granularity and selective segmentation through text prompts, and outperforms evaluated baselines across the reported tasks. Its experiments assess multi-granularity data, benchmark performance, ablations, and generated-asset generalization.

4.1. Experimental setup

  • Training annotations yield 151,492 prompt-target pairs from 57K HY3D-Bench objects; evaluation includes PartObjaverse-Tiny, held-out FLEXPARTBENCH, and Selective-190.FLEXPARTBENCH has 364 assets annotated at three granularities with no training overlap; Selective-190 contains 190 manually constructed cases across 28 benchmark objects.Source: §4.1 · p. 5
  • The evaluation reports class-agnostic mIoU, -direct mIoU without rematching for FLEXPART, and separate named-part and background IoUs for selective segmentation.Baselines are tested with their intended inputs, including ground-truth part counts for PartField and both annotated and automatic conditions for SegviGen.Source: §4.1 · pp. 5–6

4.2. Main results

  • With planner-completed prompts, FLEXPART-Auto loses 0.010 mIoU against -GT, while SegviGen-Auto loses 0.181 mIoU.The planner completes location and shape clauses from renderings after users provide part names.Source: §4.2 · p. 7
  • FLEXPART achieves the highest accuracy on PartObjaverse-Tiny and its -direct variant still outperforms all baselines without region-to-ground-truth rematching.The figure contrasts FLEXPART’s semantic intent-following with baseline failures involving occlusions, geometric seams, or imprecise boundaries.Source: Figure 3 · p. 6, §4.2 · p. 7, Table 1 · p. 7
  • FLEXPART outperforms click-driven baselines on selective segmentation, including with planner-generated prompts from part names alone.The task merges unselected geometry into one background region; click-driven baselines receive ground-truth points inside each target part.Source: Table 3 · p. 7, §4.2 · p. 7
  • FLEXPART outperforms all baselines at every granularity on FLEXPARTBENCH, and surpasses PartField by 0.255 mIoU for coarse functional grouping.Changing the text prompt shifts the partition from broad assemblies to fine-grained accessories.Source: Figure 4 · p. 6, §4.2 · p. 7, Table 2 · p. 7
  • Drop-to-background training improves full accuracy from 0.818 to 0.845, while discrete part IDs avoid the fragmentation seen with an RGB palette.The ablation also identifies drop-to-background training as essential for selective segmentation.Source: Table 4 · p. 7, §4.2 · p. 8, Figure 6 · p. 8
  • On four Hunyuan3D 3.1 meshes without ground-truth annotations, the planner constructs prompts for coarse, medium, and fine partitions from renderings and target granularity.The reported qualitative results show broad functional groups for coarse prompts and atomic decompositions for fine prompts.Source: §4.2 · p. 8, Figure 7 · p. 8

5. Conclusion

FLEXPART combines holistic text prompts, VLM-annotated multi-granularity data, and a native 3D part-identity field for intent-driven segmentation. Its reported experiments show superiority over strong baselines, while the conclusion identifies ambiguity in describing repeated or adjacent parts as a limitation.

5. Conclusion

  • FLEXPART unifies text-controlled segmentation using holistic prompts and VLM-annotated data to condition a native 3D part-identity field.Randomized background dropping is used to encourage semantic instructions over rigid geometric priors.Source: §5 · p. 8
  • The authors report significant superiority over strong baselines on multi-granularity and selective segmentation tasks.They identify ambiguity in location clauses for repeated or symmetric parts and overlapping descriptions for adjacent boundaries.Source: §5 · p. 8
  • An iterative language-model-and-user workflow for refining prompts after inspecting partitions is proposed as future work.The conclusion describes cases where sails can merge and boundaries between legs and armour can remain ambiguous.Source: §5 · pp. 8–9

A.1. Annotation pipeline

The annotation pipeline creates up to three semantic partitions from artist-authored components, then generates structured text prompts for each group. Multi-view rendering and object-level orientation rules support descriptions that remain consistent across views.

Stage 1: partitions

  • The grouping call clusters artist-authored components into coarse, middle, and fine levels, with each component assigned exactly once per level.Coarse groups represent functional assemblies; middle groups separate symmetric or functionally distinct subparts; fine groups capture semantically distinct parts.Source: §A.1 · p. 12
  • Each surviving partition maps the part IDs in each group to face labels, while duplicate levels are dropped and coarse groups need not be unions of fine groups.The three levels are requested independently, so an asset yields up to three segmentation targets.Source: §A.1 · p. 12

Stage 2: text annotation

  • Each group is annotated in a separate call using four viewpoints and three renderings that show its context, unoccluded shape, and full scene.The object-level caption is included as read-only context, and each response returns a JSON object for the group.Source: Figure 9 · p. 12, §A.1 · p. 13
  • Per-group names, spatial relations, and shapes are aggregated in group-ID order into the training prompt, alongside the object caption.The aggregation maps PartName to the item name, SpatialRelation to its location clause, and Shape to its shape clause.Source: §A.1 · p. 13, Figure 10 · p. 13

Consistent location clauses

  • Spatial descriptions use fixed global directions only when the object has a well-defined front; otherwise they use relative relations between named parts.The front decision is estimated once per object, avoiding left/right reversals across views.Source: §A.1 · pp. 13–14

A.2. Additional ablation studies

The ablations find that multi-granularity training improves performance across semantic levels, while ordered prompt–label binding is needed for direct semantic-to-channel prediction.

Multi-granularity training

  • 0.845 vs. 0.708 average mIoU: multi-granularity training improves performance across semantic levels compared with training on fine partitions alone.Fine-only training retains adequate fine-level understanding but performs poorly at coarse and medium granularities.Source: §A.2 · p. 14, Table 5 · p. 14

Prompt–label order binding

  • 0.804 vs. 0.011 Direct mIoU: ordered training restores direct prediction, while unordered training fails to establish a direct semantic-to-channel mapping.Ordered training also raises Matched mIoU to 0.845 from 0.767.Source: §A.2 · p. 14, Table 6 · p. 14

A.3. Implementation details

FLEXPART uses a text-conditioned diffusion architecture and trains it with prompt perturbations. At inference, a separate visual planner completes user-supplied part names or selects an inventory from renderings and a target granularity.

Architecture

  • The denoiser is an MM-DiT with 12 double-stream and 24 single-stream blocks; frozen Qwen3-VL-4B-Instruct encodes text, and a part-aware VAE encodes the part-ID field.Text is dropped for 10% of training samples to provide the unconditional branch for classifier-free guidance.Source: §A.3 · p. 14

Training

  • Training runs for 125K steps with AdamW and a cosine schedule; inference uses a 30-step flow-matching Euler sampler with guidance scale 6.0.The denoiser learning rate is 10^-4 and the text-projection learning rate is 5 × 10^-5.Source: §A.3 · p. 14

Prompt perturbations

  • Drop-to-background removes a uniformly sampled fraction of prompt items, up to 0.65, and assigns their faces to background; part-list shuffling also permutes items and relabels targets.Shuffling makes each label index correspond to the item’s position in the prompt list.Source: §A.3 · p. 14

Inference-time planner

  • The planner uses renderings of the bare mesh and operates either by completing user-listed names or by choosing parts from renderings and a target granularity.For selective requests, it adds background items for the unlisted surface; without part names, it decides the inventory itself.Source: §A.3 · p. 15, Figure 11 · p. 15

A.4. Additional experiments

The additional experiments define benchmark construction and baseline protocols, and report background leakage on Selective-190. The benchmark partitions are manually reviewed and frozen before model evaluation.

Benchmark construction

  • PartObjaverse-Tiny uses watertight meshes, with each new face inheriting the label of its nearest original face, and the released decomposition is used without modification.The protocol adds object captions and name, location, and shape clauses; Selective-190 uses named subsets of annotated parts and assigns the remainder to background.Source: §A.4 · p. 15

Constructing FLEXPARTBENCH

  • FLEXPARTBENCH retains 364 objects and 1,092 annotated partitions after automatic checks and human review; its fine partitions contain 10–28 parts, with a median of 23.The retained objects include 152 from PartNeXt and 212 from PartVerse-XL.Source: §A.4 · pp. 15–16

Baseline protocols

  • Baselines use released weights in their intended regimes, with method-specific inputs and settings; all text-conditioned methods receive the same English part names.For example, SAMPart3D uses one test-time optimization per object, while PartField receives the ground-truth part count.Source: §A.4 · p. 16

Background leakage on Selective-190

  • 0.024 of Selective-190 background is assigned to named parts by FLEXPART, compared with 0.171 for Find3D and 0.976 for the no-drop-to-background ablation.The leakage fraction measures ground-truth background area predicted as a named part.Source: §A.4 · p. 16

A.5. Does the model use semantics beyond part count?

The section tests whether FLEXPART follows semantic prompt content beyond part count, compares its input-surface labeling with CubePart’s generated parts, and notes broader impacts and data biases.

A.5. Does the model use semantics beyond part count?

  • With geometry and part count K fixed, generic or semantically mismatched prompts reduce mIoU, indicating predictions depend on requested semantics, not only K-way geometry.The test evaluates sensitivity to semantic conditioning, not alternative valid decompositions of the same object.Source: §A.5 · p. 16

A.6. Comparison with CubePart

  • Across three visual case studies, CubePart shows merged or fragmented parts at finer levels, while FLEXPART labels the unchanged input surface; these cases do not estimate CubePart’s overall failure rate.CubePart generates separate part meshes, whereas FLEXPART retains the input surface and predicts face labels.Source: §A.6 · p. 17, Figure 12 · p. 18

A.7. Broader impact

  • Part decompositions support editing, rigging, retopology, and semantic reuse, while FLEXPART’s training data may skew its part vocabulary and notion of part toward common categories and artists’ conventions.The paper also identifies possible misuse in repurposing copyrighted assets without attribution.Source: §A.7 · p. 17

Related papers

Loading 2610.23659v1…