Source-linked AI summary
SegviGen: Repurposing 3D Generative Model for Part Segmentation
Lin Li, Haoran Feng, Zehuan Huang, Haohua Chen, Wenbo Nie, Shaohua Hou, Keqing Fan, Pan Hu, Sheng Wang, Buyu Li, Lu Sheng
TL;DR
Existing 3D part segmentation methods either suffer from inaccurate boundaries and cross-view inconsistency or require costly, large-scale 3D annotations. SegviGen repurposes pretrained 3D generative priors through part-wise colorization, achieving stronger interactive and full segmentation with 0.32% of the training data.
Problem
Existing methods produce erroneous regions and imprecise boundaries, while native 3D approaches require costly, large-scale datasets with curated part annotations.
Method
SegviGen formulates 3D part segmentation as part-indicative colorization alongside geometry reconstruction in a unified, conditionable generative framework.
Results
SegviGen surpasses prior state of the art by 40% on interactive part segmentation and 15% on unguided full segmentation using 0.32% of the training data.
Takeaways & Limitations
Pretrained 3D generative priors transfer effectively to 3D part segmentation, supporting multiple task settings with limited supervision.
Takeaways & Limitations
Semantic ambiguity can produce too many or too few parts, while overly fine-grained 2D guidance may reduce boundary precision and smoothness.
Abstract
from arXiv · showhide
We introduce SegviGen, a framework that repurposes native 3D generative models for 3D part segmentation. Existing pipelines either lift strong 2D priors into 3D via distillation or multi-view mask aggregation, often suffering from cross-view inconsistency and blurred boundaries, or explore native 3D discriminative segmentation, which typically requires large-scale annotated 3D data and substantial training resources. In contrast, SegviGen leverages the structured priors encoded in pretrained 3D generative model to induce segmentation through distinctive part colorization, establishing a novel and efficient framework for part segmentation. Specifically, SegviGen encodes a 3D asset and predicts part-indicative colors on active voxels of a geometry-aligned reconstruction. It supports interactive part segmentation, full segmentation, and full segmentation with 2D guidance in a unified framework. Extensive experiments show that SegviGen improves over the prior state of the art by 40% on interactive part segmentation and by 15% on full segmentation, while using only 0.32% of the labeled training data. It demonstrates that pretrained 3D generative priors transfer effectively to 3D part segmentation, enabling strong performance with limited supervision. See our project page at https://fenghora.github.io/SegviGen-Page/.
1 INTRODUCTION
3D part segmentation supports important downstream applications, but existing 2D-lifting and native 3D approaches face quality, consistency, computational, and annotation challenges. SegviGen repurposes pretrained 3D generative priors through part-wise colorization to provide unified segmentation modes with limited supervision.
- Motivation: Part segmentation provides explicit 3D structure for content creation, spatial intelligence, editing, animation rigging, 3D printing, and other industrial applications.Existing methods can produce erroneous regions and imprecise boundaries that limit practical usability.
- Related approaches: 2D-to-3D lifting methods inherit domain gaps, multi-view alignment issues, computational overhead, long training cycles, cross-view inconsistency, and blurry or imprecise boundaries.These pipelines may distill 2D representations or fuse masks back-projected from multi-view images.
- Related approaches: Native 3D part segmentation avoids shortcomings of 2D priors but typically requires large-scale curated 3D annotations that are costly and inconsistent in granularity, hierarchy, and boundary definitions.This annotation burden limits the scalability of native discriminative segmentation approaches.
- Motivation: Pretrained 3D generative models encode geometry, appearance, and part-level structure from unannotated textured assets, offering priors that can support sharper boundaries with less dense annotation and task-specific training.The paper asks how these priors can be transferred effectively to improve segmentation quality and data efficiency.
- Method: SegviGen formulates part segmentation as colorization, jointly reconstructing geometry and predicting part-indicative colors while supporting interactive, full, and 2D-guided full segmentation.A 2D segmentation map can customize the decomposition in the guided full-segmentation setting.
- Results: 40% improvement on interactive part segmentation and 15% improvement on full segmentation are achieved over the prior state of the art using only 0.32% of the training data.The reported interactive gain includes IoU@1, while the full-segmentation gain is reported in overall IoU averaged across datasets.
3 METHOLODOGY
SegviGen unifies interactive, full, and 2D-guided 3D part segmentation by recasting segmentation as conditional colorization in a pretrained 3D generative latent space. The model conditions reconstruction and color prediction on geometry, task-specific inputs, timestep, and task identity.
- Overall framework: SegviGen predicts part-indicative colors on active voxels while reconstructing the underlying geometry, yielding segmentation from a unified 3D representation.The framework supports interactive segmentation, full segmentation, and full segmentation with 2D guidance.
- Overall framework: The sparse voxel representation aligns geometry and texture features in 3D, while pretrained structured latents provide transferable shape-and-appearance priors.A Sparse Compression VAE encodes voxelized assets, and the latent generator learns geometry- and texture-consistent representations.
- Task-specific conditioning: Interactive segmentation uses user clicks for binary target-part extraction, whereas full segmentation assigns distinct colors to multiple parts using 10 independently sampled palettes per shape.The shared colorization formulation preserves one model interface across segmentation settings.
- Overall framework: Training encodes geometry and part-color targets with a pretrained 3D VAE, adds noise to the color latent, and predicts the noise residual with conditional flow matching.The model conditions prediction on the noisy color latent, geometry latent, task inputs, task embedding, and timestep.
- Task-specific conditioning: 2D-guided full segmentation transfers a user-specified image-space decomposition into 3D through image-conditioning tokens injected by cross-attention.The same interface uses padded point tokens alongside image guidance, allowing finer or coarser decompositions specified by the guidance map.
- Task embedding: Task identity is encoded continuously and fused with the timestep signal so one backbone can learn task-discriminative behavior across interleaved multi-task training.A sinusoidal task encoding is mapped to an embedding and combined additively with the timestep embedding.
4.1 Setting
SegviGen is evaluated for full and interactive part segmentation against established baselines on two textured-mesh datasets, using IoU-based metrics. The experiments also assess mesh-level color transfer and report sharp, high-accuracy full segmentation from 3D input.
- Datasets: SegviGen is evaluated on PartObjaverse-Tiny and PartNeXT, using 200 and 300 textured mesh objects, respectively.
- Baselines: Full segmentation is compared against P3-SAM, Find3D, SAMPart3D, and PartField.
- Baselines: Interactive segmentation is compared against P3-SAM and Point-SAM.
- Results: SegviGen achieves high-accuracy full segmentation with sharp part boundaries using only 3D input.
- Metrics: IoU@N measures average mask overlap over foreground clicks, while full segmentation uses IoU for overall mask accuracy.
- Mesh processing: Predicted active-voxel colors transfer to the original mesh through nearest-voxel assignment and vertex-based majority voting for face labels.
4.2 Main Results
SegviGen outperforms native 3D baselines for interactive and full part segmentation, including settings with 2D guidance, while generalizing qualitatively to AI-generated meshes.
- Interactive Part-Segmentation: SegviGen achieves 42.49% IoU@1 on PartObjaverse-Tiny and 54.86% on PartNext, surpassing Point-SAM by approximately 17.6% and 31.0%, respectively.These results are reported for the most challenging one-click interactive setting.
- Interactive Part-Segmentation: As clicks increase from 1 to 10, SegviGen reaches 82.73% IoU on PartNext, compared with 65.04% for Point-SAM and 53.81% for P3-SAM.The method shows steady performance gains as user feedback increases.
- Full Segmentation: Without 2D guidance, SegviGen achieves 55.40% IoU on PartNext and 50.64% on PartObjaverse-Tiny, maintaining stronger cross-dataset performance than competing methods.On PartNext, it outperforms PartField at 41.50% and SAMPart3D at 29.62%.
- Full Segmentation: With a single-view 2D segmentation map, SegviGen reaches 62.98% IoU on PartObjaverse-Tiny and 71.53% on PartNext, achieving state-of-the-art results on both datasets.The 2D-guided mode combines 2D semantic cues with the 3D generative framework.
- Generalization: SegviGen produces plausible interactive and full segmentations on AI-generated meshes without additional training.These meshes differ from the training and benchmark assets and lack ground-truth part annotations.
- 2D Guidance: Different 2D viewpoints consistently transfer visible regions, while occluded regions may receive different but still plausible color assignments.The variation is attributed to label-assignment ambiguity rather than incorrect decomposition.
4.3 Ablation Studies and Analysis
Ablations show that explicit coordinate encoding improves later-stage interactive segmentation, while 12 sampling steps provide a practical accuracy–efficiency balance.
- Point Embedding Mechanism: The point-embedding ablation compares Explicit Coordinate Encoding with Label-based Semantic Embedding under varying click counts on PartObjaverse.The comparison is summarized in Table 3.
- Point Embedding Mechanism: Explicit Coordinate Encoding outperforms Label-based Semantic Embedding as interaction counts increase, particularly during later stages.Explicit encoding injects spatial coordinates alongside semantic vectors, whereas the label-based strategy relies on sparse-backbone locality.
- Sampling Steps: SegviGen achieves strong segmentation performance with one sampling step, while improvements begin to saturate beyond 8 steps.The flow model’s trajectory property supports good performance at low step counts.
- Sampling Steps: The authors adopt 12 sampling steps because 25 steps offer only marginal improvements while nearly doubling inference latency relative to 12 steps.This choice balances segmentation quality and computational efficiency.
4.4 Failure Cases and Limitations
SegviGen’s main limitations are semantic ambiguity in part decomposition and reduced precision when 2D guidance specifies extremely fine structures.
- Failure Cases and Limitations: Semantic ambiguity can cause SegviGen to produce more or fewer parts than expected in interactive and full segmentation.The paper attributes this to multiple valid decompositions for the same object.
- Failure Cases and Limitations: When 2D guidance becomes overly fine-grained, the resulting 3D segmentation may have reduced boundary precision and smoothness.The model cannot always reproduce highly detailed decompositions specified by the input map.
5 CONCLUSION
SegviGen repurposes pretrained 3D generative priors for accurate, globally coherent 3D part segmentation. Experiments show consistent improvements over prior methods and highlight the data efficiency of this approach.
- SegviGen repurposes pretrained 3D generative models to produce accurate and globally coherent 3D part segmentations.
- Experiments on interactive and full segmentation benchmarks show consistent improvements over prior methods.
- 3D generative priors improve the data efficiency of 3D part segmentation.