MeshSegmenter: Zero-Shot Mesh Semantic Segmentation via Texture Synthesis

1 Introduction

• Limited accurately annotated 3D data makes training-based methods struggle to generalize to unseen examples, while local mesh regions can be ambiguous across views.The introduction gives black clothing mistaken for hair as an example.Source: §1 · p. 2
• The framework uses object and region descriptions to guide fine-grained segmentation and demonstrates applications to mesh segmentation and editing.The authors report experimental evaluation across various 3D zero-shot segmentation tasks.Source: §1 · pp. 2–3
• MeshSegmenter combines 2D zero-shot segmentation of rendered meshes with multi-view aggregation to produce coherent 3D segmentation.The approach aims to use 2D models as a more efficient alternative to handling larger 3D datasets directly.Source: §1 · pp. 2–3
• Generated textures supplement geometry and help segment regions with weak geometric cues, such as a car door integrated into the body.The framework processes both textured and untextured meshes.Source: §1 · p. 3
• Face confidence revoting combines detections and segmentations across views to improve 3D consistency and correct perspective-specific errors.The paper describes the resulting segmentation as stable and reliable.Source: §1 · p. 3

Zero-shot 2D Segmentation

• Zero-shot 2D segmentation work either aligns image and semantic features for unseen categories or adapts large multimodal semantic spaces to segmentation.The related work describes recent multimodal approaches as improving generalization to unseen categories.Source: §2 · p. 4

Zero-shot 3D Segmentation

• Zero-shot 3D segmentation includes point-cloud methods and approaches that integrate 2D image-recognition models into 3D shape analysis.NeRF-related work constructs semantic fields and enforces consistency between rendered multi-view labels and ground-truth or estimated labels.Source: §2 · pp. 4–5

Single Query Segmentation

• 3DHighlighter [9] predicts mesh-vertex probabilities and uses rendered views to align them with a text prompt in CLIP space.Source: §2 · p. 5

Multiple Queries Segmentation

• SATR [2] aggregates multi-view segmentations by selecting the highest-confidence prompt per facet, while ZSC [1] uses ChatGPT-generated prompts and shape correspondence.The related work says ZSC [1] falters when single-query segmentation is required.Source: §2 · p. 5

3.1 Overview

• The overview comprises text-guided texture synthesis, 2D zero-shot segmentation, and face confidence revoting for mesh segmentation.It uses both textured and untextured meshes to provide texture and geometric information.Source: §3.1 · p. 6

3.2 Text-Guided Texture Synthesis

• Stable Diffusion generates mesh textures from the object-class description, adding color cues for regions difficult to localize from shape alone.The text used to prompt texture generation differs from the grounding description used for segmentation.Source: §3.2 · p. 6

3.3 2D zero-shot semantic segmentation

• Rendered views are processed by GroundingDINO [24] for text-guided boxes and confidence scores, then SAM [18] uses the boxes to produce masks.The camera trajectory samples azimuths at two polar angles to balance segmentation quality with viewpoint coverage.Source: §3.3 · pp. 7–8

3.4 Face Confidence Revoting (FCR)

• Revoting is designed to prevent erroneous local detections or segmentations from contaminating the final result by using evidence across viewpoints.The method is motivated by errors that can occur in individual views.Source: §3.4 · p. 9
• Face Confidence Revoting assigns positive confidence to visible masked faces and negative confidence to visible unmasked faces, then sums scores across views.Scores from textured and untextured meshes are averaged, thresholded, and smoothed with neighboring faces.Source: §3.4 · pp. 9–10

4.1 Implementation Details

• The experiments use one Nvidia A40 GPU per object-description and grounding-text case, with Kaolin rendering and GroundingDINO [24] and SAM [18] as the 2D models.The comparison includes 3DHighlighter [9], SATR [2], and official results for non-open-source ZSC [1].Source: §4.1 · p. 10

4.2 Zero-shot Mesh Semantic Segmentation

• MeshSegmenter outperforms 3DHighlighter [9] and SATR [2] on the ShapeNetPart evaluation and in single- and multiple-query user-study ratings.The evaluation randomly sampled 20 objects per category because multi-view texture generation is computationally resource-intensive; the user study used 20 meshes and 30 respondents.Source: §4.2 · p. 10, Table 1 · p. 11, Table 2 · p. 11

4.3 Application of MeshSegmenter

• For controlled stylization, MeshSegmenter localizes the prompted hair region for editing, while TEXTure [37] also colors an unspecified ribbon region red.Source: §4.3 · p. 11, Figure 7 · p. 11
• Text-specified mesh regions can be selected, stretched, warped, or deleted, and mesh segmentation can be transferred to point clouds after optimizing them into mesh structures.The point-cloud extension depends on obtaining a mapping between rendered views and the 3D representation.Source: §4.3 · p. 12, Figure 8 · p. 12, Figure 9 · p. 12

4.4 Ablation Studies

• Texture synthesis helps reduce the domain gap for 2D models and supports segmentation of regions with weak geometric information.The paper illustrates texture synthesis and face-confidence revoting ablations in Figs. 10 and 11.Source: §4.4 · p. 13, Figure 10 · p. 13, Figure 11 · p. 13
• The method reports effective segmentation with inaccurate object descriptions, while accurate class descriptions can produce more appropriate generated textures.Source: §4.4 · p. 13

• MeshSegmenter segments objects accurately with correct descriptions and also handles homogeneous descriptions such as “animals” and “dogs.”Source: Figure 12 · p. 14
• Pretrained 2D models are not designed for local-level detection and segmentation, so they can incorrectly detect entire objects as target regions.Source: §5 · p. 14
• Filtering whole-object detections and face-confidence revoting use correct detections from adjacent viewpoints to correct erroneous detections.Source: §5 · p. 14
• No rendering strategy guarantees visibility of every face, so the method uniformly samples viewpoints in fixed rotation to maximize the likelihood of seeing more faces.Source: §5 · p. 14

• MeshSegmenter extends 2D detection and segmentation models to segment diverse 3D meshes by text.Source: §6 · p. 14
• Generating textures based on object descriptions enhances the accuracy of 2D segmentation models.Source: §6 · p. 14
• A multi-view revoting module integrates views to ensure 3D consistency in segmentation results.Source: §6 · p. 14
• The framework provides stable and reliable segmentation results and has potential applications in computer graphics and computer vision.Source: §6 · p. 14