SATR: Zero-Shot Semantic Segmentation of 3D Shapes

1. Introduction

• Earlier 3D Highlighter uses CLIP-guided optimization but struggles with fine-grained segmentation, varies with initialization, and can take about 5–10 minutes per part query on a recent GPU.The paper motivates exploring off-the-shelf 2D models amid limited 3D segmentation data and higher computational costs for 3D architectures.Source: §1 · pp. 1–2
• SATR estimates 3D segmentation from multi-view 2D bounding boxes by using the underlying surface's topological properties.The paper reports that zero-shot 2D object detectors perform substantially better for this task than 2D segmentation networks.Source: §1 · p. 2
• SATR uses GLIP multi-view bounding boxes to infer 3D segmentation, refining coarse guidance with Gaussian geodesic reweighting and visibility smoothing.The authors say the combined techniques yield state-of-the-art results, especially for fine-grained queries.Source: §1 · p. 2
• 82.46%, 46.01%, and 31.9% average mIoU are reported for SATR on the coarse and fine-grained FAUST benchmarks and ShapeNetPart, respectively; fine-grained FAUST performance exceeds a baseline by at least 4% mIoU on average.The authors report state-of-the-art performance and plan to release the code and benchmarks.Source: §1 · p. 2
• The paper proposes a quantitative FAUST benchmark with coarse and fine-grained categories, manually segmenting 17 regions on one scan and propagating labels through shape correspondences.It also evaluates on ShapeNetPart.Source: §1 · p. 2

2. Related Work

• Zero-shot 2D detection evolved from word-embedding methods toward models coupling text encoders and multimodal representations with object detectors, with recent methods targeting unseen-category generalization.The passage names examples including [7], [38] [82], and [55] [99] [87] [102] [88] [78].Source: §2 · pp. 2–3
• Zero-shot 2D segmentation adapts multimodal semantic representations to segmentation, but fine-grained categories remain challenging; the paper develops segmentation from estimated bounding boxes.The passage notes annotation cost as a limiting factor and cites Figure 3 for fine-grained difficulty.Source: §2 · p. 3, Figure 3 · p. 3
• 3D semantic-field methods reconstruct annotations from multi-view renderings and can distill zero-shot 2D segmentation into NeRFs for open-vocabulary volumetric-scene segmentation.The paper focuses instead on shape segmentation and reports state-of-the-art results using a 2D object detector.Source: §2 · p. 3
• 3D Highlighter is described as the only prior zero-shot 3D mesh segmentation work, while concurrent PartSLIP uses GLIP bounding boxes to score point-cloud clusters and targets point clouds rather than meshes.3D Highlighter optimizes a CLIP-guided probability field and is reported to struggle with fine-grained predictions and initialization sensitivity.Source: §2 · p. 3

3.1. Topology-Agnostic Mesh Segmentation

• The topology-agnostic SATR-F baseline scores visible faces that fall inside detector boxes using box confidence and projected face area, then aggregates scores across views.The paper identifies segmentation leaking—unrelated surface regions included in boxes across views—as the baseline’s limitation.Source: §3.1 · pp. 4–5

3.2. Gaussian Geodesic Reweighting

• Gaussian geodesic reweighting uses distances along the mesh surface from a box’s central face to reduce leakage into nearby but semantically unrelated regions.The method fits a Gaussian to those distances to make reweighting more robust to central-face selection errors and difficult-to-tune decay rates; it performs better without normalization, but can underweight central faces.Source: §3.2 · p. 5

3.3. Visibility Smoothing

• Visibility smoothing uses q-rank mesh neighborhoods (q = 5 in the experiments) to repair central-face weights based on neighborhood visibility.It acts like a graph convolution, and the paper reports that it improves scores.Source: §3.3 · pp. 5–6

4.1. Experimental Setup

• Evaluation uses all 100 FAUST scans with manually transferred coarse and fine annotations, and 2,460 labeled ShapeNetPart shapes remeshed for inference; performance is measured with semantic segmentation mIoU.ShapeNetPart has 16 object categories and 50 annotated parts; its predicted face labels are transferred back to the original shapes after evaluation.Source: §4.1 · pp. 6–7

4.3.1 FAUST Benchmark

• 82.46% overall average mIoU on coarse-grained FAUST parts, and a 42.12% margin over 3DHighlighter on fine-grained segmentation, are reported for SATR.The paper also reports that SATR outperforms 3DHighlighter and CLIPSeg on all four coarse parts: leg, arm, head, and torso.Source: Table 2 · p. 7, Table 1 · p. 7, §4.3.1 · p. 7

4.2. Implementation Details

• Experiments use one Nvidia V100 GPU and Kaolin in PyTorch for rendering, with meshes centered and normalized inside a unit sphere and rendered at 1024 × 1024 on black backgrounds.The paper states that the same views are used across GLIP-based models.Source: §4.2 · p. 7, experiments · p. 7

4.3.2 ShapeNetPart Dataset

• SATR consistently outperforms 3DHighlighter in every ShapeNetPart shape category, with results suggesting applicability beyond human shapes.The authors note that 3DHighlighter depends on a suitable random initialization, which they say occurs infrequently.Source: §4.3.2 · p. 7, Table 3 · p. 8

4.4. Ablation Studies

• SATR remains a strong performer against DINO-SAM and GLIP-SAM, especially on the FAUST fine-grained benchmark.The comparison uses recent SAM-based segmentation methods combined with GLIP and Grounding-DINO detectors.Source: Table 4 · p. 9, §4.4 · p. 9
• Using both Gaussian geodesic re-weighting and visibility smoothing gives the best performance on FAUST coarse and fine-grained benchmarks; each component alone also beats SATR-Baseline.The results suggest the two components work in a complementary fashion.Source: §4.4 · p. 8, Table 5 · p. 9, Table 7 · p. 9
• Gaussian geodesic re-weighting outperforms other re-weighting methods, especially on the fine-grained benchmark, and is robust when the capital face is miscalculated.The alternatives include linear distance weighting and softmax normalization of geodesic distances.Source: Table 6 · p. 9, §4.4 · p. 9
• Segmentation leaking occurs when semantically unrelated surface parts are attributed to a text prompt because they often fall inside predicted bounding boxes across multiple views.Source: Figure 8 · p. 9

• SATR extracts 3D segmentation maps from multi-view bounding-box predictions using surface topology.The paper reports large improvements over previous and selected concurrent work and proposes the first benchmarks for this area.Source: §5 · p. 9

• Random view sampling has no formal guarantee of covering every triangle.The authors chose it mainly for compatibility with previous work; greedy selection or fixed uniformly distributed viewpoints are alternatives they describe.Source: A. Limitations · p. 15
• The authors could not test other types of large language models because their latest versions were not publicly available.Source: A. Limitations · p. 15
• GLIP can mistake semantically different but visually similar parts, such as an astronaut’s front and backpack.Source: A. Limitations · p. 15

• Algorithm 1 presents pseudocode for the proposed SATR method.Source: B. SATR Pseudocode · p. 15

• More rendered views generally improve mIoU on both coarse and fine-grained FAUST benchmarks.Source: §C.1 · p. 15
• Increasing the view count also increases computation time, with diminishing returns from adding many views.Source: §C.1 · p. 15

C.2. Camera View-port Sampling Approach

• On both coarse and fine-grained FAUST benchmarks, sampling camera view-ports from a normal distribution (µ = 3.14, σ = 4) yields better mIoU performance than uniform viewpoint sampling.The reported metric is mIoU.Source: §C.2 · p. 15
• The authors attribute the result to greater control over elevation and azimuth, which can yield better mesh coverage and avoid heavily occluded views.Source: §C.2 · p. 15
• More complex view selection is left as future work.Source: §C.2 · p. 15

C.3. Coloring The Input Mesh

• The color ablation compares four input-mesh colors: gray, red, blue, and natural skin color.Source: §C.3 · p. 15
• SATR takes a mesh surface, textual prompts for semantic regions or classes, and a number of rendered views as inputs.Its pseudocode also specifies a q-ring neighborhood parameter.Source: §C.3 · p. 15
• Gray input meshes give the best performance among gray, red, blue, and natural skin color.Table 2 reports results on the FAUST coarse and fine-grained benchmarks.Source: §C.3 · p. 15, Table 2 · p. 16

C.4. Summation of Reweighting Factors

• The ablation replaces multiplication of the Gaussian-geodesic and visibility-smoothing factors with addition.Source: §C.4 · p. 16
• Addition of the reweighting factors slightly increases coarse-benchmark performance but performs significantly worse than multiplication on the fine-grained benchmark.The comparison reports mIoU on both FAUST benchmarks in Table 4.Source: §C.4 · p. 16, Table 4 · p. 16
• The comparison evaluates different ways of combining reweighting factors on the FAUST coarse and fine-grained benchmarks.Source: §C.4 · p. 16, Table 4 · p. 16