PaperlayerView exact revision on arXiv

Source-linked AI summary

iSeg: Interactive 3D Segmentation via Interactive Attention

Itai Lang, Fei Xu, Dale Decatur, Sudarshan Babu, Rana Hanocka

arXiv:2404.03219v2cs.CVcs.GR

TL;DR

Text may not accurately describe all fine-grained segmentations, such as an octopus’s fourth leg or a particular point on a shape. iSeg predicts mesh regions from positive and negative surface clicks, using interactive attention to support varied click numbers and types in one model. iSeg infers segments for mesh vertices not used in training and computes complete 3D segments for clicks and regions occluded from one another.

  • Problem

    Text may not accurately describe all fine-grained segmentations, such as an octopus’s fourth leg or a particular point on a shape.

    Source: §1 · p. 1

  • Method

    iSeg predicts mesh regions from positive and negative surface clicks, using interactive attention to support varied click numbers and types in one model.

    Source: §1 · p. 2

  • Results

    iSeg infers segments for mesh vertices not used in training and computes complete 3D segments for clicks and regions occluded from one another.

    Source: §1 · p. 2

  • Takeaways & Limitations

    iSeg enables users to select meaningful regions directly across occluded surfaces in 3D.

    Source: §5 · p. 8

    iSeg may not follow mesh symmetry exactly when corresponding regions were not trained the same way.

    Source: §4.4 · p. 7

Abstract

from arXiv · show

We present iSeg, a new interactive technique for segmenting 3D shapes. Previous works have focused mainly on leveraging pre-trained 2D foundation models for 3D segmentation based on text. However, text may be insufficient for accurately describing fine-grained spatial segmentations. Moreover, achieving a consistent 3D segmentation using a 2D model is highly challenging, since occluded areas of the same semantic region may not be visible together from any 2D view. Thus, we design a segmentation method conditioned on fine user clicks, which operates entirely in 3D. Our system accepts user clicks directly on the shape's surface, indicating the inclusion or exclusion of regions from the desired shape partition. To accommodate various click settings, we propose a novel interactive attention module capable of processing different numbers and types of clicks, enabling the training of a single unified interactive segmentation model. We apply iSeg to a myriad of shapes from different domains, demonstrating its versatility and faithfulness to the user's specifications. Our project page is at https://threedle.github.io/iSeg/.

1 INTRODUCTION

iSeg addresses fine-grained 3D segmentation by letting users specify regions with surface clicks, including positive and negative clicks. It combines a 3D segmentation pipeline with interactive attention that supports varied click configurations.

1 INTRODUCTION

  • Interactive 3D segmentation is presented as a fundamental graphics problem with applications in model refinement, simulation, and analysis.The introduction identifies precise selection as useful in computer-aided design, 3D modeling, engineering, architecture, and medicine.Source: §1 · p. 1
  • Text may not accurately specify fine-grained regions such as one particular limb or a region corresponding to a point on a shape.Existing text-driven approaches use pretrained 2D foundation models for 3D segmentation.Source: §1 · p. 1
  • iSeg predicts customized mesh regions from surface clicks, with positive clicks adding regions and negative clicks excluding them.The method is designed to operate directly in 3D, while projected clicks and segmentations provide 2D foundation-model supervision during training.Source: §1 · pp. 1–2
  • Interactive attention consolidates variable numbers and types of clicks into a fixed-size representation for one flexible segmentation model.It learns representations of positive and negative clicks and their interaction with mesh points.Source: §1 · p. 2
  • The method is demonstrated on varied shapes and can infer segments for vertices not used in training, including regions occluded from one another.The paper reports training on only a small fraction of mesh vertices.Source: §1 · p. 2
  • iSeg distills inconsistent 2D foundation-model features into a coherent mesh feature field and decodes that field with user inputs.The figure describes supervision from rendered views while the 3D segmentation remains view-consistent by construction.Source: §1 · p. 2, Figure 2 · p. 2

2 RELATED WORK

Prior approaches to 3D segmentation can depend on fixed labels, training domains, or 2D views. iSeg instead uses an explicit mesh representation, a dedicated 3D decoder, and direct 3D interaction.

2.1 Non-Interactive 3D Segmentation

  • Dataset-based segmentation can be restricted to its training domain and predefined labels, while iSeg is optimized per shape without segment labels and updated through clicks.The related-work discussion also notes that unsupervised methods remain confined to learned parts that are not easy to alter.Source: §2.1 · p. 2

2.2 Lifting 2D Foundation Models to 3D

  • iSeg lifts 2D foundation-model features onto explicit meshes and learns a dedicated 3D decoder rather than using NeRF or prompt-feature correlation.Its decoder is trained using 2D segmentation masks from rendered images and projected clicks, and is described as view-consistent by construction.Source: §2.2 · p. 2, Figure 3 · p. 3

2.3 Interactive 3D Segmentation

  • 2D projection-based interaction can require guidance across views, while iSeg applies the model and clicks directly in 3D to segment occluded regions together.The cited prior approaches include projecting and fusing 2D segmentations and lifting SAM features to NeRF.Source: §2.3 · p. 3

3 METHOD

iSeg encodes mesh vertices into a 3D feature field and conditions them on positive and negative user clicks to predict mesh regions. It uses 2D foundation-model supervision during training while applying segmentation directly on the 3D shape.

3.1 Mesh Feature Field

  • The encoder distills pretrained 2D segmentation features across views into a mesh-wide, view-consistent feature field optimized independently of user clicks.Random-view rendered mesh features are trained to match the foundation model's reference features.Source: §3.1 · pp. 3–4

3.3 Segmentation Prediction

  • A single back-side click segments occluded chair-backrest surfaces, while opposite-side clicks recover the whole part despite unavailable joint 2D views.The decoder produces view-consistent segmentations from inconsistent 2D supervision, including regions not wholly visible in one projection.Source: Figure 5 · p. 4, §3.3 · p. 5

3.2 Interactive Attention Layer

  • Interactive attention attends mesh vertices to positive and negative click features, consolidating variable-size guidance into fixed-length features that are click-order invariant.Its output is also permutation equivariant to vertex order.Source: §3.2 · p. 4, Figure 4 · p. 4

3.3 Segmentation Prediction

  • The decoder combines inherent and click-conditioned vertex features to predict per-vertex probabilities for the desired region.The predicted segmentation is represented as a probability for each mesh vertex.Source: §3.3 · pp. 4–5
  • Training projects mesh probabilities and clicks into randomized views, using prompted 2D-model masks as targets and requiring second-click masks to expand or shrink.Simulated clicks and masks provide diverse training data.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments assess iSeg’s generality and fidelity across diverse meshes, then examine feature transfer and generalization to unseen vertices, views, and click counts. Results show coherent 3D segmentations that adhere to user clicks, with a noted limitation in matching mesh symmetry.

4.1 Generality of iSeg

  • Across diverse domains and geometries, iSeg segments parts ranging from thin spokes and sharp edges to flat, curved, small, and large regions.The tested meshes include humanoids, animals, musical instruments, household objects, and other categories.Source: §4 · p. 5, §4.1 · p. 6

4.2 Fidelity of iSeg

  • iSeg produces coherent 3D regions resembling ground-truth parts and achieves much higher ACC and IoU than InterObject3D and the SAM baseline on PartNet.The visual comparison shows other methods producing partial segmentations, including SAM’s omission of occluded regions.Source: §4.2 · p. 6, Table 1 · p. 6, Figure 7 · p. 7
  • In the perceptual study, participants rated iSeg much more effective than competing methods, while additional clicks let users extend or refine regions.Positive clicks add surrounding regions; negative clicks remove unwanted areas.Source: §4.2 · pp. 6–7, Table 2 · p. 6

4.3 Generic Feature Information

  • The click-independent feature field transfers semantic features across shapes, supporting cross-domain analogies such as a human’s belly corresponding to an airplane’s belly.The feature from a point on one mesh is used with another shape’s decoder.Source: §4.3 · p. 7, Figure 8 · p. 7

4.4 Generalization Capabilities

  • Training on 3% of mesh vertices still supports predictions at unseen vertices, clicks from unseen views, and segmentations with more than two clicks.The click-count generalization is attributed to the interactive attention mechanism.Source: §4.4 · p. 7
  • iSeg may not reproduce mesh symmetry exactly when corresponding regions on opposite sides were not trained alike.The paper illustrates this with different segmentations on the sides of a goat’s head.Source: §4.4 · p. 7

5 CONCLUSION

iSeg lifts features from a pretrained 2D segmentation model to a 3D mesh and uses interactive attention to handle varied positive and negative clicks. The paper presents cross-domain segmentation as a potential further application.

5 CONCLUSION

  • iSeg generates fine-grained, user-tailored segmentations of 3D meshes using features lifted from a pretrained 2D segmentation model.Its mesh feature field is presented as general and potentially useful for cross-segmentation across shape categories.Source: §5 · p. 8
  • Interactive attention learns a unified representation for varied positive and negative clicks, while 3D-consistent predictions support selecting regions across occluded surfaces.The paper also demonstrates potential cross-domain segmentation and suggests exploring applications such as key-point correspondence and texture transfer.Source: §5 · p. 8
  • The authors identify applications beyond segmentation, including key-point correspondence and texture transfer, as directions for future exploration.Cross-domain segmentation is presented as a demonstrated potential application.Source: §5 · p. 8

Supplementary Material

The supplementary material organizes additional results, evaluation and implementation details, and an ablation test across appendices A through E.

Supplementary Material

  • Appendix A contains additional results and experiments conducted with iSeg.The passage places this material in the supplementary appendices.Source: Supplementary Material · p. 12
  • Appendices B and C detail the PartNet quantitative evaluation and the perceptual user study, respectively.Source: Supplementary Material · p. 12
  • Appendix D covers an ablation test, while Appendix E details encoder and decoder implementation and comparison settings.Source: Supplementary Material · p. 12

A ADDITIONAL RESULTS

Additional results show that iSeg handles occluded regions consistently in 3D, follows fine-grained clicks, and supports stable, controllable segmentation across varied shapes. Its Euclidean-space processing can separate nearby parts in practice, though that ability may depend on the 2D supervision distinguishing them.

Fidelity

  • iSeg follows fine-grained clicks more closely than COSEG parts, selecting a horse’s head without its neck and one alien antenna rather than all three [van Kaick et al. 2011].The comparison illustrates that iSeg is not restricted to dataset-defined parts.Source: A ADDITIONAL RESULTS · p. 12, Figure 19 · p. 13

View generalization

  • iSeg recovers complete 3D regions when clicks and parts are occluded from one another in 2D views.Examples include the guitar body and bridge, whose clicks cannot be seen together from any 2D view.Source: A ADDITIONAL RESULTS · p. 12, Figure 17 · p. 12

3D consistency

  • iSeg predicts segmentation on mesh vertices, making its results 3D-consistent across views, unlike SAM’s independently generated masks.The same 3D segmentation underlies every 2D projection.Source: A ADDITIONAL RESULTS · p. 12, Figure 18 · p. 12

Full-shape segmentation and local editing

  • Repeated click sessions can parse a shape into localized, mostly disjoint parts without part-label supervision, and selected regions can support local shape edits.Examples include removing a chair bar, shortening a guitar neck, and enlarging a small region of a candle flame.Source: A ADDITIONAL RESULTS · pp. 12–13

Mesh processing in Euclidean space

  • Although iSeg processes vertex coordinates without explicitly modeling topology, it separates Euclidean-nearby parts in practice; this may depend on the 2D supervision distinguishing them.The paper cautions that iSeg may not distinguish regions when the 2D supervision model does not disambiguate them successfully.Source: A ADDITIONAL RESULTS · p. 13

Stability

  • 0.8 average neighboring-click IoU for iSeg compares with 0.7 for the SAM baseline.The evaluation used 7 meshes and 100 random clicks per mesh.Source: A ADDITIONAL RESULTS · p. 13

Granularity

  • Positive and negative clicks let users choose a whole trailer box or just its side, and adjust leg segmentation granularity by placing a negative click at different locations.The leg example varies whether the foot, lower shin, or upper calf is excluded.Source: A ADDITIONAL RESULTS · p. 13

B QUANTITATIVE EVALUATION

The evaluation compares click-based predictions with PartNet ground-truth parts on test shapes, using vertices not used during training. Predictions are binarized with Otsu thresholding, which is also applied to the baselines.

B QUANTITATIVE EVALUATION

  • Evaluation compares predictions with PartNet ground-truth labels on test shapes, using five vertices per part that were not used during training.The evaluation used 77 Knife test shapes and sampled 93 additional meshes from other categories.Source: B QUANTITATIVE EVALUATION · pp. 13–14

B.1 Segmentaiton Binarization

  • Otsu thresholding converts iSeg’s soft per-vertex scores into binary predictions, and the same binarization is applied to baselines.The authors chose this approach to minimize postprocessing and reflect method performance faithfully.Source: §B.1 · p. 14

B.2 Evaluaiton Metrics

The evaluation converts vertex predictions to face predictions and measures accuracy and IoU against the clicked part. Among the tested SAM aggregation baselines, visibility averaging performs best.

B.2 Evaluaiton Metrics

  • PartNet vertex predictions are converted to face labels, then evaluated with accuracy and IoU against the clicked part as the positive region.A face is labeled segmented only when all its vertices are segmented.Source: §B.2 · p. 14

B.3 SAM Baselines

  • Visibility Average aggregation of SAM predictions performs better than the tested union and averaging alternatives.Table 3 reports quantitative PartNet results for the SAM baselines.Source: §B.3 · p. 14, Table 3 · p. 14

C PERCEPTUAL USER STUDY

The perceptual study rates how effectively methods select a clicked region, including when the click is occluded from the viewing angle. Its reported results favor iSeg over partially effective SAM segmentations.

Granularity

  • iSeg offers control over segmentation granularity according to where the user clicks.Figure 22 illustrates this property.Source: Figure 22 · p. 14

Study scoring

  • Study examples show a clicked cubic alien region from different views and rate outcomes as completely effective, partially effective, or ineffective.The rubric assigns 5 to a complete selection and 1 when no region is selected.Source: C PERCEPTUAL USER STUDY · pp. 14–15

Study results

  • Study results describe iSeg segmentations as highly effective compared with partially effective SAM results; in the camel example, iSeg selects the full hump when the click is occluded, while SAM selects it partially.The paper connects the comparison to the difficulty of segmenting occluded 3D parts using SAM’s 2D views.Source: C PERCEPTUAL USER STUDY · p. 15

D ABLATION TEST

The ablation jointly optimized the encoder and decoder, including interactive attention, using both losses. Although it matched the original model on training vertices, it overfit and generalized less well to test vertices.

D ABLATION TEST

  • The ablation jointly optimized encoder and decoder parameters, including interactive attention, against both encoder and decoder losses.The encoder loss measures feature-field alignment with SAM embeddings; the decoder loss measures agreement with SAM segmentation masks.Source: D ABLATION TEST · p. 15
  • Joint encoder–decoder training matched the original model on training vertices but overfit and performed less well on test vertices, where iSeg generalized better after the same epochs.The ablation optimized both networks, including interactive attention, simultaneously.Source: D ABLATION TEST · p. 15

E IMPLEMENTATION DETAILS

iSeg encodes mesh vertices and combines per-vertex features with click-conditioned interactive attention to predict segmentation. The implementation uses random rendered views for training and compares iSeg with InterObject3D and other interactive techniques.

Encoder

  • The six-layer encoder maps positional encodings of vertex coordinates to 256-dimensional per-vertex features.It uses 256 neurons per layer, with ReLU and layer normalization except at the final layer, which uses hyperbolic tangent activation without normalization.Source: E IMPLEMENTATION DETAILS · p. 15

Training views

  • Encoder and decoder training used a wide range of random views, with 224 × 224 rendered images matching the SAM ViT-H resolution.The azimuth angle ranges over [0, 2π).Source: E IMPLEMENTATION DETAILS · p. 15

Decoder

  • The decoder combines a 256-dimensional interactive-attention output with each encoded vertex feature, then predicts two-class outputs with a 16-layer MLP.The MLP has a 512-unit first layer, 14 layers of 256 units, and a two-unit final layer; Softmax follows the final layer.Source: E IMPLEMENTATION DETAILS · p. 15

Compared method

  • For comparison with InterObject3D [Kontogianni et al. 2023], the experiments used its public pretrained model and a point-click volume length of 0.05.The authors report that training code was unavailable and that they followed the paper’s instructions to use channels for point clicks.Source: E IMPLEMENTATION DETAILS · p. 15

Additional comparisons

  • In additional PartNet comparisons, iSeg produces segmentations closer to ground-truth part annotations than the compared methods.The figure shows different views of segmentation results for the interactive techniques.Source: Figure 23 · p. 16

Perceptual study and stability

  • The perceptual-study example reports that participants considered iSeg highly effective, while other techniques were perceived as only partially effective.The caption attributes the difference to iSeg selecting complete 3D-consistent regions corresponding to clicked points; a separate figure shows similar segmented regions across varying positive- and negative-click locations.Source: Figure 24 · p. 17, Figure 25 · p. 18

Related papers

Loading 2404.03219v2…