PaperlayerView exact revision on arXiv

Source-linked AI summary

Segment Any 3D Gaussians

Jiazhong Cen, Jiemin Fang, Chen Yang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, Qi Tian

arXiv:2312.00860v3cs.CV

TL;DR

Promptable 3D segmentation is limited by scarce annotated 3D data and the challenge of representing multi-granularity targets efficiently. SAGA integrates SAM-derived, scale-gated affinity features into 3D-GS, achieving real-time segmentation with quality comparable to state-of-the-art methods across reported evaluations.

  • Problem

    3D promptable segmentation remains relatively unexplored because 3D data are scarce and annotation is costly, while multi-granularity ambiguity complicates segmentation.

  • Method

    SAGA distills SAM’s multi-granularity segmentation capability into affinity features attached to 3D Gaussians and adjusts feature channels with a soft 3D-scale gate.

  • Results

    SAGA achieves real-time fine-grained 3D segmentation and outperforms previous approaches by +0.4 mIoU over SA3D-GS and +0.9 mIoU over OmniSeg3D on NVOS.

  • Takeaways & Limitations

    SAGA provides a simple, efficient way to integrate promptable multi-granularity segmentation into 3D-GS without an additional bulky segmentation module.

  • Takeaways & Limitations

    SAGA has difficulty segmenting objects absent from the SAM masks used for training, especially small targets.

Abstract

from arXiv · show

This paper presents SAGA (Segment Any 3D GAussians), a highly efficient 3D promptable segmentation method based on 3D Gaussian Splatting (3D-GS). Given 2D visual prompts as input, SAGA can segment the corresponding 3D target represented by 3D Gaussians within 4 ms. This is achieved by attaching an scale-gated affinity feature to each 3D Gaussian to endow it a new property towards multi-granularity segmentation. Specifically, a scale-aware contrastive training strategy is proposed for the scale-gated affinity feature learning. It 1) distills the segmentation capability of the Segment Anything Model (SAM) from 2D masks into the affinity features and 2) employs a soft scale gate mechanism to deal with multi-granularity ambiguity in 3D segmentation through adjusting the magnitude of each feature channel according to a specified 3D physical scale. Evaluations demonstrate that SAGA achieves real-time multi-granularity segmentation with quality comparable to state-of-the-art methods. As one of the first methods addressing promptable segmentation in 3D-GS, the simplicity and effectiveness of SAGA pave the way for future advancements in this field. Our code will be released.

1 Introduction

SAGA targets the underexplored problem of promptable 3D segmentation in 3D-GS, using its explicit Gaussian structure to integrate segmentation without a bulky additional module. It addresses efficient Gaussian-level segmentation and ambiguity across segmentation granularities.

  • 3D promptable segmentation remains relatively unexplored because 3D data are scarce and annotation is costly.
  • 3D-GS offers an explicit structure that can carry segmentation as an intrinsic attribute without an additional bulky segmentation module.
  • SAGA takes 2D visual prompts and outputs the corresponding 3D target represented by 3D Gaussians.
  • SAGA must preserve 3D-GS efficiency while giving each Gaussian 3D segmentation capability and handling cases where one Gaussian belongs to targets at different granularities.
  • SAGA attaches affinity features to Gaussians, while a soft scale gate adjusts feature-channel magnitudes according to a specified 3D physical scale.

2 Related Work

Prior work extends 2D segmentation and feature similarities into radiance fields, while GARField addresses multi-granularity ambiguity but incurs repeated segmentation queries at different scales. SAGA is positioned as a more efficient 3D-GS alternative.

  • Radiance-field methods have propagated semantic or visual features from images into 3D representations for segmentation.
  • Several approaches distill 2D feature similarities into 3D features, while other methods address instance or semantic segmentation in radiance fields.
  • GARField uses 3D physical scale to address multi-granularity ambiguity in 3D segmentation.
  • GARField requires repeated queries through implicit feature fields for segmentation at different scales, reducing efficiency.

3 Method

SAGA attaches scale-gated affinity features to 3D Gaussians and trains them by distilling multi-granularity SAM masks through scale-aware contrastive learning. During inference, prompt-derived query features are compared with Gaussian features for 3D segmentation.

  • 3.1 Preliminary: 3D-GS represents a scene with colored Gaussians whose means encode positions and covariances encode scale, enabling efficient differentiable rasterization.
  • 3.2 Overall Pipeline: SAGA attaches an affinity feature to every Gaussian and uses a soft scale gate to project features into scale-specific sub-spaces.
  • 3.2 Overall Pipeline: SAGA extracts multi-granularity masks with SAM, computes each mask’s 3D physical scale from predicted depth and camera pose, and distills them into scale-gated features.
  • 3.2 Overall Pipeline: At inference, 2D visual prompts with scales become 3D scale-gated query features whose similarities with Gaussian features define the 3D target.
  • 3.3 Gaussian Affinity Feature: A scale gate maps a scalar scale s to a soft D-dimensional vector using one linear layer and a sigmoid, adjusting each affinity-feature channel’s magnitude.
  • 3.3 Gaussian Affinity Feature: The scale gate can be applied after rendering during training or directly to 3D Gaussian features during inference, with negligible scale-change overhead.
  • 3.4 Scale-Aware Contrastive Learning: SAGA’s scale-aware contrastive learning distills pixel-wise correlations from SAM masks into Gaussian affinity features through differentiable rasterization.
  • 3.4 Scale-Aware Contrastive Learning: Scale-aware pixel identity vectors encode which masks contain each pixel at a given scale, encouraging shared-mask pixels to have similar features.

4 Experiments

Experiments evaluate SAGA on promptable and open-vocabulary segmentation, efficiency, qualitative detail preservation, and feature regularization. SAGA generally achieves strong segmentation results, fast inference, fine-grained structure recovery, and benefits from smoothing and norm regularization.

  • Experimental Setup: The experiments use NVOS and SPIn-NeRF for promptable segmentation and 3D-OVS for open-vocabulary segmentation.Qualitative analyses additionally use datasets including LLFF and MIP-360.
  • Quantitative Results: +0.4 mIoU over SA3D-GS and +0.9 mIoU over OmniSeg3D on NVOS.These results are reported for SAGA on the NVOS dataset.
  • Quantitative Results: SAGA demonstrates superior results across all scenes in the 3D-OVS open-vocabulary segmentation dataset.
  • Time Consumption Analysis: SAGA has faster inference and comparable training speed to SAM-mask distillation methods including OmniSeg3D and GARField.By contrast, SA3D and SA-GS avoid training but use iterative mask refinement that incurs significant inference time.
  • Qualitative Results: SAGA achieves fine-grained segmentation at various scales and preserves thin, high-frequency structures captured by 3D-GS.Shrinking the Gaussians by 60% reveals the underlying structural modeling and completeness of the segmentation results.
  • Ablation Study: Local feature smoothing primarily eliminates outliers, while feature norm regularization improves alignment for inner Gaussians and preserves object interiors.Without smoothing, thresholded segmentation produces many false positives; without norm regularization, the apple becomes translucent at a 0.95 similarity threshold.

5 Limitation

SAGA’s affinity features are learned from multi-view 2D masks extracted by SAM, limiting segmentation of targets absent from those masks, especially small targets.

  • SAGA struggles to segment objects that do not appear in the multi-view 2D masks extracted by SAM.
  • The limitation is particularly evident for small targets, motivating better generalization during automatic mask extraction.

6 Conclusion

SAGA injects SAM’s segmentation capability into affinity features attached to 3D Gaussians, while a lightweight scale gate preserves multi-granularity segmentation with minimal overhead.

  • SAGA attaches Gaussian affinity features to all 3D Gaussians, integrating SAM-based segmentation capability into 3D-GS.
  • The lightweight scale gate adapts affinity features to different 3D physical scales with minimal computation overhead.
  • After training, SAGA achieves real-time fine-grained 3D segmentation.
  • The paper presents SAGA as one of the first methods addressing promptable segmentation in 3D-GS.

A Appendix

The appendix documents SAGA’s training and implementation, analyzes open-vocabulary segmentation and scale-gate interpretability, and evaluates robustness across additional radiance fields.

  • The appendix provides SAGA’s concrete training strategy and implementation details.
  • It details SAGA’s open-vocabulary segmentation ability and analyzes its limitation.
  • An interpretability analysis examines the underlying principle of the scale-gate mechanism.
  • The appendix evaluates robustness and generalizability by applying SAGA to additional kinds of radiance fields.

A.1 Detailed Additional Training Stategy

SAGA addresses training-data imbalance through resampling and re-weighting strategies that target scale sensitivity, positive-negative balance, and mask-size imbalance.

  • Data imbalance arises from scale insensitivity, positive-negative sample imbalance, and target-size imbalance.
  • Resampling addresses scale-sensitivity and positive-negative sample imbalance, while pixel-wise re-weighting addresses target-size imbalance.
  • Resampling: Each iteration samples Ns scales and Np pixels, forming Np × Np pixel pairs and a scale-conditioned correspondence matrix.
  • Resampling: Hard negative pairs with correspondence above 0.5 and hard positive pairs below 0.75 are added to the loss.
  • Re-weighting: Uniform pixel-pair sampling makes optimization dominated by large masks.
  • Re-weighting: Loss weights are inversely defined from mean mask sizes and min-max normalized to [1, 10] for stable training.

A.2 Implementation Details

SAGA uses consistent training and inference settings across scenes, including fixed feature and smoothing dimensions, sampled scales and pixels, and a single RTX 3090 GPU.

  • Training settings: SAGA uses 32-dimensional affinity features, K=16 for KNN local smoothing, and 10,000 training iterations across scenes.Each iteration samples eight scales and 1,000 pixels, with no loss-balance coefficient adjustments.
  • Training settings: Training samples eight different scales and 1,000 pixels per iteration for affinity-feature learning.The implementation specifies 1000^2 pixel pairs from the sampled view.
  • Models and hardware: SAGA extracts multi-view masks with SAM ViT-H and uses Open-CLIP ViT-B/16 for open-vocabulary segmentation.All training and inference runs on a single Nvidia RTX 3090 GPU.
  • Evaluation setup: NVOS evaluation uses positive and negative scribble points as reference-view prompts, then evaluates rendered target-view segmentation with IoU and pixel-wise accuracy.The supplied experiment description covers the prompt and evaluation procedure for NVOS.

A.3 Vote-based Open-vocabulary Segmentation

SAGA enables open-vocabulary 3D segmentation by clustering multi-view masks into potential 3D targets and aggregating CLIP-based text relevancy across each cluster. Its vote strategy avoids requiring a user-selected view but can confuse nested semantic granularities and lose context for isolated segments.

  • Open-vocabulary setting: SAGA uses a vote-based strategy so open-vocabulary segmentation requires only a text prompt rather than a specific queried view.This setting is stricter than approaches that convert language prompts into visual prompts in a selected view.
  • Vote graph construction: Scale-conditioned mask features cannot be clustered directly across scales, so SAGA uses segmented Gaussians as global features for multi-view mask clustering.The method samples anchor Gaussians and derives scale-conditioned segmentation results for each mask.
  • Vote graph construction: SAGA thresholds similarities between each mask feature and anchor Gaussians, then uses the IoU distance between resulting 3D sets for HDBSCAN clustering.For masks M1 and M2, the distance is defined from the intersection and union of their 3D segmentation results.
  • Vote graph construction: Masks grouped in the vote graph represent the same 3D target, with each cluster centroid corresponding to a potential 3D segmentation target.Targets may be instances or parts.
  • Vote-based inference: For each masked image, SAGA extracts a CLIP visual feature and aggregates text relevancy scores across masks associated with each cluster centroid.Semantic labels can then be assigned to clusters using the category with the highest relevancy score.
  • Limitations: Open-vocabulary voting can misclassify larger nested objects when a smaller category appears in masks at larger scales.The paper illustrates this with an egg inside noodles inside a bowl, where CLIP may label larger regions as “egg.”
  • Limitations: CLIP-based grounding also suffers when SAM isolates a texture without sufficient surrounding context, such as a texture segmented from a wall.This context limitation is shared with other CLIP-SAM methods cited by the paper.

A.4 Interpretability Analysis

The analysis examines learned scale gates and compares SAGA with alternative radiance-field representations and GARField. Results indicate that scale gating becomes more selective at larger scales, while explicit affinity features better preserve small objects during coarse segmentation.

  • Scale-gate statistics: Across 47 scenes and 1,504 scale-gate entries, 36.1% of gates are positive and 63.9% are negative.A typical scene therefore has about 12 positive and 20 negative gates.
  • Scale-gate statistics: Larger-scale inputs tend to close more gates, leaving fewer active features for coarse segmentation.The paper relates this pattern to the need for more features to capture finer-grained detail.
  • Cross-representation analysis: Adapting SAGA’s scale-gate mechanism to GARField produces competitive performance, indicating applicability beyond the original 3D-GS representation.The adaptation replaces the scale-conditioned affinity field with a scale-gated affinity field.
  • Cross-representation analysis: Applying SAGA to InstantNGP also produces competitive segmentation results, using 3D-GS queries to extract hash-grid features for clustering.Direct clustering on InstantNGP’s hash grid is described as infeasible.
  • Comparison with GARField: Compared with GARField, SAGA’s explicit affinity features show greater stability and preserve small objects during larger-scale segmentation.GARField tends to merge smaller objects into larger ones at coarse scales, whereas SAGA uses per-Gaussian features and a one-layer scale gate.
  • Additional qualitative results: Figures A4 and A5 provide additional qualitative results for SAGA.These figures extend the paper’s qualitative evidence beyond the GARField comparison.
Loading 2312.00860v3…