PaperlayerView exact revision on arXiv

Source-linked AI summary

Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields

Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang, Achuta Kadambi

arXiv:2312.03203v3cs.CV

TL;DR

NeRF-based feature-field distillation enables semantic scene tasks but is slowed by NeRF rendering and challenged by implicit feature representations. Feature 3DGS distills 2D foundation-model features into explicit 3D Gaussians using low-dimensional features and learned upsampling. Across applications, it reports comparable or better results with faster training and rendering, while remaining subject to teacher and representation limitations.

  • Problem

    NeRF-based feature-field distillation supports semantic tasks but is limited by slow rendering and challenges from implicitly represented feature fields and mismatched RGB-feature map resolutions and channels.

  • Method

    Feature 3DGS attaches semantic features to 3D Gaussians and distills 2D foundation-model embeddings through differentiable rendering, low-dimensional feature learning, and convolutional upsampling.

  • Results

    Across semantic segmentation, language-guided editing, and segment-anything applications, Feature 3DGS provides comparable or better results while being faster to train and render, with up to 2.7× speedup and 23% higher mIoU.

  • Takeaways & Limitations

    The framework provides an explicit, semantic, editable, and promptable 3D scene representation compatible with feature fields from models such as CLIP-LSeg and SAM.

  • Takeaways & Limitations

    Performance is constrained by limited student access to ground-truth features, imperfections in the teacher network, and noise-inducing floaters from the adapted 3DGS pipeline.

Abstract

from arXiv · show

3D scene representations have gained immense popularity in recent years. Methods that use Neural Radiance fields are versatile for traditional tasks such as novel view synthesis. In recent times, some work has emerged that aims to extend the functionality of NeRF beyond view synthesis, for semantically aware tasks such as editing and segmentation using 3D feature field distillation from 2D foundation models. However, these methods have two major limitations: (a) they are limited by the rendering speed of NeRF pipelines, and (b) implicitly represented feature fields suffer from continuity artifacts reducing feature quality. Recently, 3D Gaussian Splatting has shown state-of-the-art performance on real-time radiance field rendering. In this work, we go one step further: in addition to radiance field rendering, we enable 3D Gaussian splatting on arbitrary-dimension semantic features via 2D foundation model distillation. This translation is not straightforward: naively incorporating feature fields in the 3DGS framework encounters significant challenges, notably the disparities in spatial resolution and channel consistency between RGB images and feature maps. We propose architectural and training changes to efficiently avert this problem. Our proposed method is general, and our experiments showcase novel view semantic segmentation, language-guided editing and segment anything through learning feature fields from state-of-the-art 2D foundation models such as SAM and CLIP-LSeg. Across experiments, our distillation method is able to provide comparable or better results, while being significantly faster to both train and render. Additionally, to the best of our knowledge, we are the first method to enable point and bounding-box prompting for radiance field manipulation, by leveraging the SAM model. Project website at: https://feature-3dgs.github.io/

1. Introduction

NeRF-based feature-field methods extend scene representations to semantic tasks but remain slow and face representation limitations. Feature 3DGS adds semantic features to 3D Gaussians, using low-dimensional distillation and convolutional upsampling for faster rendering and improved results.

  • Motivation: NeRF-based feature-field methods support semantic tasks such as editing and segmentation, but their training and inference are natively slow.These methods add descriptive features alongside radiance information in implicit 3D fields.
  • Motivation: 3D Gaussian Splatting improves training and rendering speed over NeRF while retaining comparable or better rendered-image quality.Its explicit Gaussian representation has enabled real-time rendering applications.
  • Motivation: Native 3D Gaussian Splatting does not support jointly learning semantic features and radiance information at each Gaussian.Feature 3DGS addresses this representation limitation by augmenting each Gaussian with semantic features.
  • Approach: Feature 3DGS distills 2D foundation-model guidance into semantic features attached to 3D Gaussians and differentiably rasterizes the resulting feature vectors.The framework targets arbitrary feature fields and supports applications including semantic segmentation, language-guided editing, and instance segmentation.
  • Results: Up to 2.7× faster feature-field distillation and rendering is achieved through low-dimensional distillation followed by learned convolutional upsampling.The framework also reports up to 23% improvement on mIoU for semantic segmentation tasks.

2. Related Work

Related work develops implicit and explicit 3D scene representations for rendering and semantic understanding. NeRF-derived methods support feature-field distillation and 3D segmentation, while explicit representations target more efficient scene processing.

  • Implicit Radiance Field Representations: NeRF represents 3D scenes with coordinate-based neural networks and has achieved strong novel-view synthesis performance.Subsequent methods address aliasing, efficiency, generalization, and large-scale rendering within implicit representations.
  • Feature Fields: Feature 3DGS uses 3D Gaussian initialization with an added semantic feature and a parallel N-dimensional rasterizer for high-dimensional feature rendering.An optional convolutional speed-up module complements the rasterizer.
  • Implicit Radiance Field Representations: NeRF-based methods are hindered by slow rendering and substantial training memory usage because of their implicit design.Pure implicit radiance fields may require millions of neural-network queries to render a large-scale scene.
  • Implicit Radiance Field Representations: Tensor factorization, multiscale hash grids, and related explicit representations reduce the computational cost of implicit radiance-field processing.These approaches include Triplane, TensoRF, K-Plane, TILED, InstantNGP, and Block-NeRF.
  • Feature Fields: Semantic NeRF and Panoptic Lifting embed semantic data into 3D spaces, showing that merged noisy or inconsistent 2D labels can yield precise 3D segmentation.Other methods demonstrate 3D object segmentation with minimal user input.

3. Method

Feature 3DGS extends 3D Gaussian Splatting with explicit semantic features distilled from 2D foundation models. Its rasterization, joint optimization, and speed-up design address differing feature dimensions and resolutions while supporting promptable scene manipulation.

  • Feature rendering: A parallel N-dimensional Gaussian rasterizer jointly renders RGB images and arbitrary-dimensional feature maps at shared resolution.Joint tile-based rasterization avoids independently rendering images and feature maps with incompatible dimensions or shared attributes.
  • Feature representation: Feature 3DGS stores an N-dimensional semantic feature f alongside color and other optimizable attributes at each 3D Gaussian.Each Gaussian includes position, rotation, scale, opacity, color, and semantic feature parameters.
  • Feature distillation: Rendered student features Fs are supervised by teacher embeddings Ft extracted from ground-truth images with 2D foundation-model encoders.The model distills a large 2D teacher into an explicit 3D student through differentiable volumetric rendering.
  • Optimization: The objective combines photometric and feature losses, with rendered features resized by bilinear interpolation for per-pixel comparison.The loss is L = L_rgb + γL_f, and the implementation sets γ = 1.0 and λ = 0.2.
  • Optimization: Unlike NeRF feature distillation, equal-weighted joint optimization in the explicit representation avoids the reported sensitivity to the feature-loss weight γ.The authors report that semantic features contribute to scene understanding and physical attribute depiction.
  • Optimization and speed-up: A lightweight convolutional speed-up module renders lower-dimensional features and upsamples them to the foundation model’s target dimension.This reduces computation while enabling channel-wise communication through a learnable 1 × 1 convolution.
  • Promptable editing: Prompt similarities and softmax probabilities select 3D Gaussians for extraction, deletion, appearance modification, or novel-view editing.The method supports text-label or point-set prompts and updates Gaussian color and opacity for the selected region.

4. Experiments

Experiments evaluate Feature 3DGS for LSeg semantic segmentation, SAM prompting from novel views, and language-guided editing. Across these tasks, the method preserves or improves quality while accelerating feature rendering and inference.

  • Novel view semantic segmentation: 1.66× faster inference is reported than NeRF-DFF when the rendered feature dimension is 128, alongside more fine-grained segmentation and higher-quality feature maps.Figure 3 compares results on Replica and LLFF scenes.
  • Novel view semantic segmentation: Feature 3DGS distills semantic features to support novel-view semantic segmentation beyond conventional class-limited labels.Semantic features can represent unseen labels through embedding-space relationships, and the experiments distill LSeg features for this task.
  • Novel view semantic segmentation: Feature 3DGS achieves higher accuracy and mIoU than NeRF-DFF on Replica while producing better novel-view RGB images and segmentation masks.The comparison covers both synthetic and real scenes.
  • Segment Anything from Any View: 1.7× faster total inference is achieved for point- and box-prompted SAM segmentation while preserving nearly equivalent mask quality.The comparison includes rendering and segmentation against the naive SAM encoder-plus-decoder approach.
  • Segment Anything from Any View: SAM segmentation from novel views uses directly rendered features and only the decoder, avoiding novel-view image synthesis followed by the full encoder-decoder pipeline.The rendered feature dimension is set to 128, half of SAM’s 256-dimensional latent representation, while maintaining comparable quality.
  • Language-guided Editing: Language-guided editing queries the 3D feature field, scores Gaussians against text targets, and modifies their color and opacity for extraction, deletion, or appearance changes.The pipeline supports soft or hard selection and creates a thresholded Gaussian mask for editing.

5. Discussion and Conclusion

The paper presents Feature 3DGS as an explicit, semantic, editable, and promptable 3D representation integrating Gaussian Splatting with foundation-model feature distillation. It reports higher-quality segmentation and editing results, while identifying supervision, teacher quality, and floaters as limitations.

  • Discussion and Conclusion: Feature 3DGS integrates 3D Gaussian Splatting with 2D foundation-model feature distillation for semantic, editable, and promptable explicit 3D scenes.The demonstrated foundation models include CLIP-LSeg and SAM.
  • Discussion and Conclusion: Performance is constrained by limited student access to ground-truth features, imperfections in the teacher network, and noise-inducing floaters from the adapted 3DGS pipeline.The authors identify these factors as challenges affecting framework effectiveness and optimal performance.
  • Discussion and Conclusion: Feature 3DGS produces higher-quality SAM masks with finer details than NeRF-DFF, including successful cup-bear separation and sharper bear-leg boundaries.These differences are shown for novel-view, box-prompted segmentation.
  • Discussion and Conclusion: Feature 3DGS extracts an entire banana hidden by an apple with fewer background floaters than NeRF-DFF and supports deletion and appearance modification.Examples include deleting a car while preserving the background and changing leaves without changing a stop sign.

Fields

The supplement is organized into sections covering architecture, training and inference, teacher features, experiments, editing, ablations, and failure cases.

  • Fields: The supplement covers network architecture, training and inference settings, teacher features, Replica experiments, language-guided editing, ablations, and complex-scene failure cases.These topics are assigned to Sections A through G.

A. Details of Architectures

The architecture uses a parallel N-dimensional Gaussian rasterizer and an optional convolutional speed-up module to render and align semantic feature maps with RGB images.

  • A. Details of Architectures: The parallel N-dimensional rasterizer uses point-based alpha blending while rendering RGB images and feature maps at matching sizes.Matching sizes mitigate inconsistent spatial resolution from tile-based rasterization.

B. Training and Inference Details

The pipeline rasterizes semantic features alongside radiance information and offers a speed-up path that renders lower-dimensional features before convolutional upsampling.

  • Training and inference pipeline: Direct rendering uses feature maps matching the teacher feature dimension, including 512 for LSeg and 256 for SAM.The alternative speed-up path reduces the rendered dimension before decoding.
  • Speed-up module: The speed-up module renders a lower-dimensional feature map and upsamples it to the teacher dimension with a lightweight convolutional decoder.This avoids the cost of directly rendering high-dimensional feature maps.
  • Rasterization: Algorithm 1 initializes a Structure-from-Motion point cloud with positions, colors, covariances, opacities, and semantic features.The parallel rasterizer then renders these attributes from camera and image inputs.
  • Optimization: Training iteratively rasterizes features, backpropagates updates, performs refinement steps, and continues until convergence.Refinement includes over-reconstruction or under-reconstruction branches.
  • Optimization settings: Feature rendering uses Adam with learning rate 1e−3, while the speed-up decoder uses a separate Adam optimizer with learning rate 1e−4.Image rendering follows the 3DGS optimization strategy and exponential decay scheduling.

C. Teacher Features

The method prepares LSeg and SAM teacher features with model-specific resolutions and dimensions, then converts rendered features into formats suitable for semantic prediction and visualization.

  • LSeg features: LSeg features use CLIP ViT-L/16 encoders and have resolution 360×480 with feature dimension 512.The implementation renders dimension 128 when using the speed-up module.
  • LSeg inference: LSeg segmentation reshapes rendered features from (512, 360, 480) to (360 × 480, 512) before matching them with category text features.Semantic segmentation then applies LSeg spatial regularization blocks.
  • SAM features: SAM uses a ViT-H/16 encoder producing a 64×64 feature map with feature dimension 256 after resizing and padding inputs to 1024×1024.For non-square originals, the padded feature map is cropped to retain the original aspect ratio.
  • SAM inference: SAM inference pads the rendered cropped feature back to 64 × 64 so the SAM decoder receives equivalent semantic information.The same representation supports promptable or promptless segmentation.
  • Feature visualization: PCA reduces either 512-dimensional LSeg or 256-dimensional SAM features to three components for RGB-like visualization.The features are centered using a sampled PCA mean before projection.
  • Feature characteristics: LSeg features emphasize coarse semantic boundaries, whereas SAM features capture instance-level information and finer object details.SAM feature maps also require cropping to remove padding regions.

D. Replica Dataset Experiment

The Replica experiment evaluates feature distillation across four scenes and uses CLIP-based semantic features for segmentation and editing operations on 3D Gaussians.

  • Replica setup: The Replica evaluation uses room 0, room 1, office 3, and office 4, with 5,000 training iterations per scene using LSeg features.Each scene provides 80 trajectory images, with every eighth image selected and some semantically close labels manually relabeled.
  • Language-guided editing: Language-guided editing computes CLIP text–semantic-feature similarities for each 3D Gaussian and converts them into category scores.The text encoder is CLIP ViT-B/32 and each text feature has dimension 512.
  • Region selection: Soft selection thresholds the score column for a queried category, while hard selection chooses each Gaussian’s highest-scoring category.Selected Gaussians define the target region for editing.
  • Region selection: Hybrid selection combines soft and hard Gaussian masks with a bitwise OR operation to make selection both tunable and less vulnerable to incorrect category assignment.The method addresses limitations of threshold-only and argmax-only selection.
  • Editing limitation: Deleting a cup can remain incomplete because tiny, detailed objects are difficult to select accurately with the Gaussian mask.This limitation is most apparent in complex scenes containing objects of varied sizes.
  • Edit operations: Editing updates Gaussian opacity or color for extraction, deletion, and appearance modification operations.Appearance modification can change an object’s visual property, such as green leaves to red leaves.

F. Ablation Studies

The ablations show that lower-dimensional rendering can preserve task performance while reducing training time, motivating the choice of dimension 128 in the Replica experiment.

  • Feature dimension: 2.4× faster training makes rendered feature dimension 128 faster than dimension 256, while both achieve the best reported accuracy.Dimension 256 is slightly better on mIoU, but dimension 128 is selected for the Replica experiment.
  • Feature dimension: Dimension 128 provides the best novel view synthesis result among the evaluated LSeg feature dimensions.The study also reports PSNR, SSIM, LPIPS, and FPS for different SAM feature dimensions.
  • Speed-up module: The Speed-up Module saves time without compromising performance in the reported ablations.Figures A and B substantiate this behavior.

G. Failure Cases

The method exhibits failure cases in challenging scenes, including imperfect point-prompted masks and incomplete deletion of tiny sophisticated objects during language-guided editing.

  • Failure Cases: Point-prompted segmentation can produce coarse boundaries and small holes in complex scenes with adjacent similar objects.The paper attributes this case to low feature quality from SAM distillation and difficulty depicting the car boundary.
  • Failure Cases: Language-guided editing may fail to delete tiny sophisticated objects thoroughly in complex scenes.
  • Failure Cases: Language-guided editing selects target regions from semantic-text feature similarities before updating Gaussian opacity and color.
Loading 2312.03203v3…