PaperlayerView exact revision on arXiv

Source-linked AI summary

Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration

Kim Jun-Seong, GeonU Kim, Kim Yu-Ji, Yu-Chiang Frank Wang, Jaesung Choe, Tae-Hyun Oh

arXiv:2502.16652v1cs.CV

TL;DR

Open-vocabulary 3D scene understanding is limited by rendering-based retrieval and scene-specific feature compression. Dr. Splat directly registers CLIP embeddings to 3D Gaussians and applies general-data Product Quantization without per-scene optimization. It reports superior performance across object selection, localization, and semantic segmentation, while generalization and cosine-similarity discriminability remain limitations.

  • Problem

    Rendering-based methods can distort language embeddings, require rendering for 3D retrieval, and incur per-scene costs for compressed features.

  • Method

    Dr. Splat directly registers language-aligned CLIP embeddings to 3D Gaussians and uses Product Quantization trained on large-scale image data for compact representation without per-scene optimization.

  • Results

    Dr. Splat reports superior performance across open-vocabulary 3D object selection, 3D object localization, and 3D semantic segmentation.

  • Takeaways & Limitations

    The method balances representative language quality with computational efficiency for open-vocabulary 3D scene understanding.

  • Takeaways & Limitations

    Generalization remains an area for improvement, and cosine similarity alone may reduce discriminability for some similarity scores.

Abstract

from arXiv · show

We introduce Dr. Splat, a novel approach for open-vocabulary 3D scene understanding leveraging 3D Gaussian Splatting. Unlike existing language-embedded 3DGS methods, which rely on a rendering process, our method directly associates language-aligned CLIP embeddings with 3D Gaussians for holistic 3D scene understanding. The key of our method is a language feature registration technique where CLIP embeddings are assigned to the dominant Gaussians intersected by each pixel-ray. Moreover, we integrate Product Quantization (PQ) trained on general large-scale image data to compactly represent embeddings without per-scene optimization. Experiments demonstrate that our approach significantly outperforms existing approaches in 3D perception benchmarks, such as open-vocabulary 3D semantic segmentation, 3D object localization, and 3D object selection tasks. For video results, please visit : https://drsplat.github.io/

1. Introduction

Dr. Splat addresses limitations of rendering-based open-vocabulary 3D scene understanding by directly linking language features to 3D Gaussians and using Product Quantization for compact, efficient representation.

  • Open-vocabulary 3D scene understanding enables natural-language interpretation and referencing of 3D spatial information beyond predefined categories.The paper identifies applications in autonomous navigation, robotics, and augmented reality.
  • Dr. Splat requires no per-scene optimization and takes ∼10m, compared with OpenGaussian’s required optimization and ∼1h search preparation.Both methods are listed as using a 3D search domain with fast search and small search databases.
  • Rendering-based methods distort language embeddings and impede holistic 3D tasks such as semantic segmentation and object localization.Their reliance on rendering also makes full spatial coverage less efficient than direct 3D Gaussian methods.
  • Dr. Splat directly registers language-aligned CLIP embeddings with 3D Gaussians, bypassing intermediate rendering while preserving 3D spatial structure.The approach is designed for open-vocabulary localization, object selection, and semantic segmentation.
  • 6.25% compression ratio is achieved by storing a pre-trained Product Quantization index for each Gaussian instead of full-length or scene-specific compressed features.PQ reduces memory usage while preserving embedding richness and avoiding per-scene optimization.

2. Related Work and Motivation

The paper motivates direct 3D retrieval by showing why rendered language features are difficult to search and proposes direct feature allocation with general-data Product Quantization.

  • Language-based 3D scene understanding: Language-embedded 3D methods extend point clouds, NeRFs, and Gaussian Splatting with language knowledge for open-vocabulary 3D comprehension.These representations commonly use embeddings from 2D foundation models such as CLIP, LSeg, and DINO.
  • Preliminary of 3D Gaussian Splatting: 3D Gaussian Splatting represents scene appearance and geometry with Gaussian means, covariances, opacities, and colors, then renders 2D pixels.Its covariance is decomposed into scale and rotation matrices, and parameters are optimized against image color.
  • Language embedded 3D Gaussian Splatting: Rendering-based language distillation optimizes Gaussian embeddings so their weighted rendered features match 2D language embeddings.This replaces color rendering with language-embedding rendering while keeping the Gaussian scene representation.
  • Motivation: Direct retrieval is difficult because distributed Gaussian embeddings do not carry language information independently, while rendered weighted sums do.Compressed-feature decoders can also be incompatible with direct retrieval over per-Gaussian embeddings, producing degenerated CLIP decoding.
  • Motivation: Camera-view selection for complete coverage forms an NP-hard set covering problem, while retrieval over rendered views remains substantial for large scenes.The 3D location must then be lifted from 2D retrieval, adding another mechanism to the pipeline.
  • Motivation: A training-free algorithm directly allocates language embeddings to 3D Gaussians, enabling computation and interaction within 3D space.Unlike the concurrent OpenGaussian approach, Dr. Splat does not require per-scene codebook construction.

3. Dr. Splat

Dr. Splat directly registers language-aligned CLIP embeddings with 3D Gaussians, avoiding feature rendering and per-scene optimization. It aggregates multi-view features for Gaussian-level understanding and uses product quantization for compact storage and efficient retrieval.

  • Feature registration: Dr. Splat reconstructs a language-embedded 3D space by directly interacting with Gaussians rather than rendering feature maps.The method extracts per-pixel CLIP maps and links them to 3D Gaussians through projection relations.
  • Feature registration: Only the Top-k dominant Gaussians along each pixel ray receive mapped CLIP embeddings, after which multi-view contributions are aggregated per Gaussian.The aggregation uses per-pixel Gaussian weights derived from projection and visibility relations, and unassigned Gaussians are pruned.
  • Feature registration: Weighted averaging in the original high-dimensional feature space produces viewpoint-consistent Gaussian features without gradient-based optimization.The process is interpreted as inverse volume rendering and is intended to be faster than prior per-scene optimization methods.
  • Experiments: On LeRF-OVS, the model’s object-selection evaluation measures 2D segmentation accuracy from rendered selected Gaussians without per-scene optimization.The supplied table caption defines the evaluation protocol and notes robustness across diverse scenes.
  • Product-Quantized CLIP embeddings: Product Quantization compresses Gaussian-registered embeddings using subvector-specific codebooks and replaces each subvector with its nearest-centroid index.PQ centroids are trained from CLIP embeddings collected from over 1.2M LVIS instances, eliminating per-scene training.
  • Product-Quantized CLIP embeddings: PQ lookup reduces search complexity from O(D) to O(1) after centroid distances are tabulated, supporting efficient large-scale retrieval.The approach addresses memory and scene-specific tuning costs associated with high-dimensional Gaussian features.

4. Experiments

Experiments evaluate Dr. Splat on open-vocabulary 3D object selection, localization, and semantic segmentation, alongside ablations of its evaluation and efficiency trade-offs.

  • 3D object selection: Dr. Splat improves object selection in most scenes, exceeding counterpart models by over 0.5 mIoU and more than 4.5 mAcc.
  • 3D object selection: Dr. Splat’s geometry-aware feature registration and general PQ preserve feature distinctiveness across scene complexity.
  • 3D object localization: The volume-aware localization protocol weights Gaussians by their spatial significance, reflecting their unequal volumetric contributions.The significant score uses Gaussian scale and opacity to approximate relative ellipsoid volume.
  • 3D object localization: Dr. Splat consistently outperforms rasterization-based methods and OpenGaussian on ScanNet 3D localization.
  • 3D semantic segmentation: Semantic segmentation assigns each Gaussian the highest-activation label, and performance surpasses the baseline as Top-k increases.
  • Ablation study: PQ sub-vector sizes of 64, 128, and 256 correspond to bit-size reductions of 1/32, 1/16, and 1/8 of the original CLIP feature.
  • Ablation study: Increasing the number of Gaussians per ray improves localization but increases memory consumption and occupied Gaussians.

5. Discussion and Conclusion

Dr. Splat directly registers language embeddings to 3D Gaussians, avoiding intermediate rendering and accelerating direct 3D search with Product Quantization. Experiments report superior performance across several 3D scene-understanding tasks.

  • Dr. Splat directly registers language embeddings to 3D Gaussians, eliminating the intermediate rendering process.
  • Product Quantization further accelerates direct search over the registered 3D representations.
  • Dr. Splat achieves superior performance across 3D object selection, localization, and semantic segmentation tasks.

Supplementary Material

The supplementary material adds implementation details, experimental analyses, qualitative results, city-scale evaluations, and discussion of limitations and future directions.

  • A dedicated section addresses limitations and future directions, alongside a supplementary video.
  • The supplement covers implementation, evaluated 3D tasks, experimental setup, and a Gaussian-friendly evaluation protocol.
  • It includes search-time experiments, qualitative results, annotation analyses, and city-scale dataset evaluations.

A. Implementation Details

Dr. Splat constructs PQ codebooks before training, aggregates multiview CLIP features into Gaussian-registered embeddings, and directly refers to those embeddings at inference.

  • The pipeline has preprocessing, training, and inference stages for constructing PQ codebooks, registering embeddings, and referring to 3D Gaussians.
  • Pre-Training stage: PQ codebooks are built from CLIP embeddings extracted from object-image patches, with subvector centroids later kept frozen.
  • Pre-Training stage: The method initializes 3D Gaussians using optimized parameters from the original 3D Gaussian Splatting method after 30,000 iterations.
  • Training stage: Training extracts pixel-wise CLIP embeddings, aggregates features, and registers them using pretrained Gaussian parameters and ray-based contribution weights.
  • Training stage: Aggregated Gaussian features are quantized into 128-channel 8-bit indices, while Gaussians never selected in Top-k are pruned.
  • Inference stage: At inference, cosine similarity compares a CLIP text query with each PQ-assigned Gaussian, producing scores for downstream 3D tasks.

B. Experiment Setup

The experiments evaluate Dr. Splat on 3D object selection, open-vocabulary 3D object localization, and open-vocabulary 3D semantic segmentation using LERF-OVS and ScanNet data.

  • The study evaluates three tasks: 3D object selection, open-vocabulary 3D object localization, and open-vocabulary 3D semantic segmentation.
  • 3D object selection: For 3D object selection, text-query CLIP features are compared with 3D Gaussian embeddings using cosine similarity and thresholding.
  • 3D object selection: The 3D object selection experiments use the LeRF-OVS dataset with annotations from LangSplat and evaluate multi-view 2D segmentation accuracy.
  • 3D object localization: For 3D object localization, the method extracts a CLIP text feature and computes cosine similarity with Gaussian embeddings for text-query localization.

C. Evaluation Protocols

The paper proposes a Gaussian-friendly evaluation protocol that avoids constraining Gaussian optimization and measures volumetric significance through weighted Gaussian IoU.

  • Protocol comparison: OpenGaussian fixes Gaussian positions and count during optimization, whereas the proposed protocol updates both while preserving unconstrained scene training.
  • Protocol limitations: The paper identifies degraded rendering quality and count-only IoU as limitations of protocols that predefine Gaussian positions and numbers.
  • Semantic-label assignment: The evaluation assigns each Gaussian a semantic label by selecting the label with the highest summed Mahalanobis distance from ground-truth labeled points.
  • Volume-aware IoU: Each Gaussian receives a significance score d_i = s_ixs_iys_izα_i based on relative ellipsoid volume and opacity.
  • Volume-aware IoU: The proposed IoU weights Gaussians with higher significance scores, making the metric volume-aware.
  • Protocol validation: The metric shows high correlation with voxel-based IoU while avoiding repeated voxel-label assignment after pseudo-ground-truth Gaussians are generated.

D. Search-time Experiments

Search-time experiments compare product-quantization lookup with cosine similarity and examine evaluation protocols, additional tasks, and broader ScanNet-200 performance.

  • Product quantization: Product quantization reduces vector-distance search to lookup-table indexing over quantized subvectors.
  • Product quantization: Cosine similarity can be approximated with normalized subvector inner products, while quantization errors remain within reported bounds preserving distance correlation.
  • Search speed: With subvector sizes of 128, 64, and 32, the LUT-based approach improved search performance by approximately 2×, 6.6×, and 14.1×, respectively.The comparison averages inference time over one million feature points and 100 repeated experiments.
  • Additional task results: The supplementary experiments report favorable localization and object-selection behavior for the proposed direct 3D search compared with competing methods.
  • ScanNet-200: The method outperforms counterparts on ScanNet-200, which expands evaluation from 20 to 200 semantic categories including tail classes.

E.3. Experiments on the city-scale dataset

City-scale experiments use the Waymo San Francisco Mission Bay dataset to assess text-query localization and scalability across expansive 3D scenes.

  • Dataset: The Waymo test scenes contain approximately 12,000 images from 12 cameras, and three scene blocks are selected for large-scale evaluation.
  • Localization: The proposed method consistently localizes diverse text queries in 3D space, whereas LangSplat-m struggles with precise predictions.
  • Qualitative behavior: The qualitative results reflect objects and, to some extent, attributes such as color.
  • Scalability: On scenes with over 2.9M Gaussians and nearly 1M computations per image across more than 100 images, the method demonstrates scalable search processing.

F. Broader Applications and Limitations

Dr. Splat is presented as extensible to broader query modalities and dynamic 3D scenes, while its generalization, CLIP specificity, and threshold selection remain limitations.

  • Broader Applications: The method could support click or image queries through a self-referencing mechanism and dialogue-based interactions through integration with large language models.The passage presents these as potential extensions rather than demonstrated capabilities.
  • Broader Applications: Applying the method to canonical forms could extend it to dynamic 3D scenes beyond static environments.This adaptation is described as a possible extension for complex real-world scenarios.
  • Limitations: Generalization remains an area for improvement, and more diverse or finer-grained Product Quantization training could improve adaptability across varied contexts.The limitation is stated despite robust performance on combinations of nouns and adjectives and unfamiliar nouns.
  • Limitations: CLIP semantic associations can simultaneously activate related but distinct objects, reducing query specificity; re-ranking could mitigate this issue.The example given is a “red apple” query activating non-red apples or unrelated red objects.
  • Limitations: The method requires an appropriate similarity threshold, although the study uses a fixed threshold across scenes for stable and reproducible results.Scenario-specific optimization or dynamic adjustment could further refine localization accuracy.
Loading 2502.16652v1…