PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding

Jin-Chuan Shi, Miao Wang, Hao-Bin Duan, Shao-Hua Guan

arXiv:2311.18482v1cs.CVcs.GR

TL;DR

Open-vocabulary 3D querying is important for scene understanding, but language-embedded representations can be costly and raw semantic features strain 3D Gaussian memory. The paper introduces Language Embedded 3D Gaussians with quantized compact semantics and uncertainty-guided smoothing, achieving high visual and querying quality with real-time rendering. Its limitations include difficulty with fine-grained geometries and highly reflective or translucent objects.

  • Problem

    Language-embedded 3D representations must incorporate accurate semantics while retaining the efficiency and visual quality of current 3D scene representations.

  • Method

    The method quantizes compact semantic features on 3D Gaussians and adaptively smooths them using learned uncertainty to address memory costs and multi-view inconsistency.

  • Results

    The representation achieves state-of-the-art visual quality and language-querying performance while maintaining real-time rendering on consumer-level devices.

  • Takeaways & Limitations

    Language Embedded 3D Gaussians provide a high-accuracy, resource-efficient scene representation for open-vocabulary querying and high-quality novel view synthesis.

  • Takeaways & Limitations

    DINO features remain limited for fine-grained geometries at high resolutions, and highly reflective or translucent objects remain challenging.

Abstract

from arXiv · show

Open-vocabulary querying in 3D space is challenging but essential for scene understanding tasks such as object localization and segmentation. Language-embedded scene representations have made progress by incorporating language features into 3D spaces. However, their efficacy heavily depends on neural networks that are resource-intensive in training and rendering. Although recent 3D Gaussians offer efficient and high-quality novel view synthesis, directly embedding language features in them leads to prohibitive memory usage and decreased performance. In this work, we introduce Language Embedded 3D Gaussians, a novel scene representation for open-vocabulary query tasks. Instead of embedding high-dimensional raw semantic features on 3D Gaussians, we propose a dedicated quantization scheme that drastically alleviates the memory requirement, and a novel embedding procedure that achieves smoother yet high accuracy query, countering the multi-view feature inconsistencies and the high-frequency inductive bias in point-based representations. Our comprehensive experiments show that our representation achieves the best visual quality and language querying accuracy across current language-embedded representations, while maintaining real-time rendering frame rates on a single desktop GPU.

1. Introduction

Language-embedded 3D representations support intuitive open-vocabulary scene querying but must preserve efficiency and visual quality. Language Embedded 3D Gaussians address this challenge through compact semantic encoding and adaptive smoothing, achieving strong quality and real-time rendering.

  • Motivation: Open-vocabulary querying enables intuitive interaction with 3D scenes and supports applications including scene editing, VR/AR, autonomous driving, and robotic navigation.Visual-language features such as CLIP and DINO also cover long-tail objects more comprehensively than traditional semantic labels.
  • Motivation: Existing language-embedded representations require additional semantic prediction branches, while raw semantic features on massive numbers of 3D Gaussians create prohibitive memory demands.Trivially expanding output channels also makes high-precision, robust semantic recovery difficult.
  • Method: Language Embedded 3D Gaussians use feature quantization to compactly store semantic features and reduce computational and memory costs.The scheme leverages redundancy in local semantic features for efficient optimization and rendering on consumer devices.
  • Method: An uncertainty-guided mechanism lowers semantic feature spatial frequency to reduce ambiguity from multi-view visual inconsistency while retaining precision.The resulting language features are smoother yet still precise.
  • Results: The method achieves state-of-the-art novel view synthesis and open-vocabulary querying while supporting real-time rendering on consumer-level devices.The reported comparison covers visual quality, language query precision, and rendering speed simultaneously.

2. Related Work

Prior work combines neural or explicit 3D scene representations with language and visual-language features for semantic understanding. However, existing open-vocabulary approaches often analyze established point clouds rather than optimizing scene representations directly from multi-view images.

  • Neural Rendering: Explicit and hybrid scene representations use structures such as hash grids and point clouds to reduce the computational cost of large neural networks.These methods contrast with NeRF approaches that can suffer from slow training and rendering.
  • Language Embedded Scene Representation: NeRF-based language representations commonly add network branches to integrate semantic data, but identifying precise 3D regions remains difficult.DFF uses a language-prediction branch with pretrained-encoder supervision, while LERF assimilates CLIP features from multiscale imagery.
  • Open Vocabulary 3D Scene Understanding: Open-vocabulary scene understanding increasingly aligns 2D vision-language features with 3D point-cloud data for zero-shot learning.These methods project or align features between 3D data and 2D representations.
  • Open Vocabulary 3D Scene Understanding: Most related point-cloud methods analyze existing scene representations rather than optimizing representations from multi-view images.This distinction identifies the focus of the present work.

3. Method

The method builds a compact language-embedded Gaussian representation by quantizing dense CLIP/DINO features, rendering continuous semantic embeddings, and smoothing them adaptively using uncertainty. It addresses storage costs and multi-view semantic inconsistencies while supporting open-vocabulary querying.

  • Dense Language Feature Extraction: Dense CLIP and DINO features from multi-view images are concatenated into hybrid language feature maps for semantic representation.DINO complements CLIP by improving language-feature grouping and detail without labeled data.
  • Quantization of Language Features: Quantization compresses continuous hybrid features into discrete bases and semantic index maps, with compression controlled by the feature-space size N.Each feature selects a basis using maximum similarity, producing an image-level semantic indices map.
  • Training Process: The training pipeline initializes Gaussian scenes and semantic attributes, quantizes dense features, rasterizes Gaussian attributes into 2D maps, and optimizes semantic and smoothing losses.The pipeline follows 3D Gaussian Splatting for scene initialization and uses differentiable rasterization.
  • Compact Semantic Features on 3D Gaussians: The representation stores compact continuous semantic vectors on 3D Gaussians rather than directly embedding discrete indices, then decodes rendered features with a tiny MLP.Direct index embedding is unsuitable for differentiable rendering because indices are not continuous.
  • Semantic Feature Smoothing: Visual differences across viewpoints can produce high variance in features for the same location, complicating precise semantic learning.The cited sources identify viewing angle, illumination, specular materials, and semi-transparent materials as sources of this variance.
  • Semantic Feature Smoothing: Adaptive spatial smoothing uses learned uncertainty values to control semantic regularization and prevents the trivial solution in which every Gaussian receives maximum uncertainty.The uncertainty values are jointly optimized with compact semantic features and separately regularized.

4. Implementation Details

The implementation uses PyTorch and the CUDA rasterization kernel from 3D Gaussian Splatting to accelerate rendering while optimizing geometry, appearance, and semantic features.

  • Implementation Details: PyTorch and a modified 3D Gaussian Splatting CUDA kernel implement semantic-feature rendering and accelerate rasterization.The method retains the RGB loss and adaptive Gaussian density control used during scene reconstruction.

5. Experiments

Experiments evaluate visual quality, language-query accuracy, efficiency, and open-vocabulary coverage against DFF, LeRF, and 3DOVS. The method achieves strong qualitative and quantitative results while supporting real-time rendering and compact semantic storage.

  • Experimental Setup: The evaluation uses six annotated Mip-NeRF360 scenes, with 180 to 320 multi-view images per scene and segmentation masks for primary objects.
  • Experimental Setup: Comparisons measure PSNR, SSIM, LPIPS, mIoU, mPA, mP, mAP, FPS, host memory, video memory, and language-feature disk usage.
  • Comparisons: The method delivers the highest visual rendering quality and query accuracy across the tested indoor and outdoor scenes, while competing methods show failures or blurred boundaries on challenging objects.DFF misses some queries, 3DOVS performs poorly on complex objects, and LeRF’s grid representation limits clear boundaries.
  • Comparisons: Nearly 100 times faster inference accompanies higher rendering quality and semantic query accuracy, with lower computational demands than the compared methods.The comparison also reports host memory, video memory, and disk usage for learned language features.
  • Comparisons: Quantization enables detailed semantics in scenes with many 3D Gaussians while achieving the most efficient storage utilization among the baselines.
  • Open-vocabulary Query: Diverse queries cover visual attributes, subjective adjectives, large objects, and specific object components, demonstrating identification across vocabulary categories and granularities.
  • Ablation Study: Ablations show that adaptive uncertainty-aware smoothing improves mAP, DINO features sharpen object boundaries, and load balancing improves distinction between similarly semantic objects.Embedding uncertainty without spatial smoothing, or smoothing without accounting for multi-view semantic variance, impedes precise feature optimization.

6. Conclusion

Language Embedded 3D Gaussians compactly embed semantic features while smoothing viewpoint inconsistencies, supporting high-quality novel view synthesis and accurate open-vocabulary querying with modest resources. The method remains limited for fine-grained high-resolution geometry and highly reflective or translucent objects.

  • Conclusion: The representation stores compact, quantized semantic features on massive 3D Gaussians with minimal memory and storage requirements.Adaptive spatial smoothing lowers semantic-feature spatial frequency using learned uncertainty values.
  • Conclusion: It supports both high-quality novel view synthesis and high-accuracy open-vocabulary querying using modest computational resources.
  • Limitations and Future works: DINO features improve object boundary detection but remain weak at pinpointing fine-grained object geometries at high resolutions with CLIP-derived semantics.
  • Limitations and Future works: Highly reflective or translucent objects, including televisions and mirrors, remain challenging to detect.
  • Limitations and Future works: Advanced visual-language models and native per-pixel semantic features are identified as possible ways to overcome these limitations.

7. Load Balancing Loss

The load balancing loss maximizes use of the discrete feature space during quantization and helps avert quantization collapse. It compares feature utilization ratios with mean selection probabilities before aggregating their element-wise products.

  • Load Balancing Loss: The load balancing loss is introduced to maximize optimized feature-space utilization and avert quantization collapse.
  • Load Balancing Loss: For K quantized features, the utilization ratio of each feature in the discrete space S is calculated.
  • Load Balancing Loss: The mean selection probability for each feature is computed across the K quantizations.
  • Load Balancing Loss: The distance vector D(F_i, S) lists distances from feature F_i to each feature in S.
  • Load Balancing Loss: The loss aggregates the element-wise product of utilization ratios r and selection probabilities p.The symbol ◦ denotes element-wise multiplication.

8. Inference Strategy

During inference, rasterization and alpha blending project compact Gaussian semantics into a 2D feature map, which a decoder converts into semantic-index distributions before reconstruction with the quantized feature matrix. Text prompts then produce object-relevance maps.

  • Inference Strategy: Rasterization and alpha blending project compact semantic features from 3D Gaussians into a 2D feature map.
  • Inference Strategy: A trained MLP decoder and softmax convert the rendered feature map into a language-feature index distribution M_infer.M_infer has dimensions H × W × N.
  • Inference Strategy: Each element of M_infer represents a feature distribution over the discrete feature space S.
  • Inference Strategy: The semantic feature decoder uses 8-dimensional Gaussian semantic features and a discrete language space whose size is denoted N.
  • Inference Strategy: Multiplying M_infer by the quantized language-feature matrix S yields the language feature map F.F is the feature map for the Gaussians viewed from camera pose p_cam.
  • Inference Strategy: A provided text prompt is used to compute an object-relevance map from F in accordance with LERF.

9. Implementation Details

The discrete language feature space size N is selected per scene and controls the capacity of semantic information it can represent.

  • Implementation Details: N is set to 32 for kitchen, 64 for bonsai, and 128 for other scenes.N is a hyperparameter adjustable according to the richness of semantic content.

10. Datasets

The evaluation uses six Mip-NeRF360 scenes to assess visual and semantic embeddings quantitatively and qualitatively, with manually annotated primary-object segmentation maps.

  • Six Mip-NeRF360 scenes are selected for quantitative and qualitative evaluation of visual and semantic embeddings.The Stump scene is excluded because it has insufficient semantic content.
  • Each scene’s evaluation set is manually annotated with segmentation maps for its primary objects.Text prompts corresponding to the annotated objects are listed in Table 5.

11. More Results

The additional results compare visual quality and query accuracy across language-embedded representations and demonstrate diverse open-vocabulary queries. They also show that the method maintains detailed novel-view synthesis with reasonable memory requirements.

  • The qualitative results include comparisons of visual quality, novel-view synthesis, query accuracy, and diverse open-vocabulary queries.
  • Table 5 lists the text prompts used to evaluate open-vocabulary query quality and accuracy.
  • The evaluation materials include ground-truth segmentation masks for large and small objects, including geometrically complex, transparent, and translucent entities.Examples include bicycles, windows, and water glasses.
  • The visual-quality comparison shows that dense language features remain compatible with reasonable memory use, enabling many points to be rendered and optimized simultaneously.The reported outcome is the best visual quality with more details compared with other methods.
  • The approach supports open-vocabulary queries spanning visual attributes, general terms, materials, olfactory properties, and related actions.
Loading 2311.18482v1…