PaperlayerView exact revision on arXiv

Source-linked AI summary

LangSplat: 3D Language Gaussian Splatting

Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, Hanspeter Pfister

arXiv:2312.16084v2cs.CV

TL;DR

Existing 3D language fields face pixel-level ambiguity and slow NeRF-based rendering. LangSplat addresses these issues with language-enhanced 3D Gaussians, a scene-specific autoencoder, and SAM-based hierarchical semantics, achieving superior performance to LERF and a reported 199 × speed improvement.

  • Problem

    Existing 3D language fields are limited by image-aligned CLIP supervision, ambiguous point semantics, and time-consuming NeRF rendering.

  • Method

    LangSplat uses language-enhanced 3D Gaussians with tile-based splatting, scene-specific latent language features, and SAM-defined hierarchical semantics.

  • Results

    LangSplat significantly outperforms LERF in open-ended 3D language query tasks and achieves a 199 × speed improvement.

  • Takeaways & Limitations

    LangSplat enables precise and efficient open-vocabulary querying within 3D spaces.

Abstract

from arXiv · show

Humans live in a 3D world and commonly use natural language to interact with a 3D scene. Modeling a 3D language field to support open-ended language queries in 3D has gained increasing attention recently. This paper introduces LangSplat, which constructs a 3D language field that enables precise and efficient open-vocabulary querying within 3D spaces. Unlike existing methods that ground CLIP language embeddings in a NeRF model, LangSplat advances the field by utilizing a collection of 3D Gaussians, each encoding language features distilled from CLIP, to represent the language field. By employing a tile-based splatting technique for rendering language features, we circumvent the costly rendering process inherent in NeRF. Instead of directly learning CLIP embeddings, LangSplat first trains a scene-wise language autoencoder and then learns language features on the scene-specific latent space, thereby alleviating substantial memory demands imposed by explicit modeling. Existing methods struggle with imprecise and vague 3D language fields, which fail to discern clear boundaries between objects. We delve into this issue and propose to learn hierarchical semantics using SAM, thereby eliminating the need for extensively querying the language field across various scales and the regularization of DINO features. Extensive experimental results show that LangSplat significantly outperforms the previous state-of-the-art method LERF by a large margin. Notably, LangSplat is extremely efficient, achieving a 199 $\times$ speedup compared to LERF at the resolution of 1440 $\times$ 1080. We strongly recommend readers to check out our video results at https://langsplat.github.io/

1. Introduction

LangSplat addresses the speed, memory, and precision limitations of existing 3D language fields by combining Gaussian-based rendering, latent language features, and hierarchical semantics.

  • Existing methods distill CLIP features into 3D scenes but remain limited in speed and accuracy, constraining practical applicability.The paper identifies both the 3D modeling approach and rendering target as key design aspects.
  • CLIP embeddings are image-aligned rather than pixel-aligned, making language supervision for individual 3D points ambiguous across semantic scales.A point on a bear’s nose may correspond simultaneously to the nose, head, and bear concepts.
  • LERF produces imprecise 3D language features that fail to establish clear object boundaries.
  • LangSplat represents scenes with language-enhanced 3D Gaussians and tile-based splatting instead of NeRF for efficient high-resolution rendering.The method supervises the Gaussians with CLIP embeddings from patches across multiple training views.
  • A scene-wise language autoencoder maps CLIP embeddings to a low-dimensional latent space, reducing the memory cost of explicit language-feature modeling.Rendered latent features are decoded to obtain final language embeddings.
  • LangSplat learns hierarchical semantics using SAM to address ambiguity and improve the precision of 3D language fields.

2. Related Work

Prior work extends 3D representations and foundation-model features to dynamic scenes, segmentation, language fields, and robotic manipulation, with NeRF-based feature distillation as a recurring approach.

  • 3D Gaussian Splatting: 3D Gaussian Splatting represents scenes with collections of 3D Gaussians and enables real-time rendering at 1080p resolution.Subsequent studies extend the representation to dynamic scenes and deformable Gaussian models.
  • SAM: SAM is a foundational image-segmentation model with impressive zero-shot performance and applications across several vision tasks.Related work also explores using SAM in 3D domains.
  • 3D Language Fields: Early 3D feature fields distilled LSeg or DINO features across multiple views into NeRF representations.
  • 3D Language Fields: Later work distilled CLIP features into NeRFs for language-guided robotic manipulation and embedded semantic information into NeRF-based scene models.

3. Proposed Approach

LangSplat models 3D language fields with language-enhanced Gaussians, hierarchical SAM semantics, and scene-specific latent features. This design targets point ambiguity, costly NeRF rendering, and high memory demands while supporting open-vocabulary querying.

  • Learning Hierarchical Semantics with SAM: SAM supplies subpart, part, and whole segmentation maps that provide hierarchical semantics with clearer object boundaries.A regular grid of 32 × 32 prompts generates masks at the three semantic levels, which are filtered before full-image segmentation.
  • Learning Hierarchical Semantics with SAM: The resulting pixel-aligned CLIP embeddings align each rendered pixel with its precise semantic context, reducing point ambiguity without DINO regularization.Predefined whole, part, and subpart scales also avoid intensive searches across multiple absolute scales during querying.
  • 3D Gaussian Splatting for Language Fields: LangSplat uses 3D Gaussian Splatting with tile-based rasterization instead of NeRF for efficient high-resolution language-feature rendering.Each language-enhanced Gaussian is supervised by CLIP embeddings extracted from image patches across multiple training views.
  • Scene-Specific Language Latents: A scene-wise autoencoder compresses CLIP features into a low-dimensional latent space, where language Gaussians learn features with d = 3.The decoder reconstructs CLIP-space features during querying, while scene priors reduce memory cost compared with directly modeling D-dimensional embeddings.
  • Open-Vocabulary Querying: Rendered language embeddings are decoded back into CLIP space, enabling open-vocabulary 3D object localization and semantic segmentation.For each query, LangSplat produces relevancy maps at three semantic levels and selects the level with the highest relevancy score.

4. Experiments

LangSplat is evaluated on LERF and 3D-OVS using localization, semantic segmentation, qualitative comparisons, and ablations. It substantially improves accuracy over LERF while achieving large efficiency gains at higher image resolutions.

  • LangSplat achieves 84.3% overall localization accuracy on LERF, significantly outperforming LERF.
  • LangSplat improves LERF 3D semantic segmentation IoU by 14.0%.
  • A scene-specific autoencoder resolves the memory demands of explicitly modeling CLIP features and further improves accuracy and efficiency.Simply replacing NeRF with 3D Gaussian Splatting otherwise runs out of L1 cache memory.
  • 199 × speedup over LERF is achieved at 1440 × 1080 resolution on 3D-OVS.The reported speedup increases with image resolution, and most computation is allocated to the decoder rather than rendering.
  • On 3D-OVS, LangSplat outperforms 2D and 3D methods, including LERF and 3D-OVS, with an overall mIoU of 93.4%.The method generates object masks from the query text alone, whereas 3D-OVS requires the complete category list.

5. Conclusion

LangSplat constructs precise and efficient 3D language fields by combining language-enhanced 3D Gaussian Splatting, a scene-specific language autoencoder, and SAM-defined semantic hierarchy. Experiments report improved open-ended 3D language query performance and substantial speed gains over LERF.

  • LangSplat enables precise and efficient open-vocabulary querying within 3D spaces.
  • LangSplat extends 3D Gaussian Splatting with language features and learns a scene-specific language autoencoder to avoid NeRF-based rendering costs.
  • SAM-defined semantic hierarchy resolves the point ambiguity problem in 3D language fields.
  • 199 × speed improvement and enhanced open-ended 3D language query performance are reported relative to LERF.

A. Video Demo

The video demo visualizes LangSplat’s scene-specific latent-space language fields and contrasts them with existing blurred representations. It also reports speedups that increase at higher image resolution.

  • LangSplat can visualize encoded 3-dimensional latent features directly as RGB channels for color consistency across frames.Decoded features are instead visualized with PCA for a fair comparison; the direct visualization benefit is stated for the encoded latent space.
  • LangSplat’s video demo presents 3D-consistent, distinctly shaped language representations compared with existing blurred-boundary fields.The demo is recommended for observing the learned language fields in the scene-specific latent space.
  • 119 × speedup at 988 × 731 increases to 199 × at 1440 × 1080 compared with LERF.

B. More Implementation Details

Implementation details describe SAM-based relevancy-map selection and document a visualization caveat. The supplied table captions identify quantitative comparisons and autoencoder-dimension ablations, while the procedure thresholds and ranks semantic maps.

  • For each text query, LangSplat produces three relevancy maps corresponding to SAM-defined semantic levels.Different strategies select the best semantic level for different tasks.
  • The method smooths relevancy maps with a mean convolution filter of size 20 before selecting the semantic level.
  • PCA visualization colors are not guaranteed to remain consistent across different frames.
  • Table 6 reports accuracy scores for 3D semantic segmentation, while Table 7 ablates latent dimension d on the 3D-OVS bench scene at 1440 × 1080.
  • A 0.4 threshold converts relevancy maps into binary masks, and the map with the highest average mask-region response becomes the prediction.

C. More Quantitative Results

On the 3D-OVS dataset, LangSplat is compared with other state-of-the-art methods using the Accuracy metric for 3D semantic segmentation. The authors report consistent outperformance.

  • LangSplat consistently outperforms other state-of-the-art methods on 3D-OVS semantic segmentation using Accuracy.Accuracy is additionally employed on 3D-OVS alongside mIoU.
  • The 3D-OVS comparison uses accuracy scores reported as percentages.

D. More Ablation Study

The ablation study examines latent dimension choices for the scene-specific autoencoder and explains the selected setting. Increasing the latent dimension improves segmentation performance with only a slight time increase, while d = 3 enables direct visualization.

  • Latent Dimension Ablation: Increasing latent dimension d improves mIoU on the 3D-OVS bench scene with only a slight increase in time cost.The comparison is reported in Table 7.
  • Evaluation Clarification: A correction confirms that the mAP results reported in 3D-OVS were actually Accuracy results.The correction was made after checking with the authors of 3D-OVS.
  • Latent Dimension Ablation: The authors chose d = 3 because three-dimensional latent features can be visualized directly as RGB channels.This choice supports visualization of the learned language field in the latent space.

E. More Visualization Results

Additional visualizations compare LangSplat with prior methods across open-vocabulary localization and semantic segmentation scenes. They show more focused localization and more precise object shapes and boundaries for LangSplat in the reported examples.

  • 3D Object Localization on LERF: On LERF localization examples, LangSplat succeeds on queries where LERF fails and concentrates activations more precisely when both identify the object.Examples include “red apple,” “plate,” “waldo,” and “chopsticks.”
  • 3D Semantic Segmentation on LERF: On LERF segmentation examples, LangSplat depicts queried object shapes accurately, whereas LERF produces imprecise shapes and substantial noise.The comparison is presented in Figure 7.
  • 3D Semantic Segmentation on 3D-OVS: Figures 8–11 provide additional qualitative segmentation comparisons on blue sofa, snacks, office desk, and room scenes from 3D-OVS.The supplied visualizations cover four named 3D-OVS scenes.
  • 3D Object Localization on LERF: The localization visualizations include queried objects such as a Rubik’s cube, Waldo, spatula, pink ice cream, green apple, and a toy chair.These labels accompany the RGB, ground-truth, LERF, and Ours comparisons.
  • 3D Semantic Segmentation on 3D-OVS: The 3D-OVS segmentation visualizations include Coke Cola, Glico Pocky, chocolate biscuits, sticks, and box queries.The supplied panels are arranged with RGB, ground truth, 3D-OVS, and Ours labels.
Loading 2312.16084v2…