PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding

Yanmin Wu, Jiarui Meng, Haijie Li, Chenming Wu, Yahao Shi, Xinhua Cheng, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang, Jian Zhang

arXiv:2406.02058v2cs.CVcs.RO

TL;DR

Existing 3DGS open-vocabulary methods focus mainly on 2D pixel-level parsing and struggle with 3D point-level tasks because of weak features and inaccurate 2D–3D associations. OpenGaussian learns 3D-consistent instance features, discretizes them coarsely to finely, and associates 3D points with 2D masks and CLIP features. Across object selection, point-cloud understanding, click-based selection, and ablations, the paper reports effective 3D point-level open-vocabulary understanding.

  • Problem

    Existing 3DGS open-vocabulary methods mainly support 2D pixel-level parsing, while 3D point-level understanding is limited by weak feature expressiveness and inaccurate 2D–3D associations.

  • Method

    OpenGaussian trains distinctive, 3D-consistent instance features from SAM masks, discretizes them with a two-level codebook, and associates 3D points with 2D masks and CLIP features.

  • Results

    OpenGaussian demonstrates effective performance across 3D open-vocabulary object selection, point-cloud understanding, click-based object selection, and ablation experiments.

  • Takeaways & Limitations

    The method provides open-vocabulary 3D point-level understanding while retaining the open-vocabulary capabilities of lossless CLIP features without an additional compression or quantization network.

  • Takeaways & Limitations

    The method fixes Gaussian geometry, chooses codebook sizes empirically, excludes object-size regression for 3D detection, and does not model dynamic factors.

Abstract

from arXiv · show

This paper introduces OpenGaussian, a method based on 3D Gaussian Splatting (3DGS) capable of 3D point-level open vocabulary understanding. Our primary motivation stems from observing that existing 3DGS-based open vocabulary methods mainly focus on 2D pixel-level parsing. These methods struggle with 3D point-level tasks due to weak feature expressiveness and inaccurate 2D-3D feature associations. To ensure robust feature presentation and 3D point-level understanding, we first employ SAM masks without cross-frame associations to train instance features with 3D consistency. These features exhibit both intra-object consistency and inter-object distinction. Then, we propose a two-stage codebook to discretize these features from coarse to fine levels. At the coarse level, we consider the positional information of 3D points to achieve location-based clustering, which is then refined at the fine level. Finally, we introduce an instance-level 3D-2D feature association method that links 3D points to 2D masks, which are further associated with 2D CLIP features. Extensive experiments, including open vocabulary-based 3D object selection, 3D point cloud understanding, click-based 3D object selection, and ablation studies, demonstrate the effectiveness of our proposed method. The source code is available at our project page: https://3d-aigc.github.io/OpenGaussian

1 Introduction

OpenGaussian targets 3D point-level open-vocabulary understanding, addressing the limitations of 3DGS methods designed mainly for 2D pixel-level parsing. It learns distinctive, consistent point features and connects them to CLIP features for open-vocabulary scene understanding.

  • 3DGS supports fast training, real-time rendering, and explicit point-based representations across reconstruction, generation, and understanding tasks.
  • Existing language-augmented 3DGS methods primarily render language attributes for 2D pixel-level understanding rather than 3D point-level interaction.
  • 3D point-level matching remains difficult because existing features lack object discrimination and within-object consistency.
  • OpenGaussian trains distinctive, 3D-consistent instance features using SAM masks, without requiring cross-frame associations.
  • A two-level coarse-to-fine codebook discretizes instance features, while instance-level 2D–3D association links multiple-view CLIP features to 3D instances.
  • The method evaluates 3D point-level object selection, point-cloud understanding, click-based object selection, and module ablations.

2 Related Work

Related work combines explicit 3D representations with visual-language features to improve scene understanding, but existing approaches emphasize feature alignment, 2D projection, or semantic rendering.

  • NeRF-based methods improve novel-view synthesis but often suffer from slow training and rendering, motivating explicit representations such as voxels, hash grids, and point clouds.
  • Open-vocabulary 3D scene-understanding methods commonly align 2D vision-language features with 3D point clouds or project 3D data into 2D for zero-shot learning.
  • SAM, CLIP, and DINO have motivated semantic feature integration into NeRF for 3D segmentation, understanding, and editing.
  • Prior 3DGS methods attach semantic or language attributes to Gaussians and render feature maps, including uncertainty-aware, autoencoded, and direction-aware representations.

3 Method

OpenGaussian learns 3D-consistent, distinctive instance features from SAM masks, discretizes them with a coarse-to-fine codebook, and associates 3D instances with CLIP features through 2D masks. The method targets interactive point-level understanding while preserving geometric structure and avoiding depth-based occlusion testing.

  • 3.1 3D Consistency-Preserving Instance Feature Learning: OpenGaussian augments each 3D Gaussian with instance features and trains them using view-independent SAM masks, without cross-view correlation.The rendered features are constrained to remain consistent across views while distinguishing different objects.
  • 3.1 3D Consistency-Preserving Instance Feature Learning: Intra-mask smoothing brings features within each SAM mask toward their mean, while inter-mask contrastive loss increases separation between different instances.Together, these losses produce cross-view-consistent and distinctive instance features directly from binary masks.
  • 3.2 Two-Level Codebook for Discretization: Codebook discretization converts continuous instance features into quantized features and indices, making same-instance Gaussian features identical for object selection.The process initializes a codebook, assigns each Gaussian an index, optimizes features through quantized rendering, and updates the codebook iteratively.
  • 3.2 Two-Level Codebook for Discretization: The two-level codebook first clusters concatenated instance features and 3D coordinates, then refines each coarse cluster using instance features alone.Position-dependent coarse clustering addresses distant non-co-visible objects and partitions large scenes for finer optimization.
  • 3.2 Two-Level Codebook for Discretization: The coarse codebook uses Gaussian positions only during codebook construction, preserving the pre-trained model’s geometric structure during optimization.The resulting coarse and fine discretizations are visualized as progressively refined point features.
  • 3.3 Instance-Level 2D-3D Association without Depth Test: The association module renders each 3D instance, matches it to SAM masks using IoU and feature similarity, and transfers the selected mask’s CLIP features to its Gaussians.This instance-level association retains high-dimensional linguistic features and avoids depth-based occlusion testing.

4 Experiments

OpenGaussian is evaluated on text-based 3D object selection, point-cloud understanding, click-based selection, and component ablations. Across these experiments, it produces more discriminative and complete 3D selections than comparison methods.

  • 4.1 Open-Vocabulary Object Selection in 3D Space: OpenGaussian associates each Gaussian with a 512-dimensional CLIP feature for text-query-based 3D object selection.Selected points are rendered into multi-view images for evaluation against ground-truth object masks.
  • 4.1 Open-Vocabulary Object Selection in 3D Space: Comparison methods struggle with 3D point-level matching because compressed features lose discrimination and alpha blending weakens 2D–3D correspondence.OpenGaussian addresses these issues with distinctive features, two-level codebook discretization, and lossless feature learning.
  • 4.1 Open-Vocabulary Object Selection in 3D Space: OpenGaussian produces more discriminative 3D features and more accurate target identification than comparison methods in the reported visualizations.The feature comparisons include LERF scenes and the scenes ramen, teatime, scannet_0140_00, and scannet_0645_00.
  • 4.2 Open-Vocabulary Point Cloud Understanding: OpenGaussian significantly outperforms LangSplat and LEGaussians on ScanNet open-vocabulary point-cloud understanding across 19, 15, and 10 queried categories.CLIP features are used zero-shot in OpenGaussian, without training the Gaussian features with CLIP.
  • 4.3 Click-based 3D Object Selection: For click-based selection, OpenGaussian segments more complete 3D objects than SAGA on the LERF dataset.SAGA uses inference-time post-processing including SAM masks, statistics, region growing, and ball query.
  • 4.4 Ablation Study: Adding intra-mask smoothing to inter-mask contrastive loss improves mIoU by 3.05% and mAcc by 2.76%.The ablation identifies inter-mask contrastive loss as more crucial, while intra-mask smoothing further enhances results.
  • 4.4 Ablation Study: A single-layer codebook with k = 64 achieves only 28% mIoU, while increasing capacity alone substantially decreases performance.The ablation motivates the two-level codebook because spatially distant instances can otherwise be constrained together.
  • 4.4 Ablation Study: Combining IoU and feature distance provides an effective strategy for associating 3D instances with 2D SAM masks.The association ablation reports that each strategy can independently achieve comparable performance.

5 Conclusion

OpenGaussian targets the gap between strong pixel-level open-vocabulary methods and weaker 3D point-level understanding. It combines 3D-consistent instance features, two-level discretization, and lossless instance-level 2D–3D CLIP associations, while leaving several scope limitations for future work.

  • 5 Conclusion: OpenGaussian addresses weak 3D point-level performance caused by lossy features and inconsistent 2D–3D associations.The method trains instance features with 3D consistency, discretizes them with a two-level codebook, and uses lossless instance-level CLIP associations.
  • 5 Conclusion: The Gaussian geometric properties remain fixed, which may create inconsistencies between geometric representation and semantic content.The authors propose jointly optimizing instance features and geometric properties as future work.
  • 5 Conclusion: The method does not address open-vocabulary 3D detection through object-size regression or dynamic scene factors.The authors identify adaptive codebook values and integration with 4DGS as future directions.

A.1 Implementation Details

The implementation trains scene representations and codebooks in staged optimization, with dataset-specific schedules and fixed evaluation settings. Training uses a single 32G V100 GPU, while feature association is performed once without inference-time computation.

  • Training Strategy: Standard 3DGS is pretrained for 30,000 steps before Gaussian geometry parameters are frozen for instance-feature and codebook training.Instance features train for 10,000 steps, or 20,000 on ScanNet; the two-layer codebook trains for 30,000 steps, or 40,000 on ScanNet.
  • Training Strategy: The 2D-3D feature association step is training-free and computed once before inference.The association process takes around 1 minute, with no further computation needed during inference.
  • Training Time: Each scene is trained on a single 32G V100 GPU, taking approximately 50 minutes for LERF and 15 minutes for ScanNet.Actual memory usage is around 16–20G; LERF uses around 200 images, while ScanNet uses 100–300 images.
  • ScanNet Dataset Evaluation: ScanNet evaluation uses 10 randomly selected scenes and 19 text-query categories defined by the dataset.The listed categories include wall, floor, cabinet, bed, chair, sofa, table, door, and other indoor objects.
  • Hyperparameters: The two-level codebook and loss weights use dataset- and scene-specific hyperparameters.For example, ScanNet uses k1 = 64 and k2 = 5, while LERF generally uses k1 = 64 and k2 = 10.

A.2.1 Scene editing

OpenGaussian supports direct 3D scene editing by selecting objects for removal, insertion, or color modification after reconstructing the original scene.

  • Scene editing: The method enables object removal, object insertion, and selected-object color modification in the reconstructed scene.These edits are performed in 3D space rather than directly on the image.

A.2.2 Instance feature visualization

Feature visualizations show 3D point instance features rendered into multi-view images and across additional scenarios.

  • Instance feature visualization: The visualizations render 3D point instance features into multi-view images and present 3D point features across more scenarios.

A.2.3 Qualitative results of outdoor and real-world scenarios

Qualitative evaluations extend OpenGaussian’s visualization and retrieval demonstrations to outdoor, mobile-captured, and benchmark scenes. They highlight 3D-consistent features and coarse-to-fine discretization, while the figures also illustrate the underlying Gaussian splatting representation and editing or retrieval applications.

  • Outdoor scenarios: Waymo visualizations cover six outdoor sequences and demonstrate discretization of large scenes.The figure pairs RGB images with corresponding 3D point features.
  • Outdoor scenarios: Rendering 3D features into Waymo 2D feature maps illustrates instance features learned with 3D consistency from coarse SAM supervision.The training uses SAM masks without inter-frame associations.
  • Outdoor scenarios: The two-stage codebook produces better discretization in outdoor scenes at the fine level.
  • Real-world scenarios: In a mobile-phone office scene, OpenGaussian’s 3D point visualization shows significant object discrimination in a real-world setting.
  • Retrieval and applications: Text queries retrieve relevant Gaussians by comparing text features with language features associated with 3D Gaussians.The demonstration is shown on two ScanNet scenes.
  • Gaussian splatting representation: The 3D Gaussian representation uses spatial means and covariance matrices, projects Gaussians into 2D, and alpha-blends sorted splats to form pixels.The projection uses camera intrinsic and extrinsic matrices, while opacity and view-dependent color contribute to rendering.
Loading 2406.02058v2…