OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding

• 3DGS supports fast training, real-time rendering, and explicit point-based representations across reconstruction, generation, and understanding tasks.Source: §1 · p. 1
• Existing language-augmented 3DGS methods primarily render language attributes for 2D pixel-level understanding rather than 3D point-level interaction.Source: §1 · pp. 1–2
• 3D point-level matching remains difficult because existing features lack object discrimination and within-object consistency.Source: §1 · p. 2
• OpenGaussian trains distinctive, 3D-consistent instance features using SAM masks, without requiring cross-frame associations.Source: §1 · p. 2
• A two-level coarse-to-fine codebook discretizes instance features, while instance-level 2D–3D association links multiple-view CLIP features to 3D instances.Source: §1 · pp. 2–3
• The method evaluates 3D point-level object selection, point-cloud understanding, click-based object selection, and module ablations.Source: §1 · p. 3

• NeRF-based methods improve novel-view synthesis but often suffer from slow training and rendering, motivating explicit representations such as voxels, hash grids, and point clouds.Source: §2.1 · p. 3
• Open-vocabulary 3D scene-understanding methods commonly align 2D vision-language features with 3D point clouds or project 3D data into 2D for zero-shot learning.Source: §2.2 · p. 3
• SAM, CLIP, and DINO have motivated semantic feature integration into NeRF for 3D segmentation, understanding, and editing.Source: §2.2 · p. 3
• Prior 3DGS methods attach semantic or language attributes to Gaussians and render feature maps, including uncertainty-aware, autoencoded, and direction-aware representations.Source: §2.2 · p. 3

3.1 3D Consistency-Preserving Instance Feature Learning

• OpenGaussian augments each 3D Gaussian with instance features and trains them using view-independent SAM masks, without cross-view correlation.The rendered features are constrained to remain consistent across views while distinguishing different objects.Source: §3.1 · p. 4
• Intra-mask smoothing brings features within each SAM mask toward their mean, while inter-mask contrastive loss increases separation between different instances.Together, these losses produce cross-view-consistent and distinctive instance features directly from binary masks.Source: §3.1 · p. 4

3.2 Two-Level Codebook for Discretization

• Codebook discretization converts continuous instance features into quantized features and indices, making same-instance Gaussian features identical for object selection.The process initializes a codebook, assigns each Gaussian an index, optimizes features through quantized rendering, and updates the codebook iteratively.Source: §3.2 · p. 5
• The two-level codebook first clusters concatenated instance features and 3D coordinates, then refines each coarse cluster using instance features alone.Position-dependent coarse clustering addresses distant non-co-visible objects and partitions large scenes for finer optimization.Source: §3.2 · p. 5
• The coarse codebook uses Gaussian positions only during codebook construction, preserving the pre-trained model’s geometric structure during optimization.The resulting coarse and fine discretizations are visualized as progressively refined point features.Source: §3.2 · p. 5, Figure 3 · p. 5

3.3 Instance-Level 2D-3D Association without Depth Test

• The association module renders each 3D instance, matches it to SAM masks using IoU and feature similarity, and transfers the selected mask’s CLIP features to its Gaussians.This instance-level association retains high-dimensional linguistic features and avoids depth-based occlusion testing.Source: §3.3 · p. 6, Figure 4 · p. 6

4.1 Open-Vocabulary Object Selection in 3D Space

• OpenGaussian associates each Gaussian with a 512-dimensional CLIP feature for text-query-based 3D object selection.Selected points are rendered into multi-view images for evaluation against ground-truth object masks.Source: §4.1 · p. 7
• Comparison methods struggle with 3D point-level matching because compressed features lose discrimination and alpha blending weakens 2D–3D correspondence.OpenGaussian addresses these issues with distinctive features, two-level codebook discretization, and lossless feature learning.Source: §4.1 · p. 7
• OpenGaussian produces more discriminative 3D features and more accurate target identification than comparison methods in the reported visualizations.The feature comparisons include LERF scenes and the scenes ramen, teatime, scannet_0140_00, and scannet_0645_00.Source: §4.1 · p. 8, Figure 6 · p. 8

4.2 Open-Vocabulary Point Cloud Understanding

• OpenGaussian significantly outperforms LangSplat and LEGaussians on ScanNet open-vocabulary point-cloud understanding across 19, 15, and 10 queried categories.CLIP features are used zero-shot in OpenGaussian, without training the Gaussian features with CLIP.Source: §4.2 · pp. 8–9

4.3 Click-based 3D Object Selection

• For click-based selection, OpenGaussian segments more complete 3D objects than SAGA on the LERF dataset.SAGA uses inference-time post-processing including SAM masks, statistics, region growing, and ball query.Source: §4.3 · p. 9, Figure 7 · p. 9

4.4 Ablation Study

• Adding intra-mask smoothing to inter-mask contrastive loss improves mIoU by 3.05% and mAcc by 2.76%.The ablation identifies inter-mask contrastive loss as more crucial, while intra-mask smoothing further enhances results.Source: §4.4 · p. 9, Table 3 · p. 10
• A single-layer codebook with k = 64 achieves only 28% mIoU, while increasing capacity alone substantially decreases performance.The ablation motivates the two-level codebook because spatially distant instances can otherwise be constrained together.Source: §4.4 · p. 9, Table 5 · p. 10
• Combining IoU and feature distance provides an effective strategy for associating 3D instances with 2D SAM masks.The association ablation reports that each strategy can independently achieve comparable performance.Source: §4.4 · p. 10, Table 4 · p. 10

5 Conclusion

• OpenGaussian addresses weak 3D point-level performance caused by lossy features and inconsistent 2D–3D associations.The method trains instance features with 3D consistency, discretizes them with a two-level codebook, and uses lossless instance-level CLIP associations.Source: §5 · p. 10
• The Gaussian geometric properties remain fixed, which may create inconsistencies between geometric representation and semantic content.The authors propose jointly optimizing instance features and geometric properties as future work.Source: §5 · p. 10
• The method does not address open-vocabulary 3D detection through object-size regression or dynamic scene factors.The authors identify adaptive codebook values and integration with 4DGS as future directions.Source: §5 · p. 10

Training Strategy

• Standard 3DGS is pretrained for 30,000 steps before Gaussian geometry parameters are frozen for instance-feature and codebook training.Instance features train for 10,000 steps, or 20,000 on ScanNet; the two-layer codebook trains for 30,000 steps, or 40,000 on ScanNet.Source: §A.1 · p. 14
• The 2D-3D feature association step is training-free and computed once before inference.The association process takes around 1 minute, with no further computation needed during inference.Source: §A.1 · p. 14

Training Time

• Each scene is trained on a single 32G V100 GPU, taking approximately 50 minutes for LERF and 15 minutes for ScanNet.Actual memory usage is around 16–20G; LERF uses around 200 images, while ScanNet uses 100–300 images.Source: §A.1 · p. 14

ScanNet Dataset Evaluation

• ScanNet evaluation uses 10 randomly selected scenes and 19 text-query categories defined by the dataset.The listed categories include wall, floor, cabinet, bed, chair, sofa, table, door, and other indoor objects.Source: §A.1 · p. 14

Hyperparameters

• The two-level codebook and loss weights use dataset- and scene-specific hyperparameters.For example, ScanNet uses k1 = 64 and k2 = 5, while LERF generally uses k1 = 64 and k2 = 10.Source: §A.1 · p. 14

Scene editing

• The method enables object removal, object insertion, and selected-object color modification in the reconstructed scene.These edits are performed in 3D space rather than directly on the image.Source: §A.2.1 · p. 14

Instance feature visualization

• The visualizations render 3D point instance features into multi-view images and present 3D point features across more scenarios.Source: §A.2.2 · p. 14

Outdoor scenarios

• Waymo visualizations cover six outdoor sequences and demonstrate discretization of large scenes.The figure pairs RGB images with corresponding 3D point features.Source: §A.2.3 · p. 14, Figure 13 · p. 18
• Rendering 3D features into Waymo 2D feature maps illustrates instance features learned with 3D consistency from coarse SAM supervision.The training uses SAM masks without inter-frame associations.Source: §A.2.3 · p. 14, Figure 14 · p. 18
• The two-stage codebook produces better discretization in outdoor scenes at the fine level.Source: §A.2.3 · p. 14, Figure 15 · p. 18

Real-world scenarios

• In a mobile-phone office scene, OpenGaussian’s 3D point visualization shows significant object discrimination in a real-world setting.Source: §A.2.3 · p. 14, Figure 11 · p. 16

Retrieval and applications

• Text queries retrieve relevant Gaussians by comparing text features with language features associated with 3D Gaussians.The demonstration is shown on two ScanNet scenes.Source: §A.2.4 · p. 15, Figure 9 · p. 15

Gaussian splatting representation

• The 3D Gaussian representation uses spatial means and covariance matrices, projects Gaussians into 2D, and alpha-blends sorted splats to form pixels.The projection uses camera intrinsic and extrinsic matrices, while opacity and view-dependent color contribute to rendering.Source: §A.3 · p. 19