Rethinking Open-Vocabulary Segmentation of Radiance Fields in 3D Space

Problem setting

• The proposed setting segments 3D volumes rather than rendered 2D masks, targeting more complete semantic understanding of radiance fields.Previous methods produce masks for given viewpoints, whereas this formulation queries the semantics of 3D points.Source: §1 · p. 1, Figure 1 · p. 1

Experiments

• Experiments assess 3D and rendered 2D accuracy, training and rendering time, and consistency across viewpoints.These dimensions define the paper’s experimental comparison of its revised setting and methods.Source: §1 · p. 2

Supervision

• Directly supervising 3D points learns language embeddings in 3D space and achieves state-of-the-art accuracy in both 3D and rendered 2D segmentation.The approach changes the loss dimension from rendered embeddings to point embeddings, while removing multi-scale embeddings through masked-object encoding for multi-view consistency.Source: §1 · pp. 1–2

3DGS transfer

• Transferring the learned language field to 3DGS enables the first real-time rendering among open-vocabulary methods, at 28× the speed of the previous fastest method.This transfer avoids the memory and training-time trade-offs associated with directly storing or rendering high-dimensional language embeddings in 3DGS.Source: §1 · p. 2

Evaluation

• The proposed evaluation protocol measures agreement between estimated and ground-truth volumes represented as meshes, using F1-score for geometry and semantics together.The protocol applies to both NeRF and 3DGS and addresses evaluations that either use rendered pixels or ignore reconstructed geometry.Source: §1 · p. 2

Neural 3D representations

• NeRF represents scenes as continuous point-to-radiance and density functions, whereas 3DGS projects and alpha-blends depth-sorted Gaussians for fast rendering.These representations provide the neural scene backbones used by semantic understanding methods.Source: §2 · p. 2

Open-vocabulary radiance fields

• Closed-set semantic branches synthesize novel-view masks but require predefined classes, limiting open-vocabulary queries without retraining or additional models.VLM features such as CLIP broaden radiance-field understanding to open-set semantics through feature distillation.Source: §2 · p. 2
• LERF, FMGS, LEGaussians, OpenNeRF, and related methods distill multi-scale or pixel-level CLIP features, often using SAM-derived regions to improve object separation.Multi-scale approaches can select different scales across views for the same query, producing multi-view inconsistency; this paper uses SAM to remove scale dependence.Source: §2 · pp. 2–3

Paper’s approach

• The paper uses a point-wise semantic loss to supervise the language embeddings of sampled points directly in 3D space, ensuring precise semantics.Source: Figure 3 · p. 3

Language-embedded 3DGS

• Language-embedded 3DGS methods compress CLIP features or use global memory, trading accuracy or optimization speed against the cost of high-dimensional Gaussian embeddings.LEGaussians and LangSplat use scene-specific autoencoders with performance degradation, while uncompressed alternatives suffer longer training times.Source: §2 · p. 3

3.2 Task Redefinition

• 3D segmentation is redefined as computing relevancy scores on queried 3D points rather than rendered 2D images, with thresholded regions treated as objects.NeRF queries points along rays, whereas 3DGS queries Gaussian center positions.Source: §3.2 · p. 3

3.3 Supervising Semantics in 3D Space

• Point-wise semantic loss directly matches 3D-point embeddings to ground-truth CLIP embeddings, improving both 3D and rendered 2D segmentation.The same point-wise strategy is also applied to DINO regularization.Source: §3.3 · p. 4
• Scale-free CLIP embeddings created from tightly cropped SAM masks avoid selecting view-dependent optimal scales and promote view-consistent language fields.The approach removes background before cropping to the object masks.Source: §3.3 · p. 4

3.4 Transferring Language Field into 3DGS

• The learned language field is baked into 3DGS by querying embeddings at Gaussian centers, freezing them, and enabling 22ms transfer while retaining fast rendering.3DGS centers are initialized from the learned radiance field and kept fixed to align geometry with the language field.Source: §3.4 · p. 4, Figure 4 · p. 4
• The 3DGS pipeline alpha-blends high-dimensional language features from overlapping Gaussians and uses the resulting embeddings for 3D segmentation.The alpha value combines Gaussian opacity with the Gaussian kernel.Source: §3.4 · p. 4

3.5 3D Semantic Evaluation Protocol

• The evaluation protocol extracts meshes from segmented reconstructed points and measures F1-score against ground-truth meshes, jointly assessing semantic coverage and geometry.Precision measures correct estimated volume, recall measures covered ground-truth volume, and regular-grid points approximate volume comparisons.Source: §3.5 · pp. 4–5, Figure 5 · p. 4

4.1 Segmentation Accuracy

• Both Ours-NeRF and Ours-3DGS surpass competitors on 3D segmentation accuracy on Replica and achieve the highest rendered 2D performance on LERF and 3D-OVS.Qualitatively, they produce clear boundaries and localize complex or multiple objects that competitors sometimes miss or confuse.Source: §4.1 · p. 6, Table 2 · p. 6
• Scale-free embeddings provide view-consistent relevancy maps, whereas LERF and LangSplat vary across viewpoints because their optimal scales change.The comparison uses the napkin query and reports the competing methods’ changing optimal scales.Source: §4.1 · p. 6, Figure 8 · p. 6

4.2 Computational Time

• Ours-3DGS achieves real-time language-embedding rendering, reported as 28× faster than LEGaussians on the waldo kitchen scene.The computational comparison uses a single RTX A5000 and includes training and rendering costs.Source: §4.2 · pp. 6–7, Table 3 · p. 7

4.3 Ablation Study

• Point-wise semantic loss improves 3D understanding by removing depth-ambiguity floaters and raising F1-score by +0.0995 over 2D supervision.The ablation also reports an indirect improvement in rendered 2D segmentation accuracy.Source: §4.3 · p. 7, Table 5 · p. 7
• Scale-free embeddings improve 3D and rendered 2D segmentation with clearer object boundaries than multi-scale embeddings.The qualitative comparison shows improved coverage of the toy elephant’s head.Source: §4.3 · p. 7, Figure 9 · p. 7, Table 5 · p. 7
• NeRF initialization aligns geometry and semantics for 3DGS transfer, while SfM initialization degrades segmentation performance.NeRF initialization also yields 0.50× as many Gaussians, 0.38× training time, and 1.70× faster rendering.Source: §4.3 · p. 7, Table 4 · p. 7

• The paper reformulates the task to produce 3D segmented volumes instead of rendered 2D masks and proposes a 3D evaluation protocol.Source: §5 · p. 7
• Directly computing loss on 3D points achieves state-of-the-art segmentation accuracy in both 3D and rendered 2D.Source: §5 · p. 7
• Transferring the learned language field to 3DGS enables the first real-time rendering speed among open-vocabulary methods.Source: §5 · p. 7

A.1 Implementation Details

• NeRF and its CLIP branch are trained jointly from scratch, using OpenCLIP ViT-B/16 features and automatic ViT-H SAM masks.Source: §A.1 · p. 10
• 3DGS optimization freezes Gaussian centers, sets the rotation learning rate to 1e-4, and runs 10K iterations without densification.Source: §A.1 · p. 10
• All models and competitors are trained with a single RTX A5000.Source: §A.1 · p. 10
• The 3D-OVS evaluation view exhibits poor reconstruction quality.Source: Figure 5 · p. 10

A.2 Details of Datasets

• The evaluation fixes dataset-specific thresholds detailed in Table 9.Source: §A.2 · p. 10
• Replica is a high-quality indoor 3D mesh dataset with semantic labels for 3D segmentation using class names as text queries.Source: §A.2 · p. 10
• LERF contains real-world iPhone scenes with text queries and 2D bounding boxes, but query-dependent masks make comparisons difficult.Source: §A.2 · p. 10
• LERF evaluation uses SAM-generated masks that were manually corrected across four scenes and three non-blurry views.Source: §A.2 · p. 10
• 3D-OVS provides real-world scenes with text queries and pixel-level labels, but results are reported only for four scenes with successful reconstruction.Source: §A.2 · p. 10
• ScanNet supplies RGB-D scans, meshes, and instance-level labels, with 3D segmentation reported for scene0003 in the appendix.Source: §A.2 · p. 10

B.1 Splatting High Dimensions Embeddings on 3DGS

• 512-dimensional language embeddings require 522 KB of shared memory per streaming multiprocessor, exceeding the H100 maximum of 228 KB.Source: §B.1 · pp. 10–11
• Existing strategies trade segmentation quality, training speed, memory use, or rendering speed to accommodate high-dimensional features.Source: §B.1 · p. 11
• Feature compression accelerates training 47× but reduces mIoU by 7.30 and requires an additional decoding process.Source: §B.1 · p. 11, Table 11 · p. 11
• Ours-3DGS achieves fast training without feature compression.Source: §B.1 · p. 11

B.2 Post-process of LangSplat

• Without post-processing, LangSplat renders at 55 fps, 550× faster, but its mIoU decreases by 5.5.Source: §B.2 · p. 11, Table 12 · p. 11
• Ours-3DGS renders faster than LangSplat even without LangSplat’s post-processing.Source: §B.2 · p. 11
• Post-processing significantly improves LangSplat performance by reducing floaters.Source: Figure 6 · p. 11

C More Results

• 3D segmentation accuracy is additionally evaluated on ScanNet scene0003 and reported in Table 13.Source: C More Results · p. 11, Table 13 · p. 12
• Per-scene quantitative results are reported for Replica, LERF, and 3D-OVS in Tables 14–16.Source: C More Results · p. 11, Table 14 · p. 12, Table 15 · p. 12
• The post-processing cost consists of mean convolution on relevancy maps and binary masks across three scales, totaling 10473 ms.Source: C More Results · p. 11