PaperlayerView exact revision on arXiv

Source-linked AI summary

SceneSplat: Gaussian Splatting-based Scene Understanding with Vision-Language Pretraining

Yue Li, Qi Ma, Runyi Yang, Huapeng Li, Mengjiao Ma, Bin Ren, Nikola Popovic, Nicu Sebe, Ender Konukoglu, Theo Gevers, Luc Van Gool, Martin R. Oswald, Danda Pani Paudel

arXiv:2503.18052v2cs.CV

TL;DR

Open-vocabulary 3D understanding lacks sufficient 3D-text data and methods that learn semantics directly from 3D without relying on 2D or textual modalities. SceneSplat introduces a native 3DGS model, self-supervised learning, and a 7,916-scene dataset; it reports state-of-the-art zero-shot semantic segmentation on three fine-grained benchmarks.

  • Problem

    Open-vocabulary 3D understanding lacks large-scale 3D-text data, and current methods rely on 2D or textual modalities during training or inference.

    Source: §1 · p. 2

  • Method

    SceneSplat learns per-primitive semantic features from Gaussian parameters, with GaussSSL pretraining on unlabeled scenes and SceneSplat-7K as its indoor dataset.

    Source: §1 · p. 2

  • Results

    SceneSplat achieves state-of-the-art zero-shot semantic segmentation performance on three fine-grained benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work establishes new benchmarks and lays a foundation for future open-vocabulary 3D understanding.

    Source: §6 · p. 8

    Dataset quality depends on source data; original annotations constrain downstream tasks, and temporal inconsistency in language labels can pollute vision-language pretraining.

    Source: E. Limitations · p. 18

Abstract

from arXiv · show

Recognizing arbitrary or previously unseen categories is essential for comprehensive real-world 3D scene understanding. Currently, all existing methods rely on 2D or textual modalities during training or together at inference. This highlights the clear absence of a model capable of processing 3D data alone for learning semantics end-to-end, along with the necessary data to train such a model. Meanwhile, 3D Gaussian Splatting (3DGS) has emerged as the de facto standard for 3D scene representation across various vision tasks. However, effectively integrating semantic reasoning into 3DGS in a generalizable manner remains an open challenge. To address these limitations, we introduce SceneSplat, to our knowledge the first large-scale 3D indoor scene understanding approach that operates natively on 3DGS. Furthermore, we propose a self-supervised learning scheme that unlocks rich 3D feature learning from unlabeled scenes. To power the proposed methods, we introduce SceneSplat-7K, the first large-scale 3DGS dataset for indoor scenes, comprising 7916 scenes derived from seven established datasets, such as ScanNet and Matterport3D. Generating SceneSplat-7K required computational resources equivalent to 150 GPU days on an L4 GPU, enabling standardized benchmarking for 3DGS-based reasoning for indoor scenes. Our exhaustive experiments on SceneSplat-7K demonstrate the significant benefit of the proposed method over the established baselines.

1. Introduction

SceneSplat addresses the lack of large-scale 3DGS data and generalizable semantic learning directly on 3D by introducing a native 3DGS model, self-supervised training, and SceneSplat-7K. It reports state-of-the-art zero-shot semantic segmentation on three fine-grained benchmarks.

1. Introduction

  • Existing approaches rely on 2D or textual modalities during training or inference, while large-scale 3D-text data and end-to-end 3D semantic learning remain absent.The paper frames generalizable semantic reasoning directly on 3DGS as an open challenge.Source: §1 · p. 2
  • SceneSplat-7K supplies about 7K indoor 3DGS scenes to support large-scale scene understanding research.The dataset is curated from established indoor datasets.Source: §1 · p. 2
  • Its 3DGS encoder maps Gaussian parameters to per-primitive semantic features in one pass, without explicit 2D fusion at runtime.Vision-language supervision trains the encoder to produce CLIP-aligned embeddings.Source: §1 · p. 2
  • GaussSSL combines masked Gaussian modeling, self-distillation, and optional Language-Gaussian Feature Alignment to learn from unlabeled scenes.Its stated aim is to separate high-level semantic signals from raw Gaussian parameters.Source: §1 · p. 2
  • SceneSplat achieves state-of-the-art zero-shot semantic segmentation on three fine-grained benchmarks.The paper also reports annotation-free self-supervised training mechanisms as effective for downstream indoor segmentation.Source: §1 · p. 2

2. Related Work

Prior indoor datasets and open-vocabulary methods provide relevant foundations, but the cited work identifies a lack of large-scale indoor scene data for 3DGS and continued reliance on 2D foundation-model preprocessing. SceneSplat positions its dataset and 3D pretraining against these gaps.

3D Indoor Datasets

  • Existing indoor datasets lack large-scale support for 3DGS scene understanding, motivating a dataset built from seven widely used indoor sources.SceneSplat-7K reports 7,916 scenes and 11.27 billion 3DGS.Source: §2 · p. 2, Table 1 · p. 3

Open Vocabulary Scene Understanding

  • Prior NeRF- and 3DGS-based open-vocabulary methods use 2D foundation models, while the proposed pipeline targets feed-forward 3DGS understanding.The cited related work notes time-consuming image preprocessing with 2D foundation models.Source: §2 · p. 3

3D Representation Learning

  • 3D representation learning spans specialized architectures and self-supervised methods, including masked reconstruction adapted to point clouds, meshes, voxels, and Gaussians.The paper proposes Gaussian masking and self-distillation for 3DGS pretraining.Source: §2 · p. 3

3. SceneSplat Dataset

SceneSplat-7K is a curated collection of indoor Gaussian-splat scenes intended to support generalizable 3DGS scene understanding. Its processing filters source views and optimized scenes, and the resulting dataset contains 7,916 scenes, with 4,114 selected for pretraining.

3. SceneSplat Dataset

  • SceneSplat-7K collects about 7,000 real and synthetic indoor scenes to facilitate generalizable 3DGS scene understanding.It is presented as a carefully curated dataset of Gaussian-splat scenes.Source: §3 · p. 3

3.1. Data Processing

  • Scenes require at least 400 training frames; blurry frames are removed, and optimized scenes are filtered by PSNR before pretraining.Optimization uses gsplat, available depth loss, Markov Chain Monte Carlo compression, and opacity and scale regularization.Source: §3.1 · p. 4

3.2. Data Statistic

  • 7,916 processed scenes contain 11.27 billion Gaussians; 4,114 high-quality scenes are selected for pretraining.The dataset averages 1.42 million Gaussians per scene and covers 4.72 million RGB frames; construction required 150 NVIDIA L4 GPU-days.Source: §3.2 · p. 4
  • Average reconstruction statistics are 29.64 dB PSNR, 0.035 m depth loss, 0.897 SSIM, and 0.212 LPIPS.These figures summarize the dataset's reported reconstruction quality.Source: §3.2 · p. 4

4. Methodology

SceneSplat’s methodology pairs Gaussian-level language supervision with self-supervised learning to support open-vocabulary understanding and feature learning from unlabeled 3DGS scenes.

4. Methodology

  • The methodology first collects primitive-level language labels, then trains SceneSplat to predict vision-language features from Gaussian parameters and surroundings alongside multi-objective self-supervision.The self-supervised framework integrates reconstruction and self-distillation alignment.Source: §4 · p. 4

4.1. 3DGS Language Label Collection

  • 3DGS Language Label Collection segments views with SAMv2, extracts SigLIP2 features, and lifts feature maps to Gaussian-feature pairs using Occam’s LGS.Rather than aligning Gaussians with text embeddings or visual captions, it aligns them with VLM image embeddings, preserving richer latent semantics and avoiding scene-specific compression.Source: §4.1 · p. 4

4.2. Vision-Language 3DGS Pretraining

  • The vision-language learner predicts per-Gaussian language features using cosine, L2, and class-pooled contrastive losses, with contrastive training added after warm-starting.The resulting features support downstream open-vocabulary understanding without additional fine-tuning or 2D input.Source: §4.2 · pp. 4–5

4.3. Self Supervised Pretraining

  • GaussSSL combines masked-primitive reconstruction, augmentation-invariant student–teacher distillation, and compressed language-feature alignment for scenes with labels.Masked Gaussian Modeling embeds and masks sampled tokens, then reconstructs the masked Gaussians; language alignment predicts low-dimensional features from unmasked neighbors.Source: §4.3 · pp. 5–6

5. Experiments

The experiments evaluate vision-language pretraining for zero-shot open-vocabulary segmentation and GaussianSSL pretraining for downstream indoor segmentation. SceneSplat performs strongly across fine-grained benchmarks, while results also highlight dataset quality and inference-design trade-offs.

5.1. Vision-Language Pretraining

  • With ScanNet training, SceneSplat increases f-mIoU by 5.9% on ScanNet200 and 2.2% on Matterport3D.The reported comparisons are for zero-shot segmentation; Table 2 covers ScanNet++, Matterport3D, and ScanNet200.Source: §5.1 · p. 6, Table 2 · p. 7
  • With extended training sources, SceneSplat gains 5.7% f-mIoU on ScanNet200, 0.7% on Matterport3D, and 10.4% on ScanNet++ over concurrent work [21].The comparison reports that [21] uses 8.32× as many training scenes for its best results.Source: §5.1 · p. 6
  • SceneSplat produces competitive zero-shot segmentations, annotates some regions lacking ground-truth labels, and localizes text-query objects in 3DGS scenes.The examples include desks in Figure 3 and Robot Arm, Box, and Keyboard queries in Figure 4.Source: §5.1 · p. 6, Figure 3 · p. 6, Figure 4 · p. 6

5.2. Label-free 3DGS Pretraining

  • GaussianSSL improves over supervised-only baselines by 0.1% on ScanNet20 and 0.5% on ScanNet200, but performance drops on ScanNet++ amid pretraining-data quality variation.Against the reproduced PTv3 implementation, it improves by 0.8% on ScanNet20 and 0.9% on ScanNet200.Source: §5.2 · pp. 6–7

SceneSplat Inference Results vs. Collected Language

  • On ScanNet++, SceneSplat inference features exceed collected language labels by 4.2% f-mIoU; on Matterport3D, higher training-scene PSNR trends with higher zero-shot mIoU.The PSNR trend is reported on the 370-scene Matterport3D test split.Source: SceneSplat Inference Results vs.… · p. 7, Table 5 · p. 7, Figure 5 · p. 8
  • Majority voting aggregates predictions from neighboring Gaussians because Gaussian centers differ from evaluation point locations; 25 neighbors balance performance and inference speed.Overall mIoU increases as the number of nearest neighbors increases in the ScanNet++ validation ablation.Source: SceneSplat Inference Results vs.… · p. 7, Figure 6 · p. 8
  • 3DGS-parameter pretraining consistently outperforms point-property inputs, and applying contrastive loss later in training performs better than other tested variants.SceneSplat is also reported as 445.8× faster than the state-of-the-art language-embedded Gaussian Splatting method.Source: SceneSplat Inference Results vs.… · p. 8, Table 6 · p. 8, Table 7 · p. 8

6. Conclusion

SceneSplat operates directly on indoor 3D Gaussian splats, with SceneSplat-7K and scene-level self-supervised pretraining supporting open-vocabulary understanding. The paper reports state-of-the-art zero-shot segmentation and presents the work as a foundation for further open-vocabulary 3D understanding.

6. Conclusion

  • SceneSplat combines a single-pass semantic-feature encoder for 3D Gaussian splats with SceneSplat-7K and label-free scene-level pretraining to support open-vocabulary recognition.The dataset comprises 7,916 scenes, and the approach is reported to achieve state-of-the-art zero-shot semantic segmentation.Source: §6 · p. 8

A. Implementation Details

SceneSplat combines vision-language supervision and self-supervised learning to produce semantic features for 3D Gaussian scenes. Its implementation covers feature collection, pretraining, compression, language alignment, and model architecture.

A.1. Language Label Collection

  • Language labels align Gaussians directly with VLM image embeddings, using dynamically weighted global, background-inclusive local, and object-focused features.The weights adapt to whether a segment is integrated with its surroundings or isolated.Source: §A.1 · p. 10

A.2. Vision-Language Pretraining

  • Vision-language pretraining uses a transformer encoder-decoder on Gaussian parameters, trained for 800 epochs with extensive 3D augmentations.Training uses four H100 GPUs and a OneCycle scheduler with cosine annealing.Source: §A.2 · pp. 10–11

A.3. Masked Gaussian Modeling

  • Masked Gaussian Modeling reconstructs masked Gaussian parameters with L2 loss and attribute-specific output activations that enforce physical constraints.The reconstruction loss applies only to masked regions.Source: §A.3 · p. 11

A.4. Self-Distillation Representation Learning

  • Self-distillation combines DINO loss on pooled student and EMA-teacher features with masked iBOT alignment, using pairwise cosine similarity.Only masked regions contribute to iBOT loss; when MGM is enabled, one less-augmented global view contributes to both losses.Source: §A.4 · p. 11

A.5. Autoencoder for Feature Compression

  • A scene-specific autoencoder compresses precomputed SigLIP features into compact codes and reconstructs them, training on 3D Gaussian features rather than 2D image-level data.Its default encoder and symmetric decoder each have five layers.Source: §A.5 · pp. 11–12

A.6. Gaussian Language Alignment.

  • Gaussian language alignment applies compressed-feature alignment loss only to masked inputs and weights MGM, DINO, iBOT, and alignment objectives equally.Each loss weight is 1.0.Source: §A.6 · p. 12

A.7. Model Architecture

  • The 3D backbone uses Point Transformer with Flash Attention and sparse convolutions to improve computational efficiency while preserving geometric detail and limiting memory overhead.Source: §A.7 · p. 12

B.1. SceneSplat Zero-shot Segmentation

  • Zero-shot segmentation outperforms fully supervised segmentation for 18 ScanNet++ object classes, predominantly small objects.The compared classes include headphones, printers, pots, and clothes hangers.Source: §B.1 · p. 12

B.2. 3DGS Self-Supervised Pretraining

The section evaluates feature compression, masked Gaussian modeling, and self-distillation for 3D scene pretraining. Findings include improved compression from 3D training, sensitivity to masking choices, and dataset-dependent cross-domain classification results.

Autoencoder ablation

  • 3D-trained autoencoders perform better than 2D-trained alternatives, while increasing depth from five to six layers adds only marginal improvement.Hierarchical compression improves mIoU by +0.5% and mAcc by +0.7%.Source: §B.2 · pp. 13–14

ScanNet20 consistency issue

  • ScanNet20 zero-shot segmentation performs poorly because temporal inconsistency in SAMv2 and SigLIP2 feature collection corrupts Gaussian feature pairs.The inconsistency particularly affects large background objects such as walls and floors.Source: Table D · p. 14, §B.2 · p. 14

Language feature compression

  • The 64-dimensional latent space consistently outperforms the 16-dimensional counterpart, but the adopted 16-dimensional configuration balances efficiency and accuracy.Source: §B.2 · p. 14

Pretraining visualizations

  • Masked Gaussian Modeling reconstructs masked Gaussian parameters and predicts semantically meaningful language-aligned features; the visualizations show inputs, masked scenes, reconstructions, and feature PCA.Source: §B.2 · p. 14, Figure D · p. 16

Masked Gaussian Modeling ablation

  • A 0.4 masking ratio lowers mIoU by 0.9% versus 0.6, and a 0.15m grid lowers it by 0.5% versus 0.05m.Source: §B.2 · p. 14

Zero-shot segmentation visualizations

  • SceneSplat’s qualitative ScanNet++ results show scene segmentation and regions where the method can help annotate missing ground-truth labels.Source: Figure C · p. 15

Cross-domain classification

  • MGM achieves the best 80% accuracy on ModelNet10, while MGM alone peaks at 50% on OmniObject3D; progressively adding DINO, iBOT, and language alignment degrades classification accuracy.The passage attributes the degradation to domain gaps between distinct objects and scene-level semantics.Source: §B.2 · p. 16

C.1. Analysis

The analysis describes how input capture and scene conditions affect Gaussian Splatting reconstructions. SceneSplat-7K visualizations report an average PSNR of 28.17 dB across sources.

C.1. Analysis

  • Gaussian Splatting quality depends on dataset quality and input conditions, with fewer than 400 RGB frames often leaving indoor scenes incompletely represented.Source: §C.1 · p. 17
  • Limited filming angles omit perspectives; motion blur, unstable cameras, lighting changes, and transparent objects can produce incomplete structures, artifacts, blur, or lost surface detail.Source: §C.1 · pp. 17–18

C.2. Visualization

  • SceneSplat-7K achieves an average PSNR of 28.17 dB across its sources, and the section presents visualizations of photorealistic appearance rendering.Source: §C.2 · p. 18

D. Dataset License

SceneSplat-7K draws on indoor datasets with different licenses. Its distribution plan follows the terms of each source dataset.

D. Dataset License

  • The dataset sources carry distinct licenses, including the Apple Open Source License, ScanNet Terms of Use, and CC BY-SA 3.0.Source: D. Dataset License · p. 18
  • Data permitted for redistribution is planned for release on Hugging Face under the respective source terms.Source: D. Dataset License · p. 18
  • Sources requiring special permission will be co-hosted only after approval or hosted by the original dataset team.Source: D. Dataset License · p. 18

E. Limitations

SceneSplat-7K’s quality and downstream use are constrained by its source datasets, annotation coverage, and language-labeling process.

E. Limitations

  • Low-resolution or blurry source images can cause floating artifacts in the dataset.Dataset quality is largely influenced by the source indoor datasets.Source: E. Limitations · p. 18
  • Downstream tasks are limited by the annotations available in the original datasets.Source: E. Limitations · p. 18
  • Temporal inconsistency in the current language-labeling process can pollute vision-language pretraining.Source: E. Limitations · p. 18
  • The authors plan to add bounding boxes and language descriptions in future work.Source: E. Limitations · p. 18

F. Impact Statement

The impact statement presents SceneSplat-7K as a benchmark resource and SceneSplat as a vision-language pretraining framework for 3DGS. It reports improved 3D semantic segmentation and describes potential applications in autonomous systems and augmented reality.

F. Impact Statement

  • The framework combines knowledge distillation from 2D foundation models, contrastive learning, and masked Gaussian modeling, and significantly outperforms existing methods in 3D semantic segmentation.The authors say the dataset, model, and code are publicly released to facilitate further development of generalizable 3D scene understanding solutions.Source: F. Impact Statement · p. 18
  • SceneSplat-7K provides over 7K annotated scenes for standardized benchmarking of 3DGS-based reasoning in vision tasks.Source: F. Impact Statement · p. 18
  • The authors identify autonomous systems and augmented reality as areas with broad implications for scalable 3D scene understanding.Source: F. Impact Statement · p. 18
  • The reported 3DGS renderings have PSNRs of 45.55 dB for Replica Office0, 41.57 dB for Replica Room2, and 34.43 dB for 3RScan.Source: F. Impact Statement · p. 20

Related papers

Loading 2503.18052v2…