PaperlayerView exact revision on arXiv

Source-linked AI summary

EmbodiedSplat: Online Feed-Forward Semantic 3DGS for Open-Vocabulary 3D Scene Understanding

Seungjun Lee, Zihan Wang, Yunsong Wang, Gim Hee Lee

arXiv:2603.04254v1cs.CV

TL;DR

Embodied tasks require immediate, online, open-vocabulary 3D scene understanding, but existing approaches face computational or optimization constraints. EmbodiedSplat addresses this with feed-forward 3DGS, compact CLIP semantics, and geometry-aware features, achieving strong benchmark performance and near-real-time processing while remaining sensitive to reconstruction and domain gaps.

  • Problem

    Embodied perception needs online, real-time, whole-scene, generalizable, open-vocabulary understanding, while existing 3DGS approaches face costly rendering or per-scene optimization.

  • Method

    EmbodiedSplat combines feed-forward 3DGS with an Online Sparse Coefficient Field, CLIP Global Codebook, and 3D geometric-aware features fused with 2D semantics.

  • Results

    EmbodiedSplat achieves the best performance across reported benchmarks while maintaining short reconstruction time and near-real-time per-frame processing with real-time 2D models.

  • Takeaways & Limitations

    The framework provides an online 3DGS-based perception model for embodied scenarios with memory-efficient semantic representations and practical processing speed.

  • Takeaways & Limitations

    EmbodiedSplat inherits feed-forward 3DGS reconstruction errors, and a model trained on ScanNet fails to outperform baselines on Replica because of the domain gap.

Abstract

from arXiv · show

Understanding a 3D scene immediately with its exploration is essential for embodied tasks, where an agent must construct and comprehend the 3D scene in an online and nearly real-time manner. In this study, we propose EmbodiedSplat, an online feed-forward 3DGS for open-vocabulary scene understanding that enables simultaneous online 3D reconstruction and 3D semantic understanding from the streaming images. Unlike existing open-vocabulary 3DGS methods which are typically restricted to either offline or per-scene optimization setting, our objectives are two-fold: 1) Reconstructs the semantic-embedded 3DGS of the entire scene from over 300 streaming images in an online manner. 2) Highly generalizable to novel scenes with feed-forward design and supports nearly real-time 3D semantic reconstruction when combined with real-time 2D models. To achieve these objectives, we propose an Online Sparse Coefficients Field with a CLIP Global Codebook where it binds the 2D CLIP embeddings to each 3D Gaussian while minimizing memory consumption and preserving the full semantic generalizability of CLIP. Furthermore, we generate 3D geometric-aware CLIP features by aggregating the partial point cloud of 3DGS through 3D U-Net to compensate the 3D geometric prior to 2D-oriented language embeddings. Extensive experiments on diverse indoor datasets, including ScanNet, ScanNet++, and Replica, demonstrate both the effectiveness and efficiency of our method. Check out our project page in https://0nandon.github.io/EmbodiedSplat/.

1. Introduction

EmbodiedSplat targets embodied perception that is online, real-time, generalizable, whole-scene, and open-vocabulary. It combines feed-forward 3DGS with compact CLIP-based semantics and geometric features for online scene understanding.

  • Embodied agents need online, real-time, generalizable, whole-scene, and open-vocabulary 3D perception during exploration.
  • Existing semantic 3DGS methods incur heavy computation through repeated feature-map rendering or rely on approaches with limited online generalization.
  • EmbodiedSplat equips pretrained feed-forward 3DGS with open-vocabulary capability for streaming-image reconstruction.
  • Its Online Sparse Coefficient Field with a CLIP Global Codebook stores per-Gaussian semantics compactly without pretraining or per-scene optimization.
  • The method combines 2D CLIP semantics with 3D geometric-aware features aggregated through a 3D U-Net to compensate for missing geometric priors.
  • The framework reports improved segmentation and reconstruction efficiency, while its faster variant reaches 5-6 FPS with real-time 2D models.

2. Related Works

Prior open-vocabulary 3D understanding methods extend point clouds, NeRFs, and 3DGS with language features. Semantic 3DGS approaches primarily use rendered feature alignment, instance clustering, or direct feature lifting.

  • Point-based methods associate 3D points with projected 2D pixels or distill 2D foundation-model features into 3D networks.
  • Early semantic 3DGS methods attach learnable features to Gaussians and train them by matching rendered feature maps with 2D image features.
  • Clustering-based methods group Gaussians into instances using 2D segmentation masks before classifying each group.
  • More recent methods pursue direct feature lifting to avoid dependence on repeated 2D feature rendering.

3. Preliminaries

3DGS represents scenes with explicit Gaussian primitives and renders pixels through weighted alpha blending. EmbodiedSplat builds online processing on pretrained FreeSplat++ by selecting past frames, encoding local Gaussians, pairing overlaps, and fusing them.

  • 3DGS models a scene as Gaussian primitives defined by means, covariances, opacity, and color.
  • Projected Gaussian parameters are alpha-blended along each ray to render pixel colors.
  • The projected 2D covariance is obtained from the 3D covariance using the world-to-camera transform and projection Jacobian.
  • FreeSplat++ is adapted for online perception by selecting reference frames from prior time steps instead of using a pre-collected sequence.
  • A CNN predicts local Gaussian triplets from the current frame and references, with centers obtained by unprojecting pixels using predicted depth.
  • Overlapping local and global Gaussians are paired and fused using confidence-weighted updates, while latent features are combined with a lightweight GRU.

4. Our Methods

EmbodiedSplat maps posed image streams into semantic 3D Gaussian fields using online 2D-to-3D feature lifting, a global CLIP codebook, sparse per-Gaussian caches, and geometry-aware 3D features.

  • Overview: The feed-forward mapping transforms a posed image stream, typically exceeding 300 frames, into a 3D Gaussian field with per-Gaussian language embeddings.
  • 2D Semantic Features: Pixel-wise 2D CLIP features are unprojected directly into 3D through local Gaussian triplets rather than rasterizing 3D features into images.
  • Global Codebook: The CLIP Global Codebook accumulates instance-level features across frames and represents each Gaussian feature as a sparse combination of codebook vectors.
  • Sparse Caches and Reconstruction: Sparse index and weight caches associate each Gaussian with codebook entries and are updated online during Gaussian fusion.
  • Sparse Caches and Reconstruction: With L=6, each Gaussian uses 2(L−1)=10 cache numbers rather than a full 512- or 768-dimensional CLIP vector.
  • Codebook-based Cosine Similarity: The codebook preserves original CLIP semantics while reducing cosine-search complexity from O(MD) to codebook-scale computation plus sparse weighted sums.
  • Geometry-aware 3D Semantic Features: A sparse 3D U-Net with a memory adapter aggregates Gaussian point-cloud features and injects geometric priors into compact 3D representations.
  • 2D-3D Ensemble: During inference, 2D semantic and 3D geometric probabilities are combined through a geometric-mean ensemble.

5. Experiments

Experiments evaluate EmbodiedSplat on indoor 3D semantic segmentation, cross-domain transfer, feature and codebook ablations, memory efficiency, and qualitative online reasoning. The method achieves strong segmentation and reconstruction efficiency while combining semantic and geometric CLIP features.

  • Experimental Results: EmbodiedSplat achieves the best performance across evaluated benchmarks with the shortest reconstruction time, while EmbodiedSplat-fast reaches 5–6 FPS per-frame processing.The feed-forward design supports short reconstruction time, and real-time 2D models enable the faster variant.
  • Cross-domain 3D Semantic Segmentation: Cross-domain transfer remains within 1 mIoU of in-distribution performance for ScanNet++ →ScanNet, but ScanNet →ScanNet++ drops by 11.28 mIoU.The latter gap is attributed to poor depth estimation in challenging regions such as ceilings; RGB-D inputs reduce the difference to 44.14 mIoU versus 44.03 mIoU.
  • Ablations on 3D CLIP Features: Combining 3D geometric-aware CLIP features with 2D CLIP features improves mIoU across all evaluated indoor benchmarks.The 2D features preserve semantic generalization, while the 3D features encode geometric priors and compensate for semantic information loss during distillation.
  • Ablations on Codebook-Based Cosine Similarity: Nearly 14× faster processing results from codebook-based cosine similarity compared with naive per-Gaussian computation.On one ScanNet scene, the method processes 3.2M Gaussians using a codebook of 8.7K instance-level CLIP features.
  • Comparisons on Memory Size: 148 MB memory consumption preserves original CLIP information without pretraining, compared with 2295 MB for Occam’s LGS.The sparse coefficient field and CLIP global codebook provide the reported memory efficiency.
  • Ablations on Cache Size: Aggregating multiple instance features from multi-view images improves performance as cache size increases beyond L = 2.The ablation compares L = 2 with L = 4, 6, and 11.

6. Conclusions

EmbodiedSplat addresses online open-vocabulary 3D understanding by extending pretrained feed-forward 3DGS for embodied scenarios. Its memory-efficient semantic representation and real-time 2D integration support practical online reconstruction.

  • EmbodiedSplat provides online open-vocabulary capability for pretrained feed-forward 3DGS.
  • Its Sparse Coefficient Field and CLIP Global Codebook represent each 3D Gaussian efficiently while preserving CLIP’s semantic richness.
  • Integrating real-time 2D vision models enables near-real-time per-frame processing for embodied agents.
  • The framework is presented as a pioneering 3DGS-based 3D perception model for embodied scenarios.

7. Additional Explanations.

The additional explanations describe EmbodiedSplat’s experimental setup and online inference pipeline, from keyframe selection and local semantic Gaussian construction to feature processing and global fusion.

  • Additional Explanations: The supplementary section covers experimental settings, the EmbodiedSplat framework, and the EmbodiedSplat-fast variant.
  • Experimental Setup: Keyframes are selected when the pose distance from the last keyframe exceeds 0.1, and the selected images serve as streaming inputs.
  • Local Semantic Gaussians Field: The model builds local Gaussian fields from the current frame and 30 reference views using shared-backbone features, plane-sweep stereo, and a UNet++-like decoder.
  • Sparse Coefficient Field: The sparse coefficient field lifts pixel-level 2D CLIP features to Gaussians using local indices, weights, and an updated global codebook.
  • 3D CLIP Features: A 3D U-Net and memory adapter aggregate current local point clouds with spatially close global latents to produce geometric-aware 3D features.
  • Gaussians Fusion: During Gaussian fusion, unmatched local Gaussians are appended to the global set, while valid local-global pairs are fused by the online fusion algorithm.

8. Discussions.

The discussion situates EmbodiedSplat among semantic 3DGS approaches and identifies limitations arising from its feed-forward reconstruction backbone, domain shift, and depth estimation.

  • Feed-forward semantic 3DGS: Existing feed-forward semantic 3DGS methods generally lack whole-scene reconstruction, direct 3D referring, or online operation from streaming images.Several methods rely on pre-collected multi-view images and offline settings, while LSM handles only two or a few input views.
  • SLAM + Semantic 3DGS: Online-LangSplat and EA3D support online semantic reconstruction but require per-scene optimization, limiting near-real-time generalization to novel scenes.Their feature distillation through rendering also limits direct 3D referring.
  • Limitations: EmbodiedSplat inherits FreeSplat++’s reconstruction failures, which produce noisy semantic Gaussian fields when the underlying 3DGS is inaccurate.The limitation is tied to the feed-forward reconstruction backbone rather than semantic processing alone.
  • Limitations: Training on ScanNet leads to weaker performance on Replica because the real-world-to-synthetic domain gap degrades feed-forward 3D reconstruction and semantic lifting.The authors attribute the failure to overfitting of feed-forward 3DGS to the real-world domain.
  • Limitations: ScanNet-trained models also suffer on ScanNet++ in regions such as ceilings because inaccurate depth maps yield noisy point clouds and degrade 3D U-Net feature aggregation.Depth sensors can largely mitigate this issue.

9. Additional Experiments.

Additional experiments evaluate EmbodiedSplat across semantic segmentation, rendering, memory efficiency, and qualitative perception tasks. The results support broad compatibility, online speed, reconstruction quality, and substantial semantic-memory savings.

  • 2D models: EmbodiedSplat performs well across diverse 2D models, supporting the use of FastSAM with a pixel-level CLIP model for fast embodied inference.The authors note that the commonly used SAM+CLIP combination is unsuitable for scenarios requiring fast inference.
  • 3D semantic segmentation: EmbodiedSplat outperforms the OpenScene and EmbodiedSAM point-cloud methods in 3D semantic segmentation under matched RGB-D inputs.The authors attribute this to 3DGS-based smooth feature aggregation onto 3D points using Mahalanobis distance.
  • 2D-rendered semantic segmentation: EmbodiedSplat achieves comparable 2D-rendered segmentation performance to LangSplat despite not being optimized for a specific scene.Evaluation uses interpolated novel views from ScanNet across 10, 15, and 19 classes.
  • Novel-view synthesis: EmbodiedSplat matches the reconstruction performance of offline FreeSplat++ while adapting its inference pipeline to online operation.This preserves FreeSplat++’s advantage over earlier feed-forward 3DGS models in whole-scene reconstruction.
  • Memory efficiency: 67× average memory-efficiency improvement results from storing far fewer CLIP codebook features than Gaussians.The sparse coefficient field is constructed online, needs no pretraining or per-scene optimization, and supports real-time updates.
  • Qualitative results: EmbodiedSplat and EmbodiedSplat-fast produce clearer, more accurately classified 3D segmentation masks than the compared 3D baselines.Additional visualizations also show multi-view-consistent object-search segmentation and high-fidelity novel-view rendering with depths.
  • Online fusion: The online fusion algorithm appends unmatched local Gaussians and updates matched Gaussian coefficient caches using confidence-weighted averaging.The toy example illustrates index and weight-cache updates during online fusion.
Loading 2603.04254v1…