Source-linked AI summary
RecallGS: Geometry-Grounded Recall for Online Semantic Gaussian Splatting
Thanh-Khoi Nguyen, Thien-Phuc Tran, Minh-Triet Tran
TL;DR
Online semantic 3DGS pipelines use an additional learned 3D branch to recover geometric context, adding training and runtime costs. RecallGS instead retrieves and geometrically grounds past observations, and on ScanNet and ScanNet++ narrows the gap to EmbodiedSplat’s full 3D branch at higher speed.
Problem
Existing online semantic 3DGS pipelines use an additional learned 3D branch to recover geometric context, requiring costly training and adding GPU-memory use and latency.
Source: §1 · p. 2Method
RecallGS retrieves plausible historical frames with CLIP, verifies geometric support using alpha-compositing weights, and aggregates historical features over each Gaussian’s rendered footprint.
Source: §1 · pp. 2–3Results
On ScanNet and ScanNet++, RecallGS narrows the quality gap to EmbodiedSplat’s full 3D branch while running faster than the full model.
Source: §6 · p. 12Takeaways & Limitations
An agent’s observation history can serve as a non-parametric geometric-semantic prior without additional 3D training or per-scene optimization.
Source: §1 · p. 3The fixed grounding threshold depends on image resolution and projected footprint size, measuring visible rendering mass jointly with footprint size rather than visibility alone.
Source: §3.5 · p. 8
Abstract
from arXiv · showhide
Online semantic 3D Gaussian Splatting (3DGS) enables open-vocabulary scene understanding from streaming observations. However, existing methods recover geometric context missing from image-derived semantic features using an additionally learned 3D network, incurring extra training and runtime overhead. We observe that such geometric-semantic evidence may already exist in the agent's observation history and can be retrieved rather than learned. Based on this insight, we introduce RecallGS (Geometry-Grounded Recall for Online Semantic Gaussian Splatting), a lightweight, training-free mechanism that treats past observations as a non-parametric geometric-semantic prior. RecallGS follows a recall-ground-fuse pipeline: CLIP retrieves relevant observations, alpha-compositing weights identify geometrically supported views, and aggregate their semantic features over each Gaussian's rendered footprint. On ScanNet and ScanNet++, RecallGS achieves strong semantic segmentation performance at 2.48 FPS, demonstrating that geometric-semantic context can be effectively recovered from observation history without an additional learned 3D branch.
1 Introduction
Existing online semantic 3DGS methods learn geometric context through an extra 3D branch, prompting the question of whether observation history can supply that evidence instead. RecallGS retrieves, geometrically grounds, and fuses historical semantic evidence, with experiments showing component contributions and efficient online inference.
1 Introduction
- Existing online semantic 3DGS pipelines use an additional learned 3D branch to recover geometric context, adding training, GPU-memory, and latency costs.EmbodiedSplat-fast increases throughput from 0.75 FPS to 5.18 FPS after removing that branch, at the cost of geometric-semantic context.Source: §1 · p. 2
- A Gaussian ambiguous in the current view may match a surface seen earlier under more favorable viewpoint, illumination, or occlusion conditions.History reuse must address visually similar objects at different locations, occlusion of the queried surface, and the cost of checking every historical camera.Source: §1 · p. 2
- RecallGS treats observation history as a queryable geometric-semantic prior and routes evidence through semantic recall, geometric grounding, and compositing fusion.CLIP narrows the candidate pool, alpha-compositing weights identify geometrically supportive views, and historical features are aggregated over each Gaussian’s rendered footprint.Source: §1 · pp. 2–3
- Each RecallGS component contributes independently to semantic accuracy, and their combination supports geometric-semantic reasoning from observation history with efficient online inference.The introduction also reports that using observation history consistently improves online semantic 3DGS while maintaining efficient inference.Source: §1 · p. 3
2 Related Work
Prior language-embedded Gaussian work includes per-scene semantic optimization and feed-forward methods operating on fixed views. Online work adds persistent reconstruction and semantic features, while RecallGS differs from learned-branch approaches by retrieving and grounding evidence from observation history.
2.1 Language-Embedded Gaussian Splatting
- Per-scene methods embed open-vocabulary semantics through rendering supervision or direct 3D feature association, while feed-forward methods avoid per-scene optimization but use fixed, small view sets.Examples include LSM, SLGaussian, and SemanticSplat, which add semantic branches or cross-view aggregation to feed-forward reconstruction.Source: §2.1 · p. 3
2.2 Online Gaussian Splatting Reconstruction and Segmentation
- Online SLAM and instance-segmentation methods maintain scene information across observations, but differ in throughput or representation from an online Gaussian semantic field.Online-LangSplat and EA3D incur comparable throughput costs, while cited online instance methods use point-based representations.Source: §2.2 · p. 4
- RecallGS retrieves and geometrically grounds evidence from observation history instead of relying on EmbodiedSplat’s learned 3D U-Net semantic branch.EmbodiedSplat’s branch injects geometric context but requires additional 3D training and inference overhead.Source: §2.2 · p. 4
3 Method
RecallGS adds episodic recall to online semantic 3DGS: it retrieves past views, checks their geometric support, and reuses verified semantic evidence to refine the current representation. Its six-stage method uses a feed-forward reconstruction backbone, bounded observation history, appearance-based retrieval, rendering-based grounding, and feature fusion without additional training.
3.1 Preliminaries
- RecallGS uses FreeSplat++ to predict local Gaussian triplets, decodes renderable Gaussians for geometric grounding, and incrementally fuses overlapping local and global representations.EmbodiedSplat supplies a CLIP Global Codebook and fixed-length per-Gaussian sparse coefficients; cached dense pixel-aligned CLIP maps support replay.Source: §3.1 · p. 5
3.2 Problem Formulation
- At each timestep, the system updates its scene state from a posed RGB observation and uses only observations available so far, without future frames or offline per-scene optimization.Its episodic memory stores at most W observations with frames, poses, retrieval descriptors, and pixel-aligned features, discarding the oldest when full.Source: §3.2 · pp. 5–6
3.3 Overview
- For each incoming observation, RecallGS retrieves past views, verifies geometric compatibility, and reuses their semantic evidence to refine the current representation.The pipeline maintains episodic memory while building an online 3DGS reconstruction with a semantic field.Source: §3.3 · p. 6, Figure 1 · p. 6
3.4 Semantic Recall over Episodic Memory
- Image-level CLIP descriptors retrieve the top-scoring observations from memory, limiting geometric verification to at most Brec candidates.The image-level descriptor is used for retrieval, unlike instance-level CLIP features used to construct per-Gaussian semantics.Source: §3.4 · p. 7
3.5 Geometric Grounding of Recalled Memories
- RecallGS verifies recalled views by accumulated rendering contribution: a Gaussian is supported when its contribution exceeds τ, and camera coverage is the fraction of supported Gaussians.This geometry check addresses appearance-similar frames that may not depict the queried physical surface; the fixed threshold depends on image resolution and projected footprint size.Source: §3.5 · pp. 7–8
3.6 Compositing Fusion into the Global Codebook
- For each Gaussian, RecallGS aggregates historical CLIP features over its rendered footprint using rendering weights, fuses them with current-frame features, and passes the result onward without additional training.The historical view set is restricted to views satisfying the coverage criterion, and λ controls the historical contribution.Source: §3.6 · p. 9
4 Experiment
Experiments compare RecallGS with open-vocabulary semantic 3DGS baselines on ScanNet and ScanNet++, using EmbodiedSplat-fast as the primary baseline. RecallGS improves mIoU over that baseline while retaining online inference, and its qualitative reconstructions are more coherent.
Experimental setup
- Evaluation follows EmbodiedSplat’s dataset and protocol setup on ScanNet-10/15/19 and ScanNet++-20, with mIoU, mAcc, and FPS reported.The experiments use the same preprocessed test scenes and ground-truth annotations; all are conducted on one NVIDIA RTX5090 GPU.Source: §4 · pp. 9–10
Main results
- RecallGS improves EmbodiedSplat-fast by 3.10, 3.19, 3.38, and 3.22 mIoU on ScanNet-10/15/19 and ScanNet++-20, respectively, at 2.48 FPS versus 0.75 FPS for full EmbodiedSplat.The comparison reports corresponding mAcc gains; RecallGS is slower than EmbodiedSplat-fast at 5.18 FPS.Source: Table 1 · p. 10, §4 · p. 10
Qualitative results
- RecallGS produces more coherent semantic reconstructions with fewer fragmented or streak-like artifacts than EmbodiedSplat, especially across large surfaces and partially observed structures.The qualitative comparison reports local inconsistency in EmbodiedSplat, particularly in cluttered regions and partially observed structures.Source: §4 · p. 10
5 Ablation study
The ablations show that semantic retrieval and geometric verification complement one another, while full-footprint compositing and the chosen recall and grounding budgets further affect performance.
5.1 Historical View Selection
- The full recall-and-grounding strategy exceeds recent-history selection by approximately 1.14-1.51 mIoU; CLIP retrieval gains 0.44-0.63 mIoU, and geometric reranking adds 0.54-0.88 mIoU.The qualitative comparison also reports more coherent regions and fewer artifacts than EmbodiedSplat-fast, particularly on large surfaces and partially observed boundaries.Source: §5.1 · pp. 10–11, Table 2 · p. 11, Figure 2 · p. 11
5.2 Compositing vs. Center-Based Fusion
- The full Tα formulation improves over Projected Center by approximately 1.22-1.69 mIoU and over Center Lifting by 0.69-0.91 mIoU.It ranks views using accumulated Gaussian contribution and aggregates semantic features over each complete projected footprint; the ablation does not isolate transmittance T from opacity α.Source: §5.2 · p. 11, Table 3 · p. 12
5.3 Recall and Grounding Budget Sensitivity
- Moving from the default Brec = 16 and Bgeo = 3 lowers mIoU by 0.12-0.50 across the three ScanNet protocols.Smaller budgets can omit informative or complementary views, while larger budgets can include geometrically unsupported or redundant views; the default was chosen on the evaluation scenes.Source: Table 4 · p. 12, §5.3 · p. 12
6 Conclusion
RecallGS uses observation history as a non-parametric geometric-semantic prior without additional training. Its benefits depend on historical-view quality, and local-only grounding can overestimate visibility.
Conclusion
- RecallGS is a training-free recall-ground-fuse framework that retrieves past observations, verifies their geometric support through alpha-compositing responsibility, and fuses verified evidence into current Gaussians.It narrows the gap to EmbodiedSplat’s full 3D branch while running faster on ScanNet and ScanNet++.Source: §6 · p. 12
Limitations
- RecallGS’s quality is bounded by its past observations, limiting benefit when historical views are sparse or low quality; grounding rasterizes only current local Gaussians and may overestimate visibility because global-map occluders are not modeled.The paper suggests a future certainty or assessment gate for recalled observations, potentially using viewpoint quality, illumination, or depth confidence.Source: §6 · p. 12