PaperlayerView exact revision on arXiv

Source-linked AI summary

Gaussian Episodic Memory for Long-Term Spatiotemporal Reasoning

Ling-Chia Chen, Abdelrhman Werby, Kaan Aytekin, Kai O. Arras

arXiv:2610.33158v1cs.CVcs.RO

TL;DR

Robots need memories that capture semi-static scene changes, yet existing structured and image-based memories limit stored entities or viewpoints. GEMS combines persistent 3DGS memory with agentic evidence gathering, and reports better grounding and spatiotemporal question answering than established baselines.

  • Problem

    Structured memories can be limited to entities and attributes captured during construction, while image-based memories are constrained to previously observed viewpoints.

    Source: §I · p. 1

  • Method

    GEMS stores local scene changes as temporal deltas in a persistent 3DGS map and uses an agentic pipeline to render views and gather query-specific evidence.

    Source: §I · p. 2

  • Results

    GEMS consistently outperforms established baselines on spatiotemporal question answering, and combining viewpoint selection with cross-temporal rendering improves question-answering accuracy.

    Source: §VI · p. 7

  • Takeaways & Limitations

    GEMS-Bench provides a multi-revisit benchmark of 15 real-world and synthetic scenes with 473 question-answer pairs for spatiotemporal embodied question answering.

    Source: §I · p. 2

    On real-world scenes, sensor noise and pose-estimation errors result in imperfect Gaussian maps.

    Source: C. Spatiotemporal Question Answering · p. 7

Abstract

from arXiv · show

Real-world environments undergo continual semi-static changes, requiring a memory representation that captures both the spatial and temporal evolution of a scene. Existing spatio-temporal memories for robots typically rely either on structured representations, which are limited to the entities and attributes captured during construction, or on image-based memories, which are constrained to previously observed viewpoints. We introduce GEMS, a framework that combines a persistent spatio-temporal 3D Gaussian Splatting (3DGS) map with an agentic reasoning pipeline. The 3DGS map is reconstructed once and subsequently updated only in regions where the scene changes, storing only these modifications while preserving visual detail across time. The agentic pipeline leverages this map to render novel views on demand, iteratively accumulating evidence to answer spatio-temporal questions and ground objects from natural-language queries. To advance future research, we introduce GEMS-Bench, a spatiotemporal question-answering benchmark focused on semi-static environments. It comprises 3 real-world and 12 synthetic environments, spanning 473 question-answer pairs. We evaluate GEMS on novel-view synthesis quality across scene updates, 3D object grounding, spatiotemporal question answering, and storage efficiency. GEMS outperforms recent baselines in 3D object grounding and spatiotemporal question answering, while achieving rendering quality on par with dedicated scene-update methods. Code and data: https://abwerby.github.io/GEMS/

I. INTRODUCTION

Robots need memories that capture how semi-static environments change over time, but structured and image-based approaches limit what can be recalled or viewed. GEMS combines persistent 3D Gaussian Splatting memory with agentic reasoning, and the paper introduces GEMS-Bench for evaluating spatiotemporal reasoning.

Motivation and gap

  • Existing temporal memories may retain only predefined object states or fixed-view observations, while 3DGS approaches address static scenes or preserve only the latest state.These gaps constrain access to visual evidence and scene history.Source: §I · pp. 1–2

GEMS

  • GEMS stores scene changes as local temporal deltas, sharing unchanged Gaussians so memory grows with environmental changes rather than revisits.It maintains a persistent 3DGS map for high-fidelity rendering over time.Source: §I · p. 2
  • Its agentic pipeline selects informative viewpoints, aligns virtual-camera views across time, and segments query targets on demand for object grounding.The pipeline uses novel-view rendering to gather evidence for spatial, semantic, and temporal questions.Source: §I · p. 2

Benchmark and evaluation

  • GEMS-Bench covers 15 real and synthetic scenes and 473 question-answer pairs, while GEMS is evaluated on rendering, grounding, question answering, and storage.The benchmark includes multi-revisit scenes and posed RGB-D data.Source: §I · p. 2

II. RELATED WORK

Prior work offers spatial-semantic maps, temporal representations, 3DGS scene maps, and episodic question-answering memories, but each leaves gaps in preserving and querying visual scene history. GEMS applies query-conditioned views to a memory spanning visits.

A. Spatial-Semantic and Spatiotemporal Mapping

  • Spatial-semantic maps encode geometry and meaning, but scene graphs often capture only one snapshot; temporal maps may omit photorealistic evidence or retain only current states and predefined objects or attributes.Some methods preserve trajectories or timestamped descriptions, yet discard underlying visual detail.Source: A. Spatial-Semantic and Spatiotemporal… · p. 2

B. 3DGS-based Maps

  • 3DGS supports photorealistic rendering and local updates, but prior embodied-reasoning systems either reason over static captures or keep only the latest state; GEMS renders query-conditioned views across visits.This permits holding a viewpoint fixed while varying time.Source: B. 3DGS-based Maps · p. 2

C. Embodied Question Answering and Episodic Memory

  • Episodic question-answering memories fix information granularity or viewpoint at construction, while existing benchmarks often omit scene evolution between revisits or limit scene scale and question diversity.GEMS performs object segmentation at query time rather than relying on predefined object lists.Source: C. Embodied Question Answering and… · pp. 2–3

III. METHOD

GEMS pairs a photorealistic spatiotemporal Gaussian map with an agentic reasoning component that iteratively retrieves and renders evidence for a query. The map is initialized from the first visit and maintained as the scene changes.

III. METHOD

  • GEMS has a mapping component that maintains scene history and renders query-related views, plus an agentic component that gathers evidence through retrieval, rendering, and segmentation.The agent synthesizes its answer from the collected evidence.Source: §III · p. 3

A. Spatiotemporal Gaussian Map

  • At each of T + 1 revisits, the robot supplies a posed RGB-D sequence with images, depth maps, camera intrinsics, camera poses in a common world frame, and a frame count.These sequences form the inputs to the spatiotemporal Gaussian map.Source: A. Spatiotemporal Gaussian Map · p. 3
  • The first visit initializes the 3D Gaussian map using gsplat and GaME photometric and geometric losses, with a normalized, thresholded per-axis scale penalty against extreme anisotropic elongation.The scale penalty replaces GaME's isotropic regularizer.Source: A. Spatiotemporal Gaussian Map · p. 3

1) Cross-Visit Change Detection:

Cross-visit change detection compares each new revisit with the prior map using spatially diverse keyframes rendered at matching poses. Visual and semantic differences identify changed regions, which are classified and assigned local update operations.

1) Cross-Visit Change Detection:

  • The method selects keyframes whose poses are farther than τ_pose from previously selected keyframes and renders the prior map at each current observation pose.Matching camera poses makes differences primarily reflect scene changes rather than viewpoint changes.Source: 1) Cross-Visit Change Detection · p. 3
  • It combines visual SSIM and semantic DINOv3 feature-map similarities, takes their complement as the change score, and thresholds that score into a binary mask.The similarities are clamped to [0, 1] and combined with w = 0.3.Source: 1) Cross-Visit Change Detection · p. 3
  • Each connected changed component is classified as Add, Remove, or Refine using relative depth error and the fractions of pixels indicating newly observed or missing foreground geometry.Positive depth error indicates new geometry in front of the previous map; negative error indicates previously observed foreground is absent.Source: 1) Cross-Visit Change Detection · p. 3
  • Add and Remove trigger when their respective pixel fractions exceed τ_m; if neither does, Refine accounts for appearance changes or local reconstruction errors.Both Add and Remove may apply to one component, as when an object is relocated.Source: 1) Cross-Visit Change Detection · p. 3

2) Local Map Updates and Temporal State Storage:

GEMS updates only affected Gaussian primitives when a revisited scene changes, while storing temporal states through per-primitive lifespans rather than separate maps for every revisit.

2) Local Map Updates and Temporal State Storage:

  • On each revisit, Add, Remove, and Refine masks guide local Gaussian updates, while unchanged primitives remain static and still participate in rendering.New primitives are initialized from masked depth pixels; removed foreground primitives are deleted, with new samples added to fill potential disocclusion holes.Source: Figure 2 · p. 4, 2) Local Map Updates and Temporal… · p. 4
  • Each Gaussian stores a lifespan [b_g, d_g), allowing the map for a revisit to be recovered as a temporal slice.New primitives are appended, removed primitives are closed, and refined primitives are replaced by re-optimized copies with new lifespans.Source: 2) Local Map Updates and Temporal… · p. 4
  • The memory representation grows with accumulated scene changes rather than with the number of revisits.The retained state consists of the Gaussian primitives and keyframe poses for each revisit.Source: 2) Local Map Updates and Temporal… · p. 4

B. Agentic Spatiotemporal Question Answering

GEMS answers queries by iteratively selecting memory tools to gather visual and 3D evidence. A finite-state machine constrains tool access as the agent moves between retrieval and detailed reasoning.

B. Agentic Spatiotemporal Question Answering

  • At each iteration, an LLM planner selects a memory tool and its arguments from the query and evidence gathered so far, and the tool returns further evidence.Search ends when the planner terminates or reaches the predefined tool-call budget; an answer synthesizer then produces an answer, rationale, and confidence score from the report and evidence.Source: B. Agentic Spatiotemporal Question… · pp. 4–5
  • Cached keyframe image features and clustered revisit poses support concept search at two spatial granularities.The system renders keyframe RGB frames and caches OWLv2 features, then groups poses into spatial regions.Source: B. Agentic Spatiotemporal Question… · p. 5
  • Retrieval tools locate candidate views, while reasoning tools inspect regions, align views across revisits, ground objects, and compare or track instances.Retrieval includes scene overview, keyframe search, covisibility pruning, and evidence recall; reasoning includes time_view, relook, locate, obj_view, compare, and track.Source: B. Agentic Spatiotemporal Question… · p. 5
  • A finite-state machine restricts available tools, starting with global retrieval and alternating between detailed reasoning and retrieval rounds.The conclude control ends a round and clears raw visual context from history; exit ends the search, while report compiles the evidence.Source: B. Agentic Spatiotemporal Question… · p. 5, Figure 3 · p. 5

IV. DATASET

GEMS-Bench evaluates question answering across discrete revisits to semi-static environments using real and synthetic multi-visit RGB-D scenes. Its 473 questions cover temporal, static, and negative-premise reasoning.

IV. DATASET

  • The benchmark focuses on reasoning across discrete revisits and includes multi-visit RGB-D trajectories, camera poses, and object-level change annotations.Scene changes and questions across both splits were manually verified for correct answers and unambiguous visual references.Source: §IV · p. 5
  • GEMS-Bench contains three real-world indoor environments with four visits each and 12 synthetic scenes with at least three visits each.The real-world split has 244 question-answer pairs; the synthetic split has 229 and includes 97 annotated object-level changes.Source: §IV · p. 5
  • The benchmark has 473 question-answer pairs across Counting, Movement and Presence, State Change, Static, and Negative questions.Negative questions are 30.7%, Movement and Presence 22.8%, State Change 20.5%, Static 16.9%, and Counting 9.1%.Source: §IV · p. 5

A. Reconstruction Quality

GEMS maintains rendering quality comparable to dedicated scene-update methods while updating maps substantially faster. Its depth accuracy is strongest on CL-Splats and second-best on GaME-Aria.

A. Reconstruction Quality

  • The evaluation compares scene-update methods on two real-world benchmarks, using held-out images to measure photometric quality, geometric accuracy, and update time.The benchmarks contain one temporal update per scene; GEMS and GaME require depth, so Depth Anything 3 supplies synthesized depth for CL-Splats.Source: A. Reconstruction Quality · p. 5
  • About one minute per map update makes GEMS 18×–26× faster than GaME, with the lowest depth error on CL-Splats and second-lowest on GaME-Aria.GEMS remains within a few dB PSNR of the best photometric scores, achieved by O-SCD, which does not produce depth.Source: A. Reconstruction Quality · pp. 5–6

B. Object Grounding from Natural-Language Queries

GEMS leads the reported spatiotemporal QA comparisons and achieves the highest overall object-grounding accuracy, with one exception among grounding query categories. Its QA performance differs across real-world and synthetic scenes, where the authors report different effects from novel-view synthesis.

B. Object Grounding from Natural-Language Queries

  • 58.9% overall accuracy makes GEMS the top object-grounding method; it beats all baselines except on queries without a target mention.GEMS exceeds the other methods in spatial, color, and shape categories; KeySG has a slight edge on queries without a target mention.Source: B. Object Grounding from… · p. 6

C. Spatiotemporal Question Answering

  • 60.9% on the real-world split and 71.0% on the synthetic split for GEMS, versus 44.8% and 35.5% for STaR, respectively; GEMS leads both splits.The table reports accuracy by question category and includes an NVS-tool ablation block.Source: Table III · p. 6, C. Spatiotemporal Question Answering · p. 7
  • The QA score is the mean of LLM-matcher ratings on a 1–5 scale, rescaled to [0, 1], with GPT-5.6-Luna as judge.The evaluation builds episodic memory for each scene, then runs the agentic pipeline to answer its questions.Source: C. Spatiotemporal Question Answering · pp. 6–7
  • On the real-world split, GEMS slightly trails GEMS-all-images, while on the synthetic split it performs better despite storing no images.The authors attribute real-world degradation to sensor noise and pose-estimation errors that can produce artifacts in views far from training poses; perfect synthetic depth and poses avoid these artifacts.Source: C. Spatiotemporal Question Answering · p. 7

D. Storage Efficiency

GEMS stores changes as localized temporal deltas, so its memory grows with detected scene change rather than at a fixed rate per revisit. The reported NVS ablation finds a QA benefit on the subset of questions where the agent invoked the tools.

D. Storage Efficiency

  • After four real-world revisits, GEMS uses 1.95 GiB at fp32 or 0.98 GiB at fp16, versus 2.73 GiB for keyframes and 3.94 GiB for all frames.GEMS stores affected Gaussian primitives; its memory grows with observed change rather than incurring a fixed cost per revisit, and is more efficient when changes are localized.Source: Figure 5 · p. 7, D. Storage Efficiency · p. 7

E. Ablation Studies

  • 6.6 percentage points: NVS tools improve mean QA score across all five categories on the 245 questions where the full agent invoked them.Both tools were disabled jointly under the same Gaussian memory and inference budget, so the ablation does not separate spatial view refinement from temporal alignment.Source: E. Ablation Studies · p. 7

VI. CONCLUSION

  • GEMS stores environmental change as compact temporal deltas and answers episodic-memory questions by rendering novel views across space and time.The conclusion reports that experiments on the long-term spatiotemporal benchmark show consistent baseline outperformance in question answering.Source: §VI · p. 7
Loading 2610.33158v1…