PaperlayerView exact revision on arXiv

Source-linked AI summary

GaussMemory: Task-Driven 3D Gaussian Scene Memory for Long-Horizon Robotic Manipulation

Zhiqiang Hu, Shouren Huang, Masatoshi Ishikawa

arXiv:2608.14986v1cs.ROcs.AI

TL;DR

Long-horizon manipulation requires spatial memory because current observations may not reveal previously seen object identities or positions, while existing systems passively update all scene elements. GaussMemory learns task-driven memory updates and retrieval over persistent 3D Gaussian scenes, outperforming baselines on long-horizon benchmarks, including a 6.0-point gain over π0-FAST on VLABench Track 6.

  • Problem

    Memoryless or hand-crafted 3D memory systems cannot reliably preserve task-relevant object identity and spatial state across long-horizon manipulation.

  • Method

    GaussMemory unifies task-conditioned memory updating and retrieval through differentiable attention over persistently updated 3D Gaussian scenes within a VLA.

  • Results

    GaussMemory outperforms baselines on long-horizon tasks across LIBERO and VLABench, achieving +6.0% over π0-FAST on VLABench Track 6.

  • Takeaways & Limitations

    The system learns to update manipulated objects aggressively while leaving background largely untouched, shifting spatial memory from passive recording toward task-driven retention.

Abstract

from arXiv · show

Long-horizon robotic manipulation fundamentally relies on persistent spatial memory. However, existing 3D memory systems function merely as passive recorders: they store observations using fixed, hand-crafted rules, treating every scene element--whether a critical grasp target or an irrelevant background wall--with equal importance. In this paper, we propose a paradigm shift from passive storage to active, task-driven spatial memory. We argue that a robot's memory should not simply record what it sees, but actively learn how to remember--discovering which objects to track precisely, how aggressively to update them, and what to discard, all learned end-to-end without hand-designed rules. Crucially, this active paradigm is realized by unifying memory update and readout as two sides of the same cognitive process, enabling bidirectional flow where task needs shape update strategies and vice versa. To instantiate this vision, we introduce GaussMemory, which leverages 3D Gaussian Splatting as a persistent geometric substrate. On LIBERO, GaussMemory outperforms MemoryVLA on Goal and Long-10; on VLABench, it surpasses $π_0$-FAST by +5.2% (Track 1) and +6.0% (Track 6).

I. INTRODUCTION · II. RELATED WORK · A. Memory for Long-Horizon Manipulation

Long-horizon manipulation requires persistent spatial memory because current observations can lose task-critical information. GaussMemory addresses limitations of passive 2D and hand-crafted 3D memories by continuously updating a geometrically explicit 3D Gaussian memory whose behavior is shaped by task outcomes.

  • II. RELATED WORK: Existing temporal memories remain limited because MemoryVLA and MEM operate in 2D token space without explicit 3D geometry.Consequently, they cannot directly answer spatial queries such as the precise location of one object relative to another.
  • I. INTRODUCTION: Memoryless VLA policies fail when the current sensory observation no longer reveals information needed to choose the next action.The identical-drawer example shows that a robot may need persistent memory to locate an object after an intervening sub-task.
  • II. RELATED WORK: ConceptGraphs preserves explicit 3D geometry but relies on hand-crafted update rules that are decoupled from task performance.Its rules include thresholded matching, fixed-rate averaging, and hard pruning.
  • I. INTRODUCTION: Across prior memory categories, update and readout are treated as independent, making these systems passive recorders that assign equal importance to scene elements.GaussMemory instead enables task loss to shape memory behavior.
  • I. INTRODUCTION: 3D Gaussian Splatting provides GaussMemory with incremental scene editing, explicit geometry, and compact tokenizable features for active VLA memory.Individual Gaussians can be added, removed, or repositioned; each carries position, covariance, and opacity.
  • I. INTRODUCTION: GaussMemory continuously updates Gaussian-derived memory tokens across sub-tasks and jointly optimizes them with robot action loss for LLM-based action prediction.The paper presents this as the first integration of 3D Gaussian Splatting as temporal scene memory within a VLA framework.
  • A. Memory for Long-Horizon Manipulation: Long-horizon memory approaches include language-based, image-based, and 3D-structured systems, but they separate memory update from readout into independent non-differentiable stages.SayPlan loses geometric precision, MemoryVLA lacks 3D spatial reasoning, and ConceptGraphs builds object-centric 3D scene graphs.
  • A. Memory for Long-Horizon Manipulation: GaussMemory shifts from passive to active memory by learning its update mechanism from task outcomes while using 3D Gaussians for geometric precision.This distinguishes it from text and image memories described in the related-work comparison.

B. 3D Gaussian Splatting for Robotics · III. METHOD · A. Problem Formulation

GaussMemory uses 3D Gaussian Splatting as a persistent, temporally evolving scene memory for long-horizon manipulation. Its formulation represents language instructions as sequential sub-tasks and aims to predict actions while maintaining memory of scene changes.

  • B. 3D Gaussian Splatting for Robotics: GaussMemory uses dynamic 3D Gaussians as persistent scene memory for retrospective state tracking and future action prediction.This contrasts with ManiGaussian’s use of dynamic Gaussians as a forward world model.
  • III. METHOD: Multi-view RGB images are encoded into observation tokens Qobs and concatenated with task-conditioned readout queries Qread.The combined tokens are jointly processed by Unified Memory Attention.
  • III. METHOD: Unified Memory Attention cross-attends to a 3D Gaussian Scene Memory containing up to 32K persistent Gaussians.The architecture uses the memory as the persistent geometric substrate for processing observation and readout information.
  • B. 3D Gaussian Splatting for Robotics: Unlike GaussianVLM, GaussMemory maintains a temporally evolving Gaussian memory whose tokens directly condition robot action prediction.GaussianVLM encodes static Gaussian scenes for language-based reasoning and produces text outputs.
  • A. Problem Formulation: A high-level language instruction L decomposes into a sequence of T sub-tasks {τ1, . . . , τT}.The formulation targets long-horizon manipulation tasks specified by language.
  • A. Problem Formulation: At each time step, the robot receives multi-view RGB images and its proprioceptive state st, then predicts actions that sequentially complete all sub-tasks.The robot must simultaneously maintain persistent memory of scene changes.

B. System Overview · C. Temporal Gaussian Scene Memory

GaussMemory combines persistent temporal Gaussian scene memory, unified read/write attention, and a Prismatic VLM with a diffusion action head. Its memory stores augmented, temporally encoded Gaussian primitives organized into object-centric groups across the task horizon.

  • B. System Overview: GaussMemory comprises Temporal Gaussian Scene Memory, Unified Memory Attention for simultaneous read/write, and a Prismatic VLM with a diffusion action head.The Prismatic VLM uses Llama-2-7B, shared with OpenVLA, CogACT, and MemoryVLA.
  • C. Temporal Gaussian Scene Memory: The core data structure is a persistent set of augmented Gaussian primitives maintained across the task horizon.This structure is denoted as M_t.
  • C. Temporal Gaussian Scene Memory: Each Gaussian combines standard 3DGS attributes—position, covariance, opacity, color, and spherical harmonic features—with temporal metadata.The standard attributes are μ_i, Σ_i, α_i, c_i, and h_i.
  • C. Temporal Gaussian Scene Memory: Temporal metadata records creation or modification timestamps, sub-task index, object ID, and existence probability.The variables are t_i^c, τ_i, o_i, and e_i ∈ (0, 1).
  • C. Temporal Gaussian Scene Memory: Temporal feature encoding uses 3D Fourier positional encoding together with relative-age encoding for memory entries.Relative encoding is used for length generalization.
  • C. Temporal Gaussian Scene Memory: Encoding uses fixed Fourier frequencies, a maximum episode length T_max, and N_T frequency bands.These settings define the temporal feature representation.
  • C. Temporal Gaussian Scene Memory: Gaussian Grouping with DINO features supplies object instance IDs and organizes primitives into per-object groups G_j = {G_i : o_i = j}.This creates an object-centric organization of the temporal Gaussian memory.

D. Unified Memory Attention (UMA)

Unified Memory Attention (UMA) jointly learns what to retrieve and update through one differentiable attention mechanism. Its shared read-write pathway lets task needs shape memory behavior, including correspondence, update rates, preservation, pruning, and insertion.

  • Unified Memory Attention: UMA unifies memory retrieval and update by combining task-conditioned readout queries with observation tokens before jointly cross-attending to 3D Gaussian memory.This produces task-relevant spatial context for the LLM and soft correspondence signals for memory updates.
  • Stage 2: Self-Attention: Self-attention creates bidirectional task-scene exchange: readout queries highlight relevant observations, while observation tokens ground readout queries in the current scene.The resulting contextualized unified query is task-aware before accessing persistent memory.
  • Stage 3: Cross-Attention: Temporal-bias cross-attention queries stored Gaussian memory using learned relative-time information, with self-attention, cross-attention, and feed-forward layers repeated across UMA blocks.The temporal bias modulates attention according to temporal recency and is shared across attention heads.
  • Stage 4: Read and Write Streams: The shared cross-attention splits into read and write streams, using readout features for LLM input and observation attention rows as the correspondence matrix for memory updates.This eliminates separate matching procedures such as Sinkhorn or Hungarian.
  • Why Unifying Read and Write Matters: The unified gradient path lets task loss directly shape read-to-write coupling, write-to-read information freshness, update gates, existence probabilities, pruning, and novel-object insertion.The learned update gate preserves accumulated estimates for static background while trusting new observations for relevant objects; Gaussians with ek < 0.1 are pruned.

E. LLM Backbone and Action Generation

GaussMemory adopts the Prismatic VLM backbone used by OpenVLA, CogACT, and MemoryVLA to isolate the effects of 3D Gaussian memory and UMA. Its LLM receives a concatenated sequence combining memory, visual, language, and state tokens.

  • LLM Backbone: The method uses Prismatic VLM, pairing DINOv2 and SigLIP vision encoders with a Llama-2-7B LLM, matching OpenVLA, CogACT, and MemoryVLA.This shared backbone is intended to ensure performance differences stem from 3D Gaussian memory and UMA rather than backbone strength.
  • Action Generation: The LLM input is the concatenated token sequence Z = [Zmem; Z2D; ZL; ZS].Z2D denotes visual tokens from the fused Prismatic vision encoder combining DINOv2 and SigLIP.

F. End-to-End Training

End-to-end training combines action prediction with completion, temporal consistency, and task-relevance-weighted memory-update supervision. Gradients through η_k and A_match directly teach the memory how to update.

  • Loss Design: The total loss combines diffusion action prediction with sub-task completion, temporal consistency, and task-relevance-weighted memory-update objectives.L_complete uses binary cross-entropy, L_temporal promotes coherent static-object memory via stop-gradient consistency, and L_update supervises memory positions against ground truth.
  • End-to-End Optimization: Gradients flow through η_k and A_match, directly teaching the memory how to update.
  • Training Procedure: Training is conducted in two stages, beginning with pre-training FF-3DGS.The supplied passage truncates the description after this pre-training stage.

IV. EXPERIMENTS · A. Experimental Setup

The experiments evaluate GaussMemory across LIBERO and VLABench, using multi-view inputs, published VLA baselines, and a no-memory ablation. The setup specifies the Gaussian-memory architecture, training configuration, and evaluation protocols.

  • A. Experimental Setup: LIBERO covers four suites—Spatial, Object, Goal, and Long-10—with 40 tasks, 20 trials per task, and results averaged across three seeds and tasks.The suites target diverse spatial, semantic, and long-horizon challenges.
  • A. Experimental Setup: VLABench contains 100 task categories, with evaluation on Track 1’s 20 primitive tasks and Track 6’s 10 long-horizon composite tasks using SR and PS.The benchmark requires world knowledge, spatial understanding, semantic reasoning, and long-horizon planning.
  • A. Experimental Setup: LIBERO experiments use three views by adding lateral and overhead virtual viewpoints at ±30° to its default third-person camera.VLABench instead uses its built-in front and two side cameras.
  • A. Experimental Setup: LIBERO comparisons include 10 published VLA methods, while VLABench comparisons use officially reported results for π0, π0-FAST, and π0.5.The LIBERO baselines are OpenVLA, π0-FAST, π0, CogACT, MemoryVLA, SpatialVLA, TraceVLA, CoT-VLA, CronusVLA, and 4D-VLA.
  • A. Experimental Setup: The no-memory ablation retains the Prismatic VLM backbone while removing Gaussian memory to isolate memory’s contribution.The backbone combines Llama-2-7B, DINOv2, SigLIP, and a diffusion head.
  • A. Experimental Setup: The FF-3DGS encoder generates M = 8192 live Gaussians, pools them into approximately 20 object tokens, and uses 64 readout queries across 6 UMA layers.The model uses d = 768, a Prismatic VLM, and a 32K-Gaussian memory cap.
  • A. Experimental Setup: Training uses AdamW with lr = 2 × 10^-4, batch size 16, and 120K steps on 4×A100 GPUs.The implementation uses a 2-layer 256-unit ReLU MLP and loss weights λc = 1.0, λt = 0.1, and λu = 0.5.

B. LIBERO Results

GaussMemory achieves its strongest LIBERO gains on long-horizon tasks, where persistent 3D memory is increasingly valuable. Its 3D representation also outperforms a 2D-token alternative, while MemoryVLA remains slightly stronger on the short-horizon Spatial/Object suite.

  • Benchmark performance: 96.8% on Goal and 94.1% on Long-10 surpass MemoryVLA’s 96.4% and 93.4%, respectively.GaussMemory outperforms all baselines on these long-horizon suites, where spatial precision matters most.
  • Benchmark performance: 98.4% on Spatial/Object gives MemoryVLA an edge, reflecting the dominance of short-horizon semantic recognition.This contrasts with GaussMemory’s advantage on Goal and Long-10.
  • 3D representation ablation: 90.2% on Long-10 for the 2D-token ablation falls below both GaussMemory’s 3D model and MemoryVLA.The ablation replaces 3D Gaussians with 2D tokens, supporting the role of explicit 3D representation.
  • Memory scaling with horizon: +4.0 on Spatial, +8.6 on Goal, and +11.7 on Long-10 measure the monotonically increasing gap over the model without memory.The results indicate that persistent 3D memory becomes increasingly critical as tasks accumulate more state changes.

C. VLABench Results

GaussMemory improves VLABench success rates over π0-FAST, with larger gains on the longer-horizon Track 6. Ablations attribute the advantage to 3D geometric memory, learned active updates, unified read/write coupling, and sufficient Gaussian capacity.

  • VLABench performance: 56.4% SR on Track 1 and 24.0% on Track 6 outperform π0-FAST by +5.2% and +6.0%, respectively.Track 6 is a long-horizon composite task, and its improvement exceeds Track 1’s.
  • Representation ablation: 51.4% / 19.0% from a 2D token buffer trails the full 3D model’s 56.4% / 24.0% on Track 1/6.The ablation uses the same UMA architecture and backbone, isolating the representation effect.
  • Horizon scaling: +6.0% on Track 6 versus +5.2% on Track 1 shows larger improvement on the longer-horizon task.VLABench tasks use 500+ timesteps, compared with ∼120 for LIBERO.
  • Active memory updates: +2.4 from learned attention-derived updates raises success from 90.4 to 92.8, while the full active strategy reaches a total gain of +11.7.The learned update produces η values of 0.74 for manipulated targets and 0.06 for background.
  • Unified read/write: 91.8% for a decoupled baseline is 2.3 points below UMA’s 94.1%, demonstrating a contribution from unified read/write coupling.The baseline uses separate matching and Q-Former readout with matched parameters.
  • Memory capacity: 93.6% at 64K Gaussians indicates diminishing returns after performance scales up to the 32K default capacity.The plateau is consistent with learned existence management preventing redundant Gaussians.

D. Emergent Memory Behavior · V. CONCLUSION

GaussMemory learns task-driven memory behavior rather than passively recording scenes, aggressively updating manipulated objects while leaving background largely untouched. Its memory remains bounded through insertion and pruning dynamics, and it outperforms MemoryVLA on LIBERO Goal and Long-10.

  • D. Emergent Memory Behavior: η ≈0.74 for manipulated objects versus η ≈0.06 for background shows an emergent update strategy without hand-crafted rules.The strategy is driven by task-level action supervision and auxiliary memory-consistency losses.
  • V. CONCLUSION: GaussMemory shifts robot spatial memory from passive recording to active, task-driven learning.The conclusion presents this shift as the system’s central contribution.
  • D. Emergent Memory Behavior: The drawer’s η peaks during closing, then the bowl’s η rises during grasping, while background maintains η ≈0.06.This temporal behavior is learned through end-to-end gradient flow.
  • D. Emergent Memory Behavior: Insertion spikes during scene changes, while pruning maintains memory equilibrium well below the 32K cap.Active Gaussian count stays well below the cap throughout the LIBERO Long-10 episode.
  • V. CONCLUSION: 3D Gaussian Splatting and UMA enable the system to learn what matters, updating manipulated objects aggressively while leaving background untouched.These behaviors are discovered automatically from task performance.
  • V. CONCLUSION: GaussMemory outperforms MemoryVLA on LIBERO Goal and Long-10.The passage reports these benchmark outcomes without providing numerical scores.
Loading 2608.14986v1…