PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Spatial Memory with Elastic Test-Time Training

Ziqiao Ma, Xueyang Yu, Haoyu Zhen, Yuncong Yang, Joyce Chai, Chuang Gan

arXiv:2604.07350v1cs.CVcs.GRcs.LG

TL;DR

Long-context 4D reconstruction is limited by activation memory and unstable, fully plastic LaCT updates. The paper introduces Elastic Test-Time Training and FSM, reporting scalable high-quality reconstruction with smaller chunks, reduced overfitting, and less reliance on camera-interpolation shortcuts. The work advances LaCT toward robust multi-chunk adaptation for longer sequences.

  • Problem

    Long-context 4D reconstruction remains limited by activation-memory constraints and by LaCT fast-weight updates that can overfit and drift during inference.

  • Method

    The paper adds a Fisher-weighted elastic prior around maintained anchor parameters to LaCT and pretrains FSM on curated 3D/4D data for long-sequence reconstruction.

  • Results

    FSM supports long sequences, arbitrary timestamps, and novel view-time combinations while smaller chunks reduce overfitting and camera-interpolation shortcuts.

  • Takeaways & Limitations

    Elastic multi-chunk adaptation provides a path beyond LaCT’s bounded single-chunk setting while substantially alleviating the activation-memory bottleneck.

  • Takeaways & Limitations

    The work focuses on architectural advances rather than fully realizing arbitrarily long-sequence scaling because of limited licensable data, benchmarks, and compute.

Abstract

from arXiv · show

Large Chunk Test-Time Training (LaCT) has shown strong performance on long-context 3D reconstruction, but its fully plastic inference-time updates remain vulnerable to catastrophic forgetting and overfitting. As a result, LaCT is typically instantiated with a single large chunk spanning the full input sequence, falling short of the broader goal of handling arbitrarily long sequences in a single pass. We propose Elastic Test-Time Training inspired by elastic weight consolidation, that stabilizes LaCT fast-weight updates with a Fisher-weighted elastic prior around a maintained anchor state. The anchor evolves as an exponential moving average of past fast weights to balance stability and plasticity. Based on this updated architecture, we introduce Fast Spatial Memory (FSM), an efficient and scalable model for 4D reconstruction that learns spatiotemporal representations from long observation sequences and renders novel view-time combinations. We pre-trained FSM on large-scale curated 3D/4D data to capture the dynamics and semantics of complex spatial environments. Extensive experiments show that FSM supports fast adaptation over long sequences and delivers high-quality 3D/4D reconstruction with smaller chunks and mitigating the camera-interpolation shortcut. Overall, we hope to advance LaCT beyond the bounded single-chunk setting toward robust multi-chunk adaptation, a necessary step for generalization to genuinely longer sequences, while substantially alleviating the activation-memory bottleneck.

1. Introduction

The paper targets long-context 4D reconstruction by stabilizing chunk-wise test-time adaptation and introducing FSM for scalable spatial memory over extended observations.

  • Motivation: Building spatial memory requires compressing observations across viewpoints and time into unified 4D representations that preserve spatial structure and temporal dynamics.The capability is relevant to 4D asset generation and world modeling for embodied AI and robotics.
  • Motivation: Long-context 4D reconstruction remains difficult because temporally extended, sparsely observed videos exceed activation-memory limits and degrade beyond training context lengths.Transformer-based reconstruction models are constrained by the activation memory available for one forward pass.
  • Motivation: LaCT enables chunk-wise fast-weight adaptation without backpropagation, but fully plastic updates can drift, overfit, and become unstable during inference.These problems are especially consequential when scene dynamics evolve across space and time.
  • Approach: Elastic Test-Time Training adds consolidation inspired by elastic weight consolidation, using anchor parameters and importance estimates to stabilize LaCT fast-weight updates.The approach is designed to address uncontrolled drift and inference-time catastrophic forgetting.
  • Approach: FSM scales LaCET through pretraining on curated 3D/4D datasets containing posed images captured over time and from different cameras.The model is evaluated primarily on novel view synthesis, with scaling across data and model size and generalization to novel scenes.
  • Contributions: FSM supports long view sequences, arbitrary timestamps, and arbitrary novel view-time combinations while reducing overfitting and camera-interpolation shortcuts.The stated goal is robust multi-chunk adaptation beyond the bounded single-chunk setting while alleviating activation-memory bottlenecks.

2. Algorithmic Preliminaries

Test-Time Training adapts fast weights during inference to compress key-value information into bounded neural memory, while LaCT improves scalability by aggregating updates over large chunks. LaCET adds Fisher-weighted elastic consolidation and evolving anchors to stabilize multi-chunk adaptation without eliminating plasticity.

  • Fast Weights and Test-Time Training: TTT updates fast weights at training and inference time, unlike fixed slow weights, by learning key-value associations from token projections.Each token is projected into key, query, and value vectors, and the fast-weight function is updated using a loss between transformed keys and values.
  • Fast Weights and Test-Time Training: TTT compresses the linearly growing KV cache into a fixed-size neural memory that preserves critical key-value associations within a bounded memory budget.The updated fast weights are then applied to queries to produce output vectors as the sequence is processed.
  • Large-Chunk Test-Time Training: LaCT improves TTT scalability by aggregating losses across each chunk and sharing one fast-weight state across all queries in that chunk.Large chunks amortize computation and enable richer update-rule designs, but updates occur less frequently.
  • Fast Spatial Memory: FSM uses LaCET to process posed images across time and viewpoints, infer novel view-time combinations, and support scalable long-sequence 4D reconstruction.The architecture maintains anchor and fast weights, using online parameter importance to restore critical weights toward stable anchors.
  • Elastic Test-Time Training: LaCT’s fully plastic inference updates can drift across long, dynamic 4D scenes, causing cumulative instability and making a single chunk work best for novel view synthesis.This motivates stabilizing fast-weight dynamics while retaining their adaptability.
  • Elastic Test-Time Training: LaCET adds an EWC-inspired Fisher-weighted penalty that softly constrains important parameters near anchor weights while allowing less important parameters to adapt.The penalty is applied after the LaCT update; Fisher importance is maintained as an exponential moving average over chunks, and Streaming-EMA updates anchors as a low-pass filter over fast-weight trajectories.

3. Fast Spatial Memory (FSM)

FSM is an end-to-end feedforward model that encodes posed images with spatial, camera, and temporal information, then renders novel view-time combinations through either direct RGB prediction or explicit 4D Gaussian representations.

  • Model architecture: FSM processes posed input images augmented with camera and temporal information as visual tokens for feedforward scene representation learning.Camera parameters are converted into Plücker ray maps, while timestamps provide temporal conditioning.
  • LaCET backbone: Only input-view tokens generate gradients for fast-weight updates, while target-view tokens are synthesized independently to avoid cross-view information leakage.This design keeps each novel view prediction independent and makes inference efficient.
  • Rendering variants: FSM offers LVSM-style rendering that directly predicts target RGB patches without constructing an explicit scene representation.Target query tokens contain camera and temporal metadata, while appearance channels are zeroed before decoding.
  • Rendering variants: FSM offers LRM-style rendering that predicts pixel-aligned Gaussian primitives for an explicit 4D scene representation before rasterizing target views.The representation uses 4D Gaussian Splatting primitives and tile-based rasterization with deferred backpropagation.
  • Training objectives: FSM trains with photometric supervision combining ℓ2 and LPIPS losses, using target-view renders as supervision.The LPIPS weight µ is set to 0.5 empirically.
  • Data: Pretraining combines real and synthetic 3D/4D datasets, retaining static datasets with assigned timestamps because 4D data is limited.The dataset collection includes RealEstate10K, DL3DV, PointOdyssey, Spring, DynamicReplica, Multi-Cam Video, and Stereo4D.

4. Ablation: When and Why Elasticity Helps

The ablations show that elastic consolidation, especially streaming-EMA anchoring, reduces overfitting and improves multi-chunk behavior. LaCET is strongest under sparse inputs and reduces camera-interpolation shortcuts in continuous-view settings while retaining long-range 4D modeling.

  • Anchor policies: A fixed global anchor behaves like an importance-weighted ℓ2 regularizer, whereas streaming anchoring regularizes only within-chunk drift.Streaming without EMA lacks memory consolidation across chunks and is therefore more prone to overfitting.
  • Anchor policies: Streaming-EMA anchoring constrains cumulative fast-weight drift relative to an evolving consolidated anchor rather than the instantaneous update.Its consolidation term acts as a low-pass, importance-weighted constraint on the fast-weight trajectory.
  • Elastic consolidation: EWC reduces LaCT’s overfitting in multi-chunk settings, while streaming-EMA anchoring is critical for stable performance.The full-sequence one-chunk setup makes anchor policies equivalent, whereas four chunks contain 2048 input tokens each.
  • Test-time scaling: The scaling study varies input density on 136-frame Stereo4D clips, contrasting sparse full-span sampling with masked targets inside contiguous subsequences.The continuous-view setting reduces to frame interpolation, and the two settings converge when the full 136-frame span is used.
  • Test-time scaling: LaCET consistently outperforms LaCT under sparse spatial and temporal inputs across PSNR, SSIM, and LPIPS.Both four-chunk models degrade as sparsity increases, while LaCT’s one-chunk setup uses more activation memory and is less sustainable for longer sequences.
  • Generalization: LaCET mitigates camera-pose interpolation shortcuts in continuous-view evaluation and preserves stronger long-range 4D dynamics modeling.LaCT can exploit neighboring frames as a frame-interpolation solution instead of learning view-conditioned spatial representations.

5. Scaling LaCET for Fast Spatial Memory

FSM extends LaCET to pose-conditioned 4D reconstruction and evaluates it on 4D and 3D novel-view synthesis benchmarks. The model achieves strong dynamic-scene results and comparable static-scene performance under resolution-matched evaluation.

  • Architecture and training: FSM-LVSM and FSM-LRM use LaCET blocks for pose-conditioned 4D reconstruction, with the LVSM variant initialized from a DL3DV-pretrained LaCT backbone.The LVSM variant adds temporal encodings and continues pretraining for pose-conditioned 4D reconstruction.
  • 4D Novel View Synthesis: At 256 × 256 resolution, FSM achieves the best performance among feed-forward approaches on the NVIDIA benchmark and approaches the strongest optimization-based methods.The optimization-based methods require per-scene test-time optimization.
  • 4D Novel View Synthesis: On Stereo4D, FSM improves over prior rendering-based methods across all reported metrics.The comparison uses NVIDIA and Stereo4D benchmarks because feed-forward 4D evaluation lacks an established standard protocol.
  • 3D Novel View Synthesis: On DL3DV-140, FSM delivers performance comparable to methods evaluated at similar resolutions, preserving strong static-scene capability.Evaluation uses the minimal 256 resolution for fair comparison because metrics vary with resolution.

6. Related Work

Related work frames fast-weight adaptation and large rendering-based reconstruction models as complementary foundations for long-context 3D and 4D reconstruction. These approaches provide strong learned priors and online adaptation, but long-context modeling remains constrained by memory and temporal scalability.

  • Fast Weights and Test-Time Training: Fast weights are parameters that evolve in context to capture short-term associations and act as associative memories balancing retention and adaptation.Test-Time Training extends fast-weight adaptation to general neural components updated online.
  • Large Rendering-Based Reconstruction Models: Large Reconstruction Models use massive 3D and 4D datasets to learn strong priors over shape and appearance for view-consistent reconstruction from few posed views.Existing implementations use triplane-based NeRFs or Gaussian Splatting to encode these priors.

7. Conclusion and Limitations

The paper presents LaCET as enabling adaptation to longer sequences while acknowledging that the current work does not fully realize that potential. Its scope also assumes posed inputs and focuses on rendering architecture rather than geometrically faithful 4D reconstruction.

  • Scaling to Longer Sequences: LaCET can support fast inference-time adaptation for, in principle, arbitrarily long sequences without activation memory remaining the bottleneck.The paper focuses on architectural advances rather than fully training and scaling a model because of data, benchmark, and compute limitations.
  • Pose Estimation in Dynamic Scenes: The work assumes posed input images and does not target joint camera-intrinsic and pose estimation in dynamic scenes.Such estimation remains challenging when camera motion and scene dynamics occur together.
  • Geometrically Faithful 4D Reconstruction: Rendering quality from NVS alone does not ensure geometrically faithful or temporally consistent 4D reconstruction.The paper notes that LaCET reduces but does not eliminate interpolation of nearby context frames instead of true NVS.
  • Geometrically Faithful 4D Reconstruction: Additional depth, correspondence, multiview-consistency, or optical-flow supervision is left for future work to further mitigate geometric and motion-related limitations.The paper deliberately focuses on architectural aspects under rendering-only supervision.

A.2. Algorithm and Model Architecture

The implementation specifies the elastic test-time training coefficients and the principal dimensions of the LaCET block architecture.

  • Implementation details: Elastic test-time training uses αewc = 0.5, βewc = 0.5, and λewc = 0.5 after grid search.Each block has model dimension 768, a single-head SwiGLU fast-weight MLP with hidden dimension 1536, and a 12-head window-attention module.
  • Implementation details: The feed-forward network uses an intermediate hidden dimension of 3072, while tokenization and decoding use linear projections.The window-attention heads have dimension 64 and use QK-Norm.

A.3. Ablation Study Settings

The ablation study uses a controlled 12-LaCET-block configuration on Stereo4D, with short-window sampling and fixed training resources.

  • A controlled configuration with 12 LaCET blocks is used for the ablation study.
  • Experiments use Stereo4D, whose diverse camera trajectories and static and dynamic motion suit 4D-scene modeling.
  • Training samples 128 consecutive frames, randomly selects 64, and uses 32 input and 32 target views at 128 × 128 resolution for 32K steps.
  • The ablation-study configurations are summarized in Table 5.

A.4. Full-Scale Pre-training Settings

Full-scale FSM pre-training combines synthetic and real data with two-stage resolution training, batch-size-controlled LaCET inference, and decoder and temporal-conditioning ablations.

  • The complete FSM model is trained on a large collection of synthetic and real data.
  • FSM is pre-trained at 128 × 128 for 80K steps and fine-tuned at 256 × 256 for 10k steps, using 32 context and 32 target frames from 128-frame windows.
  • Inference batch size is fixed to 1 because LaCET averages or accumulates fast-weight updates across examples, making batch size affect adaptation dynamics.
  • The study compares LVSM-style and LRM-style decoders, including direct image-token patch reconstruction and explicit 4D Gaussian rendering.
  • Table 6 reports decoder and temporal-conditioning ablations, with additional results provided in Figures 11, 9, and 12.
  • With equal training steps, LVSM-style decoding performs better than explicit 4DGS modeling, while monocular-video training reduces camera-interpolation overfitting but slows convergence.
  • Explicit timestamp-map conditioning outperforms RoPE under the same training budget, while RoPE converges more slowly.

B.5. Failure Cases and Analysis

The reported failure analysis focuses on challenging interpolation settings, where motion consistency and space-time correspondence can break down and produce ghosting.

  • Under large camera or view interpolation, the model may preserve stale gestures or partial motion patterns instead of updating subject motion consistently.
  • These failures include ghosting artifacts, with duplicated structures remaining around moving limbs and bodies.
  • The observed behavior suggests difficulty maintaining accurate space-time correspondence and motion consistency across challenging viewpoints.
  • Additional qualitative comparisons are shown for the Stereo4D test set, NVIDIA benchmark, and DL3DV-140 benchmark.
  • Figure 10 presents a qualitative failure example, while the listed values provide an additional reported result without an accompanying metric label.
Loading 2604.07350v1…