PaperlayerView exact revision on arXiv

Source-linked AI summary

Long-horizon Video Generation with Slot Memory Addressed By Region Tokens

Eren Cetin, Julius Erbach, Vukasin Bozic, Hyojun Go, Zixiang Zhao, Nikolai Kalischek, Federico Tombari, Konrad Schindler, Dominik Narnhofer

arXiv:2610.19257v1cs.CV

TL;DR

Long-horizon video generation needs bounded memory that can recall a place when the camera returns, despite the limits of growing geometric caches and recency-based recurrent memory. SMART uses pose-addressed slots and confidence-conditioned video diffusion; across Memory Maze and Minecraft, it preserves scene-specific consistency beyond temporal-context baselines without growing storage.

  • Problem

    Long-horizon video generation must recall scene content across revisits, while explicit geometric memories grow with explored scenes and recurrent memories retain information according to recency.

    Source: §1 · p. 2

  • Method

    SMART writes and reads a fixed-capacity slot recurrence using camera-pose addresses, then conditions a video diffusion model on retrieved content and confidence.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Across Memory Maze and Minecraft, SMART preserves scene-specific consistency over substantially longer horizons than temporal-context baselines while avoiding growing storage and reconstruction dependence.

    Source: §4.2 · p. 7, §6 · p. 10

  • Takeaways & Limitations

    SMART combines constant-size recurrent storage with spatially indexed recall so retention depends on where content was seen rather than when.

    Source: §6 · p. 9

    Fixed memory capacity bounds the spatial extent retained, and sufficiently large environments can force regions to reuse slots and overwrite stored content.

    Source: §6 · p. 10

Abstract

from arXiv · show

Memory for long-horizon video generation must satisfy two competing requirements: it must remain bounded so generation can continue indefinitely, while still recovering what was observed at a place when the camera returns, regardless of how long ago it was seen. Reconstruction-based memories provide spatial recall by caching and reprojecting explicit geometry, but require growing storage and depend on estimated 3D structure. Recurrent memories keep a constant-size state, but retention is governed by temporal recurrence, causing old observations to fade with time. We introduce SMART, a slot memory addressed by region tokens, which combines bounded storage with spatially addressed recall. SMART stores content in a constant-size slot recurrence whose writes are addressed by the observing camera and whose reads are queried by the rendering camera, without constructing an explicit 3D representation or maintaining a growing cache. Consequently, recall is conditioned on inter-view geometry rather than on recency. Each read additionally provides a confidence estimate that conditions a chunk-causal diffusion transformer, allowing it to selectively trust retrieved content while relying on its generative prior elsewhere. On long trajectories in Memory Maze and Minecraft, SMART maintains high-fidelity recall far beyond its training horizon while using constant memory and per-step cost.

1 INTRODUCTION

Long-horizon video generation must preserve scene consistency across revisits without conditioning on costly full histories. SMART addresses this with bounded memory indexed by camera geometry and uses its retrieval to condition video generation.

1 INTRODUCTION

  • Long-horizon models struggle to preserve scene geometry when an agent retraces its path, motivating a persistent representation of prior observations.Such a representation can summarize past observations and provide information relevant to the current viewpoint.Source: §1 · p. 2
  • Explicit spatial memories support location-based recall but generally require storage that grows with the scene; recurrent memories stay constant-sized but forget according to recency.Explicit methods also depend on reconstructed geometry, while recurrent methods update the state over time and old observations fade.Source: §1 · p. 2
  • SMART combines fixed-capacity learned memory with spatial addressing, using camera pose to determine where observations are written and retrieved.A target-view query returns a coarse representation and confidence estimate without requiring a full 3D scene reconstruction in memory.Source: §1 · p. 2
  • A memory-conditioned diffusion model refines retrieved content into video and relies on its generative prior where memory is uninformative.The contributions pair geometry-guided writing, retrieval, and novel-view decoding with a retrieved view used as a coarse draft.Source: §1 · p. 2

2 RELATED WORK

Prior video-generation memories either retain temporal context or use explicit spatial representations. SMART instead performs camera-pose-based geometric addressing within a recurrent memory.

2 RELATED WORK

  • Camera-control methods condition generation on viewing geometry but leave open what the model retains from earlier steps.These methods use Plücker rays, geometry-aware attention, or relative-camera positional encodings.Source: §2 · pp. 2–3
  • Temporal-context methods reduce long-horizon generation cost, but recall remains indexed by recency, so old regions may be compressed or dropped even when revisited.Examples include FramePack’s bounded context and MilliVid’s hierarchical latents and coarse-to-fine rollout.Source: §2 · p. 3
  • Explicit spatial memories retrieve information relevant to the current camera, but rely on reconstructed geometry or caches that must be maintained as scenes grow.SMART instead retains spatially relevant information in a fixed-capacity learned state.Source: §2 · p. 3
  • Recurrent states can compress long sequences and selectively update slots, but prior geometric-query approaches either require explicit 3D supervision or lack low-cost recurrent memory.CUT3R predicts pointmaps from recurrent state, while GIM-World uses camera-queryable geometry supervision.Source: §2 · p. 3
  • SMART places geometric addressing inside the recurrence and decodes a target-camera query into a coarse full-frame draft for a chunk-causal video diffusion model.The generator is conditioned only on information relevant to the target view.Source: §2 · p. 3

3 METHOD

SMART uses pose-derived addresses to route writes and query a fixed-capacity slot memory, then conditions chunk-causal video generation on its retrieved draft and confidence. Its training separates memory retrieval from generator training.

3.1 PRELIMINARIES

  • Slot recurrences provide addressable state, unlike dense state-space summaries whose recoverable information diminishes with temporal distance.A slot read matches a query against stored keys and returns a weighted combination of slot values.Source: §3.1 · pp. 3–4

3.2 SPATIALLY-STRUCTURED MEMORY DESIGN

  • Each latent frame is compressed into tiled write tokens, while pose-derived region addresses encode camera geometry independently of content.Address queries and keys match by viewing geometry; tiled tokens summarize fixed image regions.Source: §3.2 · pp. 4–5
  • Each write routes to one pose-matched slot, and unselected slots remain unchanged; canonicalized values allow content to be re-posed when queried from another camera.Forgetting is tied to writes at the same address rather than elapsed time.Source: §3.2 · p. 5
  • A target-camera query retrieves and re-poses slot content, which is decoded into a coarse view estimate and uncertainty for the generator.The target view can pull relevant information from the readout through cross-attention.Source: §3.2 · p. 5

3.3 MEMORY-CONDITIONED VIDEO GENERATION

  • The diffusion transformer receives the current latent, memory draft, and confidence mask, while chunk-causal attention limits each denoising step to clean context and the current chunk.The step cost depends on context length W and chunk length N, not on total video length.Source: §3.3 · pp. 5–6

3.4 TRAINING

  • Training first fits the memory as a regressor on recall, novel-view, past/current-frame, and held-out-span queries, then trains the transformer on generation rows with corrupted histories and drafts.The memory’s confidence-aware regression permits lower confidence where the answer is unknowable; generator training also prepares for imperfect self-generated context.Source: §3.4 · p. 6

4 EXPERIMENTS

Experiments evaluate SMART on revisiting mazes and action-conditioned Minecraft trajectories. Across both, it maintains stronger long-horizon reconstruction fidelity than temporal-context baselines, with competitive compute and generation metrics.

4 EXPERIMENTS

  • Memory Maze tests enforced revisits at 64 × 64, while Minecraft tests higher-resolution, action-conditioned generation on open-terrain trajectories at 256 × 256.Memory Maze is the primary testbed; Minecraft tests whether the mechanism holds at scale.Source: §4 · p. 6
  • The evaluation paths close loops over context so reconstruction metrics primarily measure recall of observed content, although the environments use different backbones, autoencoders, and resolutions.Memory Maze retraces the observed trajectory in reverse; Minecraft continues along the action-conditioned path.Source: §4 · p. 7

4.1 IMPLEMENTATION

  • SMART is instantiated with a GSA slot recurrence using Mamba-2 decay and two diffusion-transformer backbones for the distinct generation settings.The Memory Maze backbone is camera-conditioned and Wan2.1-style; Minecraft uses an action-conditioned DiT-B initialized from MilliVid’s released rollout checkpoint.Source: §4.1 · p. 7

4.2 3D CONSISTENT VIDEO GENERATION

  • Memory Maze comparisons include full-history and windowed causal attention, DFoT, GeoCache, and a capacity-matched recurrent memory; Minecraft comparisons use released MilliVid and FramePack checkpoints.The Minecraft comparisons also include FramePack’s 7-frame full-resolution rollout.Source: §4.2 · p. 7
  • Across Memory Maze and Minecraft, SMART maintains stronger long-horizon reconstruction fidelity than temporal-context baselines and uses substantially less compute than the strongest long-context alternatives.On Minecraft it achieves the highest PSNR and SSIM across horizons and best LPIPS in three of four bands, but trails MilliVid on FID and intermediate-horizon FVD at roughly 2.7× lower compute.Source: §4.2 · p. 7

5 ANALYSIS & ABLATIONS

SMART routes writes and reads by camera pose, separating memory decay from elapsed time. Its slot reads show pose-local concentration, while ablations indicate that sparse routing and viewpoint canonicalization support longer-horizon recall.

Analysis

  • Region-token routing makes slot decay depend on being re-routed, rather than simply on elapsed time.Figure 7 compares forward-frame distance with the probability that region tokens land in the same slot.Source: §5 · p. 7, Figure 7 · p. 9
  • 1.7× uniform mass goes to slots holding the query’s place, while some heads localize and others retain global attention.The read is a softmax-attention convex combination of write tokens, making each query attributable to source frames.Source: §5 · p. 8, Figure 7 · p. 9

Ablations

  • A dense state of the same size matches SMART at training length but degrades beyond it, whereas SMART maintains quality to eight times that horizon.Sparse writes let a slot remain intact when it receives no further writes.Source: §5 · p. 8
  • Removing the SE(3) value transform lowers quality at every horizon, and halving memory heads reduces retrieval capability in longer sequences.The stored content must be expressed relative to the querying camera to be read from a new viewpoint.Source: §5 · p. 9

6 CONCLUSIONS

SMART combines constant-size recurrent storage with spatially indexed recall, updating slots for observed regions and canonicalizing stored values across viewpoints. Across Memory Maze and Minecraft, it preserves scene-specific consistency over longer horizons than temporal-context baselines; fixed capacity and modest generator backbones bound the current design.

Conclusions

  • SMART combines constant-size recurrent storage with spatially indexed recall, updating observed-region slots and canonicalizing stored values across viewpoints.A chunk-causal diffusion model refines the retrieved view as a coarse draft, using stored content where available and its generative prior elsewhere.Source: §6 · p. 9
  • Across Memory Maze and Minecraft, SMART preserves scene-specific consistency over substantially longer horizons than temporal-context baselines while avoiding growing storage and reconstruction dependence.Source: §6 · p. 10

Limitations & Future Work

  • Fixed capacity bounds the spatial extent retained: in sufficiently large environments, regions reuse slots and previously stored content is overwritten.The experiments also use relatively modest generator backbones; scaling to larger video models remains a next step.Source: §6 · p. 10

A.1 RELATION TO LINEAR RECURRENT MODELS

SMART builds on fixed-size gated-slot recurrence but changes what its memory is addressed by and asked to produce. Camera pose determines routing and queries, while frame content supplies values for target-view drafts.

A.1 Relation to Linear Recurrent Models

  • SMART builds on Raven’s fixed-slot state architecture and gated-slot attention recurrence.Each head keeps a fixed number of slots updated by a recurrence with learned decay, while a router assigns each token to slots it writes.Source: §A.1 · p. 14
  • Unlike recurrent sequence models that derive queries, keys, values, and routing from token state, SMART derives them from camera pose and frame content.Keys, queries, and routing come from pose; values come from frame content to support pose-aligned slots rather than content-similarity associations.Source: §A.1 · p. 14
  • A token writes one slot that stays stable until its place is observed again, and stored values are re-posed by the target camera when read.The output is a target-view draft read from state, not a next-token prediction; state updates after the chunk is complete.Source: §A.1 · p. 14

A.2 DATASETS

The evaluation uses Memory Maze episodes with enforced returns and Minecraft trajectories extending beyond the released test horizon. Training and evaluation data are prepared with camera poses derived from agent position and heading.

Memory Maze

  • Memory Maze evaluation uses custom 2N+21-frame episodes in a 15×15 environment: an N-frame coverage sweep, a 21-frame in-place turn, and an N-frame return leg.The return retraces positions actually walked; frames are 64 × 64 RGB from a rigidly mounted first-person camera.Source: §A.2 · p. 14
  • Memory Maze training re-renders released episodes through the same camera and cuts them into overlapping 401-frame snippets.The episodes include 29,000 training and 1,000 evaluation examples; goal spheres and target rings are hidden.Source: §A.2 · p. 14

Minecraft

  • Minecraft evaluation uses 2,048-frame trajectories, with 256 ground-truth context frames and 1,792 generated frames, extending beyond the released 1,024-frame test split.The continuations have no turn or return leg; frames are rendered at 512 × 512 and downsampled to 256 × 256.Source: §A.2 · p. 14
  • Minecraft training uses 601-frame rollouts, with camera poses derived from simulator agent position and heading.Source: §A.2 · p. 14

A.3 SMART IMPLEMENTATION DETAILS

SMART compresses each frame into a small sequence of content tokens and uses camera pose to address recurrent memory. The generator queries that memory for a confidence-weighted draft, then refines and writes back generated chunks.

A.3 SMART IMPLEMENTATION DETAILS

  • Each frame’s patch grid is compressed into J learned-query write tokens, with rotary spatial keys and causal attention making each token prefix a self-contained frame code.The tokens are laid out frame-major for recurrent memory updates.Source: §A.3 · p. 15
  • Camera pose supplies separate addressing for memory keys, queries, and routing, while content tokens provide the values written to memory.Read-only operations leave the recurrent state unchanged.Source: §A.3 · p. 15
  • Value transport places stored content in a camera-independent frame and applies the relative camera transform when the memory is read from a new view.Keys are not transformed because their moving average may combine writes from multiple cameras.Source: §A.3 · p. 15
  • A single frozen autoencoder serves every latent-space method within each dataset, so pixel metrics share a common ceiling.Memory Maze and Minecraft use different autoencoders.Source: §A.3 · pp. 15–16
  • The target-camera query returns a draft and per-cell confidence that are concatenated with the noisy latent as aligned draft and trust channels before denoising.After denoising, the generated chunk is written back into memory.Source: §A.3 · pp. 16–17
  • Both datasets use the same memory interface and N=3-frame chunks with W=6 clean context frames, while SMART is paired with different dataset-specific diffusion-transformer backbones.The Memory Maze and Minecraft instantiations use Wan-style and MilliVid backbones, respectively.Source: §A.3 · p. 16, Table 4 · p. 17

A.4 TRAINING FORMULATION

The memory is trained as a deterministic regressor that writes observed prefixes and answers camera-pose queries. Training samples revisit, fusion, and other query types, then trains a frozen-memory video generator on causal chunks.

A.4 TRAINING FORMULATION

  • The memory regressor writes each observed prefix into its state and returns a latent for a sampled query pose, with read-only queries parallelized through a scan.Training pre-encodes 401-frame Memory Maze and 601-frame Minecraft clips into 101 and 601 latent frames, respectively.Source: §A.4 · p. 17
  • 10% of targets are revisits, 45% fusion targets, 2% current-frame round trips, and 42% uniformly sampled previously written frames.Revisits test recall of an earlier view; fusion targets require combining stored viewpoints.Source: §A.4 · p. 17
  • 24 of 101 latent frames are withheld from writing, and about half of queries target spans the memory never saw.Those withheld positions are scored with plain squared error rather than confidence-weighted error.Source: §A.4 · p. 17
  • The address objective combines pose recovery with variance and covariance regularization over batched memory addresses.The variance and covariance terms use weights 0.1 and 0.001; pose uses weight 1.Source: §A.4 · p. 18
  • The video generator is trained on a frozen memory, predicting velocity for N=3-frame chunks using the preceding W=6 clean frames and memory draft.Training corrupts some context and draft frames, and sometimes replaces the whole draft with noise.Source: §A.4 · p. 18
  • Memory Maze trains a bidirectional model before memory-conditioned finetuning, whereas Minecraft initializes from MilliVid’s released rollout checkpoint and trains on sampled history boundaries.At inference, generated chunks are written back to memory in both dataset-specific procedures.Source: §A.4 · p. 18

A.5 3D CONSISTENT VIDEO GENERATION EVALUATION

SMART uses a fixed-size memory across both datasets and is evaluated against adapted baselines on shared episodes and autoencoders. Additional experiments report stronger long-horizon recall qualitatively and contrast pose-conditioned recall with action-conditioned memory.

A.5 3D CONSISTENT VIDEO GENERATION EVALUATION

  • SMART’s memory contains 98,304 floats on both datasets at every horizon, providing fixed capacity for an unbounded generation horizon.The compared FramePack and MilliVid context budgets are also 98,304 floats.Source: §A.5 · pp. 19–20
  • Baselines are adapted to the benchmarks and scored on the same episodes, with a shared autoencoder used for decoding.Memory Maze baselines are trained from scratch; Minecraft uses released rollout code unchanged.Source: §A.5 · p. 19
  • Doubling memory parameters improves Memory Maze reconstruction by 0.3 dB at 401 frames and 1.3 dB at 3,201 frames.The comparison concerns reconstruction quality of the memory alone.Source: Table 5 · p. 20

A.6 ADDITIONAL EXPERIMENTS.

  • On Memory Maze, SMART preserves previously observed geometry, layout, and appearance more consistently over the return trajectory, while baselines show increasing drift and appearance changes.The degradation becomes particularly visible in later frame ranges.Source: §A.6 · p. 20
  • The action-conditioned memory loses 7.5 dB of PSNR on seen regions as trajectory length grows, while SMART improves as more of the maze is observed.Because scoring is limited to regions seen during the forward leg, the passage attributes the loss to recall rather than invention.Source: Table 6 · p. 20, §A.6 · p. 21
  • At the 2048-frame Minecraft horizon with 256 initial context frames, only SMART faithfully retrieves the sun’s position.The figure compares generated frames from the methods in Table 2 on the same trajectory.Source: Figure 11 · p. 22
Loading 2610.19257v1…