PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Spatial Memory for Video World Models

Weijie Wang, Haoyu Zhao, Yifan Yang, Feng Chen, Zeyu Zhang, Yefei He, Zicheng Duan, Donny Y. Chen, Yuqing Yang, Bohan Zhuang

arXiv:2606.09828v1cs.CV

TL;DR

Video world models struggle to maintain 3D spatial consistency, while RGB point-cloud memories are costly and lossy. Mirage stores and queries scene information directly in diffusion latent space, achieving state-of-the-art or competitive quality across benchmarks with up to 10.57× faster generation and 55× lower cache memory.

  • Problem

    Video world models need persistent spatial memory to prevent geometric drift and maintain 3D consistency across generated frames.

  • Method

    Mirage lifts latent tokens into 3D using depth-guided back-projection and retrieves target-view latent tensors through direct latent-resolution projection.

  • Results

    Up to 10.57× faster generation and 55× less GPU memory accompany state-of-the-art or competitive performance across WorldScore and RealEstate10K.

  • Takeaways & Limitations

    Latent spatial memory provides an efficient 3D cache for world-consistent video generation without pixel-space conversion.

  • Takeaways & Limitations

    Mirage excludes moving entities from persistent memory, limiting benefits in scenes dominated by pervasive motion.

Abstract

from arXiv · show

Video world models that maintain 3D spatial consistency across generated frames typically rely on explicit point cloud memory constructed in RGB space. This design is both computationally expensive, requiring repeated rendering and VAE encoding, and inherently lossy, as the round trip through pixel space discards rich features of the learned latent representation. In this paper, we introduce \emph{latent spatial memory} for video world models, a persistent 3D cache that stores scene information directly in the diffusion latent space, avoiding pixel-space reconstruction. Building on this, we propose Mirage, a latent-space spatial memory framework that constructs the memory by lifting latent tokens into 3D via depth-guided back-projection and queries it by synthesizing novel views through direct latent-space warping. This unified formulation eliminates both the information loss of pixel-space reconstruction and the computational burden of repeated encoding and rendering. Experiments show that latent spatial memory achieves up to \textbf{10.57}$\times$ faster end-to-end video generation and \textbf{55}$\times$ reduction in memory footprint relative to explicit 3D baselines. Leveraging the geometric prior of the diffusion model, Mirage attains state-of-the-art performance on WorldScore and strong reconstruction quality on RealEstate10K.

1 Introduction

Video world models need persistent 3D memory to prevent geometric drift, but RGB point-cloud pipelines incur costly latent–pixel round trips. Mirage addresses this with latent spatial memory, enabling geometrically consistent generation while improving speed and memory efficiency.

  • Motivation: Without explicit spatial memory, video generators accumulate geometric drift despite producing individually convincing frames.This drift undermines 3D spatial consistency in world-simulation settings.
  • Motivation: RGB point-cloud memory enforces multi-view consistency but repeatedly renders and re-encodes observations, creating a computational bottleneck.The pipeline performs a latent-to-pixel-to-latent round trip at every conditioning step.
  • Latent spatial memory: Latent spatial memory stores full-channel diffusion latent tokens at world-space locations, constructed by depth-guided back-projection from an encoded frame.This avoids pixel-space conversion while retaining a persistent 3D cache.
  • Mirage: Mirage generates long, geometrically consistent rollouts through an initialize-readout-update cycle with depth-guided construction, occlusion-aware latent-resolution readout, and iterative refinement.Dynamic object exclusion is part of the iterative refinement process.
  • Results: 10.57× end-to-end faster and 55× less GPU memory in 3D cache than RGB point-cloud readout are reported for Mirage.The method achieves state-of-the-art world generation on WorldScore and competitive novel view synthesis on RealEstate10K.

2 Related Work

Prior work advances video diffusion through temporal modeling and explicit camera conditioning, while spatial-memory methods preserve consistency by maintaining persistent 3D scene representations. However, independently generated clips lack shared spatial memory, limiting long-horizon exploratory world generation.

  • Video diffusion: Video diffusion progressed from temporally extended image architectures to photorealistic, multi-second generation through large-scale pretraining on diverse video corpora.Early methods used temporal attention or 3D convolutions to model inter-frame dependencies.
  • Camera-conditioned generation: Camera-conditioned video diffusion enables controllable viewpoint changes using control modules, epipolar attention, or 3D-aware rendering signals.These approaches provide fine-grained camera control within a single generation pass.
  • Camera-conditioned generation: Independent clip generation can introduce inconsistencies when revisiting regions or extending trajectories beyond the model’s context window.The absence of shared spatial memory limits long-horizon, exploratory world generation requiring 3D coherence across sequential rounds.
  • Explicit spatial memory: Explicit spatial-memory pipelines lift RGB frames into depth-based point clouds, accumulate a persistent 3D cache, and render target-view conditioning images.This strategy is used to maintain spatial consistency across generation steps.

3 Preliminaries

Video world models generate multi-view-consistent sequences autoregressively in VAE latent space, conditioning each latent chunk on preceding frames and camera poses. Persistent RGB point-cloud memory extends geometric context but introduces costly pixel-space processing and representation mismatch for latent-space generation.

  • Video world model formulation: A video world model synthesizes a multi-view-consistent sequence together with a camera trajectory of frame extrinsics and intrinsics.The camera trajectory is represented as {(E_t, K_t)}, where E_t and K_t denote extrinsics and intrinsics.
  • Video world model formulation: Modern systems use a pretrained VAE-latent video diffusion backbone that autoregressively denoises overlapping latent-frame chunks conditioned on their predecessors.The VAE is characterized by encoder E, decoder D, spatial stride s, and latent channel count C.
  • RGB spatial memory: Persistent RGB point clouds record observed surface colors and positions, providing a long-term geometric scaffold beyond the autoregressive context.The cache is initialized from I_0 through depth-guided back-projection and expanded during rollout.
  • RGB spatial memory: RGB-cache conditioning rasterizes the target view and re-encodes it into a latent tensor, forcing a full pixel-space round trip at every target pose.Rasterization uses z-buffered projection followed by shading before VAE encoding.
  • RGB spatial memory: Pixel-space storage is costly because rasterization and VAE encoding run at pixel resolution, while RGB reconstruction can deviate from the generator’s native latent representation.These operations become increasingly expensive as the cache grows and introduce reconstruction, rasterization, and surrogate-latent errors.

4 Method

Mirage represents scene memory as latent-attributed 3D points and generates videos through repeated latent-space readout, denoising, and cache updates. It constructs memory by depth-guided back-projection, queries target views with visibility-aware z-buffering, and filters updates to preserve persistent scene content.

  • Latent spatial memory: Mirage represents memory as 3D points pairing world-space coordinates with VAE latent features in the diffusion backbone’s native input space.Each point stores a coordinate p_i and latent feature f_i directly from the VAE encoder.
  • Memory construction: Each latent-grid cell is back-projected using depth, intrinsics, and camera pose, producing one memory element per cell.The method downsamples depth and rescales intrinsics to latent resolution before back-projection.
  • Memory readout: For each target view, Mirage projects memory points onto the latent grid, retains the frontmost point with z-buffering, and retrieves its latent token.Empty cells are zero-filled, while a binary visibility mask identifies cells receiving projected points and distinguishes unseen regions from observed zero-valued features.
  • Memory readout: Mirage concatenates latent memory readouts and visibility masks into a ControlNet-style branch that injects memory features into the video diffusion backbone.When queried from the source view, the readout preserves corresponding source-view latent tokens on visible cells up to discretization and occlusion.
  • Autoregressive update: After each generated chunk, Mirage re-encodes clean VAE latents and back-projects valid static content into the cache while filtering dynamic objects and sky regions.The filter prevents transient or geometrically unreliable content from contaminating persistent memory, and chunk latents provide short-term temporal context.
  • Training: Training proceeds in two stages: first adapting only the side branch, then jointly optimizing it with rank-64 LoRA adapters on self-attention projections.Both stages use flow matching while preserving the pretrained backbone’s appearance and motion priors.

5 Experiments

Experiments show that Mirage achieves leading world-generation and novel-view results while preserving consistency, and substantially improves efficiency over RGB point-cloud memory. Ablations attribute these gains to latent-space storage, native latent-grid geometry, dynamic filtering, staged adaptation, and robust depth conditioning.

  • World generation on WorldScore: Mirage achieves the highest WorldScore Average Score, outperforming foundation video generators and improving over the memory-augmented Spatia baseline.Its advantage is strongest on the dynamic partition, while it remains competitive on the static partition and leads on 3D and photometric consistency.
  • Novel view synthesis and closed-loop consistency: Mirage achieves the best SSIM and LPIPS on RealEstate10K novel-view synthesis, remains close to Spatia on PSNR, and surpasses view-memory and 3D-aware baselines.Closed-loop evaluation emphasizes long-horizon drift because errors accumulate along a trajectory returning to its starting viewpoint.
  • Efficiency: 10.57× faster end-to-end video generation and 55× less GPU memory distinguish Mirage from the RGB pipeline at matched rollout length.The efficiency gap widens with rollout horizon because RGB rasterization and cache growth scale more rapidly.
  • Ablations: Replacing latent memory with an RGB point cloud decreases Average Score and weakens 3D and photometric consistency.The ablation supports the claim that the RGB detour discards information exploitable when memory remains in latent space.
  • Ablations: Lifting features at pixel resolution degrades 3D consistency, whereas matching geometry to the native latent grid is cheaper and more faithful to the generator.The degradation arises because pixel-resolution upsampling interpolates features outside the backbone’s pretraining distribution.
  • Ablations: Disabling dynamic filtering degrades long-horizon stability, while staged backbone freezing followed by LoRA stabilizes convergence and alternative depth predictors preserve competitive performance.Dynamic filtering prevents stale moving content from persisting; the depth branch treats projected cache features as soft geometric hints and removes severe outliers.

6 Conclusion

The conclusion presents latent spatial memory and Mirage as a video world model operating entirely in the VAE latent manifold, avoiding RGB cache round trips. It also identifies dynamic-region handling as a limitation and future-work direction.

  • Contribution: Latent spatial memory stores the video diffusion model’s latent features at world-space points instead of RGB colors.Mirage is built around this representation as a video world model operating entirely within the VAE latent manifold.
  • Efficiency: A single latent-resolution projection removes the rasterise-and-encode round trip that dominates RGB point cloud cache costs.The decode-and-re-encode pair used to grow the cache is amortised over an entire chunk and excluded from the conditioning loop.
  • Limitations and future work: The dynamic-region filter excludes moving entities because their geometry is unreliable, so persistent memory does not maintain dynamic actors across chunks.Scenes dominated by pervasive motion benefit less than scenes dominated by rigid geometry; persisting dynamic content is future work.

A Geometric Details

This appendix defines the camera, latent-grid, and projection conventions underlying the geometric construction in Eqs. 4–6. It specifies how pixel-resolution geometry is adapted to latent resolution and how valid memory points and cells are selected.

  • Conventions: Camera extrinsics map world points into camera coordinates, with the inverse providing the camera-to-world transform; intrinsics use focal lengths and a principal point.These conventions establish the coordinate transformations and pinhole-camera parameters used throughout the construction.
  • Conventions: The VAE’s spatial stride is s = 16, producing latent resolution h × w = (H/s) × (W/s) from pixel resolution H × W.Latent cells are indexed on this downsampled grid using pixel-centre homogeneous coordinates.
  • Latent-resolution intrinsics: Latent-resolution intrinsics rescale both focal lengths and the principal point by the corresponding per-axis stride ratio, preserving perspective projection after depth downsampling.The notation K is used for the latent-resolution matrix Kℓ when the context is unambiguous.
  • Projection and admissibility: The geometric operators back-project latent cells into world points through camera-ray casting and camera-to-world transformation, then project memory points onto the target latent grid.The admissible cell set retains cells with finite positive depth outside dynamic-object and sky masks.

B Additional Experimental Analysis

The appendix analyzes latent spatial memory’s efficiency, rollout scalability, cache-construction choices, and representational content. It attributes Mirage’s advantages to latent-resolution access that avoids pixel-space encoding and preserves richer scene structure than RGB caches.

  • Asymptotic cost of cache reads: Latent-cache reads cost Θ(N log N + hw), omitting the RGB pipeline’s Θ(ΦE(H, W)) encoder term and shrinking rasterisation by the squared VAE stride.The cache-memory ratio is s2 · (3/C) between the latent and RGB representations.
  • Rollout-horizon scaling: The RGB baseline exhausts GPU memory on trajectories Mirage completes, while latent-cache sorting becomes dominant only much later as the rollout horizon grows.The widening gap reflects the delayed dominance of the N log N sort in latent-cache reads.
  • Per-step timing breakdown: 98%+ of per-step RGB cost on a 257-frame rollout comes from rasterisation and VAE encoding, both absent from Mirage’s conditioning loop.Mirage replaces them with a single latent-resolution projection; the RGB pipeline also decodes at every conditioning step.
  • Depth down-sampling for cache construction: Bilinear depth down-sampling is Mirage’s default because it provides the best projected-cache coverage among tested variants.Depth must match the latent grid; bilinear can smear silhouettes and spawn phantom points, whereas nearest-neighbour preserves edges.
  • What the cache stores: Latent-cache features form semantic clusters for walls, floors, windows, and furniture that RGB point clouds cannot recover, improving 3D and photometric consistency.The visualization projects each per-point feature vector onto its top three principal components and maps them to RGB.

C Implementation Details

Mirage uses a frozen Wan2.2-TI2V-5B backbone with structured latent dimensions and a ControlNet-style conditioning branch for latent readout. Training proceeds in two flow-matching stages, while RealEstate10K preprocessing supplies geometry and masks for rigid-scene memory construction.

  • Conditioning branch: The latent readout is injected through a ControlNet-style side branch sharing patch embedding with the main network and attached at eight transformer layers.Its 48-channel input matches the VAE latent, avoiding a bridging encoder; segment-aware rotary encoding distinguishes noisy target, clean preceding, and clean reference frames.
  • Training: Training uses two flow-matching stages: side-branch-only updates at learning rate 10^-5, followed by joint side-branch and rank-64 LoRA optimization at 10^-4.Stage two applies LoRA to the q, k, v, and o projections of every self-attention layer with α = 64 and dropout 0.05.
  • Data: RealEstate10K clips are preprocessed for metric depth, intrinsics, extrinsics, entity extraction, and foreground-dynamic and sky masks.Cells inside the masks are excluded from Λt so persistent memory contains only geometry compatible with the rigid-scene assumption.
Loading 2606.09828v1…