PaperlayerView exact revision on arXiv

Source-linked AI summary

Persistent Planar Memory for Video World Models

Yuze He, Bin Tan, Zelin Gao, Yujun Shen, Yong-Jin Liu, Nan Xue, Yinghao Xu

arXiv:2610.23795v1cs.CV

TL;DR

Video world models need persistent 3D memory, yet existing representations do not combine compactness, explicit geometry, cross-view consistency, and adaptivity. PPMem uses jointly predicted planes as memory for subsequent video chunks, and reports stronger video quality than prior 3D-grounded models on DL3DV and RealEstate10K while sustaining coherence across hundreds of frames under large viewpoint changes.

  • Problem

    Existing 3D-grounded video memories do not simultaneously offer compactness, geometric explicitness, cross-view consistency, and adaptivity for long-context generation.

    Source: §1 · p. 2

  • Method

    PPMem augments a video diffusion transformer with a planar prediction head and Depth Token Fusion, merging jointly generated planes into persistent memory that conditions later chunks.

    Source: §1 · p. 2

  • Results

    PPMem outperforms Gen3C, Voyager, VMem, and SPMem in video quality on DL3DV and RealEstate10K while sustaining video quality and depth accuracy under large viewpoint changes.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The plane pool provides a compact, explicit, cross-view-consistent, adaptive 3D state, with about 100K primitives for a full scene.

    Source: §5 · p. 10

    PPMem is best suited to structured scenes; highly organic environments may be represented with less detail than by point-based alternatives, and extreme unseen camera trajectories can degrade quality.

    Source: §5 · p. 10

Abstract

from arXiv · show

Video world models that generate and explore environments at scale need a persistent 3D memory of what they have produced. Plane primitives offer a natural candidate: a compact set of oriented planes captures the dominant structural elements (e.g., walls, floors, facades) of indoor and outdoor scenes, while requiring only 7 parameters per primitive. We present PPMem, a video world model that adopts plane primitives as its persistent 3D memory. PPMem augments a video diffusion transformer with a planar prediction head and a Depth Token Fusion (DTF) module: each autoregressive chunk jointly produces RGB video and planar geometry, which merges into a persistent plane memory that conditions subsequent chunks via rendered depth. The resulting memory is three orders of magnitude more compact than per-pixel point clouds (~100K primitives, 5MB for a full scene), geometrically explicit (metric depth, normals, exportable surfaces), cross-view consistent (jointly predicted by the shared backbone), and adaptive (refined by the merging procedure as generation proceeds). On DL3DV and RealEstate10K, PPMem outperforms prior 3D-grounded world models in video quality while jointly producing metric-scale depth, surface normals, and exportable planar geometry, sustaining coherence across hundreds of frames under large viewpoint changes. Project page: https://hyzcluster.github.io/ppmem.

1 Introduction

Video world models need persistent 3D memory to generate beyond short clips, but existing representations do not jointly provide compactness, explicit geometry, cross-view consistency, and adaptability. PPMem uses jointly predicted planes as persistent memory and reports state-of-the-art video quality on two datasets while sustaining coherence across long, large-viewpoint-change sequences.

1 Introduction

  • Plane primitives encode structural elements such as walls and floors with 7 parameters each, offering a compact candidate for persistent 3D memory.The proposed memory accumulates planes through cross-chunk geometric merging.Source: §1 · pp. 1–2
  • Existing 3D-grounded representations do not simultaneously provide compactness, geometric explicitness, cross-view consistency, and adaptivity.These properties describe the explicit, persistent state sought for long-context video generation.Source: §1 · p. 2
  • PPMem achieves state-of-the-art video quality on DL3DV and RealEstate10K while accumulating under 10 MB of 3D memory.The evaluation also reports sustained visual and geometric coherence across hundreds of frames under large viewpoint changes.Source: §1 · pp. 2–3
  • PPMem adds a planar prediction head and Depth Token Fusion to a video diffusion transformer, jointly producing RGB, metric depth, surface normals, and planar geometry.Rendered depth from the persistent plane memory is re-injected into the model to condition subsequent generation.Source: §1 · p. 2

2 Related Work

Prior video world models range from camera-controlled generation with little persistent geometry to systems that use depth, point clouds, surfels, volumes, or frame banks. PPMem positions plane prediction as persistent memory for generation, distinct from prior plane methods that reconstruct geometry from already-observed images.

Camera-controlled video generation

  • Camera-controlled video models generate impressive 14–121-frame clips, but longer sequences reveal a representation gap: most lack persistent 3D state or maintain minimal geometry.Existing approaches are organized by the level of 3D representation they maintain.Source: §2 · p. 3

From depth maps to point clouds; surfels, TSDF, and frame-level memory

  • Gen3C keeps only a two-frame point buffer, while VMem, SPMem, and frame-bank methods retain geometry or views with growth, recomputation, or no explicit 3D state.Gen3C overwrites intermediate geometry; SPMem re-estimates depth and reintegrates accumulated frames as sequences grow.Source: §2 · p. 3

Joint depth and 3D generation

  • Related systems generate or use depth and 3D, but none accumulate compact, structured 3D memory for long-context video generation.Voyager conditions with per-frame normalized relative disparity; other approaches include metric-depth output, post-hoc point maps, externally estimated clouds, and static multi-view scenes.Source: §2 · p. 3

Plane-based 3D reconstruction and our position

  • Prior plane methods reconstruct geometry from observed images, whereas PPMem jointly predicts planes with generated chunks and feeds the accumulated pool back as conditioning.This generation-to-planes-to-conditioning loop has no counterpart in the cited reconstruction work.Source: §2 · p. 3

3 PPMem: Persistent Planar Memory

PPMem uses compact plane primitives as persistent 3D memory, merging geometry across autoregressive chunks and rendering it to condition future generation. Its architecture predicts video and geometry jointly, while staged training separates camera-controlled video generation from geometry learning.

3 PPMem: Persistent Planar Memory

  • PPMem accumulates jointly predicted plane primitives across chunks, rendering the persistent pool as depth conditioning for later generation.The pool typically reaches ∼100K primitives (∼5 MB) after ∼400 frames.Source: §3 · p. 4, §3.1 · pp. 4–5

3.1 Plane Primitives and Persistent Memory

  • Each plane primitive is a 7-parameter oriented 3D rectangle, and geometric merging maintains the accumulating pool without gradient-based optimization.Pool maintenance uses occlusion deletion, quality-based deduplication, and forward-looking pruning.Source: §3.1 · pp. 4–5

3.2 Model Architecture

  • PPMem adds a plane prediction head and Depth Token Fusion to Wan 2.2, using rendered pool depth and projected source views as conditioning.DTF tokenizes rendered depth and fuses it with DiT geometry features; masking helps the model handle sparse depth conditioning.Source: §3.2 · pp. 5–6

3.3 Training

  • Training first fine-tunes the video backbone for camera-controlled generation, then freezes it while training the geometry head, DTF, and feature decoder.The second stage uses per-patch depth and normal supervision, rasterization losses, and a depth smoothness regularizer.Source: §3.3 · p. 6

4 Experiments

PPMem is evaluated on short- and long-context video generation, geometric output, and ablations of its memory and prediction components. It achieves the strongest reported video quality across the short-range benchmarks and sustains perceptual fidelity across long sequences, while ablations examine its geometric and depth-conditioning choices.

4.1 Setup

  • Training uses 80K+ DL3DV and RealEstate10K clips; evaluation includes 100 scenes or clips per dataset and full 300–400-frame DL3DV sequences, with video and geometric metrics.Short-range video metrics include PSNR, LPIPS, CLIP, and DINOv2 similarity; geometric metrics include AbsRel, δ<1.25, and Normal◦.Source: §4.1 · p. 6

4.2 Short-Range Comparison

  • PPMem achieves the best results across all reported video metrics on DL3DV and RealEstate10K at both frame budgets, despite using a smaller base model than Gen3C and Voyager.Table 2 compares complete systems; Table 3 isolates the memory mechanism using the same Wan 2.2 generator.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 Long-Context Generation under Large Viewpoint Change

  • At frame 300, PPMem leads the next-best method by ∼0.27 DINOv2 similarity and maintains the highest CLIP and DINOv2 fidelity across full DL3DV sequences.Long-context evaluation uses full test sequences of 300–400 frames with large camera motion across autoregressive chunks.Source: §4.3 · p. 7, Figure 3 · p. 7
  • PPMem preserves scene structure and color consistency across the sequence, while baselines visibly degrade as the viewpoint diverges.The comparison describes VMem collapsing early, SPMem growing vague and drifting, and Gen3C and Voyager losing structural alignment.Source: §4.3 · p. 7, Figure 4 · p. 8

4.4 Ablations

  • The joint plane pool outperforms no persistent memory, point-cloud memory, and post-hoc Plana3R; δ<1.25 is 70.3 versus 64.0/55.0/57.0 across the alternatives.These are inference-only alternatives using the same checkpoint without retraining; the reported gains are particularly large on depth.Source: Table 3 · p. 8, §4.4 · p. 8
  • DTF tracks input depth scale at 1.00 ± 0.01 for scale 1.0 and, with scale augmentation, reaches identity-mapping RMSE 0.131; without depth conditioning, depth accuracy collapses (AbsRel 0.436, δ<1.25 = 30.7%).The scale-faithfulness probe rescales input depth and measures the predicted-to-ground-truth depth ratio; DTF is compared with channel concatenation.Source: §4.4 · p. 9, Figure 5 · p. 9
  • A VAE-initialized decoder raises δ<1.25 from 59.2 to 70.3 and lowers AbsRel from 0.256 to 0.210; removing normal loss collapses depth to an all-zero output.Removing rasterization loss also degrades δ<1.25 by 3.9 percentage points and increases Normal◦ error by 1.2 degrees.Source: Table 5 · p. 9, §4.4 · p. 10

5 Conclusion

PPMem uses a persistent plane pool for long-context generation, and experiments on DL3DV and RealEstate10K report sustained video quality and depth accuracy under large viewpoint changes. The conclusion also identifies scene structure and camera trajectories as scope boundaries.

5 Conclusion

  • On DL3DV and RealEstate10K, PPMem sustains video quality and depth accuracy under large viewpoint changes, outperforming Gen3C, Voyager, VMem, and SPMem.The pool contains about 100K primitives and is refined through geometric merging as generation proceeds.Source: §5 · p. 10
  • The plane pool is compact, geometrically explicit, cross-view consistent, and adaptive, with metric depth, normals, and exportable surfaces.The conclusion reports approximately 100K primitives for a full scene; planes are jointly predicted by the generation backbone and refined through merging.Source: §5 · p. 10
  • PPMem is best suited to structured scenes; planar primitives may capture less detail in highly organic environments, and unseen extreme camera trajectories can degrade quality.The paper identifies joint video-geometry optimization, dynamic planes, and scaling to open-world scenes as future work.Source: §5 · p. 10

A.1 Dataset Preprocessing

Training uses metric-geometry supervision, with a larger mixed-video dataset for the final model than for ablations. An offline incremental plane reconstruction pipeline supplies conditioning with artifacts that resemble those at inference.

A.1 Dataset Preprocessing

  • The final model trains on 80K+ DL3DV and RealEstate10K clips at 576×1024 resolution, while ablations use 5K DL3DV indoor clips at 288×512.Each clip is paired with metric depth, surface normals, and camera intrinsics and extrinsics.Source: §A.1 · p. 14
  • Dense monocular depth and normals are estimated with MoGe-2 and aligned to VGGT multi-view reconstruction using per-frame scale fitting, then globally placed in metric coordinates.The median fitted scale is applied globally to depths and camera translations.Source: §A.1 · p. 14
  • Training conditioning is generated by the same plane-based rendering process used at inference, exposing the model to holes, depth discontinuities, and partial coverage.This design uses realistic rendering imperfections rather than perfect ground-truth projections.Source: §A.1 · p. 14
  • Frames are processed in 24-frame segments; after each segment, planes are merged and jointly optimized, then rendered at the next segment’s cameras for partial-depth conditioning.Two prior-segment source views are selected by greedy union-coverage maximization and forward-warped to target cameras.Source: §A.1 · p. 14
  • Anchor, history, and overlap frames are prepended as clean context tokens; context dropout is applied with probability 0.1 during training.History and overlap frames use depth rendered from the accumulated plane pool; overlap selection maximizes camera coverage.Source: §A.1 · pp. 14–15
  • The DiT receives conditioning through concatenated RGB, visibility-mask, and depth latents, while DTF also supplies depth at pixel resolution through token-level self-attention.The concatenated input comprises six 48-channel latents, for 288 channels total.Source: §A.1 · pp. 14–15

A.2 Training Hyperparameters

Stage 2 freezes the video backbone and trains geometry-specific modules with LoRA adapters. Plane-head initialization and a rasterization-loss warmup stabilize early geometry training.

A.2 Training Hyperparameters

  • Stage 2 freezes the DiT backbone, injects LoRA adapters on self-attention, and trains only geometry-specific modules and LoRA.Stage 1 uses similar hyperparameters but has no plane loss or DTF and does not use freezing or LoRA.Source: Table S1 · p. 15
  • The plane-head MLPs begin with constant outputs: 2.0 m depth, 0.1 m extents, and an identity rotation quaternion.The depth and extent layers start with zero weights; the rotation weights are near zero.Source: §A.2 · p. 15
  • During the first 1000 rasterization-loss warmup iterations, depth and normals receive per-patch supervision while extents remain at their initialized constant.The extents receive no direct gradient during warmup, supporting stable rasterization before rendering loss is enabled.Source: §A.2 · p. 15

A.3 Inference Setup

PPMem generates video in autoregressive 25-frame chunks and updates a plane pool that conditions subsequent chunks. The merging procedure maintains that pool, while its memory footprint remains small relative to a matched point-cloud variant.

A.3 Inference Setup

  • PPMem generates video autoregressively in 25-frame chunks, reusing each chunk’s last frame as the next chunk’s first.The pipeline takes an input image, text prompt, and target camera trajectory.Source: §A.3 · p. 15
  • Each subsequent chunk renders the cumulative pool for depth conditioning, generates video with the frozen Stage 1 backbone, then predicts planes with an additional Stage 2 LoRA pass.Predicted planes are transformed into world coordinates using camera extrinsics and merged into the pool.Source: §A.3 · p. 16
  • Pool rasterization takes about 1.7 seconds for 25 views and merging about 0.3 seconds, compared with approximately 128 seconds for chunk denoising.The reported rasterization and merging costs are two orders of magnitude cheaper than denoising.Source: §A.3 · p. 16
  • At a long-rollout snapshot, the pool uses 4.8 MiB for 106K planes, versus 1.7 GiB for a matched per-pixel point-cloud variant.Maximum simultaneous nonmodel state is 41 MiB including the pool; fixed geometry-related model state adds 1.1 GiB.Source: §A.3 · pp. 16–17

A.4 Plane Pool Merging Algorithm

  • The merge uses an observation score s = r/|cos θ|, where lower scores indicate closer, more frontal observations and thus more reliable planes.Here r is distance to the creating camera and θ is the angle between the viewing ray and the surface normal.Source: §A.4 · p. 17
  • Merging removes floaters, rejects boundary planes, deduplicates nearby planes, filters distant planes, replaces lower-quality cross-chunk planes, and prunes planes likely to be viewed closer next.Within-chunk deduplication removes approximately 80% of per-chunk planes; default thresholds include 0.5 m for nearby-plane matching and 5 m for the distance filter.Source: §A.4 · p. 17

B.1 Context Configurations

The context-frame ablation finds that overlap frames recover most of the no-context depth-accuracy loss, while history and anchor frames improve perceptual and video-quality metrics. Increasing history beyond five frames yields only marginal gains as sequence length and attention cost grow.

B.1 Context Configurations

  • Overlap frames are past RGB frames selected by target-viewpoint coverage; history comes from preceding chunks, and the anchor is frame 0.Configurations were evaluated at inference without retraining, using context dropout applied during training.Source: §B.1 · p. 17, Table S2 · p. 18
  • Adding overlap, history, and anchor frames achieves the best balance across metrics; removing all context lowers δ<1.25 from 70.3 to 64.2.Overlap alone recovers most of the gap (+5.0 pp) and gives the lowest AbsRel (0.208).Source: §B.1 · p. 17, Table S2 · p. 18
  • Increasing history beyond the default of five frames raises PSNR only from 14.91 to 14.94 while increasing prefix length and attention cost.The authors retain five history frames as the quality/compute tradeoff.Source: §B.1 · p. 17, Table S2 · p. 18
  • The online accumulated plane pool stays consistent across views, unlike pairwise Plana3R reconstructions, which conflict and fragment.Figure S1 renders depth from both sources on the same generated video.Source: Figure S1 · p. 18

B.2 Plane Pool Merging Phase Ablation

The supplementary ablations examine plane-pool merging, scene planarity, and comparisons with non-persistent-memory baselines. Merging sharply reduces plane count while retaining depth accuracy, and lower planarity is associated with lower PSNR and depth accuracy.

B.2 Plane Pool Merging Phase Ablation

  • Within-chunk dedup cuts per-chunk planes by ∼4.5× (518K to 116K) while improving δ<1.25 on indoor and outdoor scenes; later additions improve δ<1.25 monotonically.Boundary rejection applies one global threshold across both scene types, trading some coverage for stricter geometry filtering.Source: §B.2 · p. 18, Table S3 · p. 18
  • Across 56 scenes, the accumulated pool compresses per-chunk predictions to ∼100K planes, with 0.241 vs. 0.210 AbsRel and 98.7% image coverage.The pool is evaluated over the complete generated trajectory; Figure S1 compares it with Plana3R reconstructions of the same videos.Source: §B.2 · p. 18

B.3 Scene Planarity

  • Across 100 DL3DV scenes, planar coverage has a Spearman correlation of 0.33 with PSNR at 121 frames; lower planarity is associated with lower PSNR and depth accuracy, while LPIPS varies mildly.Planarity is computed independently of PPMem predictions by grouping valid-depth pixels into connected planar regions; scenes are grouped into within-domain tertiles.Source: §B.3 · p. 19, Table S4 · p. 19

B.4 Comparison to Non-3D-Persistent Baselines

  • WorldPlay and MemCam are evaluated on the same DL3DV set as Table 2, with matching frame budgets; both use historical-frame context without explicit persistent 3D representation.The supplied comparison passages describe the evaluation setup, not comparative outcomes.Source: §B.4 · p. 19, Table S5 · p. 19

C Additional Qualitative Results

Additional qualitative results show PPMem maintaining scene structure across long sequences and large viewpoint changes, with a specific weakness in fine text during zoom-in. The examples cover outdoor rooftops and indoor stores.

C Additional Qualitative Results

Related papers

Loading 2610.23795v1…