PaperlayerView exact revision on arXiv

Source-linked AI summary

4LC: 4D Latent Cache for Efficient Conditioning in Dynamic World Modeling

Jianhui Zhang, Zuyao Lin, Pengxin Xu, Jingdong Wang, Xingyu Chen

arXiv:2610.17625v1cs.CVcs.AI

TL;DR

Repeated generation in a dynamic scene calls for conditions that preserve static and dynamic content across times and viewpoints, but existing pipelines can require recurring rendering and re-encoding. 4LC stores multi-encoder representations bound to shared geometry and motion, then reads them at requested times and viewpoints; a single 4LC-Video checkpoint achieves state-of-the-art control accuracy and scene consistency across five tasks without task-specific fine-tuning.

  • Problem

    Existing geometric conditioning pipelines render and re-encode RGB for each request, while prior latent caching excludes dynamic content and uses features in a specific generator’s latent space.

    Source: §1 · p. 2

  • Method

    4LC stores multiple encoders’ representations of static and dynamic content with shared geometry and motion, and 4LC-Video uses its flexible readouts to condition a shared video generator.

    Source: §1 · pp. 2–3

  • Results

    A single 4LC-Video checkpoint achieves state-of-the-art control accuracy and scene consistency across five tasks; 4LC accelerates condition acquisition by up to 68.5× with 12.6× lower peak GPU memory.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    Direct cache readout turns condition acquisition from a recurring cost into one-time encoding of observations while supplying efficient, rich conditions for dynamic world modeling.

    Source: §5 · p. 10

    4LC relies on feed-forward 4D reconstruction estimates of depth, cameras, and motion, whose errors can affect readout quality; 4LC-Video reuses only static historical content across long-term generation segments.

    Source: §5 · p. 10

Abstract

from arXiv · show

For video generation in dynamic world modeling, observed static and dynamic content should be efficiently reused to provide conditions for repeated generation at different times and viewpoints. We introduce 4LC, a 4D latent cache with composite visual representations and a flexible spatiotemporal access interface. As a latent cache, 4LC directly reuses visual representations of static and dynamic content to provide generation conditions, avoiding per-request RGB rendering and re-encoding. As a composite cache, 4LC binds visual representations from multiple visual encoders (e.g., VAE, DINO, V-JEPA, VGGT) to shared geometry and motion, allowing the composite representations to describe the dynamic world with rich information. Besides, we design a flexible spatiotemporal interface, which directly reads the cache at independently specified times and viewpoints and combines readouts across observations, providing a common conditioning format for diverse downstream tasks. Building on 4LC, we develop 4LC-Video to address multiple video generation tasks through different configurations of source observations, spatiotemporal requests, and historical reuse, including camera control, temporal control, novel-view synthesis, scene revisits, and long-term generation. Our experiments show more efficient condition acquisition, effective spatiotemporal controllability, and scene consistency across tasks using a single 4LC-Video model without task-specific fine-tuning.

1 Introduction

Repeated video generation needs conditions that preserve static structure and dynamic content across requested times and viewpoints. 4LC reuses composite cache readouts, and 4LC-Video applies them across five generation tasks with one model.

1 Introduction

  • Existing reconstruction pipelines render and re-encode RGB for each request, while prior latent caching excludes dynamic content and uses generator-specific features.Repeated processing can consume a substantial share of generation time; these gaps motivate caching dynamic content directly.Source: §1 · p. 2
  • 4LC binds multiple encoders’ representations of static and dynamic content to shared geometry and motion while retaining their formats.Motion determines cached representations’ 3D positions at requested scene times, while static content remains at its reference position.Source: §1 · p. 2
  • Its interface reads cached representations at independently specified times and viewpoints, combining observations to supplement uncovered regions without per-request rendering and re-encoding.This direct readout reduces condition-acquisition latency and peak GPU memory.Source: §1 · pp. 2–3
  • 4LC-Video conditions a shared generator on cache readouts, appends generated observations for later segments, and configures five tasks through source, request, and history choices.The tasks include camera control, temporal control, novel-view synthesis, scene revisits, and long-term generation.Source: §1 · pp. 2–3
  • Without task-specific fine-tuning, a single 4LC-Video model achieves state-of-the-art control accuracy and scene consistency across five generation tasks.4LC accelerates condition acquisition by up to 68.5× with 12.6× lower peak GPU memory, reducing its share of end-to-end generation time from 14.9% to 0.25%.Source: §1 · p. 3

2 Related Work

Prior dynamic-world methods address temporal and viewpoint control, consistency, memory, and long-term generation through several representations. Reconstruction and feature-space conditioning have distinct re-encoding and feature-scope limitations that 4LC addresses with direct composite readouts.

2 Related Work

  • Dynamic-world methods use viewpoint and time controls, point-cloud memories, view retrieval, compressed history, or separate static geometry and object trajectories.Examples include BulletTime and SpaceTimePilot for controls, Spatia for segment consistency, WorldMem and VMem for retrieval, and LiveWorld and VerseCrafter for dynamics.Source: §2 · p. 3
  • Reconstruction pipelines re-encode rendered RGB per request, while feature-space alternatives avoid target RGB encoding but remain generator-specific and static-only.4LC instead directly reads cached multi-encoder representations, which add complementary information.Source: §2 · p. 3

3 Method

4LC stores visual representations from multiple encoders with shared geometry and motion, then reads and combines cache entries at requested times and cameras to condition video generation.

3 Method

  • 4LC organizes bound visual representations in cache entries and provides construction, readout, and composition operations for conditioning a pretrained video diffusion model.Source: §3 · p. 3, Figure 2 · p. 4

3.1 4LC Construction

  • Construction back-projects source pixels into shared 3D coordinates, associates motion and encoder features with units, and appends the resulting entry to the cache.For a single source image, the motion mask is set to zero; units sharing a feature cell reuse its vector while retaining their own geometry and motion.Source: §3.1 · p. 4

3.2 4LC Readout

  • Readout propagates entries within local motion support, projects them through requested cameras, prioritizes nearby-time moving units over static units, and returns features with coverage masks.An entry built from K keyframes can provide conditions for an N-frame target sequence, including N > K.Source: §3.2 · p. 5

3.3 4LC Composition

  • Composition combines selected entry readouts at matching target times and cameras, retains the first valid values at overlaps, and stops at the coverage threshold or when entries run out.Entries remain independently reusable or evictable rather than being accumulated into one entry.Source: §3.3 · p. 5

3.4 4LC-Video

  • 4LC-Video adapts readouts, masks, and camera rays to the generator, then uses a control branch to generate observations that can extend the cache for subsequent segments.The condition adapter and control branch are trained while the visual encoders and video diffusion backbone remain frozen.Source: §3.4 · pp. 5–6, Figure 3 · p. 6

4 Experiments

The experiments evaluate 4LC-Video across five generation tasks and examine condition-acquisition efficiency, readout quality, and encoder configurations. Results describe camera and temporal control, long-term generation, and the effects of alternative conditioning paths and visual representations.

4.1 Experimental Setup

  • The evaluation uses one checkpoint without task-specific fine-tuning, with source videos for control, single-image entries for view synthesis and revisits, and accumulated entries for long-term generation.The setup also specifies Wan2.1-T2V-14B, a VACE control branch, a pretrained NeoVerse reconstructor, and 81-frame 560 × 336 inference segments.Source: §4.1 · p. 6, §4.2 · p. 7

4.2 Main Results on Multiple Video Generation Tasks

  • 4LC-Video achieves the lowest camera errors alongside the highest subject and background consistency, and retains this advantage under temporal control.For frozen-time examples, it holds the subject at frame 0 and preserves later-observed background, while NeoVerse deforms the subject inconsistently.Source: §4.2 · pp. 7–8
  • 4LC-Video earns the highest Environment and Causal scores on MBench-A, while historical reuse constrains revisited structure without replaying past motion.Each generated segment enters the cache, and later segments reuse static units from earlier entries through Composition.Source: §4.2 · p. 8, Table 4 · p. 9

4.3 Conditioning Efficiency and Quality

  • 4LC readout is 68.5× faster than 4DGS rendering plus VAE encoding and uses 12.6× lower peak GPU memory; condition acquisition falls from 14.9% to 0.25% of end-to-end time.With Wan2.1 VAE, adding DINOv2, VGGT, and V-JEPA 2.1 raises readout latency from 40.14 to 196.14 ms and peak memory from 0.385 to 1.245 GiB.Source: §4.3 · p. 9, Figure 5 · p. 9
  • Direct readouts retain comparable absolute feature similarity to re-encoded views; VGGT and V-JEPA 2.1 cluster near one in relative similarity, while VAE relative similarity is lower.Median relative similarity is 0.85 for FLUX.2 VAE and 0.71 for Wan2.1 VAE.Source: §4.3 · pp. 9–10

4.4 Ablation Studies

  • With FLUX.2 VAE, direct readout leaves subject and background consistency nearly unchanged, while Wan2.1 VAE direct readout lowers both scores and raises camera errors more.Adding V-JEPA 2.1 to FLUX.2 VAE lowers rotation and translation errors; combining all four encoders reduces translation error from 0.112 to 0.101, with all four metrics matching or exceeding FLUX.2 VAE re-encoding.Source: §4.4 · p. 10, Table 5 · p. 10

5 Conclusion

4LC stores composite visual representations of static and dynamic content and reads them at independently requested times and viewpoints. One 4LC-Video checkpoint achieves state-of-the-art control accuracy and scene consistency across five tasks; long-term generation currently reuses only static historical content.

5 Conclusion

  • 4LC stores composite visual representations of static and dynamic content and reads them at independently requested times and viewpoints.The conclusion describes these representations as combining complementary information from multiple encoders.Source: §5 · p. 10
  • A single 4LC-Video checkpoint achieves state-of-the-art control accuracy and scene consistency across five generation tasks.The cache avoids per-request RGB rendering and re-encoding by turning condition acquisition into one-time encoding of observations.Source: §5 · p. 10
  • 4LC relies on feed-forward 4D reconstruction estimates of depth, camera parameters, and motion, whose errors can affect readout quality; long-term reuse is limited to static historical content.Dynamic reuse across generated segments is left to future work.Source: §5 · p. 10

A.1 4LC Details

The appendix specifies how visual encoders align features with keyframes, how motion determines dynamic units, and how projected units are selected for readout.

A.1 4LC Details

  • FLUX.2 VAE, DINOv2, and VGGT produce one feature map per keyframe, while V-JEPA 2.1 and Wan2.1 VAE require encoder-specific alignment to keyframes.V-JEPA pairs each keyframe with its predecessor; Wan2.1 separately encodes the first frame and compresses later frames by four.Source: §A.1 · p. 16
  • The motion mask marks a unit dynamic when estimated motion displaces its source-camera projection by more than ϵ pixels per frame toward either adjacent observation.With motion estimated from multiple observations, units moving toward neither neighbor are excluded from readout.Source: §A.1 · p. 16
  • Readout assigns each unit to its nearest projected feature cell; dynamic units closest in time take priority, then static units fill remaining cells.Among remaining candidates, selection prefers the closest source camera, the frontmost surface, and projections nearest the source and target cell centers.Source: §A.1 · p. 16

A.2 4LC-Video Details

The appendix details how cache readouts align with Wan2.1 latent steps, how the model is trained, and how training conditions are varied through augmentation.

A.2 4LC-Video Details

  • For Wan2.1, target intervals of one, two, and four frames contribute four, two, and one readouts to each later latent step, respectively.For an 81-frame request, these intervals produce 81, 41, and 21 readouts over 21 latent steps.Source: §A.2 · p. 16
  • Training uses 200K clips from SpatialVID, RealCam-Vid, and OpenVid for 20K steps on 16 NVIDIA A100 GPUs with global batch size 16.The condition adapter and control branch are trained with AdamW at a learning rate of 10^-5.Source: §A.2 · p. 16
  • Augmentations vary condition availability through source-frame dropout, virtual-camera visibility culling, motion-threshold perturbation, and first-frame sampling.Geometric augmentation applies outside first-frame sampling; lateral translation and focal-length factors are specified relative to world units and the source camera.Source: §A.2 · pp. 16–17, Table 7 · p. 17
  • Representation dropout removes selected visual representations and recomputes coverage from those retained, while text dropout replaces the text condition with an empty-prompt embedding.Encoder settings specify spatial stride, feature dimension, and the number of frames between readouts.Source: Table 6 · p. 16, §A.2 · p. 17

A.3 Experimental Details

The experiments specify evaluation trajectories, cache-history selection, acquisition-cost measurement, feature-readout sampling, and independently trained ablations.

Spatiotemporal control

  • Spatiotemporal-control tests use 21 uniformly spaced keyframes and evaluate specified camera and temporal trajectories relative to each video’s first frame.Translation error excludes static-camera cases.Source: §A.3 · p. 17

Long-term generation

  • Long-term generation evaluates 401-frame MBench-A cases using up to eight sampled cache entries, ordered by Environment or Causal case type.The cases use a 0.95 coverage-ratio threshold, and historical conditions include static units.Source: §A.3 · p. 17

Conditioning efficiency

  • Condition-acquisition latency and peak GPU memory are averaged over 50 runs on an NVIDIA A100, excluding scene construction from timing.The NeoVerse reconstructor takes 1.35 s, while LiveWorld takes 20.43 s; memory includes condition-acquisition weights.Source: §A.3 · p. 17

Feature readout analysis

  • Feature-readout analysis samples 2,048 tokens per video, request, and encoder, averaging relative cosine similarities over ten anchor seeds.The five requests alter scene time or camera sequence while preserving the other source condition.Source: §A.3 · p. 17

Ablations

  • Each ablation, including RGB variants, trains its adapter and control branch independently on 10K clips for 4K steps across four GPUs.Sampling uses 20 steps.Source: §A.3 · p. 17

Long-term generation

  • Figure 6 aligns outbound and return frames by camera pose, with red boxes marking structures changed between the first and final frames.Each method occupies two rows, tracing outbound motion left to right and return motion right to left.Source: Figure 6 · p. 18

A.4 Additional Results

Additional results show 4LC-Video’s WorldScore performance, return-view consistency in long-term generation, and robustness to motion-filter thresholds.

Long-term generation

  • In two reversing-trajectory cases, 4LC-Video’s cached generated segments reproduce outbound content on return views, unlike HY-WorldPlay’s shifted layouts and missing structures.The consistency aligns with its highest Reprojection and Environment scores in Table 4.Source: §A.4 · p. 18

Generalization to WorldScore

  • Camera controllability rises from 23.53 to 86.87 and 3D consistency from 78.74 to 87.97 versus Wan2.1; 4LC-Video also leads Dynamic score and content alignment and style consistency.On WorldScore without fine-tuning, its Static score approaches those of static scene generators.Source: §A.4 · p. 18, Table 8 · p. 19

Motion threshold

  • Without motion filtering, static-structure motion estimates introduce spurious displacement and yield the worst reported camera errors and consistency, flickering, and smoothness metrics.Performance remains stable from thresholds 0.05 to 0.4, including values beyond the training-sampled range of 0.05 to 0.2.Source: §A.4 · p. 19, Table 9 · p. 19

Related papers

Loading 2610.17625v1…