Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models

1 Introduction

• Memory capacity underpins spatial and temporal consistency by retaining historical context during viewpoint shifts and long-term extrapolation.Without robust memory, generated environments can deteriorate into disconnected frames.Source: §1 · p. 2
• Hybrid Memory combines precise static-background reconstruction with prediction of unseen subject trajectories, preserving appearance and motion consistency when subjects return.Existing memory approaches [12] [31] treat environments as static canvases and can freeze, distort, or lose subjects after they leave view.Source: §1 · pp. 2–3
• 59K clips constitute HM-World, a dedicated Hybrid Memory training and evaluation dataset with decoupled camera and subject trajectories and deliberately designed exit–entry events.Introduced as the first large-scale dataset for this capability, it spans 17 scenes, 49 subjects, 10 subject motion paths, and 28 camera trajectory types.Source: §1 · pp. 2–3
• HyDRA significantly outperforms state-of-the-art approaches on HM-World in dynamic subject consistency and overall generation quality.Ablation studies further support the robustness of its design.Source: §1 · p. 3
• HyDRA compresses memory latents into informative tokens and retrieves spatiotemporally relevant motion and appearance cues during denoising to recover returning subjects.Its Memory Tokenizer and relevance-driven retrieval mechanism help preserve hidden subjects’ identity and motion.Source: §1 · p. 3

2.1 Video World Models

• High-fidelity video generation [18] [29] [34] [39] [40] increasingly provides the foundation for video world models [10] [11] [19] [23] [27].GameGen-X explores interactive game-like environments; Yume [23] extends duration autoregressively; Matrix-Game 2 [10] combines GTA-V and Unreal Engine 5 data with autoregressive denoising for controllability and visual quality. RELIC [11] supports stable long-duration generation through replayed back-propagation, while Worldplay [27] combines high-quality data and context forcing for visual quality, consistency, and real-time generation.Source: §2.1 · pp. 3–4
• Video world models struggle with static and dynamic consistency, particularly during long generation and camera motion, motivating the paper’s hybrid-memory approach.Models can lose track of previously generated content or contextual input, producing inconsistent sequences.Source: §2.1 · p. 4

2.2 Memory in Video Generation

• Existing retrieval approaches use 3D surfel indexing in Vmem and field-of-view overlap in Context-as-Memory and Worldmem.Worldmem combines external-memory retrieval with Diffusion Forcing on Minecraft data. Memory Forcing [12] adds temporal memory to balance exploration and consistency.Source: §2.2 · p. 4
• Other memory strategies include context forcing in WorldPlay [27], updatable semantic memory in MemoryPack [31], and uniform spatial down-sampling in RELIC [11].Inspired by FramePack, MemoryPack [31] retains semantically relevant memory throughout generation.Source: §2.2 · p. 4
• Most memory methods target static scenes [11] or simple dynamic environments [12] [31], leaving complex dynamic scenes insufficiently addressed.Genie 3 demonstrates strong dynamic consistency but remains closed-source with undisclosed technical details. The paper addresses this gap through hybrid-memory methods and dataset construction.Source: §2.2 · p. 4

3.1 Hybrid Memory

• Hybrid memory preserves static backgrounds while ensuring returning subjects retain their visual identities and reappear at plausible locations with consistent motion states.Static memory instead focuses on immobile elements, typically checking whether a scene looks identical when the camera revisits a previous pose.Source: §3.1 · p. 5
• Hybrid memory requires separating camera and subject motion, extrapolating unseen movement, and retrieving dynamic cues despite entangled background and subject features.Without isolating dynamic cues, historical-context retrieval can freeze subjects into backgrounds or distort them.Source: §3.1 · p. 5

3.2 Dataset Characteristics

• 17 diverse scenes and 49 distinct subjects supply HM-World’s Unreal Engine 5 rendering pipeline.People and animals are procedurally placed in groups of 1 to 3; each subject has a motion animation and follows a randomly selected trajectory from 10 predefined paths.Source: §3.2 · p. 6
• 28 distinct camera trajectories deliberately use back-and-forth movements to induce subjects’ exits and re-entries.Multiple initial positions for each camera movement further diversify the camera-motion sequences.Source: §3.2 · p. 6
• 59,225 high-fidelity video clips comprise HM-World after filtering out clips without exit-entry events.Each sample includes rendered video, a MiniCPM-V [35] caption, camera poses, per-frame subject positions, and precise subject exit-entry timestamps. The dataset addresses the scarcity of natural videos capturing suitable exit-and-re-entry events.Source: §3.1 · p. 6, §3.2 · p. 7
• HM-World combines dynamic subjects, complex camera trajectories, and explicit exit-entry events, addressing limitations of existing datasets.Context-as-Memory contains only static scenes, WorldScore’s scale is limited to 3K, and Multi-Cam Video’s subjects act in place. In 360 °-Motion, moving subjects remain visible while the camera stays static.Source: §3.2 · p. 7

4.1 Base Architecture and Camera Injection

• HyDRA uses a full-sequence video diffusion backbone comprising a causal 3D VAE [17] and a Diffusion Transformer (DiT).Each DiT block integrates dynamic retrieval attention, a projector, cross-attention, and a feedforward network. An MLP encodes the diffusion timestep to modulate the blocks.Source: §4.1 · pp. 7–8
• Flow Matching [22] trains velocity prediction on latents linearly interpolated between encoded video and Gaussian noise, with target v_t = z_0 − z_1.The VAE compresses video spatially and temporally. At inference, Gaussian noise is progressively denoised into a clean latent and decoded into video.Source: §4.1 · p. 8
• Following ReCamMaster, an MLP encodes camera rotations and translations, then broadcasts and adds the resulting features to latent features for spatial control.Each frame’s rotation matrix and translation vector form 12 parameters; encoded camera features are projected to match the latent channel dimension.Source: §4.1 · p. 8

4.2 Memory Tokenization for Retrieval

• A 3D-convolutional tokenizer compresses memory latents into spatiotemporally aware tokens, capturing long-duration motion while filtering irrelevant context and preserving motion and appearance cues.Injecting all memory latents adds overhead and irrelevant noise, while retrieval from uncoupled raw latents can lack expressiveness for subject motion and camera transformations.Source: §4.2 · pp. 8–9

4.3 Dynamic Retrieval Attention

• Dynamic Retrieval Attention replaces standard 3D self-attention, selecting top-K memory slices by spatiotemporal affinity with spatially pooled target queries.The authors report that selectively attending to relevant motion and appearance cues tracks hidden subjects, preserves spatiotemporal consistency, and reduces computational burden.Source: §4.3 · pp. 10–11
• A mandatory local temporal window augments retrieved memory with adjacent target-latent keys and values to preserve denoising stability and self-attention structure.The window is centered on each target frame, and its features are flattened and concatenated with retrieved memory features.Source: §4.3 · p. 10

5.1 Experiment Setup

• 1000 HM-World test videos, including unseen scenes and subjects, evaluate reconstruction fidelity, frame-level consistency, and dynamic subject consistency.Built on Wan2.1-T2V-1.3B [29], HyDRA encodes 77 context frames with fourfold temporal downsampling and trains for 10K iterations on 32 GPUs with batch size 32. DSC uses YOLOv11 [16] subject crops and CLIP features to compare predictions against historical context (DSCctx) and ground truth (DSCGT).Source: §5.1 · p. 11

5.2 Main Results

• 20.357 PSNR and 0.606 SSIM mark HyDRA’s across-metric lead over the baseline, DFoT, and Context-as-Memory.Against baseline PSNR 18.696 and SSIM 0.517, HyDRA also achieves DSCctx 0.827, DSCGT 0.849, Subject Consistency 0.926, and Background Consistency 0.932. It surpasses zero-shot WorldPlay [27] across all metrics, including a PSNR gap of 5.502, although WorldPlay faces a domain gap and lacks specific finetuning.Source: §5.2 · pp. 11–12, Table 2 · p. 12, Table 3 · p. 12
• HyDRA preserves subject identity and coherent motion after re-entry, whereas competing methods exhibit distortion, disappearance, or stuttering.The baseline and Context-as-Memory distort subjects and disrupt motion; DFoT loses subjects entirely. WorldPlay preserves appearance but produces stuttering movements and unnatural actions.Source: §5.2 · pp. 12–13, Figure 6 · p. 13

5.3 Ablation Study

• 1.281 PSNR and 0.014 DSCGT drops occur when the tokenizer’s temporal kernel dimension is reduced to 1, supporting temporal interaction.Spatial kernel changes have smaller effects: moving from 4×4 to 2×2 or 8×8 decreases PSNR by 0.244 or 0.127, respectively.Source: Table 4 · p. 13, §5.3 · pp. 13–14
• 19.309 PSNR with five retrieved tokens is suboptimal; retrieving 10 or 15 tokens produces better, more stable results with negligible differences.The findings indicate that a moderate token count supplies sufficient spatiotemporal information without redundant noise.Source: Table 5 · p. 14, §5.3 · p. 14
• 0.926 versus 0.908 Subject Consistency favors dynamic affinity retrieval over FOV-overlap retrieval, with improvements across every metric.Dynamic retrieval uses QK interactions to assess fine-grained spatiotemporal relevance. The FOV alternative follows Context-as-Memory, averaging source-frame camera poses for each token and relying on static geometry overlap.Source: Table 6 · p. 14, §5.3 · p. 14

6 Conclusion

• Hybrid Memory requires models to maintain static-background consistency and dynamic-subject coherence simultaneously, particularly during complex exit-and-re-entry events.Source: §6 · p. 14
• HyDRA extracts and retrieves motion and appearance cues to support consistent generation.Source: §6 · p. 14
• HM-World is introduced as the first large-scale video dataset dedicated to hybrid memory, featuring diverse scenarios and complex dynamic processes.Source: §6 · p. 14
• HyDRA significantly outperforms existing methods in the reported experiments.The conclusion does not provide numerical results.Source: §6 · pp. 14–15

Limitations and Future Work

• HyDRA’s generation consistency tends to degrade in complex scenes involving three or more subjects or severe occlusions.Future work will explore more robust memory mechanisms for intricate multi-subject dynamics and scaling to unconstrained real-world environments.Source: §6 · p. 15

Supplementary Material

• Additional generation results supplement the paper’s main presentation.The supplied introductory passage does not describe individual examples or their outcomes.Source: Supplementary Material · p. 18
• The supplementary material also includes ablation studies.The supplied passage does not specify the ablated components or findings.Source: Supplementary Material · p. 18
• Figure 1 presents results generated by HyDRA.The supplied caption does not report quantitative comparisons or identify a winning method.Source: Figure 1 · p. 18

• HyDRA maintains appearance and motion consistency in complex dynamic scenarios with exit-entry events.Source: §A.1 · p. 18
• HyDRA memorizes both backgrounds and subjects in these dynamic scenarios.Source: §A.1 · p. 18
• The generation results shown in Fig. 1 span multiple scenes, subjects, and trajectories.Source: §A.1 · p. 18

A.2 Open-Domain Results

• HyDRA shows good hybrid-memory capability in entirely unseen scenes.The passage reports this qualitative result in Fig 2.Source: §A.2 · p. 18
• The open-domain evaluation uses videos collected from the Internet that feature subject motion.Source: §A.2 · p. 18
• Back-and-forth camera movements are applied during inference to evaluate hybrid memory.Source: §A.2 · p. 18

Analysis of Retrieval Approaches

• Dynamic-affinity retrieval preserves subject appearance and motion after re-entry, whereas FOV filtering retrieves empty shots and produces inconsistent generations.During long camera movements with complex exit-and-re-entry events, feature-level correlations retrieve subject-rich keyframes rather than merely selecting nearby camera poses.Source: B More Ablation Studies · pp. 18–19, Figure 3 · p. 19
• Dynamic-affinity retrieval adapts token selection across timesteps and DiT layers, providing broader memory access and greater flexibility than fixed FOV-based selection.FOV filtering uses static 3D geometry, while dynamic affinity recomputes feature-level correlations.Source: B More Ablation Studies · pp. 19–20, Figure 4 · p. 20

Ablation on Kernel Size of Memory Tokenizer

• 2 is the temporal kernel size that maintains spatiotemporal consistency; reducing it to 1 removes temporal interaction and introduces subject inconsistencies.These qualitative observations corroborate the quantitative ablations reported in the main paper.Source: B More Ablation Studies · p. 20, Figure 5 · p. 20

Ablation on Number of Retrieved Tokens

• 5 retrieved tokens lose substantial context and cause severe artifacts, whereas settings with adequate token counts maintain subject consistency and physical plausibility.One artifact is hallucinating two giraffes instead of one.Source: B More Ablation Studies · pp. 20–21, Figure 6 · p. 21

C Additional Examples from the HM-World Dataset

• The additional examples illustrate challenges present in HM-World.Source: C Additional Examples from the… · p. 22
• The examples supplement the presentation of the proposed HM-World dataset.Source: C Additional Examples from the… · p. 22
• Figure 7 contains the additional dataset examples.Source: C Additional Examples from the… · p. 22