PaperlayerView exact revision on arXiv

Source-linked AI summary

Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models

Kaijin Chen, Dingkang Liang, Xin Zhou, Yikang Ding, Xiaoqiang Liu, Pengfei Wan, Xiang Bai

arXiv:2603.25716v2cs.CVcs.AI

TL;DR

Video world-model memory mechanisms often treat environments as static, limiting consistency when dynamic subjects leave and later re-enter view. This paper introduces Hybrid Memory, the HM-World dataset, and HyDRA’s tokenized spatiotemporal retrieval, with experiments showing improvements over existing methods in subject consistency and generation quality.

  • Problem

    Existing memory methods focus on static scenes or simple dynamics, while complex exit-and-re-entry events require preserving both background consistency and dynamic subject coherence.

  • Method

    The paper constructs HM-World and proposes HyDRA, which compresses memory into tokens and retrieves relevant motion and appearance cues through spatiotemporal attention.

  • Results

    HyDRA outperforms competing approaches across evaluation metrics, including PSNR 20.357, SSIM 0.606, DSCctx 0.827, DSCGT 0.849, Subject Consistency 0.926, and Background Consistency 0.932.

  • Takeaways & Limitations

    Hybrid Memory, HM-World, and HyDRA provide a benchmark and method for evaluating and improving consistency when dynamic subjects disappear and re-enter video frames.

  • Takeaways & Limitations

    HyDRA’s consistent generation degrades in highly complex scenes with three or more subjects or severe occlusions.

Abstract

from arXiv · show

Video world models have shown immense potential in simulating the physical world, yet existing memory mechanisms primarily treat environments as static canvases. When dynamic subjects hide out of sight and later re-emerge, current methods often struggle, leading to frozen, distorted, or vanishing subjects. To address this, we introduce Hybrid Memory, a novel paradigm requiring models to simultaneously act as precise archivists for static backgrounds and vigilant trackers for dynamic subjects, ensuring motion continuity during out-of-view intervals. To facilitate research in this direction, we construct HM-World, the first large-scale video dataset dedicated to hybrid memory. It features 59K high-fidelity clips with decoupled camera and subject trajectories, encompassing 17 diverse scenes, 49 distinct subjects, and meticulously designed exit-entry events to rigorously evaluate hybrid coherence. Furthermore, we propose HyDRA, a specialized memory architecture that compresses memory into tokens and utilizes a spatiotemporal relevance-driven retrieval mechanism. By selectively attending to relevant motion cues, HyDRA effectively preserves the identity and motion of hidden subjects. Extensive experiments on HM-World demonstrate that our method significantly outperforms state-of-the-art approaches in both dynamic subject consistency and overall generation quality. Code is publicly available at https://github.com/H-EmbodVis/HyDRA.

1 Introduction

Existing memory mechanisms largely model environments as static canvases, but dynamic subjects require identity and motion continuity across out-of-view intervals. The paper introduces Hybrid Memory, HM-World, and HyDRA to address this gap.

  • Existing methods reconstruct static environments but often lose, freeze, distort, or erase dynamic subjects when they leave and re-enter view.The gap arises because these methods do not track subjects’ independent motion during unseen intervals.
  • Hybrid Memory requires simultaneous spatial consistency for static backgrounds and motion continuity for dynamic subjects during out-of-view intervals.The model must retain appearance while predicting unseen subject trajectories.
  • HM-World is a large-scale benchmark with 59K clips, decoupled camera and subject trajectories, 17 scenes, 49 subjects, 10 subject paths, and 28 camera trajectories.Its exit-entry events are designed to evaluate hybrid coherence.
  • HyDRA compresses memory into tokens and retrieves information using spatiotemporal relevance to preserve hidden subjects’ identity and motion.Its retrieval selectively attends to relevant motion cues when subjects are poised to re-enter.
  • HyDRA significantly outperforms existing state-of-the-art methods in dynamic subject consistency and overall generation quality.

2 Related Works

Video world models have advanced high-fidelity generation, but consistency remains difficult during long-duration generation and camera motion. Existing memory methods largely target static or simple dynamic environments rather than complex scenes with moving subjects.

  • Video generation and world-model advances have established high-fidelity physical-world simulation as a research direction.Applications include autonomous driving and embodied intelligence.
  • Generation consistency remains challenging during long-duration generation and camera motion, where models can lose previously generated content or context.
  • Existing memory approaches retrieve or propagate context using structures such as 3D surfel memory, FOV overlap, and temporal memory.
  • Most existing methods target static scenes or relatively simple dynamic environments, leaving complex moving-subject scenes insufficiently optimized.The gap concerns both dataset construction and method design.

3 HM-World: Dataset

Hybrid Memory requires models to retain static backgrounds while tracking dynamic subjects through exit-and-re-entry events. HM-World addresses this need with procedurally rendered, diverse clips built from independent scene, subject, subject-trajectory, and camera-trajectory components.

  • 3.1 Hybrid Memory: Hybrid Memory requires a subject to retain visual identity and reappear at a plausible location with a consistent motion state after leaving the frame.
  • 3.1 Hybrid Memory: Hybrid Memory must decouple camera ego-motion from subjects’ independent trajectories, extrapolate motion out of view, and avoid entangling dynamic features with background context.These requirements make hybrid memory more demanding than static memory.
  • 3.2 Dataset Characteristics: HM-World was created in Unreal Engine 5 because internet videos rarely provide clean exit-entry events, combining scenes, subjects, and both trajectory types.
  • 3.2 Dataset Characteristics: The dataset uses 17 scenes, 49 subjects grouped in combinations of 1 to 3, and 28 deliberately designed camera trajectories to induce hide-and-reappear dynamics.Back-and-forth camera motion, such as a leftward pan followed by a rightward pan, can cause subjects to leave and re-enter.
  • 3.2 Dataset Characteristics: HM-World contains 59,225 high-fidelity clips annotated with videos, captions, camera poses, per-frame subject positions, and exit-entry timestamps.

4 Hybrid Dynamic Retrieval Attention

HyDRA augments a full-sequence video diffusion architecture with camera conditioning, spatiotemporal memory tokenization, and dynamic retrieval attention. It retrieves relevant historical cues while preserving local temporal context to improve consistency for hidden subjects.

  • Base Architecture: The base architecture uses a causal 3D VAE and Diffusion Transformer with dynamic retrieval attention, projection, cross-attention, and feedforward components.Flow Matching trains the model to predict velocity from noisy video latents before decoding the denoised latent sequence.
  • Camera Injection: Camera trajectories are encoded from per-frame rotations and translations, then broadcast and added to latent features as an explicit spatial condition.The camera condition is formed by flattening and concatenating pose parameters before MLP encoding.
  • Memory Tokenization: The Memory Tokenizer converts encoded memory latents Zmem into compact tokens M using 3D convolutions that capture long-duration spatiotemporal motion.The resulting tokens preserve essential motion and appearance cues while filtering irrelevant context for retrieval.
  • Dynamic Retrieval Attention: Dynamic Retrieval Attention projects target latents and memory tokens into queries, keys, and values, aligns spatial resolutions, and computes spatiotemporal affinity.Affinity uses channel-wise inner products across spatial positions, followed by Top-K selection of the most correlated memory tokens.
  • Dynamic Retrieval Attention: Retrieved memory features are concatenated with keys and values from a local temporal window, preserving denoising stability alongside long-term consistency.Across denoising queries, the model selectively attends to relevant motion and appearance cues for out-of-sight subjects.

5 Experiments

Experiments compare HyDRA with baselines and state-of-the-art methods on quantitative, qualitative, and ablation measures. HyDRA improves reconstruction, subject and background consistency, and handling of hidden-subject motion, while its components contribute measurably to performance.

  • Main Results: HyDRA is evaluated against a camera-encoder baseline, DFoT, Context-as-Memory, and zero-shot WorldPlay under the reported comparison setup.The compared models include methods trained on HM-World with matching configurations, plus a zero-shot commercial model.
  • Quantitative Comparison: 18.696 to 20.357 PSNR and 0.517 to 0.606 SSIM are the baseline-to-HyDRA improvements reported for future-frame reconstruction.HyDRA also achieves DSCctx 0.827, DSCGT 0.849, Subject Consistency 0.926, and Background Consistency 0.932.
  • Quantitative Comparison: HyDRA attains the highest reported DSCctx and DSCGT scores while preserving both static-background consistency and dynamic-subject identity and motion.The reported consistency scores indicate alignment with historical context, future states, and the static stage.
  • Qualitative Comparison: During complex exit-and-entry events, HyDRA preserves subject identity and motion coherence, whereas competing methods show distortion, incoherence, disappearance, or stuttering.The qualitative comparison identifies distinct failure patterns for the baseline, Context-as-Memory, DFoT, and WorldPlay.
  • Ablation Study: Reducing tokenizer temporal interaction causes a 1.281 PSNR and 0.014 DSCGT drop, supporting temporal processing for long-term dynamic information.The ablation varies tokenizer kernel dimensions and reports robustness to spatial-dimension changes.
  • Ablation Study: Retrieving 5 tokens gives PSNR 19.309, while 10 and 15 tokens perform better and similarly, indicating that moderate retrieval is sufficient.The study attributes the weaker 5-token result to information loss and notes negligible differences between 10 and 15 tokens.
  • Ablation Study: Dynamic affinity retrieval improves Subject Consistency from 0.908 to 0.926 over FOV overlap by assessing fine-grained spatiotemporal relevance.The FOV-based alternative relies on static geometric overlap, whereas dynamic retrieval uses QK interactions.

6 Conclusion

The paper frames Hybrid Memory as a paradigm for preserving static backgrounds and dynamic subjects through complex exit-and-re-entry events. It combines HM-World with HyDRA and reports that HyDRA significantly outperforms existing methods, while acknowledging degradation in highly complex scenes.

  • 6 Conclusion: Hybrid Memory requires simultaneous static-background consistency and dynamic-subject coherence during complex exit-and-re-entry events.The paradigm targets both spatial consistency for backgrounds and motion continuity for subjects.
  • 6 Conclusion: HM-World is introduced as a large-scale benchmark with diverse scenarios and complex dynamic processes for hybrid-memory research.The conclusion describes the dataset as purpose-built to facilitate systematic research in this setting.
  • 6 Conclusion: HyDRA is designed to extract and retrieve motion and appearance cues for consistent generation, and extensive experiments show it significantly outperforms existing methods.The conclusion presents the architecture and its reported experimental outcome together.
  • Limitations and Future Work: HyDRA’s consistency degrades in highly complex scenes with three or more subjects or severe occlusions.Future work targets more robust memory mechanisms and scaling to unconstrained real-world environments.

Supplementary Material

The supplementary material provides additional generation results and ablation studies, including visual results for HyDRA.

  • Supplementary Material: The supplementary material contains more generation results and ablation studies beyond the main paper.It is described as additional information about the work.
  • Generation Results: Supplementary Figure 1 shows HyDRA results across multiple scenes, subjects, and trajectories.The figure presents dynamic scenarios involving exit-entry events.

A.1 Generation Results

HyDRA’s generation results span varied scenes, subjects, and trajectories while maintaining background and subject consistency during exit-entry events.

  • A.1 Generation Results: HyDRA memorizes backgrounds and subjects in complex dynamic scenarios with exit-entry events, maintaining appearance and motion consistency.The results cover multiple scenes, subjects, and trajectories.

A.2 Open-Domain Results

On entirely unseen scenes, HyDRA shows good hybrid-memory capacity when open-domain videos are evaluated with back-and-forth camera movements.

  • HyDRA exhibits good hybrid-memory capacity on entirely unseen scenes with subject-motion videos and back-and-forth camera movements.The results are presented as open-domain inference examples in Fig. 2.

B More Ablation Studies

Ablations examine retrieval, tokenization, and retrieved-token count, showing that feature-level dynamic retrieval, temporal token interaction, and adequate context support consistent generation.

  • Analysis of Retrieval Approaches: Dynamic-affinity retrieval selects feature-correlated memory tokens, whereas FOV overlap can retrieve empty shots during complex exit-and-re-entry events.The FOV-based approach selects nearest camera poses, while dynamic affinity retrieves keyframes containing richer information.
  • Analysis of Retrieval Approaches: The retrieval-method comparison visualizes token index along the x-axis, DiT layers along the y-axis, and selection frequency through bubble size and color.FOV overlap yields fixed token selection, while dynamic affinity produces a more diverse retrieval distribution.
  • Analysis of Retrieval Approaches: Dynamic-affinity retrieval provides a broader memory receptive field and greater flexibility during generation.
  • Ablation on Kernel Size of Memory Tokenizer: A temporal kernel size of 2 maintains spatiotemporal consistency, while size 1 without temporal interaction produces noticeable subject inconsistencies.These qualitative findings support the reported quantitative tokenizer ablation.
  • Ablation on Number of Retrieved Tokens: Restricting retrieval to 5 tokens loses context and can produce severe artifacts, whereas adequate token counts maintain subject consistency and physical plausibility.One example hallucinates two giraffes instead of one.

C Additional Examples from the HM-World Dataset

The paper provides additional examples illustrating challenges in the HM-World dataset.

  • Additional examples are provided to further illustrate challenges in the HM-World dataset.
Loading 2603.25716v2…