Source-linked AI summary
DecMem: Towards Minute-Long Consistent World Generation with Decoupled Memory
Zhenhao Yang, Xiaoshi Wu, Zhengyao Lv, Xiaoyu Shi, Xintao Wang, Pengfei Wan, Kun Gai, Kwan-Yee K. Wong
TL;DR
Long-horizon controllable video generation still struggles with fine-grained spatio-temporal consistency, especially when recalling distant history. DecMem addresses this with Sparse Global Memory and Anchored Local Memory, and experiments report strong minute-level generation quality and consistency, though real-time performance has not yet been achieved.
Problem
Long-horizon video generation must preserve fine-grained spatio-temporal consistency, but existing approaches struggle with distant-history recall and attention dispersion.
Method
DecMem decouples global retrieval from local anchoring through Sparse Global Memory for fine-grained history access and Anchored Local Memory for stable attention.
Results
DecMem consistently surpasses current state-of-the-art baselines and enables minute-long controllable video generation with strong spatio-temporal consistency and visual quality.
Takeaways & Limitations
The architecture provides precise and efficient long-term memory for minute-long video synthesis without collapse.
Takeaways & Limitations
Real-time performance has not yet been achieved because the work focuses on memory and extrapolation generalization rather than inference acceleration via distillation.
Abstract
from arXiv · showhide
Recent advances in video generative models have promoted rapid progress in controllable world models. However, maintaining fine-grained spatio-temporal consistency under long-horizon reasoning remains a key challenge. In this work, we move beyond explicit 3D memory and coarse frame-level implicit modeling, and propose a fine-grained, learnable, and scalable memory for consistent world generation. We first identify two fundamental limitations of naïve learnable memory architectures in long-horizon extrapolation, namely computational inefficiency and attention dispersion. Through a systematic analysis of attention dispersion, we propose DecMem, a decoupled memory architecture that employs Sparse Global Memory for efficient fine-grained access to global history and Anchored Local Memory for stable and high-quality extrapolation. Extensive experiments demonstrate that DecMem significantly outperforms current state-of-the-art methods. By ensuring precise and efficient long-term memory and achieving superior extrapolation capabilities, DecMem enables minute-level controllable long video generation with high fidelity and consistency.
1 Introduction
Long-horizon controllable video generation struggles to preserve fine-grained spatio-temporal consistency as models revisit earlier scenes. DecMem addresses this with decoupled global retrieval and local anchoring, enabling minute-long generation with strong quality and consistency.
- Long-horizon world models often lose previously generated scenes during revisits, causing severe temporal inconsistencies.
- Frame-level implicit memories cannot capture sub-frame correspondences, while FOV-based retrieval uses heuristics that are not jointly optimized with generation.
- Dense attention over all historical features suffers from attention dispersion and computational inefficiency as the context grows.
- DecMem combines Sparse Global Memory for efficient fine-grained global access with Anchored Local Memory to stabilize attention distributions.
- DecMem consistently surpasses current state-of-the-art baselines for minute-long controllable videos with strong spatio-temporal consistency and visual quality.
2 Related Works
Prior work pursues long-video consistency through explicit geometric memory, implicit retrieval, or training-free sequence strategies. These approaches remain limited by estimator errors, frame-level granularity, heuristic retrieval, or weak long-range dependencies.
- Controllable world-model research has progressed from keyboard controls to mouse trajectories and prompt-based instructions.
- Explicit memory constructs 3D representations for spatial correspondence but inherits 3D-estimation limits and accumulating errors.
- Learnable implicit approaches retrieve memory at frame granularity, limiting fine-grained spatio-temporal consistency; one mechanism does not scale to minute-level generation.
- Training-free overlapping-clip strategies improve inter-clip smoothness but fail to model long-range dependencies across clips.
3 Method
DecMem addresses long-horizon world-modeling failures by separating efficient global memory retrieval from stable local attention. Its Sparse Global Memory and Anchored Local Memory jointly preserve fine-grained long-range dependencies while controlling computation and attention dispersion.
- 3.2 Attention Dispersion in Long World Simulation: Naïve dense attention degrades during long extrapolation because historical tokens dilute attention and increase computational cost.As context grows, many weakly relevant historical features receive nonzero weights, reducing attention to critical frames; training-free decay instead removes genuine long-range dependencies.
- 3 Method: DecMem decouples long-term retrieval from local stabilization through Sparse Global Memory and Anchored Local Memory.SGM accesses global history, while ALM anchors attention in recent frames to stabilize extrapolation.
- 3.3 Sparse Global Memory: SGM retrieves the top-k historical blocks for each current-frame query block, enabling fine-grained memory access without full-sequence attention.It pools each block, ranks historical blocks by relevance, and applies context-aware attention only to the retrieved blocks, preventing per-step computation from growing linearly.
- 3.4 Anchored Local Memory: ALM restricts attention to the immediate past w frames, providing a high-confidence anchor that stabilizes attention during variable-length extrapolation.Temporally adjacent frames provide the strongest visual and semantic correlation with the current frame, helping mitigate temporal drift.
- 3.4 Anchored Local Memory: A learnable gate fuses SGM and ALM outputs to trade off global consistency against extrapolation robustness.ALM supplies a stable baseline against distant long-tail distractors, while SGM preserves fine-grained retrieval over previously visited scenes.
4 Experiment
DecMem is evaluated against Minecraft world-model baselines on short-term generation, extrapolation, perceptual quality, efficiency, qualitative consistency, and ablations. It preserves quality over long horizons while supporting minute-long generation and nearly doubling inference speed versus the strongest baseline.
- Quantitative Experiments: DecMem outperforms all baselines within the training window across every evaluated metric.The evaluation uses the first eight generated frames to test retrieval and use of immediate historical context.
- Quantitative Experiments: DecMem preserves spatial consistency during extrapolation, whereas WorldMem rapidly degrades beyond the training-length threshold.The last eight generated frames evaluate performance beyond each model’s training length.
- Quantitative Experiments: 58 participants rated DecMem highest across Visual Quality, Action Controllability, and Spatio-temporal Consistency.The user study compares these three perceptual dimensions across the evaluated methods.
- Quantitative Experiments: Nearly 2x speedup over the most competitive baseline is achieved when generating 120 frames.Inference speed is measured from average generation time and reported in frames per second.
- Qualitative Experiments: DecMem preserves fine-grained details and spatio-temporal consistency in both short-term and long-term qualitative generation.The comparison initializes environments with 221 frames and generates 120 subsequent frames.
- Qualitative Experiments: Minute-long synthesis remains consistent when revisiting scenes, overcoming temporal degradation common in long-horizon generation.These results are presented as minute-long video generation results with precise memory.
- Ablation Study: Full DecMem maintains stable quality across 600-frame rollouts with near-constant computational cost, unlike dense attention and incomplete variants.Removing SGM or ALM causes long-horizon quality degradation; increasing retrieval blocks from 80 to 100 also worsens long-range PSNR and FID.
5 Conclusion
The paper identifies computational inefficiency and attention dispersion as limitations of naïve dense attention for long-horizon inference, then introduces DecMem to address them. Its sparse global retrieval and anchored local attention support efficient, minute-long controllable video generation with high consistency.
- Conclusion: DecMem combines Sparse Global Memory for efficient fine-grained long-range retrieval with Anchored Local Memory for stable attention.The architecture decouples global retrieval from local anchoring to counteract dispersion from distant noise.
- Conclusion: Extensive experiments validate DecMem for minute-long, efficient, and highly consistent controllable video generation.The conclusion attributes this capability to the proposed decoupled memory architecture.
A.1 Experiment Settings.
The experiments train DecMem on long WorldMem videos and use a latent diffusion transformer with multimodal position embeddings for controllable generation. These embeddings separately encode camera geometry, patch coordinates, and frame indices to support spatiotemporal perception.
- Training and Evaluation Details: DecMem is trained on 11 k videos containing 1500 frames at 360x640 resolution and 10 FPS, with 237 frames sampled and resized to 352x640.Training uses a two-stage strategy beginning from a pretrained full-sequence video-generation checkpoint.
- Base Model Architecture: A latent diffusion transformer serves as the base model, using spatial self-attention within frames and spatiotemporal self-attention across frames.The cross-attention module for text-to-video guidance is discarded because actions and poses control scene generation.
- Camera Embedding: Camera embeddings use projective positional encoding to model relative viewing-frustum geometry from camera intrinsics and world-to-camera extrinsics.The projected matrix is augmented to 4×4 and applied block-diagonally to camera channels.
- Spatial and Temporal Embedding: Spatial and temporal channels use axial 2D RoPE and 1D RoPE, respectively, conditioning attention on patch-coordinate offsets and relative frame distance.The spatial encoding yields translation-equivariant intra-frame perception, while temporal encoding uses t1−t2.
B More Analysis about Attention Dispersion
The analysis shows that dense attention increasingly disperses weight across negligible historical features during long-horizon inference, diluting critical attention. DecMem maintains a more stable attention distribution, whereas training-free decay only partly mitigates the problem and harms long-term memory.
- Attention Dispersion: As inference progresses, dense attention assigns increasing weight to negligible tail features while the proportion of critical weights decreases.This opposing trend quantitatively demonstrates attention dilution during long-horizon extrapolation.
- Training-Free Decay: Training-free decay reduces negligible-weight growth and preserves short-term quality, but follows a similar trend to dense attention and degrades long-term memory.Its mitigation is therefore limited to part of the attention-dispersion problem.
- DecMem Analysis: DecMem keeps irrelevant weights relatively constant and preserves a stable share of critical attention weights throughout inference.Anchored Local Memory focuses attention on important regions and prevents severe quality degradation attributed to attention dispersion.
C Comparison with Industrial-scale Model
Against industrial world models, DecMem matches visual fidelity and action controllability while improving long-horizon spatiotemporal consistency. The comparison evaluates 30-second interactive videos through perceptual user-study axes because identical actions can produce divergent ground-truth videos.
- Evaluation Protocol: 30-second interactive videos are generated from a single initial image and evaluated perceptually on visual quality, action controllability, and spatiotemporal consistency.Direct ground-truth comparison is considered meaningless because insufficient initialization information causes divergence even under identical actions.
- Results: +5.14% in long-horizon spatio-temporal consistency is achieved while visual fidelity and action controllability remain on par with advanced industrial models.The industrial-model comparison uses Matrix-Game 2.0 and WorldPlay as baselines.
D More Ablation study
Ablations examine DecMem components through 500-frame autoregressive rollouts and evaluate action classifier-free guidance. CFG sacrifices some short-range pixel fidelity but substantially improves stability and quality during long-horizon extrapolation.
- Module Ablations: Ablated variants are initialized with 221 memory-bank frames and autoregressively generate the subsequent 500 frames to assess each DecMem component.The comparison qualitatively analyzes generated samples under the Section 4.1 protocol.
- Action Classifier-Free Guidance: Action classifier-free guidance combines conditional and unconditional velocity predictions using guidance scale s during inference.Training randomly drops conditional action embeddings so the model can produce both prediction branches.
- CFG Ablation: CFG-free inference has higher PSNR within the training horizon and early extrapolation, but its quality degrades rapidly as extrapolation length increases.The comparison uses guidance scale 7.5 for the CFG branch.
- CFG Ablation: Enabling CFG substantially preserves stability and generation quality over long-horizon extrapolation despite a marginal short-range fidelity loss.The reported trade-off favors long-range quality over near-term pixel fidelity.
E More Visualization Results
Additional comparisons show DecMem maintains superior memory performance across diverse scenarios and supports up to one minute of high-fidelity inference. Evaluation uses revisitation trajectories across three stylistically distinct environments.
- Additional comparisons: DecMem consistently demonstrates superior memory performance across diverse scenarios in additional baseline comparisons.The comparisons follow the setting in Section 4.2.
- Minute-long generation: Up to one minute of inference is achieved while maintaining high fidelity.
- Evaluation setting: The evaluation uses repeated leftward and rightward camera pans through island, city, and chemical-plant environments.These revisitation trajectories probe fine-grained spatiotemporal memory capability.
G Broader Impacts and Limitations
The method targets improved spatiotemporal consistency and long-horizon extrapolation for world models, with applications spanning several creative and interactive domains. The paper also highlights misuse risks and notes that real-time performance has not yet been achieved.
- Broader impacts: The method aims to improve spatiotemporal consistency and long-horizon extrapolation in world models.
- Risks and mitigation: Potential misuse includes fraudulent forgery and large-scale disinformation enabled by temporally extended, spatiotemporally coherent video.The paper calls for forgery detection and content provenance tracing as mitigation measures.
- Limitations: Real-time performance has not yet been achieved because the work prioritizes precise memory and extrapolation generalization over inference acceleration via distillation.Future work proposes hybrid memory combining compressed global memory with fine-grained object-level memory.