Source-linked AI summary
MemFlow: Flowing Adaptive Memory for Consistent and Efficient Long Video Narratives
Sihui Ji, Xi Chen, Shuai Yang, Xin Tao, Pengfei Wan, Hengshuang Zhao
TL;DR
Long-video generation needs memory that adapts to changing prompts without imposing prohibitive computation. MemFlow retrieves prompt-relevant historical context and selectively activates memory tokens during attention. It maintains long-range consistency while achieving 18.7 FPS and only a 7.9% speed reduction versus a memory-free baseline.
Problem
Streaming video chunks need different historical cues, but fixed memory strategies struggle to provide prompt-relevant context under GPU and efficiency constraints.
Method
MemFlow uses Narrative Adaptive Memory for prompt-driven historical retrieval and Sparse Memory Activation to attend only to relevant memory tokens.
Results
MemFlow achieves long-duration contextual consistency with 18.7 FPS on one NVIDIA H100 and a 7.9% speed reduction versus the memory-free baseline.
Takeaways & Limitations
Prompt-adaptive retrieval and relevance-gated activation support narrative coherence through new events, scenario switches, and character transitions.
Abstract
from arXiv · showhide
The core challenge for streaming video generation is maintaining the content consistency in long context, which poses high requirement for the memory design. Most existing solutions maintain the memory by compressing historical frames with predefined strategies. However, different to-generate video chunks should refer to different historical cues, which is hard to satisfy with fixed strategies. In this work, we propose MemFlow to address this problem. Specifically, before generating the coming chunk, we dynamically update the memory bank by retrieving the most relevant historical frames with the text prompt of this chunk. This design enables narrative coherence even if new event happens or scenario switches in future frames. In addition, during generation, we only activate the most relevant tokens in the memory bank for each query in the attention layers, which effectively guarantees the generation efficiency. In this way, MemFlow achieves outstanding long-context consistency with negligible computation burden (7.9% speed reduction compared with the memory-free baseline) and keeps the compatibility with any streaming video generation model with KV cache.
1. Introduction
Interactive long-video generation must preserve long-range consistency while adapting memory to changing prompts and staying computationally efficient. MemFlow addresses this with prompt-driven retrieval and relevance-based memory activation, achieving long-context consistency with limited speed cost.
- Different future chunks require different historical cues, while GPU limits constrain how much memory can be stored and used.
- Existing fixed sinks, compression schemes, and trainable memory modules struggle to dynamically provide prompt-relevant historical content.
- Narrative Adaptive Memory retrieves prompt-aligned historical KV-cache frames and combines them with a condensed representation of the preceding chunk.
- Sparse Memory Activation selects only the most relevant memory tokens for attention, reducing computation while preserving generation quality.
- 7.9% speed reduction versus the memory-free baseline accompanies long-duration contextual consistency and 18.7 FPS on one NVIDIA H100.
2. Related Works
Prior long-video methods extend generation through autoregressive-diffusion hybrids, multistage synthesis, or efficient context architectures. MemFlow instead combines prompt-conditioned memory retrieval with selective activation to address long-range coherence and efficiency.
- Autoregressive-diffusion hybrids generate long videos sequentially, while multistage methods synthesize clips or keyframes separately but often lack long-horizon temporal coherence.
- MemFlow retrieves semantically aligned KV cache with the current prompt, updates memory with recent context, and applies Sparse Memory Activation to relevant tokens.
- Efficient-context methods use linear attention, condensed tokens, or dynamic context selection, but may sacrifice visual fidelity for efficiency.
3. Method
MemFlow maintains a dynamic memory bank for streaming autoregressive video generation through Narrative Adaptive Memory and Sparse Memory Activation. The method retrieves relevant historical context, incorporates recent content, and selectively attends to memory to balance coherence and efficiency.
- MemFlow refreshes memory with Narrative Adaptive Memory, selects relevant context with Sparse Memory Activation, and feeds the result to the autoregressive diffusion model for each chunk.
- The streaming long-tuning strategy trains the model end-to-end to manage memory during repeated long-duration rollouts.
- 3.2. Narrative Adaptive Memory (NAM): Narrative Adaptive Memory stores layerwise KV representations and retrieves history relevant to the incoming prompt while incorporating the latest generated chunk.
- 3.2. Narrative Adaptive Memory (NAM): Semantic retrieval scores stored frames using textual-query and visual-key relevance, retaining the top-k aligned frames and a prototype of the preceding chunk.
- 3.3. Sparse Memory Activation (SMA): Sparse Memory Activation addresses memory overhead by selecting the most relevant historical frames before attention computation.
- 3.3. Sparse Memory Activation (SMA): Attention is restricted to selected key-value pairs, reducing latency while retaining pertinent history and filtering less relevant or potentially erroneous information.
4. Experiment
MEMFLOW is evaluated on short and long single-prompt generation, multi-prompt long-video generation, and memory ablations. Across these settings, it improves semantic and consistency outcomes while retaining efficient inference.
- Long-video comparison: MEMFLOW achieves the best overall quality score in long-video evaluation, with an advantage in aesthetic quality and competitive consistency despite FramePack’s higher consistency score.The comparison uses VBench-Long metrics for visual quality, consistency, and aesthetics.
- Multi-prompt comparison: MEMFLOW shows outstanding prompt adherence and narrative coherence as videos extend to longer durations by linking prompt descriptions with corresponding subjects in previous frames.The evaluation measures clip-wise semantic adherence using CLIP scores at 10-second intervals.
- Multi-prompt comparison: In multi-prompt 60-second generation, MEMFLOW maintains narrative coherence and subject consistency without drifting or duplicated characters.Qualitative comparisons show stronger coherence than representative long-video generation alternatives.
- Long-video comparison: For 30-second single-prompt generation, MEMFLOW improves across quality and semantic metrics over competing methods while maintaining comparable efficiency.The reported comparison is against SkyReels-V2, FramePack, Self Forcing, and LongLive.
- Ablation studies: Ablations show that Narrative Adaptive Memory outperforms alternative memory mechanisms, while Sparse Memory Activation increases inference speed from 17.6 to 18.7 frames per second with minimal quality degradation.The comparisons include no memory, Frame Sink, NAM, and the full NAM+SMA model under five prompt switches over 60 seconds.
5. Conclusion
MEMFLOW equips interactive long-video generation with long-range consistency through adaptive semantic retrieval and relevance-gated memory filtering. It supports narrative coherence during complex transitions and character switching while maintaining efficient inference.
- MEMFLOW introduces Narrative Adaptive Memory to dynamically retrieve semantically aligned historical context through textual queries for long-term narrative coherence.The mechanism addresses interactive generation with changing prompts and narrative transitions.
- Sparse Memory Activation balances memory and efficiency by filtering memory through relevance-gated activation of selected tokens.This complements adaptive retrieval by limiting attention to relevant memory content.
- 18.7 FPS inference on a single NVIDIA H100 supports interactive generation with consistency, visual quality, and narrative coherence under complex transitions and character switching.