Source-linked AI summary
LongLive-RAG: A General Retrieval-Augmented Framework for Long Video Generation
Qixin Hu, Shuai Yang, Wei Huang, Song Han, Yukang Chen
TL;DR
Long-horizon autoregressive video generation can accumulate errors when it relies only on a recent context window. LongLive-RAG retrieves relevant self-generated latents as non-local context, improving long-video quality across backbones and generation lengths.
Problem
Long autoregressive video generation needs to preserve subject, background, and scene consistency despite error accumulation from discarding earlier context.
Method
LongLive-RAG retrieves relevant self-generated latents with learned embeddings and uses them as additional attention context for each new video block.
Results
LongLive-RAG achieves the best average VBench-Long rank across three AR backbones and 30s, 60s, and 120s generation lengths.
Takeaways & Limitations
Selected non-local latents provide useful context beyond the recent window, improving subject and background consistency, motion smoothness, and imaging quality.
Takeaways & Limitations
Because LongLive-RAG uses a frozen base checkpoint, final video quality remains bounded by the capability of the underlying AR generator.
Abstract
from arXiv · showhide
Autoregressive (AR) video diffusion enables variable-length synthesis, but long-horizon generation often suffers from accumulated errors and identity drift. For efficiency, existing methods commonly adopt sliding-window attention during generation. This creates an irreversible generation trajectory: once the active window accumulates appearance errors, subsequent generations can only condition on this degraded trajectory and drift further away. We address this limitation by formulating long video generation as a retrieval-augmented generation (RAG) problem. Rather than relying solely on the recent window, we treat previously generated latents as a dynamic, searchable history. We propose LongLive-RAG, a general retrieval framework for AR video generation. At each new block, LongLive-RAG uses a query embedding to retrieve relevant historical latents. This lightweight retrieval step adds only a small overhead relative to generation and lets the generator condition on non-local context instead of only the recent window. To make retrieval more discriminative, we introduce the Window Temporal Delta Loss that suppresses redundant local similarity and encourages embeddings to capture meaningful temporal changes. Together, these components help reduce error accumulation caused by sliding-window attention. Experiments across multiple AR backbones and generation lengths show improved long-video quality and the best average VBench-Long rank. To our knowledge, among open-ended AR long video generation methods, LongLive-RAG is the first to formulate self-generated latent history as content-addressable retrieval memory. Code is available at https://github.com/qixinhu11/LongLive-RAG.
1 Introduction
Long-video generation requires maintaining consistent subjects, backgrounds, and scene layouts beyond short clips, but existing history-management strategies remain limited. LongLive-RAG addresses this by retrieving useful non-local context from self-generated latent history for each new autoregressive block, using a discriminative embedding objective.
- Motivation: Long videos demand consistency across subjects, backgrounds, and scene layouts over tens or hundreds of seconds, motivating autoregressive latent-block generation beyond fixed clip lengths.AR video diffusion generates latent blocks causally and can continue beyond a fixed clip length.
- Related work: Existing approaches retain anchor tokens, extrapolate positional ranges, or compress older states into substitute tokens or recurrent memory, but limitations remain.These strategies can reduce error and improve stability while still leaving unresolved limitations.
- Motivation: Retrieval can select earlier generated context that better preserves the current block’s subject appearance, background layout, or scene structure than only recent history or fixed summaries.The model searches previously generated latents before generating a new block.
- Method: LongLive-RAG stores generated latents, encodes the latest completed latent as a query, retrieves top-K historical latents, and adds them to the generator’s local attention context.The framework is presented as a general retrieval-augmented approach for autoregressive video generation.
- Method: The Window Temporal Delta Loss reduces redundant similarity among adjacent latent embeddings, while a smoothing term stabilizes embeddings over time and preserves visual information.These objectives make retrieval more discriminative so retrieved blocks add useful context beyond the local window.
- Contributions: LongLive-RAG formulates open-ended autoregressive video generation as retrieval over self-generated latents and compares against ∞-RoPE and Deep Forcing for sliding-window limits.The framework combines lightweight retrieval, extra attention context, and an embedding objective for selecting useful non-local context.
2 Related Work
LongLive-RAG differs from prior long-video generation methods by searching self-generated history for relevant context, rather than relying only on local rollout or fixed history-access strategies.
- AR long video generation: LongLive-RAG is orthogonal to advances in local denoising, self-generated context, and streaming attention because it keeps the generator unchanged.Existing autoregressive video diffusion methods primarily improve the quality or causality of each local rollout step.
- Context visibility and memory: LongLive-RAG searches generated history and retrieves context for attention, unlike sliding windows, fixed anchors, positional extrapolation, or compressed-history tokens.These prior strategies differ in how much history they retain and how it is stored.
- Retrieval memory in video generation: Unlike retrieval in world models with geometry, camera pose, or scene coordinates, LongLive-RAG finds useful history directly from generated content during rollout.Open-ended text-to-video generation lacks explicit retrieval cues by default.
3 Method
LongLive-RAG augments a frozen autoregressive video diffusion generator with retrieval over self-generated latent history, combining retrieved non-local entries with the existing local context. An offline-trained embedding space selects relevant history while temporal-delta and smoothing terms make retrieval discriminative and stable.
- Context assembly: LongLive-RAG keeps the base autoregressive generator fixed and adds retrieved historical context entries alongside the attention sink and recent local context.It changes which historical entries are exposed to attention without adding attention layers or changing the denoising rule.
- History banks and embeddings: The retrieval encoder is trained offline on clean latents with the generator fixed, producing 1024-dimensional embeddings paired with completed block contexts.Recent entries remain in the rolling local cache and later move into the historical pool for retrieval.
- Similarity retrieval: At each block, the system embeds the most recent completed latent, ranks eligible historical embeddings by cosine similarity, and selects top-K context entries while skipping near-duplicate recent history.Retrieval occurs once before denoising and the selected context is reused for all N denoising steps.
- Inference: 4.08 ms per block and 490 ms total retrieval overhead are added during a 120s rollout, with 480 ms from encoding and 10 ms from top-K search.The overhead is dominated by encoding, while similarity search is negligible at this scale.
- Embedding-space training: The retrieval objective combines reconstruction, Window Temporal Delta Loss, and second-order smoothing to preserve visual content, reduce redundant local similarity, and stabilize the embedding trajectory.The resulting search geometry is designed specifically for non-local context selection rather than generic compression.
4 Experiments
Experiments evaluate LongLive-RAG across three autoregressive backbones and 30s, 60s, and 120s generations using VBench-Long, qualitative comparisons, and ablations. LongLive-RAG achieves the lowest average rank across evaluation blocks, while learned embeddings and K = 6 retrieval improve consistency and imaging quality.
- Evaluation protocol: Experiments use Causal-Forcing, Self-Forcing, and LongLive across 30s, 60s, and 120s generations, with all 128 refined MovieGenBench prompts.Inference uses sink size 1 and total attention window size 12; LongLive-RAG uses K = 6 retrieved slots.
- Qualitative results: On 30s rollouts, LongLive-RAG better preserves subject and background appearance than alternative history-handling methods.Compared methods can exhibit appearance shifts, duplicated subjects, background artifacts, or color artifacts.
- Quantitative results: LongLive-RAG obtains the lowest Avg. Rank across every base-model and duration block, with gains consistent from 30s through 120s.Improvements are most visible in subject consistency, background consistency, motion smoothness, and imaging quality.
- Auxiliary VLM evaluation: Auxiliary VLM exposure scores on 30s generations show the same overall trend as the VBench-Long results.Table 3 reports these scores as an auxiliary evaluation.
- Embedding space: Learned embeddings outperform random, average-pooled, and reconstruction-only retrieval, while the full objective gives the strongest overall retained-metric results.Adding LSeqDelta improves background consistency and motion smoothness over AE-only embeddings.
- Retrieval budget: K = 6 gives the strongest consistency and imaging quality under the same total attention budget, balancing retrieved non-local context with local continuation.The retrieval-budget comparison keeps the embedding fixed.
5 Conclusion
LongLive-RAG enables autoregressive video generation to retrieve relevant context from previously generated history instead of relying only on the recent window. Its retrieval embeddings combine reconstruction, Window Temporal Delta Loss, and smoothing to preserve visual content, avoid redundant nearby matches, and remain temporally stable.
- Conclusion: LongLive-RAG searches previously generated video history and retrieves relevant context for each subsequent block.This replaces exclusive reliance on the recent generation window.
- Conclusion: Retrieval embeddings use reconstruction, Window Temporal Delta Loss, and smoothing to preserve visual content, avoid redundant nearby matches, and remain stable over time.
A Limitations · B Broader Impacts
LongLive-RAG improves selection and reuse of generated history without changing the frozen base generator, so final quality remains bounded by that model’s capabilities. It may enable stable long-video synthesis while inheriting the underlying generator’s misuse risks.
- A Limitations: LongLive-RAG is built on a frozen base checkpoint.
- A Limitations: The method improves selection and reuse of generated history, but does not modify the generator itself.
- A Limitations: Final video quality remains bounded by the capability of the base autoregressive model.
- B Broader Impacts: LongLive-RAG targets improved long-horizon consistency in autoregressive text-to-video generation.
- B Broader Impacts: The method may support stable long-video synthesis for creative tools, simulation, and interactive content generation.
- B Broader Impacts: Improved long-video quality can inherit misuse risks from the underlying video generator, including deceptive or misleading generated media.
C Additional Related Work Discussion · D Why Retrieval in Latent Space?
The related work contrasts causal video generation’s context-management strategies with LongLive-RAG’s retrieval of self-generated video latents. Latent-space retrieval preserves the generator’s execution pattern, aligns search with internal states, and avoids ambiguities introduced by decoded pixels.
- C Additional Related Work Discussion: Autoregressive video generation emits frames or latent blocks causally, enabling streaming, interactive prompting, and variable-length output.Many modern generators instead denoise fixed clips jointly, tying computation to clip length.
- C Additional Related Work Discussion: Causal adaptation methods reduce exposure bias and improve local denoising, whereas LongLive-RAG selects non-local context from the self-generated trajectory.The two approaches address complementary aspects of causal rollout.
- C Additional Related Work Discussion: Sliding windows, anchors, and positional extrapolation improve scalability or stability, but fixed recent context or anchors may omit content needed by later queries.Sliding windows bound memory by retaining only recent context.
- C Additional Related Work Discussion: Compressed-history tokens and recurrent-history methods decouple memory cost from full-history length by summarizing, selecting, evicting, or representing history compactly.Their limitation is that generation often attends to the summary rather than the original history.
- C Additional Related Work Discussion: Unlike conventional retrieval-augmented generation over fixed text corpora or databases, LongLive-RAG searches self-generated video latents and inserts retrieved items as native generator context.Retrieval therefore functions as context selection for long video generation.
- C Additional Related Work Discussion: Video retrieval memories can use camera pose, 3D geometry, scene coordinates, or field-of-view overlap, but open-ended text-to-video generation lacks such explicit retrieval cues by default.This distinction motivates retrieval mechanisms suited to freely moving cameras and objects.
- D Why Retrieval in Latent Space?: Latent retrieval avoids per-block VAE decoding and transfer, while its representations already align with denoising and attention, simplifying discriminative retriever learning.Pixel-space retrieval requires higher-dimensional decoded frames and is less directly tied to the generator’s internal state.
- D Why Retrieval in Latent Space?: Pixel-space retrieval can favor temporally local neighbors and ambiguously map similar decoded appearances to different latent states, histories, or future trajectories.LongLive-RAG indexes the latent/context objects the generator produces and reuses.
E Experiment Details
The appendix specifies LongLive-RAG’s training setup, including rollout-based retrieval data construction, a shared retrieval encoder, and latent autoencoder optimization. It also describes the evaluation protocol and computational costs.
- Training data construction: Training latents are built by sampling 10% of the Self-Forcing prompt pool with seed 0 and generating 90-latent LongLive rollouts using a frozen generator.These rollouts train the retrieval encoder and are not used to filter evaluation prompts or select reported samples.
- Why single base model?: One retrieval encoder is trained unchanged across evaluated backbones in the shared WAN VAE latent space to keep LongLive-RAG general rather than backbone-specific.The appendix contrasts this with training a separate encoder for each target AR backbone or combining rollouts from all base models.
- Latent autoencoder training: The frozen base generators are paired with a convolutional latent autoencoder that independently processes latent blocks and projects encoded features into retrieval embeddings.The encoder uses stride-2 convolutions, GroupNorm, SiLU activations, residual blocks, global average pooling, and a linear projection; the decoder reconstructs the latent.
- Latent autoencoder training: Training uses AdamW with reconstruction, Window Temporal Delta, and trajectory-smoothing losses, while 10% of the latent dataset is reserved for validation and the lowest validation total-loss checkpoint is selected.Reconstruction, sequence-delta, smoothness, and total loss are tracked on both training and validation splits.
- Evaluation protocol: Evaluation uses official VBench-Long scripts, with Table 2 and Table 3 runs taking about one week on 6 NVIDIA RTX A6000 GPUs, plus separate single-GPU retrieval-overhead timing and a 30s-generation VLM evaluation.Latent encoding and top-K search are timed separately, excluding retrieval-encoder data construction and autoencoder training.
F Auxiliary VLM Evaluation
The auxiliary evaluation uses Gemini 3.1-Pro as a VLM judge to score generated videos, averaging aggregate scores from 20 randomly sampled cases among the 30s generations. The scoring prompt adds system instructions, an example, and an explicit output format to the original setup.
- Gemini 3.1-Pro serves as the VLM judge, receiving each generated video and returning an aggregate score.
- 20 cases are randomly sampled from the 30s generations for this auxiliary evaluation, following Deep Forcing and Self-Forcing++.
- The revised system prompt adds system instructions, a scoring example, and an explicit output format to the original VLM scoring setup.
VLM Evaluation Prompt
The prompt instructs an expert assessor to watch the entire video and assign one aggregate Overall Exposure Quality score based on exposure severity, duration, and visibility impact. It defines a 0–3 severity scale, requires spatial and temporal consideration, and specifies an analysis-plus-tagged-integer output format.
- Evaluation Objective: The assessor must evaluate Overall Exposure Quality across the entire video using a single aggregate score.The score should reflect the severity and duration of exposure issues.
- Scoring Scale: Scores range from 0 for catastrophic blown-out or crushed frames to 3 for moderate exposure problems.The supplied scale also defines severe issues as 1 and noticeable persistent clipping or shifts as 2.
- Evaluation Guidelines: The assessor should weigh both the spatial extent and temporal extent of each exposure problem.Mixed-quality videos should be penalized according to how distracting a bad period is.
- Output Format: The response must first summarize exposure over time and then provide the final integer inside <score></score> tags.The analysis should mention specific over- or under-exposed moments when present.
G Ablation Examples
Qualitative ablations show that retrieval-budget choices must preserve both long-range references and video continuity, while the learned LongLive-RAG embedding retrieves more useful historical context than alternative embedding spaces.
- Qualitative examples evaluate both retrieval-budget and embedding-space ablations.
- The balanced retrieval budget preserves both long-range references and video continuity.
- The learned LongLive-RAG embedding retrieves more useful historical context than random or hand-crafted alternatives.