PaperlayerView exact revision on arXiv

Source-linked AI summary

Mixture of Contexts for Long Video Generation

Shengqu Cai, Ceyuan Yang, Lvmin Zhang, Yuwei Guo, Junfei Xiao, Ziyan Yang, Yinghao Xu, Zhenheng Yang, Alan Yuille, Leonidas Guibas, Maneesh Agrawala, Lu Jiang, Gordon Wetzstein

arXiv:2508.21058v3cs.GRcs.AIcs.CV

TL;DR

Long-video generation is constrained by the need for long-term memory and the quadratic cost of dense self-attention. MoC treats generation as internal information retrieval with learned sparse routing, and it demonstrates minute-scale memory with substantial pruning, lower attention cost, and faster generation while maintaining long-video consistency. The current implementation, however, has not yet been evaluated for computation savings on sequences longer than the tested minute-scale setting.

  • Problem

    Long-video generation requires retaining and retrieving salient events over extended timelines, while dense self-attention makes long sequences computationally and memory intensive.

  • Method

    MoC replaces dense diffusion-transformer attention with content-aligned, per-head learnable sparse routing to relevant context chunks.

  • Results

    85% context pruning delivers a 2.2× generation speedup, up to 7× lower attention FLOPs, and sustained identity, action, layout, and scene consistency over minutes-long sequences.

  • Takeaways & Limitations

    Learned sparse context routing can provide minute-scale long-term video memory at a cost comparable to short-video generation without explicit 3D or FoV heuristics.

  • Takeaways & Limitations

    MoC's computation savings on sequences longer than the minute-scale evaluation setting remain unexplored, and further speedups require specialized attention-system co-design.

Abstract

from arXiv · show

Long video generation is fundamentally a long context memory problem: models must retain and retrieve salient events across a long range without collapsing or drifting. However, scaling diffusion transformers to generate long-context videos is fundamentally limited by the quadratic cost of self-attention, which makes memory and computation intractable and difficult to optimize for long sequences. We recast long-context video generation as an internal information retrieval task and propose a simple, learnable sparse attention routing module, Mixture of Contexts (MoC), as an effective long-term memory retrieval engine. In MoC, each query dynamically selects a few informative chunks plus mandatory anchors (caption, local windows) to attend to, with causal routing that prevents loop closures. As we scale the data and gradually sparsify the routing, the model allocates compute to salient history, preserving identities, actions, and scenes over minutes of content. Efficiency follows as a byproduct of retrieval (near-linear scaling), which enables practical training and synthesis, and the emergence of memory and consistency at the scale of minutes.

1 Introduction

Long-video generation requires selective long-term memory, but dense attention becomes prohibitive as sequences grow. MoC reframes the problem as learned internal retrieval, dynamically routing queries to relevant context while preserving minute-scale coherence and improving efficiency.

  • Long-video models must retain and retrieve salient events across extended timelines without drift, collapse, or identity loss.
  • Dense self-attention becomes computationally prohibitive for minute- or hour-long video sequences, making selective recall as important as computational scaling.
  • Prior approaches compress history or impose fixed sparse patterns, extending feasible horizons while hard-coding a compromise between long-range dependencies and narrative coherence.
  • MoC reformulates long-context video generation as internal information retrieval through learnable sparse attention routing over relevant video segments.
  • 85% token-pair pruning, up to 7× lower attention FLOPs, and 2.2× faster end-to-end generation enable minute-scale scenes while maintaining or surpassing dense-baseline fidelity and consistency.The speedup is measured on scenes of approximately 180k tokens.

2 Related Work

Related work addresses long-video generation through recurrent or hierarchical representations, robustness training, dense enlarged windows, sparse attention, and explicit context retrieval. These approaches improve horizon or coherence, but differ in how they manage computational scaling and historical context.

  • Long Video Generation: Existing long-video systems use recurrent states, diffusion hierarchies, autoregressive chunking, or noise-based training to extend generation beyond the few-second horizon.
  • Long Video Generation: LCT preserves multi-shot coherence by keeping attention dense across enlarged text-video windows, but FLOPs and memory retain quadratic scaling with sequence length.
  • Sparse Attention for Video Generation: Sparse-video methods compute attention over selected heads, localized 3D windows, or filtered token pairs to exploit spatiotemporal redundancy.
  • Context Learning in Visual Generation: Context-learning methods treat past frames, states, or reference images as explicit signals, retrieving historical entries or frames to preserve long-term scene consistency.

3 Method

MoC replaces dense attention with content-aligned, query-specific sparse routing that retrieves salient chunks while preserving mandatory local and textual context. Causal and distributed routing, pooling, and regularization support long-range retrieval with substantially lower computation.

  • Adaptive routing: MoC partitions multimodal streams into content-aligned chunks and routes each query to relevant chunks instead of using uniform or static attention.Chunks follow frames, shots, and modality boundaries; selected chunks are forwarded to attention while unselected tokens are skipped.
  • Adaptive routing: Mean-pooled chunk keys provide descriptors whose similarity to each query determines the top-k context selection.The query–descriptor relevance is computed by inner product, and the selected chunks supply the attention context.
  • Causal routing: Causal routing prevents reciprocal chunk selections from forming isolated feedback loops that limit communication with earlier shots.Without causal masking, mutually routing chunks can trap information in two-node cycles.
  • Adaptive routing: Per-head distributed routing lets each attention head independently select k chunks, expanding collective coverage across layers and heads.This avoids the information bottleneck of one global selection while matching different heads’ feature specializations.
  • Fixed context: Mandatory caption and intra-shot connections preserve semantic anchors and local continuity while leaving sparse capacity for long-range dependencies.Text anchors provide persistent conditioning, while intra-shot links capture trajectories, lighting continuity, and related local cues.
  • Computation efficiency: >7× fewer multiply–adds are achieved on an approximately 180k-token minute-long sequence with MoC than with dense self-attention.Using m̄ ≈1024, k = 5, C = 36, and d = 128, MoC costs approximately 2.32 × 10^12 FLOPs versus 1.66 × 10^13 for dense attention.

4 Experiments

The experiments evaluate MoC on minute-long, multi-shot text-to-video scenes against dense-attention LCT, measuring computational efficiency, video quality, and long-term coherence. MoC substantially reduces attention computation while preserving or improving key generation qualities and cross-shot consistency.

  • Experimental Setup: The main experiment uses 8-shot, 64-second 480p scenes totaling roughly 180k tokens, comparing MoC with dense-attention LCT.Each shot is 8 seconds at 12 FPS.
  • Evaluation Metrics: Evaluation uses Subject Consistency, Background Consistency, Motion Smoothness, and Dynamic Degree to assess fidelity, temporal quality, and motion.The metrics capture preservation of subjects and backgrounds, fluid movement, and the extent of motion.
  • Quantitative Results: 85% context pruning delivers a 2.2× speedup while maintaining Motion Smoothness and keeping all quality metrics high.The reported trade-off includes a slight reduction in appearance fidelity alongside improved motion diversity.
  • Quantitative Results: Dynamic Degree increases from 0.46 to 0.56 relative to dense attention, indicating greater motion diversity in the evaluated scenes.The paper reports this improvement while noting a slight reduction in appearance fidelity and maintained Motion Smoothness.
  • Qualitative Results: Qualitative results are visually indistinguishable from LCT despite pruning more than three-quarters of the attention calculation.The comparison is presented for multi-shot video generation.
  • Qualitative Coherence: MoC preserves geometric backgrounds, semantic transitions, fine-grained object details, spatial layouts, and distinct identities across shot boundaries.The qualitative examples include landmarks, architecture, signage, computer details, and multiple characters.

5 Conclusion

The conclusion presents MoC as a learned, data-driven retrieval engine that provides minute-scale video memory without explicit selection heuristics. It also identifies longer-sequence evaluation and specialized systems optimization as open directions.

  • Conclusion: MoC demonstrates that learnable sparse attention routing can serve as a data-driven memory retrieval engine for long-video generation.The model learns which historical context is salient from data rather than relying on explicit heuristics such as 3D priors or FoV selection.
  • Conclusion: The approach achieves minute-scale memory at a cost comparable to short-video generation while preserving long-term recall capabilities.The conclusion attributes this capability to scaling training data together with efficient, learnable sparse routing.
  • Conclusion: MoC's qualitative coherence spans persistent backgrounds, semantic alignment, object details, spatial layouts, and multiple distinct subjects across shots.The examples include city landmarks, sketch-to-architecture transitions, signage, computer components, and a multi-character car interior.
  • Conclusion: The authors frame removing quadratic attention as both an efficiency improvement and a route toward emergent long-term video memory.This conclusion connects the computational bottleneck with the paper's reported memory behavior.
  • Limitations and Future Work: MoC has so far been trained and tested only under LCT's setup, leaving its computation savings on longer sequences unexplored.The authors identify longer-sequence evaluation as future work.
  • Limitations and Future Work: Further speedups remain possible through hardware–software co-design, including block-sparse attention, fused operators, persistent execution, and improved K/V layouts or quantization.The current runtime relies on general-purpose variable-length attention and framework-level gathers despite a reported 7× FLOPs saving.

A Memory Complexity Analysis.

MoC reduces attention complexity through chunk-level routing, while storing routing metadata at substantially lower cost than dense attention. Its implementation keeps this overhead negligible in the overall GPU memory footprint.

  • Complexity: Sparse attention reduces computational complexity from O(L^2) to roughly O(k ·L), but introduces routing metadata for mean-pooled keys, logits, and selection indices.The metadata supports selecting a small number of relevant context chunks.
  • Routing Metadata: With N ≈L/C chunks, representative mean-pooled keys require O(N ·d) memory, or roughly 1/C of the full KV-cache memory.The typical chunk size is C ∈[10^3, 10^4].
  • Routing Metadata: Routing logits require O(L·N) = O(L^2/C) space, reducing dense attention-map storage by a factor of C.The reduction follows from routing over chunks rather than individual tokens.
  • Implementation: Routing overhead remains below 0.1% of total GPU memory because pooled representations are computed on-the-fly and sparse operations avoid intermediate expansions.The implementation uses torch.segment_reduce and a custom autograd function wrapping Flash-Attention kernels.

B MoC Implementation Benchmark.

Benchmarking against full Flash Attention 2 shows that adaptive MoC maintains near-linear scaling in FLOPs and latency as the number of shots and sequence length increase. The implementation combines sparsity with memory-efficient routing and attention execution.

  • Scaling Benchmark: MoC remains near-linear in FLOPs and latency with respect to the number of shots, and therefore sequence length L.The benchmark compares adaptive MoC with full attention implemented using Flash Attention 2.
  • Implementation Choices: On-the-fly segment_reduce pooling avoids materializing variable-length chunks in memory during routing.This is one of three implementation choices identified as central to efficiency.
  • Implementation Choices: Head-major token organization provides coalesced memory access during gather operations.The layout is designed to improve the efficiency of sparse context retrieval.
  • Implementation Choices: A custom wrapper combines routing and attention computation with Flash-Attention kernels.The benchmark attributes efficiency to the joint design of sparsity and these implementation choices.

C Dataset Details.

The dataset combines authentic multi-shot scenes from public narrative videos with additional smoothly evolving single-shot sequences. It provides scene-level captions, ordered shot clips, aligned shot descriptions, and explicit identity and transition annotations.

  • Videos are collected across movies, TV series, and documentaries, then segmented into scenes and shots using scene-boundary detection.
  • The dataset contains approximately 500K annotated scenes averaging about 5 shots per scene, or roughly 2.5M shot clips.
  • Shot captions identify people with global IDs such as “Character 1” rather than ambiguous gendered descriptions.
  • An additional augmentation contributes roughly 1M scene-level samples from long single-shot videos segmented by detected event changes.
  • Each training example includes a global scene caption, ordered shot or segment clips, aligned shot-level captions, and optional [SHOT CUT] markers for true transitions.

D Zero-shot Experiment

The zero-shot experiment replaces dense attention with highly sparse MoC routing in a pretrained DiT without fine-tuning. The resulting model retains some identity, layout, and coarse-motion consistency, supporting mean-pooled chunk descriptors as usable retrieval signals.

  • >75% sparsity preserves a certain amount of subject identity, background layout, and coarse motion without fine-tuning.
  • Freezing all pretrained weights isolates the routing benefit from any fine-tuning improvement.
  • A parameter-free arithmetic mean descriptor approximates each chunk’s first principal component and aligns with dominant foreground and background patterns.
  • The zero-shot result indicates that mean-pooled chunk keys can convert dense attention into a retrieval step even when sparse masks were absent during training.

E Single-shot Short Video Generation

On short single-shot videos, MoC performs at least as well as the dense baseline despite aggressive sparsification. However, routing overhead exceeds the computational savings at this short sequence length, slowing end-to-end generation.

  • The evaluation uses 8-second videos at 320×192 resolution and 12 FPS, yielding roughly 6,300 tokens per video.
  • MoC matches or surpasses the dense baseline across all VBench metrics despite 83% sparsification.
  • For short sequences, index gathering and pooling overhead outweigh computational savings, resulting in slower end-to-end generation.
  • The single-shot configuration uses chunk size 256 and top-k=3 with intra-chunk and forced cross-modal links enabled.

G Ablation Study

The ablations examine routing granularity and forced links. Smaller chunks can harm motion by restricting far-context access, while forced intra-shot links stabilize training and cross-modal links generally improve performance.

  • The ablations vary chunk size and top-k routing for single-shot generation, while forced-link experiments focus on multi-shot generation.
  • Chunk size and k: Tiny 64- or 128-token chunks prune aggressively but harm motion, potentially because queries lose access to far-context frames.
  • Chunk size and k: A progressive schedule that begins with larger chunks and larger k before reducing both may support more aggressive sparsification.
  • Force links and Context Drop In & Out: Removing forced intra-shot links makes training extremely unstable, whereas retaining them provides a sufficiently reasonable lower bound and greater stability.
  • Force links and Context Drop In & Out: Adding cross-modal links and Context Drop In & Out generally improves overall model performance.

H Wan-2.1-1.3B Experiment

The MoC experiment on Wan-2.1-1.3B tests whether the approach generalizes beyond the original backbone. Results show sparsity is at least on par with, and often better than, dense attention, while moderate sparsity avoids visible artifacts without fine-tuning.

  • Generalization: MoC generalizes to the Wan-2.1-1.3B open-source DiT backbone without model-wise adaptation.The experiment applies MoC only to self-attention modules using the same hyperparameters as the single-shot experiment.
  • Results: On Wan-2.1-1.3B, sparse attention is at least on par with and often better than dense attention.The comparison fine-tunes pretrained models with either dense attention or Mixture-of-Attention.
  • Practical boundary: Without fine-tuning, MoC performs reasonably well with few visible artifacts when sparsity does not become too low.This condition marks the practical boundary observed in the Wan-2.1-1.3B experiment.
Loading 2508.21058v3…