PaperlayerView exact revision on arXiv

Source-linked AI summary

MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generation

Weinan Jia, Yuning Lu, Mengqi Huang, Hualiang Wang, Binyuan Huang, Nan Chen, Mu Liu, Jidong Jiang, Zhendong Mao

arXiv:2510.18692v1cs.CV

TL;DR

Long-video generation needs longer context, but full attention’s quadratic cost makes it prohibitive. MoGA routes tokens into learned groups for sparse attention, and the resulting model generates minute-level multi-shot video while evaluations report gains over sparse and multi-shot baselines.

  • Problem

    Long-video generation needs extended context for persistent environment and character consistency, while full attention’s quadratic cost becomes prohibitive at long sequence lengths.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    MoGA uses a lightweight trainable router to assign correlated tokens to groups and performs self-attention within each group.

    Source: §1 · p. 2, §3.2 · pp. 4–5

  • Results

    Evaluations report consistent gains over sparse-attention baselines and a multi-shot generation model; against full-attention EchoShot, MoGA outperforms it on most metrics.

    Source: §1 · p. 2, §4.1 · pp. 7–8

  • Takeaways & Limitations

    In multi-shot comparisons, preserving salient-token interactions reduces FLOPs and suppresses irrelevant-content noise, alongside stronger character identity consistency and temporal scene coherence.

    Source: §4.1 · p. 7

Abstract

from arXiv · show

Long video generation with Diffusion Transformers (DiTs) is bottlenecked by the quadratic scaling of full attention with sequence length. Since attention is highly redundant, outputs are dominated by a small subset of query-key pairs. Existing sparse methods rely on blockwise coarse estimation, whose accuracy-efficiency trade-offs are constrained by block size. This paper introduces Mixture-of-Groups Attention (MoGA), an efficient sparse attention that uses a lightweight, learnable token router to precisely match tokens without blockwise estimation. Through semantic-aware routing, MoGA enables effective long-range interactions. As a kernel-free method, MoGA integrates seamlessly with modern attention stacks, including FlashAttention and sequence parallelism. Building on MoGA, we develop an efficient long video generation model that end-to-end produces minute-level, multi-shot, 480p videos at 24 fps, with a context length of approximately 580k. Comprehensive experiments on various video generation tasks validate the effectiveness of our approach.

1 Introduction

Long-video generation needs extended context for persistent environment and character consistency, but full attention scales quadratically and existing alternatives have limitations. MoGA uses a lightweight router for groupwise attention, supporting an end-to-end model that generates minute-level multi-shot video and improves over reported baselines.

1 Introduction

  • Long-form video requires scaling effective context length because persistent memory is important for maintaining environment and character consistency.The introduction motivates this need with long-form video such as movies.Source: §1 · p. 1
  • Full attention has quadratic cost, while multistage key-frame pipelines have disjoint objectives, accumulate errors across stages, and introduce hand-crafted biases.Historical-content compression also inevitably loses information.Source: §1 · p. 1
  • Coarse-to-fine sparse routing trades selection performance for coarse-stage efficiency as block size increases; MoGA instead assigns tokens individually.The introduction also couples MoGA with spatiotemporal window attention and adds shot-level textual conditioning because a global prompt cannot reliably control scene transitions or precise events.Source: §1 · p. 2
  • MoGA replaces block-level scoring with precise token assignment to groups through a lightweight router, enabling effective long-context modeling.The router is a single linear layer, and groupwise attention integrates with modern attention kernels such as FlashAttention.Source: §1 · p. 2
  • The model generates minute-level, multi-shot 480p videos at 24 fps and reports consistent improvements over sparse-attention baselines and a multi-shot generation model.The reported context length is about 580k tokens.Source: §1 · p. 2

2 Related Work

Long-video generation research spans multistage, autoregressive, and context-compression paradigms, while sparse-attention work uses static patterns or dynamic routing. The paper positions MoGA against limitations in long-video scope, block-size trade-offs, and inference-time clustering.

2.1 Long Video Generation

  • Prior long-video systems use multistage [17] [34] [37] [42] [44] [56], autoregressive [1] [6] [14] [16] [19] [45], or context-compression [7] [23] [50] paradigms.The cited approaches have limitations in duration or real-world multi-shot generation; multistage methods also pose end-to-end optimization challenges.Source: §2.1 · pp. 2–3
  • LCT [15] pioneered end-to-end multi-shot long-video generation with interleaved prompts and videos, but remains constrained by full attention’s quadratic cost.It models them within a local context window.Source: §2.1 · p. 3

2.2 Sparse Attention for Video Generation

  • Sparse video attention includes static priors and coarse-to-fine dynamic routing; static patterns struggle with evolving long-range dependencies, while block size constrains dynamic methods’ efficiency and expressiveness.The cited static methods include STA [51], SVG [41], and Radial Attention [26]; dynamic methods include VSA [52] and VMoBA [40].Source: §2.2 · p. 3
  • SVG2 [43] clusters tokens online at inference, whereas MoGA uses trainable cluster centroids for efficient routing suited to end-to-end training.The passage notes that SVG2 adds forward-pass k-means computation and is not straightforward to differentiate through.Source: §2.2 · p. 3

3 Method

The method replaces block-level selection with token routing for long-range attention, complements it with local spatiotemporal attention, and builds annotated multi-shot training clips from raw videos.

3.1 Preliminary

  • Full self-attention aggregates global information but costs O(N^2); a one-minute 480p example at 16 fps has about 384k tokens, making full attention intractable.Here N counts tokens across latent spatial and temporal dimensions.Source: §3.1 · p. 3

3.2 Mixture-of-Groups Attention (MoGA)

  • A trainable router assigns correlated tokens to groups for within-group self-attention, reducing complexity from O(N^2) to O(N^2/M) under uniform assignment.The router uses a linear projection followed by softmax gating; an auxiliary balancing loss discourages routing most tokens to only a few groups.Source: §3.2 · pp. 4–5
  • Local spatiotemporal group attention complements MoGA’s long-range coherence with bounded-compute local dependencies, while adjacent-shot keys and values preserve continuity at shot boundaries.Queries are not augmented; each token also receives per-frame attention, and the outputs from one dynamic and two static groups are averaged.Source: §3.2 · pp. 5–6

3.3 Data Pipeline

  • The data pipeline filters and segments raw videos, processes single-shot clips, then merges adjacent clips into multi-shot training samples of up to 65 seconds.Shot-level processing uses VQA and OCR, crops watermarks and subtitles when sufficient area remains, generates captions, and trims frames affected by transition overlap.Source: §3.3 · p. 6

4 Experiments

The experiments evaluate MoGA on single-shot, multi-shot, and long-video generation, with Wan2.1 and MMDiT settings, and report quantitative, qualitative, and ablation results. They assess performance, consistency, and computational efficiency across these settings.

4 Experiments

  • The experiments fine-tune Wan2.1 models for 30-second videos and use an MMDiT-based model for 60-second, 480p video generation at 24 fps with a 578k-token context.The evaluation includes single-shot, multi-shot, and long-video test sets, with VBench metrics and cross-shot CLIP and DINO scores.Source: §4 · pp. 6–7

4.1 Quantitative Results

  • MoGA improves over sparse baselines across metrics for 5-second single-shot generation and outperforms the full-attention EchoShot on most metrics for 10-second multi-shot generation.The comparisons use the evaluation settings described for each task.Source: Table 1 · p. 7, §4.1 · p. 7, Table 2 · p. 7
  • For 30-second multi-shot generation, MoGA substantially outperforms IC-LoRA+Wan under the same backbone, including with aggressive sparsity and high visual fidelity using MMDiT.MoGA with MMDiT achieves 2.26 PFLOPs versus 6.94 PFLOPs for full attention and a 1.7× speedup in training and inference on 30-second videos.Source: Table 3 · p. 7, §4.1 · p. 8, §4.3 · p. 10

4.2 Qualitative Results

  • Across 30-second comparisons, qualitative results show MoGA maintaining coherent content and shot-spanning identity, while IC-LoRA+Wan often exhibits subject drift and background inconsistency.The comparison notes IC-LoRA+Wan’s typical three-frame per-iteration cap; EchoShot also shows cross-shot inconsistencies over long temporal ranges.Source: §4.2 · pp. 8–9, Figure 5 · p. 8
  • The one-minute example preserves details across shots and avoids identity confusion across multiple faces; results also show background consistency and coherent long-form animation across styles.The one-minute video contains 1,441 frames, and background consistency is observed even when details are not explicitly specified.Source: §4.2 · p. 9, Figure 8 · p. 10, Figure 9 · p. 10

4.3 Ablation Study

  • With Wan2.1-1.3B on 30-second videos, MoGA uses 2.26 PFLOPs versus 6.94 PFLOPs for full attention and delivers a 1.7× speedup in training and inference.The reported savings occur with M=5; unlike VMoBA, MoGA adds no memory consumption.Source: §4.3 · p. 10
  • Under a fixed computational budget, cross-shot DINO and CLIP scores rise and then fall as group count increases, suggesting a moderate group count balances consistency and efficiency.The study describes this balance as yielding near-optimal consistency while maintaining computational efficiency.Source: §4.3 · p. 10, Table 4 · p. 11
  • At 71.25% sparsity, MoGA’s subject consistency is on par with or sometimes better than full attention, while combining MoGA with STGA supports strong cross-shot consistency.MoGA alone lacks local information exchange; STGA alone limits long-range shot interactions.Source: §4.3 · pp. 10–11

5 Conclusion

MoGA replaces coarse block scoring with learned group assignments, and the resulting model generates minute-level, multi-shot 480p video at 24 fps. Experiments across video-generation tasks demonstrate the approach’s effectiveness.

5 Conclusion

  • MoGA replaces coarse block-level scoring with precise, learned group assignments through a lightweight token router.The paper reports that routing tokens into coherent groups improves attention efficiency and fidelity for very long contexts.Source: §5 · p. 11
  • The MoGA-based model produces minute-level, multi-shot videos at 480p resolution and 24 fps.Source: §5 · p. 11
  • Experiments across diverse video-generation tasks demonstrate the effectiveness of the approach.Source: §5 · p. 11

A Details of the Computational Complexity

The computational-cost analysis varies group number and video duration for Wan2.1-1.3B. It reports approximately linear growth for STGA and MoGA compute of approximately 1/M of Full Attention.

A Details of the Computational Complexity

B Analysis of Group Balancing Loss

The group-balancing loss is evaluated by tracking how evenly the router assigns tokens. With the loss, the metric converges around 1; without it, the metric rises as assignments concentrate in a few groups.

B Analysis of Group Balancing Loss

Related papers

Loading 2510.18692v1…