PaperlayerView exact revision on arXiv

Source-linked AI summary

UltraViCo: Breaking Extrapolation Limits in Video Diffusion Transformers

Min Zhao, Hongzhou Zhu, Yingze Wang, Bokai Yan, Jintao Zhang, Guande He, Ling Yang, Chongxuan Li, Jun Zhu

arXiv:2511.20123v2cs.CV

TL;DR

Video diffusion transformers trained on fixed sequence lengths struggle to generate longer videos, with repetition in some models and quality degradation across models. UltraViCo treats both as consequences of attention dispersion and suppresses attention to tokens beyond the training window. It extends the reported practical extrapolation limit from 2× to 4× and also applies to controllable video synthesis and editing.

  • Problem

    Video diffusion transformers trained on fixed maximum sequence lengths struggle to generate longer videos, despite video length extrapolation being important for practical applications.

    Source: §1 · p. 1

  • Method

    UltraViCo is a training-free method that suppresses attention for out-of-window tokens using a constant decay factor to address attention dispersion.

    Source: §1 · p. 2, §3.3 · p. 7

  • Results

    UltraViCo extends the practical extrapolation limit from 2× to 4× and consistently surpasses baselines across evaluated models and extrapolation ratios.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The method also generalizes to downstream tasks including controllable video synthesis and editing.

    Source: §1 · p. 3

Abstract

from arXiv · show

Despite advances, video diffusion transformers still struggle to generalize beyond their training length, a challenge we term video length extrapolation. We identify two failure modes: model-specific periodic content repetition and a universal quality degradation. Prior works attempt to solve repetition via positional encodings, overlooking quality degradation and achieving only limited extrapolation. In this paper, we revisit this challenge from a more fundamental view: attention maps, which directly govern how context influences outputs. We identify that both failure modes arise from a unified cause: attention dispersion, where tokens beyond the training window dilute learned attention patterns. This leads to quality degradation and repetition emerges as a special case when this dispersion becomes structured into periodic attention patterns, induced by harmonic properties of positional encodings. Building on this insight, we propose UltraViCo, a training-free, plug-and-play method that suppresses attention for tokens beyond the training window via a constant decay factor. By jointly addressing both failure modes, we outperform a broad set of baselines largely across models and extrapolation ratios, pushing the extrapolation limit from 2x to 4x. Remarkably, it improves Dynamic Degree and Imaging Quality by 233% and 40.5% over the previous best method at 4x extrapolation. Furthermore, our method generalizes seamlessly to downstream tasks such as controllable video synthesis and editing.

1 INTRODUCTION

Video diffusion transformers trained at fixed lengths struggle to generate longer sequences, with periodic repetition in some models and quality degradation across all models. UltraViCo analyzes these failures through attention and suppresses out-of-window attention, extending the reported practical extrapolation limit from 2× to 4×.

1 INTRODUCTION

  • Video length extrapolation is challenging because models trained at fixed maximum sequence lengths struggle to generate longer videos, a capability described as critical for practical applications.The paper characterizes repetition as model-specific and quality degradation as universal; both worsen with longer extrapolations.Source: §1 · p. 1, Figure 2 · p. 3
  • Prior work RIFLEx addresses repetition through positional encodings but overlooks quality degradation, while attention maps directly govern how context influences outputs.The paper frames attention as a more fundamental lens because positional encodings affect attention indirectly through queries and keys.Source: §1 · p. 1
  • Attention dispersion unifies the two failure modes: out-of-window tokens dilute learned attention patterns, with repetition arising when dispersion forms periodic patterns.The passage links those periodic patterns to harmonics in positional-encoding frequencies.Source: §1 · p. 2
  • UltraViCo is a plug-and-play method that applies a constant decay factor to suppress attention for tokens beyond the training window.This reallocates attention toward reliable in-window context and is described as addressing both failure modes.Source: §1 · p. 2
  • UltraViCo consistently outperforms baselines across models and extrapolation ratios, extending the practical limit from 2× to 4× while maintaining fluid motion where prior methods collapse beyond 3×.The introduction also reports improvements in Dynamic Degree and Imaging Quality at 4× extrapolation and generalization to controllable synthesis and editing.Source: §1 · pp. 2–3

2 PRELIMINARY

The preliminary section defines video length extrapolation as generating a sequence longer than a pretrained model’s training length in a single forward generation. It reviews how rotary position encoding feeds the attention computation and narrows the analysis to the temporal axis.

2 PRELIMINARY

  • DiTs patch video into tokens and project them into queries, keys, and values; RoPE injects position into queries and keys through complex rotations.The passage describes RoPE as the positional encoding mainly adopted by modern video diffusion models.Source: §2 · p. 3
  • Query-key interactions yield attention logits, softmax normalizes them into attention scores, and those scores are applied to values to produce the output.The preliminary section defines frequency components and the resulting Q, K, attention, and output matrices.Source: §2 · p. 3
  • For temporal extrapolation, the analysis considers only the temporal subspace of M-RoPE and denotes its dimension dT as d.M-RoPE partitions dimensions into temporal, height, and width subspaces.Source: §2 · p. 3
  • Video length extrapolation adapts a pretrained model to generate a sequence longer than its training length in one forward generation, with ratio s = L′/L > 1.This target is orthogonal to methods relying on inference-time modifications.Source: §2 · p. 3

3 METHOD

Video length extrapolation exhibits model-specific repetition and universal quality degradation, both of which worsen as extrapolation grows. Attention analysis links these failures to dispersed attention, with periodic patterns as a special case, and motivates UltraViCo’s suppression of out-of-window attention.

3.1 FAILURE MODES OF VIDEO LENGTH EXTRAPOLATION

  • Both periodic repetition and universal quality degradation become more severe as extrapolation length grows, motivating analysis of their possible shared cause.Prior positional-encoding work addresses repetition but overlooks quality degradation.Source: §3.1 · p. 4

3.2.1 THE CAUSE OF CONTENT REPETITION: PERIODIC ATTENTION PATTERNS

  • HunyuanVideo’s RoPE frequencies align harmonically, concentrating 79.6% of total amplitude in a dominant component, while Wan’s largest frequency contributes 31.6%.Near-equal scores at key positions separated by the repetition period, combined with relative positional invariance, make attention rows repeat and retrieve nearly the same information; Wan lacks this row-wise periodicity. Masking harmonic-alignment positions disrupts it.Source: §3.2.1 · pp. 4–6, Figure 3 · p. 5

3.2 ATTENTION ANALYSIS OF THE CAUSE

  • Masking harmonic-alignment positions disrupts periodic attention and mitigates repetition; this intervention also improves video quality by concentrating attention.The analysis links repetition and quality degradation, prompting the investigation of attention dispersion.Source: §3.2 · p. 4, §3.2.1 · p. 6, §3.2.2 · p. 6

3.2.2 THE CAUSE OF QUALITY DEGRADATION: ATTENTION DISPERSION

  • Progressively masking out-of-window attention scores improves visual quality and motion dynamics, supporting attention dispersion as the cause of quality degradation.The proposed account is that distant frames dilute learned attention, impairing spatial detail and mixing local motion with unrelated movement; periodic attention is a special case when a frequency contributes substantially to overall amplitude.Source: §3.2.2 · pp. 6–7, Figure 5 · p. 7

3.3 ULTRAVICO

  • UltraViCo leaves in-window logits unchanged and down-scales positive out-of-window logits by a constant factor; periodic-repetition models receive stronger decay near harmonic-alignment positions.The targeted adjustment is intended to preserve learned in-window dynamics while reallocating attention from risky positions; the method is integrated with FlashAttention and SageAttention to avoid explicit mask construction for long sequences.Source: §3.3 · pp. 7–8

4 EXPERIMENTS

Across video models and extrapolation ratios, UltraViCo improves quality and eliminates repetition, extending the practical extrapolation limit from 2× to 4×. The experiments also examine decay settings, integration with long-video methods, and downstream tasks.

4.1 SETUP

  • Evaluation covers HunyuanVideo, Wan2.1-1.3B, and CogVideoX-5B, using 100 VBench prompts and measures of quality, dynamics, consistency, repetition, and user rankings.Wan and HunyuanVideo use α = 0.9 at 3× and 4×; HunyuanVideo uses β = 0.6 at 3× and 0.8 at 4×.Source: §4.1 · p. 8

4.2 RESULTS

  • Dynamic Degree and Imaging Quality on HunyuanVideo improve by 233% and 40.5%, respectively, over the previous best method at 4× extrapolation.Across models and extrapolation ratios, UltraViCo outperforms all baselines while improving video quality and eliminating content repetition.Source: §4.2 · pp. 8–9
  • On Wan at 3× extrapolation, constant, linear, and parabolic decay yield minor differences, while α = 0.9 enhances quality while preserving consistency.The sensitivity analysis reports improved quality and motion dynamics with consistency near baseline for α ≥ 0.9 and β ≥ 0.6.Source: §4.2 · pp. 9–10, Figure 8 · p. 10
  • At 6× extrapolation for 30-second Wan videos, UltraViCo extends the base model’s training window by 3× and improves long-term consistency in existing long-video methods without harming other performance.The comparison covers FreeNoise, FIFO-Diffusion, and sliding-window methods.Source: §4.2 · p. 10, Table 2 · p. 10
  • Based on VACE, UltraViCo enables 3× extrapolation in controllable video generation and editing.The paper reports additional downstream-task results in Appendix C.4.Source: §4.2 · p. 10

5 CONCLUSION

UltraViCo identifies attention dispersion as the unified cause of video length extrapolation failures and extends the practical limit from 2× to 4×. The paper also emphasizes responsible use to avoid misleading or harmful content.

5 CONCLUSION

  • UltraViCo identifies attention dispersion as the unified cause of video length extrapolation failures and extends the practical extrapolation limit from 2× to 4×.The method is training-free and suppresses attention scores for tokens beyond the training length.Source: §5 · p. 10

ETHICS STATEMENT

  • The paper emphasizes responsible use of video generation to avoid potential misleading or harmful content.Source: ETHICS STATEMENT · p. 10

USE OF LARGE LANGUAGE MODELS

The paper states that a large language model assisted only with English polishing, then reviews video-generation and length-extrapolation approaches. It also presents properties of periodic attention scores and describes the temporal, height, and width components of M-RoPE.

USE OF LARGE LANGUAGE MODELS

  • A large language model was used only to polish English writing and improve clarity; the research ideas, experiments, results, and interpretations are the authors’ own.Source: USE OF LARGE LANGUAGE MODELS · p. 15

A RELATED WORK

  • The related-work discussion describes positional-encoding adjustments for length extrapolation, while noting that these methods can perform poorly on image and video diffusion transformers.RIFLEx is described as mitigating repetition while still suffering degraded visual quality; UltraViCo is presented as addressing both repetition and quality degradation.Source: A RELATED WORK · pp. 15–16
  • Prior long-video approaches alter inference or use alternative modeling paradigms, whereas length extrapolation extends diffusion transformers’ intrinsic capacity and can integrate with those approaches.The paper mentions noise initialization, sequential denoising windows, and distributed computation as examples of long-video methods.Source: A RELATED WORK · p. 16
  • The paper notes that autoregressive methods and diffusion forcing can also generate long videos, and that UltraViCo may offer insights for their length extrapolation.Source: A RELATED WORK · p. 16

B.1 DERIVATION OF THE PERIODIC OUTPUTS

  • The appendix states that attention scores are periodic across key positions and invariant to equal query-key shifts, up to negligible error.Source: §B.1 · p. 16

B.2 DETAILS OF THE MULTIMODAL ROTARY POSITION EMBEDDING

  • M-RoPE divides each token vector into temporal, height, and width subspaces, each assigned its own frequency series.Source: §B.2 · p. 17

B.3 DERIVATION OF THE STATISTICAL ATTENTION PATTERN ¯S(∆t)

The appendix derives a statistical attention-logit pattern from temporal offsets and reports that its periodic or non-periodic characteristics persist when actual variance is included. It also discusses strict and approximate harmonic conditions and their model-specific implications.

B.3 DERIVATION OF THE STATISTICAL ATTENTION PATTERN ¯S(∆t)

  • The expected attention logits are approximated by a cosine-frequency sum after treating variance terms as nearly constant and combining trigonometric terms.The analysis focuses on temporal offsets with spatial offsets held fixed for simplicity.Source: §B.3 · pp. 17–18

B.4 CONSISTENCY OF ACTUAL ATTENTION PATTERN WITH ¯S(∆t)

  • With true variance included, HunyuanVideo retains periodic attention peaks while Wan 2.1 remains non-periodic, supporting the mean-based analysis.The appendix reports that bias angles are close to zero, except for low-amplitude components in Wan.Source: §B.4 · pp. 18–19, Figure 10 · p. 19

B.5 PROOF OF PROPOSITION 1

  • When each frequency ratio to the lowest frequency is a positive integer, the cosine sum has a period of 2π/ϕN−1.Source: §B.5 · p. 19

B.6 REMARKS ON PROPOSITION 1

  • For CogVideoX, the first two frequency ratio is 3.16, near integer 3, and the dominant component accounts for 50.80% of total amplitude, producing approximate periodicity.The passage connects this pattern to content repetition and identifies Fig. 11 as its visualization.Source: §B.6 · p. 20, Figure 11 · p. 20
  • HunyuanVideo’s strict theoretical period is 804, but its corresponding frequency contributes only 6.677% of amplitude and has little effect within the 132-token range at 4× extrapolation.The analysis therefore focuses on frequencies ϕ0 through ϕ3, whose contributions are sufficiently large and oscillatory.Source: §B.6 · p. 20

B.7 NECESSITY OF CONCENTRATING ON THE TRAINING WINDOW

On Wan at 3× extrapolation, restricting attention to the training window gives the strongest quality improvement among the tested strategies. CogVideoX at 3× exhibits reduced dynamics and quality alongside content repetition.

B.7 NECESSITY OF CONCENTRATING ON THE TRAINING WINDOW

  • Restricting Wan’s attention to the training window gives the most significant video-quality improvement at 3× extrapolation; leading- or trailing-token focus collapses video, while top-token focus yields poor quality and little dynamics.Source: §B.7 · pp. 20–21

C.1 FAILURE MODES OF COGVIDEOX

  • At 3× extrapolation, CogVideoX videos show a sharp decline in dynamic degree and visual quality, with noticeable content repetition.The figure caption describes degraded visual quality, reduced dynamics, and clear repetition.Source: §C.1 · p. 21, Figure 13 · p. 21

C.2 MORE IMPLEMENTATION DETAILS

The implementation details adapt repetition scoring for longer extrapolations and use a distinct frequency choice for Wan. Ablations find that distinguishing in-window from out-of-window tokens matters more than the decay shape, while the first frame receives a negative decay factor to address blurring.

NoRepeat Score

  • The NoRepeat Score is modified for longer extrapolations because the original RIFLEx implementation applies only when content repeats once.Its procedure filters static videos using eight sampled frames, then measures repeated-frame ratios; a threshold of 55 is used in both steps.Source: §C.2 · p. 21

Wan implementation

  • For Wan, which does not exhibit content repetition, the largest-amplitude frequency ϕ0 is used as the dominant frequency.The usual repetition-period-based frequency determination is therefore not applicable to Wan.Source: §C.2 · pp. 21–22

First-frame handling

  • The first frame’s decay factor is set negative to address blurring, hypothesized to reflect its distinct encoding in the causal video VAE.The authors hypothesize that the first frame is more sensitive to perturbations because it is encoded independently and without temporal compression.Source: §C.2 · p. 22

Decay strategies

  • Parabolic, linear, and constant decay strategies yield only minor differences; separating in-window from out-of-window tokens is the key distinction.The constant strategy uses α = 0.9, while the parabolic and linear strategies use α1 = 0.85 and α2 = 0.95, respectively.Source: §C.2 · p. 22

C.3 ADDITIONAL EXPERIMENTS OF DIFFERENT EXTRAPOLATION RATIOS AND MODELS

Additional evaluations on HunyuanVideo, Wan, and CogVideoX report that UltraViCo performs best across the tested models and extrapolation ratios. The results include substantial gains for CogVideoX at 4× and 5× extrapolation.

Settings

  • The evaluation tests 2× extrapolation on HunyuanVideo, Wan, and CogVideoX, plus CogVideoX at 3× and 4× and HunyuanVideo at 5×.The settings use 25 VBench prompts and specify model-specific α, β, and γ configurations.Source: §C.3 · p. 22

Results

  • UltraViCo achieves the best performance across all tested models and extrapolation ratios, improving CogVideoX Dynamic Degree and Imaging Quality by 200% and 13.48% at 4×.At 4×, CogVideoX baselines have Dynamic Degree ≤16 and Imaging Quality ≤56.Source: §C.3 · p. 22
  • At 5× extrapolation, UltraViCo surpasses the best baseline scores by 350% in Dynamic Degree and 47.59% in Imaging Quality.The authors describe this as indicating potential to extend to larger extrapolation ratios.Source: §C.3 · p. 22

C.4 MORE QUALITATIVE RESULTS OF OUR METHOD

The additional qualitative results show improved visual quality and motion across Wan and CogVideoX, and pose-conditioned outputs closely follow their supplied poses. Further sections report acceleration, runtime and memory costs, and integration with online attention kernels.

C.4 MORE QUALITATIVE RESULTS OF OUR METHOD

  • Across 3× and 4× extrapolation on Wan and CogVideoX, UltraViCo produces more fluid motion and richer, more natural visual results than baselines.The Wan baselines are nearly static, while CogVideoX baseline videos are blurry with dull colors.Source: §C.4 · p. 22, Figure 14 · p. 24, Figure 15 · p. 24

C.5 ACCELERATION OF ULTRAVICO VIA SPARSE ATTENTION AND DISTILLATION

  • UltraViCo achieves about 16× speed-up without compromising performance when combined with recent video-acceleration methods.Table 7 reports illustrative performance for combinations on HunyuanVideo.Source: §C.5 · p. 23

C.6 RUNTIME AND MEMORY COST

  • UltraViCo incurs almost no additional latency or memory overhead when built on FlashAttention and SageAttention.The runtime and memory comparison is presented in Table 8.Source: §C.6 · p. 23

D.1 ULTRAVICO WITH EFFIEIENT ONLINE ATTENTION

  • UltraViCo does not require materializing the full attention matrix and can be integrated into efficient online attention kernels.The appendix presents an implementation based on FlashAttention.Source: §D.1 · p. 23

C.4 MORE QUALITATIVE RESULTS OF OUR METHOD

  • Pose-guided generation closely follows the given pose conditions while maintaining high dynamic range and visual quality.Source: Figure 16 · p. 25

D.2 ABLATION ON HYPERPARAMETERS

The appendix describes sensitivity analyses of α and β and an application to segment-wise long-video generation. It also identifies illustrative tables on combining methods and on runtime and memory.

Hyperparameter sensitivity

  • The hyperparameter analysis includes a sensitivity curve and tables examining α, β, and their independent effects, but the supplied passages report no specific sensitivity outcome.Table 4 varies α with β set equal to α; Table 5 varies β with α fixed at 0.9.Source: §D.2 · p. 25, Figure 17 · p. 25, Table 4 · p. 26

Additional comparison tables

  • The appendix labels Table 7 as an illustrative performance comparison with video-acceleration methods and Table 8 as a runtime and memory comparison.Source: Table 7 · p. 26, Table 8 · p. 27

Segment-wise long-video generation

  • UltraViCo alleviates identity drift in segment-wise long-video generation when Wan2.2-TI2V uses only a few ending frames.Source: Figure 18 · p. 27

Related papers

Loading 2511.20123v2…